Index ¦ Archives ¦ RSS > Tag: apsw

APSW 3.37 is the last with Python 2 / early Python 3 support

This release of APSW (my Python wrapper for SQLite's C interface) is the last that will support Python 2, and earlier versions of Python 3 (before 3.7).

If you currently use APSW with Python 2/early Python 3, then you will want to pin the APSW version to 3.37. You will still be able to use this version of APSW with future versions of SQLite (supported till 2050). But new C level APIs won't be covered. The last C level API additions were 3.36 in June 2021 adding serialization and 3.37 in December 2021 adding autovacuum control

What does APSW support now ...

APSW supports every Python version 2.3 onwards (released 2003). It doesn't support earlier versions as there was no GIL API (needed for multi-threading support).

The downloads for the prebuilt Windows binary gives an idea of just how many Python versions that is (15). (Python 3.0 does actually work, but is missing a module used by the test suite.)

Many Python versions supported ...

Each release does involve building and testing all the combinations of 15 Python versions, 32 and 64 bit environment, and both UCS2 and UCS4 Unicode size for Python < 3.3, on multiple operating systems.

There are ~13k lines of C code making up APSW, with ~7k lines of Python code making up the test suite. It is that test suite that gives the confidence that all is working as intended.

... and why?

I wanted to make sure that APSW is the kind of module I would want to use. The most frustrating thing as a developer is that you want to change one thing (eg one library) and then find that forces you to change the versions of other components, or worse the runtime and dev tools (eg compiler).

I never made the guarantee, but it turned out to be:

You can change the APSW (and SQLite) versions, and nothing else. No other changes will be required and everything will continue to work, probably better.

This would apply to any project otherwise untouched since 2004!

There are two simple reasons:

  • Because I could - I do software development for a living, and not breaking things is a good idea (usually)
  • I would have to delete code that works

What happens next?

I am going to delete code that works, but it is mainly in blocks saying doing one thing for Python 2, another for early Python 3, and another for current Python 3.

My plan is to incrementally remove Python 2/early 3 code from the Python test suite and the C code base together, while updating documentation (only Python 3 types need to be mentioned). The test suite and coverage testing will hopefully catch any problems early.

I will be happy that the code base, testing, documentation, and tooling will all become smaller. That makes things less complex.

Other thoughts

The hardest part of porting APSW from Python 2 to 3 was the test suite which had to remain valid to both environments. For example it is easy to create invalid Unicode strings in Python 2 which I had to make sure the test suite checked.

It was about 10 times the amount of work making the Python test suite code changes, vs the C level API work. Python 3 wasn't that much different in terms of the C API (just some renaming and unification of int and long etc).

Category: misc – Tags: apsw, python


Moving to Github

I have two active open source projects, and have been hosting them at Google Code. Now I'm moving them to Github. I'm still tidying up odds and ends.

Google Code (old home) Github (new home)
https://code.google.com/p/apsw/ https://github.com/rogerbinns/apsw
https://code.google.com/p/java-mini-python/ https://github.com/rogerbinns/jia-mini-python

So let's start with some of the good things about Google Code:

  • Mercurial: I prefer Mercurial as my DVCS of choice, and it is well supported.

  • Decent Web Interface: Because code hosting sites are built by developers they tend to have idiosyncratic usability (architecture astronaut style), with Google Code being the least worst. For example on Github you can't even sort issues by priority. It is completely absurd.

  • Multiple repositories per project: The popular hosting services have an issue tracker, wiki, links etc per project. With the exception of Google Code, you only get one source repository per project. DVCS encourages a style of single purpose smaller repositories versus the large agglomeration that was more typical in the subversion and CVS days.

    It is far more pleasant being able to use a single issue tracker, wiki etc for a group of related source repositories. Only Google Code does this.

So why leave?

  • Downloads terminated: I have 14 files per release of APSW (most are Windows binaries since Windows developers rarely have compilers and related installed) and one for the other project. Google no longer want downloads, yet that is the primary way others use my projects. Google also do not believe in supporting Linux for Drive, not to mention that it is a terrible alternative.
  • No future: There is no indication of any ongoing interest in Google Code and they refuse to take money for it. They keep shutting down services, and it is good practise to not be dependent on them. (No customer service, erroneous disabling of accounts etc.)

There are three choices for where to go:

  • Sourceforge: No chance
  • Bitbucket: I used them in the past, but they couldn't support personal and corporate use at the same time. They supported Mercurial, but Atlassian means an enterprisey design (I hate using Jira). Most importantly when I asked if they would guarantee availability of a download service for any period of time they said they would not. So they are out.
  • Github: There are many things I don't like about Github, but at the end of the day they are the least worst, and only practical alternative.

This is how I converted my projects to github:

  • Use fast-export to convert the Mercurial history to Git. (In theory you can talk Mercurial to Github but it isn't worth it. Also THG lost the ability to do line by line commits.)

  • Use github pages to generate documentation into. However do not follow all those instructions - in particular the gh-pages branch needs to be created as an orphan as it has nothing to do with the master branch and its files. (Better instructions).

    The pages didn't render correctly, which was because they returned 404 for stylesheets and images. You have to do some magic to fix it.

  • Use Google Code Issues Migrator to copy all existing issues over to Github and keep the same ids.

  • Grep the source tree for all mentions of `code.google` and fix them.

  • Edit the projects at Google Code to say they have moved

Before doing the real projects I experimented on a test repository to make sure that the documentation and releases worked. Sadly releases have to be done manually because there don't appear to be any usable command line clients (not even github's one) and none of the libraries I looked at did releases either.

Category: misc – Tags: github, apsw, google

Contact me