views:

139

answers:

2

I have a Python module that I would like to upload to PyPI. So far, it is working for Python 2.x. It shouldn't be too hard to write a version for 3.x now.

But, after following guidelines for making modules in these places:

it's not clear to me how to support multiple source distributions for different versions of Python, and it's not clear if/how PyPI could support it. I envisage I would have separate code for:

  • 2.x
  • 2.6 (maybe, as a special case to use the new buffer API)
  • 3.x

How is it possible to set up a Python module in PyPI so that someone can do:

easy_install modulename

and it will install the right thing whether the user is using 2.x or 3.x?

+1  A: 

The simplest solution is to use a single source distribution.

J.F. Sebastian
I could make a single source distribution, with directories e.g. `python2` and `python3`, each containing their own `setup.py` and the module code for that Python version. Is that what you mean?
Craig McQueen
That solution would seem to conflict with the advice "You should always run the setup command from the distribution root directory" in http://docs.python.org/3.1/install/index.html#platform-variations
Craig McQueen
J.F. Sebastian
I've had a look at a few: `pyserial` uses `bdist_py2to3`; `blist` manages to write code that can be interpreted by 2.x or 3.x; `bitstring` is hacky; `crcmod` has separate 3.x code in a `py3` sub-dir (along with a separate `setup.py`. Nothing so far that looks promising for my situation.
Craig McQueen
I came across the `setup.py` for `httplib2` (http://code.google.com/p/httplib2/source/browse/setup.py) and that does something elegant with the `package_dir` parameter to `setup()`. That's looking more promising, and what I think I should do.
Craig McQueen
@Craig McQueen: Separate modules for 2.x and 3.x doubles maintenance work. It depends on how many workarounds your code required to stay with a single source for 2.x and 3.x whether it is worth it.
J.F. Sebastian
+2  A: 

I found that setup.py for httplib2 seems to have an elegant way to support Python 2.x and 3.x. So I decided to copy that method.

The task is to craft a single setup.py for the package distribution that works with all the supported Python distributions. Then with the same setup.py, you can do:

python2 setup.py install

as well as

python3 setup.py install

It should be possible to keep setup.py simple enough to be parsed with all the supported Python distributions. I've successfully done so with a package cobs that supports 2.4 through 2.6 as well as 3.1. That package includes pure Python code (separate code for Python 2.x and 3.x) and C extensions, written separately for 2.x and 3.x.

To do it:

1) I put the Python 2.x code into a python2 subdirectory, and Python 3.x code in a python3 subdirectory.

2) I put the C extension code for 2.x and 3.x in a src directory under python2 and python3.

So, the directory structure is:

root
  |
  +--python2
  |     |
  |     +--src
  |
  +--python3
  |     |
  |     +--src
  |
  +--setup.py
  +--MANIFEST.in

3) In the setup.py, I had these lines near the top:

if sys.version_info[0] == 2:
    base_dir = 'python2'
elif sys.version_info[0] == 3:
    base_dir = 'python3'

4) In the call to setup, I specified the packages as normal:

setup(
    ...
    packages=[ 'cobs', 'cobs.cobs', 'cobs.cobsr', ],

5) I specified the base directory for the Python code using a package_dir option (refer to step 3 for base_dir):

    package_dir={
        'cobs' : base_dir + '/cobs',
    },

6) For the C extensions, I gave the path:

    ext_modules=[
        Extension('cobs.cobs._cobs_ext', [ base_dir + '/src/_cobs_ext.c', ]),
        Extension('cobs.cobsr._cobsr_ext', [ base_dir + '/src/_cobsr_ext.c', ]),
    ],

That was about it for setup.py. The setup.py file is parsable by both Python 2.x and 3.x.

7) Finally, if you build a source distribution using:

python2 setup.py sdist

then it will by default pull in only the files that are specifically needed to build for that Python. E.g. in the above case, you would only get the files under python2 in the source distribution, but not those under python3. But for a complete source distribution, you want to include the files for both 2.x and 3.x. To do that, create a MANIFEST.in file that contains something like this:

include *.txt
recursive-include python2 *
recursive-include python3 *

To see what I did, see the cobs source code on PyPI or BitBucket.

Craig McQueen