Configuring extension modules with distutils/setuptools

Question

I have a Python project with mutiple extension modules written in C, which talk to a third-party library. However, depending on the user's environment and options some modules should not be built, and some compiler flags should be enabled/disabled. The problem is that I have to build the list of extension modules before I call setup(), and ideally I'd like to use a distutils.Command subclass to handle the user options. Right now I have a few options:

1. Require a "python setup.py configure" command be run before building the modules, store the information in a pickle file, and use it to generate the extensions list next time the script runs. This is how my project currently works, which seems quite silly.

2. Manually scrape options out of sys.argv and use them to build the list. This is not a long-term solution because I will eventually want to run some scripts to check the settings before building.

3. Subclass build_ext from distutils, do my configuration in the beginning of the run() method (possibly also using options sent via (2)) and directly modify self.distribution.ext_modules before building. I'm afraid this may confuse setuptools, however, as it may assume the list of extension modules is fixed when setup() is called. It also means that when setup() is called with a command other than build_ext the list of extension modules is empty.

Is there a preferred way to do this?

Answer 1

My own experience with changing distutils has been weak and shaky, so all I can offer are pointers. Take a look at numpy. That has an entire submodule (numpy.distutils) with ways to work with (or work around) distutils. Otherwise, ask the distutils mailing list.

Answer 2

I would subclass distutils.core.Distribution and pass it with distutils.core.setup(distclass=CustomDistribution) - this gives you access to the command-line parameters in the same way that the normal setup has them, and you can do things like adjusting the extensions list in the CustomDistribution.__init__ method. But I agree with dalke, the way of distutils is full of pain...

Answer 3

Is there a preferred way to do this?

From my experience working with other people's modules, I can say there is certainly not consensus on the right way to do this.

I have tried and rejected subclassing bits of distutils -- I found it fragile and difficult to get working across different Python versions and different systems.

For our code, after trying the types of things you are considering, I have settled on doing detection and configuration right in setup.py before the main call to setup(). This is admittedly a bit ugly, but it means that someone trying compile your stuff has one place to figure out e.g. why the include path is wrong. (And they certainly don't need to be experts on distutils internals).