Python distutils builds extensions differently on different machines.

Question

I have been working on a Python extension module with lots of files. While building on one machine, python setup.py build will happily detect changed files, build just those files, and link the whole thing together, just like make. On another machine, however, a single change to any file triggers a recompile of all sources.

Just to be clear. Both machines detect when the package is up to date and won't do anything. It is only when a single file changes that their behavior diverges.

Why is the second machine doing this?

Machine 1 (Does proper per-file dependency check and build.)

Python 2.6.4 (r264:75706, Feb 15 2010, 17:06:03) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

setuptools-0.6c11-py2.6

LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: CentOS
Description: CentOS release 5.4 (Final)
Release: 5.4
Codename: Final

Machine 2 (Rebuilds everything when a single source file changes.)

Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.

setuptools-0.6c11-py2.6

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 10.04 LTS
Release: 10.04
Codename: lucid

Answer 1

I've tracked this down to a change in distutils between Python 2.6.4 and Python 2.6.5. Two methods in distutils.ccompiler.CCompiler, namely, _setup_compile and _prep_compile, have had the same chunk of code removed:

if self.force:
    skip_source = {}            # rebuild everything
    for source in sources:
        skip_source[source] = 0
elif depends is None:
    # If depends is None, figure out which source files we
    # have to recompile according to a simplistic check. We
    # just compare the source and object file, no deep
    # dependency checking involving header files.
    skip_source = {}            # rebuild everything
    for source in sources:      # no wait, rebuild nothing
        skip_source[source] = 1

    n_sources, n_objects = newer_pairwise(sources, objects)
    for source in n_sources:    # no really, only rebuild what's
        skip_source[source] = 0 # out-of-date
else:
    # If depends is a list of files, then do a different
    # simplistic check.  Assume that each object depends on
    # its source and all files in the depends list.
    skip_source = {}
    # L contains all the depends plus a spot at the end for a
    # particular source file
    L = depends[:] + [None]
    for i in range(len(objects)):
        source = sources[i]
        L[-1] = source
        if newer_group(L, objects[i]):
            skip_source[source] = 0
        else:
            skip_source[source] = 1

This code checks each source file against its target object, and marks it to be skipped if it is older than the target. I don't know why it was removed, but it explains the discrepancy. When I put it back in as a test, the compiler reverts to per-source dependency analysis.