What's wrong with this `setup.py`?

Question

I've been having problems withe getting setup.py to do the sdist thing correctly. I boiled it down to this. I have the following directory structure:

my_package\
    my_subpackage\
        __init__.py
        deep_module.py
    __init__.py
    module.py
    setup.py

And here's what I have in setup.py:

#!/usr/bin/env python

from distutils.core import setup
import distutils

setup(
    name='a',
    version='0.1',
    description='a',
    author='a',
    author_email='[email protected]',
    url='http://a.org',
    packages=['my_package','my_package.my_subpackage'],
    package_dir={'': '..'},
    license= "a",
    long_description = 'aaa',

)

(The 'aaa' stuff is just placeholder.)

Anyway, it works okay when I do setup.py install, but when I try to do setup.py sdist, a few curious things happen:

1. A MANIFEST file is created.

2. A copy of the my_package folder is created inside the existing my_package folder (though it misses a few of the setup-related files I think.)

3. A dist folder is created, inside it a zipfile, inside that a folder with the package name, but inside that folder there isn't the whole package like I hoped but only two files, setup.py and PKG-INFO.

What am I doing wrong? How do I make sdist work?

Answer 1

Instead of this:

my_package\
    my_subpackage\
        __init__.py
        deep_module.py
    __init__.py
    module.py
    setup.py

Try this:

my_package_source\
    setup.py
    README.txt
    my_package\
        my_subpackage\
            __init__.py
            deep_module.py
        __init__.py
        module.py

You don't actually need a README, it's just for illustrative purpose for what kind of things sit in the root directory of your project's folder.

=== EDIT ======================================

I should elaborate. After you run it your directory should then look something like this:

my_package_source\
    setup.py
    README.txt
    MANIFEST
    PKG-INFO
    dist\
        my_package_0.X.tar.gz (or .zip on windows I believe)
    my_package\
        my_subpackage\
            __init__.py
            deep_module.py
        __init__.py
        module.py

Use the package under the dist directory to distribute.

Answer 2

The problem is well explained here:

Setuptools has many silent failure modes. One of them is failure to include all files in sdist release (well not exactly a failure, you could RTFM, but the default behavior is unexpected). This post will serve as a google-yourself-answer for this problem, until we get new, shinier, Distribute solving all of our problems.

As comments point out, the bug (misdesign) is actually in distutils -- setuptools just fails to fix it (if you're using svn, things are actually a bit better).

I can reproduce your problem as you observe it, i.e., shortening file names a bit, I have:

$ ls -lR
total 8
-rw-r--r--  1 aleax  eng    0 Oct 24 11:25 __init__.py
-rw-r--r--  1 aleax  eng    0 Oct 24 11:25 modu.py
drwxr-xr-x  4 aleax  eng  136 Oct 24 11:25 mysub
-rw-r--r--  1 aleax  eng  323 Oct 24 11:26 setup.py

./mysub:
total 0
-rw-r--r--  1 aleax  eng  0 Oct 24 11:25 __init__.py
-rw-r--r--  1 aleax  eng  0 Oct 24 11:25 deepmod.py

and running python setup.py sdist produces (as well as warnings):

$ ls -lR
total 16
-rw-r--r--  1 aleax  eng  104 Oct 24 11:35 MANIFEST
-rw-r--r--  2 aleax  eng    0 Oct 24 11:25 __init__.py
drwxr-xr-x  3 aleax  eng  102 Oct 24 11:35 dist
-rw-r--r--  2 aleax  eng    0 Oct 24 11:25 modu.py
drwxr-xr-x  5 aleax  eng  170 Oct 24 11:35 mypack
drwxr-xr-x  4 aleax  eng  136 Oct 24 11:25 mysub
-rw-r--r--  1 aleax  eng  323 Oct 24 11:26 setup.py

./dist:
total 8
-rw-r--r--  1 aleax  eng  483 Oct 24 11:35 a-0.1.tar.gz

./mypack:
total 0
-rw-r--r--  2 aleax  eng    0 Oct 24 11:25 __init__.py
-rw-r--r--  2 aleax  eng    0 Oct 24 11:25 modu.py
drwxr-xr-x  4 aleax  eng  136 Oct 24 11:35 mysub

./mypack/mysub:
total 0
-rw-r--r--  2 aleax  eng  0 Oct 24 11:25 __init__.py
-rw-r--r--  2 aleax  eng  0 Oct 24 11:25 deepmod.py

./mysub:
total 0
-rw-r--r--  2 aleax  eng  0 Oct 24 11:25 __init__.py
-rw-r--r--  2 aleax  eng  0 Oct 24 11:25 deepmod.py

One solution is to change the directory layout as follows (from the current mypack dir):

$ mkdir mypack
$ mv __init__.py modu.py mysub/ mypack
$ touch README.txt

so getting:

$ ls -lR
total 8
-rw-r--r--  1 aleax  eng    0 Oct 24 11:37 README.txt
drwxr-xr-x  5 aleax  eng  170 Oct 24 11:37 mypack
-rw-r--r--  1 aleax  eng  323 Oct 24 11:26 setup.py

./mypack:
total 0
-rw-r--r--  1 aleax  eng    0 Oct 24 11:25 __init__.py
-rw-r--r--  1 aleax  eng    0 Oct 24 11:25 modu.py
drwxr-xr-x  4 aleax  eng  136 Oct 24 11:25 mysub

./mypack/mysub:
total 0
-rw-r--r--  1 aleax  eng  0 Oct 24 11:25 __init__.py
-rw-r--r--  1 aleax  eng  0 Oct 24 11:25 deepmod.py

(and getting rid of one of the warnings, the one about README -- the one about missing MANIFEST.in clearly remains;-). Also change one line of setup.py to:

package_dir={'': '.'},

Now, after python setup.py sdist, you do get a decent tarball:

$ tar tvf dist/a-0.1.tar.gz 
drwxr-xr-x aleax/eng         0 2009-10-24 11:40:05 a-0.1/
drwxr-xr-x aleax/eng         0 2009-10-24 11:40:05 a-0.1/mypack/
-rw-r--r-- aleax/eng         0 2009-10-24 11:25:30 a-0.1/mypack/__init__.py
-rw-r--r-- aleax/eng         0 2009-10-24 11:25:30 a-0.1/mypack/modu.py
drwxr-xr-x aleax/eng         0 2009-10-24 11:40:05 a-0.1/mypack/mysub/
-rw-r--r-- aleax/eng         0 2009-10-24 11:25:30 a-0.1/mypack/mysub/__init__.py
-rw-r--r-- aleax/eng         0 2009-10-24 11:25:30 a-0.1/mypack/mysub/deepmod.py
-rw-r--r-- aleax/eng       156 2009-10-24 11:40:05 a-0.1/PKG-INFO
-rw-r--r-- aleax/eng         0 2009-10-24 11:37:41 a-0.1/README.txt
-rw-r--r-- aleax/eng       322 2009-10-24 11:39:46 a-0.1/setup.py

the MANIFEST file is still created in your current directory of course, but I hope that's not a problem.