How do you organize Python modules?

Question

When it comes to organizing python modules, my Mac OS X system is a mess. I've packages lying around everywhere on my hdd and no particular system to organize them.

How do you keep everything manageable?

Answer 1

Maybe PEP8 and easy_install can help you?

Answer 2

In addition to PEP8 and easy_install, you should check out virtualenv. Virtualenv allows you to have multiple different python library trees. At work, we use virtualenv with a bootstrapping environment to quickly set up a development/production environment where we are all in sync w.r.t library versions etc. We generally coordinate library upgrades.

Answer 3

The "Modules" Python documentation page is a useful guide on organising code, specifically the "packages" sections

Answer 4

I keep all the source for my packages inside ~/Packages/ , and then I do a standard install with "python2.5 setup.py install" on them. This tosses into (for me) /Library/Frameworks/Python/Versions/current/lib/python2.5/site-packages/ . For the development of my own software, I have aliases set up to switch between trunk/ branches/1.0, etc, by pre-prending onto PYTHONPATH. (I have to run 'setup.py build_ext --inplace' in each of these directories before they will import properly.)

It's worth noting that Python2.6 has a per-user site-packages directory, which you may find more convenient.

Answer 5

My advice is to try to put everything into your site-packages directory(ies) unless you have a good reason not to. And I try to avoid easy_install because I find that it tends to cruft up my sys.path with egg locations, but that's just me. Some people find it useful.

If you have lots of programs that use different libraries that may conflict with each other, you may also want to check out virtualenv.

Answer 6

There are several families of Python componentry.

1. The stuff that comes with Python. This takes care of itself.

2. The stuff that you got with easy_install. This, also, takes care of itself.

3. The packages that you had to get some other way, either as TARballs or SVN checkouts. Create a Components folder. Put the downloads or the SVN's in there first. Every Single Time. Do installs from there.

4. The packages that you wrote that are reusable. I have a Projects folder with each project in that folder. If the project is a highly reusable thing, it has a setup.py and I actually run the install as if I downloaded it. I don't have many of these, but a few. Some of them might become open source projects.

5. The final applications you write. I have a folder in Projects with each of these top-level applications. These are usually big, rambling things (like Django sites) and don't have setup.py. Why? They're often pretty complex with only a few server installations to manage, and each of those server installations is unique. These generally rely on PYTHONPATH to identify their parts.

Notice the common theme. Either they're Components you downloaded or they're Projects you're working on.

Also, I keep this separate (to an extent) from the client. I have a master directory of Client folders, each of which has Projects and each project has Sales and Delivery. Not all projects have both sales and delivery.

Answer 7

My advice:

• Read Installing Python Modules.
• Read Distributing Python Modules.
• Start using easy_install from setuptools. Read the documentation for setuptools.
• Always use virtualenv. My site-packages directory contains setuptools and virtualenv only.
• Check out Ian Bicking's new project pyinstall.
• Follow everything Ian Bicking is working on. It is always goodness.
• When creating your own packages, use distutils/setuptools. Consider using paster create (see http://pythonpaste.org) to create your initial directory layout.