Organizing Python projects with shared packages

Question

What is the best way to organize and develop a project composed of many small scripts sharing one (or more) larger Python libraries?

We have a bunch of programs in our repository that all use the same libraries stored in the same repository. So in other words, a layout like

trunk
    libs
        python
        utilities
    projects
        projA
        projB

When the official runs of our programs are done, we want to record what version of the code was used. For our C++ executables, things are simple because as long as the working copy is clean at compile time, everything is fine. (And since we get the version number programmatically, it must be a working copy, not an export.) For Python scripts, things are more complicated.

The problem is that, often one project (e.g. projA) will be running, and projB will need to be updated. This could cause the working copy revision to appear mixed to projA during runtime. (The code takes hours to run, and can be used as inputs for processes that take days to run, hence the strong traceability goal.)

My current workaround is, if necessary, check out another copy of the trunk to a different location, and run off there. But then I need to remember to change my PYTHONPATH to point to the second version of lib/python, not the one in the first tree.

There's not likely to be a perfect answer. But there must be a better way.

Should we be using subversion keywords to store the revision number, which would allow the data user to export files? Should we be using virtualenv? Should we be going more towards a packaging and installation mechanism? Setuptools is the standard, but I've read mixed things about it, and it seems designed for non-developer end users (of which we have none).

Answer 1

The much better solution involves not storing all your projects and their shared dependencies in the same repository.

Use one repository for each project, and externals for the shared libraries.

Make use of tags in the shared library repositories, so consumer projects may use exactly the version they need in their external.

Edit: (just copying this from my comment) use virtualenv if you need to provide isolated runtime environments for the different apps on the same server. Then each environment can contain a unique version of the library it needs.

Answer 2

If I'm understanding your question properly, then you definitely want virtualenv. Add in some virtualenvwrapper goodness to make it that much better.