Python Code Organization Question : Eggs + Packages + Buildout + Unit Tests + SVN

Question

I have several python projects that share common modules. Until now, I've been ... ahem ... keeping multiple copies of the common code and synchronizing by hand. But I'd clearly prefer to do something else.

It looks to me now, as if zc.Buildout maybe what I need. I guess that what I should be doing is putting each reusable component of my system into a separate egg, and then using buildout to assemble them into projects.

I'm also thinking that for any particular module, I should put the unit-tests into a separate package or egg, so that I'm not also installing copies of the component's unit-tests in every project. I only want to unit-test in a place where my library is developed, not where it's just being used.

So maybe I want something like this

projects
  lib1
    tests
    code
  lib2
    tests
    code
  app1
    tests 
    appcode
  app2
    tests
    appcode

etc.

Where both app1 and app2 are independent applications with their own code and tests, but are also including and using both lib1 and lib2. And lib1/test, lib1/code, lib2/test, lib2code, app1, app2 are separate eggs. Does this sound right?

However, I now get confused. I assume that when I develop app1, I want buildout to pull copies of lib1, lib2 and app1 into a separate working directory rather than put copies of these libraries under app1 directly. But how does this work with my SVN source-control? If the working directory is dynamically constructed with buildout, it can't be a live SVN directory from which I can check the changes back into the repository?

Have I misunderstood how buildout is meant to be used? Would I be better going for a completely different approach? How do you mix source-control with module-reuse between projects?

Update : thanks to the two people who've currently answered this question. I'm experimenting more with this.

Answer 1

This is why you have the site module. It sets the internal sys.path to include all packages and modules from

• lib/site-packages -- including directories, eggs and .pth files.
• PYTHONPATH

This way there is exactly one working copy of your libraries.

There are an unlimited ways to make use of this. Here are two.

1. In each lib, write a setup.py that deploys your lib properly. When you make changes, you do an svn up to collect the changes and a python setup.py install to deploy the one working copy that every application shares.

2. In each app, either depend on things being in the PYTHONPATH environment variable. Be sure that projects/lib1 and projects/lib2 are won the PYTHONPATH. Each app then shares the one working copy of the various libraries.

Answer 2

I've use the following structure quite effectively. in SVN.

Lib1/
   branches/
   tags/
   trunk/
     lib1/
     tests/
     setup.py
Lib2
   branches/
   tags/
   trunk/
     lib2/
     tests/
     setup.py
App1
   branches/
   tags/
   trunk/
     app1/
     tests/
     setup.py
App2
   branches/
   tags/
   trunk/
     app2/
     tests/
     setup.py

I would then create my dev workspace( I use eclipse/pydev) as follows, checking out from either trunk or a branch.

Lib1/
   lib1/
   tests/
   setup.py
Lib2/
   lib2/
   tests/
   setup.py
App1/
   app1/
   tests/
   setup.py
App2/
   app2/
   tests/
   setup.py

I would then use either eclipse project dependencies setup python path, which works well with eclipse code completion. setup.py also works but does not support having multiple workspaces well.

For deployment, I use create a single zip with the following structure.

App1/
   lib1-1.1.0-py2.5.egg/
   lib2-1.1.0-py2.5.egg/
   app1/
   sitecustomize.py

App2/
   lib1-1.2.0-py2.5.egg/
   lib2-1.2.0-py2.5.egg/
   app2/
   sitecustomize.py

I don't use setup install because I want to support multiple versions of the app, also I have some control of the runtime environment, so I don't package python with my deployment but should be easy to add Python into the deployment package if it's needed.

Answer 3

I'd consider each application and library an egg and use one of the examples already given in terms of laying it out in SVN. Really, the VCS end of things should not be an issue.

Then, for testing each application/library or combination, I'd set up a virtualenv and install each package either via setup.py develop or via actually installing it. Virtualenvwrapper is also a helpful tool to manage these environments as you can simply do things like:

mkvirtualenv lib1-dev

And then later:

workon lib1-dev

Virtualenv uses the PYTHONPATH to gain finer control of the packages installed. Likewise, you can use create environments with:

virtualenv --no-site-packages some-env

And it will leave out any references to your actual system site-packages. This is helpful because not only can you test your library/application, but you can also verify you have the correct dependencies on installation.

Answer 4

Perhaps this might help you out...

http://www.stereoplex.com/two-voices/fez-djangoskel-django-projects-and-apps-as-eggs