Pylons app deployment with privately developed dependencies

Question

In my organization, we have a couple of internally developed Python packages. For sake of example, let's call them Foo and Bar. Both are developed in separate Git repositories. Foo is a Pylons application that uses certain library functions from Bar. Neither is publicly distributed.

When we deploy Foo, we typically export the latest revision from source control and run setup.py develop within our virtualenv. This works okay.

The problem is that we'll need some way of distributing Bar for every environment where we deploy Foo. We obviously can't put 'Bar' in setup.py's install_requires (as easy_install won't find be able to find it on any website). I can't find any way of automatically obtaining/installing privately developed dependencies.

Is there an easier to way to manage this? I feel like I'm missing the point of Python packaging and distribution.

Answer 1

You can create a package repository. The steps are basically:

1. Create an egg with setup.py bdist_egg
2. Copy the created egg from dist to a directory served by Apache
3. Add the url to the directory exposed by Apache to the easy_install command with the -f switch

Note that Apache is not necessarily required, but it automatically generates a directory listing that easy_install can deal with.

If you are using buildout, there are config options to do the same thing as -f and I am pretty sure there is something you can use in pip as well.

Answer 2

At my work we use setuptools to create packages specific to the OS. We happen to use RedHat so we call bdist_rpm to create rpm package. We find that works better than eggs because we can do dependency management in the packages for both python and non-python libs.

We create the rpms on our continuous integration machine and the move them to a YUM repo where they can be pushed out via a YUM update or upgrade.

Answer 3

When using setuptools, in setup.py you can specify HTTP, FTP and SVN locations where easy_install should look for packages:

http://peak.telecommunity.com/DevCenter/setuptools#dependencies-that-aren-t-in-pypi

You can either publish Bar in some "secret" location, or, I haven't tried it but maybe HTTP basic auth works:

setup(
    ...
    dependency_links = [
        "http://user:[email protected]/private-repository/"
    ],
)