Best practice for installing python modules from an arbitrary VCS repository

Question

I'm newish to the python ecosystem, and have a question about module editing.

I use a bunch of third-party modules, distributed on PyPi. Coming from a C and Java background, I love the ease of easy_install <whatever>. This is a new, wonderful world, but the model breaks down when I want to edit the newly installed module for two reasons:

1. The egg files may be stored in a folder or archive somewhere crazy on the file system.
2. Using an egg seems to preclude using the version control system of the originating project, just as using a debian package precludes development from an originating VCS repository.

What is the best practice for installing modules from an arbitrary VCS repository? I want to be able to continue to import foomodule in other scripts. And if I modify the module's source code, will I need to perform any additional commands?

Answer 1

You can use the PYTHONPATH environment variable or symlink your code to somewhere in site-packages.

Answer 2

Packages installed by easy_install tend to come from snapshots of the developer's version control, generally made when the developer releases an official version. You're therefore going to have to choose between convenient automatic downloads via easy_install and up-to-the-minute code updates via version control. If you pick the latter, you can build and install most packages seen in the python package index directly from a version control checkout by running python setup.py install.

If you don't like the default installation directory, you can install to a custom location instead, and export a PYTHONPATH environment variable whose value is the path of the installed package's parent folder.

Answer 3

Pip lets you install files gives a URL to the Subversion, git, Mercurial or bzr repository.

pip install -e svn+http://path_to_some_svn/repo#egg=package_name

Example: pip install -e hg+https://[email protected]/ianb/cmdutils#egg=cmdutils

If I wanted to download the latest version of cmdutils. (Random package I decided to pull).

I installed this into a virtualenv (using the -E parameter), and pip installed cmdutls into a src folder at the top level of my virtualenv folder.

pip install -E thisIsATest -e hg+https://[email protected]/ianb/cmdutils#egg=cmdutils

$  ls thisIsATest/src
cmdutils

Answer 4

Are you wanting to do development but have the developed version be handled as an egg by the system (for instance to get entry-points)? If so then you should check out the source and use Development Mode by doing:

python setup.py develop

If the project happens to not be a setuptools based project, which is required for the above, a quick work-around is this command:

python -c "import setuptools; execfile('setup.py')" develop

Almost everything you ever wanted to know about setuptools (the basis of easy_install) is available from the the setuptools docs. Also there are docs for easy_install.

Development mode adds the project to your import path in the same way that easy_install does. An changes you make will be available to your apps the next time they import the module.

As others mentioned, you can also directly use version control URLs if you just want to get the latest version as it is now without the ability to edit, but that will only take a snapshot, and indeed creates a normal egg as part of the process. I know for sure it does Subversion and I thought it did others but I can't find the docs on that.