Creating multiple Python modules in different directories that share a portion of the package structure.

Question

I'm working on a Django project that contains a single application. The application will be released under the GPL so I want to develop it separately from the project - a personal site using the app. I'm attempting to use a package structure based on my domain name for both the project and the app, and that's where I'm running into problems.

Here's my file structure (with __init__.py files where appropriate):

$HOME/django-sites/mydomain
$HOME/django-apps/mydomain/cms

And my PYTHONPATH:

$HOME/django-sites:$HOME/django-apps

If I fire up a Python interpreter (from any directory on the filesystem) I can import classes from the site, but not the application. If I reverse the order of the two entries in the PYTHONPATH (apps first, then sites) I can import from the app but not the site.

It looks like Python's only attempting to import from the first entry in the PYTHONPATH that contains the first portion of the package name. Is that correct? Is this expected behavior? If so, I can only stick modules in package structures like domain/app1, domain/app2 if they live in the same directory structure - regardless of the PYTHONPATH.

It's not show-stopper because I can rename the site, but it's much different than I was expecting. The Python tutorial mentions __path__, but I have no idea how to use it:

Packages support one more special attribute, __path__. This is initialized to be a list containing the name of the directory holding the package’s __init__.py before the code in that file is executed. This variable can be modified; doing so affects future searches for modules and subpackages contained in the package.

While this feature is not often needed, it can be used to extend the set of modules found in a package.

Has anyone else come across this? Is there something I can do with __path__ to make this function as expected?

Answer 1

Your analysis seems about right. The python path is used to locate modules to import; python imports the first one it finds. I'm not sure what you could do to work around this other than name your modules different things or put them in the same location in the search path.

Answer 2

You basically have two modules named the same thing (mydomain). Why not set up your PYTHONPATH like so?

$HOME/django-sites:$HOME/django-apps/mydomain

It would avoid your import problems.

Answer 3

Here's how __path__ in a package's __init__.py is intended to be used:

$ export PYTHONPATH=$HOME/django-sites
$ ls -d $HOME/django*
django-apps/  django-sites/ 
$ cat /tmp/django-sites/mydomain/__init__.py
import os

_components = __path__[0].split(os.path.sep)
if _components[-2] == 'django-sites':
  _components[-2] = 'django-apps'
  __path__.append(os.path.sep.join(_components))

$ python -c'import mydomain; import mydomain.foo'
foo here /tmp/django-apps/mydomain/foo.pyc
$

as you see, this makes the contents of django-apps/mydomain part of the mydomain package (whose __init__.py resides under django-sites/mydomain) -- you don't need django-apps on sys.path for this purpose, either, if you use this approach.

Whether this is a better arrangement than the one @defrex's answer suggests may be moot, but since a package's enriching its __path__ is a pretty important bit of a python programmer's bag of tools, I thought I had better illustrate it anyway;-).