Python module search path problem

Question

Hi, I am trying to work on a dev environment but am find problems in that python seems to be using modules from the site-packages directory. I want it to be using the modules from my dev directory.

sys.path returns a bunch of dirs, like this

['', '/usr/lib/python26.zip', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/usr/lib/python2.6/site-packages' etc

This is good, it's using the current directory as the first place of lookup (at least this is how I understand it to be).

Ok now if I create say a file called command.py in the current directory, things work as I would expect them.

>>> import commands
>>> commands.__file__
   'commands.pyc'

I then exit out of the python shell, and start another one. I then do this.

>>> import foo.bar.commands

Now, what I'm expecting it to do is go down from the current directory to ./foo/bar/ and get me the commands module from there. What I get though is this

>>> foo.bar.commands.__file__
    '/usr/lib/python2.6/site-packages/foo/bar/commands.pyc'

Even though from my current directory there is a ./foo/bar/commands.py

Using imp.find_module() and imp.load_module() I can load the local module properly. Whats actually interesting (although I don't really know what it means) is the last line that is printed out in this sequence

>>> import foo.bar.commands
>>> foo.bar.commands.__file__
   '/usr/lib/python2.6/site-packages/foo/bar/commands.pyc'
>>> foo.bar.__file__
   '/usr/lib/python2.6/site-packages/foo/bar/__int__.pyc'
>>> foo.__file__
    './foo/__init__.pyc'

So if it can find the foo/init.pyc in the local dir why can't it find the other files in the local dir?

Cheers

Answer 1

You mention that there's a foo directory under your current directory, but you don't tell us whether foo/__init__.py exists (even possibly empty): if it doesn't, this tells Python that foo is not a package. Similarly for foo/bar/__init__.py -- if that file doesn't exist, even if foo/__init__.py does, then foo.bar is not a package.

You can play around a little by placing .pth files and/or setting __path__ explicitly in your packages, but the basic, simple rule is to just place an __init__.py in every directory that you want Python to recognize as a package. The contents of that file are "the body" of the package itself, so if you import foo and foo is a directory with a foo/__init__.py file, then that's what you're importing (in any case, the package's body executes the first time you import anything from the package or any subpackage thereof).

If that is not the problem, it looks like some other import (or explicit sys.path manipulation) may be messing you up. Running python with a -v flag makes imports highly visible, which can help. Another good technique is to place an

import pdb; pdb.set_trace()

just before the import that you think is misbehaving, and examining sys.path, sys.modules (and possibly other advanced structures such as import hooks) at that point - is there a sys.modules['foo'] already defined, for example? Interactively trying the functions from standard library module imp that locate modules on your behalf given a path may also prove instructive.

Answer 2

What is foo doing in /usr/lib/python2.6/site-packages? It sounds like you have created foo in your local directory but that is not necessarily the one you are importing.

Try getting rid of the foo/bar in site-packages

Make sure your directory structure looks like this

/foo/__init__.py  
    /bar/__init__.py
        /commands.py

Also, it is a good idea to not reuse python standard library names for your own modules -- can you call your commands.py something else?