Python accessing modules from package that is distributed over different directories

Question

Hi,

I have a question regarding one single module that is distributed over multiple directories.

Let's say I have these two file and directories:

~/lib/python
   xxx
      __init__.py
      util
         __init__.py
         module1.py
         module2.py
~/graphics/python
   xxx
      __init__.py
      misc
         __init__.py
         module3.py
         module4.py

So then in my Python modules, I did this:

import sys
pythonlibpath = '~/lib/python'
if pythonlibpath not in sys.path: sys.path.append(pythonlibpath)
import xxx.util.module1

which works.

Now, the problem is that I need xxx.misc.module3, so I did this:

import sys
graphicslibpath = '~/graphics/python'
if graphicslibpath not in sys.path: sys.path.append(graphicslibpath)
import xxx.misc.module3

but I get this error:

ImportError: No module named misc.module3

It seems like it somehow still remembers that there was a xxx package in ~/lib/python and then tries to find misc.module3 from there.

How do I get around this issue?

Answer 1

You can't without an extreme amount of trickery that pulls one package structure into the other. Python requires that all modules in a package be under a single subdirectory. See the os source to learn how it handles os.path.

Answer 2

Python does indeed remember that there was a xxx package. This is pretty much necessary to achieve acceptable performance, once modules and packages are loaded they are cached. You can see which modules are loaded by looking the the dictionary sys.modules.

sys.modules is a normal dictionary so you can remove a package from it to force it to be reloaded like below:

import sys
print sys.modules
import xml
print sys.modules
del sys.modules['xml']
print sys.modules

Notice that after importing the xml package it is the dictionary, however it is possible to remove it from that dictionary too. This is a point I make for pedagogical purposes only, I would not recommend this approach in a real application. Also if you need to use your misc and util packages together this would not work so great. If at all possible rearrange your source code structure to better fit the normal Python module loading mechanism.