Putting separate python packages into same namespace?

Question

Hey everyone,

I'm developing a python framework that would have "addons" written as separate packages. I.e.:

import myframework
from myframework.addons import foo, bar

Now, what I'm trying to arrange is so that these addons can be distributed separately from core framework and injected into myframework.addons namespace.

Currently my best solution to this is the following. An add-on would be deployed (most likely into {python_version}/site-packages/ like so:

fooext/
fooext/__init__.py
fooext/myframework/
fooext/myframework/__init__.py
fooext/myframework/addons/
fooext/myframework/addons/__init__.py
fooext/myframework/addons/foo.py

The fooext/myframework/addons/__init__.py would have the pkgutil path extension code:

import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)

The problem is that for this to work, the PYTHONPATH needs to have fooext/ in it, however the only thing it would have is the parent install directory (most likely, the above-mentioned site-packages).

The solution to this is to have extra code in myframework/addons/__init__.py which would tranverse sys.path and look for any modules with a myframework sub-package, in which case it adds it to sys.path and everything works.

Another idea I had is to write the addon files directly to myframework/addons/ install location, but then it would make development and deployed namespace differ.

Is there a better way to accomplish this or perhaps a different approach to the above distribution problem altogether?

Answer 1

It sounds like what you're after can be accomplished quite neatly with import hooks.

This is a way of writing custom loading code, which can be associated with a package (or in your case a framework), to perform the loading of all sub-packages and modules, rather than using python's default loading mechanism. Then you can install the loader in site-packages as a base package or under your framework.

When a package is found to be associated with the loader (which can simply be hard-coded to the relative path if necessary), it will then always use the loader to load all add-ons for example. This has the advantage of not requiring any fiddling of the PYTHONPATH, which is generally worth keeping as short as possible.

The alternative to this is to use the init files to redirect the import call for a sub-module to the one you want it to pick up, but this is a little bit messy.

More information on import hooks can be found here:

http://www.python.org/dev/peps/pep-0302/

Answer 2

Is there a better way to accomplish this or perhaps a different approach to the above distribution problem altogether?

Possibly. Python's module/package setup is generally tricky to tamper with dynamically like this, but its object/class system is open and extensible in a well-defined way. When modules and packages don't quite have the features you need to encapsulate your project nicely you can use classes instead.

For example you could have the extension functionality in a completely different package, but allow it to inject classes into your basic framework through a specific interface. eg. myframework/_​​_​init​_​​_.py containing a basic application wrapper:

class MyFramework(object):
    """A bare MyFramework, I only hold a person's name
    """
    _addons= {}
    @staticmethod
    def addAddon(name, addon):
        MyFramework._addons[name]= addon

    def __init__(self, person):
        self.person= person
        for name, addon in MyFramework._addons.items():
            setattr(self, name, addon(self))

Then you could have extension functionality in a myexts/helloer.py, that keeps a reference to its 'owner' or 'outer' MyFramework class instance:

class Helloer(object):
    def __init__(self, owner):
        self.owner= owner
    def hello(self):
        print 'hello '+self.owner.person

import myframework
myframework.MyFramework.addAddon('helloer', Helloer)

So now if you just "import myframework", you only get the basic functionality. But if you also "import myexts.helloer" you also get the ability to call MyFramework.helloer.hello(). Naturally you can also define protocols for addons to interact with the basic framework behaviour, and each other. You can also do things like inner classes a subclass of the framework can override to customise without having to monkey-patch classes that might affect other applications, if you need that level of complexity.

Encapsulating behaviour like this can be useful, but it's typically annoying work to adapt module-level code you've already got to fit this model.

Answer 3

Setuptools has the ability to lookup package "entry points" (functions, objects, whatever) by name. Trac uses this mechanism to load its plugins, and it works well.

Answer 4

See namespace packages:

http://www.python.org/dev/peps/pep-0382/

or in setuptools:

http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages