How do I create a namespace package in Python?

Question

In Python, a namespace package allows you to spread Python code among several projects. This is useful when you want to release related libraries as separate downloads. For example, with the directories Package-1 and Package-2 in PYTHONPATH,

Package-1/namespace/__init__.py
Package-1/namespace/module1/__init__.py
Package-2/namespace/__init__.py
Package-2/namespace/module2/__init__.py

the end-user can import namespace.module1 and import namespace.module2.

What's the best way to define a namespace package so more than one Python product can define modules in that namespace?

Answer 1

You have your Python namespace concepts back to front, it is not possible in python to put packages into modules. Packages contain modules not the other way around.

A Python package is simply a folder containing a __init__.py file. A module is any other file in a package (or directly on the PYTHONPATH) that has a .py extension. So in your example you have two packages but no modules defined. If you consider that a package is a file system folder and a module is file then you see why packages contain modules and not the other way around.

So in your example assuming Package-1 and Package-2 are folders on the file system that you have put on the Python path you can have the following:

Package-1/
  namespace/
  __init__.py
  module1.py
Package-2/
  namespace/
  __init__.py
  module2.py

You now have one package namespace with two modules module1 and module2. and unless you have a good reason you should probably put the modules in the folder and have only that on the python path like below:

Package-1/
  namespace/
  __init__.py
  module1.py
  module2.py

Answer 2

There's a standard module, called pkgutil, with which you can 'append' modules to a given namespace.

With the directory structure you've provided:

Package-1/namespace/__init__.py
Package-1/namespace/module1/__init__.py
Package-2/namespace/__init__.py
Package-2/namespace/module2/__init__.py

You should put those two lines in Package-2/namespace/__init__.py:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

From now on, you should be able to distribute those two packages independently.

Answer 3

This section should be pretty self-explainatory.

In short, put that namespace code in _init_.py, update setup.py to declare namespace and you are free to go.

Answer 4

I ran into this same issue trying to run nose on a series of directories like Package-1 and Package-2. My Package-1 namespace is self contained, and represents a base distribution. Package-2 is distributed as a separate egg, adding a subpackage "namespace.extend" that has dependencies on namespace.

I use numerous features (extension entry points, developer installation) of setuptools. This represents the real tradeoff between pkgutil vs pkg_resources for me. Nose's directory-by-directory importer can understand pkgutil with no additional work. On the other hand, it seems like all the work I put in egg-info is lost. I'm not the only one ... the zope discussion referenced above eventually (I think) led to them abandoning pkg_resources because they were invested in setuptools. At least they talked about it.

So I have a fairly advanced question. What happens if namespace is imported using setuptools and namespace.extend is imported by a simple manipulation of sys.path? Can I force the regular import to: 1. understand the pkg_resources stuff and actually create the package namespace and 2. perform whatever setuptools voodoo would normally be done on namespace.extend even though I am importing it using the filesystema and sys.path?

Thanks! Eli