Python Plugin-System - HOWTO

Question

Hello,

I'm writing a Python application which stores some data. For storing the data i've wrote a Connection class with abstract methods (using Python's abc module). This class is the super class all storage back-ends derive from. Each storage back-end has only one purpose, e.g. storing the data in plain text files or in a XML file.

All storage backends (inclusive the module where the super class is located) are in one package called 'data_handler'. And each back-end is in one module.

My application should be able the store data in multiple back-ends simultaneously and determinate at runtime which storage back-ends are available. To do this i had the idea to write a singleton class where each back-end have to register at their import. But this seems to be not so good in a dynamic language (please correct me if I misinterpret this). Another way could be the import of the package with import data_handler and then get the __file__ attribute of the package and search all Python files in the dir for subclasses of the super Connection class.

What method should I use, or are there other (maybe better) methods to do this.

Stefan

Answer 1

But should your application always write to all backends? If not, you could use (as usual) another layer of indirection, e.g.

storage = Storage()
storage.use(TextBackend, XMLBackend, YamlBackend)
storage.write(data)

Or something like this, with Storage being a dispatcher, which would just loop over the backends and call the appropriate serializer.

This is of course, very coarse.

Answer 2

Do not walk the filesystem (!) and scan the Python source code of the backends! That's an ugly hack at the best of times, and even worse here because you don't need anything like it at all! Registering all the classes on import is perfectly OK.

---

Store the backends in a class attribute instead of an instance attribute; that way, all Storage instances will look at the same set of backends:

>>> class Storage(object):
...     backends = set()
...
...     def register(self, backend):
...             self.backends.add(backend)
...

Every backend can register itself by instantiating its own Storage, which has access to the class-level backends attribute:

>>> foo = Storage()
>>> foo.register("text")
>>> bar = Storage()
>>> bar.register("xml")

You can read this attribute by instantiating another Storage, which will read the same variable:

>>> baz = Storage()
>>> baz.backends
{'xml', 'text'}

---

You could even store the backend instances in a class attribute of Connection, and register each backend upon instantiation:

>>> class Connection(object,metaclass=abc.ABCMeta):
...     @abc.abstractmethod
...     def register(self, backend):
...             pass
...
...     backends = set()
...
>>> class TextBackend(Connection):
...     def register(self):
...             super().backends.add(self)
...
...     def __init__(self):
...             self.register()
...
>>> class XMLBackend(Connection):
...     def register(self):
...             super().backends.add(self)
...
...     def __init__(self):
...             self.register()
...
>>> foo = TextBackend()
>>> bar = XMLBackend()
>>> Connection.backends
{<__main__.XMLBackend object at 0x027ADAB0>, \
<__main__.TextBackend object at 0x027ADA50>}

Answer 3

If these backends are going to be distributed in various Python distributions, you might want to look at setuptools/distribute entry points. Here's an article on how you might use these for dynamic plugin finding services:

http://aroberge.blogspot.com/2008/12/plugins-part-6-setuptools-based.html

Answer 4

Is discovering the back-ends at runtime a strict requirement or would static enumeration of them in the code do?

This feature will be nice to note have to edit the code when I add a new back-end

---

But should your application always write to all backends?

I will have a class where I can register available handler. And the data shall be written to each registered handler. But not all available handlers have to be registered.

Answer 5

you could use a function like this one:

def loadClass(fullclassname):
    sepindex=fullclassname.rindex('.')    
    classname=fullclassname[sepindex+1:]
    modname=fullclassname[:sepindex]
    #dynmically import the class in the module
    imod=__import__(modname,None,None,classname)
    classtype=getattr(imod,classname)
    return classtype

where fullclassname is the fully dotted qualifiant for the class you want load.

example (pseudo code,but the idea is there):

for package availability scanning, only perform some globbing , then for finding final class name, you may declare a Plugin class in each of your modules that has a getStorage()

 #scan for modules , getPluginPackagesUnder (to be defined) returns the dotted name for all packages under a root path (using some globbing, listdir or whatever method) 

    pluginpackages=getPluginPackagesUnder("x/y/z")
    storagelist=[]
    for pgpck in plunginpackages:
        pluginclass=loadClass("%s.Plugin"%pgpck)
        storageinstance=Plugin().getStorage()
        storagelist.append(storageinstance)

so, you can dynamically scan for your existing storage plugins

Answer 6

Sorry I have to answer in this way because I created the question without having a account hier at stackoverflow.com. Now I have a account but I can't comment because I don't have enough reputations and the system does not track this question as mine (My fault, next time I'll login before)

I'll will try the example from katrielalex, it looks very good.

Thanks for all the answers :)

Stefan