Bundle a Python app as a single file to support add-ons or extensions?

Question

There are several utilities — all with different procedures, limitations, and target operating systems — for getting a Python package and all of its dependencies and turning them into a single binary program that is easy to ship to customers:

• http://wiki.python.org/moin/Freeze
• http://www.pyinstaller.org/
• http://www.py2exe.org/
• http://svn.pythonmac.org/py2app/py2app/trunk/doc/index.html

My situation goes one step further: third-party developers will be wanting to write plug-ins, extensions, or add-ons for my application. It is, of course, a daunting question how users on platforms like Windows would most easily install plugins or addons in such a way that my app can easily discover that they have been installed. But beyond that basic question is another: how can a third-party developer bundle their extension with whatever libraries the extension itself needs (which might be binary modules, like lxml) in such a way that the plugin's dependencies become available for import at the same time that the plugin becomes available.

How can this be approached? Will my application need its own plug-in area on disk and its own plug-in registry to make this tractable? Or are there general mechanisms, that I could avoid writing myself, that would allow an app that is distributed as a single executable to look around and find plugins that are also installed as single files?

Answer 1

You should be able to have a plugins directory that your application scans at runtime (or later) to import the code in question. Here's an example that should work with regular .py or .pyc code that even works with plugins stored inside zip files (so users could just drop someplugin.zip in the 'plugins' directory and have it magically work):

import re, os, sys
class Plugin(object):
    """
    The base class from which all plugins are derived.  It is used by the
    plugin loading functions to find all the installed plugins.
    """
    def __init__(self, foo):
        self.foo = foo
    # Any useful base plugin methods would go in here.

def get_plugins(plugin_dir):
    """Adds plugins to sys.path and returns them as a list"""

    registered_plugins = []

    #check to see if a plugins directory exists and add any found plugins
    # (even if they're zipped)
    if os.path.exists(plugin_dir):
        plugins = os.listdir(plugin_dir)
        pattern = ".py$"
        for plugin in plugins:
            plugin_path = os.path.join(plugin_dir, plugin)
            if os.path.splitext(plugin)[1] == ".zip":
                sys.path.append(plugin_path)
                (plugin, ext) = os.path.splitext(plugin) # Get rid of the .zip extension
                registered_plugins.append(plugin)
            elif plugin != "__init__.py":
                if re.search(pattern, plugin):
                    (shortname, ext) = os.path.splitext(plugin)
                    registered_plugins.append(shortname)
            if os.path.isdir(plugin_path):
                plugins = os.listdir(plugin_path)
                for plugin in plugins:
                    if plugin != "__init__.py":
                        if re.search(pattern, plugin):
                            (shortname, ext) = os.path.splitext(plugin)
                            sys.path.append(plugin_path)
                            registered_plugins.append(shortname)
    return registered_plugins

def init_plugin_system(cfg):
    """
    Initializes the plugin system by appending all plugins into sys.path and
    then using load_plugins() to import them.

        cfg - A dictionary with two keys:
        plugin_path - path to the plugin directory (e.g. 'plugins')
        plugins - List of plugin names to import (e.g. ['foo', 'bar'])
    """
    if not cfg['plugin_path'] in sys.path:
        sys.path.insert(0, cfg['plugin_path'])
    load_plugins(cfg['plugins'])

def load_plugins(plugins):
    """
    Imports all plugins given a list.
    Note:  Assumes they're all in sys.path.
    """
    for plugin in plugins:
        __import__(plugin, None, None, [''])
        if plugin not in Plugin.__subclasses__():
            # This takes care of importing zipped plugins:
            __import__(plugin, None, None, [plugin])

So lets say I have a plugin named "foo.py" in a directory called 'plugins' (that is in the base dir of my app) that will add a new capability to my application. The contents might look like this:

from plugin_stuff import Plugin

class Foo(Plugin):
    """An example plugin."""
    self.menu_entry = {'Tools': {'Foo': self.bar}}
    def bar(self):
        return "foo plugin!"

I could initialize my plugins when I launch my app like so:

plugin_dir = "%s/plugins" % os.getcwd()
plugin_list = get_plugins(plugin_dir)
init_plugin_system({'plugin_path': plugin_dir, 'plugins': plugin_list})
plugins = find_plugins()
plugin_menu_entries = []
for plugin in plugins:
    print "Enabling plugin: %s" % plugin.__name__
    plugin_menu_entries.append(plugin.menu_entry))
add_menu_entries(plugin_menu_entries) # This is an imaginary function

That should work as long as the plugin is either a .py or .pyc file (assuming it is byte-compiled for the platform in question). It can be standalone file or inside of a directory with an init.py or inside of a zip file with the same rules.

How do I know this works? It is how I implemented plugins in PyCI. PyCI is a web application but there's no reason why this method wouldn't work for a regular ol' GUI. For the example above I chose to use an imaginary add_menu_entries() function in conjunction with a Plugin object variable that could be used to add a plugin's methods to your GUI's menus.

Hopefully this answer will help you build your own plugin system. If you want to see precisely how it is implemented I recommend you download the PyCI source code and look at plugin_utils.py and the Example plugin in the plugins_enabled directory.

Answer 2

Here is another example of a Python app that uses plugins: OpenSTV. Here, the plugins can only be Python modules.