Dynamic loading of uncompiled python plugins in py2exe compile code

Question

My Python application is constructed as such that some functionality is available as plugins. The plugin architecture currently is very simple: I have a plugins folder/package which contains some python modules. I load the relevant plugin as follows:

plugin_name = blablabla
try:
    module = __import__(plugin_name, fromlist='do_something')
except ImportError:
   #some error handling ...

and then execute:

try:
    loans = module.do_something(id_t, pin_t)
except xxx:
    # error handling

I compile the application to a Windows binary using py2exe. This works fine, except for that the fact that all plugins are (and have to be) included in the binary. This is not very practical, since for each new plugin, I have to recompile and release a new version of my application. It would be better if a new plugin (i.e. python file) could be copied to some application plugin folder, and that the Python code in the file code be interpreted on-the-fly by my application.

What is the best approach to do so?

(I've though of reading each line of the selected plugin file, and applying an exec statement to it. But there might be better ways...)

Answer 1

I suggest you use pkg_resources entry_points features (from setuptools/distribute) to implement plugin discovery and instantiation: first, it's a standard way to do that; second, it does not suffer the problem you mention AFAIK. All you have to do to extend the application is to package some plugins into an egg that declare some entry points (an egg may declare many plugins), and when you install that egg into your python distribution, all the plugins it declares can automatically be discovered by your application. You may also package your application and the "factory" plugins into the same egg, it's quite convenient.

Answer 2

I'm not sure you have to put plugin files in the zip library. This may be because you're using the default for py2exe packaging your script.

You could try using compressed = False (as documented in py2exe ListOfOptions ) which would eliminate the library.zip generated by py2exe, and possibly allow you to have access to python modules (your plugins are python modules, I presume, from the import) in a "normal" way, instead of being forced to package them in your zip or binary.

Answer 3

PyInstaller lets you import external files as well. If you run it over your application, it will not package those files within the executable. You will then have to make sure that paths are correct (that is, your application can find the modules on the disk in the correct directory), and everything should work.

Answer 4

If you don't mind that plugin's will be release as .py files, you can do something like the following. Put all your plugin's under a "plugin" subdir, and create an empty "_init_.py". Doing runtime, it will import the package along with all the modules in that directory. Check Dive In Python for explanation... but here's what I finally end up using.

def load_plugin(path):
import imp
"""Load plugin from directory and return list of modules"""
files = os.listdir( path )
test = re.compile(".py$", re.IGNORECASE)          
files = filter(test.search, files)                     
filenameToModuleName = lambda f: os.path.splitext(f)[0]
moduleNames = sorted(map(filenameToModuleName, files))
f, filename, desc = imp.find_module('plugin')
plugin = imp.load_module('plugin', f, filename, desc)
modules = []

#print moduleNames
for m in moduleNames:
    # skip any files starting with '__', such as __init__.py
    if m.startswith('__'):
        continue
    try:
        f, filename, desc = imp.find_module(m, plugin.__path__)
        modules.append( imp.load_module(m, f, filename, desc))
    except ImportError:
        continue
return modules