Dynamic loading of python modules

Question

In python how do you dynamically add modules to a package while your programming is running.

I want to be able to add modules to the package directory from an outside process, and be able to use those new modules in my program:

import package

def doSomething(name):
    pkg = __import__("package." + name)
    mod = getattr(pkg, name)
    mod.doSomething()

How do I do this?

Answer 1

Add the module directory to sys.path and use the normal import statement.

Answer 2

Your code is almost correct.

See __import__ function.

def doSomething(name):
    name = "package." + name
    mod = __import__(name, fromlist=[])
    mod.doSomething()

Answer 3

To detect changes to a directory, on Linux, you can use pyinotify (here is a nice working example); on a Mac, fsevents (via the PyObjC package that comes with your Mac); on Windows, Directory Change Notifications via win32api (or the Python standard library ctypes module). AFAIK, nobody's wrapped up these various approaches into one portable package. (Of course, worst case, you can fall back to "lower tech" approaches such as periodic polling, as Tim Golden's article, perhaps with a touch of "alerts from an external process" via a signal, etc).

Once you do have the notification and the name of the new or modified module, the code you show in the question should work.

Answer 4

Bastien already answered the question, anyway you may find useful this function I use to load all the modules from a subfolder in a dictionary:

def loadModules():
    res = {}
    import os
    # check subfolders
    lst = os.listdir("services")
    dir = []
    for d in lst:
        s = os.path.abspath("services") + os.sep + d
        if os.path.isdir(s) and os.path.exists(s + os.sep + "__init__.py"):
            dir.append(d)
    # load the modules
    for d in dir:
        res[d] = __import__("services." + d, fromlist = ["*"])
    return res

This other one is to instantiate an object by a class defined in one of the modules loaded by the first function:

def getClassByName(module, className):
    if not module:
        if className.startswith("services."):
            className = className.split("services.")[1]
        l = className.split(".")
        m = __services__[l[0]]
        return getClassByName(m, ".".join(l[1:]))
    elif "." in className:
        l = className.split(".")
        m = getattr(module, l[0])
        return getClassByName(m, ".".join(l[1:]))
    else:
        return getattr(module, className)

A simple way to use those functions is this:

mods = loadModules()
cls = getClassByName(mods["MyModule"], "submodule.filepy.Class")
obj = cls()

Obviously you can replace all the "services" subfolder references with parameters.

Answer 5

One trick with Bastien's answer... The __import__() function returns the package object, not the module object. If you use the following function, it will dynamically load the module from the package and return you the module, not the package.

def my_import(name):
    mod = __import__(name)
    components = name.split('.')
    for comp in components[1:]:
        mod = getattr(mod, comp)
    return mod

Then you can do:

mod = my_import('package.' + name)
mod.doSomething()