Delayed loading of modules in python

Question

I'm writing a python applet for data acquisition from scientific instruments and I'm trying to figure out the best way to manage my device "drivers".

Each driver is represented by a separate python module in the package that is my program. They each implement some standard interface, but mostly it's a gigantic list of commands (and function mappings) that are specific to each device and bus involved. In the long run, (I'm writing this for my lab group and am planning on supporting a few dozen or so devices) I want to avoid loading all of them in at once. Instead, at run-time, I want to read in a list of modules into a dictionary/list and then load them only when they are actually needed.

When the user wants to use a new device, he selects the driver to use, and passes the name along to the driver subsystem which then checks to see if that driver is in the list of loaded modules, and if it's not, it calls the __import__ function and loads the driver in then instantiates a device object using the driver and hands it back to the user to use.

My question is: What is the best way to get a list of all modules in a relative way? What I mean is, if I know that the drivers are located in ..drivers is there a way to get a tidy list of modules in that subpacakage? To illustrate: usually I just call from ..drivers import driver_name to import the module, but since I'm not guaranteed to be in the package directory, can't just us os to get a list of module names.

In any case, any ideas (even maybe a better way to accomplish what I want - loadable "drivers") would be appreciated.

Answer 1

There is pkgutil which has iter_modules. (Note: The Python 2 documentation doesn't mention that function, but it works just fine for me on Python 2.6.)

You can also do it manually using __file__, os.path and os.listdir.

Answer 2

If the location of the driver packages is known, then you could use the absolute path to the driver folder(s).

I think that a more robust approach would be to have a list of all known drivers for your program, perhaps along with a bit of metadata so that users can choose them more easily. You could have the drivers register themselves, or register them yourself if you know of likely locations. One advantage of having the drivers register themselves is that you might have python files other than drivers in the driver package directories.

Edit

For a driver to "register itself," you might do something like this:

if __name__ == "__main__":
    import mamapackage
    mamapackage.register("thisdriver", 
               os.path.abspath(__file__), 
               "A description of this driver")

Then in your own package, provide a "register" function that updates a list of drivers, with their locations and descriptions (a simple text file should do for starters).

Answer 3

Why not make a package of all the drivers? Then you could

import drivers
drivers.load("device1")

where the load function maintains a mapping of device name to module in package? Then you could dynamically import each of the required modules when necessary...