Dynamic Loading of Python Modules

Question

I'm trying to dynamically load modules I've created.

Right now this works properly:

import structures.index

But if I try the same thing by importing it dynamically, it fails.

struct = __import__("structures.index")

Error supplied is:

Error ('No module named structures.index',)

Any ideas why?

---

Edit: When using full scope (it sort of works?):

struct = __import__("neoform.structures.index")

This doesn't throw any errors, however, it isn't loading the index module, it's loading the "neoform" module instead.

The result of "struct" is:

<module 'neoform' from '/neoform/__init__.py'>

Also, as a side question, how can I then instantiate a class within a dynamically loaded module? (assuming all the modules contain a common class name).

Edit: Solution: (thanks coonj & Rick) This ended up being what worked. Not sure why (yet), but the fromlist had to be something "anything apparently, since it worked when I put the letter "a" as a value (strange, given that the file only had 1 class in it).

def get_struct_module(self, name):
    try:
        return = __import__("neoform.structures." + name, fromlist='*')
    except ImportError, e:
        self.out.add("Could not load struct: neoform.structure." + name + "\n\n" + "Error " + str(e.args))

Answer 1

Java programmer here, but I think you need the imp module

Answer 2

I'm not sure what "it fails" means, so I'll just mention that __import__('structures.index') should, in fact, work, but it doesn't assign the module name in the current scope. To do that (and then use a class in the dynamically imported module), you'll have to use:

structures = __import__('structures.index')
structures.index.SomeClass(...)

The complete details on __import__ are available here.

Edit: (based on question edit)

To import neoform.structures.index, and return the index module, you would do the following:

structures = __import__('neoform.structures.index', 
                        fromlist=['does not in fact matter what goes here!'])

So if you have a list of package names packages, you can import their index modules and instantiate some MyClass class for each using the following code:

modules = [ __import__('neoform.%s.index' % pkg, fromlist=['a']) 
            for pkg in packages ]
objects = [ m.MyClass() for m in modules ]

Answer 3

To import sub-modules, you need to specify them in the fromlist arg of __import__()
Fo example, the equivalent of:

import structures.index

is:

structures = __import__('structures', fromlist=['index'])

To do this in a map is a little more tricky...

import mod1.index
import mod2.index
import mod3.index

For those imports, you would want to define a new function to get the index sub-module from each module:

def getIndexMods(mod_names):
  mod_list = map(lambda x: __import__(x, fromlist='index'))
  index_mods = [mod.index for mod in mod_list]
  return index_mods

Now, you can do this to get references to all index modules:

index_mods = getIndexMods(['mod1', 'mod2', 'mod3'])

Also, if you want to grab sub-modules that are not named 'index' then you could do this:

mod1, mod2, mod3 = map(lambda x,y: __import__(x, fromlist=y), 
  ['mod1', 'mod2', 'mod3'], ['index1', 'index2', 'index3'])

Answer 4

Why on earth would you replace

import structures.index

with

map(__import__, ["structures.index"])

The first one (a) works, (b) is dynamic and (c) is directly supported. What possible use case is there for replacing easy-to-change, plain-text source with something more complex?

In short: don't do this. It doesn't have any value.

---

Edit

The "I'm getting the import from a database" is a noble effort, but still not sensible. What code block depends on those imports? That whole code block -- imports and all -- is what you want to execute. That whole code block -- imports, statements and everything -- should be a plain old python module file.

Import that block of code from the file system. Use the database to identify which file, the author of the file -- anything you want to use the database for. But simply import and execute the module the simplest possible way.

Answer 5

Use full scope ("neoform.structures.index") with this helper method.

def import_module(name):
    mod = __import__(name)
    components = name.split('.')
    for comp in components[1:]:
        mod = getattr(mod, comp)
    return mod

module = import_module("neoform.structures.index")
# do stuff with module

Answer 6

>>> import imp
>>> fm = imp.find_module('index', ['./structures']) # for submodule
>>> mymod = imp.load_module('structures.index', *fm)
>>> mymod
<module 'structures.index' from './structures/index.pyc'>
>>> x = mymod.insideIndex()
Initialising index class...

Voila!