How to load modules dynamically on package import?

Question

Given the following example layout:

test/
  test.py
  formats/
    __init__.py
    format_a.py
    format_b.py

What I try to archive is, that whenever I import formats, the __init__.py looks for all available modules in the formats subdir, loads them and makes them available (right now simply through a variable, supported_formats). If theres a better, more pythonic or otherwise approach to dynamically loading stuff on runtime, based on physical available files, please tell.

My Approach

I tried something like this (in __init__.py):

supported_formats =  [__import__(f[:f.index('.py')]) for f in glob.glob('*.py')]

So far I just get it to work, when I run __init__.py from the command line (from the formats subdir or from other directories) . But when I import it from test.py, it bails on me like this:

ImportError: No module named format_a.py

Same when I import it from the python interpreter, when I started the interpreter in an other directory but the formats subdir.

Here's the whole code. It also looks for a specific class and stores one instance of each class in an dict, but loading the modules dynamically is the main part I don't get:

def dload(get_cls=True, get_mod=True, key=None, fstring_mod='*.py', fstring_class=''):
  if p.dirname(__file__):
    path = p.split(p.abspath(__file__))[0]
    fstring_mod = p.join(path, fstring_mod)
    print >> sys.stderr, 'Path-Glob:', fstring_mod
  modules = [p.split(fn)[1][:fn.index('.py')] for fn in glob.glob(fstring_mod)]
  print >> sys.stderr, 'Modules:', ', '.join(modules)
  modules = [__import__(m) for m in modules]
  if get_cls:
    classes = {} if key else []
    for m in modules:
      print >> sys.stderr, "-", m
      for c in [m.__dict__[c]() for c in m.__dict__ if c.startswith(fstring_class)]:
        print >> sys.stderr, " ", c
        if key:
          classes[getattr(c, key)] = c
        else:
          classes.append(c)
    if get_mod:
      return (modules, classes)
    else:
      return classes
  elif get_mod:
    return modules

_supported_formats = dload(get_mod=False, key='fid', fstring_mod='format_*.py', fstring_class='Format')

My Idea

The whole messing with filesystem-paths and the like is probably messy anyway. I would like to handle this with module namespaces or something similar, but I'm kinda lost right now on how start and how to address the modules, so they reachable from anywhere.

Answer 1

A module is searched in sys.path. You should be able to extend sys.path with the path to your module. I'm also not really sure whether you can load a module on sys.path with a 'module.py' convention, I would think without '.py' is preferred.

This is obviously not a solution, but may be handy nonetheless.

Answer 2

I thought if you did something that, 'formats' would be your package, so when you tell it import formats you should be able to access the rest of the modules inside that package, so, you would have something like formats.format_a.your_method

Not sure though, I'm just a n00b.

Answer 3

There are two fixes you need to make to your code:

1. You should call __import__(m, globals(), locals()) instead of __import__(m). This is needed for Python to locate the modules within the package.

2. Your code doesn't remove the .py extension properly since you call index() on the wrong string. If it will always be a .py extension, you can simply use p.split(fn)[1][:-3] instead.

Answer 4

First you must make it so that your code works regardless of the current working directory. For that you use the __file__ variable. You should also use absolute imports.

So something like (untested):

supported_formats = {}
for fn in os.listdir(os.path.dirname(__file__)):
    if fn.endswith('.py'):
        exec ("from formats import %s" % fn[:-3]) in supported_formats

Answer 5

Here's the code I came up with after the corrections from interjay. Still not sure if this is good style.

def load_modules(filemask='*.py', ignore_list=('__init__.py', )):
  modules = {}
  dirname = os.path.dirname(__file__)
  if dirname:
    filemask = os.path.join(dirname, filemask)
  for fn in glob.glob(filemask):
    fn = os.path.split(fn)[1]
    if fn in ignore_list:
      continue
    fn = os.path.splitext(fn)[0]
    modules[fn] = __import__(fn, globals(), locals())
  return modules