Using Python multiprocessing while importing a module via file path

Question

I'm writing a program which imports a module using a file path, with the function imp.load_source(module_name,module_path). It seems to cause a problem when I try to pass objects from this module into a Process.

An example:

import multiprocessing
import imp

class MyProcess(multiprocessing.Process):
    def __init__(self,thing):
        multiprocessing.Process.__init__(self)
        self.thing=thing
    def run(self):
        x=self.thing


if __name__=="__main__":
    module=imp.load_source('life', 'C:\\Documents and Settings\\User\\workspace\\GarlicSim\\src\\simulations\\life\\life.py')
    thing=module.step
    print(thing)
    p=MyProcess(thing)
    p.start()

Note: for this code to "work", you must substitute the parameters I gave to imp.load_source with something else: It has to be some Python file on your computer, preferably not in the same folder. Then, in thing=module.step, instead of step put in some random function or class that is defined in that .py file.

I am getting the following traceback:

<function step at 0x00D5B030>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python26\lib\multiprocessing\forking.py", line 342, in main
    self = load(from_parent)
  File "C:\Python26\lib\pickle.py", line 1370, in load
    return Unpickler(file).load()
  File "C:\Python26\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\Python26\lib\pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "C:\Python26\lib\pickle.py", line 1124, in find_class
    __import__(module)
ImportError: No module named life

So what do I do?

EDIT:

I'm using Python 2.6.2c1 on Win XP.

Answer 1

test.py on any folder:

import multiprocessing
import imp

class MyProcess(multiprocessing.Process):
    def __init__(self,thing):
        multiprocessing.Process.__init__(self)
        self.thing=thing
    def run(self):
        print 'running...', self.thing()


if __name__=="__main__":
    module=imp.load_source('life', '/tmp/life.py')
    thing=module.step
    print(thing)
    p=MyProcess(thing)
    p.start()

life.py on /tmp:

def step():
    return 'It works!'

Running test.py:

$ python test.py
<function step at 0xb7dc4d4c>
running... It works!

I just tested and it works, so you must be doing something else wrong. Please edit your question and paste the real code that is not working.

I am Using ubuntu Jaunty 9.04 with default python (Python 2.6.2 release26-maint, Apr 19 2009, 01:56:41). Don't know if your problem is windows-only, because I don't have windows available to test.

That's a reason that mangling with import paths is a bad idea. You'll never get it right for all platforms/environments, and your users will probably get angry. Better just use the python way of finding modules, which is by placing the module on the python module search path. That would work consistently everywhere.

Answer 2

I've just done the following, running on XP with Python 2.5...

D:\Experiments\ModuleLoading\test.py

import imp

if __name__=="__main__":
    module=imp.load_source('life', r'D:\Experiments\ModuleLoading\somefolder\life.py')
    thing=module.step
    print(thing)

D:\Experiments\ModuleLoading\somefolder\step.py

def step():
    return 'It works!'

...and running the script gives:

D:\jcage\Projects\Experiments\ModuleLoading>test.py
<function step at 0x00A0B470>

...so try that first and make sure the module can load without multiprocessing?

[Edit] Okay, so it's definitely a problem importing into the forked process. There's some bits and pieces in the documentation which are specific to windows:

More picklability

Ensure that all arguments to Process.__init__() are picklable. This means, in particular, that bound or unbound methods cannot be used directly as the target argument on Windows — just define a function and use that instead. Also, if you subclass Process then make sure that instances will be picklable when the Process.start() method is called.

Global variables

Bear in mind that if code run in a child process tries to access a global variable, then the value it sees (if any) may not be the same as the value in the parent process at the time that Process.start() was called. However, global variables which are just module level constants cause no problems.

Safe importing of main module

Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).

[Edit2] Is there any reason why you can't do the importing in the process? I think the problem is that when you launch the new process it's not running in the same address space so trying to access functions in the original thread won't work.

You could do this for D:\Experiments\ModuleLoading\test.py instead:

from multiprocessing import Process
import imp

class MyProcess(Process):
    def __init__(self):
        Process.__init__(self)

    def run(self):
        print 'run called...'
        module=imp.load_source('life', r'D:\Experiments\ModuleLoading\somefolder\life.py')
        print 'running...', module.step()

if __name__=="__main__":
    p=MyProcess()
    p.start()

Answer 3

Probably it does not work because of placing of import code into main block. Code below works on Windows XP, Python 2.6. Then life module will also be imported in new process.

import multiprocessing
import imp

class MyProcess(multiprocessing.Process):
  def __init__(self,thing):
    multiprocessing.Process.__init__(self)
    self.thing=thing
  def run(self):
    print("Exiting self.thing")
    self.thing()
    print("Finished")

life=imp.load_source('life', r'd:\temp5\life.py')

if __name__=="__main__":
  p=MyProcess(life.step)
  p.start()