Multiprocessing: using Pool.map on a function defined in a class

Question

Hi, when i run something like

from multiprocessing import Pool
p = Pool(5)
def f(x):
     return x*x
p.map(f, [1,2,3])

it works fine. However, putting this as a function of a class

class calculate(object):
 def run(self):
  def f(x):
   return x*x

  p = Pool() 
  return p.map(f, [1,2,3])

cl = calculate()
print cl.run()

gives me the following error.

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/sw/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/sw/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/sw/lib/python2.6/multiprocessing/pool.py", line 225, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

I've seen a post from Alex Martelli dealing with the same kind of problem, but it wasn't explicit enougth.

Answer 1

Functions defined in classes (even within functions within classes) don't really pickle. However, this works:

def f(x):
    return x*x

class calculate(object):
    def run(self):
    p = Pool()
    return p.map(f, [1,2,3])

cl = calculate()
print cl.run()

Answer 2

There is currently no solution to your problem, as far as I know: the function that you give to map() must be accessible through importing your module. This is why robert's code works: the function f() can be obtained by importing the following code:

def f(x):
    return x*x

class Calculate(object):
    def run(self):
        p = Pool()
        return p.map(f, [1,2,3])

if __name__ == '__main__':
    cl = Calculate()
    print cl.run()

I actually added a "main" section, because this follows the recommendations for the Windows platform ("Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects").

I also added an uppercase letter in front of Calculate, so as to follow PEP 8. :)