tags:

views:

128

answers:

2

I am trying to load the function in a remote environment using cPickle. But I got the error "the 'module' object has no attribute ..." . Where I really stuck is the namespace has already contain that attributes , even though it fails to load Please Help

import inspect
import cPickle as pickle
from run import run


def get_source(func): 
 sourcelines = inspect.getsourcelines(func)[0]
 sourcelines[0] = sourcelines[0].lstrip()
 return "".join(sourcelines)

def fun(f):
 return f()

def fun1():
 return 10 

funcs = (fun, fun1) 

sources = [get_source(func) for func in funcs]

funcs_serialized = pickle.dumps((fun.func_name,sources),0)

args_serialized = pickle.dumps(fun1,0) 

#Creating the Environment where fun & fun1 doesnot exist 
del globals()['fun']
del globals()['fun1']

r = run() 

r.work(funcs_serialized,args_serialized) 

Here is run.py

import cPickle as pickle

class run():
 def __init__(self):
  pass

 def work(self,funcs_serialized,args_serialized):

  func, fsources = pickle.loads(funcs_serialized)

  fobjs = [compile(fsource, '<string>', 'exec') for fsource in fsources]

    #After eval fun and fun1 should be there in globals/locals
  for fobj in fobjs:
   try: 
    eval(fobj)
    globals().update(locals())
   except:
    pass

  print "Fun1 in Globals: ",globals()['fun1']   
  print "Fun1 in locals: ",locals()['fun1']   
  arg = pickle.loads(args_serialized)

The error is

Fun1 in Globals:  <function fun1 at 0xb7dae6f4>
Fun1 in locals:  <function fun1 at 0xb7dae6f4>
Traceback (most recent call last):
  File "fun.py", line 32, in <module>
    r.work(funcs_serialized,args_serialized) 
  File "/home/guest/kathi/python/workspace/run.py", line 23, in work
    arg = pickle.loads(args_serialized)
AttributeError: 'module' object has no attribute 'fun1'
A: 

The module name of the function is saved into the pickle, when you are doing the loads it is looking for fun1 in __main__ or whereever it was originally

gnibbler
So how fun1 is integrate into the module, say run
Harikrishnan R
A: 

From http://docs.python.org/library/pickle.html#what-can-be-pickled-and-unpickled:

Note that functions (built-in and user-defined) are pickled by “fully qualified” name reference, not by value. This means that only the function name is pickled, along with the name of module the function is defined in. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.

You deleted the reference to fun1 in the module that defines fun1, thus the error.

Jeremy Brown
I intentionally deleted the reference , but during eval() it is again recreated in the namespace
Harikrishnan R
Without looking too deeply into it I would imagine that the eval defines the function within the global scope of the run module rather than the module that originally defined it (likely `__main__` ?). A better exercise of your testwould be to define the functions in another module ("my_funcs.py") that maintains a reference to the function object. Remember that function objects are not really pickled - just the fully qualified names. The module in which it is defined must be importable and it must have the function name as an attribute.
Jeremy Brown
Are u familiar with parallel python? they are doing exactly the same way, they pickle the func and its arguments and sent to run function.but when they do the run(), it executes perfectly. I want to know how they achieve it, is it better to contact parallel python ?
Harikrishnan R
No I am not familiar with it. Regardless of how they do it (read their source if needed), you are missing the point about pickling functions. Function objects themselves _are not pickled_. A fully qualified name resolving to the function object is pickled. For what you want to do (dynamically define the function in the unpickling environment), you would need to create a module that is importable via the same path (if function is defined in mylib.funcs, then you need to be able to import mylib.funcs) and that module object _must_ have a reference to the function object using the same attr name.
Jeremy Brown