ImportError: No module named copy_reg pickle

Question

I'm trying to unpickle an object stored as a blob in a MySQL database. I've manually generated and stored the pickled object in the database, but when I try to unpickle the object, I get the following rather cryptic exception:

ImportError: No module named copy_reg

Any ideas as to why this happens?

Method of Reproduction

Note: Must do step 1 on a Windows PC and steps 3 and 4 on a Linux PC.

1) On a Windows PC:

file = open("test.txt", "w")
thing = {'a': 1, 'b':2}
cPickle.dump(thing, file)

2) Manually insert contents of text.txt into blob field of MySQL database running on linux

3) In Python running on a linux machine, fetch the contents of column from MySQL

4) Assuming that you put the contents of the blob column into a variable called data, try this:

cPickle.loads(rawString)

Answer 1

It seems this might be caused by my method of exporting the pickled object.

This bug report seens to suggest that my issue can be resolved by exporting to a file writen in binary mode. I'm going to give this a go now and see if this solves my issue.

UPDATE: This works. The solution is to make sure you export your pickled object to a file open in binary mode, even if you are using the default protocol 0 (commonly referred to as being "text")

Correct code based on orignal example in question:

file = open("test.txt", 'wb')
thing = {'a': 1, 'b':2}
cPickle.dump(thing, file)

Answer 2

Also, simply running dos2unix (under linux) over the (windows-created) pickle file solved the problem for me. (Haven't tried the open mode 'wb' thing.) Dan

Answer 3

just an interactive python session to show that you don't need any particular code to have this problem:

do something like this on a windows machine

Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle, re
>>> empty_string = re.compile("^$")
>>> pickle.dump([empty_string,1,1.23,'abc'], file('m:/mario/test-b.dump','wb'))
>>> pickle.dump([empty_string,1,1.23,'abc'], file('m:/mario/test-t.dump','wt'))
>>>

and then try to retrieve the data from a linux box

Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) 
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> pickle.load(file('/home/mario/.gvfs/transfer on 192.168.0.4/mario/test-b.dump'))
/usr/lib/python2.6/pickle.py:1124: DeprecationWarning: The sre module is deprecated, please import re.
  __import__(module)
[<_sre.SRE_Pattern object at 0xb7d42420>, 1, 1.23, 'abc']
>>> pickle.load(file('/home/mario/.gvfs/transfer on 192.168.0.4/mario/test-t.dump'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/pickle.py", line 1370, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
ImportError: No module named sre
>>>

the error message can be even more confusing if you are just pickling base types. this is what I get with the list [12, 1.2, '']:

ValueError: insecure string pickle