GAE: ValueError: insecure string pickle

Question

I'm having trouble unpickling objects from Google App Engine. I am running Windows 7. Here is the procedure:

1. Create a CSV with one of the fields being pickle.dumps([[('CS', 2110), ('CS', 3300), ('CS', 3140)]]), or some similar argument.

The CSV looks something like this:

INFO,2210,"CS 2110, 3300, 3140","(lp0
(lp1
(S'CS'
p2
I2110
tp3
a(g2
I3300
tp4
a(g2
I3140
tp5
aa."

CS,3110,CS 2110 or equivalent experience,"(lp0
(lp1
(S'CS'
p2
I2110
tp3
aa."

MSE,4102,"MATH 2210, 2230, 2310, or 2940","(lp0
(lp1
(S'MATH'
p2
I2210
tp3
a(g2
I2230
tp4
a(g2
I2310
tp5
aa(lp6
(g2
I2940
tp7
aa."

(Yes, those are \ns produced by pickle.dumps())

1. Load this file into the google app engine devserver:

appcfg.py upload_data --config_file="DataLoader.py" --filename="pre_req_data.csv" --kind=Course --url=http://localhost:8083/remote_api "appdir"

Course model:

class Course(db.Model):
    dept_code = db.StringProperty()
    number = db.IntegerProperty()
    raw_pre_reqs = db.StringProperty(multiline=True)
    original_description = db.StringProperty()

    def getPreReqs(self):
        pickle.loads(str(self.raw_pre_reqs))

DataLoader.py:

class CourseLoader(bulkloader.Loader):
    def __init__(self):
        bulkloader.Loader.__init__(self, 'Course',
                                   [('dept_code', str),
                                    ('number', int),
                                    ('original_description', str),
                                    ('raw_pre_reqs', str)
                                   ])

loaders = [CourseLoader]

1. Confirm that the data is successfully loaded:

1. Try to unpickle:

   class MainHandler(webapp.RequestHandler):

   def get(self):
       self.writeOut('cock!')
       self.writeOut('\n')
   
   
   courses = Course().all()
   for c in courses:
       self.writeOut("%s => %s" % (c.raw_pre_reqs, c.getPreReqs()))
   
   
   def writeOut(self, string):
       self.response.out.write(string)
   

2. Observe error:

   Traceback (most recent call last):

   File "C:\Program Files\Google\google_appengine\google\appengine\ext\webapp_init_.py", line 511, in call handler.get(*groups)

   File "main.py", line 30, in get self.writeOut("%s => %s" % (c.raw_pre_reqs, c.getPreReqs()))

   File "src\Models.py", line 17, in getPreReqs pickle.loads(str(self.raw_pre_reqs))

   File "C:\Python26\lib\pickle.py", line 1374, in loads return Unpickler(file).load()

   File "C:\Python26\lib\pickle.py", line 858, in load dispatchkey

   File "C:\Python26\lib\pickle.py", line 966, in load_string raise ValueError, "insecure string pickle"

   ValueError: insecure string pickle

What am I doing wrong here?

Answer 1

Pickle is a binary format, and CSV isn't binary-safe. You need to encode your pickle - say, using base64.b64encode - if you want to transport it inside a text format.

Answer 2

Pickle can be a binary format, but by default it is completely ASCII safe(protocol 0). Read the pickle docs for specifics: pickle.dump.

It will usually have line breaks so you have to take that into consideration when using a line-based format such as CSV.

If you are reading someone else's pickle, they may have used the binary protocol, but the output you pasted looks like normal pickling.