How to save data with Python?

Question

I am working on a program in Python and want users to be able to save data they are working on. I have looked into cPickle; it seems like it would be a fast and easy way to save data, it seems insecure. Since entire functions, classes, etc can be pickled, I am worried that a rogue save file could inject harmful code into the program. Is there a way I can prevent that, or should I look into other methods of saving data, such as directly converting to a string (which also seems insecure,) or creating an XML hierarchy, and putting data in that.

I am new to python, so please bear with me.

Thanks in advance!

EDIT: As for the type of data I am storing, it is mainly dictionaries and lists. Information such as names, speeds, etc. It is fairly simple right now, but may get more complex in the future.

Answer 1

You need to give us more context before we can answer: what type of data are you saving, how much is there, how do you want to access it?

As for pickles: they do not store code. When you pickle a function or class, it is the name that is stored, not the actual code itself.

Answer 2

You could do something like:

to write

• Pickle
• Sign pickled file
• Done

to read

• Check pickled file's signature
• Unpickle
• Use

I wonder though what makes you think that the data files are going to be tampered but your application is not going to be?

Answer 3

From your description JSON encoding is the secure and fast solution. There is a json module in python2.6, you can use it like this:

import json
obj = {'key1': 'value1', 'key2': [1, 2, 3, 4], 'key3': 1322}
encoded = json.dumps(obj)
obj = json.loads(encoded)

JSON format is human readable and is very similar to the dictionary string representation in python. And doesn't have any security issues like pickle. If you don't have python2.6 you can install cjson or simplejson

You can't use JSON to save python objects like Pickle. But you can use it to save: strings, dictionaries, lists, ... It can be enough for most cases.

To explain why pickle is insecure. From python docs:

Most of the security issues surrounding the pickle and cPickle module involve unpickling. There are no known security vulnerabilities related to pickling because you (the programmer) control the objects that pickle will interact with, and all it produces is a string.

However, for unpickling, it is never a good idea to unpickle an untrusted string whose origins are dubious, for example, strings read from a socket. This is because unpickling can create unexpected objects and even potentially run methods of those objects, such as their class constructor or destructor ... The moral of the story is that you should be really careful about the source of the strings your application unpickles.

There are some ways to defend yourself but it is much easier to use JSON in your case.

Answer 4

You should use a database of some kind. Storing in pickle format isn't a good idea (in most cases). You may consider:

• SQLite - (included in Python 2.5+) fast and simple, but requires knowledge of SQL and DB-API
• buzhug - non-SQL, file based database with pythonic syntax
• SQL database - you may use interface to some of DBMS (like MySQL, PostreSQL etc.), but it's only good for larger amount of data (thousands of records).

You may find some other solutions here.

Answer 5

Who -- specifically -- is the sociopath who's going through the effort to break a program by hacking the pickled file?

It's Python. The sociopath has your source. They don't need to fool around hacking your pickle file. They can just edit your source and do all the "damage" they want.

Don't worry about "insecurity" unless you're involved in litigation with organized crime syndicates.

Don't worry about "a rogue save file could inject harmful code into the program". No one will bother with a rogue save file when they have the source.

Answer 6

You might enjoy working with the y_serial module over at http://yserial.sourceforge.net

which reads like a tutorial but operationally offers working code for serialization and persistance. The commentary discusses some of the pros and cons relevant to issues raised here.

It's designed to be a general solution to warehousing compressed Python objects with SQLite (with almost no SQL fuss ;-)

Hope this helps.