Easiest way to persist a data structure to a file in python?

Question

Let's say I have something like this:

d = { "abc" : [1, 2, 3], "qwerty" : [4,5,6] }

What's the easiest way to progammatically get that into a file that I can load from python later?

Can I somehow save it as python source (from within a python script, not manually!), then import it later?

Or should I use JSON or something?

Answer 1

If you want to save it in an easy to read JSON-like format, use repr to serialize the object and eval to deserialize it.

repr(object) -> string

Return the canonical string representation of the object. For most object types, eval(repr(object)) == object.

Answer 2

Use the pickle module.

import pickle
d = { "abc" : [1, 2, 3], "qwerty" : [4,5,6] }
afile = open(r'C:\d.pkl', 'wb')
pickle.dump(d, afile)
afile.close()

#reload object from file
file2 = open(r'C:\d.pkl', 'rb')
new_d = pickle.load(file2)
file2.close()

#print dictionary object loaded from file
print new_d

Answer 3

Take your pick: Python Standard Library - Data Persistance. Which one is most appropriate can vary by what your specific needs are.

pickle is probably the simplest and most capable as far as "write an arbitrary object to a file and recover it" goes—it can automatically handle custom classes and circular references.

For the best pickling performance (speed and space), use cPickle at HIGHEST_PROTOCOL.

Answer 4

Try the shelve module which will give you persistent dictionary, for example:

import shelve
d = { "abc" : [1, 2, 3], "qwerty" : [4,5,6] }

shelf = shelve.open('shelf_file')
for key,val in d.items():
    shelf[key] = val

shelf.close()

....

# reopen the shelf
shelf = shelve.open('shelf_file')
print shelf # => {'qwerty': [4, 5, 6], 'abc': [1, 2, 3]}

Answer 5

Just to add to the previous suggestions, if you want the file format to be easily readable and modifiable, you can also use YAML. It works extremely well for nested dicts and lists, but scales for more complex data structures (i.e. ones involving custom objects) as well, and its big plus is that the format is readable.

Answer 6

JSON has faults, but when it meets your needs, it is also:

• simple to use
• included in the standard library as the json module
• interface somewhat similar to pickle, which can handle more complex situations
• human-editable text for debugging, sharing, and version control
• valid Python code
• well-established on the web (if your program touches any of that domain)

Answer 7

You also might want to take a look at Zope's Object Database the more complex you get:-) Probably overkill for what you have, but it scales well and is not too hard to use.