How do I store then retrieve Python-native data structures into and from a file?

Question

I am reading an XML file and reorganizing the desired data into Python data structures (lists, tuples, etc.)

For example, one of my XML parser modules produces the following data:

# data_miner.py
animals = ['Chicken', 'Sheep', 'Cattle', 'Horse']
population = [150, 200, 50, 30]

Then I have a plotter module that roughly says, e.g.:

# plotter.py
from data_miner import animals, population

plot(animals, population)

Using this method, I have to parse the XML file every time I do a plot. I'm still testing other aspects of my program and the XML file doesn't change as frequently for now. Avoiding the parse stage would dramatically improve my testing time.

This is my desired result:
In between data_miner.py and plotter.py, I want a file that contains animals and population such that they can be accessed by plotter.py natively (e.g. no change in plotting code), without having to run data_miner.py every time. If possible, it shouldn't be in csv or any ASCII format, just a natively-accessible format. plotter.py should now look roughly like:

# plotter.py

# This line may not necessarily be a one-liner.
from data_file import animals, population

# But I want this portion to stay the same
plot(animals, population)

Analogy:
This is roughly equivalent to MATLAB's save command that saves the active workspace's variables into a .mat file. I'm looking for something similar to the .mat file for Python.

Recent experience:
I have seen pickle and cpickle, but I'm not sure how to get it to work. If that is the right tool to use, example code would be very helpful. There may also be other tools that I don't know yet.

Answer 1

Pickling is good if you have Python-specific objects to save. If they're just generic data in some basic container type then JSON is fine.

>>> json.dumps(['Chicken', 'Sheep', 'Cattle', 'Horse'])
'["Chicken", "Sheep", "Cattle", "Horse"]'
>>> json.dump(['Chicken', 'Sheep', 'Cattle', 'Horse'], sys.stdout) ; print
["Chicken", "Sheep", "Cattle", "Horse"]
>>> json.loads('["Chicken", "Sheep", "Cattle", "Horse"]')
[u'Chicken', u'Sheep', u'Cattle', u'Horse']

Answer 2

The pickle module, or its faster equivalent cPickle, should serve your needs well.

Specifically:

# data_miner.py
import pickle

animals = ['Chicken', 'Sheep', 'Cattle', 'Horse']
population = [150, 200, 50, 30]

with open('data_miner.pik', 'wb') as f:
  pickle.dump([animals, population], f, -1)

and

# plotter.py
import pickle

with open('data_miner.pik', 'rb') as f:
    animals, population = pickle.load(f)

print animals, population

Here, I've made data_miner.py quite explicit regarding what needs to be saved (always an excellent idea to be very explicit unless you have extremely specific reasons to do otherwise). Some things (such as modules and open files) cannot be pickled anyway, so a simple pickling of globals() would not work.

If you absolutely must, you could make a copy of globals() while removing all objects whose types make them unsuitable for saving; or, perhaps better, religiously use a leading _ in every name you don't want to save (so import pickle as _pickle, with open ... as _f, and so forth) and exclude from the copy of globals() all names with a leading underscore == with such an approach, the pickle.load would retrieve a dict, then the variables of interest would be extracted from it by indexing. However, I would strongly recommend the simple alternative of saving a list (or dict, if you want;-) with the specific values that are actually of interest, rather than taking a "wholesale" approach.

Answer 3

pickle was designed for this. Use pickle.dump to write an object to a file and pickle.load to read it back.

>>> data
{'animals': ['Chicken', 'Sheep', 'Cattle', 'Horse'], 'population': [150, 200, 50, 30]}
>>> f = open('spam.p', 'wb')
>>> pickle.dump(data, f)
>>> f.close()
>>> f = open('spam.p', 'rb')
>>> pickle.load(f)
{'animals': ['Chicken', 'Sheep', 'Cattle', 'Horse'], 'population': [150, 200, 50, 30]}

Answer 4

As already suggested, pickle is usually used here. Keep in mind that not everything is serializable (i.e. files, sockets, database connections).

With simple data structures you can also chose json or yaml. The latter is actually pretty readable and editable.