Is something like ConfigParser appropriate for saving state (key, value) between runs?

Question

I want to save a set of key, value pairs (string, int) between runs of a Python program, reload them on subsequent runs and write the changes to be available on the next run.

I don't think of this data as a configuration file, but it would fit the ConfigParser capabilities quite well. I would only need two [sections]. It's only a few hundred pairs and very simple so I don't think it's necessary to do an actual database.

Is it appropriate to use ConfigParser in this way? I've also considered using Perl and XML::Simple. What about that? Is there a way to do this in bash without Python or Perl?

Answer 1

Well, you have better options. You can for example use pickle or json format. Pickle serializing module is very easy to use.

import cPickle
cPickle.dump(obj, open('save.p', 'wb')) 
obj = cPickle.load(open('save.p', 'rb'))

The format is not human readable and unpickling is not secure against erroneous or maliciously constructed data. You should not unpickle untrusted data.

If you are using python 2.6 there is a builtin module called json. It is as easy as pickle to use:

import json
encoded = json.dumps(obj)
obj = json.loads(encoded)

Json format is human readable and is very similar to the dictionary string representation in python. And doesn't have any security issues like pickle.

If you are using an earlier version of python you can simplejson instead.

Answer 2

For me, PyYAML works well for these kind of things. I used to use pickle or ConfigParser before.

Answer 3

Re doing it in bash: If your strings are valid identifiers, you could use environment variables and env.

Answer 4

ConfigParser is a fine way of doing it. There are other ways (the json and cPickle modules already mentioned may be useful) that are also good, depending on whether you want to have text files or binary files and if you want code to work simply in older versions of Python or not.

You may want to have a thin abstraction layer on top of your chosen way to make it easier to change your mind.

Answer 5

Sounds like a job for a dbm. Basically it is a hash that lives external to your program. There are many implementations. In Perl it is trivial to tie a dbm to a hash (i.e. make it look like a dbm is really a normal hash variable). I don't know if there is any equivalent in mechanism in Python, but I would be surprised if there weren't.

Answer 6

If you can update the state key by key then any of the DBM databases will work. If you need really high performance and compact storage then Tokyo Cabinet - http://tokyocabinet.sourceforge.net/ is the cool toy.

If you want to save and load the whole thing at once (to maybe keep old versions or some such) and don't have too much data then just use JSON. It's much nicer to work with than XML. I don't know how the JSON implementation is in Python, but in Perl the JSON::XS module is insanely fast.