Python Pickle Help

Question

I'm not sure why this Pickle example is not showing both of the dictionary definitions. As I understand it, "ab+" should mean that the pickle.dat file is being appended to and can be read from. I'm new to the whole pickle concept, but the tutorials on the net don't seem to go beyond just the initial storage.

import cPickle as pickle

def append_object(d, fname):
    """appends a pickle dump of d to fname"""
    print "append_hash", d, fname
    with open(fname, 'ab') as pickler:
        pickle.dump(d, pickler)

db_file = 'pickle.dat'

cartoon = {}
cartoon['Mouse'] = 'Mickey'
append_object(cartoon, db_file)

cartoon = {}
cartoon['Bird'] = 'Tweety'
append_object(cartoon, db_file)

print 'loading from pickler'
with open(db_file, 'rb') as pickler:
    cartoon = pickle.load(pickler)
print 'loaded', cartoon

Ideally, I was hoping to build up a dictionary using a for loop and then add the key:value pair to the pickle.dat file, then clear the dictionary to save some RAM.

What's going on here?

Answer 1

I started to edit your code for readability and factored out append_object in the process.

There are multiple confusions here. The first, is that pickle.dump writes a Python object in its entirety. You can put multiple objects in a pickle file, but each needs its own load. The code did what you asked of it and loaded the first dictionary you wrote to the file. The second dictionary was there waiting to be read but it isn't a concatenation to the first, it is its own loadable.

Don't underestimate the importance of names. append_object isn't a great name, but it is different than append_to_object.

If you are opening a file for reading, just open it for reading and the same for writing or appending. Not only does it make your intentions more clear but it prevents silly errors.

Answer 2

Don't use pickle for that. Use a database.

Python dbm module seems to fit what you want perfectly. It presents you with a dictionary-like interface, but data is saved to disk.

Example usage:

>>> import dbm
>>> x = dbm.open('/tmp/foo.dat', 'c')
>>> x['Mouse'] = 'Mickey'
>>> x['Bird'] = 'Tweety'

Tomorrow you can load the data:

>>> import dbm
>>> x = dbm.open('/tmp/foo.dat', 'c')
>>> print x['Mouse']
Mickey
>>> print x['Bird']
Tweety