shelve gives strange error

Question

Hi,

I'm trying to put some sites i crawled into a shelve, but the shelve won't accept any Site-objects. It will accept lists, strings, tuples, what have you, but as soon as i put in a Site-object, it crashes when i try to get the contents of the shelve

So when i fill up my shelve like this:

def add_to_shelve(self, site):
    db = shelve.open("database")
    print site, site.url
    for word in site.content:
        db[word] = site.url #site.url is a string, word has to be one too

shelve.open("database")['whatever'] works perfectly.

But if I do this:

def add_to_shelve(self, site):
    db = shelve.open("database")
    print site, site.url
    for word in site.content:
        db[word] = site #site is now an object of Site

shelve.open("database")['whatever'] errors out with this error message:

AttributeError: 'module' object has no attribute 'Site'

I'm completely stumped, and the pythondocs, strangely, don't have much info either. All they say is that the key in a shelve has to be a string, but the value or data can be "an arbitrary object"

Answer 1

It seems that Python is looking for a constructor for a 'Site' object, and not finding it. I have not used shelve, but I recall the rules for what can be pickled are byzantine, and suspect the shelve rules are similar.

Try adding the line:

Site = sitemodule.Site

(with the name of the module providing 'Site') before you try unshelving. This ensures that a Site class can be found.

Answer 2

It looks like you refactored your code after saving objects in the shelve. When retrieving objects from the shelve, Python rebuilds the object, and it needs to find the original class that, presumably, you have moved. This problem is typical when working with pickle (as the shelve module does).

The solution, as pduel suggests, is to provide a backwards-compatibility reference to the class in the same location that it used to be, so that pickle can find it. If you re-save all the objects, thereby rebuilding the pickles, you can remove that backwards-comatibility referece.