Really weird issue with shelve (python)

Question

I create a file called foo_module.py containing the following code:

import shelve, whichdb, os

from foo_package.g import g

g.shelf = shelve.open("foo_path")
g.shelf.close() 

print whichdb.whichdb("foo_path")  # => dbhash
os.remove("foo_path")

Next to that file I create a directory called foo_package than contains an empty __init__.py file and a file called g.py that just contains:

class g:
    pass

Now when I run foo_module.py I get a weird error message:

Exception TypeError: "'NoneType' object is not callable" in ignored

But then, if I rename the directory from foo_package to foo, and change the import line in foo_module.py, I don't get any error. Wtf is going on here?

Running Python 2.6.4 on WinXP.

Answer 1

It seems to be an exception in a shutdown function registered by the shelve module. The "ignored" part is from the shutdown system, and might get its wording improved sometime, per Issue 6294. I'm still hoping for an answer on how to eliminate the exception itself, though...

Answer 2

I think you've hit a minor bug in 2.6.4's code related to the cleanup at end of program. If you run python -v you can see exactly at what point of the cleanup the error comes:

# cleanup[1] foo_package.g
Exception TypeError: "'NoneType' object is not callable" in  ignored

Python sets references to None during the cleanup at the end of program, and it looks like it's getting confused about the status of g.shelf. As a workaround you could set g.shelf = None after the close. I would also recommend opening a bug in Python's bug tracker!

Answer 3

This is indeed a Python bug, and I've posted a patch to the tracker issue you opened (thanks for doing that).

The problem is that shelve's del method calls its close method, but if the shelve module has already been through cleanup, the close method fails with the message you see.

You can avoid the message in your code by adding 'del g.shelf' after g.shelf.close. As long as g.shelf is the only reference to the shelf, this will result in CPython calling the shelve's del method right away, before the interpreter cleanup phase, and thus avoid the error message.

Answer 4

After days of hair loss, I finally had success using an atexit function:

  import atexit
  ...
  cache = shelve.open(path)
  atexit.register( lambda : cache.close() )

It's most appropriate to register right after opening. This works with multiple concurrent shelves.

(python 2.6.5 on lucid)