DEADLOCK_WRAP error when using Berkeley Db in python (bsddb)

Question

I am using a berkdb to store a huge list of key-value pairs but for some reason when i try to access some of the data later i get this error:

try:
    key = 'scrape011201-590652'
    contenttext = contentdict[key]
except:
    print the error


<type 'exceptions.KeyError'> 'scrape011201-590652' in 
contenttext = contentdict[key]\n', '  File "/usr/lib64/python2.5/bsddb/__init__.py",
line 223, in __getitem__\n    return _DeadlockWrap(lambda: self.db[key])  #   
self.db[key]\n', 'File "/usr/lib64/python2.5/bsddb/dbutils.py", line 62, in 
DeadlockWrap\n    return function(*_args, **_kwargs)\n', '  File 
"/usr/lib64/python2.5/bsddb/__init__.py", line 223, in <lambda>\n    return 
_DeadlockWrap(lambda: self.db[key])  # self.db[key]\n']

I am not sure what DeadlockWrap is but there isnt any other program or process accessing the berkdb or writing to it (as far as i know,) so not sure how we could get a deadlock, if its referring to that. Is it possible that I am trying to access the data to rapidly? I have this function call in a loop, so something like

for i in hugelist:
    #try to get a value from the berkdb
    #do something with it

I am running this with multiple datasets and this error only occurs with one of them, the largest one, not the others.

Answer 1

I'm pretty certain the DeadlockWrap stuff is not relevant here. It's simply a way to automagically provide retries with a back-off strategy. In other words, if the database manipulation fails, it waits a little bit then tries again, a number of times before finally failing.

You seem to be getting a KeyError from your dictionary get operation which is more likely to be due to the fact that the key you're using doesn't actually exist in the database.

Try your code with something like:

try:
    key = 'scrape011201-590652'
    if not contentdict.has_key(key):
        print "Urk!, No record for %s"%(key)
    contenttext = contentdict[key]
except:
    print the error

This should show you if the record doesn't exist in the table (by outputting the Urk! message). As to what you do in that case, it depends on your architecture. You would probably want to return either None or an empty string. You may also want to do exactly what you're doing now (raising an exception).

Answer 2

contenttext = contentdict[key] if contentdict.has_key(key) else None