Problems with Threading in Python 2.5, KeyError: 51, Help debugging?

Question

I have a python script which runs a particular script large number of times (for monte carlo purpose) and the way I have scripted it is that, I queue up the script the desired number of times it should be run then I spawn threads and each thread runs the script once and again when its done.

Once the script in a particular thread is finished, the output is written to a file by accessing a lock (so my guess was that only one thread accesses the lock at a given time). Once the lock is released by one thread, the next thread accesses it and adds its output to the previously written file and rewrites it.

I am not facing a problem when the number of iterations is small like 10 or 20 but when its large like 50 or 150, python returns a KeyError: 51 telling me element doesn't exist and the error it points out to is within the lock which puzzles me since only one thread should access the lock at once and I do not expect an error.

This is the class I use:

class errorclass(threading.Thread):

    def __init__(self, queue):
        self.__queue=queue
        threading.Thread.__init__(self)

    def run(self):
        while 1:
              item = self.__queue.get()
              if item is None: break
              result = myfunction()
              lock = threading.RLock()
              lock.acquire()
              ADD entries from current thread to entries in file and
              REWRITE FILE
              lock.release()

queue = Queue.Queue()

for i in range(threads):
    errorclass(queue).start()

for i in range(desired iterations):
    queue.put(i)
for i in range(threads):
    queue.put(None)

Python returns with KeyError: 51 for large number of desired iterations during the adding/write file operation after lock access, I am wondering if this is the correct way to use the lock since every thread has a lock operation rather than every thread accessing a shared lock? What would be the way to rectify this?

Answer 1

Create the lock and pass it into errorclass.__init__ so they do share an instance. Otherwise each thread is locking itself out of re-entering their own critical section which is exactly a no-op.

Answer 2

What you have right now is a new lock for every iteration in each thread's run method. In effect, there is no locking going on at all. If you want to protect writes to a file, you need to make sure that all threads that access the same file use the same lock object. The simplest way to do that is to create it at the global level:

lock = threading.RLock()
class errorclass(...):
    ...
    def run(self)
        ... # get items and process
        with lock:
            ADD entries from current thread to entries in file and REWRITE FILE

You'll need to use from __future__ import with_statement to use the with statement in Python 2.5, but it makes sure you never forget to unlock the lock, not even when an error occurs. (If you need Python 2.4-and-earlier compatibility, you'll have to use a try/finally instead.)