Adding objects to queue without interruption

Question

I would like to put two objects into a queue, but I've got to be sure the objects are in both queues at the same time, therefore it should not be interrupted in between - something like an atomic block. Does some one have a solution? Many thanks...

queue_01.put(car)
queue_02.put(bike)

Answer 1

You could use a Condition object. You can tell the threads to wait with cond.wait(), and signal when the queues are ready with cond.notify_all(). See, for example, Doug Hellman's wonderful Python Module of the Week blog. His code uses multiprocessing; here I've adapted it for threading:

import threading
import Queue
import time

def stage_1(cond,q1,q2):
    """perform first stage of work, then notify stage_2 to continue"""
    with cond:
        q1.put('car')
        q2.put('bike')
        print 'stage_1 done and ready for stage 2'
        cond.notify_all()
def stage_2(cond,q):
    """wait for the condition telling us stage_1 is done"""
    name=threading.current_thread().name
    print 'Starting', name
    with cond:
        cond.wait()
        print '%s running' % name
def run():
    # http://www.doughellmann.com/PyMOTW/multiprocessing/communication.html#synchronizing-threads-with-a-condition-object
    condition=threading.Condition()
    queue_01=Queue.Queue()
    queue_02=Queue.Queue()    
    s1=threading.Thread(name='s1', target=stage_1, args=(condition,queue_01,queue_02))
    s2_clients=[
        threading.Thread(name='stage_2[1]', target=stage_2, args=(condition,queue_01)),
        threading.Thread(name='stage_2[2]', target=stage_2, args=(condition,queue_02)),
        ]
    # Notice stage2 processes are started before stage1 process, and yet they wait
    # until stage1 finishes
    for c in s2_clients:
        c.start()
        time.sleep(1)
    s1.start()
    s1.join()
    for c in s2_clients:
        c.join()

run()

Running the script yields

Starting stage_2[1]
Starting stage_2[2]
stage_1 done and ready for stage 2  <-- Notice that stage2 is prevented from running until the queues have been packed.
stage_2[2] running
stage_2[1] running

Answer 2

To atomically add to two different queues, acquire the locks for both queues first. That's easiest to do by making a subclass of Queue that uses recursive locks.

import Queue # Note: module renamed to "queue" in Python 3
import threading

class MyQueue(Queue.Queue):
    "Make a queue that uses a recursive lock instead of a regular lock"
    def __init__(self):
        Queue.Queue.__init__(self)
        self.mutex = threading.RLock()

queue_01 = MyQueue()
queue_02 = MyQueue()

with queue_01.mutex:
    with queue_02.mutex:
        queue_01.put(1)
        queue_02.put(2)