Python Queue - yet another question

Question

Hi, I am reading up on how to utilize Python Queues to send and receive short messages between nodes. I am simulating a set of nodes that are in a nice tree structure. I want some of these nodes to send a fixed-size data to its parent. Once this parent receives data from some of its child-nodes, it will "process" it and send a "aggregate" packet to its parent...and so on.

To do this, I was told that queues would be useful to pass messages and a quick readup on it tells me that it will suit my needs. However, I am finding it a bit difficult to implement a basic setup and test my understanding -- 1 producer (that generates a message packet) and 1 consumer (the worker thread that dequeues the task and processes it).

I have searched and read many posts here and elsewhere...and I understand all the queue methods. What I still do not understand is how to associate or bind a queue to 2 given nodes.

I want node-1 and node-2 to send messages to node-3. For this basic scenario, I have to somehow create one (or 2) queues and "associate" it with node-1 and node-2 which uses it to place messages into and node-3. And, node-3 must also be "listening"/"associated" with this queue to "get" or dequeue a task.

If node-1 and node-2 are to be 'producers', I should have them as 2 separate threads and node-3 being the 3rd thread. Then, I must create one queue Q. Then, I should have node-1 and node-2 create messages, 'put' them into the queue. Node-3 will have to be 'somehow' notified /waken-up to 'get' these messages from Q and process it.

I have seen

http://docs.python.org/library/queue.html#module-Queue

Here is the (untested) code I have so far:

=================================================================

import threading
import queue

q = Queue.Queue(2)   # create a queue of size 2.

# worker is node-3 which received msgs from 1 and 2 and fuses them.
def worker():
    while True:
        msg = q.get()
        fuse_msgs(msg)
        q.task_done()

# generate 3 worker threads. Node-3 could be both a producer and consumer. Each
# thread represents each node that will be a potential producer/consumer or both.
# need something like t1 - thread-1 for node-1 ...

for i in range(3):
     t = Thread(target=worker)
     t.setDaemon(True)
     t.start()

# How can I make node-1 and node-2 to put items into a specified queue???



for msg in source():
    q.put(item)

q.join()

=========================================

Am I going in the right direction? Please let me know where I am wrong and what I am misunderstanding...

Any help is appreciated. I have hundreds of nodes and links. If I get this fundamentals straight, I will be able to move on smoothly...

Thanks, B.R.Srini.

Answer 1

I'm not commenting on the python code in particular, but as far as your queues are designed, it seems you just need one queue in the node 1,2,3 scenario you were describing. Basically, you have one queue, where you have node 1 and node 2 putting messages to, and node 3 reading from.

You should be able to tell node-3 to do a "blocking" get on the queue, so it will just wait until it sees a message for it to process, and leave nodes 1 and 2 to produce their output as fast as possible.

Depending on the processing speed of each node, and the traffic patterns you expect, you will probably want a queue deeper than 2 messages, so that the producing nodes don't have to wait for the queue to be drained.