Why does this python queue code process items multiple times?

Question

The following is a testcase I created. Why does every process print the number 1 to 5 and are the numbers not divided over the processes?

code:

#!/usr/bin/python
from subprocess import *

from Queue import Queue
from Queue import Empty

import multiprocessing
from multiprocessing import Process

def main():
    r = Runner()
    r.run()

class Runner(object):
    processes = []

    def run(self):
        q = Queue()
        for t in range(1,6):
            q.put(t)

        for pi in range(1,4):
            p = Process(target=self.runFromQueue, args=(q,))
            p.start()
            self.processes.append(p)

        for p in self.processes:
            p.join()

        print "Finished!"

    def runFromQueue(self, q):
        try:
            while True:
                number = q.get_nowait()
                print str(number)
                q.task_done()

        except Empty:
            pass


if __name__ == "__main__":
    main()

Ouput:

$ ./test_threads.py 
1
2
3
4
5
1
1
2
3
2
4
3
5
4
5
Finished!

Expected ouput:

$ ./test_threads.py 
1
2
3
4
5
Finished!

Answer 1

The Queue package is not process aware, it only works for threads. The following happens in your example:

1. Create Queue and fill with numbers
2. Fork 4 processes. This copies the memory content into each subprocess, including the filled Queue
3. Each process empties its copy of the queue

You have to use the Queue class provided by multiprocessing.