Python scoping and threading question

Question

I have one thread that inserts into the queueStream (not shown here) and FlowController which is another thread that pops from the queue if the queue is not empty.

I verified that the data is inserted into the queue correctly with the debug code in addToQueue()

Problem is, the 'if queueStream' statement in FlowController always sees the queueStream as empty, and instead goes to the else statement.

I'm new to Python and I feel I'm missing some simple scoping rules of some kind. I am using the 'global queueStream' but that appears to not be doing anything.

Thanks for any help.

from stream import *
from textwrap import TextWrapper
import threading
import time


queueStream = []

class FlowController(threading.Thread):
    def run(self):
        global queueStream
        while True:
            if queueStream:
                print 'Handling tweet'
                self.handleNextTweet()
            else:
                print 'No tweets, sleep for 1 second'
                time.sleep(1)

    def handleNextTweet(self):
        global queueStream
        status = queueStream.pop(0)
        print self.status_wrapper.fill(status.text)
        print '\n %s  %s  via %s\n' % (status.author.screen_name, status.created_at, status.source)


def addToQueue(status):
    print 'adding tweets to the queue'
    queueStream.append(status)

    #debug
    if queueStream:
        print 'queueStream is non-empty'

if __name__ == '__main__':
    try:
        runner = RunStream()
        runner.start()
        flow = FlowController()
        flow.start()
    except KeyboardInterrupt:
        print '\nGoodbye!'

EDIT::::::::::::

Thanks for the help so far. The Queue documentation is nice and has helped me write cleaner code since the get() function blocks (cool!). Anyway, it still did not solve my problem, but I printed out the queueStream instance before passing it to FlowController and after, and they had two different memory locations. That is why I believe nothing is being popped from the queue in FlowController. Does that mean that Python passes queueStream by value and not by reference? If so, how do I get around that?

from stream import *
from textwrap import TextWrapper
from threading import Thread
from Queue import Queue
import time


class FlowController(Thread):
    def __init__(self, queueStream):
        Thread.__init__(self)
        self.queueStream=queueStream

    def run(self):
        while True:
            status = self.queueStream.get()
            print self.status_wrapper.fill(status.text)
            print '\n %s  %s  via %s\n' % (status.author.screen_name, status.created_at, status.source)


def addToQueue(status):
    print 'adding tweets to the queue'
    queueStream.put(status)

queueStream = Queue()
if __name__ == '__main__':
    try:
        runner = RunStream()
        runner.start()
        flow = FlowController(queueStream)
        flow.start()
    except KeyboardInterrupt:
        print '\nGoodbye!'

Answer 1

It's hard to debug this problem without seeing RunStream. So I tried to dream up a simple RunStream that might exhibit the problem.

I wasn't able to reproduce the problem, but this code seems to work. If it does work and is similar enough to your RunStream, perhaps you can compare this code to your own to find what's wrong.

import threading
import time
import Queue
import sys
import random

class FlowController(threading.Thread):
    def __init__(self,queueStream):
        threading.Thread.__init__(self)        
        self.queueStream=queueStream
    def run(self):
        while True:
            if not self.queueStream.empty():
                print 'Handling tweet'
                self.handleNextTweet()
            else:
                print 'No tweets, sleep for 1 second'
                time.sleep(1)
    def handleNextTweet(self):
        status = self.queueStream.get()
        print(status)

class RunStream(threading.Thread):
    def __init__(self,queueStream):
        threading.Thread.__init__(self)
        self.queueStream=queueStream
    def run(self):
        i=0
        while True:
            addToQueue(self.queueStream,i)
            i+=1
            time.sleep(random.randint(0,2))

def addToQueue(queueStream,status):
    print 'adding tweets to the queue'
    queueStream.put(status)
    if not queueStream.empty():
        print 'queueStream is non-empty'

if __name__ == '__main__':
    queueStream = Queue.Queue()
    try:
        runner = RunStream(queueStream)
        runner.daemon=True
        runner.start()
        flow = FlowController(queueStream)
        flow.daemon=True
        flow.start()
        time.sleep(100)
    except KeyboardInterrupt:
        pass
    finally:
        print('Bye!')

Answer 2

I'm not a python expert, but I believe you have to declare your global variables even in module level functions

def addToQueue(status):
    global queueStream
    print 'adding tweets to the queue'
    queueStream.append(status)

    #debug
    if queueStream:
        print 'queueStream is non-empty'