how to put a function and arguments into python queue?

Question

Hello

I have a python program with 2 threads ( let's name them 'source' and 'destination' ). Source thread sometimes post a message to destination thread with some arguments. Than destination thread picks a message it must call a corresponding function with aruments saved in message.

This task can be solved multiple ways. The easy one is tu put a big 'if...if..if' in destination thread's message pick cycle and call function according to received message type and saved arguments. But this will result in huge amounf of code ( or big lookup table ) and adding new messages / handler function will evolve additonal step to write code in message pick cycle.

Since python treats functions as first-class objects and have tuples, i want to put a function and argumens inside a message, so than destination thread picks a message it just call a functon saved within a message without any knowledge what function it is.

I can write a code for a functions with specified number of arguments:

from Queue import *
from thread import *
from time import *

q = Queue()

def HandleMsg( arg1, arg2 ) :
  print arg1, arg2

def HandleAnotherMsg( arg1, arg2, arg3 ) :
  print arg1, arg2, arg3

def DestinationThread( a ) :
  while True :
    (f, a, b) = q.get()
    f( a, b )

start_new_thread( DestinationThread, ( 0, ) )
print "start"
sleep( 1 )
q.put( (HandleMsg, 1, 2) )
sleep( 1 )
print "stop"

The question is: how to modify a code so i can put() a function with any number of arguments in queue? for example HandleAnotherMsg() ? Using q.put( (HandleAnotherMsg, 1, 2, 3) ) will rise a compilation error :(

Answer 1

Why don't you subclass Queue?

class MyQueue(Queue):
  # by using *args, you can have a variable number of arguments
  def put(self,*args):
    for arg in args:
       Queue.put(self,arg)

or, why don't you put a list?

list = [function_obj]
for arg in function_args:
   list.append(arg)
queue.put(list)

Answer 2

So simple:

def DestinationThread( a ) :
  while True :
    items = q.get()
    func = items[0]
    args = items[1:]
    func(*args)

Answer 3

from Queue import *
from thread import *
from time import *

q = Queue()

def HandleMsg( arg1, arg2 ) :
  print arg1, arg2

def HandleAnotherMsg( arg1, arg2, arg3 ) :
  print arg1, arg2, arg3

def DestinationThread() :
  while True :
    f, args = q.get()
    f(*args)

start_new_thread( DestinationThread, tuple() )
print "start"
sleep( 1 )
q.put( (HandleMsg, [1, 2]) )
sleep( 1 )
q.put( (HandleAnotherMsg, [1, 2, 3]) )
sleep( 1 )
print "stop"

Answer 4

It sounds like you want to use the apply() intrinsic or its successor:

def f(x. y): print x+y

args = ( 1, 2 )

apply(f, args) # old way

f(*args) # new way

Answer 5

I've used a similar construct before:

class Call:
    def __init__(self, fn, *args, **kwargs):
        self.fn = fn
        self.args = args
        self.kwargs = kwargs

    def __call__(self):
        return self.fn(*self.args, **self.kwargs)


x = Call(zip, [0,1], [2,3], [4,5])

You should then be able to pass x to your other thread and call it from there:

x() # returns the same as zip([0,1], [2,3], [4,5])

Answer 6

You can create an abstract message class with a run method. Then for each function that need to be transmitted via the queue, subclass and implement the function as the run method. The sending thread will create an instance of the proper sub class and put it into the queue. The receiving thread will get an object from the queue and blindly execute the run mehtod.

This is usually called the Command pattern (Gamma et al.)

Example:

class Message (object):
    """abstract message class"""
    def __init__(self, **kwargs):
        self.kwargs = kwargs

    def run(self):
        pass


class MessageOne (Message):
    """one message class"""
    def run(self):
         # perform this emssage's action using the kwargs

The sender will instantiate and send a message:

queue.put(MessageOne(one='Eins', two='Deux'))

The receiver simply gets a message object and execute it run method (without having to it..else.. thru the available types of messages):

msg = queue.get()
msg.run()

Answer 7

Another interesting option is simply to pass in a lambda.

q.put(lambda: HandleMsg(1,2))
q.put(lambda: HandleAnother(8, "hello", extra="foo"))

def DestinationThread() :
   while True :
      f = q.get()
      f()