Python multiprocessing handling sessions

Question

I have a script receiveing data from a socket, each data contains a sessionid that a have to keep track of, foreach incomming message, i'm opening a new process with the multiprocessing module, i having trouble to figure out a way to keep track of the new incoming messages having the same sessionid. For example:

100100|Hello --

(open a new process)

100100|Hello back at you

(proccess replies)

100101|Hello

(open a new process)

100101|Hello back at you

(new proccess replies)

100100|wasap? --

(open a new process)

when a new message from sessionid 100100 comes... how could i sent it to the process that is handling those particular messages?

until know the main process is opening a new process for each incomming message another process is writing data on the socket, but is giving me real trouble finding out a way to handle each session process and sending data to them...

I need some guidance cause a never work with multiprocessing before...

Thanks...

Answer 1

How about storing the processes in a dictionary? E.g. in pseudo code:

def __init__(self):
    self.sessions = {}
def handle(self, sessionid, data):
    proc = self.sessions.get(sessionid)
    if prod is None:
        proc = self.create_process()
        self.sessions[sessionid] = proc
    proc.handle(data)

Answer 2

I am not sure about the requirement to open a separate process for every message received from the sockets. I guess, you have a reason for doing all that you have mentioned.

My understanding is, you have written a server side socket that listens for client connection, accepts the client connection, receives the data and dispatch these data to various processes you have created to handle these messages.

I am assuming that session id remains same for a client connection. Each client sends data tagged with it's session id. If this is the case, then you can solve easily by the fact that in unix - forks duplicate file / socket descriptors. when you launch a multiprocessing.Process(), you can pass it the socket descriptor. Ensure that you close the socket in parent process or the process in which you listen for client connections. This way each process will handle the connection and will receive only messages for the single client tagged with the same session id.

I hope this answers your need.

Answer 3

To communicate with processes created suing multiprocessing you can use the classes Queue and Pipe (also from the multiprocessing module). Here is a short example of using a Queue to send a message to a process:

from multiprocessing import Process, Queue

def f(q):
    print 'f(), waiting...'
    print q.get()

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    q.put('Hello from main!')
    p.join()

More informatin can be found in the Python docs.