Python multiprocessing with twisted's reactor

Question

Dear everyone I am working on a xmlrpc server which has to perform certain tasks cyclically. I am using twisted as the core of the xmlrpc service but I am running into a little problem:

class cemeteryRPC(xmlrpc.XMLRPC):

    def __init__(self, dic):
        xmlrpc.XMLRPC.__init__(self)


    def xmlrpc_foo(self):
        return 1


    def cycle(self):
        print "Hello"
        time.sleep(3)


class cemeteryM( base ):

    def __init__(self, dic):   # dic is for cemetery
        multiprocessing.Process.__init__(self)
        self.cemRPC = cemeteryRPC()


    def run(self):
        # Start reactor on a second process
        reactor.listenTCP( c.PORT_XMLRPC, server.Site( self.cemRPC ) )
        p = multiprocessing.Process( target=reactor.run )
        p.start()

        while not self.exit.is_set():
            self.cemRPC.cycle()
            #p.join()


if __name__ == "__main__":

    import errno
    test = cemeteryM()
    test.start()

    # trying new method
    notintr = False
    while not notintr:
        try:
            test.join()
            notintr = True 
        except OSError, ose:
            if ose.errno != errno.EINTR:
                raise ose
        except KeyboardInterrupt:
            notintr = True

How should i go about joining these two process so that their respective joins doesn't block?

(I am pretty confused by "join". Why would it block and I have googled but can't find much helpful explanation to the usage of join. Can someone explain this to me?)

Regards

Answer 1

Do you really need to run Twisted in a separate process? That looks pretty unusual to me.

Try to think of Twisted's Reactor as your main loop - and hang everything you need off that - rather than trying to run Twisted as a background task.

The more normal way of performing this sort of operation would be to use Twisted's .callLater or to add a LoopingCall object to the Reactor.

e.g.

from twisted.web import xmlrpc, server
from twisted.internet import task
from twisted.internet import reactor

def runEverySecond():
    print "a second has passed"

class Example(xmlrpc.XMLRPC):          
    def xmlrpc_add(self, a, b):
        return a + b

    def timer_event(self):
        print "one second"

r = Example()
m = task.LoopingCall(r.timer_event)
m.start(1.0)

reactor.listenTCP(7080, server.Site(r))
reactor.run()

Answer 2

Hey asdvawev - .join() in multiprocessing works just like .join() in threading - it's a blocking call the main thread runs to wait for the worker to shut down. If the worker never shuts down, then .join() will never return. For example:

class myproc(Process):
    def run(self):
        while True:
            time.sleep(1)

Calling run on this means that join() will never, ever return. Typically to prevent this I'll use an Event() object passed into the child process to allow me to signal the child when to exit:

class myproc(Process):
    def __init__(self, event):
        self.event = event
        Process.__init__(self)
    def run(self):
        while not self.event.is_set():
            time.sleep(1)

Alternatively, if your work is encapsulated in a queue - you can simply have the child process work off of the queue until it encounters a sentinel (typically a None entry in the queue) and then shut down.

Both of these suggestions means that prior to calling .join() you can send set the event, or insert the sentinel and when join() is called, the process will finish it's current task and then exit properly.