RPC for multiprocessing, design issues

Question

Hey everyone, what's a good way to do rpc across multiprocessing.Process'es ?

I am also open to design advise on the following architecture: Process A * 10, Process B * 1. Each process A has to check with proces B on whether a particular item needs to be queried.

So I was thinking of implementing multiprocessing.Pipe() object for all the As, and then have B listen to each of them. However, I realize that Multiprocessing.Pipe.recv is BLOCKING. so I don't really know how I can go about doing this. (if I use a loop to check which one has things sent through the other end that the loop will be blocked).

There are suggestions for me to use twisted, but I am not sure how I should go about doing this in twisted: Should I create a defer on each pipe.handler from all the processes A and then when recv() receives something it goes on and complete a certain routine? I know personally twisted does not mix well with multiprocessing, but I have done some testing on twisted that are child processes of an multiprocessing implementation and I think this time it's workable.

Any recommendations?

Answer 1

Personally, I always tend to lean towards socket-based RPC, because that frees me from the confine of a single node if and when I need to expand more. Twisted offers a great way to handle socket-based communications, but of course there are other alternatives too. HTTP 1.1 is a great "transport" layer to use for such purposes, as it typically passes firewalls easily, is easily migrated into HTTPS if and when you need security, too. As for payloads over it, I may be somewhat of an eccentric for favoring JSON, but I've had a great time with it compared with XML or many other encodings. Though I have to admit that, now that Google's protobufs have been open-sourced, they're tempting too (especially as they ARE what we use internally, almost exclusively -- one sure gets used to them;-). Pity no specific RPC implementation of protobufs over HTTP has been open-sourced... but it's not THAT hard to cook up one for yourself;-).

Answer 2

I'm satisfied with using REST-ful transaction design.

This means using HTTP instead of pipes.

If Process B has a queue of things for the various Process A's to do, it would work like this.

Process B is an HTTP server, with RESTful URI's that handle queries from process A's. B is implemented using Python wsgiref or werkzeug or some other WSGI implementation.

Mostly, B responds to GET requests from A. Each GET request takes the next thing off the queue and responds with it. Since B will have multiple concurrent requests, some kind of single-threaded queue is essential. The easiest way to assure this is to assure that the WSGI server is single-threaded. Each request is relatively fast, so single-threaded processing works out quite nicely.

B has to have it's queue loaded, so it probably also responds to POST requests to enqueue things, too.

Process A is an HTTP client, making requests of the RESTful URI's that Process B provides. A is implemented using urllib2 to make requests of B. A makes GET requests of B to get the next thing from the queue.

Answer 3

You know Windows ATOM API. Its use involves the development of controls, but you can use one's own for IPC, see D-Bus.

Answer 4

Have you looked at MPI? http://en.wikipedia.org/wiki/Message%5FPassing%5FInterface.

It is broadly available on UNIX/Linux/etc. I believe it can be had on Windows. Basically it provides all the plumbing that you'll have to build on top of RPC mechanisms, and it has many years of development and refinement behind it. It is a spec for an API, originally done in C so works with C++ too, and there are Python implementations of it out there too.