Detect if a process is already running and collaborate with it.

Question

I'm trying to create a program that starts a process pool of, say, 5 processes, performs some operation, and then quits, but leaves the 5 processes open. Later the user can run the program again, and instead of it starting new processes it uses the existing 5. Basically it's a producer-consumer model where:

1. The number of producers varies.
2. The number of consumers is constant.
3. The producers can be started at different times by different programs or even different users.

I'm using the builtin multiprocessing module, currently in Python 2.6.4., but with the intent to move to 3.1.1 eventually.

Here's a basic usage scenario:

1. Beginning state - no processes running.
2. User starts program.py operation - one producer, five consumers running.
3. Operation completes - five consumers running.
4. User starts program.py operation - one producer, five consumers running.
5. User starts program.py operation - two producers, five consumers running.
6. Operation completes - one producer, five consumers running.
7. Operation completes - five consumers running.
8. User starts program.py stop and it completes - no processes running.
9. User starts program.py start and it completes - five consumers running.
10. User starts program.py operation - one procucer, five consumers running.
11. Operation completes - five consumers running.
12. User starts program.py stop and it completes - no processes running.

The problem I have is that I don't know where to start on:

1. Detecting that the consumer processes are running.
2. Gaining access to them from a previously unrelated program.
3. Doing 1 and 2 in a cross-platform way.

Once I can do that, I know how to manage the processes. There has to be some reliable way to detect existing processes since I've seen Firefox do this to prevent multiple instances of Firefox from running, but I have no idea how to do that in Python.

Answer 1

Take a look at these different Service Discovery mechanisms: http://en.wikipedia.org/wiki/Service_discovery

The basic idea is that the consumers would each register a service when they start. The producer would go through the discovery process when starting. If it finds the consumers, it binds to them. If it doesn't find them it starts up new consumers. In most all of these systems, services can typically also publish properties, so you can have each consumer uniquely identify itself and give other information to the discovering producer.

Bonjour/zeroconf is pretty well supported cross-platform. You can even configure Safari to show you the zeroconf services on your local network, so you can use that to debug the service advertisement for the consumers. One side advantage of this kind of approach is that you could easily run the producers on different machines than the consumers.

Answer 2

There are a couple of common ways to do your item #1 (detecting running processes), but to use them would first require that you slightly tweak your mental picture of how these background processes are started by the first invocation of the program.

Think of the first program not as starting the five processes and then exiting, but rather as detecting that it is the first instance started and not exiting. It can create a file lock (one of the common approaches for preventing multiple occurrences of an application from running), or merely bind to some socket (another common approach). Either approach will raise an exception in a second instance, which then knows that it is not the first and can refocus its attention on contacting the first instance.

If you're using multiprocessing, you should be able simply to use the Manager support, which involves binding to a socket to act as a server.

The first program starts the processes, creates Queues, proxies, or whatever. It creates a Manager to allow access to them, possibly allowing remote access.

Subsequent invocations first attempt to contact said server/Manager on the predefined socket (or using other techniques to discover the socket it's on). Instead of doing a server_forever() call they connect() and communicate using the usual multiprocessing mechanisms.

Answer 3

You need a client-server model on a local system. You could do this using TCP/IP sockets to communicate between your clients and servers, but it's faster to use local named pipes if you don't have the need to communicate over a network.

The basic requirements for you if I understood correctly are these:
1. A producer should be able to spawn consumers if none exist already.
2. A producer should be able to communicate with consumers.
3. A producer should be able to find pre-existing consumers and communicate with them.
4. Even if a producer completes, consumers should continue running.
5. More than one producer should be able to communicate with the consumers.

Let's tackle each one of these one by one:

(1) is a simple process-creation problem, except that consumer (child) processes should continue running, even if the producer (parent) exits. See (4) below.

(2) A producer can communicate with consumers using named pipes. See os.mkfifo() and unix man page of mkfifo() to create named pipes.

(3) You need to create named pipes from the consumer processes in a well known path, when they start running. The producer can find out if any consumers are running by looking for this well-known pipe(s) in the same location. If the pipe(s) do not exist, no consumers are running, and the producers can spawn these.

(4) You'll need to use os.setuid() for this, and make the consumer processes act like a daemon. See unix man page of setsid().

(5) This one is tricky. Multiple producers can communicate with the consumers using the same named pipe, but you cannot transfer more than "PIPE_BUF" amount of data from the producer to the consumer, if you want to reliably identify which producer sent the data, or if you want to prevent some kind of interleaving of data from different producers.

A better way to do (5) is to have the consumers open a "control" named pipe (/tmp/control.3456, 3456 being the consumer pid) on execution. Producers first set up a communication channel using the "control" pipe. When a producer connects, it sends its pid say "1234", to the consumer on the "control" pipe, which tells the consumer to create a named pipe for data exchange with the producer, say "/tmp/data.1234". Then the producer closes the "control" pipe, and opens "/tmp/data.1234" to communicate with the consumer. Each consumer can have its own "control" pipes (use the consumer pids to distinguish between pipes of different consumers), and each producer gets its own "data" pipe.. When a producer finishes, it should clean up its data pipe or tell the consumer to do so. Similarly, when the consumer finishes, it should clean up its control pipes.

A difficulty here is to prevent multiple producers from connecting to the control pipes of a single consumer at the same time. The "control" pipe here is a shared resource and you need to synchronize between different producers to access it. Use semaphores for it or file locking. See the posix_ipc python module for this.

Note: I have described most of the above in terms of general UNIX semantics, but all you really need is the ability to create daemon processes, ability to create "named" pipes/queues/whatever so that they can be found by an unrelated process, and ability to synchronize between unrelated processes. You can use any python module which provides such semantics.