How to implement a master/watchdog script in python?

Question

I need it to open 10 processes, and each time one of them finishes I want to wait few seconds and start another one. It seems pretty simple, but somehow I can't get it to work.

Answer 1

I've done this same thing to process web statistics using a semaphore. Essentially, as processes are created, the semaphore is incremented. When they exit, it's decremented. The creation process is blocked when the semaphore blocks.

This actually fires off threads, which run external processes down execution path a bit.

Here's an example.

    thread_sem = threading.Semaphore(int(cfg.maxthreads))
    for k,v in log_data.items():
        thread_list.append(ProcessorThread(int(k), v, thread_sem))
        thread_list[-1].start()

And then in the constructor for ProcessorThread, I do this:

    def __init__(self, siteid, data, lock_object):
        threading.Thread.__init__(self)
        self.setDaemon(False)
        self.lock_object = lock_object
        self.data = data
        self.siteid = siteid
        self.lock_object.acquire()

When the thread finishes it's task (whether successfully or not), the lock_object is released which allows for another process to begin.

HTH

Answer 2

I'm not 100% clear on what you're trying to accomplish, but have you looked at the multiprocessing module, specifically using a pool of workers?