Multiprocessing launching too many instances of Python VM

Question

I am writing some multiprocessing code (Python 2.6.4, WinXP) that spawns processes to run background tasks. In playing around with some trivial examples, I am running into an issue where my code just continuously spawns new processes, even though I only tell it to spawn a fixed number.

The program itself runs fine, but if I look in Windows TaskManager, I keep seeing new 'python.exe' processes appear. They just keep spawning more and more as the program runs (eventually starving my machine).

For example,
I would expect the code below to launch 2 python.exe processes. The first being the program itself, and the second being the child process it spawns. Any idea what I am doing wrong?

import time
import multiprocessing


class Agent(multiprocessing.Process):
    def __init__(self, i):
        multiprocessing.Process.__init__(self)
        self.i = i

    def run(self):
        while True:
            print 'hello from %i' % self.i
            time.sleep(1)


agent = Agent(1)
agent.start()

Answer 1

When I run this in Linux with python2.6, I see a maximum of 4 python2.6 processes and I can't guarantee that they're all from this process. They're definitely not filling up the machine.

Need new python version? Linux/Windows difference?

Answer 2

I don't see anything wrong with that. Works fine on Ubuntu 9.10 (Python 2.6.4).

Are you sure you don't have cron or something starting multiple copies of your script? Or that the spawned script is not calling anything that would start a new instance, for example as a side effect of import if your code runs directly on import?

Answer 3

It looks like you didn't carefully follow the guidelines in the documentation, specifically this section where it talks about "Safe importing of main module".

You need to protect your launch code with an if __name__ == '__main__': block or you'll get what you're getting, I believe.

I believe it comes down to the multiprocessing module not being able to use os.fork() as it does on Linux, where an already-running process is basically cloned in memory. On Windows (which has no such fork()) it must run a new Python interpreter and tell it to import your main module and then execute the start/run method once that's done. If you have code at "module level", unprotected by the name check, then during the import it starts the whole sequence over again, ad infinitum