How do you share data between a parent and forked child process in Python?

Question

I'm pretty sure one would do this using the os.plock(op) function, but I have no idea how. Also, if there's a better way, I'd be grateful to find out. Code snippets are very welcome. Thanks!

Answer 1

This is pretty much Python-independent! It's a classic example of Unix interprocess communication. One good option is to use popen() to open a pipe between the parent and child processes, and pass data/messages back and forth along the pipe.

Take a look at the subprocess module, which can set up the necessary pipes automatically while spawning child processes.

Answer 2

You have two options: os.popen* in the os module, or you can use the subprocess module to the same effect. The Python manual has pretty documentation and examples for popen and subprocess.

Answer 3

Subprocess replaces os.popen, os.system, os.spawn, popen2 and commands. A simple example for piping would be:

p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

You could also use a memory mapped file with the flag=MAP_SHARED for shared memory between processes.

multiprocessing abstracts both pipes and shared memory and provides a higher level interface. Taken from the Processing documentation:

from multiprocessing import Process, Pipe

def f(conn):
    conn.send([42, None, 'hello'])
    conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(child_conn,))
    p.start()
    print parent_conn.recv()   # prints "[42, None, 'hello']"
    p.join()

Answer 4

Take a look at the multiprocessing module new in python 2.6 (also available for earlier versions a pyprocessing

Here's an example from the docs illustrating passing information using a pipe for instance:

from multiprocessing import Process, Pipe

def f(conn):
    conn.send([42, None, 'hello'])
    conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=f, args=(child_conn,))
    p.start()
    print parent_conn.recv()   # prints "[42, None, 'hello']"
    p.join()