Python multiprocessing/threading with shared variables that only will be read

Question

Considering the code below. I would like to run 3 experiments at a time. The experiments are independent, the only thing they share is the Model object which they only read.

As there are seemingly no hard things in threading this out, how can I best do this in Python? I would like to use a pool or so to make sure that only three experiments run at a time. Shall I use multi-processing? If yes, how is the shortest and most concise?

#!/usr/bin/env python2.6
import time

class Model:
    name = ""
    def __init__(self,name):
        self.name = name

class Experiment:
    id = 0
    model = None
    done = False

    def __init__(self,id,model):
        self.id = id
        self.model = model

    def run(self):
        for _ in range(0,60):
            print "Hey %s from experiment %d" % (self.model.name, id)
            time.sleep(1)
        self.done = True


if __name__ == "__main__":
    experiments = []
    model = Model("statictistical model")
    for i in range(0,5):
        experiments.append(Experiment(i, model))

    #How to run 3 experiments at the same time

Answer 1

You must always keep in mind that threads do not really run parallel, if that is what you actually want.

I do not unterstand what the actual problem is...? If you only want 3 threads running the same time, why not just start only 3 threads?

Answer 2

Check the docs, specifically:

http://docs.python.org/library/multiprocessing.html#module-multiprocessing.pool

There really are a lot of examples there that should get you on your way. For instance, I could come up with:

#!/usr/bin/env python2.6
import time
import multiprocessing

class Model:
    name = ""
    def __init__(self,name):
        self.name = name

def run_experiment(id, model):
    print "Experiment %d is starting" % id
    for _ in range(0,60):
        print "Hey %s from experiment %d" % (model.name, id)
        time.sleep(1)
    print "Experiment %d is done" % id
    return "Result for %d" % id


if __name__ == "__main__":
    model = Model("statictistical model")
    experiments = ((i, model) for i in range(0, 5))
    pool = multiprocessing.Pool(3)

    results = [pool.apply_async(run_experiment, experiment) for experiment in experiments]
    for result in results:
        r = result.get()
        # do something with r
        # or nothing, i suppose...

Do also pay attention to what the docs say about using the multiprocessing module:

Functionality within this package requires that the __main__ method be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.Pool examples will not work in the interactive interpreter