Python threads in embedded Python: How?

Question

While experimenting with Python's (python.org) C API, I found myself wondering how to properly spawn threads via Python's threading package when Python itself is embedded in a C program. Functions PyEval_EvalCode and kin appear to terminate threads it "owns" as soon as the C function finishes evaluating a block of Python code. Roughly, running the following Python from C ...

import threading, time

class MyThread(threading.Thread):
    def __init__(self, num):
        self.num = num
        threading.Thread.__init__(self)
    def run(self):
        print "magic num = %d" % self.num

for x in xrange(1000):
    MyThread(x).start()

... will stop entirely as soon as the for loop finishes and control is returned back out of the PyEval_EvalCode (or such) C function. We can observe this produces truncated output.

I hypothesized this behavior after using the following tactic: We can dictate when control is returned, and therefore to a degree the output, by sleeping after spawning the slew of threads:

for x in xrange(100):
    MyThread(x).start()

# Don't leave just yet; let threads run for a bit
time.sleep(5) # Adjust to taste

I suspect a possible approach lies in creating a new system thread dedicated to embedding and running Python. After spawning Python threads, the Python code would sleep on a semaphore or something until it is told to shut down. The question would then be how can the thread be signaled for a tidy shutdown? Similarly, the "main" block can simply join() on all threads; the threads would then need to be signaled from C.

Solutions greatly appreciated ..

Answer 1

Have you included the pthread library? Python will fall back to dummy threads if it detects that real threads are not available