Embedding Python thread safe

views:

answers:

+1 Q:

Embedding Python thread safe

Hi everyone,

I'm trying to use Python in a module for an analysis software of vehicle bus systems. For this I have to embed Python in a thread safe manner, since there can be multiple instances of the module witch work independently. I could use a mutex to guard all access to Python and create an unique (python) module for every thread. Obviously this is the easiest way out but comes with the price of not being able to scale across multiple cores. Or I could modify my module to spawn new processes which intern uses Python and connect to them via shared memory. This gives me a performance penalty and costs more time to implement but scales great.

My question: witch one do you think makes more sense? Is there any other way to embed Python thread safe or even in a way that scales over multiple cores.

Kind regards Moritz

edit: I'm using CPython

+3 A:

If you're CPU bound, Python can only scale to multiple core using the multiprocessing library. However, if you're I/O bound, then threading is usually sufficient.

If you want easy thread safety, then use Queue for all message passing.

Lie Ryan 2010-10-15 14:25:00

does the multiprocessing library realy help me? I don't use multiple threads within one Python interpreter, I want to use multiple Python interpreter running in multiple ( C++ ) threads.

JustMaximumPower 2010-10-15 14:32:27

The multiprocessing library can handle spawning subprocess (i.e. it spawns multiple python interpreter) and interprocess-communication (e.g. data sharing, synchronization), and in doing so, avoids the GIL (Global Interpreter Lock). I'd suggest on looking at it before trying to spawn multiple interpreters on C++ threads (which will replicate most of what they do in the multiprocessing library).

Lie Ryan 2010-10-15 14:39:24

Again I don't want sup processes from within Python I want, if at all, to spawn new processes from C. So are you suggesting that I look into the Multiprocessing source code? Or how does the multiprocessing library help me? I'm not trying to be cocky, but I don't see how your answer helps me.

JustMaximumPower 2010-10-19 07:55:57

You are already seeing the solution before seeing the problem. There is no difference between subprocess spawned by python and subprocess spawned by C. You just need a slightly different way of dividing the tasks, instead of dividing the tasks in C, you now divide tasks in Python. Also, since you're writing for a bus, take extra care to ensure that you are actually using multiprocessing since the library will silently fallback to threading if it thinks it cannot work with the bus' system.

Lie Ryan 2010-10-19 11:38:27

To follow-up on my question: I went ahead in implemented it using Processes with intern use Python. A good text why the multiprocessing library does not help can be found here: http://pkaudio.blogspot.com/2010/04/whey-multiprocessing-doesnt-always-work.html It was not written by myself but that guy has the same problem as I have. I'm thankful for everybody who tried to help me.

JustMaximumPower 2010-10-22 08:37:07

ansaurus

tags:

views:

answers:

Embedding Python thread safe

related questions