ansaurus

Question

Answer 1

+1 A:

cPython is not thread safe. That is the purpose of the GIL, which must be used whenever accessing or modifying the interpreter state.

If you need threading and python, then you will need to use an implementation other than cPython (the standard one), such as IronPython or Jython, both of which are perfectly robust in the case of threading. There are some modified cPython's such as Stackless python that might work better as well.

TokenMacGuy 2010-10-22 18:52:55

So are you saying c extensions simply don't work in multithreaded environment? Or that you need to fiddle around with the GIL to make it work?

c00kiemonster 2010-10-22 18:57:15

I've never really attempted to get a multithreaded extension to work with cPython, so I don't know the right answer, but yes, you will need to fiddle with the GIL if you wish to interact with the interpreter state from multiple threads.

TokenMacGuy 2010-10-22 19:02:17

@c00kiemonster "which must be used whenever *accessing or modifying* the interpreter state"

pst 2010-10-22 19:03:22

This answer does not pass my sniff test. The question was about a "simple" C extension being called in a multithreaded Python environmnt. This just says to me that the C code is not written to be reentrant. No reason to start throwing out "can't use cPython" without much more evidence. Question to poster: does your extension use global variables?

Bill Gribble 2010-10-24 11:13:49

Answer 2

A:

I finally managed to get rid of the problem, but in a pretty long winded way. Here goes.

I spent a very very long time trying to make sense of the documentation for c extensions and their thread-safeness. On one of the many google trajectories that night I stumbled on this page describing how to use numpy arrays in c extensions. Since my problems seemed to be performance related (the original c extension worked for smaller datasets) I suspected that my implementation of looping through the python lists and using PyList_GetItem to get the data into their c array counterparts was not up to scratch. (I deduced the following actual number crunching in the c extension wasn't the issue since it was very generic c without any special stuff at all.)

Hence I decided to undertake a complete rewrite of the c extension and my calling python script to use numpy arrays instead of lists. It took a good two days including all the debugging. But now it works like a charm. All datasets are processed ok, there are no signs of any bus errors nor segmentation faults.

TLDR: Use numpy arrays instead of python lists when working with large datasets and python c extensions to avoid bus errors and segmentation faults.

c00kiemonster 2010-10-24 11:04:05

ansaurus

tags:

views:

answers:

Python C extension not threadsafe?

related questions