Concurrency: Are Python extensions written in C/C++ affected by the Global Interpreter Lock?

Question

One of Python's strongest points is the ease of writing C and C++ extensions to speed up processor intensive parts of the code. Can these extensions avoid the Global Interpreter Lock or are they also restricted by the GIL? If not, then this "ease of extension" is even more of a killer feature than I previously realized. I suspect the answer is not a simple yes-or-no but I am not sure, so I am asking the question here on StackOverflow.

Answer 1

C/C++ extensions to Python are not bound by the GIL. However, you really need to know what you're doing. From http://docs.python.org/c-api/init.html:

The global interpreter lock is used to protect the pointer to the current thread state. When releasing the lock and saving the thread state, the current thread state pointer must be retrieved before the lock is released (since another thread could immediately acquire the lock and store its own thread state in the global variable). Conversely, when acquiring the lock and restoring the thread state, the lock must be acquired before storing the thread state pointer.

Why am I going on with so much detail about this? Because when threads are created from C, they don’t have the global interpreter lock, nor is there a thread state data structure for them. Such threads must bootstrap themselves into existence, by first creating a thread state data structure, then acquiring the lock, and finally storing their thread state pointer, before they can start using the Python/C API. When they are done, they should reset the thread state pointer, release the lock, and finally free their thread state data structure.

Answer 2

Yes, calls to C extensions (C routines called from Python) are still subject to the GIL.

However, you can manually release the GIL inside your C extension, so long as you are careful to re-assert it before returning control to the Python VM.

For information, take a look at the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros: http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock

Answer 3

Here's a long article I wrote for python magazine that touches on the c-extension/GIL/threading thing. It's a bit long at 4000 words, but it should help.

http://jessenoller.com/2009/02/01/python-threads-and-the-global-interpreter-lock/

Answer 4

Check out Cython, it has similar syntax to Python but with a few constructs like "cdef", fast numpy access functions, and a "with nogil" statement (which does what it says).