Explain Python extensions multithreading

Question

Python interpreter has a Global Interpreter Lock, and it is my understanding that extensions must acquire it in a multi-threaded environment. But Boost.Python HOWTO page says the extension function must release the GIL and reacquire it on exit.

I want to resist temptation to guess here, so I would like to know what should be GIL locking patterns in the following scenarios:

1. Extension is called from python (presumably running in a python thread).
2. And extension's background thread calls back into Py_* functions.

And a final question is, why the linked document says the GIL should be released and re-acquired?

Answer 1

Whenever Python is interpreting bytecode the GIL is being held by the currently running thread. No other Python thread can run until it manages to acquire the GIL.

When the interpreter has called into native code that code has two options regarding the GIL:

1. It could do nothing at all.
2. It could release the GIL while it works and then reacquire it before returning to Python

If the native code makes a lot of calls back to Python's runtime it should take option 1: you can never call Python's runtime safely unless you hold the GIL (with a few exceptions such as making the call to acquire the GIL if you don't have it).

If the native code does a lot of work which doesn't involve Python in any way then you can use option 2: as soon as you release the GIL Python will be able to schedule another Python thread so you get some parallelism. If you don't release the GIL then none of Python's other threads can execute while your Boost code is running: that's why the docs tell you to release and reacquire the GIL.

If you go this way then you must be careful to make all your access to Py_* functions before you release the GIL or after you reacquire it. This may mean you have to make local copies of data as you cannot safely access Python data types such as list or dictionary elements while the GIL is released.

If your Boost code needs to call back into Python while the GIL is released then you need to acquire the GIL, make the call, release the GIL. Try to avoid making such calls from threads that weren't created by Python as they need additional work to be able to acquire the GIL.