pthread with callback to python VM

Question

Let's say I have a python script which loads a shared library (SL) through ctypes.

• The SL sets up a pthread T1
• The python script configures callbacks through the SL i.e. python script calls functions from the SL with references to python callables

Now, let's say T1 calls a "callback" function, are the following assumptions true:

• the callback function on the Python side is executed within the context of T1
• I can use a queue to communicate between T1 and the Python VM
• I would need to poll the said queue on the Python VM side

I understand all the concepts of threading, shared state etc. but I haven't dug very deep on the multi-threading side with Python. Since there is an adaptation layer which I do not know enough at the moment (ctypes), I am afraid I'll be missing some key aspects of the process.

Answer 1

Polling the queue isn't normally necessary (you can devote another thread on the Python side to doing blocking .get calls on it), but that's not a big deal. Problem is, with such an arrangement, you might get caught by the GIL -- see the three links from this wikipedia page for ample treatments thereof.

When you interface to/from C with C (or Cython) code using the Python C API, you can release and acquire the GIL pretty simply, at least, hopefully avoiding deadlocks and the like; with ctypes, GIL operations are automated in callback to/from C situations, so if there's any other lock in play a deadlock is a risk (since things are not within your control you can't easily ensure Djikstra's Banker Algorithm is applied).