Realtime processing and callbacks with Python and C++

Question

I need to write code to do some realtime processing that is fairly computationally complex. I would like to create some Python classes to manage all my scripting, and leave the intensive parts of the algorithm coded in C++ so that they can run as fast as possible. I would like to instantiate the objects in Python, and have the C++ algorithms chime back into the script with callbacks in python. Something like:

myObject = MyObject()
myObject.setCallback(myCallback)
myObject.run()

def myCallback(val):
    """Do something with the value passed back to the python script."""
    pass

Will this be possible? How can I run a callback in python from a loop that is running in a C++ module? Anyone have a link or a tutorial to help me do this correctly?

Answer 1

Have a look at Boost.Python. Its tutorial starts here.

Answer 2

Here is an example of how to do python callbacks using Cython. It might be worth looking at Pyrex as well. Both can make integrating C/C++ with Python rather easy.

Answer 3

We do what you are doing a lot at work. We like python but it's just not fast enough sometimes.
Boost and Swig are both good for doing that. You should also check out this link on Python Performance they talk a little about NumPy which may help you.

Answer 4

I suggest using Boost.Python as suggested by ChristopheD. A gotcha would be if the C++ extension is running in it's own thread context (not created by Python). If that's the case, make sure to use the PyGILState_Ensure() and PyGILState_Release() functions when calling into Python code from C++.

From the docs (http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock):

Beginning with version 2.3, threads can now take advantage of the PyGILState_*() functions to do all of the above automatically. The typical idiom for calling into Python from a C thread is now:

PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

/* Perform Python actions here.  */
result = CallSomeFunction();
/* evaluate result */

/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate)

I recommend making the callbacks short & sweet - to limit the need to perform exception handling in C++ code. If you're using wxPython, you could use it's robust async event system. Or the callbacks could put events on a Queue and you could have a thread devoted to asynchronously executing callback/event code.

Even with Boost.Python magic, you'll have to get familiar with this portion of the Python C API when dealing with threads. (Don't forget to wrap the C++ functions with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS to release the GIL!)