communication with long running tasks in python

Question

I have a python GUI app that uses a long running function from a .so/.dll it calls through ctypes.

I'm looking for a way to communicate with the function while it's running in a separate thread or process, so that I can request it to terminate early (which requires some work on the C side before returning a partial result). I suppose this will need some kind of signal receiving or reading from pipes, but I want to keep it as simple as possible.

What would you consider the best approach to solve this kind of problem? I am able to change the code on both the python and C sides.

Answer 1

If you're on *nix register a signal handler for SIGUSR1 or SIGINT in your C program then from Python use os.kill to send the signal.

Answer 2

You said it: signals and pipes.

It doesn't have to be too complex, but it will be a heck of a lot easier if you use an existing structure than if you try to roll your own.

Answer 3

There's two parts you'll need to answer here: one if how to communicate between the two processes (your GUI and the process executing the function), and the other is how to change your function so it responds to asynchronous requests ("oh, I've been told to just return whatever I've got").

Working out the answer to the second question will probably dictate the answer to the first. You could do it by signals (in which case you get a signal handler that gets control of the process, can look for more detailed instructions elsewhere, and change your internal data structures before returning control to your function), or you could have your function monitor a control interface for commands (every millisecond, check to see if there's a command waiting, and if there is, see what it is).

In the first case, you'd want ANSI C signal handling (signal(), sighandler_t), in the second you'd probably want a pipe or similar (pipe() and select()).

Answer 4

You mention that you can change both the C and Python sides. To avoid having to write any sockets or signal code in C, it might be easiest to break up the large C function into 3 smaller separate functions that perform setup, a small parcel of work, and cleanup. The work parcel should be between about 1 ms and 1 second run time to strike a balance between responsiveness and low overhead. It can be tough to break up calculations into even chunks like this in the face of changing data sizes, but you would have the same challenge in a single big function that also did I/O.

Write a worker process in Python that calls those 3 functions through ctypes. Have the worker process check a Queue object for a message from the GUI to stop the calculation early. Make sure to use the non-blocking Queue.get_nowait call instead of Queue.get. If the worker process finds a message to quit early, call the C clean up code and return the partial result.