Problems replacing a Python extension module while Python script is executing

Question

I'm trying to solve the following problem: Say I have a Python script (let's call it Test.py) which uses a C++ extension module (made via SWIG, let's call the module "Example"). I have Test.py, Example.py, and _Example.so in the same directory.

Now, in the middle of running Test.py, I want to make a change to my Example module, recompile (which will overwrite the existing .so), and use a command to gracefully stop Test.py which is still using the old version of the module (Test.py has some cleaning up to do, which uses some stuff which is defined in the Example module), then start it up again, using the new version of the module. Gracefully stopping Test.py and THEN recompiling the module is not an option in my case.

The problem is, as soon as _Example.so is overwritten and Test.py tries to access anything defined in the Example module (while gracefully stopping), I get a segmentation fault. One solution to this is to explicitly name the Example module by appending a version number at the end, but I was wondering if there was a better solution (I don't want to be importing Example_1_0)?

Answer 1

What you could is compile to the temporary name (Example_1_0) but as soon as the Test.py is stopped, rename to _Example.so and then restart Test.py.

---

Edit:

Since you're running multiple instances, you might consider using some type of stack/generator/symlink threading to link it all together to do your own "garbage collection" on _Example.so:

You could have the main tester script running, launching the Test.py scripts with subprocess. Each Test.py could take _ExampleXXX.so as a command line argument. Then you keep a reference count to each .so file - when the refcount drops to zero, that version of the module is eliminated and the process is respawned with the newest version of _Example.so.

It might be a bit tricky, but you may just be able to use

while True:
    #Do stuff
    for p in myprocesses:
        retcode = p.poll() # Set to [None][1] if the process hasn't finished
        # Do something with the return code

or some similar logic.

Answer 2

You could, on starting Test.py, copy the Example.* files to a temp folder unique for that instance (take a look at tempfile.mkdtemp, it can create safe, unique folders), add that to sys.path and then import Example; and on Test.py shutdown remove that folder (shutils.rmtree) at the cleanup stage.

This would mean that each instance of Test.py would run on its own copy of the Example module, not interfering with the others, and would update to the new one only upon relaunch.

You would need the Example.* files not to be on the same folder as Test.py for this, otherwise the import would get those first. Just storing them on a subfolder should be fine.