Dynamically loading two libpython versions

Question

I have a program which embeds both python2 and python3 interpreters. The libpython shared libraries are dlopen()ed by the respective commands which provide access to the interpreters and each interpreter maintains its own state.

This all works just fine if the user only uses pure python modules or builtins. Trying to load a C extension (like termios) then complains "undefined symbol: PyExc_TypeError". This happens because the C extensions aren't linked against libpython. Python upstream doesn't think this is a problem.

To get around that, I can change the dlopen() calls in my program for the libpython shared libraries to use RTLD_GLOBAL. As soon as I do that, however, trying to use both the python2 and python3 interpreters in the same session of the program causes it to ABRT in the process of calling Py_Initialize for whichever interpreter was invoked second. Using only one of the interpreters works fine.

Any idea how to get this to work when the C extensions won't be linked against libpython, therefore requiring the use of RTLD_GLOBAL?

Answer 1

Sorry, but this won't work the way you want it to. The solution ordinarily would involve linking each extension to versioned libpython symbols; or one could have a namespace-capable linker, such that one could map each library to a different namespace, rather than a global one. Unfortunately neither of these options are easily applied, so you're probably stuck with a multi-process model. Simply fork and have one process link to each version of Python. The tough bit then is how to share whatever data led you to require two distinct Python interpreters in the first place. Perhaps a description of what problem led to the question may help find a better solution?