Dynamic linking and Python SWIG (C++) works in C++ fails in python

Question

I have a library for which I have created a python wrapper using SWIG. The library itself accepts user provided functions which are in an .so file that is dynamically linked. At the moment I'm dealing with one that I have created myself and have managed to get the dynamic linking working... in C++. When I attempt to run it in python I get undefined symbol errors. These symbols are ones that are not present in the provided .so file but are present in the main program (essentially they are the functions that allow the provided module to access data from the main program).

I do not get any errors running a short test program in C++, but a short test program in python with this wrapper (that worked previously) fails. I can't think of an explanation as to why it would fail in C++ and not in the python. What worries me slightly is the idea that the C++ isn't working properly but isn't telling me, and the python is picking up errors that the C++ isn't. Yet the result returned by the C++ is accurate, so this seems unlikely.

Any thoughts how this is possible and hence how I could fix it?

Thanks.

Update: I have added this code to the top of the program:

import dl
sys.setdlopenflags(dl.RTLD_NOW | dl.RTLD_GLOBAL)

This gets rid of the runtime error, but unfortunately allows a second problem to arise (still due to linking). The functions that are being called from within the dynamically linked library that are part of the main program are not returning the correct values. They are returning 0. What's more it's evident that they are not even being run at all. The question becomes what is actually being run, why is it different to the C++, and how do I fix THAT?

Thanks again.

Update- A potentially clearer explanation Python imports a module, which is my C++ library that has been wrapped by SWIG. This C++ library uses dlopen and dlsym to obtain functions from a user provided .so file. The user provided file calls to functions that are part of the C++ library in order to do it's job. The function calls from the .so file to the C++ library are the part that is failing, that is they fail to call the function and simply return 0. However this failure only occurs when the test code is written in python. C++ test code that uses the library works fine.

Answer 1

The python interpreter is probably loading your wrapper .so without making it's symbols available to other dynamic link libraries (to avoid symbol conflicts). Try adding the following lines just before importing your wrapper:

import dl
sys.setdlopenflags(dl.RTLD_NOW | dl.RTLD_GLOBAL)

Answer 2

A solution is to make sure that python is preloading the C++ main library in the global scope. This is not a very elegant solution, and I don't want to do it, but it makes it work for the moment.

After a little poking around here and recognising the LD_LIBRARY_PATH environment variable that I have to set every time I start the terminal in order for it to even find the main C++ library that has been SWIGed, I noticed the LD_PRELOAD environment variable. Upon setting this to the filename of the main C++ library the program worked.

I suspect this is because it "can be used to selectively override functions in other shared libraries".

If anyone comes up with a better answer than setting environment variables it'd be awesome, as I'm not sure how portable this is.

Edit: The original problem is that the functions that the user provided library is looking for are not in the global scope. In order to fix this simply using python's "dl.open" to open the main library's .so file, using dl.RTLD_NOW and dl.RTLD_GLOBAL.

Success!