Linux/C++ How to debug release application

Question

Hello, I have linux c++ multithreaded application. Now it's tested on production servers and have segfault. Problem is that I cannot reproduce that error on any of my test servers and have no access to production servers. I have no dump or any other useful information. Only line: segfault at 0000000046bf0fb8 rip 000000000048ac6b rsp 0000000046bf0fa0 error 6

I would like to ask community can I get from such line some information which will help decrease area of possible places where I should search. I cannot run debug build on production because of its slow speed. What can I add to release which help me debug? This bug looks like multithread bug, and hard to reproduce. But I'm not sure, because application works with a lot of different emails from MTA.

Platform: Linux

Compiler line: g++ -O3 -D_REENTRANT

Thank you.

upd.: Thanks for your answers. I can include debug information. I'd like to know base methods of debugging release builds. For example I have dump and release version. How should I continue. What should I read about that? Can you explain in few words how you debug your application if possible? Thank you.

Answer 1

I've been reading the gdb manuals recently, and they recommend leaving the debugging symbols in e.g. g++ -g.

Since you don't have access to the production server maybe include some basic logging functionality that'll output a data to a text file. You should be able to narrow down roughly where the error occurs, depending on which data has been output to your log file.

Answer 2

As Andy mentioned, leave the debugging symbols in when you make your release builds.

If this makes the size of the finished executable unacceptably large, then you can make a copy of the final executable and run it through strip to remove the debug symbols. This way you have two executables that are identical except one has debug symbols and the other does not. Put the one without symbols on the production server. When it segfaults, debug against the copy of the executable that still contains debug symbols.

Answer 3

You can (and should) build release executables with debug information. If you don't want to distribute executables containing the debug information, then you can separate the debug information and install it later for debugging. That's what we do in our application.

Answer 4

You can use gdb to get a backtrace of your program at the point where it segfaults even though you did not build your application with the debug flags. This will at least give you an idea where your application segfaults.

gdb <your_app_exe>
gdb> run
gdb> backtrace

or

gdb <your_app_exe>
gdb> core-file <generated_core_file>

Answer 5

well i found another solution, which i m using very frequently, we normally get the stack (which we got in this case).

i have a executable which we deploy on some embedded platform. let say my executable is server. i use addr2line -e ./server and i paste the stack which i got from customer. it will give u the details of line where problem occur.

it might help you.

Thanks