Debugging a segv

Question

I am sure a lot of you have experienced this. Sometimes while debugging a crash you add a line of trace and viola the crash disappears. This generally points to a memory corruption and you look for buffer overflows etc around the code. To be more specific the same occurs for even single threaded applications.

How does though a line of trace prevent the crash from occurring?

The trace line is a piece of code so should go to the code segment which is as it is read only right?

It should have no impact on the local stack of execution? How does it impact pointer offsets to make some other piece of memory to get over written which doesn't manifest it self(still bad).

Answer 1

Please see here for my answer to another SO question. I am not 100% certain what environment you are running on but I guess it is one of the unix variants... The link I have given demonstrates a simple trick to trap the SIGSEGV handler by wiring up to the signal handler and anytime during execution of the code, the handler gets intercepted and dumps the stack trace to a file for reading.

Hope this helps you in tracking down the problem, Best regards, Tom.

Answer 2

First off, a trace can affect not just the code segment but the data segment if the trace includes some sort of string. This can completely change the layout of your memory.

If your segfault is being caused because you are dereferencing some random memory, the specific value may have changed because of how the data has been shifted around. Where the old memory layout caused you to read an invalid memory address, the new memory layout now points into something valid.

One recommendation I have is to stop debugging by adding traces and use a debugger instead.

Answer 3

A couple of folks have already mentioned timing changes. Even if your code is single-threaded, you might be calling API that either starts threads on your behalf, or implements some other form of asynchronous communication.

As far as this statement is concerned:

The trace line is a piece of code so should go to the code segment

That depends on what your tracing code actually does. Even something as simple as:

write(0, "Hello, World!\n", 14);

Will at the least add additional data storage for the constant string, which will shift the location of other constant data, and possibly change the start of other memory segments, affecting the location of code and/or heap, as well. If this is the first reference to that function, it'll add a relocation record to the object file, which might cause other shifts in the memory layout.

A more-complex tracing call that calls fwrite or printf, for example, will almost certainly end up allocating some memory for temporary buffers, etc.

If you can reproduce the bug (with the original code, obviously) when running in the debugger, then you should be able to at least determine which code is triggering the segfault. If the debugger also causes the bug not to happen, you might consider enabling core dumps, and debugging based on the dump.

Failing that, you can try running the code with a debugging malloc package (there are several available). Even if "the bug" doesn't reproduce, you'll likely find some improper pointer manipulation that way.

Last suggestion: Re-compile your code with all warnings enabled on the compiler, and seriously consider all of the warnings so generated. Note that with gcc, -Wall does not turn on all warnings.