What does the GDB backtrace message "0x0000000000000000 in ?? ()" mean?

Question

What does it mean when it gives a backtrace with the following output?

#0  0x00000008009c991c in pthread_testcancel () from /lib/libpthread.so.2
#1  0x00000008009b8120 in sigaction () from /lib/libpthread.so.2
#2  0x00000008009c211a in pthread_mutexattr_init () from /lib/libpthread.so.2
#3  0x0000000000000000 in ?? ()

The program has crashed with a standard signal 11, segmentation fault. My application is a multi-threaded FastCGI C++ program running on FreeBSD 6.3, using pthread as the threading library.

It has been compiled with -g and all the symbol tables for my source are loaded, according to info sources.

As is clear, none of my actual code appears in the trace but instead the error seems to originate from standard pthread libraries. In particular, what is ?? () ????

EDIT: eventually tracked the crash down to a standard invalid memory access in my main code. Doesn't explain why the stack trace was corrupted, but that's a question for another day :)

Answer 1

Make sure you compile with debug symbols. (For gcc I think that is the -g option). Then you should be able to get more interesting information out of GDB. Don't forget to turn it off when you compile the production version.

Answer 2

gdb wasn't able to extract the proper return address from pthread_mutexattr_init; it got an address of 0. The "??" is the result of looking up address 0 in the symbol table. It cannot find a symbolic name, so it prints a default "??"

Unfortunately right offhand I don't know why it could not extract the correct return address.

Answer 3

Maybe the bug that caused the crash has broken the stack (overwritten parts of the stack)? In that case, the backtrace might be useless; no idea what to do in that case...

Answer 4

Something you did cause the threading library to crash. Since the threading library itself is not compiled with debugging symbols (-g), it cannot display the source code file or line number the crash happened on. In addition, since it's threads, the call stack does not point back to your file. Unfortunately this will be a tough bug to track down, you're gonna need to step through your code and try and narrow down when exactly the crash happens.

Answer 5

I could be missing something, but isn't this indicative of someone using NULL as a function pointer?

#include <stdio.h>

typedef int (*funcptr)(void);

int
func_caller(funcptr f)
{
    return (*f)();
}

int
main()
{
    return func_caller(NULL);
}

This produces the same style of a backtrace if you run it in gdb:

rivendell$ gcc -g -O0 foo.c -o foo
rivendell$ gdb --quiet foo
Reading symbols for shared libraries .. done
(gdb) r
Starting program: ...
Reading symbols for shared libraries . done

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000000
0x00000000 in ?? ()
(gdb) bt
#0    0x00000000 in ?? ()
#1    0x00001f9d in func_caller (f=0) at foo.c:8
#2    0x00001fb1 in main () at foo.c:14

This is a pretty strange crash though... pthread_mutexattr_init rarely does anything more than allocate a data structure and memset it. I'd look for something else going on. Is there a possibility of mismatched threading libraries or something. My BSD knowledge is a little dated, but there used to be issues around this.