Solaris Core dump analysis

Question

I use pstack to analyze core dump files in Solaris

How else can I analyze the core dump from solaris?

What commands can be used to do this?

What other information will be available from the dump?

Answer 1

GDB can be used.

It can give the call that was attempted prior to the dump.

http://sourceware.org/gdb/

http://en.wikipedia.org/wiki/GDB

Having the source is great and if you can reproduce the errors even better as you can use this to debug it.

Worked great for me in the past.

Answer 2

You can use Solaris modular debugger,mdb, or dbx. mdb comes with SUNWmdb (or SUNWmdb x for the 64 bits version) package.

A core file is the image of your running process at the time it crashed.

Depending on whether your application was compiled with debug flags or not,you will be able to view an image of the stack, hence to know which function caused the core, to get the value of the parameters that were passed to that function, the value of the variables, the allocated memory zones ...

On recent solaris versions, you can configure what the core file will contain with the coreadm command ; for instance, you can have the mapped memory segment the process were attached to.

Refer to MDB documentation and dbx documentation. The GDB quick reference card is also helpful once you know the basics of GDB.

Answer 3

Ben Rockwood has just published a two part article on this topic :

http://cuddletech.com/blog/pivot/entry.php?id=965

http://cuddletech.com/blog/pivot/entry.php?id=966

Answer 4

The pflags command is also useful for determining the state each thread was in when it core dumped. In this way you can often pinpoint the problem.

Answer 5

Attach to the process image using the dbx debugger:

dbx [executable_file_name] [coredump_file_name]

It is important that there were no changes to the executable since the core was dumped (i.e. it wasn't rebuilt).

You can see the stack trace to see where the program crashed with dbx command "where".

You can move up and down the stack with command "up" and "down", or jump to the exact stack frame with "frame [number]", with the numbers seen in the output of "where".

You can print the value of variables or expressions with "print [expr]" command.

Have fun.

Answer 6

If the core dump is from a program you wrote or built, then use whichever debugger you would normally use to debug the running application. They should all be able to load core files. If you're not picky about debuggers, and you're using Solaris, I would recommend dbx. It will help to get the latest FCS version of Sun Studio with patches, or else the latest Express version of Sun Studio. It's also very helpful if you can load the core file into the debugger on the same system where the core file was created. If the code in the libraries is different from when the core file was created, then stack trace won't be useful when it goes through libraries. Debuggers also use OS helper libraries for understanding the libthread and runtime linker data structures, so IF you need to load the core file on a different machine, you'll want to make sure the helper libraries installed on the OS match the system data structures in the OS. You can find out everything you never wanted to know about these system libraries in a white paper tha twas written a few years ago.

http://developers.sun.com/solaris/articles/DebugLibraries/DebugLibraries_content.html

Answer 7

I guess any answer to this question should start with a simple recipe:

For dbx, the recipe is:

% dbx a.out core
(dbx) where
(dbx) threads
(dbx) thread t@3
(dbx) where