views:

361

answers:

5

I'm developing an application that runs on a small Linux-based SBC (~32MB RAM). Sadly, my app recently became too large to run under GDB anymore. Does anyone know of any good, lightweight debugging methods that I can use in embedded Linux? Even being able to view a thread's stack trace would be extremely helpful.

I should mention that this application is written in C++ and runs multiple threads, so gdbserver is a no-go as it doesn't work with multithreaded apps.

Thanks in advance,

Maha

+2  A: 

I've heard of people doing hacks like running the application in an emulator like QEMU and then running GDB (or things like valgrind) on that. It sounds painful, but if it works....

Would you get anywhere with libunwind (to get stack traces) and printf-style logging?

Brooks Moses
Thanks for the pointers. I looked at running under an emulator, but that's definitely a painful way to go and would probably eat up more time than I can spend.I'm doing printf-style logging right now, but this is a fairly complex application and it's sometimes tricky to figure out exactly which component's causing a problem in the first place. Libunwind definitely looks like a helpful tool, I'll give it a shot.
Maha
+1  A: 

Serial port printing is the most light weight I can think of ~~~ Easily seen in a Host PC, and simple and light weight code inside your app~~

If you do not have a serial port, once we used an GPIO port and simulated a serial port using it. It worked perfectly well, but was a bit slow :-( ~~~

Alphaneo
+3  A: 

gdbserver definitely works with multi-threaded applications, I'm working on an embedded project right now with >25 threads and we use gdbserver all the time.

info threads

lists all the threads in the system

thread <thread number from info threads>

switches to that thread of execution.

thread apply XXX <command>

Runs on the thread designated by XXX, which can also be 'all'. So if you want the back trace from all running threads do

thread apply all bt

Once you're in the execution flow of a given threads all your typical commands work as they would in a single threaded process.

Andrew Myers
Do you have to run gdb/gdbserver with special arguments? I'm running on an ARM processor. When I run 'gdbserver localhost:12345 myapp' and then run the same version of gdb on my host and enter the command 'target remote 10.0.150.92:12345', the debugger gets confused as it thinks only one thread is running (breaks with every context switch , and 'info threads' reports only 1 running thread).
Maha
I don't have to run with special arguments when I debug, our project is also on an ARM. My process for debugging remotely sounds the same as yours.On target: gdbserver localhost:10000 myappOn host: arm_v5t-le-gdb myappOn host gdb command line: target remote <target ip>:<port>By same version of gdb you mean the arm build right? Is there any reason your app would be getting signals such as SIGUSR1/2 or something during context switches? That will cause the debugger to stop. The application on both the target and host must be built with debug symbols, we NFS mount the host for that.
Andrew Myers
My host machine is an x86 system, and the target system runs an ARM processor. Is your host is also an ARM system? If not, maybe I missed something in my GDB build (I built GDB 7.0 for ARM, then built it separately for x86). My app's definitely not generating SIGUSR1/2--I've verified that it's breaking on context switches as the debugger thinks only one thread is running.
Maha
My host machine is also x86. You need gdbserver built as an arm-linux application as you did, however you can't use a normal gdb build on the host to remotely debug the application. The host side must be built to run on x86 but load and debug an ARM binary, it's a sort of hybrid application. To be honest I'm not sure what the complete build process is, my company bought our tool-chain from Montavista. Googling turned up http://www.gnuarm.com/home.html which looks like it may have precompiled binaries of what you're looking for, I can't download it myself right now though.
Andrew Myers
Thanks for your help. I'll try to get GDB built with host-target debugging enabled--I've played with a few of the configure options but I haven't quite gotten a configuration that works. I'll post my configure options when I get it working.
Maha
Might also be worth giving a prebuilt version from http://www.codesourcery.com/sgpp/lite/arm a try -- I'm pretty sure the GDB from that will work fine even if you're not using the compiler from it. (Disclaimer: I work for them.)
Brooks Moses
A: 

Is there a reason why you have built your own debugger? I am developing a Linux system using an ARM processor (AT91SAM926x) and we are using both compiler and debugger from CodeSourcery. I do not think that they have released a version with GDB 7 yet but I am debugging multithreaded C++ applications using the gdbserver tool without any problems.

Anders Holmberg
We're using a toolchain supplied by our SBC manufacturer. Unfortunately they don't supply a pre-built GDB, so I'm on my own.
Maha
One thing you could try is to build an older version of GDB. GDB 7 is very new and I have read some reports of major bugs (not ARM related though). We are running version 6.7.
Anders Holmberg
A: 

Gdbserver does indeed work with multithreaded applications. However you do need to compile a cross target debugger for your host to get it to work with your target gdb.

See this article for a detailed description of how to do it:

Remote cross-target debugging with GDB and GDBserver

mikelong
Thanks for the link. I followed the instructions there but they didn't work for me. Trying to build the arm-native binaries with his method failed--make dies, complaining that the C compiler can't generate executables. I don't know why it does this, as GDB's build process is usually smart enough not to test executables it generates when cross-compiling.
Maha