Debugging techniques

Question

Debugging is the most time consuming activity of programming. So using appropriate tools and techniques is paramount to efficiency and productivity.

What are your favorite debugging techniques, and in which cases do you apply each of them?

There are many orthogonal criteria to consider:

• Programming languages (tools usually are language specific, and there are maybe some techniques that are applicable only within specific languages)
• Applications (web applications, client side, server side, singlethreaded versus multithreaded)
• Environment (how many tiers, is it on a stored procedure, is it on an embedded device)

But I'm mainly interested on techniques (and tools, if there are any) which are generally applicable and that you find most useful for finding and squashing bugs.

Answer 1

The most simple way would be to "print" them, or set breakpoints and inspect data. I find Eclipse's possibility to "eval" code when the debugger hits a breakpoint, very useful.

Answer 2

Well,

Let me be the first to mention GDB, the GNU debugger. A very handy utility for debugging C++ and some other languages. It's pretty universal, runs on any system (including Windows with Cygwin) and is very well documented.

Answer 3

The one that I've used for the longest time and is pretty much universal:

When in doubt, print more out.

Answer 4

I'd say that a thorough automated test suite is the best debugging tool in any language. A logging engine is a good second tool. Then the integrated debugger for your language/IDE of choice... which must have at least variable inspection, conditional breakpoints and remote debugging.

Answer 5

I find the best debugging session is one that never happens. If you employ a really good Test First (TDD) mentality then debugging doesn't have to be time consuming at all. If you write tests before you write your code then you will always have coverage, and your tests will direct your attention to the exact location where the problem is.

Debugging really becomes much less of a hassle because you simply have to run your tests, find the one that broke, and then you can use whatever built in debugging your IDE offers.

Answer 6

Tests & Testing!

I hope I'm not stating the obvious here.

Answer 7

When I can't isolate a problem with standard methods I dig out my old watcom C++ compiler, build the program or the relevant part with it and fire on the watcom debugger.

Why? Once you hit a breakpoint or get an exception that masterpiece of a debugger lets you step back in time and see what caused the problem at the first place.

Works for C, C++ and Fortran.

Answer 8

Well, if you have a reasonable idea of how the code works, the symptoms of the bug should give you a good idea of where to look and what to look for. Then it's just a matter of reading the code intelligently, to find out what exactly the problem is.

If this is unbearably cumbersome, one or two well-placed text output statements should usually give you the information you need.

I used to be a big fan of "stepping through the code" using whatever debugger is available for the criteria you listed, but I'm starting to lean towards this reading approach now - the reason being the sentence you opened your question with:

Debugging is the most time consuming activity of programming.

It seems to me that simplifying the whole process (by removing all the tools) and applying your mind instead, speeds things up a lot. This effect is probably caused by the fact that having so many tools makes me too lazy to think for myself.

Answer 9

Conditional breakpoints

Many people don't use them. Conditional breakpoints cut down debugging time by filtering out breaks that don't fit the scenario you're investigating.

Answer 10

This is my approach to finding bugs/debug:

1. Stay away from the debugger.
2. Make up your own theory of where/what the problem is.
3. Get as much data about the problem (reconstruct the scenario that made the bug appear)
4. Try out your theory.
5. Talk to other developers (rubber ducking might help)
6. Start using the debugger, and if you can't solve the problem in a short while, go to point 1.

Answer 11

Usually I use this approach with a debugger.

1. Track down the buggy code portion to the exact line
2. See what is causing the problem. Its usually one of the variables which have a wrong value
3. Think: Why did the variable have a wrong value. Come out with a theory which explains it.
4. Test your theory. If it works, bug solved.

More often than not, its the theory thinking part that is crucial. It is all about pattern recognition. Ask, why would it cause this output. All bugs have a pattern to it. Whenever I come across bugs which I can't even think of a theory, you got to give yourself more clues. Introduce other values into your function. See what it produces. More often than they can shed some clues on where the problem lies. Keep going until you get more clues. Usually that does the trick.

Sometimes I feel that the mental state of the programmer is important too. If you feel tired, take a break. If you feel tense, take a break. Believe me, your mind works away while you're having a break. Happy debugging!

Answer 12

(Note: Unix specific)

When it comes to tools, here are some that can help in particular situations:

• System call tracing (ktrace on BSD, strace on Linux, dtruss on Mac OS X, truss on Solaris)

• dtrace, if you're lucky enough to have a system that supports it; learn it! alternatively, use Instruments on
  Mac OS X which uses dtrace in the background. But the amount of stuff you can do with dtrace (using its D language), is awesome.

• lsof, list information about open files.

• tcpdump, for network debugging

There are a lot more tools that doesn't come to mind right now, but these are tools that should be in every POSIX programmers arsenal.

Answer 13

I find that whatever I'm debugging - be it C, C++, Perl or even HTML or CSS - if I pare down the source to the absolute bare minimum (or even transplant the erroneous section into a minimal program or page), the issue quickly becomes apparent.

Answer 14

One technique I'm surprised hasn't been mentioned is to make a call graph. E.g. In the area of problem, what functions call other functions. Make that into a diagram, and it becomes easier to navigate the code and have a holistic view of things.

I wouldn't recommend this for the simple bugs, as the time it takes is too much, but for more complicated issues it is helpful.

I would also add that doing this by hand (if time allows) is very useful to gaining an understanding of the code in question.

Answer 15

A technique I use for debugging code written by others:

Another technique which should be used only carefully is to do minor refactoring of the buggy area. Usually, I have found that areas with bugs are needlessly complex, so simplifying them incrementally helps in two ways: it is preventative (e.g. less bugs are likely to be in code that is straightforward) and it aids in understanding the code better. Often when I do this, I also find I need to rename variables or functions because they were not named in a manner that indicated the previous coder understood the problem in question!

Ultimately, the key is to understand the code in question.

Answer 16

Some techniques I use for debugging field issues (e.g. stuff that needs to be resolved quickly) is:

1. Analyze the debug log of the application (e.g. something that logs unexpected exceptions like null reference exceptions). Ideally, the stack trace info is in the log. In .Net if the PDB file is also distributed, the line number where the exception occurred is also provided.
2. Attempt to Reproduce it. Or If that fails, read the code to see if the failure scenario is obvious. This allows me to identify the function in question.
3. See if I can find the input data into the function that is failing. If so, does it need to be scrubbed/normalized? If so, add that code. Does it need additional scenarios to be handled? If so, add that code.

Answer 17

Tools for deployment problems:

The SysInternals tools are useful for debugging deployment problems on Windows. Procmon allows you to see what process (and callstack) is touching files or the registry. Procexp will allow you to search for open handles, and inspect which modules are loaded in the process.

Answer 18

Find a simple consistent way to reproduce the bug first!

Many problems are difficult because there isn't a simple way to consistently reproduce the bug. If you have a long but consistent way to reproduce the bug, then spend time trying to make a short and consistent way to reproduce the bug.

If you have an inconsistent way to reproduce the bug then try to figure out how to make it consistent.

(Use all the tools you want. Including a debugger, modifying code, whatever. Just realize you are attempting to discover clues to help make a way to consistently reproduce the bug.)

If you don't have a way to reproduce the bug then any fix is a shot in the dark. You really don't know if you've fixed anything.

Answer 19

Source history is helpful.

Any significant problem will cross out of code you own. Source history helps find who owns what in a large code base.

It also helps in reading code that you didn't write, to get it in small purposeful chunks via the diffs for a checkin.

Finally if what you are tracking is a recent regression, then binary searching back to identify the bad checkin can work.

Answer 20

Debugging is usually very contextual, based on the problem you are debugging. Some broad tips are

1. Try to isolate the problem to figure out which line of code, function fails.
2. Check all the variables, see if they hold incorrect values.
3. Dangling memory, memory overwrite etc can be tracked using a memory breakpoint. Some debuggers support this and I know Visual Studio does.
4. Get to know the machine architecture, and calling conventions of methods. This will be really useful when figuring out how data is passed across functions.

Answer 21

Learn to use your debugger well, use conditional breakpoints. Dynamic breakpoints (as in NuMega SoftIce/W) or as in JavaScript debugger calls (or Win32 DebugBreak()) help you break into a debugger when you hit that rare/impossible situation. So, it also helps to have debugging in mind when you code complex applications.

It also helps to know how to "look" at log files (for server based apps). I use find, grep, xargs, tail, screen (new tool) to help me get my job done. You can do a lot by using a scripting language.

Above all, understanding what is happening is important: your brain is the best tool that you can possess.

Having access to a nice editor (XEmacs works for me) which can let you filter the view of a log file can make a big difference :-)

Answer 22

Be purposeful.

Some advice here seems to be 'don't use a debugger at all'. This is based on devs tending to grab a debugger and step around without purpose.

That is of course bad, however the 'solution' of avoiding useful tools is worse. Please do use every tool available ... just understand what you are trying to do with the tool.

Answer 23

If I have a good understanding of the code, say, like the back of my hand, then I'll start to rationalize and track down any source of the problem in my head. I'll come to a conclusion and then go test my hypothesis. If I find the bug or problem and fix it without creating another problem, great. If I don't fix it with one swoop, then I'll continue using the aid of the built in debugger.

If I'm not familiar with the code I'll try to think of where the most logical place for the problem to be. I'll start testing and prodding to find out where it is, then I'll try to fix it.

I have to say that my favorite technique to use is using the breakpoint in Visual Studio. Its such a handy little thing that I forget how helpful it can really be until I run into some extraneous result.

Answer 24

1. I think about invariants for each function I write.

2. I put a bunch of assertions in my code to test the invariants. I put them in to check that the program really does what I believe it should be doing. I use asserts() even with relatively time-consuming conditions.

3. I investigate every assertion failure. About 15% of my assert conditions are wrong, the rest are true program bugs. With the remaining 15%, I have learned something new.

4. I check for impossible. I have a macro to wrap every system call with, and that macro (usually) aborts the program if the "impossible" should happen [e.g. failing to lock a mutex]. Obviously, if the failure is expected (e.g. seeing invalid input while reading a file), I use normal if() to handle it.

5. Crashing is good. The sonner my program crashes due to some bug, the closer it will be to the source, and the easier it will be to find out the cause of the bug from a core-dump or a repeated run.

6. Logging and tracing facilities to debug nondeterministic events (e.g. clients arriving, receiving signals, etc.)

7. Last resort: single-stepping with a debugger. When I find the problem the hard way, I put additional assertions into the code.

The last problem I spent much time debugging was a now-corrected bug in Solaris signal handling: sometimes the signal was not delivered on the alternate stack (established by sigaltstack()) and the signal context consequently overwrote my local data (I was coding preemptive task scheduler in user-mode). Nasty.

A previous problem that took a lot of my time was a compiler optimizer bug where generated code accessed a global variable through an unitialized pointer. "Fun"!

Otherwise, I spent very little time in debugger -- most of the problems are discovered early with assertions. And even though I code in C++, somehow I manage to keep out of trouble with pointers and memory allocation (I don't remember the last time I had a memory leak, or reading from freed memory, etc.) that was actually my fault.

Answer 25

For small embedded systems: Always figure out a way to communicate debug messages out from the systems. For example sending RS232 data, debug printouts on an alphanumeric display, blink codes using a LED, toggling a bit somewhere which can be monitored with a oscilloscope or logic analyzer.

If possible, request this possibility from the hardware designers early on in design process.

Answer 26

I am currently reading Why Programs Fail: A Guide to Systematic Debugging by Andreas Zeller. I can only highly recommend this book. It is a really seminal overview about debugging theory and practice. What I liked most so far is Chapter 6 on "Scientific Debugging".

Answer 27

My debugging technique is reasonably simple:

• Establish what the correct behaviour should be.
• Reproduce the incorrect behaviour (the bug).
• Devise a theory to explain the bug.
• Make a prediction based on the theory.
• Devise an experiment to test the prediction.
• Run the experiment and observe the results.
• Interpret the results of the experiment.
• When appropriate, fix the bug.
• Devise an experiment to test the bug fix.
• Run the experiment and observe the results.
• Interpret the results of the experiment.

The hard part comes when it's difficult or impossible to reproduce the bug. That's where skill, hard-earned experience, and luck come into play.

Answer 28

Using printf() is a perfectly legitimate way to debug. Yes there are other ways. You can use interactive debuggers such as gdb.

If you are on Windows, grab Microsoft Visual Studio Express. It's free and has one of the best interactive debuggers around.

An interactive debugger will let you step through code line-by-line, examine the contents of variables and set break-points to stop execution at specific points. Life would be real hard without these tools.

Answer 29

You should use a debugger. There are several out there: GDB, Visual Studio, Eclipse, XCode, etc.

Answer 30

Sure, using a Debugger to set Breakpoints and then using the Watch-Function to have a look at the state of your program.

Under Windows, windbg and the Visual Studio Debugger would come into my mind, under Linux I think gdb used to be one, not sure what the current state is.

What is your Operating System and C Compiler?

Answer 31

printf debugging has its place of course, it's not necessarily a bad technique. But there are LOTS of better ways:

http://stackoverflow.com/questions/91527/debugging-techniques

Answer 32

Your essential debugging tools should include...

• Logging: sounds like you have a handle on this already :)

• Breakpoints: tell the program to stop on a given line of code if and when it gets there

• Callstack: ability to see the chain of function calls that lead to the current point as well as the parameters they were passed

• Locals: ability to see what variables are defined in the current scope and what their values are

• Watches: ability to evaluate an expression in the debugger and see what it's value is

• Disassembly: see what the compiled instructions corresponding to your code at a specific point are

• Registers: see the state of the CPU at a given point in your program

Some advanced types of breakpoints:

• Conditional Breakpoints: stop on a breakpoint only if a given conditional expression is satisfied

• Memory Breakpoints: given a memory address and a number of bytes (the size of the value that you care about), break if the value stored at that address changes

Visual Studio provides all of these capabilities. The GNU tool for debugging is gdb and will provide most or all of these abilities as well.

Answer 33

Yes, there is:

#ifdef DEBUG
#define DBG x printf x
#else
#define DBG x do {} while (0)
#endif

Vinko gave a excellent link to the discussion about debugging.
Stay away from the debugger is good advice. In 90% of the cases the debugger just slow you down rather helping you.

The debugger will not help you debug the problem discovered by the customer and not reproducible on your setup, well designed logging system will do that.

printf is not necessary an answer, but using a macro you can easily design your logging system to use fprintf or whatever you need.

I would also suggest to make the DEBUG macro more advanced and have level and module parameters that will control the verbosity of your output.

In response to This CarlOS post:

It is a bad practice that production code in release mode produce debug output. Just assume that I debug my code while your code is running and I'm using DebugView or tail -f /var/log/messages to see the debug output of my program and I will get tons of messages from your code. Be nice to other developers; control your code output :)

But you can instead make the function into a macro and control your debug output at run time:

void my_printf(int section, int level, ...)
{
    if (global_debug_off)
        return; 
    //do you stuff here
}

Using a function instead just if(debug) printf ... will give you flexibility to change your output device easily for example from stderr to a file or network or whatever.

Answer 34

When it comes to C I usually create a DPRINTF macro during development that expands to a printf call if DEBUG is defined (using C99 variadic macros):

#ifdef DEBUG
# define DPRINTF(...) fprintf(stderr, __VA_ARGS__);
#else
# define DPRINTF(...)
#endif

Then I insert DPRINTF statements as needed for debugging and then comment them out when I'm done with them inserting a note describing what I put them in there for. Before the production release I go back and review all of the debugging print statements and translate some of them into logging statements.

Other debugging tools I use include gdb, ltrace, strace, and lint although for the vast majority of the bugs I encounter, the printf debugging method is usually considerably faster.

Answer 35

If you are debugging, you know something horribly confused you. Don't do it! Step back, write down invariants and understand what you are doing.

assert is a nice way to accomplish this.

Answer 36

There are tools to debug, which are generally specific to particular platforms, and then there are ways to debug, which are more generic approaches common to lots of problem sets.

I recommend a read of Debugging by David Agans

Answer 37

If you are unlucky enough to have either a multi-threaded program or a multi-process systems (or multi-process system consisting of multi-threaded programs) where those programs do different things depending on timing interactions, then using a debugger is really hard, and variations on the debugging print statements may be necessary. Sometimes, even those can change the timing characteristics enough to change the behaviour of the program -- leading to 'heisenbugs'. (You can get heisenbugs without needing multi-threading or multi-processing; those just make them more likely.)

Answer 38

In response to Ilya: I was thinking something like this

typedef void(* debugproto)  (char *string);

void dummy (char *string)
{
}

void debug(char *string)
{
    //printf or something like output to a file.
    printf("%s",string);
}

debugproto global_debug_proc = dummy;

...
if (global_debug)
    global_debug_proc = debug;
...
global_debug_proc("Yay debug mode on!!!");

This way it will only check for global_debug once, but I don't know if it's OK to do that.

Answer 39

1. Log Trace messages
2. Log warnings
3. Log errors

Have a configurable switch to turn the amount of logging. No debugger can substitute this. It also helps you to debug issues in other environments where you cannot attach debugger (like QA, Staging , Prod etc

Just like programming, logging is an art

Answer 40

I work in the mobile phone industry and we often encounter situations that an on-PC software simulator simply won't simulate. So we have to put the application on the physical phone and hope we can replicate the issue that is being investigated.

Here are the conditions: we don't have a debugger and we don't always get synchronous logging messages (oh, and the OEM throws in a ton of extra messages that can't be turned off so that you'll have some next extra reading to do).

Since the logging system is asynchronous, by the time you suspect you should have gotten a message, the phone will have crashed. So we have to try alternate ways of getting info, a lot of which have already been mentioned, but here are a few extra:

• File logging: print our logs to file; this is problematic in that writing to a file may cause more problems or may simply be too slow. Not always available (not all phones have file system).
• Memory management: put checks in to ensure there aren't any leaks near the area of suspicion. Also consider low or fragmented conditions.
• Assertions: Mentioned before, but worth mentioning again. I've broken code on purpose before just to understand the expected behavior. Sometimes it offers an indirect view of the issue.
• "When in doubt, print more out," as somebody mentioned above. Despite the asynchronous nature of the logs, sometimes enough data will reveal a pattern that can be extrapolated into an idea for a solution.
• Consider the network conditions; time of day, weekday or weekend, which operator, etc.

When push comes to shove, it all comes down to process of elimination. What elements in the code can be controlled enough so that a pattern might emerge?

Answer 41

NO JOKE - TALK TO A ROCK

Many of the above answers are fantastic and I've made my vote. But this is no joke. When all else fails, we have a pet rock that we talk to and explain to it (him, her?) what the program is supposed to be doing.

When it gets to thAT point we get a beer but it works. The rock can make us feel like morons (for explaining to a rock why our program doesn't work)

Answer 42

In PHP, the

<pre>

HTML tag, along with

var_dump()

and

print_r()

are your friends.

Answer 43

In .NET, using the $exception keyword in the Immediate Window.

I often have the misfortune to debug code like the following

try {
   //something that fails
} catch {
   //do nothing
}

With a breakpoint in the catch block you can get access the exception by typing in $exception

Answer 44

When debugging C++ code with Visual Studio I frequently put "eax" into my watch window to see the return of the last function.

Answer 45

My debugging techniques, tips and tricks are documented at my blog at http://www.technochakra.com/.

It covers assembly language debugging, using hit count breakpoints beyond their limits, reducing the data set and other such notes, GDB, Visual Studio stuff, etc. Give it a read if you see something you like.