How to debug a program without a debugger?

Question

Interview question-

Often its pretty easier to debug a program once you have trouble with your code.You can put watches,breakpoints and etc.Life is much easier because of debugger.

But how to debug a program without a debugger?

One possible approach which I know is simply putting print statements in your code wherever you want to check for the problems.

Are there any other approaches other than this?

As its a general question, its not restricted to any specific language.So please share your thoughts on how you would have done it?

EDIT- While submitting your answer, please mention a useful resource (if you have any) about any concept. e.g. Logging
This will be lot helpful for those who don't know about it at all.(This includes me, in some cases :)

UPDATE: Michal Sznajderhas put a real "best" answer and also made it a community wiki.Really deserves lots of up votes.

Answer 1

One word: Logging.

Your program should write descriptive debug lines which include a timestamp to a log file based on a configurable debug level. Reading the resultant log files gives you information on what happened during the execution of the program. There are logging packages in every common programming language that make this a snap:

Java: log4j

.Net: NLog or log4net

Python: Python Logging

PHP: Pear Logging Framework

Ruby: Ruby Logger

C: log4c

Answer 2

• use println/log in code
• use DB explorer to look at data in DB/files
• write tests and put asserts in suspicious places

Answer 3

I guess you just have to write fine-grain unit tests.

I also like to write a pretty-printer for my data structures.

Answer 4

Just a couple suggestions:

1) Asserts. This should help you work out general expectations at different states of the program. As well familiarize yourself with the code

2) Unit tests. I have used these at times to dig into new code and test out APIs

Answer 5

More generally, you can monitor side effects and output of the program, and trigger certain events in the program externally.

A Print statement isn't always appropriate. You might use other forms of output such as writing to the Event Log or a log file, writing to a TCP socket (I have a nice utility that can listen for that type of trace from my program), etc.

For programs that don't have a UI, you can trigger behavior you want to debug by using an external flag such as the existence of a file. You might have the program wait for the file to be created, then run through a behavior you're interested in while logging relevant events.

Another file's existence might trigger the program's internal state to be written to your logging mechanism.

Answer 6

Actually you have quite a lot of possibilities. Either with recompilation of source code or without.

With recompilation.

• Additional logging. Either into program's logs or using system logging (eg. OutputDebugString or Events Log on Windows). Also use following steps:
  • Always include timestamp at least up to seconds resolution.
  • Consider adding thread-id in case of multithreaded apps.
  • Add some nice output of your structures
  • Do not print out enums with just %d. Use some ToString() or create some EnumToString() function (whatever suits your language)
  • ... and beware: logging changes timings so in case of heavily multithreading you problems might disappear.
  • More details on this here.
• Introduce more asserts
• Unit tests
• "Audio-visual" monitoring: if something happens do one of
  • use buzzer
  • play system sound
  • flash some LED by enabling hardware GPIO line (only in embedded scenarios)

Without recompilation

• If your application uses network of any kind: Packet Sniffer or I will just choose for you: Wireshark
• If you use database: monitor queries send to database and database itself.
• Use virtual machines to test exactly the same OS/hardware setup as your system is running on.
• Use some kind of system calls monitor. This includes
  • On Unix box strace or dtrace
  • On Windows tools from former Sysinternals tools like http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx, ProcessExplorer and alike
  • In case of Windows GUI stuff: check out Spy++ or for WPF Snoop (although second I didn't use)
  • Consider using some profiling tools for your platform. It will give you overview on thing happening in your app.
• [Real hardcore] Hardware monitoring: use oscilloscope (aka O-Scope) to monitor signals on hardware lines
• Source code debugging: you sit down with your source code and just pretend with piece of paper and pencil that you are computer. Its so called code analysis or "on-my-eyes" debugging
• Source control debugging. Compare diffs of your code from time when "it" works and now. Bug might be somewhere there.

And some general tips in the end:

• Do not forget about Text to Collumns and Pivot Table in Excel. Together with some text tools (awk, grep or perl) give you incredible analysis pack. If you have more than 32K records consider using Access as data source.
• Basics of Data Warehousing might help. With simple cube you may analyse tons of temporal data in just few minutes.
• Dumping your application is worth mentioning. Either as a result of crash or just on regular basis
• Always generate you debug symbols (even for release builds).
• Almost last but not least: most mayor platforms has some sort of command line debugger always built in (even Windows!). With some tricks like conditional debugging and break-print-continue you can get pretty good result with obscure bugs
• And really last but not least: use your brain and question everything.

In general debugging is like science: you do not create it you discover it. Quite often its like looking for a murderer in a criminal case. So buy yourself a hat and never give up.

Answer 7

like everyone else said:

• Logging
• Asserts
• Extra Output

&

your favorite task manager or process explorer

links here and here

Answer 8

On *nix systems, strace and/or dtrace can tell you an awful lot about the execution of your program and the libraries it uses.

Answer 9

Another thing I have not seen mentioned here that I have had to use quite a bit on embedded systems is serial terminals.

You can cannot a serial terminal to just about any type of device on the planet (I have even done it to embedded CPUs for hydraulics, generators, etc). Then you can write out to the serial port and see everything on the terminal.

You can get real fancy and even setup a thread that listens to the serial terminal and responds to commands. I have done this as well and implemented simple commands to dump a list, see internal variables, etc all from a simple 9600 baud RS-232 serial port!

Answer 10

I think the rest of the interview might go something like this...

Candidate: So you don't buy debuggers for your developers?
Interviewer: No, they have debuggers.

Candidate: So you are looking for programmers who, out of masochism or chest thumping hamartia, make things complicated on themselves even if they would be less productive?
Interviewer: No, I'm just trying to see if you know what you would do in a situation that will never happen.

Candidate: I suppose I'd add logging or print statements. Can I ask you a similar question?
Interviewer: Sure.

Candidate: How would you recruit a team of developers if you didn't have any appreciable interviewing skill to distinguish good prospects based on relevant information?

Answer 11

Spy++ (and more recently Snoop for WPF) are tremendous for getting an insight into Windows UI bugs.

Answer 12

First of all, what does debugging actually do? Advanced debuggers give you machine hooks to suspend execution, examine variables and potentially modify state of a running program. Most programs don't need all that to debug them. There are many approaches:

1. Tracing: implement some kind of logging mechanism, or use an existing one such as dtrace(). It usually worth it to implement some kind of printf-like function that can output generally formatted output into a system log. Then just throw state from key points in your program to this log. Believe it or not, in complex programs, this can be more useful than raw debugging with a real debugger. Logs help you know how you got into trouble, while a debugger that traps on a crash assumes you can reverse engineer how you got there from whatever state you are already in. For applications that you use other complex libraries that you don't own that crash in the middle of them, logs are often far more useful. But it requires a certain amount of discipline in writing your log messages.

2. Program/Library self-awareness: To solve very specific crash events, I often have implemented wrappers on system libraries such as malloc/free/realloc which extensions that can do things like walk memory, detect double frees, attempts to free non-allocated pointers, check for obvious buffer over-runs etc. Often you can do this sort of thing for your important internal data types as well -- typically you can make self-integrity checks for things like linked lists (they can't loop, and they can't point into la-la land.) Even for things like OS synchronization objects, often you only need to know which thread, or what file and line number (capturable by __FILE__, __LINE__) the last user of the synch object was to help you work out a race condition.

3. If you are insane like me, you could, in fact, implement your own mini-debugger inside of your own program. This is really only an option in a self-reflective programming language, or in languages like C with certain OS-hooks. When compiling C/C++ in Windows/DOS you can implement a "crash-hook" callback which is executed when any program fault is triggered. When you compile your program you can build a .map file to figure out what the relative addresses of all your public functions (so you can work out the loader initial offset by subtracting the address of main() from the address given in your .map file). So when a crash happens (even pressing ^C during a run, for example, so you can find your infinite loops) you can take the stack pointer and scan it for offsets within return addresses. You can usually look at your registers, and implement a simple console to let you examine all this. And voila, you have half of a real debugger implemented. Keep this going and you can reproduce the VxWorks' console debugging mechanism.

4. Another approach, is logical deduction. This is related to #1. Basically any crash or anomalous behavior in a program occurs when it stops behaving as expected. You need to have some feed back method of knowing when the program is behaving normally then abnormally. Your goal then is to find the exact conditions upon which your program goes from behaving correctly to incorrectly. With printf()/logs, or other feedback (such as enabling a device in an embedded system -- the PC has a speaker, but some motherboards also have a digital display for BIOS stage reporting; embedded systems will often have a COM port that you can use) you can deduce at least binary states of good and bad behavior with respect to the run state of your program through the instrumentation of your program.

5. A related method is logical deduction with respect to code versions. Often a program was working perfectly at one state, but some later version is not longer working. If you use good source control, and you enforce a "top of tree must always be working" philosophy amongst your programming team, then you can use a binary search to find the exact version of the code at which the failure occurs. You can use diffs then to deduce what code change exposes the error. If the diff is too large, then you have the task of trying to redo that code change in smaller steps where you can apply binary searching more effectively.

Answer 13

Peer review. You have been looking at the code for 8 hours and your brain is just showing you what you want to see in the code. A fresh pair of eyes can make all the difference.

Version control. Especially for large teams. If somebody changed something you rely on but did not tell you it is easy to find a specific change set that caused your trouble by rolling the changes back one by one.

Answer 14

Binary search in time is also a method: If you have your source code stored in a version-control repository, and you know that version 100 worked, but version 200 doesn't, try to see if version 150 works. If it does, the error must be between version 150 and 200, so find version 175 and see if it works... etc.

Answer 15

A nice read would be Delta Debugging from Andreas Zeller. It's like binary search for debugging