views:

242

answers:

3

What is statistical debugging? I haven't found a clear, concise explanation yet, but the term certainly sounds impressive.

Is it just a research topic, or is it being used somewhere, for actual development? In other words: Will it help me find bugs in my program?

A: 

It sounds like statistical sampling. When you buy a product, there's a good chance that not every single product coming off the "assembly line" has been checked for quality.

Statistical sampling calls for checking a certain percentage of products to (almost) ensure they're all problem-free.

It minimizes the effort at the risk of some problems sneaking through.

To be honest, unless you're checking every single execution path and every single possible input value, you're already doing this in your testing. The amount of effort required to test everything for any but the most simplistic systems is not worth it. The extra cost would make your product a non-compete item.

Note that statistical sampling doesn't just involve testing every 100th unit. There are ways to target the sampling to improve the chance of catching problems. For example, if historical data suggests most errors are introduced at a specific phase, target that phase. If one of your developers is more problematic than others, check his stuff more closely.

From what I can see from a cursory glance at some research papers, statistical debugging is just that - targeting areas based on past history of problems.

I know we already do this for our software. Since any bugs that get fixed have to pass unit and system tests that replicate the problem (and our TDD says these tests should be written before trying to fix the bug), those tests are automatically added to the regression test suite so that those areas that cause more problems naturally are tested more often in the future.

paxdiablo
i would downvote this answer because it's wrong, but since you're trying to be helpful (instead of just googling) that wouldn't be fair ;-)
Steven A. Lowe
Saw the smiley but, if it *is* wrong, let me know why. The first bit was conjecture but the rest was based on actual investigation (you're right though, I'm no domain expert). Still, I'd rather delete it than mislead someone (or worse, get downvoted by an actual domain expert).
paxdiablo
possibly not wrong, just incomplete - there are a number of statistical debugging methods, including gathering user data from program crashes, automated testing, etc. Frankly the OP could have searched google and found the same information so this whole thread is probably pointless...
Steven A. Lowe
I don't think it's useless (maybe because I was the one who created it)! The information on the web seems useless or at least there is no easy explanation out there. Google gives me links to research papers, which are hard to understand. So I hoped to get some SO-typical understandable answer.
It isn't useless because it's the same sort of quality control that "real" manufacturers use (the ones who, if their cars explode, find themselves at the pointy end of a billion-dollar lawsuit). Same as optimization: find/target problem areas. No point testing code that's been bugfree for 20 years.
paxdiablo
+1  A: 

that's when you ship software saying "well, it probably works..." ;-)

EDIT: it's a research topic where machine learning and statistical clustering are used to try to find patterns in programs that are good predictors of bugs, to identify where more bugs are likely to hide.

Steven A. Lowe
Would downvote you, @SAL, since I prefer to see humor without helpfulness in comments rather than answers. But that wouldn't be fair.
paxdiablo
@Pax: That wouldn't be fair since your answer was a made-up guess, too? OK, edited to include something "useful".
Steven A. Lowe
+3  A: 

I created statistical debugging, along with various wonderful collaborators across the years. I wish I'd noticed your question months ago! But if you are still curious, perhaps this late answer will be better than nothing.

At a very high level, statistical debugging is the idea of using statistical models of program success/failure to track down bugs. These statistical models expose relationships between specific program behaviors and eventual success or failure of a run. For example, suppose you notice that there's a particular branch in the program that sometimes goes left, sometimes right. And you also notice that runs where the branch goes left are fine, but runs where the branch goes right are 75% more likely to crash. So there's a statistical correlation here which may be worth investigating more closely. Statistical debugging formalizes and automates that process of finding program (mis)behaviors that correlate with failure, thereby guiding developers to the root causes of bugs.

Getting back to your original question:

Is it just a research topic, or is it being used somewhere, for actual development?

It is mostly a research topic, but it is out there in the "real" world in two ways:

  1. The public deployment of the Cooperative Bug Isolation Project hunts for bugs in various Open Source programs running under Fedora Linux. You can download pre-instrumented packages and every time you use them you're feeding us data to help us find bugs.

  2. Microsoft has released Holmes, an implementation of statistical debugging for .NET. It's nicely integrated into Visual Studio and should be a very easy way for you to use statistical debugging to help find your own bugs in your own code. I've worked closely with Microsoft Research on Holmes, and these are good smart people who know how to put out high-quality tools.

One warning to keep in mind: statistical debugging needs ample raw data to build good statistical models. In CBI's public deployment, that raw data comes from real end users. With Holmes, I think Microsoft assumes that the raw data will come from in-house automated unit tests and manual tests. What won't work is code with no runs at all, or with only failing runs but no successful counterexamples. Statistical debugging works off of the contrast between good and bad runs, so you need to feed it both. If you want bug-hunting tools without runs, then you'll need some sort of static analysis. I do research on that too, but that's not statistical debugging. :-)

I hope this helped and was not too long. I'm happy to answer any follow-up questions. Happy bug-hunting!

Ben Liblit