How many bugs is too many ?

Question

I sometimes get frustrated at my work due to the perceived buggyness of my code. But I think I am producing no more, and probably less than most people. For example I have just finished developing a project that took 3 months where the testers found around 10 genuine bugs and probably 10 other non-bug changes (ie. can I change the color of this button). I think this is not bad after 3 months of development but other people are looking at me as if I just checked in the biggest load of rubbish. What do you think?

Edit: I will add some more info.

1. I didn't just check it all in at the end. I was checking in continuously throughout and doing my own testing as I went. Obviously I found and fixed many bugs as I went.
2. First round of testing by external people was after two months. Most bugs were in areas I knew had issues and not tested thoroughly myself but as the software was still 'under development'
3. Second round of testing was after 3 months and like first version were in areas I had not been that thorough with, although I don't think it was that many issues.
4. Most of the bugs were in pretty visible parts of the application so it possibly made things worse. These parts while visible they were not really integral or used that often by the user. In fact some of the requests were put on hold as not essential.

Answer 1

One hair in soup is too many and in head too few. It all depends

Answer 2

We have no context to judge you, so I'd pay much more attention to those in your immediate surroundings than us here.

That said, I don't think you can strictly go by the numbers. If you missed some major details a customer would be immediately aware of, I'm going to be reluctant to be on projects with you. If you missed goofy feature-creep items that's more understandable.

Writing software for customers whose daily activities you have no idea about is a huge problem. It may be that your coworkers have an intuition from spending time with customers that you don't have yet, hence their reaction.

Answer 3

Perception of the quality of a software product can vary greatly between the programmer and a user. This is partly because one sees it from inside and the other sees it from outside.

If they are judging only by the bug count found in testing, then that is unfortunate, because it bears little correlation to how they'll enjoy the product when they actually use it. I'd see that as the real test of quality. Sit them down in front of the product and then see what the reaction is.

As much as the code is beautiful from your point of view as a programmer, it is an unfortunate (or fortunate, depending on your perspective) fact of life that the software will be judged on the user's experience using it.

See also: number of WTFs per minute.

Answer 4

If you check in your code only after three months of work then I think there's something seriously wrong.

As for projects with a short checkin-cycle personally I think every bug is one too many. Always fix bugs before writing new code (#5 in the Joel Test).

Answer 5

Relatively speaking, 10 bugs in 3 months of code might be very good. If I did a project that took 3 months I would not be surprised if 200 bugs could be found in it.

Of course, it also makes a difference whether they are large bugs or small ones, and whether they're the kinds of things you should have caught while coding. If one of those 10 bugs is a major issue that is a pretty standard thing to watch out for, then maybe those other people are somewhat justified in their thoughts.

Answer 6

How worried you should be does depend on the kind of bugs they are. How severe they are and how likely it is you should have spotted them before the testers.

If it's case of that the sum net and tax totals on an invoice isn't matching the gross then you should be worried. If it's a case of two people editing the same invoice at the same time causes something weird to happen 1 time in 5 then you can be less bothered.

Answer 7

It really depends. Firefox has about 15000 bugs, but it is considered to be quite usable. On the other hand, I wouldn't like to have a bug in my pacemaker.

Answer 8

It's very subjective: one person's trivial bug might be a blocker to someone else, or even a feature to another. It might be that the requirements weren't clear or correct, and that was reflected in the code and picked up by the testers. The quality of your testers also plays a part, the better they are the more relevant defects they'll find.

Answer 9

(not be to used as an excuse) but bugs in code in large (non-trivial) programs is part of the game. That's why we have test departments. Even the best programmer produces bugs as it's mathematically impossible to prove that a piece of (non-trivial) software dont't have any bugs.

http://amartester.blogspot.com/2007/04/bugs-per-lines-of-code.html

Answer 10

I'm just a perfectionist, but I tend to go over everything and test it with things I know used to break it (and things I know used to work) to make sure my new code fixes my old problems (and doesn't "fix" any old solutions). I have a bad habit of getting hung up fixing a bug instead of adding something I needed to add. This is probably a good thing, but who knows?

I assume you spend 3 months writing and finding bugs, and that your final result after 3 months of work had 10 bugs. This is alright. I wouldn't necessarily say this is good or bad - it depends on how much of your 3 months you spend fixing bugs. If you're just spitting out code for 3 months and then check it in, 1) you're doing it wrong, and 2) that's pretty amazing. If, however, you're spending those 3 months checking your code and making sure it works, which I suspect you are, I might suggest spending just a little more time checking it to weed out at least 5 of those 10 bugs. If they're corner cases that happen with just the wrong, uncommon combination of inputs, then that's somewhat forgivable, but if they're fairly common, you should consider taking a little extra time to track them down.

Not to say you're doing bad work, but if every 3 months the rest of your team has to track down and fix 10 bugs you introduce to get your features up and running, I can understand them getting a little annoyed.

Answer 11

From a perfectionist view, 1 bug is too many. How relevant the perfectionist view is happens to be very subjective.

If it's shrink-wrapped code, as few bugs as possible is better as it will increase sales (you wont have word of mouth saying its buggy).

In high-safety apps, zero bugs is really critical (pacemakers, hospital equipment, nuclear power stations) but these also tend to be specified with formal languages that allow you to properly analyse the design, along with long trial periods and longer test periods to really destruction test your code.

In apps written for specific customers, how many bugs are they paying you to remove? Are they paying for quick or quality? Severe bugs in an app that has been under budgeted and rushed through to production is just to be expected.

Also, was this 10 bugs while the testers were testing pre-production code? If so, then that's pretty good (last project I worked on, I sometimes managed 10 bugs in a day... but then fixed them all the next day because they were small things that I just hadn't considered fully :)). But bug counts before the system goes live hardly count (unless it's a week before release and you have issues meeting the original requirements)

Answer 12

As a rough metric, a three month project is 25,000-50,000 lines of code, and so you should expect to find ten bugs per 1000 lines in in-house testing, so 100-500 bugs would be typical.

If your in-house test department are reporting 0.2 bugs/1000 lines for what sounds like a solo effort (did anyone code review it?), then either you're very good, or they aren't very being thorough enough.

Answer 13

The number of "external" bugs for a programmer per 1'000 lines of code (KLOC) is a constant. The figure varies from 3-4 for very good programmers to 10 for average programmers to "lots".

This means: For every 1000 lines of code, you will find that many "bugs" (from silly typos in a string to completely f*** up code in a method to disastrous design which leads to a complete rewrite).

Note that we usually make a lot more mistakes but the better a programmer gets, the more bugs they find before the code leaves their desk (hence "external" bugs -> bugs which other people find). Also, good programmers will tend to make few disastrous bugs and more "silly" bugs (which are easy to fix but more embarrassing: "how could I oversee this?").

Interestingly, the number of bugs is constant with respect to the language, so very good assembler programmers make 4 bugs/KLOC, so very good C programmers make 4 bugs/KLOC, so very good Java programmers make 4 bugs/KLOC, etc.

This means that languages which need fewer lines of code to achieve the desired result, have an advantage.

Now to the question "when are there too many bugs?" Take a step back. As you develop, you will have many bugs (most of them internal). The final product will have fewer bugs because most will be found during development. This means that it matters when you look. It also means that the number of bugs per se is not a good indication of "health".

To figure our when your project starts to get sick, you must either count your bugs somehow (how many lines have I written, how many bugs did I find, how many could I fix). This is a lot of effort and often has a negative ROI. These things are hard to measure and it takes time to write down the bugs you find somewhere and this all distracts you from your goal: Write code.

My approach is to start writing more tests when more bugs sprout. The felt amount of "buggyness" (or uneasiness on my side) is a good indication whether something is wrong. So if you feel that you create two new bugs with every line of code, it's a sure signal that something has to be done. If you write lots of lines of code and everything falls into place as if by magic, then why waste time on measuring how well everything goes?

By writing tests, I can keep the bugs in check and I can make sure that they stay dead.

[EDIT] Some links with info where I got my numbers:

• http://www.fastcompany.com/magazine/06/writestuff.html
• http://www.zdnetasia.com/news/hardware/0,39042972,38005284,00.htm
• http://lwn.net/Articles/115530/
• http://www.joelonsoftware.com/items/2007/01/02.html

Answer 14

Here is a single comparison point (I would be interested to see others): a quick bug query tells me that in my developer job this past year I fixed 107 bugs, and assuming there's a rough correspondence between what I fixed and what I originally authored, that's maybe about 25 bugs created per three months.

As people have said, there are many confounds to finding a reasonable/meaningful interepretation of this data (how severe are the bugs, what kind of system/product do you work on, ...), but I'd be interested to see five or ten other people contribute their own "real world data" just to try to average out some anecdotal evidence.

Answer 15

Its hard to say how many bugs is a lot. I think metrics involving X bugs per Y lines of code are as flawed as any metric using lines of code (eg: measuring productivity).

My observation is that developers and users/business people have very different views of what bugs mean. For a developer, every bug found and fixed increases the developer's confidence in the code, since you can see that it is getting more robust. For a user, every bug found (even if fixed) decreases their confidence.

It reminds me a little of my undergrad software engineering course dealing with testing: a successful test is not one which passes but one which fails, since it has successfully detected a bug, and will thus lead to an improvement. In a similar way successful code is not successful because no bugs were found, but because the bugs were found and fixed. If few/no bugs are found in a piece of software I would simply assume it has not been adequately tested yet.

Answer 16

Some people would rather not publicise when they find bugs in their own code especially for start-ups with no bug tracking.

I personally consider it a positive behaviour for developers to find as many bugs in their software as possible. Of course I am talking about professional and competent developers - it is not use leaving bugs in their to earn brownie points in this way. But if I get 1 bug for the same amount of code that another developer unearths 10 bugs, I would not look down in this developer especially if we are considering "big" projects

Answer 17

If others perceive your code as being buggy then it is too buggy. Perception is more important than reality. It doesn't matter if you only had 1 bug. To be perceived as buggy (even if it was only 1 bug) that would mean you created some fairly visible bugs.

Like someone else suggested, we can't really tell for your situation if 10 bugs is too many because we would need to know the type of bugs. I would suggest that you look at the types of bugs you are creating to see if there are any similarities. Did you not understand the requirements? Were there many off by one errors? Logic errors? etc... Then evaluate whether these are bugs that you should have caught. If so, figure out a way to not inject those types of bugs next time.

Logic/Math/Processing errors are almost always cases that you should have caught. Requirements and interface errors are more understandable as words mean different things to different people no matter how detailed you write them down. However, if you understood what the requirement meant and still had the bug then that requirement error is not understandable.

Someone pointed out that 3 or 4 errors per kSLOC is average, that seems high too me, but I've been on projects that were worse (so maybe it is not). However, those same projects did not have very good developers either. IMO, if it is not an obscure error or a misinterpretation then the bug should not be in the code. If it is then you aren't doing your job properly. So maybe 10 is too many, maybe not, it depends on the types of errors you are creating.

Answer 18

The quantity of bugs is less important than the length of time it takes to detect them. The longer it takes to identify a bug the more expensive it will be to track down and fix, and the more likely it will be that the fix for that bug might break something else. Test early and Test often (TEATO?) is the best approach.