Should we be tracking defects in things other than code?

Question

At various times in my career I have encouraged staff I worked with and/or managed to track defects in artifacts of the development process other than source code (i.e. requirements, tests, design). Each time the request has been met with astonishment, confusion and resistance. It seems so obvious to me that I'm always a little shocked when people resist the idea.

What we get from this exercise is a picture of where bugs are created and where they are found (in what part of the process). If we are building bad requirements then we we'll know it and can work to improve them.

Is anyone else collecting information on defects not in source code?

Answer 1

Yes, track them all.

Documentation, design docs, requirements, etc.

I am also as astonished as you when I hear "arguments" against it.
At the very least the tracking system should be able to identify where the defect was found and what part of the process it was injected.

Answer 2

Yes, definitely. The artifacts surrounding your code--models, specs, doco, requirements info, use cases, etc--can all contain errors that affect the code itself.

Answer 3

Normally bug tracking systems have an assumption that they're a list of things that are to be fixed or implemented. Tracking bugs in requirements or other documentation (e.g. task lists) doesn't seem like it's the same thing. It's more a matter of keeping records so you can trend problems and evaluate if you're making fewer of them.

I'm tracking them, but outside of our bug tracking system.

Answer 4

Well duh... anything you can improve, do what can to improve!

Treating it all as bug tracking makes sense - opinion will vary, as you note - but using one tracking system would give a coherent big picture of it all, let tasks be assigned, etc. Maybe a demo, a slide show or something aimed at using these systems in ways beyond the original source code tracking - pictures convince more than words.

Answer 5

I've normally tracked the source of all defects. They may get fixed in the code, but they don't necessarily get caused by that.

Wrong requirement, wrongly interpreted requirement, bad design, developer brainf*rt, bad documentation, wrong test, missing test, outdated test, code that doesn't do what the developer does, tool/compiler error (very rare, in my view), build system problem....

To me, they're all "the system doesn't do what the customer wants it to do", and all indicate something must be changed in order to make it do what the customer wants it to do. Arguing whether it's defect or feature, or a source code bug or some other issue distracts from addressing the issues to me.

Answer 6

Absolutely. Just look at Ubuntu Bug #1.

Answer 7

One biggie that no one seems to have mentioned is to start a database of bad smells and traps for use when performing peer reviews.

This is an invaluable resource for the peers actually performing the review.

It definitely pays off in the long term. This should also be a live document, database, etc. that is added to as:

• bugs are fixed
• as peers perform reviews, and
• as new blood arrives to join the team(s) bringing with them new knowledge and experience.

HTH.

cheers,

Rob

Answer 8

aboslutely. if your process is well enough along to trace back source of defect to orgin great. it helps customers and designers qualify the constraints in which they operate.

customer: develop robot to cut grass where all blades of grass are to be cut to a precise uniform length

designer: we will use left-handed kindergarten scissors mounted perpendicular to the ground ensure crisp/precise cuts

QA: cuts are precise.

customer: why does it take the robot 6 days to cut grass. we need in in 30 mins or less!

clearly tracking the source of the performance defect can help in molding conversations and improving the process going forward.

Answer 9

We track bugs in software, errors in documents, errors in drawings, and requests for new features all using the same track tool.