Software Metrics in Agile Methodologies

Question

Agile methodologies are rather prevalent these days, but I cannot seem to find much documentation on what metrics are most useful and why. I have found many more things saying that some traditional metrics like LOC and code coverage of tests are not appropriate, leaving two main questions:

1. Why are those two (and other) metrics inappropriate?
2. What metrics are best for Agile and why?

Even with an Agile process, wouldn't you want to know how much code coverage you have with your unit tests? Or is it simply that this metric (and others) just are not as useful as other metrics like cyclomatic complexity and velocity?

Answer 1

1.1) LOC are easy to answer

• They are really dependent of the language you use! The same feature might have a big difference when written on JAVA or on Ruby, for example

• A not well written software might have more lines than a good one!

1.2) Code coverage

• IMHO you should use metric, although its not perfect, it should give you a nice understanding on where your code needs more tests.

• Just one point you should take care here is that it is also dependent of the language. There could be some situations where you have a class or method that you really don't need to test! For example a class with only getters and setters.

2) From (1) you just mentioned code metrics, but judging from your question about velocity, you are interested on metrics on all the creation process, so I would list some:

• Velocity: The classic one and, if used well, it can enhance quite well an agile team performance, since you will know what your team can really do on a fixed time.

• Burn up and burn down charts : they can give you a good notion about how the team is performing during the interaction (sprint)

There are some articles on InfoQ about this. Here and here.

Answer 2

Irrespective of methodology, there are some basic metrics that can and should be used.
According to S. Kahn, the most important are the following three:

• size of product
• number of defects found in final phase of testing
• and number of defects found in the field.

If those are all you track, there's at least five ways they can be used:

• calculate product defect rate (A)
• calculate test defect rate (B)
• determine a desirable goal for A and monitor the performance
• determine a desirable goal for B and monitor the performance
• assess correlation between A and B
• if correlation is found, form metric of test effectiveness (B/A * 100%)

Although not necessarily fun to read, Metrics and Models of Software Quality Engineering provides an excellent in-depth software engineering and metrics overview.

Answer 3

As for question 1, I don't see any reason those metrics would be bad in an Agile process.

LOC provides you with a relative size measurement. While it may not always be useful to compare numbers between projects, it can provide you with a rate of growth within the project. If you can get it, the number of lines changed within a sprint may be useful as well to track a rate or refactoring.

Code coverage (of lines of code) gives you a general sense of whether or not your team is meeting a minimum bar of automated testing within a project.

As for question 2, keep the items above and here are a few more:

• LOC versus test count. If you can, maintain separate ratios for unit, integration and system tests.
• Average number of acceptance criteria versus test scenarios (or tests) for each story. It can help provide a better sense of whether or not your testing against the story's intent.
• Number of defects discovered
• Amount of work discovered (this is often captured by Agile tracking software) that wasn't part original estimates. It will help you judge if you are doing 'enough' planning.
• Tracking consistencies, or lack thereof, of velocity sprint to sprint
• While probably not popular and probably potentially dangerous, tracking estimates to work completed for each developer. While teams are supposed to be self organized and driven, not all teams are capable of dealing with human problems.

Answer 4

Agile is a business oriented thing, Agile is about maximizing the customer value while minimizing waste to provide the most optimal ROI. This is what should get measured. And to do so, I use the system that Mary Poppendieck recommends. This system is based on three holistic measurements that must be taken as a package:

1. Cycle time
   • From product concept to first release or
   • From feature request to feature deployment or
   • From bug detection to resolution
2. Business Case Realization (without this, everything else is irrelevant)
   • P&L or
   • ROI or
   • Goal of investment
3. Customer Satisfaction
   • e.g. Net Promoter Score

Sure, at the team level you can track things like test coverage, cyclomatic complexity, conformance to coding standards, etc, but high quality is not an end in itself, it's just a mean. Don't misinterpret me, I'm not saying high quality doesn't matters, high quality is mandatory to achieve sustainable pace (and we include "no increase of the technical debt" in our Definition of Done) but still, the goal is to deliver value to the customer in a fast and profitable way.

Answer 5

Just to add

Why LOC and Code Coverage of Tests are less than ideal:

Agile emphasizes outcome, not output (see Agile Manifesto). These two simply track output. Also, they do not properly measure refactoring, which is a vital aspect of Agile processes.

Another metric to consider would be Running Tested Features. I can't describe any better than this: http://xprogramming.com/articles/jatrtsmetric/