Supporting Code Metrics with Case Studies

Question

I'm principally interested in case studies on code metrics, relating code readability to defect reduction, that justify taking seriously cyclomatic complexity or some similar metric. Wikipedia has this example:

A number of studies have investigated cyclomatic complexity's correlation to the number of defects contained in a module. Most such studies find a strong positive correlation between cyclomatic complexity and defects: modules that have the highest complexity tend to also contain the most defects. For example, a 2008 study by metric-monitoring software supplier Enerjy analyzed classes of open-source Java applications and divided them into two sets based on how commonly faults were found in them. They found strong correlation between cyclomatic complexity and their faultiness, with classes with a combined complexity of 11 having a probability of being fault-prone of just 0.28, rising to 0.98 for classes with a complexity of 74.

This is good, but I'm hoping to know if there are more studies (or perhaps similar studies for other metrics, such as SLOC).

I also found an article at IBM that promotes monitoring CC values, but it lacks case-study support showing ROI figures. Then there is Coding Horror article on "arrow code" which sites a summary of a case study, but does not offer the case study(ies) themselves nor the actual numbers which justified the conclusion:

Studies show a correlation between a program's cyclomatic complexity and its error frequency. A low cyclomatic complexity contributes to a program's understandability and indicates it is amenable to modification at lower risk than a more complex program. A module's cyclomatic complexity is also a strong indicator of its testability.

Certainly cyclomatic complexity (CC) will help spot arrow-code, but I still need case studies that show ROI values. For example, "organization X incorporated a max CC of 10 on methods/functions, and reduced defects 20% in the following development iteration."

Without that kind of data, it is difficult to get management to care. Can anyone point me to a few hard studies? Even just one would help...

Answer 1

"...I still need case studies that show ROI values."

Why is ROI so hard?

Here's why.

Individual programmer productivity varies by at least one and sometimes two orders of magnitude.

http://forums.construx.com/blogs/stevemcc/archive/2008/03/27/productivity-variations-among-software-developers-and-teams-the-origin-of-quot-10x-quot.aspx

Individual variability trumps any other effect you might be looking for. You can't do a "head-to-head", "apples-to-apples" comparison. When you compare two similar teams using different techniques (i.e., different complexity thresholds) you find that individual performance differences simply dominate the data and almost everything is noise.

"Without that kind of data, it is difficult to get management to care."

If management doesn't care about quality, you have big problems. ROI numbers aren't going to influence management to change the environment.

You have to run your own experiments on your own code in your own organization.

Gather Cyclomatic complexity, defect rates, problem tickets, crashes, anything you can. Try to correlate complexity with other bad metrics. An argumentative manager can always win by pointing out the individual differences among members of teams.

Use real data in your real organization. That's the best you can do. And it's not "some study" or "some whitepaper" It's your actual organization.

Answer 2

In this case, you get a bit more from the original article than Wikipedia's. His technical paper about how the data gathering and such shows a 95% confidence level in the conclusions.

You're right that this doesn't give ROI information directly. At least for this study, that would be fairly difficult -- for example, they used open-source projects for their training data, and actual costs for open-source projects are usually difficult to even estimate, much less measure. At the same time, they did use what I'd consider at least a reasonable proxy for true ROI data: they searched through the source control system for each of their "training" projects looking for check-ins that appeared to be related to fixing bugs, defects, etc. They then used a naive Bayes algorithm to find correlation between the metrics they used, and the problems that had been identified in the code. While undoubtedly open to at least some improvement, it appears to me that these results should mean at least something.

It's also worth noting that the same people who did the study keep a running index on a large number of open-source projects. If you wanted more in the way of solid data, you can check their index back against the source control logs for some of those projects, you could probably use their data to come up with more in the way of direct ROI type results. One note, however: their index is based on quite a few source code metrics, not just cyclomatic complexity, so I'm not sure exactly how much it would tell exclusively about CC as opposed to the other metrics they look at.

Answer 3

Here are some for object-oriented metrics:

1. The Optimal Class Size for Object-Oriented Software,
2. Predicting Maintainability with Object-Oriented Metrics - An Empirical Comparison,
3. Empirical analysis of CK metrics for object-oriented design complexity: implications for software defects