What measurements to you use to improve your processes?

Question

This question was originally asking 'What KPIs do you use to in a software development organisation'. Unfortunately it seems that KPI is a four-letter word, and the immediate assumption is that KPIs are always mis-used (maybe they are?).

So, I've hopefully improved the question to get at the underlying goals which I originally thought KPIs were useful for. Assuming you have some process for how you (or your organisation) develops software. Secondly assume, that you (or your team) wants to become better at developing and delivery software. Finally, assume that one of the ways to improve is by refining your process.

Given all this, how do you know if your process refinements are having a positive impact? If these are KPIs, or SMART goals please provide individual, or groups, of KPIs/SMART goals that you have found the be effective. If it is some other mechanism please explain what it is. Finally, I guess, if you don't think that improving processes is a useful thing, I guess you can explain that too.

The areas of improvement I think would be useful are: quality, timeliness of releases, productivity, flexibility. If there are other aspects of an individual or team of developers, then that would be interesting to know.

Clarifying notes:

The question is not about how to best adapt or change a process, or what a good process improvement process is (be it Kaizen, retrospectives, etc). Nor is it about root-cause analysis or other approaches used to determine what specific aspects of a process should be improved.

The use of measures to determine if process improvement has been achieved, should not be confused with ongoing, as it happens, process improvement. (That is a good thing, but it isn't what the question is about!)

The process could be anything; scrum, agile, extreme, waterfall, ad-hoc. This question isn't about which process is best for certain types of software development, rather it is how to improve that process over time.

Obviously the specific metric will depend on the process that is involved, and the perceived problem that is trying to be improved. This question is simply designed to get examples of metrics used, which would obviously span a number of different processes and improvement areas.

The metric need not be something that is used all the time, for example, it could just be used while testing if a process change works. (For example, it could be too expensive - time or money wise - to measure and track at all times, so you just track it will tweaking the process).

It is taken as a given that if implemented poorly, the use of metrics can have a detrimental effect as developers game the system or otherwise. It is assumed that the person implementing the process change is aware of this problem and has taken effective steps to mitigate it.

All software organisations are different and how they fit into their company, so will have different specific things expected of them within the company, however I would think that product quality, productivity, flexibility and timeliness of releases are applicable to most if not all organisations. (With obviously different emphasis depending on the specific organisation.)

This question has nothing to do with source lines of code! In particular, I am not interested in measuring programmer productivity, especially in terms of SLOCs or # of bugs fixed, or any other naive measurements. I'm interested in higher-level way a team or individual measures their improvement. I am not interested in using a single KPI to measure anyone's performance. I am interested in using a range of KPIs to measure and improve my team's software development processes.

I know of horror stories about KPIs being misused and being ineffective (you don't need to search very hard to find them), but I can't believe that no-one out there tries to continually improve their processes, so there must be some good examples of KPIs out there.

I know all about the drawbacks of simplistic metrics that are applied to individual software programmers. I am really hoping to get examples of KPIs or alternative strategies that people have found as useful, rather than all the reasons why I shouldn't be using KPIs.

I'm mostly interested in processes and performance related to a development organisation within a larger company, as opposed to a software development company as a whole. For example, a software company should ensure that the products have the right features for the market, but generally that is product management's role, rather than engineering. And yes, there is a complete other discussion as to why and to what extent engineers should be involved in product management, but that is a separate discussion.

Answer 1

When I hear Key Performance Indicator I get a little worried because usually the next thing done is to link performance to reward and then you can get unstuck very quickly. I am always reminded of the software firm that decided on a reward system around bug fixing - the testers would be rewarded for finding bugs and the developers rewarded for fixing bugs. Development ground to a halt as an instant black market formed around the insertion, detection and correction of bugs.

Your organisational KPIs should be customer focussed. Depending on the type of software product you are making, you can measure it in the following ways:

• Sales - Is your product meeting customer requirements? You may be able to measure this in terms of the ratio of software presentations to sales or visits to your web site's purchase page to actual purchases
• Quality - Is your software understandable and reliable? How many support calls per customer do you get per day? Are the questions about how to do something or errors?
• Customer satisfaction - How satisfied are your customers with your product? Survey your customers and find out what you could be doing to increase their satisfaction then survey them again later to find out if you've improved. (Don't annoy your customers by asking a lot of questions or doing it too frequently)

Yes, these indicators seem to have nothing to do with the base level software metrics like bugs found and lines of code produced. However, the problem with bugs found is then you have to grade the severity of the bugs, and refactoring will often reduce your lines of code. Timeliness only matters if you are meeting your customer's expectations of timely delivery.

Concentrate on the business goals. If you have customers buying your software, they don't need a lot of support to use it and they are happy, then your software organisation is successful. No measure of bugs detected, schedule slips or anything else will matter if you don't have those three things in place.

If your software project is like the majority out there, it will be late, over budget, ship with less features than anticipated and have bugs. Don't beat yourself up over these things, deal with them and move on. Yes, you need bug databases, source control, testing and a way of measuring project velocity but in the end if you don't meet the business outcomes then you can't be successful, regardless of how polished and shiny your code is and how few bugs it has.

Update to try to address the revised question

KPIs as you desire to use them are difficult when delivering an intangible product that is also often a moving target. Will your KPIs used this year on an accounting system have the same meaning next year when you are implementing a document management system?

Let's take as an example a profession where KPIs are used widely - lawyers. Measuring lawyers uses KPIs such as: average billed hours worked per day; hours billed per month; age of debtors ledger; average age of unbilled work; percent of billed fees written off; and so on. You should notice a trend here - all these KPIs relate to willingness (or not) of clients to pay for the services rendered. This is the final arbiter of success and is why I suggested (above) some ways you could use this type of measurement as KPIs for your software business.

When you try to get down to having KPIs that don't relate to your client's willingness to pay for the value you are providing, then we get down to problems with what we are measuring, how you are measuring it and what differences there are in the measurement or what is being measured this year as opposed to last year.

"Dollars paid by clients" has a fixed value year to year - arbitrary metrics like "bugs in software", "timeliness of release" and "flexibility" don't have a fixed value and an increase in the KPI may not have a direct relationship to the underlying value that is meant to be measured by the KPI, such as "more bugs means lower quality".

For example, after the Columbia disaster, I recall the investigation team came up with several hundred recommendations and items to be investigated. Did these newly discovered "bugs" mean the space shuttle suddenly had a lot less quality? Actually, after the investigation the space shuttle had more quality. So a KPI around bugs can easily be distorted by an extensive QA session and more bugs reported may actually mean your software has higher quality.

Productivity in terms of timeliness of releases is easily distorted by commercial factors, such as a client throwing money at you to do some custom development for them. Your release schedule will slip but your business will improve.

As for flexibility, I can't even hazard a guess at how you would measure something so intangible.

About the only measurement I can think of that has value this side of the client's wallet is project velocity - how much did we estimate we would do last iteration/cycle/release and how much did we actually get done? Then plug this figure into the time available for the next iteration/cycle/release to estimate how much you will probably be able to get done this time. You can display the time remaining in a burn down chart or similar during the iteration.

The rest comes down to process which I don't think you can pin down to KPIs. All you can do is make sure your developers know what everyone is doing (daily developer meetings), your extended team gets input (weekly or fortnightly team meetings), you understand what worked last time and what didn't (retrospectives) and above all you have transparent effective communication.

Unfortunately, I don't think there are any magic KPIs like you are after (but don't overlook the relevance of getting money from clients as a KPI).

Answer 2

By far the best single indicator is "tested functionality delivered and accepted". In the Agile world, we usually measure "functionality" in terms of "user stories" but it can be in any convenient form as long as it's measuring actual functionality delivered and tested, acceptable to the customer.

The usual other measures, like SLOC, SLOC/staff-hour, etc, fail because of Charlie's First Law of Management, which is:

People will deliver whatever they're being rewarded to deliver.

Set up your measures as SLOC, and you'll get lots of SLOC. Use SLOC/hr, you'll get lots of SLOC/ht. Give them bonuses for working overtime, you'll get lots of overtime.

Oh, and remember the correlary, too:

What people are delivering is what they think will be rewarding to deliver.

If you're not getting what you want, ask why it's rewarding to do the stuff that's getting done.

Answer 3

Basing your KPIs around Cost, Quality & Schedule would be a good start. Consider what are the attributes for each of those you want to measure.

Being able to split each of these measures to show the cost of bugs would be useful - lots of bug fix effort late in the project means cost/schedule blowout. Being able to profile which parts of the codebase are buggy could help in targeting additional testing & possible code re-writes - typically 80% of the bugs will come from 20% of the code. Knowing where that is will allow your team to focus better.

EDIT: Look at measures like Cost of Quality (CoQ) and Cost of Poor Quality (CoPQ).

Measures like productivity are always hard to quantify - for example, using LOC/day leads to a debate about what exactly is a line of code? It can also lead to silly code formatting to "boost" productivity if the developers don't understand why these things are being tracked or perceive them as personal measurements. Even if LOC/day is not measured at the developer level, you can still get team rivalry leading to the same result.

EDIT: There are some good discussions to be found on Steve McConnell's Construx website. [Yes, that's the Steve McConnell of Code Complete fame]

Answer 4

The best use for Key Performance Indicators is for driving (or steering, if your prefer). For course-corrections in real-time.

(See Dashboards are for Driving for more blather about this sub-topic. Caveat: I am the author of the blathering article.)

So, the question back to you is: are you trying to evaluate performance after the fact, when it is too late to do anything about it, or are you trying to find KPIs that can help you stay on course?

If the former, any metric your organization cares about (bug count, ship-date slippage, lines of code with comments, customer return percentages, etc.) will be fine. Measure away and good luck getting any better in between shipping products and upgrades ;-)

If the latter, choose velocity. Assuming you are using test-driven development (TDD) of course.

EDIT: so it's the former. Well, here's why you are probably out of luck:

Suppose that you decide that "quality" is best quantified by measuring the number of bugs reported by customers as your post-process KPI. Let's assume that you are using TDD, and say that your team delivers Product #1 in 6 months, and after 6 months in the field you find that you have 10 customer-reported bugs. So now what, exactly, are you going to do to improve your process? Test more? Test specifically for more things like the causes of the bugs that were reported? It seems to me that you would already be testing, and when bugs are discovered - whether by the customer or not - you add a regression test for the specific bug and additional unit tests to make sure that there are no more similar bugs. In other words, your post-process improvement response will be no different than your in-process improvement response, so this KPI is really of no significant help in improving your process. The point is that the way you improve your process remains the same regardless of whether the bugs are discovered 6 months after release or two days into coding. So while this might be a shiny KPI to put on a manager's wall or a departmental newsletter, it really will not change your process-improvement mechanisms. (And beware of putting too much stock in this KPI because it can be wildly influenced by factors beyond your control!). In short, knowing the number of bugs does not help you improve.

(There is another danger here, one commonly found not just in business but also in the military, and that is the illusion that the post-mortem analysis revealed valuable information, so the lessons learned post-mortem are vigorously applied to the next project, which is probably not the same as the last project. This is known as "fighting the last war".)

Suppose the number of customer returns/refunds is your KPI of choice for "quality" - if this number is 5, what does this tell you? The specific reasons why customers requested a refund may be some indication of quality problems ("too slow", "doesn't interface with XYZ system", etc.), but the mere number of such incidents tell you nothing. A variance against an expected return percentage might tell you if quality was improving, but again the number does not help you improve. You need more information than the number can give you.

So for "timeliness of releases" what measurement would be appropriate? Number of days of ship-date slippage? Percent overrun based on original estimates? It doesn't matter, because again the numbers do not help you improve.

If you can measure "productivity" after the product is done then you can probably measure it while the product is being developed (e.g. velocity), the difference is that productivity less than expected during development can be improved immediately, while an overall productivity number measured after development is completed is too gross, too averaged, to be of any use. One could only guess at why it was lower than expected 6 months later...

I have no idea how one might measure "flexibility", that sounds like marketing jargon ;-)

I hope I haven't pounded this nail too hard or too far, but I don't think that there is anything useful that you can measure after-the-fact that you cannot measure while in progress. And there are a lot of after-the-fact measurements that are useless without knowing the causes.

Answer 5

No process is going to help you improve what you do except actually getting everyone together and figuring out what is working and what isn't working. For the team I'm currently leading, we do this through a series of Retrospectives (of which I'd highly recommend this book). The teams generally know what parts they want to improve - the trick is giving them the empowerment to actually measure and improve those things.

Yes, you certainly still need someone looking at a Macro level. If you look at an organization like Toyota, they have a Chief Engineer who straddles that line between business and production (for a good explanation, see Scott Bellware's blog post). In our organization we have someone similar - my boss was one of the initial developers of our product nearly 20 years ago, and stays highly active in the tech side, but is heavily invested in the customer side as well. My job is also to look at the teams as a whole to suggest improvements.

To measure, we first make sure that any improvements we strive for are things our teams can actually change, and then use something akin to the SMART goals so that any improvements are measurable. We have a Big, Visible Wall which we post the notes from the retrospective on. This happens to also be where we hold our daily stand-ups, so it gives us focus on what is going on.

For rolling up statistics to our executive meetings, we focus on code delivery - not lines of code delivered. I purposely kicked the team off measuring in nebulous units meaning that we don't report up that we worked x number of hours, or days, or whatever. What they do see is a trending chart of how well we are delivering our features and how we are improving. We'll also include interesting tidbits when the team feels they want to share it.

The best part about all of this is that we can try things for a month, and then reevaluate it just 4 weeks later. This creates a much lower barrier to entry for trying new things, since the team knows that if it is impacting them, we'll either cancel it immediately, or we'll reevaluate and find better ways at the next retrospective.

The bad part is that it isn't exactly what you are looking for. There isn't one metric or set of metrics we continually follow. We watch trends at all times, and measure the ones that we think are interesting - but only for a little bit, and only when the team is setting out to achieve a specific goal because of them. But in general, I'm quite happy with how it works, and I've seen a marked improvement in the involvement of the teams in improving the process. We aren't quite Kaizen, but we're getting better every day.

Answer 6

I did process improvement professionally for 14 years. Here is my advice, stop trying to quantify and start talking to people. Measurement works fine for a specific problem (once you know the problem, you have a better idea what to measure) and for repeatble processes like manufacturing. Your people know exactly where the problem areas are, so do your customers and users (from a very different perspective). Flow chart (use industrial engineering symbols not computer programming symbols) out the actual process for areas where there are concerns(not what we pretend the process is, you will need to observe as well as ask questions). Once you see the whole flow of the process look for delays, areas where the work is duplicated, areas where there is unnecessary process (usually due to adding more steps to the process to account for human error, thus creating many more potential areas for human error). Question the need for each step and whether there is a better way to do each step. Test potential changes and see if in fact they are an imporvement (way too many times they make the situation worse not better). Do not under any circumstances only talk to managers when getting a feel for problems or when flow charting. You will not get a true picture and will thus solve the wrong problem.

Answer 7

Benno, I'm answering your comment but didn't have enough characters there for the answer.

This depends on the problem you are solving. For instance suppose the issue is that the time from when the developer checks in the code until it is actually placed in production seems too long. Then you would get a baseline measurement of how long it is taking. then you would put in your change and then measure for a period of time to see if it now takes less time. You might also check something like the number of times the solution was determined not to work and sent back for rework before and after as well to make sure that the solution is not faster but lower quality.

Now the problem in IT with these measurements is that it may take quite some time to accumulate enough data as some problems do not re-occur frequently. In this case you may have to start by relying on subjective data until you can accumulate enough data to know if the change was good or not. But do not ask if something is an improvement until the users have gotten used to it. The first week or two of a new process, you will meet resistance to change and thus will get bad subjective results if you ask too early.

Another thing to be wary of is that if people know you are measuring something, they will be afraid that their personal performance is being measured and thus will game the system to get good results. It is often best if you can take measurements based on some system already in place (we have a a system that manages requests for software changes, we could query the database to find out historically how many requests missed the deadline, how many we reopened after being closed or are related to past requests etc., what the time differential between developer finishing and code being moved to production etc. are). You may also have to consider eliminating severe outliers, especially if the time spans the time period of both the old and new system. For instance we have one request that has been in Qa for over a 100 days not becasue it is bad but because QA has an avaliability problem and this one is lowest priority so it keeps getting bumped for higher prioroity work. This time would not be valuable in measuring the improvement of time, becasue the factor making the time so long isn't the process you are trying to fix. If you graph the data, you will easily see the outliers that might need excluding.

Answer 8

You can get lot of ideas about KPIs and Examples of Dashboards at http://www.dashboardzone.com

It has kpis by industry and functional areas.

Answer 9

Understanding waste and value-stream mapping will show you where you need to make improvements, and from that knowledge, you will learn what you need to measure. Principles of Lean and Kanban apply here. Understanding waste and it's effects on producing software will start you down the specific path to betterment that is inevitably specific to your organization. You can't take a cookie-cutter approach. Read (or listen to) "The Goal" and "Lean Thinking" for more on this really amazing and eye-opening perspective of what's wrong and how to fix it.