Measurements and targets: The use and abuse of metrics

Software development is a social activity. As such, it does not lend itself very well to measurements. Sure, we can measure a whole lot of things about software development but we can never contend that a given set of metrics is exhaustive.  Plus, it is costly to regularly measure and track too many things. So, we restrict ourselves to a subset of feasible measurements.

For instance, although return on investment is a useful metric, it is often not feasible to measure and track it for a piece of software. Thus, in practice, we settle for things that only paint part of the picture. It is important to acknowledge this. Often, enterprise IT loses sight of this truth. It deludes itself that the reported metrics constitute the big picture (From the school of “If you can’t measure it, it doesn’t exist.”).

What is worse, in the name of continuous improvement, these metrics are converted into targets. Teams now have an incentive to work towards local optima. No wonder large swathes of big enterprise IT are in a desperate mess. New causes of failure are discovered all the time. Heads roll and the new heads track a slightly different subset of metrics.

Automatically converting metrics to targets also has adverse psychological effects. Here is an example from day trading. What is more important? Making more winning trades than losing trades or making money? Obviously it is the second. But the moment you start tracking win-loss ratio it may become a target it itself. Emotional conditioning will also come into play and we will try to have less losing trades.

A software team can get severely constrained when a velocity target is imposed on it. At a certain threshold of constraints, team members lose a sense of empowerment. They play it completely safe and lose initiative. We experience this phenomenon sometimes when we deal with call centres. Government sponsored healthcare and education in western economies often exhibits similar characteristics. It is not just bureaucracy. In an effort to scale and centrally control efficient delivery of services, a raft of metrics is imposed on the practitioners. Performance of schools/hospitals is then tracked against these metrics (they now become targets). Teachers and doctors get frustrated, lose initiative and just play by the book much to the exasperation of parents and patients.

Measurements should not automatically become targets. When measurements indicate something is wrong, it calls for a conversation in context, not for a rating downgrade. Conversations in context may reveal that things are still ok in the larger scheme of things, the measurements only point to a local sub-optima. Trouble is, it is easy to scale management by numbers. It is very difficult to scale management by context. Perhaps it is worth asking if scaling is more important than delivering a quality experience. Besides, too much scale creates “too big to fail” monsters that have to bailed out with the money of the innocent in times of crisis. This is not just true of macroeconomics. It is true of macro IT as well.

Dashboards promote ignorance

It is possible to collect a wide range of metrics on a project. Metrics about the codebase, unit tests, performance tests, build, delivery progress etc. It is also possible to define  red, amber, green thresholds for each metric. Some go for standardized organization wide thresholds while others are comfortable with project specific ones. The obvious next step is to create a dashboard that rolls up all these metrics into a single red, amber, green project status indicator. Middle managers spend a good part of their life managing inputs to these dashboards. Some "programme managers" assiduously refer to their dashboards as balanced scorecards.

Trouble is, most of the time, an overall green status indicator doesn't mean anything. All it says is that the things under measurment seem ok. But there always will be many more things not under measurement. To celebrate 'green' indicators is to ignore the unknowns. The value of a metric is not when it is green. Lets take unit test coverage for example. Is 100% test coverage a cause for celebration? What if most tests don't have any assertions? When measurements become targets, they encourage gaming. On the other hand, a coverage of 50% at least tells you unambiguously that half the code is not covered (or there is an error in measurement).

One may argue that it is only a matter of measuring even more so that the overall green becomes truly indicative of overall project health. This is somewhat unrealistic in a fast paced knowledge work environment. Tools and technologies keep changing. Measurement tools don't keep pace. It takes significant effort to maintain the measurement infrasturture on a project.

Avoiding subjectivity in assessments is also cited as a reason for resorting to metrics. But that's like saying that a person can be certified healthy without the judgement of a physician by a dashboard that rolls up a hundred diagnostic tests.

In my experience, metrics are much more useful when they report bad news than when they are green. In the words of the Way of Testivus, "Good tests fail". Unfortunately, the tendency to roll up metrics into dashboards promotes ignorance. We forget that we are only see what is under measurement. We only act when something is not green.

Then comes the final argument. Don't blame the tool for human laxity. Dashboards cannot promote ignorance or even wisdom. Unfortunately, tools aren't value neutral.

Tools aren't value neutral

If we accept that language influences the way we think1, then we don't have to make a big leap to see that tools (and technologies) influence the way we think (and act) as well2.  Therefore, it is erroneous to say things like the following:
“The internet isn’t a pro or anti-democratic technology. It is value neutral”
“Powerpoint isn’t good or bad. It is how you use it.”
Couple of telling quotes from McLuhan:
“We shape our tools, and thereafter our tools shape us”
“Our conventional response to all media, name that it is how they are used that counts, it the numb stance of the technological idiot.”
Why am I writing this in the middle of posts on software excellence? Because I need this for my forthcoming posts :)
1. Language influences the way we think
http://www.edge.org/3rd_culture/boroditsky09/boroditsky09_index.html
http://www.amazon.com/Language-Thought-Action-S-I-Hayakawa/dp/0156482401
2. Tools (and technologies) influence the way we act
http://www.amazon.com/Understanding-Media-Extensions-Marshall-McLuhan/dp/0262631598
http://www.amazon.com/Shallows-What-Internet-Doing-Brains/dp/0393072223