Metrics can be awesome. They can also create a horrible culture of counterproductive behavior as people try to back into desired numbers.
I love data. I’m pretty sure anyone who knows me knows this about me. But that’s okay, I’m not ashamed. I love data. There I said it again.
I’ve used data, and metrics specifically, to help me run my shop for over a decade. Starting from a wee development shop to a much larger, multi-departmental, global shop, I’ve used metrics to help make small — and occasionally, large — course corrections along the way to the goal posts.
But before we start, let me acknowledge that metrics can and do get used poorly. Done incorrectly, metrics can produce counterproductive behavior as people try to back into desired numbers. We’ve all seen it and it makes me weep. So before we get into the depths of how to use metrics, let’s spend a bit of time talking about how NOT to use metrics.
Don’t let metrics drive your business
People who love data sometimes lose sight of the fact that metrics are not the actual goal. They are instrumentation — crude instrumentation — that helps you approximate how you’re doing against the real business goal.
Use metrics as a starting point for discussion with your team. Use it to discuss anomalies. Use it to test hypotheses. Use it to spot trends before your team can “feel” the trends. But please, don’t use them as the end goal in and of themselves. That’s the definition of bureaucracy.
Don’t overreact
If you spot a change from one reporting period to the next, don’t (over)react. It often takes time for a real trend-line to emerge. Make a note of it and make sure to investigate next time. Maybe set up a mid-point to check its direction.
Don’t get fixated on your recipe
We’ve all seen it. The person whose full-time job is to gather bad data, scrub it, massage it, and then report on it. Like a French chef, they start with a gallon of produce only to end up with a tablespoon of sauce. But this is less tasty.
Keep in mind that there are often many ways to get at the same data point. Does it have to be the exact one metric that you thought up while driving to work? Can it be a metric that leverages existing data? Don’t make someone gather a bunch of data and create something new when there’s more likely than not something that already exists that’s close enough. Close enough. Remember that and your team will thank you.
A side benefit to this is that when you use data that is produced for another reason (ideally as an outcome from a business activity, or what I call “natural” metric), it’s more likely that the data will be higher quality. So you get higher quality with less work. Enjoy this rare phenomenon.
Don’t overdo it
Having too many metrics will drive your team crazy and you won’t know what to do with them. Start with a small number and increase as needed. Be merciless about dropping ones that are no longer useful. If you don’t have a specific action that you’d do if a metric goes up and down, consider dropping it.
Also don’t measure too often. Unless you’re in a special situation, metrics are unlikely to change daily or weekly. I would suggest tying it to a normal business reporting period such as quarters.
The Fun and Profit Part
Now that I’ve spent a fair amount of time talking about what not to do, let’s talk about what to do.
I try to create metrics around a trio that counterbalance each other. The goal is to get a holistic perspective so that one metric doesn’t drive behavior to the detriment of other important business goals. My counterbalanced metrics trio consists of the following — one metric to measure capacity, another to measure efficiency, and one more to measure quality. Let me use these in an example from product development to show it in action.
Capacity: How Much Are We Doing?
The capacity metric is perhaps the most obvious and common thing people want to measure. How much of X — whether it be printers, billable hours, or so on — are we producing? For development, we use “Ideal Days Produced” (abbreviated as ID) as our capacity metric. It’s a measure of the person-power that goes into a quarter, which lines up nicely with our release cycle (well, it used to but that’s another story…).
Ideal Days Produced helps us understand how much pure output our development team was capable of producing. If it went up and other metrics weren’t suffering as a result, great! If it went down, then something was amiss, and we needed to dig in a bit more. Usually, it came down to distractions of some sort or the other — a big customer launch that consumed a lot of developer cycles but not toward the product, or maybe a killer bug that required a lot of developer cycles to fix.
Efficiency: Are We Focused?
The efficiency metric is often overlooked. Perhaps many think it’s just a reflection of capacity, but I’ve found it to be an invaluable counterbalancing metric. In our development shop, we measure efficiency as “Ideal Days Produced / Person-Days” (abbreviated as ID/PD). This provides us with a way of understanding how efficient the entire team is during the cycle. Note that this is not an individual measure of effectiveness. I feel metrics are a poor way to manage an individual so all of our metrics are at an aggregate level.
Since it is closely related, we do typically view our efficiency metric, ID/PD, along with our capacity metric, ID. If ID and ID/PD are both going up, we say yay! We are getting more efficient, and as a result, producing more. If ID is going up and ID/PD is going down, it’s typically the result of adding new headcount that hasn’t quite become fully productive yet. If ID/PD is going down by itself, it typically means the team has gotten distracted for some reason. Sometimes a highly productive developer is out for a vacation. There are a lot of reasons, but it’s always worth investigating. If ID/PD is going up, but ID isn’t, it could be due to the same amount of work being unexpectedly done by a smaller team if the PD number is going down — hopefully due to productivity gains.
Quality: Are We Doing It Well?
The quality metric is one of the hardest to measure quantitatively, and will often require some creative thinking. Luckily, in development, the number of critical and high bugs post-release (abbreviated B for “bugs”) works pretty well. We only measure up to 30 days post-release because we’ve found that beyond that, issues tend to be more operational than development oriented. But quality is really important to measure because that’s often the first behavior to go out the door when metrics are implemented — especially if bonuses are tied to capacity-oriented metrics.
Our quality metric is often the metric we don’t want to go down. We’ve gotten it to the 80/20 mark where we feel that, in order to improve it, we’d have to really sacrifice productivity. So mostly we use it as a measure of when things are getting sloppy. If B starts to go up, we check around. Because a bug can be introduced by a single poorly written line, it doesn’t always mean the team is rushing ahead. But if the overall bug trend starts to increase and capacity, ID, is increasing, then we would likely make a concerted effort to slow down the next development. If it’s going down, then we do a happy jig and pat each other on the backs.
And Beyond
Beyond the ability to spot trends in organizational behavior, metrics can also be used for their predictive value. Once we establish a baseline of metrics, we use them to help plan future development cycles. In other parts of our organization, we might use them to help plan and allocate headcount or budget.
This predictive value becomes critical in helping you understand how to best position your business as you reach important milestones. Do you need to ramp up development capacity because a competitor has created a new product or significantly differentiated feature set? Do you need to hire more headcount to accommodate decreasing efficiency as your technical debt piles up (but yet, you don’t have the ability to majorly refactor)?
Hopefully, this gives you some food for thought on metrics. I haven’t gone at all into the art of actually selecting metrics. There’s a fair amount written on the topic already tucked under concepts like the Balanced Scorecard. Plus, it really is very specific to the business. Just remember, whichever operational area you are trying to measure, create the trio so you keep a holistic perspective on the business.