This article was originally posted on swombat.com in February 2011.
Over the past few months there have quite a few articles about metrics, which are, of course, a very important topic for startups. Metrics like these can and should drive your startup’s development, once you get past the initial part of the hypothesis testing phase.
Yet there seems to be a surprising lack of clarity in many people’s minds about what kind of metrics are useful, how to make them actionable rather than vanity metrics, how to use them, and when to use them. Here’s my attempt to clear things up.
1. Why you should care
Metrics have an aura of scientific validity about them, and they take effort to measure. This means that if you do measure metrics, you’re likely to care about them and act on them. If you measure the wrong metrics, or you measure them wrongly, or you draw incorrect conclusions from what you measure, you are quite likely to make incorrect decisions based on them.
So, if you’re going to look at metrics and act on them, do it correctly. If you spend time and energy calculating metrics and end up making the wrong decision because of those metrics, that is much worse than not measuring metrics and going by gut feeling.
2. When not to use metrics
The first misconception, which I fell prey to as well in the past, is when to use metrics. If your traffic is too low for metrics to be meaningful, then measuring metrics is largely a waste of time, and making decisions based on statistical noise is potentially harmful.
If your traffic is very low, you may still be able to measure things like conversion rates and run A/B tests over a long period of time, but it will be a very slow process, and you are likely to get much better and quicker results by observing user activity directly and letting your subconscious do the work of spotting potential patterns. Don’t forget to double-check those patterns before acting on them, though. The brain is a wonderful pattern-matching machine that will see patterns in the most unlikely places.
3. Actionable metrics
An actionable metric is one which directly leads to some kind of action. As Eric Ries put it, “if you cannot fail, you cannot learn“. If your measurement does not lead you directly to some kind of decision, it’s not actionable. And, if the measurement cannot lead to both outcomes for the decision (e.g. “build out the new feature” or “pull the feature”), it is also not a real actionable metric.
For example, measuring your monthly traffic is a classic vanity metric. Are you going to do something differently if the traffic is higher? Lower? The same? Why?
Another, less obvious example: if you put in a new feature, and you measure its usage, based on the idea that it is “testing your hypothesis that the feature is useful”, this measurement is not an actionable metric unless you’re willing to pull the feature if not enough people are using it. Many people will measure the new feature usage after they’ve already made the decision to keep the feature anyway, under the illusion that it’s a meaningful metric.
Measuring metrics without linking them up to a hypothesis that you’re testing is largely a waste of time, and may lead you to make incorrect decisions.
There are reasons to measure non-actionable metrics, but those are to do with managing the business).
4. Management metrics
There is a good reason to measure non-actionable metrics: managing the business. In order to manage the business effectively, you need to know how much money came in. You need to know how much tax you owe to the government.
A related point: you need management metrics when pitching your business. User count is largely pointless to measure, but if you’re pitching to investors, or even customers, it’s a metric that, if it looks good, will make them feel good (that’s what a vanity metric is…), so it is useful to measure it even though it is probably not actionable.
But, management metrics are not actionable. They are subject to external influences, and internal biases (you tend to see what you want to see in them). Don’t make the mistake of driving your product design on the basis of vanity, management metrics.
5. How to use metrics posts
How then, to use a post like this one or this one, which gives a set of metrics without actions. It looks useful, but is it?
Yes, it is. What these metrics posts do is tell you what are the fundamental metrics that your tests should boil down to. Measuring “how many people used a feature”, for example, is not a fundamental metric. What metrics posts really point out are KPIs for your type of business.
Measuring the KPIs by themselves can often be a useful management metric, but it is not actionable. However, any actionable metric should boil down to a KPI. So, when you put in a new feature, don’t measure “how many people used the new feature”, measure “whether people who had access to the new feature were more likely to pay” or “whether people who had access to the new feature stayed on the site longer”.
6. External influences
One more key point to make about metrics is that there are many external factors that can affect your measurements. Don’t be caught out like this guy. Any test should be run with both options presented in parallel to compensate for seasonable changes. If you measure conversion with one landing page on Friday and another on Sunday, what you’re measuring is a combination of your landing page changes with the weekly cycle of traffic. So, all tests must be run in parallel.
In addition to that, if your users are interconnected, you need to be careful about how you group users together. Don’t show one user one set of features, and a different set of feature to their colleague, or you will lose both and falsify your metrics. Remember that you don’t need to have the same number of people on the A and B sides of the A/B test – you just need enough people on either side.
Management metrics are particularly susceptible to external influences, so do not use them to drive product design decisions.
7. Unmeasurable things
Finally, no metrics overview would be complete without mentioning that there are many things in the running of a business which cannot be turned into statistics. You should measure everything you can, but don’t fall into the trap of thinking you can measure things like design, product vision or even the fitness of your first few key hires.
Trying to A/B test things which are not measurable is foolish and leads to bad decisions. Even worse, ignoring key things just because they’re not measurable is selective blindness. Be very aware that there are many things in the running of a business which cannot be measured but are worth doing.
In conclusion
I’ve made a number of points that I hope will be useful to you. The key takeaway is that using metrics incorrectly is worse than using no metrics at all. Metrics are a knife. Sharp, accurate, strong, useful – but make sure you don’t hold it by the blade:
- Care about correct metrics usage; using them incorrectly will lead you to bad decisions.
- Don’t use metrics when you don’t have the traffic; use user activity streams instead.
- Actionable metrics are metrics which drive a decision directly; don’t act on vanity metrics.
- It’s ok to measure management metrics, but be aware that they’re not actionable.
- Metrics posts provide KPIs; make sure your measurements boil down to KPIs, but don’t measure the KPIs by themselves, except as a management metric.
- Beware of external influences, they can falsify your tests.
- Don’t discard unmeasurable things, and don’t force unmeasurable things into an A/B test mold.
This is by no means a complete list – there are many more articles even on this site, let alone in other places. It takes more than one article to learn how to use metrics correctly, but hopefully this is a good starting point.
Thoughts from 2017
I find myself much further from this topic than I was then, but I believe it’s still correct. In a way, I am neither more nor less skeptical of metrics: they have their use, and if you know how to use them they will make a big difference to your business.
I would add a note of caution from a few years of experience though: be careful when linking metrics to performance or even job security. This tends to lead to serious distortions in people’s behaviour – they will optimise for the metrics that they are being rewarded for, at the expense of everything else, including the long-term success of the organisation. When it comes to motivation, intrinsic motivators are far more powerful, lasting, and less distorting. They’re also a lot harder. Maybe check this page out for many thoughts on the topic.