Metrics that mislead: averages, denominators and vanity numbers

A metric does not have to be wrong to mislead. Most of the figures that produce bad decisions are calculated correctly and read badly — an average standing in for a distribution, a ratio whose denominator moved, a proxy that stopped tracking what it was chosen to represent.

Below: the average that hides the problem, the denominator nobody checks, survivorship, proxies that drift, the vanity number, and what to put on the report instead.

These are not exotic statistical traps. They are five ordinary reading errors, each of which has produced real decisions in real committees, and each of which is visible once you know to look for it.

The common thread: the number was fine and the question it was asked to answer was different from the question it actually answers.

That distinction matters for how the problem gets fixed. None of what follows is solved by better data quality or a better tool — the inputs were already correct. It is solved by changing what the report shows and how the figure is framed, which is cheaper than any of the alternatives usually proposed.

The average that hides the problem

An average response time of four hours can mean every case took about four hours, or it can mean most took twenty minutes and a handful took three days. Those are different operations with the same headline.

The fix is not sophisticated: show a percentile alongside the average — the 90th is usually enough — or show the distribution outright. The moment the two numbers diverge, the average stops being the story.

This one matters more than it appears because complaints come from the tail, never from the mean. A team optimising the average can improve the reported figure while making the experience that generates the complaints worse.

The denominator nobody checks

Conversion, adoption rate, incidents per user, cost per transaction — every ratio has a denominator, and denominators move. A conversion rate that improved because the top of the funnel shrank has not improved.

The habit that prevents it costs nothing: report the numerator and denominator next to the ratio. Three numbers instead of one, and the whole class of error disappears.

It is worth being specific about why this is so common. Ratios are attractive precisely because they compress two facts into one, and the compression is lossy in exactly the direction that flatters whoever is presenting.

Survivorship

A satisfaction score computed from current customers says nothing about the ones who left, and the ones who left are the population the question was really about.

The same shape appears everywhere: uptime measured only over systems still in service, project success rates that exclude cancelled projects, supplier performance averaged over suppliers still under contract.

The check is one question — who is missing from this population, and would they have answered differently? Where the answer is yes, the metric is not wrong so much as answering a narrower question than the one being asked of it.

Proxies that drift

Most operational metrics are proxies. Tickets closed stands in for work done; logins stand in for engagement; training completions stand in for capability. Each was a reasonable proxy at the moment it was chosen.

They drift for two reasons. The thing being measured changes — and, more reliably, the proxy becomes a target, at which point people optimise the proxy rather than the outcome. Tickets get split, logins get automated, courses get clicked through.

The defence is to periodically re-ask why each metric was chosen and whether the link still holds. In practice almost nobody does this, which is why organisations accumulate dashboards full of measures that no longer correspond to anything.

The comparison that is not one

Two figures placed side by side invite a comparison, and the invitation is often unearned. Month against month with different working-day counts, a region against a region with different product mixes, this year against a year that contained a one-off event — each of these produces a difference that means nothing.

In Colombia and Mexico the calendar version of this is routine rather than exotic: the two countries do not share a holiday calendar, and neither of them shares one with the head office that usually built the report. A regional dashboard comparing months across both markets is comparing different amounts of time.

The remedy is to normalise deliberately — per working day, per active customer, per unit of whatever actually drives the volume — and to say on the report which normalisation was applied. A comparison whose basis is not stated will be read as like-for-like whether or not it is.

The lag nobody accounts for

Some metrics report an outcome that was determined weeks earlier. Sales closed this month were largely decided last quarter; incidents this week often originate in a change made a month ago; churn reported today reflects an experience the customer had well before deciding.

Reading those as feedback on this week's actions produces confident, wrong conclusions — and worse, it produces course corrections that arrive after the thing they were correcting has already changed.

Where the lag is known, it belongs written on the report next to the metric. Where it is not known, establishing it is usually a single afternoon of looking at when each recorded event actually happened versus when it appeared.

The vanity number

A figure that only ever goes up — total users registered, cumulative documents processed, hours saved since inception — cannot indicate a problem, and a metric that cannot indicate a problem is decoration.

They persist because they are pleasant and because removing one looks like hiding something. The test to apply is direct: what value of this number would cause us to change course? If there is no answer, it does not belong on a decision-making report.

It can still belong somewhere else. Cumulative figures are legitimate for communication and for recognising progress; the error is placing them where decisions get made and letting them crowd out the measures that can actually move against you.

What to put on the report instead

Fewer measures, each with its denominator visible, at least one that can move in the wrong direction, and a written definition attached to each — including the date the definition took effect.

And a stated owner per metric, because a number nobody owns is a number nobody investigates when it moves. That is the difference between a report that gets read and one that gets presented.

The reduction is the hard part. Committees accumulate metrics because adding one is easy and removing one requires arguing that it never mattered, so the count only ever rises until somebody deliberately cuts it.

Where to start

With the report that already exists. Take each figure on it and ask what decision it informs, what its denominator is, and who would notice if it were wrong. The ones with no answer are the candidates for removal.

A data and analytics maturity assessment runs that exercise systematically across source, quality, governance, model and consumption, and produces an analytics plan built on measures that survive being questioned.

Frequently asked questions

Why can a correct metric still mislead?

Because it answers a narrower question than the one being asked of it. An average hides a distribution, a ratio hides a moving denominator, and a proxy stops tracking the outcome it was chosen to represent.

What should be shown alongside an average?

A percentile — the 90th is usually enough — or the distribution. Complaints come from the tail, so a team optimising the average can improve the figure while worsening the experience.

What is the denominator problem?

Every ratio has one and denominators move. A conversion rate that rose because the top of the funnel shrank has not improved. Reporting numerator and denominator beside the ratio removes the whole class of error.

What is survivorship in a metric?

Measuring only the population still present — current customers, systems still in service, projects not cancelled. The check is whether the missing group would have answered differently.

Why do proxy metrics stop working?

Because the proxy becomes a target and people optimise it instead of the outcome: tickets get split, logins get automated, training gets clicked through.

How do you identify a vanity metric?

Ask what value of it would cause a change of course. If there is no answer, it does not belong on a report used for decisions, though it may be legitimate for communication.