--- title: Metrics that mislead the executive committee | SUMāTO description: "Why a correct metric can still lead to a wrong decision: averages hiding distributions, moving denominators, survivorship and proxies that drift." image: https://sumatogroup.com/hubfs/BRANDING/SUM%C4%81TO%20%7C%20LOGO%201000x500.png --- [Skip to content](https://sumatogroup.com/en/insights/blog/metricas-que-enganan#main-content) - [INSIGHTS](https://sumatogroup.com/en/insights) - [SUPPORT](https://sumatogroup.com/en/support) - [CONTACT](https://sumatogroup.com/en/contact) EN [Español](https://sumatogroup.com/insights/blog/metricas-que-enganan) [English](https://sumatogroup.com/en/insights/blog/metricas-que-enganan) [![SUMāTO Group — home](https://sumatogroup.com/hs-fs/hubfs/BRANDING/SMT%20-%20LOGO.png?width=40&height=40&name=SMT%20-%20LOGO.png)](https://sumatogroup.com/en) - [HOME](https://sumatogroup.com/en/) - About #### SUMāTO - [About us→](https://sumatogroup.com/en/about-us) - [Terms→](https://sumatogroup.com/en/legal) - [Legal→](https://sumatogroup.com/en/legal) - [Cookies→](https://sumatogroup.com/en/legal) - [Data protection→](https://sumatogroup.com/en/legal) #### METHODOLOGIES - [Design Thinking→](https://sumatogroup.com/en/methodologies#design-thinking) - [Lean Startup→](https://sumatogroup.com/en/methodologies#lean-startup) - [PMI→](https://sumatogroup.com/en/methodologies#pmi) - [Scrum→](https://sumatogroup.com/en/methodologies#scrum) #### Vendors - [AWS→](https://sumatogroup.com/en/vendors#aws) - [Cisco→](https://sumatogroup.com/en/vendors#cisco) - [Dahua→](https://sumatogroup.com/en/vendors#dahua) - [Fortinet→](https://sumatogroup.com/en/vendors#fortinet) - [Huawei→](https://sumatogroup.com/en/vendors#huawei) - [Microsoft→](https://sumatogroup.com/en/vendors#microsoft) - [OCI→](https://sumatogroup.com/en/vendors#oci) - [Panduit→](https://sumatogroup.com/en/vendors#panduit) - Capabilities #### TECHNOLOGY - [Artificial Intelligence→](https://sumatogroup.com/en/artificial-intelligence) - [Data Analytics→](https://sumatogroup.com/en/data-analytics) - [Automation→](https://sumatogroup.com/en/automation-rpa) - [Cybersecurity→](https://sumatogroup.com/en/cybersecurity) - [Cloud→](https://sumatogroup.com/en/cloud) #### SEGMENTS - [SMB→](https://sumatogroup.com/en/smb) - [Enterprise→](https://sumatogroup.com/en/enterprise) - [Government→](https://sumatogroup.com/en/government) - Consulting #### Assessments - [AI Readiness→](https://sumatogroup.com/en/ai-readiness-assessment) - [Analytics→](https://sumatogroup.com/en/data-analytics-maturity-assessment) - [Cloud→](https://sumatogroup.com/en/cloud-readiness-assessment) - [Cybersecurity→](https://sumatogroup.com/en/cybersecurity-assessment) - [Enterprise Architecture→](https://sumatogroup.com/en/enterprise-architecture-assessment) - [IT Maturity→](https://sumatogroup.com/en/it-maturity-assessment) - [IT Strategy→](https://sumatogroup.com/en/technology-strategy-assessment) - [Process Automation→](https://sumatogroup.com/en/process-automation-assessment) #### Consulting & Architecture - [AI First→](https://sumatogroup.com/en/ai-first) - [BCP→](https://sumatogroup.com/en/business-continuity-plan) - [DRP→](https://sumatogroup.com/en/disaster-recovery-plan) - [Enterprise Architecture→](https://sumatogroup.com/en/enterprise-architecture-togaf) - [Enterprise Transformation→](https://sumatogroup.com/en/enterprise-transformation) - [IT Strategic Plan→](https://sumatogroup.com/en/it-strategic-plan) - [Strategic Consulting→](https://sumatogroup.com/en/strategic-consulting) - Operations #### INFRASTRUCTURE - [Data Center→](https://sumatogroup.com/en/data-center) - [Managed Services→](https://sumatogroup.com/en/managed-services) - [VDI→](https://sumatogroup.com/en/vdi) - [Intelligent Video Surveillance→](https://sumatogroup.com/en/video-surveillance) #### SECURITY - [NOC→](https://sumatogroup.com/en/noc) - [SOC→](https://sumatogroup.com/en/soc) #### USERS - [Modern Desktop→](https://sumatogroup.com/en/modern-desktop) - [Help Desk→](https://sumatogroup.com/en/help-desk) - Industries Industries - [Banking & Finance→](https://sumatogroup.com/en/banking-finance) - [Insurance→](https://sumatogroup.com/en/insurance) - [Government→](https://sumatogroup.com/en/government) - [Healthcare→](https://sumatogroup.com/en/healthcare) - [Telecommunications→](https://sumatogroup.com/en/telecommunications) - [Retail & Consumer→](https://sumatogroup.com/en/retail) - [Manufacturing→](https://sumatogroup.com/en/manufacturing) - [Energy, Oil & Gas→](https://sumatogroup.com/en/energy-oil-gas) - [Education→](https://sumatogroup.com/en/education) - [Logistics & Transportation→](https://sumatogroup.com/en/logistics-transport) - [Legal Services→](https://sumatogroup.com/en/legal-services) - [Engineering & Construction→](https://sumatogroup.com/en/engineering-construction) - Resources #### CONTENT - [Blog→](https://sumatogroup.com/en/insights) - [Use cases→](https://sumatogroup.com/en/use-cases) #### EVENTS - [Webinars→](https://sumatogroup.com/en/webinars) EN [Español](https://sumatogroup.com/insights/blog/metricas-que-enganan) [English](https://sumatogroup.com/en/insights/blog/metricas-que-enganan) Search - There are no suggestions because the search field is empty. [Datos y Analítica](https://sumatogroup.com/en/insights/tag/datos-y-analítica) # Metrics that mislead: averages, denominators and vanity numbers [Andrés Lozada](https://sumatogroup.com/en/insights/author/andres-lozada) · Oct 11, 2022, 8:00:00 AM · 8 min read **A metric does not have to be wrong to mislead. Most of the figures that produce bad decisions are calculated correctly and read badly — an average standing in for a distribution, a ratio whose denominator moved, a proxy that stopped tracking what it was chosen to represent.** Below: the average that hides the problem, the denominator nobody checks, survivorship, proxies that drift, the vanity number, and what to put on the report instead. These are not exotic statistical traps. They are five ordinary reading errors, each of which has produced real decisions in real committees, and each of which is visible once you know to look for it. The common thread: the number was fine and the question it was asked to answer was different from the question it actually answers. That distinction matters for how the problem gets fixed. None of what follows is solved by better data quality or a better tool — the inputs were already correct. It is solved by changing what the report shows and how the figure is framed, which is cheaper than any of the alternatives usually proposed. ## The average that hides the problem An average response time of four hours can mean every case took about four hours, or it can mean most took twenty minutes and a handful took three days. Those are different operations with the same headline. The fix is not sophisticated: show a percentile alongside the average — the 90th is usually enough — or show the distribution outright. The moment the two numbers diverge, the average stops being the story. This one matters more than it appears because complaints come from the tail, never from the mean. A team optimising the average can improve the reported figure while making the experience that generates the complaints worse. ## The denominator nobody checks Conversion, adoption rate, incidents per user, cost per transaction — every ratio has a denominator, and denominators move. A conversion rate that improved because the top of the funnel shrank has not improved. The habit that prevents it costs nothing: report the numerator and denominator next to the ratio. Three numbers instead of one, and the whole class of error disappears. It is worth being specific about why this is so common. Ratios are attractive precisely because they compress two facts into one, and the compression is lossy in exactly the direction that flatters whoever is presenting. ## Survivorship A satisfaction score computed from current customers says nothing about the ones who left, and the ones who left are the population the question was really about. The same shape appears everywhere: uptime measured only over systems still in service, project success rates that exclude cancelled projects, supplier performance averaged over suppliers still under contract. The check is one question — who is missing from this population, and would they have answered differently? Where the answer is yes, the metric is not wrong so much as answering a narrower question than the one being asked of it. ## Proxies that drift Most operational metrics are proxies. Tickets closed stands in for work done; logins stand in for engagement; training completions stand in for capability. Each was a reasonable proxy at the moment it was chosen. They drift for two reasons. The thing being measured changes — and, more reliably, the proxy becomes a target, at which point people optimise the proxy rather than the outcome. Tickets get split, logins get automated, courses get clicked through. The defence is to periodically re-ask why each metric was chosen and whether the link still holds. In practice almost nobody does this, which is why organisations accumulate dashboards full of measures that no longer correspond to anything. ## The comparison that is not one Two figures placed side by side invite a comparison, and the invitation is often unearned. Month against month with different working-day counts, a region against a region with different product mixes, this year against a year that contained a one-off event — each of these produces a difference that means nothing. In Colombia and Mexico the calendar version of this is routine rather than exotic: the two countries do not share a holiday calendar, and neither of them shares one with the head office that usually built the report. A regional dashboard comparing months across both markets is comparing different amounts of time. The remedy is to normalise deliberately — per working day, per active customer, per unit of whatever actually drives the volume — and to say on the report which normalisation was applied. A comparison whose basis is not stated will be read as like-for-like whether or not it is. ## The lag nobody accounts for Some metrics report an outcome that was determined weeks earlier. Sales closed this month were largely decided last quarter; incidents this week often originate in a change made a month ago; churn reported today reflects an experience the customer had well before deciding. Reading those as feedback on this week's actions produces confident, wrong conclusions — and worse, it produces course corrections that arrive after the thing they were correcting has already changed. Where the lag is known, it belongs written on the report next to the metric. Where it is not known, establishing it is usually a single afternoon of looking at when each recorded event actually happened versus when it appeared. ## The vanity number A figure that only ever goes up — total users registered, cumulative documents processed, hours saved since inception — cannot indicate a problem, and a metric that cannot indicate a problem is decoration. They persist because they are pleasant and because removing one looks like hiding something. The test to apply is direct: what value of this number would cause us to change course? If there is no answer, it does not belong on a decision-making report. It can still belong somewhere else. Cumulative figures are legitimate for communication and for recognising progress; the error is placing them where decisions get made and letting them crowd out the measures that can actually move against you. ## What to put on the report instead Fewer measures, each with its denominator visible, at least one that can move in the wrong direction, and a written definition attached to each — including the date the definition took effect. And a stated owner per metric, because a number nobody owns is a number nobody investigates when it moves. That is the difference between a report that gets read and one that gets presented. The reduction is the hard part. Committees accumulate metrics because adding one is easy and removing one requires arguing that it never mattered, so the count only ever rises until somebody deliberately cuts it. ## Where to start With the report that already exists. Take each figure on it and ask what decision it informs, what its denominator is, and who would notice if it were wrong. The ones with no answer are the candidates for removal. A [data and analytics maturity assessment](https://sumatogroup.com/en/data-analytics-maturity-assessment) runs that exercise systematically across source, quality, governance, model and consumption, and produces an [analytics](https://sumatogroup.com/en/data-analytics) plan built on measures that survive being questioned. ## Frequently asked questions ### Why can a correct metric still mislead? Because it answers a narrower question than the one being asked of it. An average hides a distribution, a ratio hides a moving denominator, and a proxy stops tracking the outcome it was chosen to represent. ### What should be shown alongside an average? A percentile — the 90th is usually enough — or the distribution. Complaints come from the tail, so a team optimising the average can improve the figure while worsening the experience. ### What is the denominator problem? Every ratio has one and denominators move. A conversion rate that rose because the top of the funnel shrank has not improved. Reporting numerator and denominator beside the ratio removes the whole class of error. ### What is survivorship in a metric? Measuring only the population still present — current customers, systems still in service, projects not cancelled. The check is whether the missing group would have answered differently. ### Why do proxy metrics stop working? Because the proxy becomes a target and people optimise it instead of the outcome: tickets get split, logins get automated, training gets clicked through. ### How do you identify a vanity metric? Ask what value of it would cause a change of course. If there is no answer, it does not belong on a report used for decisions, though it may be legitimate for communication. Next step [Data Analytics](https://sumatogroup.com/en/data-analytics)[Data Maturity Assessment](https://sumatogroup.com/en/data-analytics-maturity-assessment)[Automation and RPA](https://sumatogroup.com/en/automation-rpa) [Datos y Analítica](https://sumatogroup.com/en/insights/tag/datos-y-analítica) ![Andrés Lozada](https://sumatogroup.com/hs-fs/hubfs/SPEAKERS/AL.jpeg?width=56&height=56&name=AL.jpeg) Andrés Lozada Oct 11, 2022, 8:00:00 AM [LinkedIn](https://www.linkedin.com/in/andreslozada/) ### Explore more from SUMāTO [Enterprise AI](https://sumatogroup.com/en/artificial-intelligence) [Enterprise Transformation](https://sumatogroup.com/en/enterprise-transformation) [Strategic Consulting](https://sumatogroup.com/en/strategic-consulting) [AI Agent](https://sumatogroup.com/en/artificial-intelligence) [AI Contact Center](https://sumatogroup.com/en/artificial-intelligence) [Cybersecurity](https://sumatogroup.com/en/cybersecurity) ### Related Posts #### [Automating e-invoicing in Colombia and Mexico](https://sumatogroup.com/en/insights/blog/automatizar-facturacion-electronica-colombia-mexico) **E-invoicing is already automated in the part you can see: the document is issued, validated and delivered without anyone touching it. The manual work...** #### [Process orchestration: coordinating automations that already work](https://sumatogroup.com/en/insights/blog/orquestacion-procesos-coordinar) **Orchestration is what you need when the individual automations already work and the process still does not. It coordinates the order, the waiting,...** #### [Data residency: where the data lives once it moves to the cloud](https://sumatogroup.com/en/insights/blog/residencia-datos-nube-colombia-mexico) **Data residency is where information is physically stored. Neither Colombia nor Mexico forbids it leaving the country, but both condition...** ![SUMāTO](https://sumatogroup.com/hs-fs/hubfs/BRANDING/SUM%C4%81TO%20%7C%20LOGO%201000x500.png?width=200&height=100&name=SUM%C4%81TO%20%7C%20LOGO%201000x500.png) Strategic technology planning consultants. AI, Analytics, Cloud and Cybersecurity ## Navigation [Home](https://sumatogroup.com/en) [Capabilities](https://sumatogroup.com/en/artificial-intelligence) [Consulting](https://sumatogroup.com/en/strategic-consulting) [Operations](https://sumatogroup.com/en/managed-services) [Industries](https://sumatogroup.com/en/banking-finance) [Resources](https://sumatogroup.com/en/insights) ## SUMāTO [About](https://sumatogroup.com/en/about-us) [Terms](https://sumatogroup.com/en/legal#terminos) [Legal & Privacy](https://sumatogroup.com/en/legal) [Data protection](https://sumatogroup.com/en/legal) Cookies ## [Contact](https://sumatogroup.com/en/contact) [sales@sumatogroup.com](mailto:sales@sumatogroup.com) Mexico HQ Mexico City, Mexico [+52 55 8897 5791](tel:+525588975791) Bogotá Bogotá, Colombia [+57 601 724 5059](tel:+576017245059) © 2026 SUMāTO Group. All rights reserved. ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Andrés Lozada", "url" : "https://sumatogroup.com/en/insights/author/andres-lozada" }, "datePublished" : "2022-10-11T13:00:00.000Z", "headline" : "Metrics that mislead the executive committee | SUMāTO", "mainEntityOfPage" : { "@id" : "https://sumatogroup.com/en/insights/blog/metricas-que-enganan", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://sumatogroup.com/hubfs/BRANDING/Logo_SUMATO_Original%20-%201000x500.png" }, "name" : "SUMāTO Group" } } ``` ```json { "@context" : "https://schema.org", "@type" : "FAQPage", "mainEntity" : [ { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "Because it answers a narrower question than the one being asked of it. An average hides a distribution, a ratio hides a moving denominator, and a proxy stops tracking the outcome it was chosen to represent." }, "name" : "Why can a correct metric still mislead?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "A percentile — the 90th is usually enough — or the distribution. Complaints come from the tail, so a team optimising the average can improve the figure while worsening the experience." }, "name" : "What should be shown alongside an average?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "Every ratio has one and denominators move. A conversion rate that rose because the top of the funnel shrank has not improved. Reporting numerator and denominator beside the ratio removes the whole class of error." }, "name" : "What is the denominator problem?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "Measuring only the population still present — current customers, systems still in service, projects not cancelled. The check is whether the missing group would have answered differently." }, "name" : "What is survivorship in a metric?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "Because the proxy becomes a target and people optimise it instead of the outcome: tickets get split, logins get automated, training gets clicked through." }, "name" : "Why do proxy metrics stop working?" }, { "@type" : "Question", "acceptedAnswer" : { "@type" : "Answer", "text" : "Ask what value of it would cause a change of course. If there is no answer, it does not belong on a report used for decisions, though it may be legitimate for communication." }, "name" : "How do you identify a vanity metric?" } ] } ``` ```json { "@context" : "https://schema.org", "@id" : "https://sumatogroup.com/#organization", "@type" : "Organization", "address" : { "@type" : "PostalAddress", "addressCountry" : "MX", "addressLocality" : "Huixquilucan", "addressRegion" : "Estado de México", "postalCode" : "52787", "streetAddress" : "Av. Vialidad de la Barranca No. 6, Torre 1, Suite 400, Piso 4, Col. Bosques de las Palmas" }, "alternateName" : [ "SUMāTO Group", "SUMATO Group", "Sumato Group", "SUMATO", "SUMaTO", "SUMaTO Group", "SUMTO", "SUMTO Group" ], "areaServed" : [ { "@type" : "Country", "name" : "México" }, { "@type" : "Country", "name" : "Colombia" }, { "@type" : "Place", "name" : "Latinoamérica" } ], "contactPoint" : { "@type" : "ContactPoint", "areaServed" : "Latinoamérica", "availableLanguage" : [ "es", "en" ], "contactType" : "sales", "email" : "sales@sumatogroup.com" }, "description" : "SUMāTO is a Latin American technology consulting and integration firm founded in 2016, with a presence in Mexico and Colombia. It designs, implements and operates artificial intelligence, data analytics, automation, cybersecurity and cloud on the systems a client already runs, under governance frameworks such as NIST AI RMF and ISO/IEC 42001.", "foundingDate" : "2016", "knowsAbout" : [ "Inteligencia Artificial", "IA Generativa", "Agentes de IA", "Analítica de Datos", "Big Data", "Automatización de Procesos (RPA)", "Ciberseguridad", "Computación en la Nube", "Continuidad del Negocio y Recuperación ante Desastres", "Arquitectura Empresarial", "Transformación Digital" ], "legalName" : "SUMāTO Group", "location" : [ { "@type" : "Place", "address" : { "@type" : "PostalAddress", "addressCountry" : "MX", "addressLocality" : "Huixquilucan", "addressRegion" : "Estado de México", "postalCode" : "52787", "streetAddress" : "Av. Vialidad de la Barranca No. 6, Torre 1, Suite 400, Piso 4, Col. Bosques de las Palmas" }, "name" : "SUMāTO MX", "telephone" : "+52 55 8897 5791" }, { "@type" : "Place", "address" : { "@type" : "PostalAddress", "addressCountry" : "CO", "addressLocality" : "Bogotá", "streetAddress" : "Cra. 45 # 103-34, Of. 202" }, "name" : "SUMāTO CO", "telephone" : "+57 601 724 5059" } ], "logo" : { "@type" : "ImageObject", "height" : 500, "url" : "https://sumatogroup.com/hubfs/BRANDING/SUM%C4%81TO%20%7C%20LOGO%201000x500.png", "width" : 1000 }, "name" : "SUMāTO", "sameAs" : [ "https://www.linkedin.com/company/sumatogroup", "https://www.youtube.com/@sumatogroup", "https://torre.ai/teams/SUMaTOGroup", "https://www.cbinsights.com/company/sumto-group", "https://elioplus.com/profiles/channel-partners/57295/sumato-group" ], "telephone" : "+52 55 8897 5791", "url" : "https://sumatogroup.com" } ``` ```json { "@context" : "https://schema.org", "@id" : "https://sumatogroup.com/#website", "@type" : "WebSite", "description" : "Technology consulting in AI, data, automation, cybersecurity and cloud across Latin America.", "inLanguage" : "en", "name" : "SUMāTO", "publisher" : { "@id" : "https://sumatogroup.com/#organization" }, "url" : "https://sumatogroup.com" } ``` ```json { "@context" : "https://schema.org", "@type" : "BreadcrumbList", "itemListElement" : [ { "@type" : "ListItem", "item" : "https://sumatogroup.com/en", "name" : "Home", "position" : 1 }, { "@type" : "ListItem", "item" : "https://sumatogroup.com/en/insights/blog/metricas-que-enganan", "name" : "Metrics that mislead: averages, denominators and vanity numbers", "position" : 2 } ] } ```