# Percentile summary > An average hides the slow requests users actually feel; the viewer needs the tail. - Canonical: https://patterns.konigi.com/dashboards/percentile-summary - Group: Data information - Level: implementation - Status: published - Updated: September 9, 2026 - Also called: p50/p95/p99, latency percentiles, tail latency --- ## Problem The average response time looks fine. A meaningful number of people are having a bad time anyway, and the viewer needs to see them before those people write in. ## Solution Report the tail of the distribution instead of its centre. p50 says what a typical request felt like. p95 and p99 say what the unlucky ones felt like, and the unlucky ones are the ones who churn. Two things make this harder than it looks, and both are invisible on the panel. The first is that percentiles computed from histograms are estimates. Prometheus interpolates linearly inside whichever bucket contains the quantile, so the error is bounded by the bucket width and by nothing about the data itself. Their own documentation carries the example: a true p95 near 320ms reported as 443ms, because nearly every observation landed in one wide 300–450ms bucket. The number on the tile has three significant figures and one of them is real. The second is that percentiles don't average. A p95 per instance does not aggregate into a p95 across instances by any arithmetic. You have to sum the buckets and compute the quantile from the total, which is why Prometheus says flatly that aggregating precomputed quantiles rarely makes sense. Dashboards do it anyway, usually without meaning to, by putting a percentile in a panel and letting the panel's own rollup average it across the window. Then there's what the measurement never saw. Gil Tene's coordinated omission: when a system stalls, the thing measuring it usually stalls too, so the slow period produces fewer samples rather than more. The percentile improves at exactly the moment the system gets worse. ## Use when The metric is a latency or a duration, the distribution has a tail, and somebody is accountable for that tail. Which describes most SLOs. ## Don't use when The population is small. A p99 over forty requests is the slowest request with extra steps, and calling it a percentile lends it a confidence it hasn't earned. Also don't reach for it when the viewer's real question is "what shape is this", because a percentile is one number and the shape is the thing they asked about. ## Trade-offs Percentiles compress a distribution into a handful of numbers, and every one of them hides the multi-modal case: two populations, one fast and one slow, produce a p95 that describes neither. Computing them exactly is expensive, so nearly everyone computes them approximately and nobody says by how much. And the ladder is seductive. p99, p99.9, p99.99—each rung needs an order of magnitude more data to mean anything, and buys less than the last. ## Checklist - Which percentiles, and why those? Is p99 there because someone needs it, or because it looks rigorous? - Over what window, and how many samples land in it? - Are these computed from histogram buckets, and if so, where are the boundaries? - Does the interesting range sit inside one wide bucket, where the estimate is worst? - Is any percentile on this page being averaged, by the query or by the panel's rollup? - Do the percentiles cover the same window as the request count shown beside them? - Is latency timed from intended dispatch or from when the request actually went out? - Is p50 shown next to the tail, so the viewer sees the spread and not just the edge? - Does the panel say what "good" is for this percentile, and who decided? - Would a histogram or a heatmap answer this viewer's real question better? ## Compare **Grafana** renders whatever the query returns, which puts the whole correctness burden on whoever wrote the `histogram_quantile` expression, and will then average that result across the rollup window without comment. **Honeycomb** argues the pattern itself is the problem and shows a heatmap instead, on the grounds that a line through a percentile hides the distribution that explains it; BubbleUp then compares the points inside a drawn region against the baseline outside it, answering "why is the tail slow" rather than "how slow is the tail". **Netdata** keeps per-second resolution, so its windows are short enough that the aggregation trap mostly doesn't get a chance to spring. **Sentry** attaches percentiles to a transaction rather than to a service, so the tail arrives already scoped to a code path, which is the form an engineer can act on. ## Related Histogram is the same data before it was reduced to a few numbers, and often the better answer. Heatmap is that distribution over time. KPI tile is the container a percentile usually lands in, and it inherits every problem above on top of its own. Time series is what a percentile looks like plotted, and it's where the averaging mistake usually happens. Metric targets is where somebody writes down what the tail is allowed to be. ## Examples ### Honeycomb Query-first; heatmaps and BubbleUp replace the dashboard-of-panels model with draw-a-region cross-filtering. **Trace / cart checkout** — captured September 11, 2026, Honeycomb sandbox, public dataset (signed out). analytical, dense density, light theme, desktop-web. Seventy-one spans over 3.288 seconds for one checkout, and the shape gives the answer away before you read a single duration. Two thirds of the way down, getDiscounts runs for 2.576s—more than three quarters of the whole request —and underneath it nine visible SELECT spans step down and to the right in a staircase, each starting after the last one finished. The badge on the parent says 19. Nineteen queries in a loop, run one at a time, and the waterfall says so by its outline rather than by any number. That is the shape worth learning: siblings overlapping means concurrency, siblings in a staircase means something that should have been one query. The panel top right is the other good idea here—it plots the distribution of this span's duration across the whole dataset and marks where this particular trace fell, so you can see whether you are looking at a normal request or the tail before you start optimising. Hotspots in this capture: - [Trace waterfall](https://patterns.konigi.com/dashboards/trace-waterfall) — Indentation is causality, length is duration, horizontal position is when it started. Six levels deep here. - [Trace waterfall](https://patterns.konigi.com/dashboards/trace-waterfall) — The staircase: nineteen SELECTs one after another inside getDiscounts. Batch the query, don't add a machine. - [Percentile summary](https://patterns.konigi.com/dashboards/percentile-summary) *(this pattern)* — This span's duration against the whole distribution, with this trace marked—so you know if you're looking at the tail. - [Detail on demand](https://patterns.konigi.com/dashboards/detail-on-demand) — Selecting a span fills the right pane with its fields. The waterfall never moves while you read. - [Overview then detail](https://patterns.konigi.com/dashboards/overview-then-detail) — A minimap of all 71 spans above the list, so the shape of the whole trace is visible before you scroll it. - [Categorical series palette](https://patterns.konigi.com/dashboards/categorical-series-palette) — Five services, five hues, and the name in a column beside every one. Colour is never carrying it alone. ## Related patterns - [Histogram and distribution](https://patterns.konigi.com/dashboards/histogram) — The viewer needs the shape of a population, not a summary statistic of it. - [Heatmap](https://patterns.konigi.com/dashboards/heatmap) — Thousands of events per second can't be drawn as points; the viewer needs density. - [KPI tile](https://patterns.konigi.com/dashboards/kpi-tile) — A person needs to know the current value of one number, and whether it's fine, before they look at anything else. - [Time series](https://patterns.konigi.com/dashboards/time-series) — The viewer needs to see how a value changed over time and spot the moment it changed. - [Metric targets](https://patterns.konigi.com/dashboards/metric-targets) — The viewer wants the dashboard to know what 'good' is. ## Sources - [Prometheus, Histograms and summaries](https://prometheus.io/docs/practices/histograms/) - [Tyler Treat, Everything You Know About Latency Is Wrong (on Gil Tene's coordinated omission)](https://bravenewgeek.com/everything-you-know-about-latency-is-wrong/) - [Honeycomb, Handle Unruly Outliers with Log Scale Heatmaps](https://www.honeycomb.io/blog/handle-unruly-outliers-with-log-scale-heatmaps) --- Screenshots on patterns.konigi.com are reproduced for commentary and criticism. Product names and marks belong to their owners.