Problem
An average can hide the slowest experiences. People need to see how durations vary beyond the middle of the data.
Solution
Show clearly labeled percentiles alongside the metric and time window. For example, p95 is the value at or below which 95% of observations fall. Use the same definition consistently across panels.
Explain when values are estimates, especially when calculated from grouped data. Do not average percentiles from different groups to claim a combined percentile; calculate it from suitable underlying data.
Keep sample size and measurement gaps in view. Very high percentiles need enough observations to be useful, and missing slow requests can make results look better than they are.
Use when
Durations vary widely and people need to monitor slower experiences as well as typical ones.
Don’t use when
The population is small. A p99 over forty requests is the slowest request with extra steps, and calling it a percentile lends it a confidence it hasn’t earned. Also don’t reach for it when the viewer’s real question is “what shape is this”, because a percentile is one number and the shape is the thing they asked about.
Trade-offs
A few percentiles still hide the full distribution and separate clusters. Estimates depend on how data was collected and grouped. High percentiles can vary sharply with small samples.
Checklist
- Which percentiles, and why those? Is p99 there because someone needs it, or because it looks rigorous?
- Over what window, and how many samples land in it?
- Are these computed from histogram buckets, and if so, where are the boundaries?
- Does the interesting range sit inside one wide bucket, where the estimate is worst?
- Is any percentile on this page being averaged, by the query or by the panel’s rollup?
- Do the percentiles cover the same window as the request count shown beside them?
- Is latency timed from intended dispatch or from when the request actually went out?
- Is p50 shown next to the tail, so the viewer sees the spread and not just the edge?
- Does the panel say what “good” is for this percentile, and who decided?
- Would a histogram or a heatmap answer this viewer’s real question better?
Compare
Grafana renders whatever the query returns, which puts the whole correctness burden on whoever wrote the histogram_quantile expression, and will then average that result across the rollup window without comment. Honeycomb argues the pattern itself is the problem and shows a heatmap instead, on the grounds that a line through a percentile hides the distribution that explains it; BubbleUp then compares the points inside a drawn region against the baseline outside it, answering “why is the tail slow” rather than “how slow is the tail”. Netdata keeps per-second resolution, so its windows are short enough that the aggregation trap mostly doesn’t get a chance to spring. Sentry attaches percentiles to a transaction rather than to a service, so the tail arrives already scoped to a code path, which is the form an engineer can act on.
Related
Histogram shows the full distribution. Heatmap shows how it changes over time. KPI tile can display a selected percentile.
Time series plots that value across intervals. Metric targets defines the desired limit.