Dashboards / Data information / Percentile summary

Percentile summary

An average hides the slow requests users actually feel; the viewer needs the tail.

Updated September 9, 2026

Problem

The average response time looks fine. A meaningful number of people are having a bad time anyway, and the viewer needs to see them before those people write in.

Solution

Report the tail of the distribution instead of its centre. p50 says what a typical request felt like. p95 and p99 say what the unlucky ones felt like, and the unlucky ones are the ones who churn.

Two things make this harder than it looks, and both are invisible on the panel.

The first is that percentiles computed from histograms are estimates. Prometheus interpolates linearly inside whichever bucket contains the quantile, so the error is bounded by the bucket width and by nothing about the data itself. Their own documentation carries the example: a true p95 near 320ms reported as 443ms, because nearly every observation landed in one wide 300–450ms bucket. The number on the tile has three significant figures and one of them is real.

The second is that percentiles don’t average. A p95 per instance does not aggregate into a p95 across instances by any arithmetic. You have to sum the buckets and compute the quantile from the total, which is why Prometheus says flatly that aggregating precomputed quantiles rarely makes sense. Dashboards do it anyway, usually without meaning to, by putting a percentile in a panel and letting the panel’s own rollup average it across the window.

Then there’s what the measurement never saw. Gil Tene’s coordinated omission: when a system stalls, the thing measuring it usually stalls too, so the slow period produces fewer samples rather than more. The percentile improves at exactly the moment the system gets worse.

Use when

The metric is a latency or a duration, the distribution has a tail, and somebody is accountable for that tail. Which describes most SLOs.

Don’t use when

The population is small. A p99 over forty requests is the slowest request with extra steps, and calling it a percentile lends it a confidence it hasn’t earned. Also don’t reach for it when the viewer’s real question is “what shape is this”, because a percentile is one number and the shape is the thing they asked about.

Trade-offs

Percentiles compress a distribution into a handful of numbers, and every one of them hides the multi-modal case: two populations, one fast and one slow, produce a p95 that describes neither. Computing them exactly is expensive, so nearly everyone computes them approximately and nobody says by how much. And the ladder is seductive. p99, p99.9, p99.99—each rung needs an order of magnitude more data to mean anything, and buys less than the last.

Checklist

Which percentiles, and why those? Is p99 there because someone needs it, or because it looks rigorous?
Over what window, and how many samples land in it?
Are these computed from histogram buckets, and if so, where are the boundaries?
Does the interesting range sit inside one wide bucket, where the estimate is worst?
Is any percentile on this page being averaged, by the query or by the panel’s rollup?
Do the percentiles cover the same window as the request count shown beside them?
Is latency timed from intended dispatch or from when the request actually went out?
Is p50 shown next to the tail, so the viewer sees the spread and not just the edge?
Does the panel say what “good” is for this percentile, and who decided?
Would a histogram or a heatmap answer this viewer’s real question better?

Compare

Grafana renders whatever the query returns, which puts the whole correctness burden on whoever wrote the histogram_quantile expression, and will then average that result across the rollup window without comment. Honeycomb argues the pattern itself is the problem and shows a heatmap instead, on the grounds that a line through a percentile hides the distribution that explains it; BubbleUp then compares the points inside a drawn region against the baseline outside it, answering “why is the tail slow” rather than “how slow is the tail”. Netdata keeps per-second resolution, so its windows are short enough that the aggregation trap mostly doesn’t get a chance to spring. Sentry attaches percentiles to a transaction rather than to a service, so the tail arrives already scoped to a code path, which is the form an engineer can act on.

Histogram is the same data before it was reduced to a few numbers, and often the better answer. Heatmap is that distribution over time. KPI tile is the container a percentile usually lands in, and it inherits every problem above on top of its own. Time series is what a percentile looks like plotted, and it’s where the averaging mistake usually happens. Metric targets is where somebody writes down what the tail is allowed to be.

Wireframe — the pattern's anatomy, not any one product's version of it

Implementation

The mean is the number nobody experiences. Latency is reported at p50, p95 and p99 because the tail is what users feel, and an average hides exactly the part that pages someone.

Tokens: --foreground--muted-foreground--border--status-warn--chart-1

LightDarkCode

p50: 275ms
p95: 443ms±
p99: 693ms±

p95 is interpolated across the 300ms–450ms bucket, which holds most of the observations. Anywhere in that bucket is consistent with the data.

LatencyPercentiles.tsxComputes quantiles the way histogram_quantile does, and prints the bucket a wide estimate came from.

/**
 * Percentiles from a histogram are estimates, and the panel should say so.
 *
 * This computes quantiles the way Prometheus's histogram_quantile does:
 * find the bucket the rank falls in, then interpolate linearly across it. The
 * error is bounded by that bucket's width and by nothing about the data, so
 * the component prints the bucket beside any estimate whose bucket is wide
 * relative to the value. A p95 of 443ms from a 300–450ms bucket has three
 * significant figures and one of them is real.
 */
export type Bucket = { le: number; count: number };

const ms = (v: number) => (v >= 1000 ? `${v / 1000}s` : `${Math.round(v)}ms`);

/** Which bucket holds the rank, and where in it the quantile interpolates to. */
export function quantile(buckets: Bucket[], q: number) {
  const total = buckets.reduce((n, b) => n + b.count, 0);
  const rank = q * total;
  let below = 0, lo = 0;
  for (const b of buckets) {
    if (below + b.count >= rank) {
      const frac = b.count ? (rank - below) / b.count : 0;
      return { value: lo + frac * (b.le - lo), lo, hi: b.le };
    }
    below += b.count;
    lo = b.le;
  }
  const last = buckets[buckets.length - 1];
  return { value: last.le, lo, hi: last.le };
}

/** An estimate whose bucket spans more than a quarter of its value is thin. */
const WIDE = 0.25;

export function LatencyPercentiles({ buckets, quantiles = [0.5, 0.95, 0.99], ticks, width = 400, height = 160 }: {
  buckets: Bucket[];
  quantiles?: number[];
  /** Axis labels, in ms. Defaults to every bucket boundary. */
  ticks?: number[];
  width?: number;
  height?: number;
}) {
  const max = buckets[buckets.length - 1].le;
  const peak = Math.max(...buckets.map((b) => b.count));
  const x = (v: number) => (v / max) * width;
  const plotH = height - 24;
  const estimates = quantiles.map((q) => {
    const e = quantile(buckets, q);
    return { q, ...e, wide: (e.hi - e.lo) / e.value > WIDE };
  });
  const flagged = estimates.find((e) => e.wide);

  return (
    <div>
      {/* The summary itself: three figures, the tail beside the middle. */}
      <dl className="flex gap-6">
        {estimates.map((e) => (
          <div key={e.q}>
            <dt className="text-xs text-muted-foreground">p{Math.round(e.q * 100)}</dt>
            <dd className={`text-2xl font-semibold tabular-nums ${e.wide ? "text-status-warn" : "text-foreground"}`}>
              {ms(e.value)}
              {e.wide && <span className="ml-1 text-xs font-normal">±</span>}
            </dd>
          </div>
        ))}
      </dl>

      <svg viewBox={`0 0 ${width} ${height}`} className="mt-4 w-full" aria-hidden="true">
        {buckets.map((b, i) => {
          const lo = i ? buckets[i - 1].le : 0;
          const h = (b.count / peak) * (plotH - 20);
          return <rect key={b.le} x={x(lo)} y={plotH - h} width={x(b.le) - x(lo)} height={h} className="fill-chart-1" />;
        })}
        {flagged && (
          <rect x={x(flagged.lo)} y={plotH - (buckets.find((b) => b.le === flagged.hi)!.count / peak) * (plotH - 20)}
            width={x(flagged.hi) - x(flagged.lo)} height={(buckets.find((b) => b.le === flagged.hi)!.count / peak) * (plotH - 20)}
            fill="none" className="stroke-status-warn" strokeWidth="2" />
        )}
        {estimates.map((e) => (
          <g key={e.q} className={e.wide ? "stroke-status-warn fill-status-warn" : "stroke-border fill-muted-foreground"}>
            <line x1={x(e.value)} y1={14} x2={x(e.value)} y2={plotH} strokeDasharray="4 4" strokeWidth={e.wide ? 2 : 1} />
            <text x={x(e.value)} y={10} textAnchor="middle" className="text-[9px]" stroke="none">p{Math.round(e.q * 100)} {ms(e.value)}</text>
          </g>
        ))}
        <line x1={0} y1={plotH} x2={width} y2={plotH} className="stroke-border" />
        {(ticks ?? buckets.map((b) => b.le)).map((t) => (
          <text key={t} x={x(t)} y={height - 4} textAnchor={t === max ? "end" : "middle"} className="fill-muted-foreground text-[9px]">{ms(t)}</text>
        ))}
      </svg>

      {flagged && (
        <p className="mt-2 text-xs text-muted-foreground">
          p{Math.round(flagged.q * 100)} is interpolated across the {ms(flagged.lo)}–{ms(flagged.hi)} bucket, which holds most of the observations. Anywhere in that bucket is consistent with the data.
        </p>
      )}
    </div>
  );
}

demo.tsxHow it is called: the bucketed histogram. p95 lands in the wide bucket and is marked as an estimate.

import { LatencyPercentiles } from "./LatencyPercentiles";

/**
 * Uneven buckets, roughly Prometheus's shape, with most observations in one
 * wide bucket from 300 to 450ms. p95 lands in it and gets the ± and the
 * outline; p50 lands in a narrow one and does not.
 */
export default function Demo() {
  return (
    <LatencyPercentiles
      buckets={[
        { le: 100, count: 30 },
        { le: 200, count: 50 },
        { le: 250, count: 60 },
        { le: 300, count: 40 },
        { le: 450, count: 130 },
        { le: 700, count: 7 },
        { le: 1000, count: 3 },
      ]}
      ticks={[100, 300, 450, 1000]}
    />
  );
}