Trace waterfall

One request touched twenty services and the viewer needs to see where the time went.

Updated September 10, 2026

Problem

A request took 4.2 seconds and touched twenty services. It passed through a gateway, a dozen services, two databases and a cache, and every one of those has a dashboard saying it is healthy. The viewer needs to see where the 4.2 seconds actually went.

Solution

Draw each unit of work as a horizontal bar on a shared time axis, indented under whatever called it. Length is duration, horizontal position is when it started, indentation is causality. Read down the left edge for structure and across for time.

The data model does most of the design work. In OpenTelemetry a span is a unit of work carrying a name, start and end timestamps, a parent span ID, attributes, events, links and a status of Ok, Error or Unset. A root span is the one with no parent_id. Children share the parent’s trace ID and reference its span ID. That structure is already a tree with timings attached, which is why every implementation of this pattern looks broadly alike.

The reading skill the waterfall teaches is the gap. A bar that starts late was waiting, and the empty space to its left is the actual finding. Nested bars that fill their parent mean the work is where you think. A parent much longer than the sum of its children means time went somewhere nobody instrumented, which is the most useful thing a trace can tell you and the thing a summary metric never will.

Sibling bars overlapping means concurrency; siblings in a staircase mean a sequential loop that probably shouldn’t be one. Those two shapes are worth learning to spot, because they are the difference between “add a machine” and “batch the query”.

Use when

A single slow or failed request needs explaining and the work crosses process boundaries. This is the view you land on from the tail of a latency distribution, not the view you monitor.

Don’t use when

The question is about the population rather than the instance. One trace is an anecdote; a p99 that moved needs a heatmap or a distribution first, and the waterfall afterwards to explain one example of it.

Trade-offs

A waterfall shows exactly one request, and the request you happened to open may be unrepresentative. Deep traces produce hundreds of spans and the interesting one is often collapsed three levels down. Wide dynamic range is brutal on the axis: a 4-second span and a 2-millisecond span on one linear scale means the short one is invisible, which is precisely the sub-span that turns out to run four hundred times. Clock skew between hosts can render a child starting before its parent. And the pattern only shows what was instrumented, so the most important gap on the screen is by definition unlabelled.

Checklist

Can the viewer get here from an aggregate, or must they already have a trace ID?
Is the root span’s total duration stated as a number, not just a bar?
Is the gap between a parent and the sum of its children visible or does the layout hide it?
What happens with several hundred spans, and what is collapsed by default?
Are errored spans distinguishable at a glance, and does the status field drive that?
Can the viewer see span attributes without leaving the waterfall?
Are repeated identical spans grouped, or does one N+1 loop fill the screen?
Does the axis handle three orders of magnitude, and is a log option available?
Can this trace be reached from, and get back to, the logs for the same request?
Is clock skew across hosts handled or at least flagged?

Compare

Jaeger is the reference implementation most people picture and keeps the view deliberately plain: a timeline, a collapsible tree, and span detail on click, with the trace ID as the unit of navigation. Honeycomb treats the waterfall as the last step of an analysis loop rather than an entry point, so you arrive holding a group of events you already narrowed down, and the single trace is evidence for a conclusion rather than a place to start guessing. Datadog links the waterfall to the service it belongs to in both directions, so a slow span carries you back out to that service’s dashboards and monitors. Sentry comes at it from the error rather than the metric, attaching the trace to an exception so the question is why this request failed rather than why it was slow.

Service map is the same relationships aggregated instead of instanced. Drill-down is how the viewer arrives here. Hover detail is how span attributes get read without losing the shape. Percentile summary is usually the panel that sent them looking. Log tail is the other view of the same request, and the two should link.

Wireframe — the pattern's anatomy, not any one product's version of it

Implementation

Bars positioned by start time, not stacked. A span that starts late was waiting; a span that is wide was slow. Those are different problems and only the offset separates them.

Tokens: --foreground--card--muted-foreground--border--status-warn--status-critical--chart-1

LightDarkCode

POST /checkout · 800ms · 8 spans · 335ms in the root not covered by any child

TraceWaterfall.tsxOrders spans depth-first from their parent ids, shades any wait longer than 50ms, and states how much of the root no child covers.

/**
 * The OpenTelemetry shape, reduced to what the drawing needs: a name, when it
 * started and ended, and who called it. A root is the span with no parent.
 */
export type Span = {
  id: string;
  parentId?: string;
  name: string;
  /** Milliseconds from the trace's start. */
  start: number;
  end: number;
  status?: "ok" | "error" | "unset";
};

/** Empty space to a bar's left longer than this is shaded. A bar that starts
 *  late was waiting, and the wait is the finding, not the bar. */
const GAP_MS = 50;
const ROW = 20;
const LABEL_W = 180;

const ms = (v: number) => (v ? `${Math.round(v)}ms` : "0");

/** Depth-first, so a child always follows its parent, siblings by start time. */
function order(spans: Span[]) {
  const kids = (id?: string) => spans.filter((s) => s.parentId === id).sort((a, b) => a.start - b.start);
  const out: { span: Span; depth: number; gap: number }[] = [];
  const walk = (parent: Span | undefined, depth: number) => {
    let latest = parent?.start ?? 0;
    for (const s of kids(parent?.id)) {
      out.push({ span: s, depth, gap: s.start - latest });
      latest = Math.max(latest, s.end);
      walk(s, depth + 1);
    }
  };
  walk(undefined, 0);
  return out;
}

/** How much of a parent no child covers. Longer than its children means
 *  time went somewhere nobody instrumented, which no summary metric says. */
function uncovered(parent: Span, spans: Span[]) {
  const kids = spans.filter((s) => s.parentId === parent.id).sort((a, b) => a.start - b.start);
  let covered = 0, cursor = parent.start;
  for (const k of kids) {
    covered += Math.max(0, k.end - Math.max(k.start, cursor));
    cursor = Math.max(cursor, k.end);
  }
  return parent.end - parent.start - covered;
}

export function TraceWaterfall({ spans, onSelect, width = 640 }: {
  spans: Span[];
  /** Attributes belong beside the waterfall, not on another page. */
  onSelect?: (span: Span) => void;
  width?: number;
}) {
  const rows = order(spans);
  const root = rows[0].span;
  const total = root.end - root.start;
  const x = (t: number) => LABEL_W + ((t - root.start) / total) * (width - LABEL_W - 20);
  const axisY = rows.length * ROW + 12;
  const missing = uncovered(root, spans);

  return (
    <div className="rounded-lg border bg-card p-4">
      <p className="text-xs text-muted-foreground">
        <span className="font-medium text-foreground">{root.name}</span> · {ms(total)} · {rows.length} spans · {ms(missing)} in the root not covered by any child
      </p>
      <svg viewBox={`0 0 ${width} ${axisY + 18}`} className="mt-3 w-full text-[9px]" role="list">
        {[0.5, 0.75].map((f) => <line key={f} x1={x(f * total)} y1={0} x2={x(f * total)} y2={axisY} className="stroke-border" strokeDasharray="3 4" />)}
        {rows.map(({ span, depth, gap }, i) => {
          const y = i * ROW;
          const waited = gap > GAP_MS;
          const fill = span.status === "error" ? "fill-status-critical" : waited ? "fill-status-warn" : depth ? "fill-chart-1" : "fill-foreground";
          return (
            <g key={span.id} role="listitem" onClick={() => onSelect?.(span)} className={onSelect ? "cursor-pointer" : undefined}>
              {waited && <rect x={x(span.start - gap)} y={y + 2} width={x(span.start) - x(span.start - gap)} height={ROW - 2} className="fill-status-warn/20" />}
              <text x={16 + depth * 12} y={y + 13} className={depth ? "fill-muted-foreground" : "fill-foreground"}>{span.name}</text>
              <rect x={x(span.start)} y={y + 5} width={Math.max(1, x(span.end) - x(span.start))} height={10} rx={2} className={fill}>
                <title>{`${span.name}: ${ms(span.end - span.start)}${waited ? `, after waiting ${ms(gap)}` : ""}`}</title>
              </rect>
            </g>
          );
        })}
        <line x1={x(0)} y1={axisY} x2={x(total)} y2={axisY} className="stroke-border" />
        {[0, 0.5, 1].map((f) => (
          <text key={f} x={x(f * total)} y={axisY + 14} textAnchor="middle" className="fill-muted-foreground">{ms(f * total)}</text>
        ))}
      </svg>
    </div>
  );
}

demo.tsxHow it is called: eight spans of one checkout, with an overlap, a staircase and a gap.

import { TraceWaterfall, type Span } from "./TraceWaterfall";

/**
 * One checkout, 800ms, eight spans. pricing.quote and tax.quote overlap,
 * which is concurrency. The two db.selects step one after the other under
 * cart.load, which is a loop. payment.charge starts 192ms after anything
 * else finished, and that wait is shaded because it is the finding.
 */
const SPANS: Span[] = [
  { id: "root", name: "POST /checkout", start: 0, end: 800 },
  { id: "auth", parentId: "root", name: "auth.verify", start: 8, end: 95 },
  { id: "cart", parentId: "root", name: "cart.load", start: 100, end: 180 },
  { id: "items", parentId: "cart", name: "db.select items", start: 108, end: 138 },
  { id: "prices", parentId: "cart", name: "db.select prices", start: 140, end: 168 },
  { id: "pricing", parentId: "root", name: "pricing.quote", start: 188, end: 298 },
  { id: "tax", parentId: "root", name: "tax.quote", start: 200, end: 298 },
  { id: "payment", parentId: "root", name: "payment.charge", start: 490, end: 678 },
];

export default function Demo() {
  return <TraceWaterfall spans={SPANS} onSelect={(s) => console.log("span", s.id)} />;
}