Problem
A chart shows when a metric changed, but the events that might explain it are recorded elsewhere.
Solution
Place events on the chart’s time axis with a marker and short label. Make details available by pointer, keyboard, or touch.
Use automatic events for records such as deployments and manual notes for findings people want to preserve. Let people filter event types so frequent markers do not cover the data.
Show a time span for an event with duration, such as maintenance. Keep the event’s source and timestamp available, and avoid presenting a nearby event as proof of what caused a change.
Use when
People use charts to investigate changes and need to compare them with recorded events.
Don’t use when
The event stream is high-volume and unfiltered. Forty markers is worse than none, because it converts the chart into a chart with a fence in front of it.
Trade-offs
Markers compete with the data and other chart guides. Nearby events can look like causes even when the connection is uncertain. Manual notes depend on people maintaining them, and stored annotations need to survive dashboard changes.
Checklist
- Where do these events come from, and does anything guarantee they keep arriving?
- Are annotations tagged, and does each panel show only the tags it needs?
- At the busiest time range, how many markers appear, and is the chart still readable?
- Is a span drawn as a span, or flattened into a line?
- Does the marker say what happened, or only that something did?
- Can the viewer get from a marker to the underlying deploy, ticket or incident?
- Do annotations survive the dashboard being rebuilt or duplicated?
- Do they compete visually with thresholds and bands on the same chart?
- Does a marker near a spike risk implying a cause nobody has verified?
- Who can add one, and is that the right set of people?
Compare
Grafana splits the pattern cleanly into stored annotations added by hand and query-driven ones from any data source, then filters both by tag, which is the most complete treatment and also the one that most needs someone to curate the tags. Datadog treats events as a first-class stream that overlays any graph, so deploys and alerts appear on charts without per-dashboard configuration, at the cost of needing suppression when the stream is noisy. Honeycomb leans on markers tied to deploys as the main annotation, on the reasoning that in practice the question is almost always whether a release caused it. Sentry goes further and makes the release an object rather than a marker, attaching regressions to it directly, which answers the causal question rather than only placing it on an axis.
Related
Time series provides the time axis. Comment and annotate supports discussion of findings. Threshold line adds another chart reference.
Status history shows changes in state over time. Zoom and pan helps people inspect the period around an event.