Problem
Latency doubled at 14:07 and stayed there. Somewhere there is a deploy log, a change ticket, or a person who knows why, and the viewer is looking at a chart that shows the effect and nothing about the cause.
Solution
Put the event on the same axis as the data. A vertical line at the moment, a marker with a label, and detail on hover. The correlation the viewer would otherwise construct by cross-referencing two systems becomes something they see.
Grafana’s model is a good one to think in, because it separates two sources of the same visual. Built-in annotations are added by ctrl or cmd-clicking a graph, stored in Grafana itself against the -- Grafana -- data source, and fetched by a query that exists on every dashboard. Query-driven annotations come from a data source: any query returning events becomes markers. The first is a person saying “this mattered”. The second is a system saying “this happened”.
Both are necessary and they age differently. Automatic annotations from deploys, config changes and incidents are complete and never stop arriving, which is also their failure mode: a chart with forty deploy markers a day has a picket fence where a signal should be. Manual annotations are sparse, high-value, and depend on someone remembering.
Tags are what make the automatic kind survivable, because they let a panel choose which events it cares about. Grafana filters annotation queries by tag for exactly this reason.
The other useful distinction is point versus region. A deploy is a moment. A maintenance window, a traffic experiment or a degraded period is a span, and drawing a span as a line loses the duration that was the whole point. Grafana handles this with a Time regions query type defining From and To.
Use when
Charts are read to explain rather than only to monitor, and the causes live in systems the viewer would otherwise have to go and check.
Don’t use when
The event stream is high-volume and unfiltered. Forty markers is worse than none, because it converts the chart into a chart with a fence in front of it.
Trade-offs
Annotations occupy the same visual channel as thresholds and anomaly bands, so a chart can quickly have three kinds of vertical furniture competing with the series. They imply causation by adjacency: a marker near a spike reads as the explanation whether or not it is, which is a real analytical hazard on a page people trust. Manual ones are only as good as the discipline behind them, and that discipline decays. And annotations stored in the dashboard rather than in the data are lost when the dashboard is rebuilt.
Checklist
- Where do these events come from, and does anything guarantee they keep arriving?
- Are annotations tagged, and does each panel show only the tags it needs?
- At the busiest time range, how many markers appear, and is the chart still readable?
- Is a span drawn as a span, or flattened into a line?
- Does the marker say what happened, or only that something did?
- Can the viewer get from a marker to the underlying deploy, ticket or incident?
- Do annotations survive the dashboard being rebuilt or duplicated?
- Do they compete visually with thresholds and bands on the same chart?
- Does a marker near a spike risk implying a cause nobody has verified?
- Who can add one, and is that the right set of people?
Compare
Grafana splits the pattern cleanly into stored annotations added by hand and query-driven ones from any data source, then filters both by tag, which is the most complete treatment and also the one that most needs someone to curate the tags. Datadog treats events as a first-class stream that overlays any graph, so deploys and alerts appear on charts without per-dashboard configuration, at the cost of needing suppression when the stream is noisy. Honeycomb leans on markers tied to deploys as the main annotation, on the reasoning that in practice the question is almost always whether a release caused it. Sentry goes further and makes the release an object rather than a marker, attaching regressions to it directly, which answers the causal question rather than only placing it on an axis.
Related
Time series is the chart this decorates. Comment and annotate is the collaborative version, where the note is addressed to a person rather than to the record. Threshold line competes for the same visual channel. Status history is the aggregated form of the same events. Zoom and pan is how a viewer gets from a marker to the detail around it.