Problem
People watching a metric need to know when it requires attention without keeping the dashboard open.
Solution
Let people create an alert from the chart using the same query and threshold. Keep the relationship visible so later edits do not leave the chart and alert describing different conditions.
Ask how long the condition must last, who receives the alert, and what should happen when data stops arriving. Make the expected response clear. An alert should help someone decide what to do.
Show existing rules before creating another one, and explain what happens to a rule when its panel is changed or deleted.
Use when
The metric has a meaningful alert condition and someone is responsible for responding.
Don’t use when
Nobody would act on it. An alert nobody can respond to trains a team to ignore the channel, and the cost lands on the next alert rather than this one. Also avoid it when the condition needs several signals: a per-panel rule encourages single-metric thinking.
Trade-offs
Rules attached to individual panels can multiply across dashboards and send duplicate notifications. Static thresholds need review as conditions change. Missing data requires its own behavior so silence is not mistaken for normal operation.
Checklist
- Is this symptom or cause, and if cause, is it definite and imminent?
- Could a person act on this at 3am, and would the action be more than restarting something?
- Is there a duration condition, or does a single spike page?
- What happens on no data, and was that chosen deliberately?
- Does the rule use the same query as the panel, or a copy that can drift?
- Who owns this rule, and do they know?
- Does the same condition exist on another dashboard?
- What happens to the rule if the panel or dashboard is deleted?
- Is the threshold still right for current traffic, and when was that checked?
- How often has this fired, and how often did it lead to action?
Compare
Grafana creates alert rules from panels and forces the no-data decision at rule-creation time with four explicit mappings, which is the most honest treatment of the failure mode most products leave implicit. Prometheus keeps rules in version-controlled config rather than in a UI, so they get reviewed, diffed and owned like code, at the cost of the create-it-while-looking-at-it immediacy this pattern is named for. Datadog makes the monitor the primary object and lets dashboards reference it, which inverts the relationship and stops the same condition being redefined per dashboard. Sentry derives alerting from issue behaviour—new, regressed, spiking—rather than from a threshold on a line, which sidesteps threshold staleness entirely and only works because its data has a natural notion of “new”.
Related
Threshold line shows the alert boundary. Metric targets records the goal and its owner. Error and stale state handles missing or failed data.
Semantic status color shows an active alert. Status history records changes in alert state.