An error-budget burn rate is the speed at which a service spends the failures allowed by its service-level objective. A high burn rate sustained over a short interval can justify a page; a smaller rate sustained for hours may need a different response. The alert should describe user-visible failures and include enough traffic to make a ratio meaningful.
Make this comfortable
SLO burn-rate alerts: page on budget consumption, not isolated spikes
Operational decision
A claims API targets 99.9 percent successful requests over thirty days. Compute a failure ratio from counters, compare it with the allowed error fraction, and require both a short and a long window before paging. The PromQL fragment shows the five-minute ratio only; a complete alert also checks a longer window, excludes health checks from the user denominator, and handles low traffic. A 2 percent error ratio is twenty times the allowed 0.1 percent failure fraction. Review how much budget that burns over the chosen windows before selecting thresholds. Route the page to the owning team with a runbook that shows affected operations, current deployment revision, and a safe stop action. Test the alert with a controlled synthetic failure and confirm the notification reaches an operator.
sum(rate(claim_requests_total{result="error"}[5m]))
/ clamp_min(sum(rate(claim_requests_total[5m])), 0.001)Cost and verification
Long windows lower noise but delay detection; short windows detect quickly and can page on brief bursts. Multiple windows use more rule evaluation and require care around sparse traffic. A ratio without a minimum request count can behave oddly when almost no requests occur. Track alert firing alongside consumed error budget and operator action. A page that reliably fires but offers no useful response should be revised, not celebrated as coverage.
Common Mistakes
- Do not page on every single failed request.
- Do not divide by a sparse traffic rate without a low-volume policy.
- Do not call a ratio alone a complete multiwindow alert.
Connected lessons
- DevOps: delivery, infrastructure, and reliable operations
- SLOs and error budgets: turn reliability into a decision
- Alert design: page on impact and include a first action
- Observability: join metrics, logs, and traces
- Incident response: contain impact, then learn
Practice and check
devops
operations