🚨 Alerting Systems

Why Alert at All?

An alerting system is the smoke detector for your application. It sits quietly and watches your metrics and logs; when the pattern of "trouble" appears, it sounds the alarm — ideally before anyone smells smoke. The goal isn't to notify you of everything that happens. It's to notify you of the few things that need a human decision, right now.

Notice the diamond: a good alert doesn't fire the instant a number crosses a line. It waits to confirm the problem is real and sustained. That single idea — patience before paging — is what separates a trusted alerting system from a noisy one.

Alert on Symptoms & SLOs

Here's the most important principle in this lesson: alert on symptoms your users feel, not on internal causes.

A doctor doesn't wire a patient to alarms for every internal fluctuation — a slightly elevated white-cell count doesn't sound a siren. The alarms are for symptoms that matter: the heart stops, breathing fails. Your system is the same. High CPU is a cause; it might mean trouble, or it might just be a busy afternoon doing useful work. Users don't feel CPU. They feel slow responses and errors.

SLOs make "how bad is too bad" concrete

An SLO (Service Level Objective) is a target you commit to, like "99.9% of requests succeed over 30 days." That budget of allowed failure — the error budget — turns vague worry into math. Alert when you're burning through the budget fast enough to miss the target, and you alert on exactly what matters, no sooner.

Writing Alert Rules

In the Prometheus world, alerts are just queries with a threshold and a duration. The for clause is the "patience" from our diagram — the condition must hold continuously before the alert fires, which filters out momentary blips.

# alert_rules.yml
groups:
  - name: service_slos
    rules:
      # SYMPTOM: users are getting errors. Fire if 5xx rate exceeds 5%
      # of traffic, sustained for 5 minutes (not a one-second spike).
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "Error rate above 5% on {{ $labels.service }}"
          description: "Current error rate: {{ $value | humanizePercentage }}"
          runbook: "https://wiki.example.com/runbooks/high-error-rate"

      # SYMPTOM: the site is slow. Fire if 95th-percentile latency
      # stays above 1s for 10 minutes.
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 1
        for: 10m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "p95 latency above 1s"
          description: "p95 is {{ $value }}s over the last 10 minutes"

Routing, Grouping & Inhibition

Once an alert fires, Alertmanager decides who hears about it and how. Three features do the heavy lifting of noise reduction.

Routing — send it to the right team

Match on labels to direct alerts. Critical ones go to the pager; everything else goes to chat.

# alertmanager.yml
route:
  group_by: ['alertname', 'service']
  group_wait: 30s        # buffer briefly so related alerts arrive together
  group_interval: 5m     # wait before sending an update for a group
  repeat_interval: 4h    # re-notify a still-firing alert only every 4h
  receiver: 'team-chat'  # default
  routes:
    - matchers: [ severity="critical" ]
      receiver: 'pagerduty'
    - matchers: [ team="frontend" ]
      receiver: 'frontend-chat'

receivers:
  - name: 'pagerduty'
    pagerduty_configs:
      - routing_key: '<your-integration-key>'
  - name: 'team-chat'
    slack_configs:
      - api_url: '<your-slack-webhook>'
        channel: '#alerts'
        send_resolved: true

Grouping — one notification, not a hundred

If a database dies and forty services all error at once, you want one grouped notification ("40 services failing") rather than forty pages. group_by and group_wait handle this automatically.

Inhibition — silence the redundant

When a whole cluster is down, you don't also need "high latency" warnings for every service on it. Inhibition suppresses the lesser alert while the bigger one is firing:

inhibit_rules:
  # If a critical alert is firing, mute the matching warning-level one
  - source_matchers: [ severity="critical" ]
    target_matchers: [ severity="warning" ]
    equal: ['alertname', 'service']

Notification Channels & Severity

Match the intrusiveness of the channel to the urgency of the problem. Waking someone at 3 a.m. for a cosmetic issue is how you lose their trust; emailing a total outage is how you lose customers.

On-Call & Escalation

An alert that fires but is never acknowledged is worse than no alert — it creates a false sense of coverage. An escalation path guarantees that if the first responder doesn't answer, someone else does.

Tools like PagerDuty and Grafana OnCall manage the schedule and the escalation timers. Alertmanager just hands the alert off with a routing key:

receivers:
  - name: 'pagerduty'
    pagerduty_configs:
      - routing_key: '<your-pagerduty-integration-key>'
        severity: '{{ .CommonLabels.severity }}'
        description: '{{ .CommonAnnotations.summary }}'

Avoiding Alert Fatigue

Remember the boy who cried wolf. A system that pages constantly for non-issues trains its responders to ignore it — and then misses the one alert that mattered. Alert fatigue is a real, studied phenomenon: in hospitals, staff exposed to hundreds of alarms a day respond measurably slower to critical ones. The same applies to engineers.

Measure your alert quality

• Signal-to-noise ratio — what fraction of alerts led to real action? Aim high.
• Alert frequency — how many pages per on-call shift? More than a couple is a smell.
• Time to acknowledge / resolve — rising times can mean people are tuning out.

Cut the noise

• Use a for window so brief blips never page
• Group related alerts into a single notification
• Inhibit redundant alerts under a bigger one
• Silence alerts during planned maintenance windows
• Delete alerts that fire repeatedly without ever needing action

Practice & Quiz

- alert: ServiceDown
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
      / sum(rate(http_requests_total[5m])) by (service) > 0.05
  for: 3m
  labels:
    severity: critical
  annotations:
    summary: "Success rate below 95% on {{ $labels.service }}"
    description: "5xx rate is {{ $value | humanizePercentage }}"

route:
  receiver: 'team-chat'      # default for non-critical
  routes:
    - matchers: [ severity="critical" ]
      receiver: 'pagerduty'

receivers:
  - name: 'team-chat'
    slack_configs:
      - api_url: '<webhook>'
        channel: '#alerts'
  - name: 'pagerduty'
    pagerduty_configs:
      - routing_key: '<key>'

Best Practices & Pitfalls

✅ Do

• Alert on user-facing symptoms (errors, latency) and SLO burn, not raw causes
• Add a for window so momentary spikes don't page anyone
• Attach a runbook link and clear context to every alert
• Match channel intrusiveness to severity — pager for P1, chat for P3
• Define an escalation path so no critical alert goes unanswered
• Review alert quality regularly and delete the noisy ones

❌ Don't

• Page for every threshold cross, including harmless blips
• Alert on causes like CPU when what users feel is latency and errors
• Send non-urgent alerts through intrusive channels
• Fire a hundred alerts for one root cause — group and inhibit instead
• Leave alerts without an owner or a documented response

Summary

📚 Additional Resources

• Prometheus — Alertmanager documentation
• Grafana — Alerting documentation
• Google SRE Workbook — Alerting on SLOs
• PagerDuty — Incident response guide

🚀 What's Next?

You've now completed Week 13's monitoring-and-logging arc — you can measure, understand, and be alerted about your system. Next you'll pull it all together in the Production Checklist: the final review of readiness, security, scaling, and reliability before you flip the switch and go live.