🐤 Canary Releases

What Is a Canary Release?

The name comes from coal mining: miners carried a canary underground because the bird reacted to toxic gas long before humans did. A distressed canary was an early warning to get out. In software, a canary release is a small, expendable sample of production traffic sent to the new version — if it shows distress (errors, slowness), you pull it before the whole user base is affected.

Concretely: you deploy the new version alongside the current one, then route a tiny percentage — often 1–5% — of live traffic to it. You compare the canary's metrics against the stable version. If it holds up, you increase the percentage in stages until it serves everyone. If it doesn't, you route that traffic back and the vast majority of users never noticed.

Canary vs Blue-Green

Both strategies run two versions and both avoid downtime, but they answer a different question. Blue-green asks "is the new environment ready to take over?" Canary asks "how is the new version behaving with real users, right now, at small scale?"

Splitting the Traffic

The heart of a canary is weighted routing: telling the infrastructure "send this fraction here, the rest there." Where you configure it depends on your stack.

Load balancer: weighted target groups (AWS)

# An ALB listener that sends 90% to current and 10% to the canary.
DefaultActions:
  - Type: forward
    ForwardConfig:
      TargetGroups:
        - TargetGroupArn: !Ref CurrentTargetGroup
          Weight: 90
        - TargetGroupArn: !Ref CanaryTargetGroup
          Weight: 10
      # Keep a user pinned to one version for their session:
      TargetGroupStickinessConfig:
        Enabled: true
        DurationSeconds: 3600

Service mesh: Istio traffic splitting

# Istio VirtualService: split by weight between two subsets.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app
spec:
  hosts: [ "my-app" ]
  http:
    - route:
        - destination: { host: my-app, subset: stable }
          weight: 90
        - destination: { host: my-app, subset: canary }
          weight: 10

If you're routing inside a Node service instead of at the edge, hash the user to a stable bucket so each user consistently gets one version:

// Deterministic canary routing: the same user always lands the same way.
function hashToPercent(id) {
  let h = 0;
  for (let i = 0; i < id.length; i++) {
    h = ((h << 5) - h) + id.charCodeAt(i);
    h |= 0; // force 32-bit integer
  }
  return Math.abs(h) % 100;                 // 0..99
}

function canaryMiddleware(canaryPercent) {
  return (req, res, next) => {
    const id = req.session?.userId || req.ip;
    // Below the threshold => canary; the SAME id always maps the same way.
    req.serviceVersion = hashToPercent(id) < canaryPercent ? 'canary' : 'stable';
    next();
  };
}

app.use(canaryMiddleware(10)); // send ~10% of users to the canary

Watching the Right Metrics

A canary is only as good as the signals you compare. The whole point is to catch a regression in the small sample before it reaches everyone, so you must decide in advance which numbers matter and what "worse" means.

The strongest technique is a relative comparison: canary metric divided by stable metric over the same window. A ratio near 1.0 means the canary behaves like the baseline; a ratio well above 1.0 for errors or latency is your signal to roll back.

# Prometheus recording rules comparing canary to baseline (a ratio).
groups:
  - name: canary_analysis
    rules:
      # Error-rate ratio: canary errors vs baseline errors.
      - record: canary:error_ratio
        expr: |
          (sum(rate(http_requests_total{version="canary",status=~"5.."}[5m]))
             / sum(rate(http_requests_total{version="canary"}[5m])))
          /
          (sum(rate(http_requests_total{version="stable",status=~"5.."}[5m]))
             / sum(rate(http_requests_total{version="stable"}[5m])))

      # Latency ratio: average canary latency vs baseline.
      - record: canary:latency_ratio
        expr: |
          (sum(rate(http_request_duration_seconds_sum{version="canary"}[5m]))
             / sum(rate(http_request_duration_seconds_count{version="canary"}[5m])))
          /
          (sum(rate(http_request_duration_seconds_sum{version="stable"}[5m]))
             / sum(rate(http_request_duration_seconds_count{version="stable"}[5m])))

canary:error_ratio   → 1.02   // ~2% more errors: within tolerance
canary:latency_ratio → 1.35   // 35% slower: exceeds a 15% threshold → roll back

Progressive Rollout

You rarely jump from 5% to 100%. Instead you climb a staircase, pausing at each step to analyze. If any step fails its thresholds, you abandon the climb and route everything back to stable.

A minimal analysis loop makes the promote-or-rollback decision explicit. Read the ratios, compare to thresholds, and act:

#!/usr/bin/env bash
set -euo pipefail

PROM="http://prometheus:9090"
ERROR_MAX=1.10    # allow at most 10% more errors than baseline
LATENCY_MAX=1.15  # allow at most 15% more latency than baseline
BAKE=600          # analyze each stage for 10 minutes
INTERVAL=30

query() { curl -s -G "$PROM/api/v1/query" \
  --data-urlencode "query=$1" | jq -r '.data.result[0].value[1] // "1"'; }

deadline=$(( $(date +%s) + BAKE ))
while [ "$(date +%s)" -lt "$deadline" ]; do
  error_ratio=$(query 'canary:error_ratio')
  latency_ratio=$(query 'canary:latency_ratio')
  echo "error_ratio=$error_ratio latency_ratio=$latency_ratio"

  if (( $(echo "$error_ratio > $ERROR_MAX" | bc -l) )); then
    echo "Error ratio too high — rolling back the canary."; exit 1
  fi
  if (( $(echo "$latency_ratio > $LATENCY_MAX" | bc -l) )); then
    echo "Latency ratio too high — rolling back the canary."; exit 1
  fi
  sleep "$INTERVAL"
done

echo "Stage healthy — promote to the next traffic weight."

Automating with Flagger

Hand-rolling the loop above teaches the mechanics, but in production you'll lean on a tool that does the ramp-and-analyze cycle for you. On Kubernetes, Flagger (paired with a service mesh) and Argo Rollouts are the popular choices. You declare the steps and metric thresholds; the operator drives the rollout and rolls back automatically on failure.

# Flagger Canary: raise traffic 10% at a time up to 50%,
# checking metrics every minute and rolling back after 5 failures.
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: my-app
  namespace: prod
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  provider: istio
  service:
    port: 80
    targetPort: 8080
  analysis:
    interval: 1m        # analyze once per minute
    threshold: 5        # roll back after 5 failed checks
    maxWeight: 50       # ramp the canary up to 50% before promoting
    stepWeight: 10      # increase by 10 percentage points each step
    metrics:
      - name: request-success-rate
        thresholdRange: { min: 99 }   # require >= 99% success
        interval: 1m
      - name: request-duration
        thresholdRange: { max: 500 }  # p99 latency under 500ms
        interval: 1m

The same idea in Argo Rollouts, where you spell out the traffic weights and pause durations directly:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  replicas: 5
  selector:
    matchLabels: { app: my-app }
  template:
    metadata:
      labels: { app: my-app }
    spec:
      containers:
        - name: my-app
          image: my-app:v2
          ports: [ { containerPort: 8080 } ]
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 10m }   # bake at 5%
        - setWeight: 25
        - pause: { duration: 10m }   # bake at 25%
        - setWeight: 50
        - pause: { duration: 10m }   # bake at 50%
        # then Argo promotes to 100% automatically

Practice & Quiz

Best Practices & Pitfalls

✅ Do

• Start small (1–5%) and increase in deliberate stages
• Compare canary vs stable as ratios over the same time window
• Define automatic rollback thresholds before you deploy
• Use session affinity so a user stays on one version
• Give each stage a bake time long enough to reveal slow-burning bugs
• Keep database changes backward-compatible (expand-contract, dual-write)
• Include a business metric — technical health can look fine while conversions tank

❌ Don't

• Don't jump straight to a large percentage — that defeats the purpose
• Don't rely on a human staring at dashboards; automate the decision
• Don't start a canary so small that the metrics are just noise
• Don't make breaking schema changes while both versions run side by side
• Don't forget to eventually decommission the old version after full promotion

Summary

📚 Additional Resources

• Google Cloud — Canary test pattern
• Flagger — Progressive delivery with Istio
• Argo Rollouts — Canary strategy

🚀 What's Next?

Both blue-green and canary run a separate copy of the new version. But the most common day-to-day strategy simply upgrades your existing instances a few at a time — no second environment, minimal extra capacity. That's the Rolling Update, and it's next.