π΅π’ Blue-Green Deployment
Your app is live and thousands of people are using it right now. You need to ship a new version β but "taking the site down for maintenance" is not an option. Blue-green deployment solves this the way a stage crew changes sets between acts: build the new scene completely off to the side, then flip one switch so the audience never sees the dark stage.
Week 13 · Day 2 (Tuesday: Deployment Strategies) · Lecture 1
π― Learning Objectives
By the end of this lesson, you will be able to:
- Explain what a blue-green deployment is and how it delivers zero-downtime releases
- Describe how a load balancer or DNS record switches all traffic from one environment to the other in seconds
- Perform an instant rollback by pointing traffic back at the previous environment
- Apply the expand-contract pattern to keep database migrations backward-compatible across a switch
- Externalize session state so users are not logged out when the active environment changes
- Implement a real Kubernetes blue-green switch with a health-gated cutover script
Estimated Time: 70 minutes
Practice: Design a blue-green setup for a three-tier app and write the traffic-switch + rollback logic.
In This Lesson
Why Deployment Strategies Matter
A deployment strategy is simply how you replace the running version of your app with a new one. The naΓ―ve approach β stop the old version, start the new one β leaves a window where nobody can use the site. For a hobby project that might be fine. For an e-commerce checkout, every minute of downtime is lost revenue and lost trust.
Modern strategies exist to shrink that window to zero. The three you'll study this week each make a different trade-off between speed, cost, and risk:
stop old, start new β has downtime"] A --> C["Blue-Green
switch between two full environments"] A --> D["Canary
send a small percent of traffic to the new version"] A --> E["Rolling Update
replace instances batch by batch"]
π£οΈ The highway construction analogy
Picture renovating a busy highway. Recreate closes the whole road β fast work, furious drivers. Rolling update closes one lane at a time β traffic slows but keeps moving. Blue-green builds a brand-new highway alongside the old one and redirects everyone at once when it's ready. Canary opens a single lane of the new highway to a trickle of cars first, then widens it as confidence grows.
This lesson focuses on blue-green: the cleanest mental model and the easiest to reason about, because at any moment every user sees exactly one version.
How Blue-Green Works
You run two identical production environments, named "Blue" and "Green." They are identical in every way β same infrastructure, same configuration, same scale β except for the version of the application code they run. Only one is live (serving real users) at a time; the other is idle, ready to become the next live environment.
The lifecycle of one release looks like this:
The key ideas
- Identical environments. The only difference is the app version. Keep them in sync with the same infrastructure-as-code so there is no "works on Blue but not Green" surprise.
- A switch in front. A load balancer, router, or DNS record decides which environment is live. Changing that one setting redirects everyone.
- Atomic cutover. Traffic moves all at once. No user ever gets a mix of old and new β a huge simplification compared to strategies where versions overlap.
- Instant rollback. The old environment is still running. If the new one misbehaves, flip the switch back.
- Shared backing services. Databases, queues, and caches are usually shared, which is exactly where the hard parts live (covered below).
The Traffic Switch & Rollback
"Flip the switch" can mean different things depending on where the switch lives. Each level trades simplicity for speed.
| Switch level | Mechanism | Cutover speed | Watch out for |
|---|---|---|---|
| DNS | Repoint a DNS record to the new environment | Slow (minutes to hours) | TTL caching means some clients keep hitting the old IP |
| Load balancer | Change the listener's target group / backend pool | Instant (seconds) | Provider-specific, but the recommended default |
| Orchestrator | Update a Service selector (Kubernetes, ECS) | Instant | Integrated with CI/CD; more moving parts to learn |
β οΈ Why DNS is a trap for the cutover
DNS records are cached by resolvers and browsers for the duration of their TTL. Even with a 60-second TTL, some clients ignore it and keep the old address for far longer. That means a period of mixed traffic where you can't cleanly say "everyone is on Green now." Prefer a load balancer switch, whose effect is immediate and total.
Rollback is the whole point
The reason blue-green feels safe is that a rollback is the same operation as the deploy, just aimed the other way. There is no rebuild, no re-deploy, no waiting β the previous version is still warm and running.
# Cut over to Green (make it live)
aws elbv2 modify-listener \
--listener-arn "$LISTENER_ARN" \
--default-actions Type=forward,TargetGroupArn="$GREEN_TG_ARN"
# Something's wrong? Roll back instantly by pointing at Blue again.
aws elbv2 modify-listener \
--listener-arn "$LISTENER_ARN" \
--default-actions Type=forward,TargetGroupArn="$BLUE_TG_ARN"
β Keep the old environment warm
Do not tear down Blue the instant Green goes live. Keep it running for a "bake" period (often a few hours to a day) so that if a subtle bug surfaces under real load, rollback is one command away. Only scale Blue down once you're confident.
Database Migrations: The Hard Part
Here's the catch that trips up every newcomer: Blue and Green usually share one database. You can flip application code atomically, but you cannot keep two copies of your users' live data in sync. So the schema must work with both versions of the app during the switch and rollback window.
old code ignores it"] C["Add a new table"] --> B D["Change a column type"] --> E["Medium risk
needs a transition"] F["Drop a column or table"] --> G["High risk
breaks rollback if old code needs it"]
The expand-contract pattern
Never make a breaking change in one step. Split it across multiple deploys so that at every moment, the live schema is compatible with the code that could be running:
- Expand. Add the new column/table alongside the old. Both versions of the app can now read the database.
- Migrate & deploy. Ship the new code that writes to both old and new, and backfill existing rows in the background.
- Contract. Only after the new version is proven stable (and rollback is no longer needed) do you drop the old column in a later release.
-- STEP 1 (Expand): additive change, safe for both app versions
ALTER TABLE users ADD COLUMN phone_number VARCHAR(20);
-- Old code never touches this column, so it keeps working.
-- STEP 3 (Contract): only in a FUTURE release, after v1.1 is trusted
-- ALTER TABLE users DROP COLUMN legacy_phone;
During the transition, the application writes to both fields so a rollback to the old code still finds the data it expects:
// Transitional write path (runs in the new version)
async function updateUserPhone(userId, phone) {
const updates = { legacy_phone: phone }; // keep the old column populated...
// ...and also fill the new column when the schema has it.
if (await columnExists('users', 'phone_number')) {
updates.phone_number = phone;
}
await db.update('users', userId, updates);
}
π‘ Decouple "deploy" from "release"
A feature flag lets you ship code that is dormant until you turn it on. Combined with blue-green, this means the risky new feature can be enabled after the switch, and disabled instantly without touching the deployment. Deploy strategies move code; feature flags control behavior. Use both.
Handling State & Sessions
If your app keeps user sessions in the memory of a specific server, switching environments logs everyone out β their session lived on Blue, and Blue is no longer serving them. The fix is to externalize state so it survives the cutover.
| Kind of state | Where it should live |
|---|---|
| User sessions / carts | A shared store like Redis, not server memory |
| Cached data | A shared cache (Redis/Memcached) both environments read |
| Background jobs | A durable queue, so work isn't lost when a worker stops |
| WebSocket connections | Drain gracefully; ask clients to reconnect to the new environment |
// Sessions in Redis survive the blue-green switch because BOTH
// environments read from the same store β not from local memory.
import express from 'express';
import session from 'express-session';
import { RedisStore } from 'connect-redis';
import { createClient } from 'redis';
const app = express();
const redisClient = createClient({ url: process.env.REDIS_URL });
await redisClient.connect();
app.use(session({
store: new RedisStore({ client: redisClient }),
secret: process.env.SESSION_SECRET,
resave: false,
saveUninitialized: false,
cookie: { secure: process.env.NODE_ENV === 'production', maxAge: 86400000 } // 1 day
}));
// A user logged in on Blue stays logged in on Green β the cookie
// points at a session that lives in shared Redis, not on any one box.
app.get('/profile', (req, res) => {
if (!req.session.user) return res.redirect('/login');
res.json({ user: req.session.user });
});
β The golden rule: keep app servers stateless
If any two requests from the same user could safely hit different servers, blue-green (and every other zero-downtime strategy) becomes dramatically simpler. Push all durable state to databases, caches, and queues.
A Real Kubernetes Switch
In Kubernetes the "switch" is a Service selector. A Service routes to whichever Pods match its label selector, so changing one label β version: blue to version: green β instantly moves all traffic.
# The Service is the switch. Its selector decides which Deployment is live.
apiVersion: v1
kind: Service
metadata:
name: my-app
spec:
selector:
app: my-app
version: blue # flip to "green" to cut over
ports:
- port: 80
targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app-green
spec:
replicas: 3
selector:
matchLabels: { app: my-app, version: green }
template:
metadata:
labels: { app: my-app, version: green }
spec:
containers:
- name: my-app
image: my-app:1.1
ports:
- containerPort: 8080
readinessProbe: # gate: no traffic until this passes
httpGet: { path: /health, port: 8080 }
initialDelaySeconds: 5
periodSeconds: 5
A safe cutover script deploys Green, waits for it to be healthy, smoke-tests it, then flips the selector β with a built-in rollback if the switch doesn't take:
#!/usr/bin/env bash
set -euo pipefail
# 1. Roll out the green version and wait until every pod is ready.
kubectl apply -f deployment-green.yaml
kubectl rollout status deployment/my-app-green --timeout=120s
# 2. Smoke-test green privately BEFORE it takes live traffic.
if ! kubectl run smoke --rm -i --restart=Never --image=curlimages/curl -- \
-sf http://my-app-green/health; then
echo "Green failed its health check β aborting, blue stays live."
exit 1
fi
# 3. Flip the switch: point the Service at green.
kubectl patch service my-app -p '{"spec":{"selector":{"version":"green"}}}'
echo "Traffic switched to green."
# 4. Verify, and roll back automatically if green isn't serving v1.1.
sleep 10
if ! curl -sf http://my-app/version | grep -q '1.1'; then
echo "Live traffic not reaching green β rolling back to blue."
kubectl patch service my-app -p '{"spec":{"selector":{"version":"blue"}}}'
exit 1
fi
echo "Deployment healthy. Keeping blue warm for the bake period."
β οΈ A health check must mean "ready for real traffic"
A check that only returns 200 OK because the process started is worthless β the pod might not have a database connection yet. Make readiness probes verify the dependencies that requests actually need, so the switch never sends users to a half-initialized environment.
Practice & Quiz
ποΈ Exercise 1: Design the switch and rollback
Goal: For a three-tier app (React SPA, Node API, PostgreSQL) with Redis-backed sessions, sketch a blue-green plan. Decide: where does the switch live? How is the database change made safe? How do sessions survive the cutover? What is the exact rollback step?
π‘ Hint
Put the switch at the load balancer (not DNS). Use expand-contract for the schema. Sessions already live in Redis, which both environments share. The rollback is a single load-balancer change back to the previous target group.
β Solution
- Switch: ALB listener forwards to Blue or Green target group. Cutover = one
modify-listenercall. - Database: Apply additive migrations first (expand). New API writes to both old and new columns. Drop old columns only in a later release (contract).
- Sessions: Already externalized to Redis, so a user on Blue stays logged in on Green.
- Rollback: Point the listener back at the Blue target group β instant, because Blue is still running.
ποΈ Exercise 2: Order the migration steps
Goal: You must rename users.email_addr to users.email during a blue-green release without breaking rollback. Put these steps in the correct order: (a) drop email_addr, (b) add email and copy data, (c) deploy code reading email and writing both, (d) confirm stability for a day.
β Solution
Order: b β c β d β a. Add the new column and backfill (expand), deploy code that writes both columns, bake to confirm the new version is trusted, then drop the old column in a later release (contract). A one-step rename would break the old code the moment you rolled back.
π― Quick Quiz
Question 1: What is the main advantage of switching traffic at the load balancer instead of via DNS?
Question 2: Why must you avoid dropping a column in the same release that switches to the new version?
Question 3: Where should user sessions live so they survive a blue-green switch?
Best Practices & Pitfalls
β Do
- Keep Blue and Green truly identical using infrastructure-as-code
- Gate the switch on a deep health check that verifies real dependencies
- Use expand-contract so every migration is backward-compatible
- Externalize sessions, cache, and jobs so app servers stay stateless
- Keep the old environment warm for a bake period before scaling it down
- Automate both the switch and the rollback β never do the cutover by hand
β Don't
- Don't rely on DNS for an instant cutover β TTL caching causes mixed traffic
- Don't drop columns or make breaking schema changes in the switch release
- Don't tear down Blue the moment Green goes live β you lose your instant rollback
- Don't let configuration drift between the two environments
- Don't ship a "started = healthy" probe that lets traffic hit a half-ready app
β οΈ The cost trade-off
Running two full production environments roughly doubles infrastructure cost during a deploy. Many teams accept this only around releases (spin Green up, cut over, scale Blue down) rather than 24/7. If double capacity is unacceptable, a rolling update β covered later this week β needs far less headroom.
Summary
π Key Takeaways
- Blue-green runs two identical environments; only one is live at a time
- A load-balancer switch moves all traffic atomically in seconds β no mixed versions
- Rollback is instant because the previous environment is still warm
- The shared database is the hard part: use the expand-contract pattern to stay backward-compatible
- Externalize state (Redis sessions, durable queues) so app servers are stateless and survive the switch
- Gate the cutover on deep health checks, and keep old capacity warm during a bake period
π Additional Resources
- AWS β Blue-green deployments overview
- Google Cloud β Application deployment and testing strategies
- Kubernetes β Service (the routing switch)
π What's Next?
Blue-green flips everyone at once. But what if you want to test a new version on just a sliver of real users first, watching the metrics before committing? That gradual, risk-minimizing approach is the next lesson: Canary Releases.
π Great work!
You can now ship a new version with zero downtime and an instant escape hatch. Next up, we make the rollout even more cautious.