🛠️ Container Management Patterns
Getting a container running is the easy part. Keeping it healthy, watching what it uses, catching it when it falls, and shipping new versions without breaking things — that's the daily work of running containers in production. These patterns apply whether you orchestrate with Kubernetes, Docker Swarm, or plain Docker.
Week 11 · Day 5 (Friday: Container Orchestration Basics) · Lecture 3
🎯 Learning Objectives
By the end of this lesson, you will be able to:
- Add a health check so the platform knows when a container is truly ready and alive
- Choose the right restart policy so crashed containers recover automatically
- Set resource limits to stop one container from starving its neighbors
- Monitor live containers with
docker statsand related commands - Explain the container logging model and why apps should log to stdout
- Compare rolling, blue-green, and canary deployment strategies at a high level
Estimated Time: 65 minutes
Practice: Add a health check and restart policy to a service, then read its live resource usage.
In This Lesson
The Container Lifecycle
To manage a container you first need to picture its life. A container is created, then runs, and eventually stops — either because it finished, crashed, or was told to. Good management hooks into these transitions: a health check tells you whether "running" actually means "working," a restart policy decides what happens when it stops unexpectedly, and limits keep it well-behaved while it runs.
Think of a container like an employee. Hiring them (starting) isn't enough — you want regular check-ins to confirm they're okay (health checks), a plan for when they call in sick (restart policy), and a fair budget so one person doesn't hog every resource (limits). The rest of this lesson walks each of those.
💡 Cattle, not pets
A guiding idea in container ops: treat containers as interchangeable "cattle," not beloved "pets." You don't nurse a sick container back to health — you replace it. Every pattern here assumes containers are disposable and your configuration is what's precious.
Health Checks
A container can be "running" while the app inside it is frozen, deadlocked, or still starting up. A health check is a command the platform runs on a schedule to ask "are you actually okay?" If the check fails enough times, the orchestrator marks the container unhealthy and can replace it or stop sending it traffic.
In a Dockerfile
FROM node:20-alpine
WORKDIR /app
COPY . .
RUN npm ci --omit=dev
EXPOSE 3000
# Ask the app's own /health route every 30s; fail after 3 misses
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD wget --quiet --tries=1 --spider http://localhost:3000/health || exit 1
CMD ["node", "server.js"]
The check exits 0 for healthy or 1 for unhealthy. The --start-period gives a slow-booting app a grace window before failures count. In your app, expose a lightweight endpoint that returns 200 only when real dependencies (like the database) are reachable:
// A minimal health endpoint in an Express app
app.get('/health', async (req, res) => {
try {
await db.ping(); // confirm a real dependency works
res.status(200).json({ status: 'ok' });
} catch (err) {
res.status(503).json({ status: 'unhealthy' }); // signals "replace me"
}
});
📖 Liveness vs. readiness
Kubernetes splits the idea in two. A liveness probe asks "is it alive, or should I restart it?" A readiness probe asks "is it ready to receive traffic yet?" A pod can be alive but not ready — for example, still warming a cache — so it stays up but gets no requests until ready.
Restart Policies
When a container exits unexpectedly, what should happen? A restart policy answers that. With plain Docker you set it per container; in an orchestrator it's part of the service definition and works hand-in-hand with self-healing.
| Policy | Behavior | Use it for |
|---|---|---|
no | Never restart (the default) | One-off tasks and scripts |
on-failure | Restart only on a non-zero exit, optionally capped | Batch jobs that may fail transiently |
always | Restart whenever it stops, even after a reboot | Long-running services |
unless-stopped | Like always, but respects a manual stop | Services you sometimes stop by hand |
# Plain Docker: restart this container up to 5 times on failure
docker run -d --restart on-failure:5 --name api ghcr.io/acme/api:2.1
# In a Swarm/Compose stack, restart policy lives under deploy:
services:
api:
image: ghcr.io/acme/api:2.1
deploy:
restart_policy:
condition: on-failure
delay: 5s # wait before each retry
max_attempts: 3 # then give up and report the failure
window: 120s # time window used to judge success
⚠️ Watch for crash loops: If an app crashes instantly on start, an aggressive always policy makes it restart forever, burning CPU. That's why real orchestrators add exponential backoff — waiting longer between each retry — and why capping attempts matters.
Resource Limits
By default a container can use as much CPU and memory as the host allows. That's dangerous: one runaway process can starve every other container on the machine — the "noisy neighbor" problem. Setting limits (a ceiling) and reservations (a guaranteed minimum) keeps the whole host stable.
# Plain Docker: cap at half a CPU and 256MB of memory
docker run -d --cpus="0.5" --memory="256m" --name web nginx:1.27
# Kubernetes: requests are the guaranteed minimum, limits the ceiling
resources:
requests:
cpu: "100m" # 100 millicores = 0.1 CPU, guaranteed
memory: "128Mi"
limits:
cpu: "500m" # may burst to 0.5 CPU
memory: "256Mi" # killed if it exceeds this
⚠️ CPU and memory behave differently at the limit
CPU is compressible — hit the limit and the container is simply slowed down (throttled). Memory is not — exceed the memory limit and the container is killed with an OOMKilled ("out of memory") error. Set memory limits with a real margin above observed usage.
Monitoring & Logging
You can't manage what you can't see. Two feeds matter: metrics (how much CPU, memory, and network a container uses) and logs (what the app is saying).
Live metrics with docker stats
# A live, top-like view of every running container's resource use
docker stats
# Just one container, one snapshot (no live refresh)
docker stats --no-stream web
Output
$ docker stats --no-stream
CONTAINER NAME CPU % MEM USAGE / LIMIT MEM % NET I/O PIDS
a1b2c3 web 0.14% 28.4MiB / 256MiB 11.1% 1.2kB / 640B 5
d4e5f6 api 2.03% 96.7MiB / 256MiB 37.8% 84kB / 71kB 18
Reading logs
# Follow a container's logs live, showing the last 100 lines first
docker logs -f --tail 100 web
# In Swarm or Kubernetes, target the service or pod instead
docker service logs web
kubectl logs -f deployment/web-deployment
The golden rule: log to stdout
Container apps should write logs to standard output and standard error, not to files inside the container. Why? Containers are disposable — a log file dies with the container. When the app logs to stdout, the platform captures the stream and a central system can collect it. This is one of the twelve-factor app principles.
For metrics at scale, teams pair Prometheus (collects and stores time-series metrics) with Grafana (dashboards and alerts). At an intro level, know that docker stats is your quick local check, and Prometheus plus Grafana is where you graduate to for a real cluster.
Deployment Strategies
Shipping a new version is a management problem too: how do you replace v1 with v2 without downtime or a scary all-at-once switch? Three common strategies trade off risk, speed, and cost.
Rolling update
Replace old copies with new ones a few at a time. The app stays up throughout, and if a new copy is unhealthy the rollout can pause. This is the default in both Kubernetes and Swarm.
Blue-green
Run the new version (green) as a full, separate environment alongside the current one (blue). Test green in isolation, then flip all traffic to it at once. Rollback is instant — just flip back. It's safe but needs double the resources during the switch.
Canary
Release the new version to a small slice of traffic first — say 5% — watch its error rates and latency, then widen gradually if it behaves. It catches problems with real users while limiting the blast radius.
| Strategy | Downtime | Rollback | Extra cost |
|---|---|---|---|
| Rolling | None | Gradual (roll back the same way) | Low |
| Blue-green | None | Instant flip | High (2x during switch) |
| Canary | None | Fast (pull the small slice) | Low to medium |
💡 Start simple: Rolling updates cover the vast majority of needs and come free with your orchestrator. Reach for blue-green or canary when a release is high-stakes and you need extra safety.
Practice & Quiz
🏋️ Exercise 1: Make a service self-managing
Goal: Write the deploy: and healthcheck: sections for a Compose service api (image ghcr.io/acme/api:2.1) that runs 3 replicas, restarts on failure up to 3 times, limits memory to 256MB, and probes http://localhost:3000/health every 30 seconds.
💡 Hint
Replicas, restart policy, and resource limits go under deploy:. The healthcheck: block is a sibling of image: and uses a test: command that exits non-zero on failure.
✅ Solution
services:
api:
image: ghcr.io/acme/api:2.1
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000/health"]
interval: 30s
timeout: 3s
retries: 3
deploy:
replicas: 3
restart_policy:
condition: on-failure
max_attempts: 3
resources:
limits:
memory: 256M
🏋️ Exercise 2: Diagnose OOMKilled
Goal: A container keeps dying and its status shows OOMKilled. What does that mean, and what are two reasonable fixes?
✅ Solution
OOMKilled means the container exceeded its memory limit and was terminated, because memory is not compressible. Two fixes: (1) raise the memory limit to a value comfortably above the app's real peak usage (check with docker stats), or (2) fix the app's memory use — a leak, an unbounded cache, or loading too much data at once. Simply removing the limit is not a good fix, since it just moves the risk to the whole host.
🎯 Quick Quiz
Question 1: Why should a containerized app log to stdout instead of a file?
Question 2: What happens when a container exceeds its memory limit?
Question 3: Which deployment strategy sends a small slice of traffic to the new version first?
Best Practices & Pitfalls
✅ Do
- Add a health check that verifies real dependencies, not just that the process is up
- Set a restart policy so transient crashes recover without you
- Always set memory limits (and requests) to protect the host from noisy neighbors
- Log to stdout/stderr in a structured format like JSON, with request IDs for context
- Prefer rolling updates by default; save blue-green and canary for risky releases
❌ Don't
- Don't run production containers with no limits and no restart policy
- Don't write logs to files inside the container — they vanish when it's replaced
- Don't nurse a broken container back to health; replace it (cattle, not pets)
- Don't set a health check so strict that a slow startup gets the container killed — use a start period
⚠️ A missing health check hides failures
Without a health check, an orchestrator only knows whether the process is running — not whether it's serving requests. A frozen app can keep receiving traffic that all times out. The health check is what turns "running" into "actually working."
Summary
🎉 Key Takeaways
- Health checks let the platform tell "running" from "actually working," and drive readiness and self-healing
- Restart policies recover crashed containers automatically; watch for crash loops and use backoff
- Resource limits prevent noisy-neighbor problems — CPU throttles at the limit, memory gets OOMKilled
- Monitor with
docker statsand central logging; apps should log to stdout - Rolling, blue-green, and canary deployments trade off risk, speed, and cost — rolling is the sensible default
📚 Additional Resources
- Docker — HEALTHCHECK instruction reference
- Docker — Restart policies and auto-starting containers
- Docker — Runtime resource constraints
- Kubernetes — Liveness, readiness, and startup probes
- The Twelve-Factor App — Logs
🚀 What's Next?
You've covered orchestration and the patterns that keep containers healthy. Time to put it all together in a hands-on build: next you'll containerize a full-stack application with Docker Compose, wiring a web app, an API, and a database into one coordinated stack.
🎉 You can run containers like a pro!
Health, restarts, limits, monitoring, and safe rollouts are the difference between a demo and a production system that stays up.