Skip to main content

🛠️ Container Management Patterns

Getting a container running is the easy part. Keeping it healthy, watching what it uses, catching it when it falls, and shipping new versions without breaking things — that's the daily work of running containers in production. These patterns apply whether you orchestrate with Kubernetes, Docker Swarm, or plain Docker.

Week 11 · Day 5 (Friday: Container Orchestration Basics) · Lecture 3

🎯 Learning Objectives

By the end of this lesson, you will be able to:

  • Add a health check so the platform knows when a container is truly ready and alive
  • Choose the right restart policy so crashed containers recover automatically
  • Set resource limits to stop one container from starving its neighbors
  • Monitor live containers with docker stats and related commands
  • Explain the container logging model and why apps should log to stdout
  • Compare rolling, blue-green, and canary deployment strategies at a high level

Estimated Time: 65 minutes

Practice: Add a health check and restart policy to a service, then read its live resource usage.

In This Lesson

The Container Lifecycle

To manage a container you first need to picture its life. A container is created, then runs, and eventually stops — either because it finished, crashed, or was told to. Good management hooks into these transitions: a health check tells you whether "running" actually means "working," a restart policy decides what happens when it stops unexpectedly, and limits keep it well-behaved while it runs.

stateDiagram-v2 [*] --> Created Created --> Running: start Running --> Paused: pause Paused --> Running: unpause Running --> Stopped: exit or stop Stopped --> Running: restart policy Stopped --> [*]: removed

Think of a container like an employee. Hiring them (starting) isn't enough — you want regular check-ins to confirm they're okay (health checks), a plan for when they call in sick (restart policy), and a fair budget so one person doesn't hog every resource (limits). The rest of this lesson walks each of those.

💡 Cattle, not pets

A guiding idea in container ops: treat containers as interchangeable "cattle," not beloved "pets." You don't nurse a sick container back to health — you replace it. Every pattern here assumes containers are disposable and your configuration is what's precious.

Health Checks

A container can be "running" while the app inside it is frozen, deadlocked, or still starting up. A health check is a command the platform runs on a schedule to ask "are you actually okay?" If the check fails enough times, the orchestrator marks the container unhealthy and can replace it or stop sending it traffic.

In a Dockerfile

FROM node:20-alpine
WORKDIR /app
COPY . .
RUN npm ci --omit=dev
EXPOSE 3000

# Ask the app's own /health route every 30s; fail after 3 misses
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD wget --quiet --tries=1 --spider http://localhost:3000/health || exit 1

CMD ["node", "server.js"]

The check exits 0 for healthy or 1 for unhealthy. The --start-period gives a slow-booting app a grace window before failures count. In your app, expose a lightweight endpoint that returns 200 only when real dependencies (like the database) are reachable:

// A minimal health endpoint in an Express app
app.get('/health', async (req, res) => {
  try {
    await db.ping();                 // confirm a real dependency works
    res.status(200).json({ status: 'ok' });
  } catch (err) {
    res.status(503).json({ status: 'unhealthy' }); // signals "replace me"
  }
});

📖 Liveness vs. readiness

Kubernetes splits the idea in two. A liveness probe asks "is it alive, or should I restart it?" A readiness probe asks "is it ready to receive traffic yet?" A pod can be alive but not ready — for example, still warming a cache — so it stays up but gets no requests until ready.

Restart Policies

When a container exits unexpectedly, what should happen? A restart policy answers that. With plain Docker you set it per container; in an orchestrator it's part of the service definition and works hand-in-hand with self-healing.

PolicyBehaviorUse it for
noNever restart (the default)One-off tasks and scripts
on-failureRestart only on a non-zero exit, optionally cappedBatch jobs that may fail transiently
alwaysRestart whenever it stops, even after a rebootLong-running services
unless-stoppedLike always, but respects a manual stopServices you sometimes stop by hand
# Plain Docker: restart this container up to 5 times on failure
docker run -d --restart on-failure:5 --name api ghcr.io/acme/api:2.1
# In a Swarm/Compose stack, restart policy lives under deploy:
services:
  api:
    image: ghcr.io/acme/api:2.1
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s            # wait before each retry
        max_attempts: 3      # then give up and report the failure
        window: 120s         # time window used to judge success
⚠️ Watch for crash loops: If an app crashes instantly on start, an aggressive always policy makes it restart forever, burning CPU. That's why real orchestrators add exponential backoff — waiting longer between each retry — and why capping attempts matters.

Resource Limits

By default a container can use as much CPU and memory as the host allows. That's dangerous: one runaway process can starve every other container on the machine — the "noisy neighbor" problem. Setting limits (a ceiling) and reservations (a guaranteed minimum) keeps the whole host stable.

A reservation guarantees a minimum share while a limit caps the maximum a container may use Host resource budget for one container reservation guaranteed min limit: hard ceiling may burst up to the limit →
Reserve what the app always needs; cap what it may ever grab. Memory over the limit gets the container killed; CPU over the limit just gets throttled.
# Plain Docker: cap at half a CPU and 256MB of memory
docker run -d --cpus="0.5" --memory="256m" --name web nginx:1.27
# Kubernetes: requests are the guaranteed minimum, limits the ceiling
resources:
  requests:
    cpu: "100m"        # 100 millicores = 0.1 CPU, guaranteed
    memory: "128Mi"
  limits:
    cpu: "500m"        # may burst to 0.5 CPU
    memory: "256Mi"    # killed if it exceeds this

⚠️ CPU and memory behave differently at the limit

CPU is compressible — hit the limit and the container is simply slowed down (throttled). Memory is not — exceed the memory limit and the container is killed with an OOMKilled ("out of memory") error. Set memory limits with a real margin above observed usage.

Monitoring & Logging

You can't manage what you can't see. Two feeds matter: metrics (how much CPU, memory, and network a container uses) and logs (what the app is saying).

Live metrics with docker stats

# A live, top-like view of every running container's resource use
docker stats

# Just one container, one snapshot (no live refresh)
docker stats --no-stream web

Output

$ docker stats --no-stream
CONTAINER   NAME   CPU %   MEM USAGE / LIMIT   MEM %   NET I/O       PIDS
a1b2c3      web    0.14%   28.4MiB / 256MiB    11.1%   1.2kB / 640B  5
d4e5f6      api    2.03%   96.7MiB / 256MiB    37.8%   84kB / 71kB   18

Reading logs

# Follow a container's logs live, showing the last 100 lines first
docker logs -f --tail 100 web

# In Swarm or Kubernetes, target the service or pod instead
docker service logs web
kubectl logs -f deployment/web-deployment

The golden rule: log to stdout

Container apps should write logs to standard output and standard error, not to files inside the container. Why? Containers are disposable — a log file dies with the container. When the app logs to stdout, the platform captures the stream and a central system can collect it. This is one of the twelve-factor app principles.

graph LR A["App logs to stdout"] --> B["Container runtime captures stream"] B --> C["Log agent on each node"] C --> D["Central store: Elasticsearch or Loki"] D --> E["Dashboard: Kibana or Grafana"]

For metrics at scale, teams pair Prometheus (collects and stores time-series metrics) with Grafana (dashboards and alerts). At an intro level, know that docker stats is your quick local check, and Prometheus plus Grafana is where you graduate to for a real cluster.

Deployment Strategies

Shipping a new version is a management problem too: how do you replace v1 with v2 without downtime or a scary all-at-once switch? Three common strategies trade off risk, speed, and cost.

Rolling update

Replace old copies with new ones a few at a time. The app stays up throughout, and if a new copy is unhealthy the rollout can pause. This is the default in both Kubernetes and Swarm.

graph LR S0["4x v1"] --> S1["3x v1 + 1x v2"] S1 --> S2["2x v1 + 2x v2"] S2 --> S3["4x v2"]

Blue-green

Run the new version (green) as a full, separate environment alongside the current one (blue). Test green in isolation, then flip all traffic to it at once. Rollback is instant — just flip back. It's safe but needs double the resources during the switch.

Canary

Release the new version to a small slice of traffic first — say 5% — watch its error rates and latency, then widen gradually if it behaves. It catches problems with real users while limiting the blast radius.

StrategyDowntimeRollbackExtra cost
RollingNoneGradual (roll back the same way)Low
Blue-greenNoneInstant flipHigh (2x during switch)
CanaryNoneFast (pull the small slice)Low to medium
💡 Start simple: Rolling updates cover the vast majority of needs and come free with your orchestrator. Reach for blue-green or canary when a release is high-stakes and you need extra safety.

Practice & Quiz

🏋️ Exercise 1: Make a service self-managing

Goal: Write the deploy: and healthcheck: sections for a Compose service api (image ghcr.io/acme/api:2.1) that runs 3 replicas, restarts on failure up to 3 times, limits memory to 256MB, and probes http://localhost:3000/health every 30 seconds.

💡 Hint

Replicas, restart policy, and resource limits go under deploy:. The healthcheck: block is a sibling of image: and uses a test: command that exits non-zero on failure.

✅ Solution
services:
  api:
    image: ghcr.io/acme/api:2.1
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000/health"]
      interval: 30s
      timeout: 3s
      retries: 3
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure
        max_attempts: 3
      resources:
        limits:
          memory: 256M

🏋️ Exercise 2: Diagnose OOMKilled

Goal: A container keeps dying and its status shows OOMKilled. What does that mean, and what are two reasonable fixes?

✅ Solution

OOMKilled means the container exceeded its memory limit and was terminated, because memory is not compressible. Two fixes: (1) raise the memory limit to a value comfortably above the app's real peak usage (check with docker stats), or (2) fix the app's memory use — a leak, an unbounded cache, or loading too much data at once. Simply removing the limit is not a good fix, since it just moves the risk to the whole host.

🎯 Quick Quiz

Question 1: Why should a containerized app log to stdout instead of a file?

Question 2: What happens when a container exceeds its memory limit?

Question 3: Which deployment strategy sends a small slice of traffic to the new version first?

Best Practices & Pitfalls

✅ Do

  • Add a health check that verifies real dependencies, not just that the process is up
  • Set a restart policy so transient crashes recover without you
  • Always set memory limits (and requests) to protect the host from noisy neighbors
  • Log to stdout/stderr in a structured format like JSON, with request IDs for context
  • Prefer rolling updates by default; save blue-green and canary for risky releases

❌ Don't

  • Don't run production containers with no limits and no restart policy
  • Don't write logs to files inside the container — they vanish when it's replaced
  • Don't nurse a broken container back to health; replace it (cattle, not pets)
  • Don't set a health check so strict that a slow startup gets the container killed — use a start period

⚠️ A missing health check hides failures

Without a health check, an orchestrator only knows whether the process is running — not whether it's serving requests. A frozen app can keep receiving traffic that all times out. The health check is what turns "running" into "actually working."

Summary

🎉 Key Takeaways

  • Health checks let the platform tell "running" from "actually working," and drive readiness and self-healing
  • Restart policies recover crashed containers automatically; watch for crash loops and use backoff
  • Resource limits prevent noisy-neighbor problems — CPU throttles at the limit, memory gets OOMKilled
  • Monitor with docker stats and central logging; apps should log to stdout
  • Rolling, blue-green, and canary deployments trade off risk, speed, and cost — rolling is the sensible default

📚 Additional Resources

🚀 What's Next?

You've covered orchestration and the patterns that keep containers healthy. Time to put it all together in a hands-on build: next you'll containerize a full-stack application with Docker Compose, wiring a web app, an API, and a database into one coordinated stack.

🎉 You can run containers like a pro!

Health, restarts, limits, monitoring, and safe rollouts are the difference between a demo and a production system that stays up.