Docker Swarm Overview

Understanding Docker's native orchestration solution for container clustering

What is Docker Swarm?

Docker Swarm is Docker's native clustering and orchestration solution that turns a pool of Docker hosts into a single, virtual Docker host. It allows you to deploy and manage a cluster of Docker nodes to run containerized applications at scale with built-in orchestration capabilities.

The Beehive Analogy

Docker Swarm can be compared to a beehive:

  • Individual bees are like Docker containers—each performs specific tasks
  • Worker bees are like worker nodes in the swarm—they execute the actual tasks
  • Queen bee is like the manager node—it coordinates and directs the workers
  • Honeycomb structure represents the overlay network—it provides organization and communication paths
  • Bee dance communication is like service discovery—sharing information about where resources are located

Just as bees work together in a highly organized manner to achieve complex goals, Docker Swarm coordinates containers across multiple hosts to build resilient, scalable applications.

Docker Swarm Architecture

flowchart TB subgraph "Docker Swarm Cluster" subgraph "Manager Nodes" M1[Manager 1\nLeader] M2[Manager 2] M3[Manager 3] M1 --- M2 M1 --- M3 M2 --- M3 end subgraph "Worker Nodes" W1[Worker 1] W2[Worker 2] W3[Worker 3] W4[Worker 4] end M1 --> W1 M1 --> W2 M1 --> W3 M1 --> W4 subgraph "Services" S1[Service 1\n3 Replicas] S2[Service 2\n5 Replicas] end W1 --- T1[Task 1.1] W2 --- T2[Task 1.2] W2 --- T3[Task 1.3] W1 --- T4[Task 2.1] W3 --- T5[Task 2.2] W3 --- T6[Task 2.3] W4 --- T7[Task 2.4] W4 --- T8[Task 2.5] S1 -.-> T1 S1 -.-> T2 S1 -.-> T3 S2 -.-> T4 S2 -.-> T5 S2 -.-> T6 S2 -.-> T7 S2 -.-> T8 end

Key Features of Docker Swarm

Docker Swarm Key Features Cluster Management Declarative state model Self-organizing and self-healing nodes Automatic load balancing Scalability Scale services up or down with a single command Balance across nodes Security Automatic TLS encryption for control plane Certificate rotation Mutual authentication Simplicity Easy to set up and use Standard Docker CLI Docker Compose support Low learning curve Service Discovery Automatic DNS-based service discovery Internal load balancing Round-robin by default Load Balancing Ingress networking External and internal load balancing Multiple strategies Rolling Updates Progressive deployment with rollback Multi-host Networking Overlay networks across nodes

Cluster Management

Simplicity and Integration

Networking Features

Security Features

Rolling Updates and Scaling

Docker Swarm vs. Kubernetes

Both Docker Swarm and Kubernetes are container orchestration platforms, but they have different focuses and strengths:

Feature Docker Swarm Kubernetes
Setup and Installation Very simple, built into Docker More complex, requires additional setup
Learning Curve Gentle, familiar Docker commands Steeper, many new concepts to learn
Scalability Good for small to medium deployments Excellent for large-scale deployments
Feature Set Focused, core orchestration features Comprehensive, extensive capabilities
Community & Ecosystem Smaller community, fewer integrations Massive community, vast ecosystem
Auto-healing Basic auto-healing capabilities Advanced self-healing across various levels
Load Balancing Native load balancing with routing mesh Various load balancing options (Services, Ingress)
Rolling Updates Supported with basic controls Extensive update and rollback strategies
Monitoring & Logging Basic, requires additional tools Rich ecosystem of monitoring solutions
Industry Adoption Less widely adopted Industry standard, widespread adoption

When to Choose Docker Swarm

When to Choose Kubernetes

Docker Swarm Architecture

Node Types

A Docker Swarm cluster consists of two types of nodes:

A node can be both a manager and a worker simultaneously.

flowchart TB subgraph "Manager Nodes" direction TB M1[Manager 1\nLeader] -- Raft Consensus --> M2[Manager 2] M1 -- Raft Consensus --> M3[Manager 3] M2 -- Raft Consensus --> M3 M1 -.- SwarmState[Swarm State] M2 -.- SwarmState M3 -.- SwarmState end subgraph "Worker Nodes" W1[Worker 1] W2[Worker 2] W3[Worker 3] end M1 -- "Control API\nAssign Tasks" --> W1 M1 -- "Control API\nAssign Tasks" --> W2 M1 -- "Control API\nAssign Tasks" --> W3 W1 -- Status Updates --> M1 W2 -- Status Updates --> M1 W3 -- Status Updates --> M1

Manager Nodes

Manager nodes handle the control plane of the Swarm:

Worker Nodes

Worker nodes are responsible for running containers:

Raft Consensus

Swarm uses the Raft consensus algorithm for manager node coordination:

Recommended manager node configurations:

Key Docker Swarm Concepts

Services

Services are the primary building blocks in Docker Swarm:

# Create a replicated service with 3 replicas
docker service create --name webapp --replicas 3 -p 80:80 nginx

# Create a global service (one per node)
docker service create --name monitoring --mode global monitoring-agent:latest

Tasks

Tasks are the individual instances of a service:

Stacks

Stacks are groups of related services that share dependencies:

# Deploy a stack from a compose file
docker stack deploy -c docker-compose.yml myapp

Networking

Swarm provides several network types:

# Create an overlay network
docker network create --driver overlay mynetwork

# Create a service connected to the network
docker service create --name myservice --network mynetwork myimage

Volumes

Docker Swarm supports data persistence through volumes:

# Create a service with a volume
docker service create --name db --mount type=volume,source=dbdata,target=/var/lib/mysql mysql

Secrets

Secrets allow secure storage of sensitive data:

# Create a secret
echo "mypassword" | docker secret create db_password -

# Use the secret in a service
docker service create --name db --secret db_password mysql

Configs

Configs are similar to secrets but for non-sensitive configuration:

# Create a config from a file
docker config create nginx_conf nginx.conf

# Use the config in a service
docker service create --name web --config source=nginx_conf,target=/etc/nginx/nginx.conf nginx

Setting Up a Docker Swarm Cluster

Initialize a Swarm Cluster

Creating a new Swarm is straightforward:

# Initialize a new swarm on the first node (becomes a manager)
docker swarm init --advertise-addr 192.168.1.10

# This command outputs a token for adding worker nodes
# Example output:
# Swarm initialized: current node (dxn1zf6l61qsb1josjja83ngz) is now a manager.
# 
# To add a worker to this swarm, run the following command:
#     docker swarm join --token SWMTKN-1-49nj1cmql... 192.168.1.10:2377

Joining Worker Nodes

Add additional nodes to the swarm as workers:

# On worker nodes, run the join command with the token
docker swarm join --token SWMTKN-1-49nj1cmql... 192.168.1.10:2377

# Verify node joined
# Example output:
# This node joined a swarm as a worker.

Adding Manager Nodes

For high availability, add additional manager nodes:

# Get the manager join token on an existing manager
docker swarm join-token manager

# Output will include the join command and token for managers
# Join a new manager node
docker swarm join --token SWMTKN-1-61ztec5kyafptydic... 192.168.1.10:2377

Inspecting the Swarm

View information about the swarm and its nodes:

# List all nodes in the swarm
docker node ls

# Inspect a specific node
docker node inspect node-id

# View swarm information
docker info

Node Management

Manage nodes in the swarm:

# Promote a worker to manager
docker node promote worker-node-id

# Demote a manager to worker
docker node demote manager-node-id

# Drain a node for maintenance (move tasks away)
docker node update --availability drain node-id

# Make a node active again
docker node update --availability active node-id

# Remove a node from the swarm (run on the node to remove)
docker swarm leave

# Force remove a node (run on a manager)
docker node rm node-id

Managing Services in Docker Swarm

Creating Services

Deploy services to the swarm:

# Create a simple service
docker service create --name webserver --replicas 3 -p 80:80 nginx

# Create a service with constraints
docker service create --name db \
  --constraint 'node.labels.type == database' \
  --replicas 1 \
  -e MYSQL_ROOT_PASSWORD=password \
  mysql:5.7

# Create a service with resource limits
docker service create --name cpu-limited \
  --limit-cpu 0.5 \
  --limit-memory 512M \
  --reserve-cpu 0.1 \
  --reserve-memory 128M \
  --replicas 3 \
  nginx

Inspecting Services

View information about services:

# List all services
docker service ls

# Inspect a specific service
docker service inspect service-name

# View service logs
docker service logs service-name

# View tasks (containers) for a service
docker service ps service-name

Scaling Services

Adjust the number of replicas:

# Scale a service up
docker service scale webserver=5

# Scale multiple services at once
docker service scale webserver=3 api=2 db=1

Updating Services

Apply rolling updates to services:

# Update the image used by a service
docker service update --image nginx:1.19 webserver

# Update with specific update configurations
docker service update \
  --update-parallelism 2 \
  --update-delay 10s \
  --update-failure-action rollback \
  --image nginx:1.19 \
  webserver

Rolling Back Updates

Revert to previous service configurations:

# Rollback a service to its previous configuration
docker service update --rollback webserver

Removing Services

Clean up services when no longer needed:

# Remove a service
docker service rm webserver

# Remove multiple services
docker service rm webserver api db

Docker Swarm Networking

Network Types

Docker Swarm works with several network types:

flowchart TD subgraph "Node 1" direction TB C1[Container 1] C2[Container 2] C1 --- OverlayN1[Overlay Network] C2 --- OverlayN1 end subgraph "Node 2" direction TB C3[Container 3] C4[Container 4] C3 --- OverlayN2[Overlay Network] C4 --- OverlayN2 end OverlayN1 -.- OverlayN2 Client[External Client] --- LoadBalancer[Routing Mesh] LoadBalancer --- C1 LoadBalancer --- C3

Creating Overlay Networks

# Create an overlay network
docker network create --driver overlay my-network

# Create encrypted overlay network
docker network create --driver overlay --opt encrypted my-secure-network

# Create with custom subnet
docker network create --driver overlay --subnet 10.0.9.0/24 my-network

Connecting Services to Networks

# Create a service connected to a network
docker service create --name backend --network my-network backend-image

# Connect an existing service to a network
docker service update --network-add my-network frontend

# Disconnect a service from a network
docker service update --network-rm my-network frontend

The Routing Mesh

Docker Swarm includes a routing mesh for distributing incoming traffic:

# Publish a port using the routing mesh
docker service create --name web --replicas 3 --publish 8080:80 nginx

Deploying Stacks with Docker Compose

Docker Compose for Swarm

Docker Compose files can be used to deploy multi-service applications to Swarm:

Compose File for Swarm

# docker-compose.yml
version: '3.8'

services:
  webapp:
    image: nginx:latest
    ports:
      - "80:80"
    deploy:
      replicas: 3
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
    networks:
      - webnet

  visualizer:
    image: dockersamples/visualizer
    ports:
      - "8080:8080"
    deploy:
      placement:
        constraints:
          - node.role == manager
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - webnet

networks:
  webnet:
    driver: overlay

Stack Deployment

# Deploy a stack
docker stack deploy -c docker-compose.yml myapp

# List all stacks
docker stack ls

# List services in a stack
docker stack services myapp

# List tasks in a stack
docker stack ps myapp

# Remove a stack
docker stack rm myapp

Swarm-specific Compose Options

Docker Compose for Swarm includes additional options:

# Example of deploy options
deploy:
  mode: replicated
  replicas: 3
  placement:
    constraints:
      - node.role == worker
      - node.labels.region == east
  resources:
    limits:
      cpus: '0.5'
      memory: 512M
    reservations:
      cpus: '0.1'
      memory: 128M
  update_config:
    parallelism: 2
    delay: 10s
    failure_action: rollback
  restart_policy:
    condition: on-failure
    delay: 5s
    max_attempts: 3
    window: 120s

Managing Secrets and Configs

Working with Secrets

Secrets provide secure storage for sensitive data:

# Create a secret from a file
docker secret create my_secret_file ./secret.txt

# Create a secret from standard input
echo "my secret data" | docker secret create my_secret_data -

# List secrets
docker secret ls

# Inspect a secret
docker secret inspect my_secret_file

# Remove a secret
docker secret rm my_secret_file

Using Secrets in Services

# Create a service with a secret
docker service create \
  --name db \
  --secret db_password \
  --secret db_root_password \
  -e MYSQL_PASSWORD_FILE=/run/secrets/db_password \
  -e MYSQL_ROOT_PASSWORD_FILE=/run/secrets/db_root_password \
  mysql:5.7

# Update a service to add a secret
docker service update --secret-add my_new_secret my_service

Working with Configs

Configs store non-sensitive configuration data:

# Create a config from a file
docker config create nginx_conf nginx.conf

# List configs
docker config ls

# Inspect a config
docker config inspect nginx_conf

# Remove a config
docker config rm nginx_conf

Using Configs in Services

# Create a service with a config
docker service create \
  --name webserver \
  --config source=nginx_conf,target=/etc/nginx/nginx.conf \
  nginx

# Update a service to add a config
docker service update --config-add source=app_conf,target=/app/config.json my_service

Secrets and Configs in Compose Files

# docker-compose.yml with secrets and configs
version: '3.8'

services:
  db:
    image: mysql:5.7
    environment:
      MYSQL_PASSWORD_FILE: /run/secrets/db_password
      MYSQL_ROOT_PASSWORD_FILE: /run/secrets/db_root_password
    secrets:
      - db_password
      - db_root_password

  web:
    image: nginx
    configs:
      - source: nginx_conf
        target: /etc/nginx/nginx.conf

secrets:
  db_password:
    file: ./db_password.txt
  db_root_password:
    file: ./db_root_password.txt

configs:
  nginx_conf:
    file: ./nginx.conf

Scaling and Load Balancing

Scaling Services

Scale services to handle varying loads:

# Scale a service to 5 replicas
docker service scale myservice=5

# Scale multiple services at once
docker service scale web=5 api=3 cache=2

# Scale using update command
docker service update --replicas=10 myservice

Load Balancing

Swarm provides several load balancing mechanisms:

# Using routing mesh with published ports
docker service create \
  --name web \
  --replicas 5 \
  --publish published=8080,target=80 \
  nginx

# Using host mode (bypasses routing mesh)
docker service create \
  --name web \
  --replicas 5 \
  --publish mode=host,published=8080,target=80 \
  nginx

Placement Constraints

Control where services run in the swarm:

# Place service only on manager nodes
docker service create \
  --name mgr-only \
  --constraint "node.role == manager" \
  nginx

# Place service on specific node
docker service create \
  --name specific-node \
  --constraint "node.hostname == worker-03" \
  nginx

# Place service based on node labels
docker service create \
  --name use-labels \
  --constraint "node.labels.zone == east" \
  nginx

Resource Constraints

Limit and reserve resources for services:

# Set resource limits and reservations
docker service create \
  --name resource-limited \
  --limit-cpu 0.5 \
  --limit-memory 512M \
  --reserve-cpu 0.1 \
  --reserve-memory 128M \
  nginx

Monitoring and Troubleshooting

Monitoring Commands

Docker provides several commands to monitor the swarm:

# Check swarm status
docker info

# List nodes and their status
docker node ls

# List services and their status
docker service ls

# View tasks for a service
docker service ps my-service

# View logs for a service
docker service logs my-service

Node and Container Metrics

# View container resource usage
docker stats

# Check system-wide information
docker system df

Common Issues and Solutions

Issue Possible Causes Troubleshooting Steps
Service creation fails
  • Image not found
  • Network issues
  • Resource constraints
  • Check image availability with docker pull
  • Verify network connectivity
  • Check available resources with docker node inspect
Tasks stuck in "Pending" state
  • Placement constraints not met
  • Insufficient resources
  • Node availability issues
  • Check constraints with docker service inspect
  • Verify node resources
  • Check node status with docker node ls
Node fails to join swarm
  • Network connectivity issues
  • Firewall blocking required ports
  • Invalid join token
  • Check network connectivity between nodes
  • Verify ports 2377, 7946, and 4789 are open
  • Generate a new join token
Manager node not available
  • Network partition
  • Node failure
  • Loss of quorum
  • Check network connectivity
  • Use docker swarm init --force-new-cluster to recover
  • Add more manager nodes for redundancy

External Monitoring Tools

Additional tools for monitoring Docker Swarm:

Real-world Docker Swarm Use Cases

Web Application Deployment

Sample stack for a typical web application:

# docker-compose.yml for a web application
version: '3.8'

services:
  web:
    image: nginx:latest
    ports:
      - "80:80"
    deploy:
      replicas: 3
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
    volumes:
      - web_content:/usr/share/nginx/html
    networks:
      - frontend
      - backend

  api:
    image: myapp/api:latest
    deploy:
      replicas: 2
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
    environment:
      - DB_HOST=database
      - DB_NAME=myapp
      - DB_USER=user
    secrets:
      - db_password
    networks:
      - backend

  database:
    image: postgres:13
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.role == db
      restart_policy:
        condition: on-failure
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD_FILE=/run/secrets/db_password
    volumes:
      - db_data:/var/lib/postgresql/data
    networks:
      - backend

  cache:
    image: redis:alpine
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    networks:
      - backend

networks:
  frontend:
  backend:
    driver: overlay
    attachable: true

volumes:
  web_content:
  db_data:

secrets:
  db_password:
    file: ./db_password.txt

CI/CD Environments

Swarm is ideal for CI/CD environments:

Small to Medium Production Workloads

Swarm excels at simpler production deployments:

Edge Computing and IoT

Docker Swarm's lightweight nature makes it suitable for:

Docker Swarm Best Practices

Planning and Architecture

Security

Performance

High Availability

Operations

Hands-on Exercises

Exercise 1: Setting Up a Docker Swarm Cluster

Create a basic Swarm cluster with manager and worker nodes:

  1. Set up three virtual machines (using Vagrant, VirtualBox, or cloud providers)
  2. Install Docker on each machine
  3. Initialize a Swarm on the first node
  4. Join the second node as a manager
  5. Join the third node as a worker
  6. Verify the Swarm status and node roles
  7. Experiment with node management commands

Exercise 2: Deploying a Multi-service Application

Deploy a web application with multiple services:

  1. Create a Docker Compose file for a web app with:
    • Frontend service (Nginx or similar)
    • Backend API (Node.js, Python, or similar)
    • Database (MongoDB, PostgreSQL, or similar)
  2. Configure appropriate networks, volumes, and replicas
  3. Deploy the stack to the Swarm
  4. Verify all services are running correctly
  5. Test service discovery between containers
  6. Test external access to the application

Exercise 3: Implementing High Availability and Scaling

Practice Swarm's HA and scaling capabilities:

  1. Deploy a service with multiple replicas
  2. Test the service's availability
  3. Simulate a node failure (stop Docker or shut down a VM)
  4. Observe how Swarm reschedules containers
  5. Scale services up and down
  6. Implement and test a rolling update strategy
  7. Practice rollback of a failed update

Summary and Next Steps

Docker Swarm provides a straightforward, integrated solution for container orchestration that's especially valuable for teams already familiar with Docker. While it may not have the extensive feature set of Kubernetes, its simplicity and tight Docker integration make it an excellent choice for many use cases.

Key Takeaways

Learning Path

To continue your Docker Swarm journey:

  1. Practice: Set up a Swarm cluster and deploy various applications
  2. Explore tools: Try monitoring and visualization tools like Portainer or Swarm Visualizer
  3. Container management: Learn about container management patterns
  4. Production deployment: Explore production deployment strategies with Swarm

In the next lecture, we'll explore container management patterns that apply to both Docker Swarm and other orchestration platforms.

Additional Resources