What is Docker Swarm?
Docker Swarm is Docker's native clustering and orchestration solution that turns a pool of Docker hosts into a single, virtual Docker host. It allows you to deploy and manage a cluster of Docker nodes to run containerized applications at scale with built-in orchestration capabilities.
The Beehive Analogy
Docker Swarm can be compared to a beehive:
- Individual bees are like Docker containers—each performs specific tasks
- Worker bees are like worker nodes in the swarm—they execute the actual tasks
- Queen bee is like the manager node—it coordinates and directs the workers
- Honeycomb structure represents the overlay network—it provides organization and communication paths
- Bee dance communication is like service discovery—sharing information about where resources are located
Just as bees work together in a highly organized manner to achieve complex goals, Docker Swarm coordinates containers across multiple hosts to build resilient, scalable applications.
Docker Swarm Architecture
Key Features of Docker Swarm
Cluster Management
- Declarative state: Define the desired state of your services and Swarm makes it happen
- Self-healing: Maintains the desired number of tasks despite node failures
- Manager nodes: Handle cluster management tasks using Raft consensus algorithm
- Worker nodes: Execute the containers assigned by managers
- High availability: Multiple manager nodes provide fault tolerance
Simplicity and Integration
- Seamless Docker integration: Built into Docker Engine (no additional installation needed)
- Familiar Docker CLI: Use standard Docker commands with Swarm-specific options
- Docker Compose compatibility: Deploy multi-container applications with Compose files
- Easier learning curve: Simpler than Kubernetes for small to medium deployments
Networking Features
- Overlay networks: Secure container-to-container communication across nodes
- Service discovery: Automatic DNS-based discovery for services
- Load balancing: Built-in load balancing for service requests
- Ingress networking: Routing mesh for exposing services to external clients
Security Features
- Automatic TLS: Encrypts communications between nodes
- Certificate rotation: Automatically rotates manager certificates
- Mutual authentication: Nodes authenticate with each other using TLS
- Secret management: Securely distribute sensitive data to services
Rolling Updates and Scaling
- Scaling: Easily scale services up or down with a single command
- Rolling updates: Update services with zero downtime
- Health checks: Monitor service health and replace failing tasks
- Rollback capability: Quickly revert to previous service versions
Docker Swarm vs. Kubernetes
Both Docker Swarm and Kubernetes are container orchestration platforms, but they have different focuses and strengths:
| Feature | Docker Swarm | Kubernetes |
|---|---|---|
| Setup and Installation | Very simple, built into Docker | More complex, requires additional setup |
| Learning Curve | Gentle, familiar Docker commands | Steeper, many new concepts to learn |
| Scalability | Good for small to medium deployments | Excellent for large-scale deployments |
| Feature Set | Focused, core orchestration features | Comprehensive, extensive capabilities |
| Community & Ecosystem | Smaller community, fewer integrations | Massive community, vast ecosystem |
| Auto-healing | Basic auto-healing capabilities | Advanced self-healing across various levels |
| Load Balancing | Native load balancing with routing mesh | Various load balancing options (Services, Ingress) |
| Rolling Updates | Supported with basic controls | Extensive update and rollback strategies |
| Monitoring & Logging | Basic, requires additional tools | Rich ecosystem of monitoring solutions |
| Industry Adoption | Less widely adopted | Industry standard, widespread adoption |
When to Choose Docker Swarm
- You already have Docker expertise and want a simple solution
- You have small to medium-sized applications
- You need a quick setup for development or testing
- Your team has limited operational resources
- You prefer simplicity over extensive features
When to Choose Kubernetes
- You need to scale to very large deployments
- You require advanced features and extensive customization
- You want to leverage a vast ecosystem of tools and extensions
- You have complex microservices architectures
- You have the operational resources to manage complexity
Docker Swarm Architecture
Node Types
A Docker Swarm cluster consists of two types of nodes:
- Manager nodes: Control the cluster state and orchestrate tasks
- Worker nodes: Execute the tasks (containers) assigned by managers
A node can be both a manager and a worker simultaneously.
Manager Nodes
Manager nodes handle the control plane of the Swarm:
- Cluster management: Maintain desired state and reconcile actual state
- Orchestration: Schedule tasks onto worker nodes
- Swarm API: Process commands from Docker CLI and API
- Raft consensus: Use Raft consensus algorithm for distributed state
- High availability: Recommended to have 3, 5, or 7 manager nodes for redundancy
Worker Nodes
Worker nodes are responsible for running containers:
- Task execution: Run the containers assigned by manager nodes
- Resource provision: Provide CPU, memory, and storage resources
- Status reporting: Report task status back to managers
Raft Consensus
Swarm uses the Raft consensus algorithm for manager node coordination:
- Leader election: One manager is elected as the leader
- Distributed state: Cluster state is replicated across all managers
- Fault tolerance: Can tolerate (N-1)/2 node failures (where N is the number of managers)
- Quorum: Requires a majority of managers to be available for operations
Recommended manager node configurations:
- 3 managers: Can tolerate 1 node failure
- 5 managers: Can tolerate 2 node failures
- 7 managers: Can tolerate 3 node failures
Key Docker Swarm Concepts
Services
Services are the primary building blocks in Docker Swarm:
- Definition: A service defines a container image, number of replicas, ports, networks, etc.
- Replicated services: Multiple identical tasks distributed across the cluster
- Global services: One task per node (useful for monitoring agents, etc.)
- Service modes: Replicated (default) or global
# Create a replicated service with 3 replicas
docker service create --name webapp --replicas 3 -p 80:80 nginx
# Create a global service (one per node)
docker service create --name monitoring --mode global monitoring-agent:latest
Tasks
Tasks are the individual instances of a service:
- Definition: A task is a running container that is part of a service
- Scheduling: The scheduler assigns tasks to nodes based on various constraints
- Lifecycle: Tasks have states like new, pending, running, complete, etc.
- Failure handling: If a task fails, Swarm creates a new task to maintain the desired state
Stacks
Stacks are groups of related services that share dependencies:
- Definition: A multi-service application defined in a Compose file
- Deployment: Deployed and managed as a single entity
- Organization: Services in a stack share a unique namespace
# Deploy a stack from a compose file
docker stack deploy -c docker-compose.yml myapp
Networking
Swarm provides several network types:
- Overlay networks: Multi-host networks for container-to-container communication
- Ingress network: A special overlay network for routing external traffic to services
- Docker_gwbridge: A bridge network connecting overlay networks to host network
# Create an overlay network
docker network create --driver overlay mynetwork
# Create a service connected to the network
docker service create --name myservice --network mynetwork myimage
Volumes
Docker Swarm supports data persistence through volumes:
- Local volumes: Standard Docker volumes on a single node
- Volume drivers: Integration with external storage systems
# Create a service with a volume
docker service create --name db --mount type=volume,source=dbdata,target=/var/lib/mysql mysql
Secrets
Secrets allow secure storage of sensitive data:
- Definition: Sensitive data like passwords, certificates, or API keys
- Storage: Securely stored in the Swarm's internal database (encrypted at rest)
- Access: Only available to services that are granted access
- Implementation: Mounted as in-memory filesystem at /run/secrets/
# Create a secret
echo "mypassword" | docker secret create db_password -
# Use the secret in a service
docker service create --name db --secret db_password mysql
Configs
Configs are similar to secrets but for non-sensitive configuration:
- Definition: Configuration files, scripts, or other non-sensitive data
- Storage: Stored in the Swarm's internal database
- Access: Only available to services that are granted access
- Implementation: Mounted as files in the container's filesystem
# Create a config from a file
docker config create nginx_conf nginx.conf
# Use the config in a service
docker service create --name web --config source=nginx_conf,target=/etc/nginx/nginx.conf nginx
Setting Up a Docker Swarm Cluster
Initialize a Swarm Cluster
Creating a new Swarm is straightforward:
# Initialize a new swarm on the first node (becomes a manager)
docker swarm init --advertise-addr 192.168.1.10
# This command outputs a token for adding worker nodes
# Example output:
# Swarm initialized: current node (dxn1zf6l61qsb1josjja83ngz) is now a manager.
#
# To add a worker to this swarm, run the following command:
# docker swarm join --token SWMTKN-1-49nj1cmql... 192.168.1.10:2377
Joining Worker Nodes
Add additional nodes to the swarm as workers:
# On worker nodes, run the join command with the token
docker swarm join --token SWMTKN-1-49nj1cmql... 192.168.1.10:2377
# Verify node joined
# Example output:
# This node joined a swarm as a worker.
Adding Manager Nodes
For high availability, add additional manager nodes:
# Get the manager join token on an existing manager
docker swarm join-token manager
# Output will include the join command and token for managers
# Join a new manager node
docker swarm join --token SWMTKN-1-61ztec5kyafptydic... 192.168.1.10:2377
Inspecting the Swarm
View information about the swarm and its nodes:
# List all nodes in the swarm
docker node ls
# Inspect a specific node
docker node inspect node-id
# View swarm information
docker info
Node Management
Manage nodes in the swarm:
# Promote a worker to manager
docker node promote worker-node-id
# Demote a manager to worker
docker node demote manager-node-id
# Drain a node for maintenance (move tasks away)
docker node update --availability drain node-id
# Make a node active again
docker node update --availability active node-id
# Remove a node from the swarm (run on the node to remove)
docker swarm leave
# Force remove a node (run on a manager)
docker node rm node-id
Managing Services in Docker Swarm
Creating Services
Deploy services to the swarm:
# Create a simple service
docker service create --name webserver --replicas 3 -p 80:80 nginx
# Create a service with constraints
docker service create --name db \
--constraint 'node.labels.type == database' \
--replicas 1 \
-e MYSQL_ROOT_PASSWORD=password \
mysql:5.7
# Create a service with resource limits
docker service create --name cpu-limited \
--limit-cpu 0.5 \
--limit-memory 512M \
--reserve-cpu 0.1 \
--reserve-memory 128M \
--replicas 3 \
nginx
Inspecting Services
View information about services:
# List all services
docker service ls
# Inspect a specific service
docker service inspect service-name
# View service logs
docker service logs service-name
# View tasks (containers) for a service
docker service ps service-name
Scaling Services
Adjust the number of replicas:
# Scale a service up
docker service scale webserver=5
# Scale multiple services at once
docker service scale webserver=3 api=2 db=1
Updating Services
Apply rolling updates to services:
# Update the image used by a service
docker service update --image nginx:1.19 webserver
# Update with specific update configurations
docker service update \
--update-parallelism 2 \
--update-delay 10s \
--update-failure-action rollback \
--image nginx:1.19 \
webserver
Rolling Back Updates
Revert to previous service configurations:
# Rollback a service to its previous configuration
docker service update --rollback webserver
Removing Services
Clean up services when no longer needed:
# Remove a service
docker service rm webserver
# Remove multiple services
docker service rm webserver api db
Docker Swarm Networking
Network Types
Docker Swarm works with several network types:
- Overlay networks: Multi-host networks for container communication
- Ingress network: Special overlay network for routing mesh
- Bridge networks: Single-host networks (not directly swarm-aware)
- Macvlan and IPvlan: For direct connection to physical network
Creating Overlay Networks
# Create an overlay network
docker network create --driver overlay my-network
# Create encrypted overlay network
docker network create --driver overlay --opt encrypted my-secure-network
# Create with custom subnet
docker network create --driver overlay --subnet 10.0.9.0/24 my-network
Connecting Services to Networks
# Create a service connected to a network
docker service create --name backend --network my-network backend-image
# Connect an existing service to a network
docker service update --network-add my-network frontend
# Disconnect a service from a network
docker service update --network-rm my-network frontend
The Routing Mesh
Docker Swarm includes a routing mesh for distributing incoming traffic:
- Ingress routing: Allows any node to accept connections for any service
- Internal load balancing: Distributes requests across service tasks
- Port publishing: Makes services available on published ports
# Publish a port using the routing mesh
docker service create --name web --replicas 3 --publish 8080:80 nginx
Deploying Stacks with Docker Compose
Docker Compose for Swarm
Docker Compose files can be used to deploy multi-service applications to Swarm:
- Compose file version 3+ is required for Swarm mode
- Some Compose features are not supported in Swarm mode
- Additional Swarm-specific features are available
Compose File for Swarm
# docker-compose.yml
version: '3.8'
services:
webapp:
image: nginx:latest
ports:
- "80:80"
deploy:
replicas: 3
update_config:
parallelism: 1
delay: 10s
restart_policy:
condition: on-failure
networks:
- webnet
visualizer:
image: dockersamples/visualizer
ports:
- "8080:8080"
deploy:
placement:
constraints:
- node.role == manager
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- webnet
networks:
webnet:
driver: overlay
Stack Deployment
# Deploy a stack
docker stack deploy -c docker-compose.yml myapp
# List all stacks
docker stack ls
# List services in a stack
docker stack services myapp
# List tasks in a stack
docker stack ps myapp
# Remove a stack
docker stack rm myapp
Swarm-specific Compose Options
Docker Compose for Swarm includes additional options:
- deploy: Swarm-specific deployment configuration
- placement constraints: Control where services run
- resources: Specify resource limitations
- update_config: Configure rolling updates
- restart_policy: Define restart behavior
# Example of deploy options
deploy:
mode: replicated
replicas: 3
placement:
constraints:
- node.role == worker
- node.labels.region == east
resources:
limits:
cpus: '0.5'
memory: 512M
reservations:
cpus: '0.1'
memory: 128M
update_config:
parallelism: 2
delay: 10s
failure_action: rollback
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
Managing Secrets and Configs
Working with Secrets
Secrets provide secure storage for sensitive data:
# Create a secret from a file
docker secret create my_secret_file ./secret.txt
# Create a secret from standard input
echo "my secret data" | docker secret create my_secret_data -
# List secrets
docker secret ls
# Inspect a secret
docker secret inspect my_secret_file
# Remove a secret
docker secret rm my_secret_file
Using Secrets in Services
# Create a service with a secret
docker service create \
--name db \
--secret db_password \
--secret db_root_password \
-e MYSQL_PASSWORD_FILE=/run/secrets/db_password \
-e MYSQL_ROOT_PASSWORD_FILE=/run/secrets/db_root_password \
mysql:5.7
# Update a service to add a secret
docker service update --secret-add my_new_secret my_service
Working with Configs
Configs store non-sensitive configuration data:
# Create a config from a file
docker config create nginx_conf nginx.conf
# List configs
docker config ls
# Inspect a config
docker config inspect nginx_conf
# Remove a config
docker config rm nginx_conf
Using Configs in Services
# Create a service with a config
docker service create \
--name webserver \
--config source=nginx_conf,target=/etc/nginx/nginx.conf \
nginx
# Update a service to add a config
docker service update --config-add source=app_conf,target=/app/config.json my_service
Secrets and Configs in Compose Files
# docker-compose.yml with secrets and configs
version: '3.8'
services:
db:
image: mysql:5.7
environment:
MYSQL_PASSWORD_FILE: /run/secrets/db_password
MYSQL_ROOT_PASSWORD_FILE: /run/secrets/db_root_password
secrets:
- db_password
- db_root_password
web:
image: nginx
configs:
- source: nginx_conf
target: /etc/nginx/nginx.conf
secrets:
db_password:
file: ./db_password.txt
db_root_password:
file: ./db_root_password.txt
configs:
nginx_conf:
file: ./nginx.conf
Scaling and Load Balancing
Scaling Services
Scale services to handle varying loads:
# Scale a service to 5 replicas
docker service scale myservice=5
# Scale multiple services at once
docker service scale web=5 api=3 cache=2
# Scale using update command
docker service update --replicas=10 myservice
Load Balancing
Swarm provides several load balancing mechanisms:
- Routing mesh: Distributes incoming traffic across all nodes
- Service-level load balancing: Distributes traffic across service replicas
- Round-robin (default): Rotates requests across available tasks
- External load balancers: Can be integrated for more advanced scenarios
# Using routing mesh with published ports
docker service create \
--name web \
--replicas 5 \
--publish published=8080,target=80 \
nginx
# Using host mode (bypasses routing mesh)
docker service create \
--name web \
--replicas 5 \
--publish mode=host,published=8080,target=80 \
nginx
Placement Constraints
Control where services run in the swarm:
# Place service only on manager nodes
docker service create \
--name mgr-only \
--constraint "node.role == manager" \
nginx
# Place service on specific node
docker service create \
--name specific-node \
--constraint "node.hostname == worker-03" \
nginx
# Place service based on node labels
docker service create \
--name use-labels \
--constraint "node.labels.zone == east" \
nginx
Resource Constraints
Limit and reserve resources for services:
# Set resource limits and reservations
docker service create \
--name resource-limited \
--limit-cpu 0.5 \
--limit-memory 512M \
--reserve-cpu 0.1 \
--reserve-memory 128M \
nginx
Monitoring and Troubleshooting
Monitoring Commands
Docker provides several commands to monitor the swarm:
# Check swarm status
docker info
# List nodes and their status
docker node ls
# List services and their status
docker service ls
# View tasks for a service
docker service ps my-service
# View logs for a service
docker service logs my-service
Node and Container Metrics
# View container resource usage
docker stats
# Check system-wide information
docker system df
Common Issues and Solutions
| Issue | Possible Causes | Troubleshooting Steps |
|---|---|---|
| Service creation fails |
|
|
| Tasks stuck in "Pending" state |
|
|
| Node fails to join swarm |
|
|
| Manager node not available |
|
|
External Monitoring Tools
Additional tools for monitoring Docker Swarm:
- Prometheus: Metrics collection and alerting
- Grafana: Visualization dashboard for metrics
- cAdvisor: Container resource usage and performance analysis
- Docker Swarm Visualizer: Visual representation of the swarm
- ELK Stack: Logging solution (Elasticsearch, Logstash, Kibana)
Real-world Docker Swarm Use Cases
Web Application Deployment
Sample stack for a typical web application:
# docker-compose.yml for a web application
version: '3.8'
services:
web:
image: nginx:latest
ports:
- "80:80"
deploy:
replicas: 3
update_config:
parallelism: 1
delay: 10s
restart_policy:
condition: on-failure
volumes:
- web_content:/usr/share/nginx/html
networks:
- frontend
- backend
api:
image: myapp/api:latest
deploy:
replicas: 2
update_config:
parallelism: 1
delay: 10s
restart_policy:
condition: on-failure
environment:
- DB_HOST=database
- DB_NAME=myapp
- DB_USER=user
secrets:
- db_password
networks:
- backend
database:
image: postgres:13
deploy:
replicas: 1
placement:
constraints:
- node.labels.role == db
restart_policy:
condition: on-failure
environment:
- POSTGRES_DB=myapp
- POSTGRES_USER=user
- POSTGRES_PASSWORD_FILE=/run/secrets/db_password
volumes:
- db_data:/var/lib/postgresql/data
networks:
- backend
cache:
image: redis:alpine
deploy:
replicas: 1
restart_policy:
condition: on-failure
networks:
- backend
networks:
frontend:
backend:
driver: overlay
attachable: true
volumes:
web_content:
db_data:
secrets:
db_password:
file: ./db_password.txt
CI/CD Environments
Swarm is ideal for CI/CD environments:
- Deploy test environments on demand
- Run automated tests in containers
- Facilitate blue-green deployments
- Implement canary releases
Small to Medium Production Workloads
Swarm excels at simpler production deployments:
- E-commerce platforms
- Content management systems
- Small business applications
- Internal enterprise tools
Edge Computing and IoT
Docker Swarm's lightweight nature makes it suitable for:
- Edge computing deployments
- IoT gateway orchestration
- Remote location deployments
- Environments with limited resources
Docker Swarm Best Practices
Planning and Architecture
- Manager nodes: Use 3, 5, or 7 manager nodes for optimal fault tolerance
- Manager placement: Distribute manager nodes across availability zones
- Node sizing: Choose appropriate node sizes based on workload demands
- Network design: Plan overlay networks based on security and isolation needs
Security
- TLS: Use TLS for securing node communications (enabled by default)
- Secrets: Store sensitive information in Swarm secrets, not environment variables
- Regular updates: Keep Docker Engine updated with security patches
- Network segmentation: Use overlay networks to isolate services
Performance
- Layer optimization: Use multistage builds for smaller images
- Resource limits: Set appropriate CPU and memory limits for services
- Monitoring: Implement proper monitoring for early problem detection
- Storage optimization: Use volume drivers appropriate for your workload
High Availability
- Replicated services: Use replication for service availability
- Health checks: Implement container health checks for better resilience
- Update configurations: Use rolling updates with proper testing
- Quorum protection: Maintain manager node quorum for cluster operations
Operations
- Infrastructure as code: Use compose files for declaring infrastructure
- Rolling updates: Configure update parallelism and delay for smooth deployments
- Backup: Regularly backup Swarm state and volumes
- Documentation: Document your Swarm setup and service configurations
Hands-on Exercises
Exercise 1: Setting Up a Docker Swarm Cluster
Create a basic Swarm cluster with manager and worker nodes:
- Set up three virtual machines (using Vagrant, VirtualBox, or cloud providers)
- Install Docker on each machine
- Initialize a Swarm on the first node
- Join the second node as a manager
- Join the third node as a worker
- Verify the Swarm status and node roles
- Experiment with node management commands
Exercise 2: Deploying a Multi-service Application
Deploy a web application with multiple services:
- Create a Docker Compose file for a web app with:
- Frontend service (Nginx or similar)
- Backend API (Node.js, Python, or similar)
- Database (MongoDB, PostgreSQL, or similar)
- Configure appropriate networks, volumes, and replicas
- Deploy the stack to the Swarm
- Verify all services are running correctly
- Test service discovery between containers
- Test external access to the application
Exercise 3: Implementing High Availability and Scaling
Practice Swarm's HA and scaling capabilities:
- Deploy a service with multiple replicas
- Test the service's availability
- Simulate a node failure (stop Docker or shut down a VM)
- Observe how Swarm reschedules containers
- Scale services up and down
- Implement and test a rolling update strategy
- Practice rollback of a failed update
Summary and Next Steps
Docker Swarm provides a straightforward, integrated solution for container orchestration that's especially valuable for teams already familiar with Docker. While it may not have the extensive feature set of Kubernetes, its simplicity and tight Docker integration make it an excellent choice for many use cases.
Key Takeaways
- Docker Swarm is Docker's native clustering and orchestration solution
- It offers simplicity and integration with the standard Docker CLI
- The architecture consists of manager and worker nodes using Raft consensus
- Services, tasks, and stacks are the primary building blocks
- Swarm provides features like service discovery, load balancing, and secrets management
- Docker Compose can be used to deploy multi-service applications
- Swarm is ideal for small to medium deployments and Docker-centric teams
Learning Path
To continue your Docker Swarm journey:
- Practice: Set up a Swarm cluster and deploy various applications
- Explore tools: Try monitoring and visualization tools like Portainer or Swarm Visualizer
- Container management: Learn about container management patterns
- Production deployment: Explore production deployment strategies with Swarm
In the next lecture, we'll explore container management patterns that apply to both Docker Swarm and other orchestration platforms.