Docker Architecture

Introduction to Docker Architecture

In our previous lecture, we explored the fundamental concepts of containers and how they revolutionize application deployment. Now, we'll dive deeper into Docker's architecture - the components that work together to create, run, and manage containers efficiently.

Understanding Docker's architecture is like understanding how a car works under the hood. While you can drive a car without knowing all its internal mechanics, understanding these components helps when you need to troubleshoot problems, optimize performance, or customize your experience.

Docker's Client-Server Architecture

Docker uses a client-server architecture with several distinct components that work together.

Key Components

• Docker Client: The primary way users interact with Docker
• Docker Daemon (dockerd): The persistent process that manages Docker objects
• Docker REST API: The interface between the client and daemon
• Docker Objects: Images, containers, networks, volumes, plugins, etc.

Think of this architecture like a restaurant:

• You (the user) place an order (command) with the waiter (Docker client)
• The waiter communicates your order to the kitchen (Docker daemon) via a standardized order form (REST API)
• The kitchen prepares your meal (manages containers, images, etc.)
• Finally, the waiter serves your food (returns results to you)

Docker Client

The Docker client is the primary way users interact with Docker. It's a command-line tool that allows you to issue commands to the Docker daemon.

Docker Client Functions

• Accepts commands from users (docker build, run, pull, etc.)
• Translates these commands into API calls to the Docker daemon
• Displays results and output from the daemon back to the user
• Can connect to a local or remote Docker daemon

Common Docker Client Commands

# Build an image
docker build -t myapp:1.0 .

# Run a container
docker run -d -p 8080:80 myapp:1.0

# Pull an image from a registry
docker pull nginx:latest

# List running containers
docker ps

# View container logs
docker logs container_id

# Stop a container
docker stop container_id
            

Docker Client Locations

The Docker client can communicate with:

• A local Docker daemon (default)
• A remote Docker daemon (via TCP/IP)
• Multiple Docker daemons

# Connect to a remote Docker daemon
docker -H tcp://remote-docker-host:2375 info

# Use environment variable to set default daemon
export DOCKER_HOST=tcp://remote-docker-host:2375
            

This client flexibility is similar to how you might use a TV remote control (client) to control different TVs (daemons) by simply pointing it at the one you want to interact with.

Docker Daemon (dockerd)

The Docker daemon (dockerd) is the persistent background process that manages Docker objects such as images, containers, networks, and volumes.

Docker Daemon Responsibilities

• Listens for Docker API requests from the Docker client
• Manages Docker objects (images, containers, networks, volumes)
• Builds and stores images
• Creates and manages containers
• Sets up networking for containers
• Manages volume mounts
• Communicates with other daemons for distributed features

The Docker daemon is like the engine and control systems of a car. You don't directly interact with it, but it's responsible for all the actual work - from processing fuel (images) to turning the wheels (running containers).

The Daemon Runtime Architecture

Docker's architecture has evolved over time. Initially, the Docker daemon directly managed container execution, but now it delegates this to specialized containerd runtime.

containerd and runc

• containerd: A core container runtime that manages the container lifecycle
• runc: A lightweight, portable container runtime that implements the OCI specification

This modular architecture is like how a modern restaurant might have specialized stations: The head chef (Docker daemon) coordinates everything, but delegates specific tasks to the sous chef (containerd), who in turn works with line cooks (runc) to prepare specific dishes (run containers).

Docker REST API

The Docker REST API is the interface used for communication between the Docker client and the Docker daemon. It provides a way to control the daemon programmatically.

API Characteristics

• RESTful API using HTTP
• Supports JSON data format
• Versioned API for backward compatibility
• Can be accessed directly or via the Docker client
• Enables third-party tools to interact with Docker

Sample API Endpoints

# List containers
GET /containers/json

# Create a container
POST /containers/create

# Start a container
POST /containers/{id}/start

# List images
GET /images/json

# Create an image (build)
POST /build
            

The REST API is like a common language that allows different tools (Docker CLI, Docker Desktop, Portainer, etc.) to communicate with the Docker daemon using the same vocabulary and grammar.

Using the Docker API Directly

While most users interact with Docker through the client, you can use the API directly for advanced use cases:

# Using curl to list containers via the API
curl --unix-socket /var/run/docker.sock http:/v1.41/containers/json

# Using curl to create a container
curl -X POST --unix-socket /var/run/docker.sock \
  -H "Content-Type: application/json" \
  -d '{"Image":"nginx"}' \
  http:/v1.41/containers/create
            

This direct API access is particularly useful for:

• Creating custom automation tools
• Integrating Docker with other systems
• Building alternative user interfaces for Docker
• Debugging Docker issues

Docker Objects

Docker objects are the entities managed by Docker to run your applications. Understanding these objects is essential for effective Docker usage.

Images

A Docker image is a read-only template with instructions for creating a container. Images:

• Are built from a Dockerfile
• Are composed of multiple layers
• Are stored locally or in registries
• Have a unique tag (e.g., nginx:latest, node:14-alpine)
• Share common layers between images to save space

Images are like blueprints for houses. They contain all the specifications and materials needed, but you don't live in the blueprint - you live in the house built from it.

Containers

A container is a runnable instance of an image. Containers:

• Are created from images
• Have their own filesystem, network, and process space
• Are isolated from other containers and the host system
• Can be started, stopped, moved, and deleted
• Can be connected to one or more networks
• Can have storage attached to them

To extend our blueprint analogy: if an image is a blueprint, a container is the actual house built from that blueprint, with people living inside and modifications made by the residents.

Networks

Docker networks allow containers to communicate with each other and with the outside world.

• Bridge: The default network driver, isolating containers while allowing them to communicate
• Host: Removes network isolation between container and host
• None: Completely isolates a container from the network
• Overlay: Connects multiple Docker daemons across hosts (used with Docker Swarm)
• Macvlan: Assigns MAC addresses to containers, making them appear as physical devices on the network

Docker networks are like the plumbing and electrical systems in a house development. They allow communication between houses (containers) and with the outside world, with different types of connections for different needs.

Volumes

Docker volumes provide persistent storage for container data.

• Persist data beyond the container lifecycle
• Can be shared between containers
• Are more efficiently managed by Docker than bind mounts
• Support different storage drivers (local, NFS, cloud storage, etc.)
• Can be backed up and restored independently of containers

Volumes are like storage units that can be attached to houses (containers). When you move to a new house, your stored belongings remain intact in the storage unit and can be attached to your new home.

Docker Registry

A Docker registry is a repository for Docker images. It allows you to store and distribute images for use by Docker installations.

Types of Registries

• Docker Hub: Docker's public registry (similar to GitHub for code)
• Private Registries: Self-hosted or cloud-based private repositories
• Cloud Provider Registries: Amazon ECR, Google Container Registry, Azure Container Registry

Working with Registries

# Pull an image from Docker Hub
docker pull nginx:latest

# Tag a local image for a specific registry
docker tag myapp:1.0 username/myapp:1.0

# Push an image to Docker Hub
docker push username/myapp:1.0

# Pull from a private registry
docker pull registry.example.com/myapp:1.0
            

A Docker registry is like a library where blueprints (images) are stored, cataloged, and shared. Anyone with access to the library can check out a copy of a blueprint to build their own structure (container).

Registry Security

Registries typically include security features such as:

• Authentication to control who can push/pull images
• Image signing to verify image integrity
• Vulnerability scanning to detect security issues
• Access controls to manage permissions for different images

Docker Storage

Docker provides several options for storing data, each with different use cases and characteristics.

Storage Drivers

Storage drivers manage the container's writable layer and how images are stored. Common storage drivers include:

• overlay2: The preferred storage driver for most Linux distributions
• btrfs: Uses the Btrfs filesystem
• zfs: Uses the ZFS filesystem
• devicemapper: Uses Linux device mapper
• aufs: Older storage driver

Storage drivers control how container layers are stored and accessed on the host system, similar to how different filesystems (NTFS, ext4, etc.) manage files on your computer.

Storage Types

Docker offers different ways to persist data beyond the container lifecycle:

Volume Commands

# Create a named volume
docker volume create mydata

# Run a container with a volume
docker run -v mydata:/data nginx

# List volumes
docker volume ls

# Inspect a volume
docker volume inspect mydata

# Remove a volume
docker volume rm mydata

# Remove all unused volumes
docker volume prune
            

Docker Networking

Docker networking enables communication between containers and with the outside world. Understanding Docker's networking model is essential for building multi-container applications.

Network Drivers

Docker supports several network drivers to accommodate different scenarios:

Network Commands

# List networks
docker network ls

# Create a custom bridge network
docker network create mynetwork

# Run a container with a specific network
docker run --network=mynetwork nginx

# Connect a running container to a network
docker network connect mynetwork container_id

# Disconnect a container from a network
docker network disconnect mynetwork container_id

# Inspect a network
docker network inspect mynetwork

# Remove a network
docker network rm mynetwork
            

Evolution of Docker Architecture

Docker's architecture has evolved significantly since its inception, becoming more modular and standards-based.

Early Docker (Pre-1.11)

• Monolithic architecture
• Docker daemon handled all container operations
• Limited separation of concerns

Modern Docker (Post-1.11)

• Modular architecture
• Docker daemon delegates container execution to containerd
• containerd uses runc to create and run containers
• Compliance with OCI (Open Container Initiative) standards

Benefits of the Evolution

• Modularity: Components can be developed and upgraded independently
• Reusability: Lower-level components like containerd can be used by other projects
• Standards Compliance: OCI compatibility ensures interoperability
• Stability: More focused components with clearer responsibilities

This evolution is like how early cars had integrated systems built by a single manufacturer, but modern cars use standardized components (brakes, transmissions, etc.) from specialized suppliers that adhere to industry standards.

Docker Desktop Architecture

Docker Desktop is a popular tool for developers to run Docker on Windows and macOS. Its architecture differs from Docker on Linux due to the need for virtualization.

Windows Architecture

On Windows, Docker Desktop can use two different backends:

• WSL 2 Backend: Uses Windows Subsystem for Linux 2, providing better performance and compatibility
• Hyper-V Backend: Uses Windows' native hypervisor

macOS Architecture

On macOS, Docker Desktop uses a lightweight virtual machine:

• Uses a lightweight Linux VM based on LinuxKit
• Provides native-like experience with transparent virtualization
• Includes file system sharing between macOS and the VM
• Optimizes resource usage for developer machines

Docker Desktop's architecture is like running a simulation of a foreign environment (Linux) within your native system (Windows/macOS). This simulation is seamless enough that you can interact with it as if it were native, but it's actually running in a translation layer.

Practical Architecture Considerations

Docker Performance

Understanding Docker's architecture helps optimize performance:

• Choose appropriate storage drivers for your workload
• Use appropriate network drivers for your communication patterns
• Properly size container resources (CPU, memory)
• Consider the impact of filesystem operations, especially on Windows/macOS
• Optimize image size and layer caching

Security Implications

Docker's architecture has security implications to consider:

• The Docker daemon runs with root privileges by default
• Container isolation is strong but not as complete as VMs
• Access to the Docker socket (/var/run/docker.sock) is equivalent to root access
• Images may contain vulnerabilities or malicious code
• User namespace mapping can reduce privilege risks

Orchestration Considerations

When moving beyond single-host Docker:

• Container orchestration tools (Kubernetes, Swarm) add additional architectural layers
• Cross-host networking requires overlay networks or other solutions
• Distributed storage becomes important for persistent data
• Service discovery and load balancing become essential

Hands-On: Exploring Docker Architecture

Let's explore Docker's architecture through commands that reveal its components and behavior.

Examining the Docker Installation

# Check Docker version and architecture details
docker version

# View detailed Docker system information
docker info

# List the Docker client configuration
docker config ls

# Show Docker daemon configuration
cat /etc/docker/daemon.json  # Linux/macOS
notepad %PROGRAMDATA%\\docker\\config\\daemon.json  # Windows
            

Exploring Docker Components

# List running containers
docker ps

# View container details
docker inspect container_id

# Examine container stats
docker stats

# List networks
docker network ls

# View network details
docker network inspect bridge

# List volumes
docker volume ls

# Examine volume details
docker volume inspect volume_name
            

Image and Layer Inspection

# Pull an image to examine
docker pull nginx:latest

# View image details
docker image inspect nginx:latest

# See the image layers
docker history nginx:latest

# View storage usage
docker system df -v
            

Conclusion

Docker's client-server architecture provides a powerful, flexible system for containerizing applications. By understanding the relationships between the Docker client, Docker daemon, and Docker objects, you can better leverage Docker for your development and deployment needs.

Key takeaways from this lecture:

• Docker uses a client-server architecture with distinct components
• The Docker client communicates with the Docker daemon via a REST API
• The Docker daemon manages containers, images, networks, and volumes
• Docker's architecture has evolved to become more modular and standards-based
• Different operating systems require different approaches to run Linux containers
• Understanding Docker's architecture helps with debugging, optimization, and security

In our next lecture, we'll focus on Docker commands and how to use them effectively for managing containers, images, networks, and volumes.

Practice Activities

Activity 1: Explore Docker System Information

1. Run docker version and identify the client and server components
2. Run docker info and note the storage driver and network driver being used
3. Investigate the Docker daemon configuration on your system
4. Research how to change the default storage driver and what considerations apply

Activity 2: Investigate Container Isolation

1. Run two containers based on the same image:

   docker run -d --name container1 alpine sleep 1000
   docker run -d --name container2 alpine sleep 1000

2. Use docker exec to create a file in each container:

   docker exec container1 sh -c "echo 'hello from container1' > /file.txt"
   docker exec container2 sh -c "echo 'hello from container2' > /file.txt"

3. Verify that the files are different in each container
4. Investigate the container filesystem layers:

   docker inspect container1 | grep -A 10 GraphDriver
   docker inspect container2 | grep -A 10 GraphDriver

5. Stop and remove the containers when done

Activity 3: Explore Docker Networks

1. List the default Docker networks with docker network ls
2. Create a custom bridge network: docker network create mynetwork
3. Run containers on different networks and test connectivity:

   docker run -d --name default_container nginx
   docker run -d --name custom_container --network mynetwork nginx

4. Attempt to ping between containers in different networks
5. Connect a container to multiple networks:

   docker network connect bridge custom_container

6. Test connectivity again
7. Clean up the containers and networks when done

Activity 4: Examine Image Layers

1. Pull a large image: docker pull node:latest
2. Examine its layers: docker history node:latest
3. Create a simple Dockerfile that uses this image as a base
4. Build your custom image: docker build -t mylayer .
5. Compare the layers between the base image and your custom image
6. Modify your Dockerfile to optimize layer caching and rebuild
7. Compare build times and layer reuse

Challenge: Docker Architecture Diagram

Create a comprehensive diagram of Docker's architecture showing the relationships between:

• Docker client
• Docker daemon
• containerd
• runc
• Docker objects (images, containers, networks, volumes)
• Host system (kernel, filesystem)

Include annotations explaining each component's role and how they interact. Use a tool like draw.io, Lucidchart, or even paper and pencil. Present your diagram to the class and explain your understanding of Docker's architecture.

Additional Resources

Official Documentation

• Docker Architecture Overview
• Docker Storage Documentation
• Docker Networking Documentation
• Docker Engine API Documentation

Deep Dives

• Docker Execution Drivers and libcontainer
• containerd: The Container Runtime That Powers Docker
• Open Container Initiative (OCI)
• containerd GitHub Repository

Books

• "Docker Deep Dive" by Nigel Poulton (Chapter on Docker Architecture)
• "Docker in Action" by Jeff Nickoloff and Stephen Kuenzli (Chapter 2: Running Software in Containers)
• "Docker: Up & Running" by Sean P. Kane and Karl Matthias (Chapter 3: Docker Fundamentals)

Video Resources

• Docker Architecture Explained
• How Docker Works Behind the Scenes
• Understanding Docker Internals