Development Workflows with Docker

Docker: Beyond Production

While Docker is often discussed in the context of production deployments, it's equally transformative for development environments. Using Docker during development solves the age-old problem: "It works on my machine," by creating consistent, reproducible environments that match production as closely as possible.

The Kitchen Analogy

Think of development with Docker like cooking in a professional kitchen versus a home kitchen:

Traditional development is like each chef cooking at home with different equipment, ingredients, and setups before bringing dishes to a restaurant. When things don't work the same at the restaurant, it's frustrating and time-consuming to debug.
Docker development is like giving each chef a standardized kitchen setup with identical equipment and ingredients. Everything is packaged together, ensuring that dishes prepared in development kitchens will taste the same when cooked in the production kitchen.

With Docker, when your application "tastes good" in development, it will "taste good" in production too.

Traditional vs. Docker Development

Benefits of Docker in Development

Real-world Impact on Development

Eliminating "works on my machine" problems: No more environment-specific bugs that only appear in certain setups
Simplified onboarding: New developers can be productive in minutes instead of days
Multiple projects without conflicts: Work on many projects with different dependencies without interference
Matching production: Test in environments that mirror production, catching deployment issues early
Disposable environments: Break things and experiment freely without fear of damaging your system
Debugging production issues: Reproduce and debug production problems in identical local environments

Common Development Workflow Patterns

Pattern 1: The Build and Run Workflow

This simple workflow rebuilds the container for each code change:

# 1. Build Docker image
docker build -t my-app .

# 2. Run the container
docker run -p 3000:3000 my-app

# 3. When code changes, stop container and repeat

Best for: Simple applications, learning Docker, applications where builds are fast

Drawbacks: Slow feedback loop, inefficient for large applications

Pattern 2: The Volume Mount Workflow

This workflow mounts your source code as a volume, enabling live code reloading:

# 1. Build initial image with all dependencies
docker build -t my-app-dev .

# 2. Run with source code mounted as volume
docker run -p 3000:3000 -v $(pwd):/app my-app-dev

# 3. Code changes are immediately available in the container

Best for: Active development, interpreted languages (JavaScript, Python, Ruby, etc.)

Drawbacks: May have performance issues on certain platforms, requires filesystem watching

Pattern 3: Docker Compose Development Workflow

Using Docker Compose to manage multi-container development environments:

# docker-compose.yml
version: '3.8'

services:
  app:
    build: 
      context: .
      dockerfile: Dockerfile.dev
    ports:
      - "3000:3000"
    volumes:
      - ./:/app
      - /app/node_modules
    environment:
      - NODE_ENV=development

  db:
    image: postgres:14
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=password

volumes:
  postgres_data:

Best for: Real-world applications with multiple services and dependencies

Drawbacks: More complex setup, requires understanding Docker Compose

Pattern 4: Development Containers (Dev Containers)

Using your IDE (like VS Code) to connect directly to a development container:

Best for: Modern development teams, sophisticated IDEs, full development experience

Drawbacks: May require additional setup and configuration

Setting Up a JavaScript Development Environment

Project Structure

my-js-app/
├── src/               # Application source code
├── package.json       # Node.js dependencies
├── Dockerfile         # Production Dockerfile
├── Dockerfile.dev     # Development Dockerfile
├── docker-compose.yml # Development environment config
└── .dockerignore      # Files to exclude from builds

Development Dockerfile (Dockerfile.dev)

FROM node:18-alpine

WORKDIR /app

# Copy package files first for better caching
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy application code
COPY . .

# Expose development server port
EXPOSE 3000

# Command to run development server
CMD ["npm", "run", "dev"]

Docker Compose Configuration

version: '3.8'

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile.dev
    ports:
      - "3000:3000"
    volumes:
      - ./:/app
      - /app/node_modules
    environment:
      - NODE_ENV=development
      - PORT=3000
      # Add other environment variables as needed
    
  # Optional database service
  db:
    image: mongodb:latest
    volumes:
      - mongo_data:/data/db
    ports:
      - "27017:27017"

volumes:
  mongo_data:

Dockerignore File

# .dockerignore
node_modules
npm-debug.log
build
.git
.github
.vscode
*.md
.env
.env.local

Running the Development Environment

# Start the development environment
docker-compose up

# View logs
docker-compose logs -f app

# Execute commands in the container
docker-compose exec app npm install some-package

# Restart the container
docker-compose restart app

# Stop the environment
docker-compose down

Optimizing Volume Mounts for Development

Proper volume mount configuration is crucial for efficient development workflows with Docker.

Basic Source Code Mounting

volumes:
  - ./:/app

This mounts your entire project directory to the /app directory in the container.

Avoiding node_modules Mounting

volumes:
  - ./:/app
  - /app/node_modules

The second volume line creates an anonymous volume for node_modules, preventing the host's node_modules from overriding the container's.

Selective Path Mounting

volumes:
  - ./src:/app/src
  - ./public:/app/public
  - ./package.json:/app/package.json

Only mount the directories and files that need to change during development.

Platform-specific Considerations

Volume performance can vary across platforms:

Linux: Native performance, typically fast
macOS: Uses osxfs or gRPC FUSE, can be slower without optimizations
Windows: Performance varies based on filesystem type (WSL2 is recommended)

Docker Volume Performance Tools

docker-sync: Tool to improve volume performance on macOS and Windows
Mutagen: File synchronization for Docker, works well on macOS
WSL2: Use Windows Subsystem for Linux 2 for better performance on Windows

Managing Environment Variables in Development

Environment variables are key to configuring your application across different environments.

Using .env Files

# .env
API_URL=http://localhost:3001
DEBUG=true
NODE_ENV=development

Reference in docker-compose.yml:

services:
  app:
    # ... other configuration
    env_file:
      - .env

Direct Environment Variable Definition

services:
  app:
    # ... other configuration
    environment:
      - NODE_ENV=development
      - PORT=3000
      - DB_HOST=db
      - DB_NAME=myapp

Using Docker Compose Variable Substitution

# .env
APP_PORT=3000
DB_PORT=5432

# docker-compose.yml
services:
  app:
    # ... other configuration
    ports:
      - "${APP_PORT}:3000"
  
  db:
    # ... other configuration
    ports:
      - "${DB_PORT}:5432"

Development vs. Production Environments

Use different .env files for different environments:

# For development
docker-compose --env-file .env.development up

# For production
docker-compose --env-file .env.production up -d

Debugging Applications in Docker

Debugging containerized applications requires specific techniques but can be just as powerful as traditional debugging.

Accessing Container Logs

# View logs of a running container
docker logs -f container_id

# Through Docker Compose
docker-compose logs -f service_name

Executing Commands in Running Containers

# Open a shell in a running container
docker exec -it container_id sh

# Run a specific command
docker exec -it container_id npm run test

# With Docker Compose
docker-compose exec service_name sh

Remote Debugging Node.js Applications

Update your start command to enable inspection:

# package.json
{
  "scripts": {
    "dev": "node --inspect=0.0.0.0:9229 index.js"
  }
}

Configure in Docker Compose:

services:
  app:
    # ... other configuration
    ports:
      - "3000:3000"
      - "9229:9229"  # Expose debug port

Connect from Chrome DevTools or your IDE to localhost:9229

Node.js Debugging with VS Code

// .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "attach",
      "name": "Docker: Attach to Node",
      "port": 9229,
      "address": "localhost",
      "localRoot": "${workspaceFolder}",
      "remoteRoot": "/app",
      "restart": true
    }
  ]
}

Testing Strategies with Docker

Docker provides consistent environments for various types of testing.

Unit Testing Inside Containers

# Run tests in the container
docker-compose exec app npm test

# One-off test container
docker-compose run --rm app npm test

Dedicated Test Service in Docker Compose

services:
  app:
    # Main application service
    # ...
  
  test:
    build:
      context: .
      dockerfile: Dockerfile.dev
    volumes:
      - ./:/app
      - /app/node_modules
    command: npm test
    environment:
      - NODE_ENV=test

Integration Testing with Multiple Services

services:
  app:
    # Application service
    # ...
  
  db:
    # Database service
    # ...
  
  test:
    build: ./test
    depends_on:
      - app
      - db
    environment:
      - APP_URL=http://app:3000
      - DB_HOST=db

CI/CD Pipeline Testing

# GitHub Actions example
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run Tests
        run: docker-compose -f docker-compose.test.yml up --build --exit-code-from test

Team Workflow Strategies

Standardized Development Environment

Ensure all team members use the same Docker-based environment:

Include all Docker configuration in version control
Document container-based workflows in README
Create setup scripts for common tasks
Consider using Make or similar tools to simplify commands

Example Makefile for Common Tasks

# Makefile
.PHONY: up down build test shell logs

up:
	docker-compose up -d

down:
	docker-compose down

build:
	docker-compose build

test:
	docker-compose run --rm app npm test

shell:
	docker-compose exec app sh

logs:
	docker-compose logs -f

Development/Production Configuration Split

# Base configuration (docker-compose.yml)
version: '3.8'
services:
  app:
    build: .
    # Common configuration...

# Development overrides (docker-compose.override.yml)
version: '3.8'
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile.dev
    volumes:
      - ./:/app
    environment:
      NODE_ENV: development

# Production configuration (docker-compose.prod.yml)
version: '3.8'
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile.prod
    environment:
      NODE_ENV: production

Git Hooks for Docker Workflows

#!/bin/sh
# .git/hooks/pre-commit
docker-compose run --rm app npm run lint
docker-compose run --rm app npm test

Advanced Development Techniques

Multi-stage Development Builds

Using multi-stage builds for development:

# Dockerfile with development and production stages
FROM node:18-alpine AS base
WORKDIR /app
COPY package*.json ./

FROM base AS development
RUN npm install
COPY . .
CMD ["npm", "run", "dev"]

FROM base AS builder
RUN npm install
COPY . .
RUN npm run build

FROM nginx:alpine AS production
COPY --from=builder /app/build /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Using different targets in docker-compose.yml:

services:
  app:
    build:
      context: .
      target: development  # Use development stage for dev

VS Code Development Containers

Using VS Code's Remote - Containers extension:

// .devcontainer/devcontainer.json
{
  "name": "Node.js Development",
  "dockerComposeFile": "../docker-compose.yml",
  "service": "app",
  "workspaceFolder": "/app",
  "extensions": [
    "dbaeumer.vscode-eslint",
    "esbenp.prettier-vscode"
  ],
  "settings": {
    "terminal.integrated.shell.linux": "/bin/sh"
  }
}

Docker BuildKit Features for Development

# Enable BuildKit
export DOCKER_BUILDKIT=1

# Use BuildKit cache mounts for faster builds
# Dockerfile
FROM node:18-alpine

WORKDIR /app
COPY package*.json ./

RUN --mount=type=cache,target=/root/.npm \
    npm install

COPY . .
CMD ["npm", "run", "dev"]

Real-world Development Workflow Examples

Example 1: React Frontend Development

Optimized setup for React.js development:

# Dockerfile.dev
FROM node:18-alpine

WORKDIR /app

COPY package*.json ./
RUN npm install

COPY . .

EXPOSE 3000
CMD ["npm", "start"]

# docker-compose.yml
version: '3.8'

services:
  frontend:
    build:
      context: .
      dockerfile: Dockerfile.dev
    ports:
      - "3000:3000"
    volumes:
      - ./:/app
      - /app/node_modules
    environment:
      - CHOKIDAR_USEPOLLING=true  # For Hot Reload on Windows/Mac
      - WDS_SOCKET_PORT=3000  # For WebpackDevServer

Example 2: Full Stack MERN Application

Development setup for MongoDB, Express, React, and Node.js:

# Project structure
mern-app/
├── client/          # React frontend
├── server/          # Express backend
└── docker-compose.yml

# docker-compose.yml
version: '3.8'

services:
  client:
    build:
      context: ./client
      dockerfile: Dockerfile.dev
    ports:
      - "3000:3000"
    volumes:
      - ./client:/app
      - /app/node_modules
    environment:
      - REACT_APP_API_URL=http://localhost:5000
    depends_on:
      - server

  server:
    build:
      context: ./server
      dockerfile: Dockerfile.dev
    ports:
      - "5000:5000"
    volumes:
      - ./server:/app
      - /app/node_modules
    environment:
      - MONGO_URI=mongodb://mongo:27017/mernapp
      - PORT=5000
    depends_on:
      - mongo

  mongo:
    image: mongo:latest
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db

volumes:
  mongo-data:

Example 3: Microservices Development

Setup for developing microservices with shared dependencies:

# Project structure
microservices/
├── gateway/
├── service-a/
├── service-b/
├── shared-lib/
└── docker-compose.yml

# docker-compose.yml
version: '3.8'

services:
  gateway:
    build:
      context: .
      dockerfile: ./gateway/Dockerfile.dev
    volumes:
      - ./gateway:/app
      - ./shared-lib:/shared-lib
      - /app/node_modules
    ports:
      - "8000:8000"
    environment:
      - SERVICE_A_URL=http://service-a:3000
      - SERVICE_B_URL=http://service-b:3001

  service-a:
    build:
      context: .
      dockerfile: ./service-a/Dockerfile.dev
    volumes:
      - ./service-a:/app
      - ./shared-lib:/shared-lib
      - /app/node_modules

  service-b:
    build:
      context: .
      dockerfile: ./service-b/Dockerfile.dev
    volumes:
      - ./service-b:/app
      - ./shared-lib:/shared-lib
      - /app/node_modules

Common Development Issues and Solutions

Issue	Cause	Solution
Slow volume performance	File system performance on macOS/Windows	Use WSL2 on Windows Try docker-sync or Mutagen Mount only necessary directories
Node_modules conflicts	Volume mounting overriding container dependencies	Use anonymous volume: `- /app/node_modules` Run npm install inside the container
Hot reloading not working	File watching issues in containers	Set `CHOKIDAR_USEPOLLING=true` Configure webpack poll interval Ensure correct ports are exposed
Permission issues	User ID differences between host and container	Use `--user $(id -u):$(id -g)` Set file permissions in Dockerfile Use volume bind mount options
Container keeps restarting	Application crashes or exits	Check logs: `docker-compose logs` Add `tty: true` and `stdin_open: true` Run with `--init` flag

Hands-on Exercises

Exercise 1: Create a Development Environment for a Node.js Application

Set up a Docker-based development environment for a simple Node.js API:

Create a Node.js Express application with a basic API
Write a development-focused Dockerfile
Configure Docker Compose with appropriate volumes and ports
Add environment variables for development configuration
Test hot reloading by making changes to the code
Add a database service (MongoDB or PostgreSQL)

Exercise 2: Optimize Development Volume Performance

Improve the performance of volume mounts in a Docker development environment:

Start with the environment from Exercise 1
Benchmark initial file access performance
Optimize volume configurations for your OS
Try different mounting strategies (selective paths vs. entire directory)
Configure node_modules correctly
Re-benchmark and document performance improvements

Exercise 3: Set Up a Full-stack Development Environment

Create a development environment for a full-stack JavaScript application:

Set up a React frontend with hot reloading
Configure a Node.js backend API
Add a database container
Configure networking between services
Set up environment variables for each service
Create a convenient testing workflow
Add a Makefile or npm scripts for common tasks

Summary and Next Steps

Docker transforms development workflows by providing consistent, isolated environments that closely match production. The key benefits include:

Eliminating "works on my machine" problems through environment consistency
Faster onboarding for new team members
Isolation between projects and dependencies
Closer parity between development and production
Easier testing and debugging across environments

Effective Docker development workflows use strategies like:

Properly configured volume mounts for code changes
Environment variable management
Efficient debugging techniques
Consistent team practices
Development-specific optimizations

In the next lecture, we'll explore hot reloading in Docker containers, which further enhances the development experience by providing instant feedback as you code.