Debugging Containerized Applications

Techniques, tools, and strategies for effective debugging in Docker environments

The Debugging Challenge in Containers

Debugging is a critical skill for all developers, but containerized applications introduce unique challenges. Containers add an additional layer of abstraction between you and your code, which can make traditional debugging approaches more difficult. However, with the right techniques and tools, debugging in Docker can be just as effective — and sometimes even more powerful — than debugging traditional applications.

The Surgery Analogy

Think of debugging a containerized application like performing surgery:

  • Traditional debugging is like open surgery — you have direct access to everything, but it's messy, invasive, and the environment isn't sterile or controlled.
  • Container debugging is more like laparoscopic surgery — you're working through smaller interfaces with specialized tools. It requires different techniques and might seem more constrained at first, but it's actually more precise, controlled, and reproducible.

Just as modern surgery has evolved to be less invasive yet more effective, modern debugging in containers can be more powerful despite the additional abstraction layer.

Container Debugging Challenges

flowchart TD A[Containerized Application] --> B[Limited Access] A --> C[Ephemeral Nature] A --> D[Configuration Complexity] A --> E[Network Isolation] A --> F[Resource Constraints] B --> B1[No direct filesystem access] B --> B2[No GUI for desktop tools] C --> C1[Containers may be recreated] C --> C2[State lost on restart] D --> D1[Multiple configuration layers] D --> D2[Environment variables] D --> D3[Volume mounts] E --> E1[Custom networks] E --> E2[Port mappings] F --> F1[CPU/Memory limits] F --> F2[Disk space issues]

The Layers of Container Debugging

Effective debugging of containerized applications requires understanding the different layers where issues can occur.

Host System Docker installation, resources, OS settings Docker Configuration Dockerfiles, Compose files, volumes, networks Container Runtime Container lifecycle, resource usage, processes Application Environment Dependencies, configuration, environment variables Application Code Business logic, frameworks, libraries

Different debugging techniques and tools are needed depending on which layer you're troubleshooting:

Understanding which layer a problem is occurring in helps you choose the right debugging approach.

Basic Debugging Techniques

Container Logs

Docker logs are the first line of defense for debugging containerized applications:

# View logs for a specific container
docker logs container_id

# Follow logs in real-time
docker logs -f container_id

# Show timestamps
docker logs -t container_id

# Show only the last N lines
docker logs --tail=100 container_id

# Show logs since a specific time
docker logs --since=1h container_id

# With Docker Compose
docker-compose logs service_name
docker-compose logs -f service_name

Log output includes everything written to stdout and stderr by the process running inside the container.

Executing Commands in Running Containers

Accessing a shell inside a running container allows for interactive debugging:

# Open a shell in a running container
docker exec -it container_id /bin/sh
# or
docker exec -it container_id bash

# With Docker Compose
docker-compose exec service_name sh

Once inside the container, you can:

Container Inspection

Docker provides detailed information about containers:

# Get detailed information about a container
docker inspect container_id

# Filter for specific information
docker inspect -f '{{ .NetworkSettings.IPAddress }}' container_id
docker inspect -f '{{ .Config.Env }}' container_id
docker inspect -f '{{ .Mounts }}' container_id

Monitoring Container Resources

# View resource usage statistics
docker stats

# View processes running in a container
docker top container_id

# With Docker Compose
docker-compose top service_name

Effective Logging Strategies

Structured Logging

Structured logs (usually JSON) provide more context and are easier to parse:

// Example of structured logging in Node.js
const winston = require('winston');

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.json(),
  defaultMeta: { service: 'user-service' },
  transports: [
    new winston.transports.Console()
  ]
});

// Usage
logger.info('User logged in', { 
  userId: '123', 
  timestamp: new Date().toISOString(),
  requestId: 'abc-123-xyz'
});

Log Levels

Use appropriate log levels to filter noise and highlight important events:

Centralized Logging

For multi-container applications, centralized logging is essential:

# docker-compose.yml with ELK stack for logging
version: '3.8'

services:
  app:
    build: .
    logging:
      driver: "json-file"
      options:
        max-size: "200k"
        max-file: "10"
    # ... other configuration
        
  elasticsearch:
    image: elasticsearch:7.17.0
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
        
  logstash:
    image: logstash:7.17.0
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    depends_on:
      - elasticsearch
        
  kibana:
    image: kibana:7.17.0
    ports:
      - "5601:5601"
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    depends_on:
      - elasticsearch

volumes:
  elasticsearch-data:

Log Collection in Production

For production environments, consider:

Debugging Frontend Applications in Containers

Source Maps

Source maps are essential for debugging minified JavaScript:

// webpack.config.js for development
module.exports = {
  mode: 'development',
  devtool: 'source-map',  // Generate source maps
  // ... rest of config
};

Remote Debugging with Chrome DevTools

For React, Vue, or other frontend applications:

# Dockerfile.dev
FROM node:18-alpine

WORKDIR /app

COPY package*.json ./
RUN npm install

COPY . .

EXPOSE 3000
CMD ["npm", "start"]
# docker-compose.yml
version: '3.8'

services:
  frontend:
    build:
      context: .
      dockerfile: Dockerfile.dev
    ports:
      - "3000:3000"  # Application port
      - "9229:9229"  # Debug port (for Node.js internals)
    volumes:
      - ./:/app
      - /app/node_modules
    environment:
      - NODE_ENV=development
      - CHOKIDAR_USEPOLLING=true

With this setup, you can:

  1. Open Chrome and navigate to http://localhost:3000
  2. Open Chrome DevTools (F12 or right-click → Inspect)
  3. Use the Elements, Console, Network, and Sources panels as usual
  4. Set breakpoints, inspect variables, and debug normally

React Developer Tools in Containers

Install React DevTools in the container:

# Dockerfile.dev with React DevTools
FROM node:18-alpine

WORKDIR /app

# Install React DevTools
RUN npm install -g react-devtools

COPY package*.json ./
RUN npm install

COPY . .

EXPOSE 3000 8097
CMD ["sh", "-c", "react-devtools & npm start"]

Make sure to expose port 8097 in your docker-compose.yml:

ports:
  - "3000:3000"
  - "8097:8097"  # React DevTools port

Debugging Node.js Applications in Containers

Node.js Inspector Protocol

Node.js provides a built-in inspector protocol for debugging:

# Start Node.js with inspector
node --inspect=0.0.0.0:9229 app.js

# Or for immediate breakpoint at start
node --inspect-brk=0.0.0.0:9229 app.js

Key points for container-based debugging:

Express.js Debugging Setup

# package.json
{
  "scripts": {
    "start": "node src/index.js",
    "dev": "nodemon src/index.js",
    "debug": "node --inspect=0.0.0.0:9229 src/index.js",
    "debug:watch": "nodemon --inspect=0.0.0.0:9229 src/index.js"
  }
}

# docker-compose.yml
services:
  api:
    build:
      context: .
      dockerfile: Dockerfile.dev
    ports:
      - "4000:4000"  # API port
      - "9229:9229"  # Debug port
    volumes:
      - ./:/app
      - /app/node_modules
    command: npm run debug:watch

Debugging with Chrome DevTools

To connect Chrome DevTools to a Node.js container:

  1. Open Chrome and navigate to chrome://inspect
  2. Click "Configure..." and add localhost:9229
  3. Under "Remote Target", you should see your Node.js application
  4. Click "inspect" to open DevTools connected to your Node.js process
  5. Set breakpoints, inspect variables, view the call stack, etc.

VS Code Debugging

// .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "attach",
      "name": "Docker: Attach to Node",
      "port": 9229,
      "address": "localhost",
      "localRoot": "${workspaceFolder}",
      "remoteRoot": "/app",
      "restart": true
    }
  ]
}

With this configuration, you can:

  1. Start your containerized application with the debug flag
  2. In VS Code, go to the Run panel (Ctrl+Shift+D)
  3. Select "Docker: Attach to Node" and press play
  4. Use VS Code's rich debugging features: breakpoints, watches, call stack, etc.

Debugging Multi-container Applications

Inspecting Container Communication

Tools for debugging container networking:

# Install network tools in a container
docker exec -it container_id sh -c "apt-get update && apt-get install -y iputils-ping net-tools curl"

# Check DNS resolution
docker exec -it container_id nslookup service_name

# Test connectivity to another service
docker exec -it container_id curl http://service_name:port

# View network settings
docker network inspect network_name

Simulating Network Conditions

Testing with network latency or failures:

# Add network latency
docker exec -it container_id tc qdisc add dev eth0 root netem delay 100ms

# Simulate packet loss
docker exec -it container_id tc qdisc add dev eth0 root netem loss 10%

# Remove network conditions
docker exec -it container_id tc qdisc del dev eth0 root

Debugging Microservices Interactions

sequenceDiagram participant C as Client participant G as API Gateway participant S1 as Service 1 participant S2 as Service 2 participant DB as Database C->>G: Request G->>S1: Forward Request S1->>DB: Query Data DB-->>S1: Response S1->>S2: Call Service 2 S2->>DB: Query Data DB-->>S2: Response S2-->>S1: Response S1-->>G: Response G-->>C: Response Note over G,S1: Debug point 1: API Gateway logs Note over S1,S2: Debug point 2: Service communication Note over S2,DB: Debug point 3: Database queries

Using Distributed Tracing

Implement distributed tracing for complex microservices:

# docker-compose.yml with Jaeger for tracing
version: '3.8'

services:
  # Your application services...
  
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "5775:5775/udp"
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5778:5778"
      - "16686:16686"  # UI
      - "14268:14268"
      - "9411:9411"
    environment:
      - COLLECTOR_ZIPKIN_HOST_PORT=9411

Then in your services, initialize a tracer:

// Node.js example with Jaeger tracer
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');
const { NodeTracerProvider } = require('@opentelemetry/node');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');

// Initialize tracer
const provider = new NodeTracerProvider();
const exporter = new JaegerExporter({
  serviceName: 'my-service',
  endpoint: 'http://jaeger:14268/api/traces'
});

provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

registerInstrumentations({
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation()
  ]
});

Advanced Debugging Techniques

Core Dumps in Containers

Capturing core dumps from crashing containers:

# Enable core dumps in the host
echo "/tmp/cores/core.%e.%p" | sudo tee /proc/sys/kernel/core_pattern
sudo mkdir -p /tmp/cores
sudo chmod 777 /tmp/cores

# Run container with core dump support
docker run --ulimit core=-1 --security-opt seccomp=unconfined -v /tmp/cores:/tmp/cores my-image

Analyzing core dumps:

# Install debugging tools in the container
docker exec -it container_id apt-get update && apt-get install -y gdb

# Analyze a core dump
docker exec -it container_id gdb /path/to/executable /tmp/cores/core.file

Profiling Applications in Containers

For Node.js applications:

# Run Node.js with profiling enabled
docker-compose exec service_name node --prof app.js

# Generate a processed log
docker-compose exec service_name node --prof-process isolate-0xNNNNNNNN-v8.log > processed.txt

# CPU profiling with Chrome DevTools
docker-compose exec service_name node --inspect=0.0.0.0:9229 app.js

Memory Leak Detection

Tracking memory usage and detecting leaks:

# Monitor memory usage
docker stats container_id

# For Node.js applications
docker exec -it container_id node --inspect=0.0.0.0:9229 --expose-gc app.js

Then in Chrome DevTools, you can:

  1. Take heap snapshots
  2. Compare snapshots to find growing objects
  3. Use the Memory timeline to track allocations

Using strace for System Call Tracing

# Install strace in the container
docker exec -it container_id apt-get update && apt-get install -y strace

# Trace system calls for a process
docker exec -it container_id strace -p process_id

# Attach to a new process
docker exec -it container_id strace command

Debugging in Production

Debugging Strategies for Production

Production environments require different approaches:

Health Checks and Monitoring

Implement health checks to detect issues early:

# docker-compose.yml with health checks
services:
  api:
    build: ./api
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s

Create a health endpoint in your application:

// Express.js health endpoint
app.get('/health', (req, res) => {
  // Basic health check
  res.status(200).json({ status: 'UP' });
  
  // More comprehensive health check
  const health = {
    status: 'UP',
    timestamp: new Date(),
    services: {
      database: isDatabaseConnected ? 'UP' : 'DOWN',
      cache: isCacheConnected ? 'UP' : 'DOWN',
      // Check other dependencies
    },
    memory: process.memoryUsage(),
    uptime: process.uptime()
  };
  
  const overallStatus = Object.values(health.services).every(s => s === 'UP') ? 200 : 500;
  res.status(overallStatus).json(health);
});

Implementing Circuit Breakers

Circuit breakers help prevent cascading failures:

// Node.js example with Opossum
const CircuitBreaker = require('opossum');

// Function to make an API call
async function apiCall() {
  const response = await fetch('http://api-service/endpoint');
  if (!response.ok) throw new Error(`HTTP error ${response.status}`);
  return response.json();
}

// Create a circuit breaker
const breaker = new CircuitBreaker(apiCall, {
  timeout: 3000,              // Time in ms before timeout
  errorThresholdPercentage: 50, // Error threshold percentage
  resetTimeout: 30000         // Time to wait before trying again
});

// Use the circuit breaker
breaker.fire()
  .then(data => console.log(data))
  .catch(err => console.error(err));

// Listen for events
breaker.on('open', () => console.log('Circuit breaker opened'));
breaker.on('close', () => console.log('Circuit breaker closed'));
breaker.on('halfOpen', () => console.log('Circuit breaker half-open'));

Crash Recovery Strategies

Implement strategies to handle container crashes:

# docker-compose.yml with restart policies
services:
  api:
    build: ./api
    restart: unless-stopped
    # or
    # restart: on-failure:5
    # for maximum 5 restart attempts

Graceful shutdown in Node.js:

// Handle graceful shutdown
process.on('SIGTERM', () => {
  console.log('SIGTERM received, shutting down gracefully');
  
  // Close server connections
  server.close(() => {
    console.log('HTTP server closed');
    
    // Close database connections
    mongoose.connection.close(false, () => {
      console.log('Database connections closed');
      process.exit(0);
    });
  });
  
  // Force close after timeout
  setTimeout(() => {
    console.error('Could not close connections in time, forcefully shutting down');
    process.exit(1);
  }, 10000);
});

Framework-specific Debugging Techniques

React

Debugging React applications in containers:

// React Error Boundary example
class ErrorBoundary extends React.Component {
  constructor(props) {
    super(props);
    this.state = { hasError: false, error: null, errorInfo: null };
  }
  
  componentDidCatch(error, errorInfo) {
    this.setState({
      hasError: true,
      error: error,
      errorInfo: errorInfo
    });
    // Log the error
    console.error("Error caught by boundary:", error, errorInfo);
  }
  
  render() {
    if (this.state.hasError) {
      return (
        <div className="error-container">
          <h2>Something went wrong.</h2>
          <details>
            <summary>Error Details</summary>
            <p>{this.state.error && this.state.error.toString()}</p>
            <p>Component Stack: {this.state.errorInfo.componentStack}</p>
          </details>
        </div>
      );
    }
    return this.props.children;
  }
}

Express.js

Debugging Express applications:

// Using debug module
const debug = require('debug')('app:routes');

app.get('/users', (req, res) => {
  debug('GET /users called with query:', req.query);
  // Route handler code
});

// Error handling middleware
app.use((err, req, res, next) => {
  console.error('Error:', err);
  
  // Log details only in development
  if (process.env.NODE_ENV === 'development') {
    res.status(500).json({
      error: err.message,
      stack: err.stack,
      details: err
    });
  } else {
    // In production, return minimal error info
    res.status(500).json({
      error: 'Internal Server Error'
    });
  }
});

MongoDB

Debugging MongoDB interactions:

// Enable Mongoose debug mode
mongoose.set('debug', process.env.NODE_ENV === 'development');

// Detailed connection logging
mongoose.connect(uri, {
  useNewUrlParser: true,
  useUnifiedTopology: true
})
.then(() => console.log('MongoDB connected'))
.catch(err => {
  console.error('MongoDB connection error:', err);
  process.exit(1);
});

// Log slow queries
const UserSchema = new mongoose.Schema({
  // Schema definition
});

// Add query logging
if (process.env.NODE_ENV === 'development') {
  UserSchema.pre('find', function() {
    console.time(`User.find execution time`);
    this._startTime = Date.now();
  });
  
  UserSchema.post('find', function() {
    console.timeEnd(`User.find execution time`);
    console.log(`User.find took ${Date.now() - this._startTime}ms`);
  });
}

PostgreSQL

Debugging PostgreSQL interactions:

// Node.js with pg module
const { Pool } = require('pg');

const pool = new Pool({
  host: process.env.DB_HOST,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
  database: process.env.DB_NAME,
  // Add query logging in development
  ...(process.env.NODE_ENV === 'development' ? {
    log: (msg) => console.log(msg),
    query_timeout: 10000  // Longer timeout for debugging
  } : {})
});

// Wrap query with timing
async function queryWithLogging(text, params) {
  const start = Date.now();
  try {
    const res = await pool.query(text, params);
    const duration = Date.now() - start;
    
    if (duration > 100) {  // Log slow queries
      console.log('Slow query:', { text, duration, rows: res.rowCount });
    }
    
    return res;
  } catch (e) {
    console.error('Query error:', e);
    console.error('Query was:', text, params);
    throw e;
  }
}

Common Issues and Solutions

Problem Possible Causes Debugging Steps
Container won't start
  • Dockerfile errors
  • Command issues
  • Volume mount problems
  • Port conflicts
  • Check docker logs
  • Use docker-compose config to validate
  • Try running with a shell command first
  • Check for running containers with docker ps
Services can't communicate
  • Network misconfiguration
  • Service not ready
  • Incorrect hostnames
  • Firewall issues
  • Check network with docker network inspect
  • Test connectivity with ping and curl
  • Ensure service is actually running and listening
  • Use depends_on with health checks
Volume mount issues
  • Incorrect paths
  • Permission problems
  • Missing volumes
  • Docker for Windows/Mac issues
  • Check paths with docker inspect
  • Verify file permissions
  • Use absolute paths
  • Check Docker file sharing settings
Performance issues
  • Resource constraints
  • I/O bottlenecks
  • Network latency
  • Inefficient code
  • Use docker stats to monitor resources
  • Run profiling tools
  • Check disk and network I/O
  • Increase container resource limits if needed
Environment variables not set
  • Missing in Dockerfile/Compose
  • .env file issues
  • Scope problems
  • Check with docker exec env
  • Verify .env file is in correct location
  • Check variable scope in docker-compose
  • Use printenv inside container

Hands-on Exercises

Exercise 1: Basic Container Debugging

Practice fundamental container debugging techniques:

  1. Create a simple Express API with a deliberate bug
  2. Containerize the application with Docker
  3. Start the container and observe the failure
  4. Use container logs to identify the issue
  5. Access a shell in the container to further investigate
  6. Fix the bug and rebuild the container
  7. Verify the fix worked

Exercise 2: Remote Debugging with Node.js

Set up and use remote debugging with a Node.js application:

  1. Create a Node.js/Express application with some complexity
  2. Configure the application for debugging (--inspect flag)
  3. Set up Docker Compose with the appropriate ports and volumes
  4. Create a VS Code launch configuration for remote debugging
  5. Set breakpoints in your code
  6. Start the container and attach the debugger
  7. Step through code execution, inspect variables, and explore the call stack

Exercise 3: Debugging Multi-container Communication

Troubleshoot issues in a multi-container application:

  1. Create a three-service application (Frontend, API, Database)
  2. Introduce networking issues between containers
  3. Use logging to identify where communication breaks down
  4. Inspect network configuration with Docker commands
  5. Test connectivity between containers
  6. Resolve the issues and verify the application works end-to-end
  7. Implement proper health checks to detect similar issues early

Summary and Best Practices

Key Takeaways

Debugging Best Practices

Additional Resources