🪵 Log Aggregation

Why Aggregate Logs?

Imagine a detective investigating a case where witnesses are scattered across a dozen towns, each keeping a private diary in a different language. To reconstruct what happened, the detective would have to visit every town, translate each diary, and line the timelines up by hand. That's debugging a distributed system without log aggregation.

Log aggregation hires a courier to collect every diary, translate them into one language, and file them in a single searchable archive. Now the detective asks one question — "what happened to order 12345 between 14:30 and 14:33?" — and gets every relevant entry from every service, in order.

The Log Pipeline

Every aggregation system, whatever the branding, is the same four stages. Learn the shape once and every tool becomes "which piece plays which role."

Structured Logging with pino

The foundation of any log pipeline is the format your app emits. A human sentence like User 12 failed login from 1.2.3.4 is fine for one person reading one file, but a machine can't reliably filter it. Structured logging emits JSON, where every field is a queryable key.

Pino is the fast, modern choice for Node — it logs JSON by default and adds barely any overhead:

// logger.js
const pino = require('pino');

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  // Redact sensitive paths so secrets never reach disk (more on this below)
  redact: ['req.headers.authorization', 'password', '*.creditCard'],
  base: { service: 'user-service', env: process.env.NODE_ENV },
});

module.exports = logger;

// Using it — the second arg is structured context, not string concatenation
const logger = require('./logger');

logger.info({ userId: 12, route: '/login' }, 'Login attempt');
logger.warn({ userId: 12 }, 'User not found');
logger.error({ err, orderId: '12345' }, 'Payment failed');

{"level":30,"time":1715176337345,"service":"user-service","env":"production","userId":12,"route":"/login","msg":"Login attempt"}

Notice we log to stdout, not a file. In a containerized world that's the rule: write logs to standard output and let the platform (Docker, Kubernetes) capture and ship them. Your app shouldn't know or care where logs end up.

Log Levels

Levels let you turn the volume up or down without changing code. In development you might show everything down to debug; in production you show info and above, so the noise stays manageable and the bill stays lower.

Correlation IDs

A single user action fans out across services, each logging independently. A correlation ID (also called a request ID or, when it spans services, a trace ID) is one identifier attached to every log line for that action — the thread that lets you pull the whole story out of millions of lines.

In Express, generate the ID once per request and bind it to a child logger so you never have to pass it around manually:

const { randomUUID } = require('crypto');
const baseLogger = require('./logger');

app.use((req, res, next) => {
  // Reuse an incoming ID from upstream, or mint a fresh one
  const correlationId = req.headers['x-correlation-id'] || randomUUID();

  // A child logger stamps EVERY log on this request with the ID
  req.log = baseLogger.child({ correlationId });

  // Pass it downstream and back to the client for support tickets
  res.setHeader('x-correlation-id', correlationId);
  next();
});

app.get('/users/:id', (req, res) => {
  req.log.info({ userId: req.params.id }, 'Fetching user');
  // ...every log via req.log now carries the same correlationId
  res.json({ id: req.params.id });
});

Aggregation Stacks: ELK, Loki, Cloud

Three families dominate. They all implement the collect-store-analyze pipeline; they differ in how they index and what they cost.

The ELK / Elastic Stack

The classic open-source stack: Elasticsearch (stores and full-text-indexes logs), Logstash or lightweight Beats (collect and process), Kibana (visualize). It indexes the full content of every log, so search is powerful — but that indexing is storage-hungry.

# filebeat.yml — ship JSON logs to Elasticsearch
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
    json.keys_under_root: true    # promote JSON fields to top level
    json.add_error_key: true

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "app-logs-%{+yyyy.MM.dd}"    # daily time-based index

setup.kibana:
  host: "kibana:5601"

Grafana Loki — "like Prometheus, but for logs"

Loki takes a leaner approach: it indexes only a small set of labels (service, level, env) and stores the log body compressed and unindexed. That makes it dramatically cheaper than Elasticsearch, and it lives in the same Grafana pane as your metrics — so you jump from a latency spike straight to the logs behind it.

# promtail-config.yml — Loki's collector, scraping container logs
scrape_configs:
  - job_name: app
    static_configs:
      - targets: [localhost]
        labels:
          job: user-service
          env: production
          __path__: /var/log/app/*.log

Cloud-native — CloudWatch, Cloud Logging, Azure Monitor

If you run on a cloud provider, the managed option is often the least work. AWS CloudWatch Logs ingests stdout from Lambda and ECS automatically; GCP has Cloud Logging; Azure has Monitor Logs. You trade some flexibility and portability for zero infrastructure to run.

Retention & Security

Tiered retention

Logs are expensive to keep and most are never read. Use tiered storage that ages data out automatically:

• Hot (1–7 days) — fast, searchable, for active debugging
• Warm (weeks to months) — cheaper, for trend analysis
• Cold / archive (months to years) — object storage like S3, for compliance

# Elasticsearch Index Lifecycle Management (ILM) policy
policy:
  phases:
    hot:
      actions:
        rollover: { max_size: "50gb", max_age: "1d" }
    warm:
      min_age: "7d"
      actions:
        forcemerge: { max_num_segments: 1 }
    delete:
      min_age: "90d"      # logs older than 90 days are removed
      actions:
        delete: {}

Never log secrets or PII

This is the rule with teeth. A logged password or credit-card number is a breach waiting to happen and can violate GDPR or HIPAA. Redact at the source:

// pino redaction — these paths are replaced with [Redacted] before write
const logger = require('pino')({
  redact: {
    paths: ['password', 'req.headers.authorization', 'user.ssn', '*.creditCard'],
    censor: '[Redacted]',
  },
});

// This is now SAFE — the password field is scrubbed automatically
logger.info({ user: { name: 'Ada', password: 'hunter2' } }, 'User created');
// => {"user":{"name":"Ada","password":"[Redacted]"},"msg":"User created"}

Practice & Quiz

const logger = require('pino')();
logger.info({ orderId: 'A-1001', total: 49.99 }, 'Order placed');
// => {"level":30,...,"orderId":"A-1001","total":49.99,"msg":"Order placed"}

const { randomUUID } = require('crypto');
const baseLogger = require('pino')();

app.use((req, res, next) => {
  const correlationId = req.headers['x-correlation-id'] || randomUUID();
  req.log = baseLogger.child({ correlationId });
  res.setHeader('x-correlation-id', correlationId);
  next();
});

Best Practices & Pitfalls

✅ Do

• Log structured JSON to stdout and let the platform ship it
• Attach a correlation ID to every request and pass it downstream
• Choose log levels by the action required, and set the threshold per environment
• Configure redaction in the logger so secrets can't leak
• Set retention and lifecycle policies so old logs age out automatically

❌ Don't

• Log passwords, tokens, API keys, or PII — ever
• Build log messages with string concatenation instead of fields
• Write logs to local files inside a container (they die with the container)
• Keep everything forever — cost and compliance both bite you
• Log at debug in production and drown in noise

Summary

📚 Additional Resources

• Pino — API & redaction documentation
• Winston — logger documentation
• Grafana Loki — official documentation
• Elastic Stack (ELK) — documentation
• AWS CloudWatch Logs — documentation

🚀 What's Next?

You can now measure your system (metrics) and understand it (logs). The final piece is being told when something breaks — without drowning in noise. Next up: Alerting Systems, where you'll set thresholds on symptoms and SLOs, build escalation paths, and design alerts people actually trust.