📈 Application Monitoring

Why Monitor?

Think of your running application like a patient in a hospital. Monitoring is the bedside equipment: the heart-rate monitor beeping steadily, the blood-pressure cuff, the chart at the foot of the bed. Without it, the first sign of trouble is a code blue — the equivalent of a customer emailing "your site is down." With it, a nurse notices the numbers drifting and steps in early.

Observability is a bigger idea than monitoring. Monitoring answers questions you thought to ask in advance ("is CPU above 80%?"). Observability is the property of being able to ask new questions of your system — ones you never anticipated — just from the data it already emits. A modern production service should be observable enough that when something weird happens at 3 a.m., you can diagnose it without shipping new code.

This lesson focuses on the metrics and error side of the picture. The next lesson dives deep into logs, and the one after into alerting — together they form Week 13's monitoring-and-logging arc.

The Three Pillars of Observability

Almost every observability tool sorts telemetry into three signal types. Learn the differences and you'll always know which one to reach for.

Metrics — numbers measured over time

A metric is a numeric value sampled at regular intervals: requests per second, memory in use, error count. They're tiny to store and easy to aggregate (sum, average, percentile), which makes them perfect for dashboards and alerts. The four classic metric types:

Logs — timestamped events

A log line records that a discrete thing happened, with context attached. The single biggest upgrade you can make is moving from human-sentence logs to structured JSON:

// ❌ Unstructured — hard for machines to filter or aggregate
// [2026-05-08 14:32:17] ERROR Failed to process payment for order 12345

// ✅ Structured JSON — every field is queryable
{
  "timestamp": "2026-05-08T14:32:17.345Z",
  "level": "error",
  "service": "payment-service",
  "traceId": "abc123def456",
  "message": "Failed to process payment",
  "orderId": "12345",
  "errorCode": "CC_DECLINED"
}

We'll go deep on logging in the next lesson — for now, just note that logs carry the context a metric spike can't.

Traces — the journey of one request

In a microservice system, a single click might touch an API gateway, an auth service, a database, and a cache. A trace stitches those hops together into one timeline made of spans, so you can see exactly where the 800 ms went.

Each arrow above becomes a span with a start time, a duration, and a parent — and every span shares the same trace ID, which is the thread that ties your metrics, logs, and traces together.

What to Measure: RED & USE

You could measure a thousand things. Two battle-tested checklists tell you which handful actually matter.

The RED method — for services

Popularised by Weaveworks, RED is the user's-eye view of a request-serving service:

• Rate — requests per second
• Errors — how many of those requests failed
• Duration — how long they took (as a distribution, not just an average)

The USE method — for resources

Brendan Gregg's USE method looks at the machine underneath — CPU, memory, disk, network:

• Utilization — percentage of time the resource is busy
• Saturation — how much work is queued up waiting
• Errors — error events on that resource

The rule of thumb: RED for the code you wrote, USE for the infrastructure it runs on. Instrument RED first — it's what your users feel.

Health & Readiness Endpoints

Before dashboards, give your service two tiny endpoints that answer yes/no questions. Load balancers and orchestrators like Kubernetes poll these constantly to decide whether to send you traffic.

• Liveness (/healthz) — "Am I running at all?" If this fails, restart me.
• Readiness (/readyz) — "Am I ready to serve?" If my database is still connecting, keep traffic away until I say ready.

const express = require('express');
const app = express();

// Liveness: cheap, no dependencies. Just proves the process responds.
app.get('/healthz', (req, res) => {
  res.status(200).json({ status: 'ok', uptime: process.uptime() });
});

// Readiness: check the things you need to actually serve requests.
app.get('/readyz', async (req, res) => {
  try {
    await db.query('SELECT 1');        // is the database reachable?
    res.status(200).json({ status: 'ready' });
  } catch (err) {
    // 503 tells the load balancer: not ready, hold traffic
    res.status(503).json({ status: 'not-ready', reason: err.message });
  }
});

Metrics with Prometheus

Prometheus is the de-facto open-source metrics system. The model is simple: your app exposes a plain-text /metrics page, and Prometheus scrapes (polls) it every few seconds, storing each sample as time-series data you can query and alert on.

Instrumenting RED metrics in Express

Using the official prom-client library, one small middleware captures Rate, Errors, and Duration for every route:

const express = require('express');
const client = require('prom-client');
const app = express();

// Registry holds all our metrics; default metrics add CPU/memory/event-loop.
const register = new client.Registry();
client.collectDefaultMetrics({ register });

// Duration histogram — buckets in seconds. This one object gives us
// Rate (count), Errors (by status_code label), and Duration (buckets).
const httpDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 2, 5],
});
register.registerMetric(httpDuration);

// Middleware: start a timer, stop it when the response finishes.
app.use((req, res, next) => {
  const endTimer = httpDuration.startTimer();
  res.on('finish', () => {
    // Use req.route?.path (the pattern) not req.path, or high-cardinality
    // values like /users/42 would explode into thousands of series.
    const route = req.route ? req.route.path : req.path;
    endTimer({ method: req.method, route, status_code: res.statusCode });
  });
  next();
});

app.get('/', (req, res) => res.send('Hello, world!'));

// Expose metrics for Prometheus to scrape.
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

app.listen(3000, () => console.log('Listening on 3000'));

# HELP http_request_duration_seconds HTTP request duration in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{method="GET",route="/",status_code="200",le="0.05"} 42
http_request_duration_seconds_bucket{method="GET",route="/",status_code="200",le="0.1"} 47
http_request_duration_seconds_count{method="GET",route="/",status_code="200"} 48
http_request_duration_seconds_sum{method="GET",route="/",status_code="200"} 1.83

Dashboards & Error Tracking

Grafana — turning metrics into pictures

Grafana queries Prometheus (and many other sources) and renders dashboards. A solid RED dashboard for a service has three core panels:

# PromQL queries behind a RED dashboard
# Rate — requests per second, per route
rate(http_request_duration_seconds_count[5m])

# Errors — percentage of 5xx responses
sum(rate(http_request_duration_seconds_count{status_code=~"5.."}[5m]))
  / sum(rate(http_request_duration_seconds_count[5m]))

# Duration — 95th percentile latency
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route))

Add deployment annotations (a vertical line each time you ship) and you can instantly see if a release moved the numbers.

APM & error tracking with Sentry

Metrics tell you the error rate went up. They don't tell you the stack trace. That's the job of an error tracker / APM tool like Sentry, which captures every exception with its stack, request context, and the release it happened on — then groups duplicates so ten thousand crashes become one issue.

const Sentry = require('@sentry/node');

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  environment: process.env.NODE_ENV,
  release: process.env.GIT_SHA,      // ties errors to a specific deploy
  tracesSampleRate: 0.1,             // sample 10% of transactions for perf
});

// Sentry's Express error handler goes AFTER your routes, so it catches
// anything they throw and reports it with full request context.
Sentry.setupExpressErrorHandler(app);

Distributed Tracing with OpenTelemetry

OpenTelemetry (OTel) is the vendor-neutral standard for generating traces, metrics, and logs. You instrument once against the OTel API, then point it at any backend — Jaeger, Tempo, Datadog, Honeycomb — without touching your code again.

The best part is auto-instrumentation: OTel patches popular libraries (Express, HTTP, database drivers) so you get spans for free.

// tracing.js — load this BEFORE anything else with: node -r ./tracing app.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

const sdk = new NodeSDK({
  serviceName: 'product-service',
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();
console.log('OpenTelemetry tracing started');

Need a span around your own business logic? Create one manually:

const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('product-service');

async function getProduct(id) {
  return tracer.startActiveSpan('getProduct', async (span) => {
    span.setAttribute('product.id', id);
    try {
      const product = await db.getProduct(id);
      return product;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: 2 }); // 2 = ERROR
      throw err;
    } finally {
      span.end();
    }
  });
}

Practice & Quiz

async function checkDatabase() {
  // pretend this pings the DB; flip it to test both paths
  return true;
}

// TODO: add app.get('/readyz', ...) that returns 200 {status:'ready'}
// when checkDatabase() resolves true, else 503 {status:'not-ready'}

app.get('/readyz', async (req, res) => {
  try {
    const ok = await checkDatabase();
    if (ok) return res.status(200).json({ status: 'ready' });
    return res.status(503).json({ status: 'not-ready' });
  } catch (err) {
    return res.status(503).json({ status: 'not-ready', reason: err.message });
  }
});

const client = require('prom-client');
const requests = new client.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'status_code'],
});

app.use((req, res, next) => {
  res.on('finish', () => {
    requests.inc({ method: req.method, status_code: res.statusCode });
  });
  next();
});

Best Practices & Pitfalls

✅ Do

• Instrument the RED metrics for every service before anything fancy
• Expose separate /healthz (liveness) and /readyz (readiness) endpoints
• Keep metric labels bounded — method, route pattern, status code
• Include a shared trace/correlation ID across metrics, logs, and traces
• Measure percentiles (p95/p99), not just averages — averages hide the slow tail

❌ Don't

• Check external dependencies inside a liveness probe
• Label metrics with high-cardinality values (user IDs, request IDs, full URLs)
• Rely on averages — a 200 ms average can hide a 4 s p99
• Alert on raw CPU when what users feel is latency and errors (more on this in the alerting lesson)

Summary

📚 Additional Resources

• Prometheus — Documentation & overview
• Grafana — Official documentation
• OpenTelemetry — JavaScript SDK docs
• Sentry — Node.js error & performance monitoring
• Google SRE Book — Monitoring distributed systems

🚀 What's Next?

You can now measure that something is wrong. Next you'll learn to find out why by centralising the second pillar — logs — in Log Aggregation: structured logging with pino/winston, correlation IDs, and shipping to ELK, Loki, or CloudWatch.