🔗 Service Communication Patterns

The Network Changes Everything

In a monolith, one part of the app calls another with an ordinary function call: instantaneous, reliable, strongly typed, and impossible to get "half done." The moment you split those parts into separate services, that same call becomes a network request — and everything you took for granted disappears.

Think of the difference between asking a colleague a question when they're sitting next to you versus phoning them. In person, you get an answer immediately and you know they heard you. On the phone, the call might not connect, there's a delay, the line can drop mid-sentence, and sometimes you're left wondering whether they got your message at all. Every microservice call is a phone call, and this lesson is about making those calls dependable.

Two big questions shape every design decision here:

• Does the caller wait? Synchronous means "ask and block until the answer comes back." Asynchronous means "fire a message and carry on." That single choice drives coupling, resilience, and consistency.
• What happens when it fails? Not if — when. A resilient service assumes its dependencies will be slow or down and degrades gracefully instead of dragging everyone down with it.

Synchronous: REST and gRPC

In synchronous communication the caller sends a request and waits — its own work is paused until the response arrives or the call fails. It's the most natural style because it mirrors a normal function call, and it's the right choice when the caller genuinely needs the answer now to keep going (a user is waiting on a page load, an order can't proceed until payment clears).

REST over HTTP

REST is the default for service-to-service calls: plain HTTP, JSON bodies, familiar verbs and status codes, debuggable with curl. Version the path so you can evolve the contract without breaking callers, and always set a timeout — a call with no timeout can hang forever and take your service down with it.

// Order Service calling the Product Service over REST.
// NOTE: never leave a network call without a timeout.
async function getProduct(productId) {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), 3000); // 3s budget

    try {
        const res = await fetch(
            `http://product-service/api/v1/products/${productId}`,
            { signal: controller.signal }
        );
        if (!res.ok) {
            // A 404 is a real answer; a 500 is a failure worth retrying.
            throw new Error(`Product service responded ${res.status}`);
        }
        return await res.json();
    } finally {
        clearTimeout(timer); // always clean up the timer
    }
}

gRPC for high-performance internal calls

When two services talk a lot and you control both ends, gRPC is often a better fit than REST. You define the contract once in a .proto file; gRPC generates strongly typed clients and servers, serializes with compact binary Protocol Buffers instead of verbose JSON, and rides HTTP/2 so it can stream and multiplex. The trade-off: it's less human-readable and not directly callable from a browser.

// product.proto — the single source of truth for both sides.
// gRPC generates the client and server stubs from this file.
syntax = "proto3";
package product;

service ProductService {
  rpc GetProduct (GetProductRequest) returns (Product);
  // Server-streaming: push a live feed of price changes.
  rpc WatchPrice (GetProductRequest) returns (stream PriceUpdate);
}

message GetProductRequest { string product_id = 1; }

message Product {
  string id = 1;
  string name = 2;
  double price = 3;
  bool in_stock = 4;
}

message PriceUpdate {
  string product_id = 1;
  double new_price = 2;
}

Asynchronous: Events and Queues

In asynchronous communication the caller hands off a message and immediately moves on — it does not wait for the work to finish. This is like sending an email instead of making a phone call: you don't sit on the line, and the recipient handles it when they're ready. A message broker (RabbitMQ, Apache Kafka, AWS SQS/SNS) sits in the middle, holding messages until a consumer is available.

Events vs. commands

An event is a statement of fact about something that already happened — OrderPlaced, PaymentCaptured, InventoryReserved. The publisher announces it and doesn't know or care who listens. Any number of services can subscribe and react independently. This is what truly decouples services: the Order service has no idea the Notification, Analytics, and Inventory services all react to its events.

Picture a bulletin board in a shared office. The Order service pins a note reading "Order 1234 was placed." It has no idea who reads the board — but Inventory, Notification, and Analytics each check it and act when they see something relevant. Adding a new reader never requires changing the writer.

// Publishing an event with Kafka. The producer is fire-and-forget
// from the caller's point of view — it does not wait for consumers.
const { Kafka } = require('kafkajs');
const kafka = new Kafka({ clientId: 'order-service', brokers: ['kafka:9092'] });
const producer = kafka.producer();

async function publishOrderPlaced(order) {
    await producer.connect();
    await producer.send({
        topic: 'orders',
        messages: [{
            key: order.id, // same key -> same partition -> ordered per order
            value: JSON.stringify({
                type: 'OrderPlaced',
                data: { orderId: order.id, userId: order.userId, total: order.total },
                occurredAt: new Date().toISOString()
            })
        }]
    });
}

// A consumer. Because events can be delivered more than once,
// consumers MUST be idempotent — processing the same event twice
// must not send two emails or ship two boxes.
async function startNotificationConsumer() {
    const consumer = kafka.consumer({ groupId: 'notification-service' });
    await consumer.connect();
    await consumer.subscribe({ topic: 'orders', fromBeginning: false });

    await consumer.run({
        eachMessage: async ({ message }) => {
            const event = JSON.parse(message.value.toString());
            if (event.type !== 'OrderPlaced') return;

            // Guard against duplicate delivery with a processed-ids check.
            if (await alreadyHandled(event.data.orderId)) return;

            await sendConfirmationEmail(event.data);
            await markHandled(event.data.orderId);
        }
    });
}

Service Discovery

Hardcoding http://192.168.1.42:3002 works right up until that instance is replaced, scaled to five copies, or moved to a new host — which, in a container orchestrator, happens constantly. Service discovery solves this: instances register themselves in a central service registry as they start, and callers look up healthy instances by logical name instead of address.

There are two shapes for this:

• Client-side discovery — the caller asks the registry for instances and picks one itself (often round-robin or random). More control, more logic in every client. Tools: Consul, etcd, Eureka.
• Server-side discovery — the caller hits a fixed load balancer or DNS name, and that layer consults the registry and forwards the request. Simpler clients. This is what Kubernetes gives you for free: a Service name like product-service resolves via cluster DNS to healthy pods.

# In Kubernetes, a Service IS your discovery mechanism.
# Other pods just call http://product-service and the cluster
# routes to a healthy, ready pod behind this stable name.
apiVersion: v1
kind: Service
metadata:
  name: product-service
spec:
  selector:
    app: product          # routes to every pod labelled app=product
  ports:
    - port: 80            # stable virtual port callers use
      targetPort: 3002    # actual container port

The registry only forwards to instances that pass their health checks, so a crashed or overloaded instance is quietly taken out of rotation. That's why every service should expose a lightweight /health endpoint that reports whether it can actually serve traffic — not just whether the process is running.

Resilience Patterns

Failures in a distributed system are normal operating conditions, not exceptions. The goal isn't to prevent them — you can't — but to contain them so one sick service doesn't take down the whole system. Four patterns do most of the work.

1. Timeouts

The most important and most forgotten. A call with no timeout can wait forever; while it waits, it holds a thread, a connection, and memory. Enough hung calls and your service runs out of resources and dies — killed not by its own bug but by a slow dependency. Every network call gets a timeout budget.

2. Retries with exponential backoff and jitter

Many failures are transient: a brief network blip, a pod restarting. Retrying often works. But retry naively and you make things worse — a struggling service gets hammered by a synchronized storm of retries. The fix is exponential backoff (wait longer after each failure) plus jitter (randomize the wait so callers don't all retry in lockstep). Only retry idempotent operations, and cap the attempts.

// Retry with exponential backoff + jitter. Only for idempotent calls!
async function withRetry(fn, { attempts = 3, baseMs = 200 } = {}) {
    for (let i = 0; i < attempts; i++) {
        try {
            return await fn();
        } catch (err) {
            const isLast = i === attempts - 1;
            if (isLast) throw err; // give up, let the caller decide

            // 200ms, 400ms, 800ms ... plus random jitter to de-sync callers.
            const backoff = baseMs * 2 ** i;
            const jitter = Math.random() * baseMs;
            await new Promise(r => setTimeout(r, backoff + jitter));
        }
    }
}

// Usage: safe because a GET is idempotent.
const product = await withRetry(() => getProduct('prod_123'));

3. Circuit breaker

If a service is clearly down, retrying just wastes time and piles on load. A circuit breaker wraps a call and watches for failures. After too many, it "trips" to OPEN and fails fast — instantly, without even attempting the call — for a cooldown period. Then it goes HALF-OPEN and allows one trial request: succeed and it closes (healthy again); fail and it re-opens. It's the electrical breaker in your home: trip on overload, wait, test, reset.

// A minimal circuit breaker. In production reach for a library
// like 'opossum', but understanding the mechanics matters.
class CircuitBreaker {
    constructor(action, { threshold = 5, cooldownMs = 10000 } = {}) {
        this.action = action;
        this.threshold = threshold;
        this.cooldownMs = cooldownMs;
        this.failures = 0;
        this.state = 'CLOSED';   // CLOSED healthy | OPEN failing fast | HALF_OPEN testing
        this.openedAt = 0;
    }

    async fire(...args) {
        if (this.state === 'OPEN') {
            if (Date.now() - this.openedAt < this.cooldownMs) {
                throw new Error('Circuit open — failing fast');
            }
            this.state = 'HALF_OPEN'; // cooldown done: allow ONE trial
        }
        try {
            const result = await this.action(...args);
            this.failures = 0;
            this.state = 'CLOSED';    // success: fully recovered
            return result;
        } catch (err) {
            this.failures++;
            if (this.failures >= this.threshold) {
                this.state = 'OPEN';
                this.openedAt = Date.now();
            }
            throw err;
        }
    }
}

4. Bulkheads

Named after the watertight compartments in a ship's hull: if one floods, the others keep the ship afloat. In software, a bulkhead isolates resources so a failure in one area can't drain the pool everyone shares. Give each downstream dependency its own limited pool of connections or concurrent calls — then a hung dependency exhausts only its own bulkhead, and calls to healthy services keep flowing.

Sagas & Eventual Consistency

Here's the problem microservices make genuinely hard. In a monolith, "reserve inventory, charge the card, create the order" is one database transaction: it all commits or it all rolls back. Split those across three services with three databases and you can't use a single ACID transaction anymore — there's no shared database to roll back.

The answer is the saga pattern: model the process as a sequence of local transactions, one per service. Each step publishes an event that triggers the next. If a step fails, you don't roll back — you run compensating transactions that undo the completed steps, walking the process backward to a consistent state.

Think of booking a trip: reserve the flight, then the hotel, then the rental car. If the car booking fails, you don't have a magic undo button — you explicitly cancel the hotel and cancel the flight. Those cancellations are the compensating transactions. The saga is the coordinator that knows the forward steps and their matching undos.

// An orchestration-style saga: one coordinator drives the steps
// and runs compensations in reverse if anything fails.
async function placeOrderSaga(order) {
    const done = []; // track completed steps for rollback

    try {
        await inventory.reserve(order.items);
        done.push(() => inventory.release(order.items)); // compensation

        const receipt = await payment.charge(order.userId, order.total);
        done.push(() => payment.refund(receipt.id));     // compensation

        await orders.confirm(order.id);
        return { status: 'CONFIRMED' };
    } catch (err) {
        // Something failed — undo completed steps in REVERSE order.
        for (const compensate of done.reverse()) {
            await compensate(); // each must be idempotent and retry-safe
        }
        await orders.cancel(order.id);
        return { status: 'CANCELLED', reason: err.message };
    }
}

Sagas come in two flavors. Orchestration (above) uses a central coordinator that tells each service what to do — easy to follow, but the coordinator is a hub. Choreography has no coordinator: each service listens for events and reacts, publishing its own events in turn — very decoupled, but the overall flow is spread across services and harder to trace.

Eventual consistency

Sagas mean your system passes through temporarily inconsistent states — the order exists but isn't confirmed yet, inventory is reserved but payment hasn't cleared. This is eventual consistency: given no new changes, all services will converge to a consistent state, but not instantly. That's a real shift from the strong, immediate consistency of a single database.

Avoiding the Distributed Monolith

You met this villain in the last lesson; here's how communication choices create it. A distributed monolith is a set of services that look independent but are wired together so tightly they must move as one. Communication is the usual culprit: every user action fans out into a long chain of synchronous calls, so services can't deploy, scale, or fail independently.

In this chain, one request touches five services synchronously. The latencies add up. If any single service is slow, the whole request is slow. If any is down, the whole request fails. And because a change often ripples through several links, you end up coordinating releases — the exact thing microservices were supposed to eliminate. You've paid for a distributed system and gotten a monolith's coupling on top.

The cures are the patterns from this lesson, applied on purpose:

• Prefer asynchronous events for anything that doesn't need an immediate answer — it breaks the availability coupling of long synchronous chains.
• Give each service its own data so callers don't reach across boundaries.
• Let services keep local read-copies of what they frequently need (populated by events) instead of calling out on every request.
• Wrap every synchronous call in timeouts, circuit breakers, and fallbacks so a failure stays contained.
• Draw boundaries so that the common operations stay within one service rather than fanning out.

Practice & Quiz

Best Practices & Pitfalls

✅ Do

• Set a timeout on every network call — no exceptions
• Prefer asynchronous events for anything that doesn't need an immediate answer; it's what actually decouples services
• Retry only idempotent operations, with exponential backoff and jitter, and a bounded number of attempts
• Wrap synchronous calls in circuit breakers and provide a sensible fallback
• Make event consumers idempotent and configure dead-letter queues for messages that keep failing
• Use an idempotency key for non-idempotent operations like payments before retrying them
• Version your APIs and event schemas so you can evolve without breaking consumers

❌ Don't

• Make everything a synchronous call — long chains recreate the distributed monolith
• Retry immediately in a tight loop; you'll turn a blip into a retry storm
• Hardcode host addresses instead of using service discovery / DNS
• Assume a message is delivered exactly once — design for at-least-once
• Reach for a distributed transaction across services; use a saga with compensations instead
• Demand strong consistency everywhere when eventual consistency would do

Summary

📚 Additional Resources

• microservices.io — Messaging (asynchronous communication)
• microservices.io — Circuit Breaker pattern
• microservices.io — Saga pattern
• Martin Fowler — Microservice Trade-Offs
• Martin Fowler — CircuitBreaker

🚀 What's Next?

You now know how services talk to each other and stay resilient while doing it. But clients — browsers and mobile apps — shouldn't have to know about your dozens of services, their addresses, or their auth schemes. Next up: API Gateways — the single front door that routes, authenticates, rate-limits, aggregates, and shields your services from the outside world, plus the Backend-for-Frontend pattern and the pitfalls of a gateway that grows into a god object.