☁️ Cloud Storage Integration

Why Object Storage?

A traditional filesystem organizes data in a tree of folders. Object storage throws the tree away: every file (an "object") lives in a flat namespace inside a "bucket," addressed by a unique key. That simplification is what makes it scale to trillions of objects. You never think about disks, partitions, or free space — you just put objects and get them back.

Think of your local disk as a filing cabinet in your office: fast to reach, but finite, and gone if the building burns down. Object storage is a bank vault with unlimited safe-deposit boxes, mirrored across several cities. You trade a little latency for durability, capacity, and the ability to run many app servers that all share the same files.

Amazon S3 is the most widely used implementation, and because so many other providers (Cloudflare R2, DigitalOcean Spaces, Backblaze B2, MinIO) copy its API, the code you learn here transfers to all of them by changing one endpoint line. We'll focus on S3 with the official AWS SDK v3.

Setting Up the S3 Client

The AWS SDK v3 is modular — instead of pulling in one giant aws-sdk package, you install only the clients and helpers you use. For S3 that means two small packages.

// npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

The SDK v3 uses a command pattern: you create a client once, then send it typed command objects like PutObjectCommand. This keeps each operation explicit and tree-shakeable.

const { S3Client } = require('@aws-sdk/client-s3');

// Create ONE client and reuse it across requests.
const s3 = new S3Client({
  region: process.env.AWS_REGION, // e.g. "us-east-1"
  // In production, OMIT credentials entirely — the SDK reads them from the
  // environment, an IAM role, or the AWS credentials file automatically.
  // Only pass them explicitly for local development:
  credentials: process.env.AWS_ACCESS_KEY_ID
    ? {
        accessKeyId: process.env.AWS_ACCESS_KEY_ID,
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
      }
    : undefined
});

const BUCKET = process.env.AWS_S3_BUCKET;

module.exports = { s3, BUCKET };

Uploading & Reading Objects

Combine what you built last lesson — Multer memory storage with validation — with a PutObjectCommand. The buffer goes straight from the request to the bucket; your server's disk is never touched.

const express = require('express');
const multer = require('multer');
const crypto = require('crypto');
const { PutObjectCommand } = require('@aws-sdk/client-s3');
const { s3, BUCKET } = require('./s3');

const app = express();
const upload = multer({
  storage: multer.memoryStorage(),
  limits: { fileSize: 10 * 1024 * 1024 } // 10 MB
});

app.post('/api/upload', upload.single('file'), async (req, res) => {
  if (!req.file) return res.status(400).json({ error: 'No file uploaded' });

  // Random, unguessable object key — same reasoning as safe filenames on disk.
  const key = `uploads/${crypto.randomUUID()}`;

  await s3.send(new PutObjectCommand({
    Bucket: BUCKET,
    Key: key,
    Body: req.file.buffer,
    ContentType: req.file.mimetype,        // so browsers render it correctly
    ServerSideEncryption: 'AES256'         // encrypt at rest — free and easy
  }));

  // Persist the key (not a URL) in your database; URLs are generated on demand.
  res.status(201).json({ ok: true, key });
});

To read the object back on the server — say, to re-process it — you send a GetObjectCommand. Its Body is a readable stream you can pipe.

const { GetObjectCommand } = require('@aws-sdk/client-s3');

async function streamObjectToResponse(key, res) {
  const { Body, ContentType } = await s3.send(
    new GetObjectCommand({ Bucket: BUCKET, Key: key })
  );
  res.setHeader('Content-Type', ContentType);
  // Body is a Node Readable stream in the SDK v3 — pipe it, don't buffer it.
  Body.pipe(res);
}

Presigned URLs

Best practice is to keep your bucket private — block all public access. But then how does a browser display a private image? The answer is a presigned URL: a normal S3 URL with a cryptographic signature and an expiry baked into the query string. Anyone holding the URL can perform exactly one operation (read this one object) for exactly as long as it's valid, then it stops working.

It's like a hotel key card. The front desk (your server, which holds the credentials) programs a card that opens one room until checkout. The guest never learns the master key; the card is useless tomorrow.

const { GetObjectCommand } = require('@aws-sdk/client-s3');
const { getSignedUrl } = require('@aws-sdk/s3-request-presigner');

// Generate a read URL that self-destructs after 5 minutes.
async function getReadUrl(key) {
  const command = new GetObjectCommand({ Bucket: BUCKET, Key: key });
  return getSignedUrl(s3, command, { expiresIn: 300 }); // seconds
}

app.get('/api/files/:id/url', async (req, res) => {
  // In real code, look up the key by id AND check the user may access it.
  const key = `uploads/${req.params.id}`;
  const url = await getReadUrl(key);
  res.json({ url }); // the browser puts this straight into  or 
});

• AWS SDK for JavaScript v3 — S3 client reference
• Amazon S3 — Working with presigned URLs
• Amazon S3 — Configuring CORS
• MDN — Cross-Origin Resource Sharing (CORS)

🚀 What's Next?

Your files now live safely in the cloud — but they're the raw bytes the user uploaded, often huge and in the wrong format. Next we transform them on the way in: Image Processing with sharp, where you'll resize, convert to WebP, generate thumbnails, and strip metadata before storing.