Load Testing with Artillery

Creating and Running Effective Load Tests for Web Applications

Introduction to Artillery

In our previous lecture, we explored various performance testing tools available in 2025. Today, we'll dive deep into Artillery, a modern, developer-friendly load testing platform that has gained significant popularity for its ease of use and powerful capabilities.

Artillery is a Node.js-based load testing tool designed to help developers and QA engineers test the performance and reliability of their applications under various load conditions. It supports testing HTTP APIs, WebSockets, Socket.IO, gRPC, and more, making it versatile for modern web applications.

Artillery in the Testing Ecosystem

flowchart TD A[Testing Ecosystem] --> B[Functional Testing] A --> C[Performance Testing] A --> D[Security Testing] C --> E[Load Testing] C --> F[Stress Testing] C --> G[Scalability Testing] E --> H[Artillery] H --> I[HTTP APIs] H --> J[WebSockets] H --> K[Socket.IO] H --> L[gRPC] H --> M[Browser Testing]

Why Choose Artillery?

Artillery has gained popularity among developers and QA teams for several compelling reasons:

The Orchestra Analogy

Think of Artillery as the conductor of an orchestra. Just as a conductor directs various instruments to play in harmony, Artillery orchestrates virtual users to interact with your application in a coordinated manner. The test script is your musical score, detailing when and how each section (user) should perform. The resulting performance metrics are like the audience's reaction, telling you whether your application performed beautifully or hit some sour notes under pressure.

Installing Artillery

Before we can start load testing, we need to install Artillery. Since it's a Node.js package, we'll use npm for installation.

Prerequisites

Installation Options

You can install Artillery either globally or as a project dependency.

Global Installation (For CLI access anywhere)

npm install -g artillery
                
Project Installation (Recommended for teams and CI/CD)

# Create a new project if needed
mkdir load-testing && cd load-testing
npm init -y

# Install Artillery as a dev dependency
npm install --save-dev artillery
                

Verifying Installation

After installation, verify that Artillery is working correctly:


# If installed globally
artillery -V

# If installed locally
npx artillery -V
                

You should see output similar to:


___ __ _ ____ _____/ | _____/ /_(_) / /__ _______ __ ___ /____/ /| | / ___/ __/ / / / _ \/ ___/ / / /____/ /____/ ___ |/ / / /_/ / / / __/ / / /_/ /____/ /_/ |_/_/ \__/_/_/_/\___/_/ \__ / /____/ 

VERSION INFO:
Artillery: 2.0.22
Node.js: v18.16.0
OS: darwin
                

Quick Start: Your First Artillery Test

Let's start with a simple test to get a feel for Artillery. The quick command allows you to run basic load tests without creating test scripts.


# Basic syntax
artillery quick --count 10 --num 20 https://example.com

# Breaking it down:
# --count 10: Create 10 virtual users
# --num 20: Each user makes 20 requests
# https://example.com: The target URL to test
                

While the quick command is useful for simple tests, real-world load testing requires more structured and comprehensive test scenarios. Let's dive into creating proper Artillery test scripts.

Creating Artillery Test Scripts

Artillery test scripts are typically written in YAML (though JavaScript is also supported) and consist of two main sections:

Basic Structure of a Test Script


# basic-test.yml
config:
  target: "https://api.example.com"
  phases:
    - duration: 60
      arrivalRate: 5
      name: "Warm up phase"
    - duration: 120
      arrivalRate: 10
      name: "Sustained load phase"
  
scenarios:
  - name: "API testing scenario"
    flow:
      - get:
          url: "/users"
      - think: 2
      - get:
          url: "/products"
                

Understanding the Config Section

The config section sets up the test environment and load parameters:

Load Phases in Detail

Load phases control how Artillery generates traffic to your application:

Common Load Testing Patterns

graph LR A[Load Patterns] --> B[Constant Load] A --> C[Ramp Up] A --> D[Spike] A --> E[Step Load] B --> F["duration: 300
arrivalRate: 10"] C --> G["duration: 300
arrivalRate: 5
rampTo: 50"] D --> H["Multiple phases with
rapid increase then decrease"] E --> I["Multiple phases with
increasing arrivalRate"]
Example Load Patterns

# Constant load - 10 new users per second for 5 minutes
phases:
  - duration: 300
    arrivalRate: 10
    name: "Constant load"

# Ramp up - start with 5 new users per second, increase to 50 over 5 minutes
phases:
  - duration: 300
    arrivalRate: 5
    rampTo: 50
    name: "Ramp up load"

# Spike test - quickly ramp up to high load, then back down
phases:
  - duration: 60
    arrivalRate: 5
    name: "Warm up"
  - duration: 60
    arrivalRate: 50
    name: "Spike"
  - duration: 120
    arrivalRate: 5
    name: "Recovery"
                

Understanding the Scenarios Section

The scenarios section defines the user journeys to simulate:

Example Scenario with Multiple Steps

scenarios:
  - name: "User browsing products"
    flow:
      # Step 1: Visit homepage
      - get:
          url: "/"
      
      # Step 2: Wait a bit (simulating user reading)
      - think: 3
      
      # Step 3: Search for products
      - get:
          url: "/search?q=laptop"
          headers:
            Accept: "application/json"
      
      # Step 4: View a specific product
      - get:
          url: "/products/{{ $randomNumber(1, 100) }}"
      
      # Step 5: Add to cart (POST request with payload)
      - post:
          url: "/cart"
          json:
            productId: "{{ $randomNumber(1, 100) }}"
            quantity: 1
      
      # Step 6: Wait before checkout
      - think: 5
      
      # Step 7: Proceed to checkout
      - get:
          url: "/checkout"
                

Working with Dynamic Data

Real-world testing often requires dynamic data, such as random values, capturing values from responses, or using external data sets. Artillery provides several methods for handling dynamic data.

Built-in Variables and Functions

Artillery includes several built-in variables and functions you can use in your test scripts:


# Random number generation
- get:
    url: "/products/{{ $randomNumber(1, 100) }}"

# Random string
- post:
    url: "/users"
    json:
      username: "user_{{ $randomString(8) }}"
      email: "user_{{ $randomString(8) }}@example.com"

# Random item from an array
- get:
    url: "/search?category={{ $randomItem(['electronics', 'clothing', 'books', 'home']) }}"

# Current timestamp
- post:
    url: "/events"
    json:
      timestamp: "{{ $timestamp }}"
                

Capturing and Reusing Values

You can capture values from responses and reuse them in subsequent requests:


# Capture a value from response
- get:
    url: "/products"
    capture:
      - json: "$.products[0].id"
        as: "productId"

# Use the captured value in a subsequent request
- get:
    url: "/products/{{ productId }}/details"
                

Using Data from External Files

For more complex testing, you can load test data from external CSV or JSON files:


# In your test script
config:
  target: "https://api.example.com"
  phases:
    - duration: 60
      arrivalRate: 10
  payload:
    path: "users.csv"
    fields:
      - "username"
      - "password"

scenarios:
  - name: "Login with different users"
    flow:
      - post:
          url: "/login"
          json:
            username: "{{ username }}"
            password: "{{ password }}"
                

With a CSV file like this:


username,password
user1,pass1
user2,pass2
user3,pass3
...
                

Custom JavaScript Functions

Sometimes built-in functions aren't enough for complex scenarios. Artillery allows you to define custom JavaScript functions for more advanced test logic.

Processor Files

Create a JavaScript file with your custom functions:

custom-functions.js

// Generate a random valid credit card number
function generateCreditCardNumber() {
  // Simple example - not a real credit card algorithm
  const prefix = '4111'; // Visa prefix
  const length = 16;
  let number = prefix;
  
  for (let i = prefix.length; i < length - 1; i++) {
    number += Math.floor(Math.random() * 10);
  }
  
  // Add a valid checksum digit (simplified)
  number += '0';
  
  return number;
}

// Generate a formatted address
function generateAddress() {
  const streets = ['Main St', 'Oak Ave', 'Maple Rd', 'Broadway'];
  const cities = ['New York', 'Los Angeles', 'Chicago', 'Houston'];
  const states = ['NY', 'CA', 'IL', 'TX'];
  
  const streetNumber = Math.floor(Math.random() * 1000) + 1;
  const street = streets[Math.floor(Math.random() * streets.length)];
  const city = cities[Math.floor(Math.random() * cities.length)];
  const state = states[Math.floor(Math.random() * states.length)];
  const zip = Math.floor(Math.random() * 90000) + 10000;
  
  return {
    street: `${streetNumber} ${street}`,
    city,
    state,
    zip: zip.toString()
  };
}

// Make function available to Artillery
module.exports = {
  generateCreditCardNumber,
  generateAddress
};
                

Then reference this file in your test script:

Using Custom Functions in Test Script

config:
  target: "https://api.example.com"
  phases:
    - duration: 60
      arrivalRate: 5
  processor: "./custom-functions.js"

scenarios:
  - name: "Checkout process"
    flow:
      # Add product to cart
      - post:
          url: "/cart"
          json:
            productId: "{{ $randomNumber(1, 100) }}"
            quantity: 1
      
      # Checkout with custom data
      - post:
          url: "/checkout"
          json:
            creditCard: "{{ generateCreditCardNumber() }}"
            shippingAddress: "{{ generateAddress() }}"
                

Hooks for Request Processing

Artillery provides hooks to execute custom logic at different stages of request processing:

hooks.js

// Add custom authentication header before each request
function addAuthHeader(requestParams, context, ee, next) {
  // Generate or retrieve auth token
  const authToken = `token-${Date.now()}`;
  
  // Add to headers
  requestParams.headers = requestParams.headers || {};
  requestParams.headers['Authorization'] = `Bearer ${authToken}`;
  
  return next();
}

// Process response data after request
function processResponse(requestParams, response, context, ee, next) {
  // Extract useful information from response
  if (response.statusCode === 200 && response.body) {
    const body = JSON.parse(response.body);
    
    // Store something from the response for later use
    if (body.sessionId) {
      context.vars.sessionId = body.sessionId;
    }
  }
  
  return next();
}

module.exports = {
  addAuthHeader,
  processResponse
};
                

And in your test script:


config:
  target: "https://api.example.com"
  phases:
    - duration: 60
      arrivalRate: 5
  processor: "./hooks.js"

scenarios:
  - name: "API with auth"
    flow:
      - get:
          url: "/secure-resource"
          beforeRequest: "addAuthHeader"
          afterResponse: "processResponse"
                

Running Artillery Tests

Once you've created your test script, it's time to run the test and analyze the results.

Basic Test Execution


# Run a test script
artillery run my-test.yml

# Run with verbose output
artillery run --verbose my-test.yml

# Run with a specific environment
artillery run --environment production my-test.yml
                

Generating Reports

Artillery can generate detailed HTML reports from your test runs:


# Generate a JSON report file
artillery run --output report.json my-test.yml

# Convert JSON report to HTML
artillery report report.json
                
Sample Artillery HTML Report
Artillery Test Report Test run: 2025-05-05 12:30:45 Summary Virtual users created: 500 Requests completed: 4,500 Test duration: 300s Response Time Distribution 0-100ms 101-200ms 201-300ms 301-400ms 401+ms Request Rate (per second)

A visual representation of what an Artillery HTML report might look like

Understanding Test Results

Artillery reports include several key metrics to help you evaluate your application's performance:

Performance Metrics Interpretation

When analyzing load test results, pay special attention to:

  • p95 and p99 Response Times: These show performance for the slowest 5% and 1% of requests, respectively. High values here often indicate performance problems that affect some users.
  • Error Rates: Error rates above 1% typically indicate problems that need immediate attention.
  • Response Time Stability: Look for sudden spikes in response times, which may indicate memory leaks or resource exhaustion.

Advanced Artillery Features

Testing with Multiple Environments

Artillery allows you to define multiple environments in a single test script:


config:
  environments:
    development:
      target: "http://localhost:3000"
      phases:
        - duration: 60
          arrivalRate: 2
    
    staging:
      target: "https://staging.example.com"
      phases:
        - duration: 120
          arrivalRate: 5
    
    production:
      target: "https://api.example.com"
      phases:
        - duration: 300
          arrivalRate: 10
          rampTo: 50

# To run with a specific environment:
# artillery run --environment production my-test.yml
                

Artillery Plugins

Artillery's functionality can be extended with plugins:


# Install a plugin
npm install -D artillery-plugin-expect

# Use the plugin in your test script
config:
  target: "https://api.example.com"
  phases:
    - duration: 60
      arrivalRate: 5
  plugins:
    expect: {}  # Load the 'expect' plugin for response validation

scenarios:
  - name: "API with validation"
    flow:
      - get:
          url: "/users/1"
          expect:
            - statusCode: 200
            - contentType: "application/json"
            - hasProperty: "data.email"
                

Browser-Based Load Testing with Playwright

Artillery can leverage Playwright for browser-based load testing:


# Install the Playwright engine
npm install -D @artillery/playwright

# Create a Playwright test script (playwright-test.yml)
config:
  target: "https://example.com"
  phases:
    - duration: 60
      arrivalRate: 2
  engines:
    playwright: {}

scenarios:
  - engine: playwright
    testFunction: browserTest

# Create the test function (browser-test.js)
async function browserTest(page) {
  // Navigate to the home page
  await page.goto('https://example.com');
  
  // Click the login button
  await page.click('#login-button');
  
  // Fill the login form
  await page.fill('#username', 'testuser');
  await page.fill('#password', 'password123');
  
  // Submit the form
  await page.click('#submit-button');
  
  // Wait for dashboard to load
  await page.waitForSelector('#dashboard');
  
  // Perform actions on the dashboard
  await page.click('#profile-tab');
  
  // Verify element is visible
  await page.waitForSelector('#profile-info');
}

module.exports = { browserTest };
                

Distributed Load Testing

For large-scale tests, Artillery supports distributed testing using multiple machines:

Distributed Testing Architecture

graph TD A[Test Controller] --> B[Worker 1] A --> C[Worker 2] A --> D[Worker 3] A --> E[Worker N] B --> F[Target Application] C --> F D --> F E --> F

Artillery provides multiple options for distributed testing, including AWS and Azure-based solutions, to generate larger loads than possible from a single machine.

Integrating with CI/CD Pipelines

Integrating Artillery into your CI/CD pipeline enables automated performance testing as part of your development workflow.

Setting Performance Thresholds

Define performance thresholds to automatically fail the build if performance degrades:


config:
  target: "https://api.example.com"
  phases:
    - duration: 60
      arrivalRate: 5
  thresholds:
    # 95th percentile response time must be below 200ms
    http.response_time.p95: 200
    # 99th percentile response time must be below 500ms
    http.response_time.p99: 500
    # Less than 1% of requests should fail
    http.codes.404: 1
    http.codes.500: 1
                

GitHub Actions Integration

Example GitHub Actions workflow for performance testing:


# .github/workflows/performance.yml
name: Performance Testing

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  performance-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Start test server
        run: npm run start:test-server &
        env:
          PORT: 3000
      
      - name: Wait for server
        run: npx wait-on http://localhost:3000
      
      - name: Run performance tests
        run: npx artillery run --output report.json performance-test.yml
      
      - name: Generate HTML report
        run: npx artillery report report.json
      
      - name: Upload test report
        uses: actions/upload-artifact@v3
        with:
          name: artillery-report
          path: |
            report.json
            report.html
                

Best Practices for Artillery Load Testing

Test Design

Execution

Analysis

Real-World Case Study: E-commerce Company

An e-commerce company implemented Artillery load testing in their CI/CD pipeline with the following results:

  • Identified a database query that degraded performance under load before deployment
  • Discovered that their CDN configuration was causing delays for certain regions
  • Verified that their system could handle 3x the expected Black Friday traffic
  • Reduced average response time by 45% by fixing issues identified in load tests
  • Implemented automated performance regression testing for all major releases

By integrating Artillery into their development workflow, they prevented several potential outages and improved customer experience significantly.

Practical Activity: Building a Complete Load Test

Let's create a comprehensive load test for a sample e-commerce API. This activity will walk you through creating, running, and analyzing a realistic load test.

Setup

  1. Create a new directory for your load test project:
    mkdir ecommerce-load-test && cd ecommerce-load-test
  2. Initialize a new npm project and install Artillery:
    npm init -y
    npm install --save-dev artillery
  3. Create a basic test script file:
    touch ecommerce-test.yml

Creating the Test Script

Open ecommerce-test.yml in your editor and add the following content:


config:
  target: "https://example-ecommerce-api.com"  # Replace with your target API
  phases:
    - duration: 60
      arrivalRate: 5
      name: "Warm up phase"
    - duration: 120
      arrivalRate: 10
      rampTo: 30
      name: "Ramp up phase"
    - duration: 300
      arrivalRate: 30
      name: "Sustained load phase"
  variables:
    productIds:
      - "1001"
      - "1002"
      - "1003"
      - "1004"
      - "1005"
    categories:
      - "electronics"
      - "clothing"
      - "books"
      - "home"
      - "sports"

scenarios:
  - name: "Browse and purchase flow"
    weight: 70
    flow:
      # Step 1: Visit homepage
      - get:
          url: "/"
          headers:
            Accept: "application/json"
      
      # Step 2: Browse a category
      - get:
          url: "/categories/{{ $randomItem(categories) }}"
          headers:
            Accept: "application/json"
      
      # Step 3: Search for products
      - get:
          url: "/search?q=popular&sort=price"
          headers:
            Accept: "application/json"
          capture:
            - json: "$.products[0].id"
              as: "firstProductId"
      
      # Step 4: View a product (either from search results or random)
      - think: 2
      - get:
          url: "/products/{{ firstProductId || $randomItem(productIds) }}"
          headers:
            Accept: "application/json"
      
      # Step 5: Add to cart
      - think: 1
      - post:
          url: "/cart"
          headers:
            Content-Type: "application/json"
            Accept: "application/json"
          json:
            productId: "{{ firstProductId || $randomItem(productIds) }}"
            quantity: "{{ $randomNumber(1, 3) }}"
      
      # Step 6: View cart
      - think: 2
      - get:
          url: "/cart"
          headers:
            Accept: "application/json"
      
      # Step 7: Checkout
      - think: 3
      - post:
          url: "/checkout"
          headers:
            Content-Type: "application/json"
            Accept: "application/json"
          json:
            payment:
              type: "credit_card"
              cardNumber: "4111111111111111"
              expiryMonth: "{{ $randomNumber(1, 12) }}"
              expiryYear: "{{ $randomNumber(2025, 2030) }}"
              cvv: "{{ $randomNumber(100, 999) }}"
            shipping:
              name: "Test User"
              address: "123 Test St"
              city: "Test City"
              zipCode: "12345"
              country: "US"
  
  - name: "Browse only flow"
    weight: 30
    flow:
      # Step 1: Visit homepage
      - get:
          url: "/"
          headers:
            Accept: "application/json"
      
      # Step 2: Browse categories
      - get:
          url: "/categories"
          headers:
            Accept: "application/json"
      
      # Step 3: View a category
      - think: 1
      - get:
          url: "/categories/{{ $randomItem(categories) }}"
          headers:
            Accept: "application/json"
      
      # Step 4: View multiple products
      - loop:
          - think: 2
          - get:
              url: "/products/{{ $randomItem(productIds) }}"
              headers:
                Accept: "application/json"
        count: 3
                

Adding Custom Functions

Create a file named custom-functions.js with helper functions:


// Generate a random address
function generateAddress() {
  const streets = ['Main St', 'Oak Ave', 'Pine Rd', 'Maple Ln', 'Cedar Blvd'];
  const cities = ['New York', 'Los Angeles', 'Chicago', 'Houston', 'Phoenix'];
  const states = ['NY', 'CA', 'IL', 'TX', 'AZ'];
  
  return {
    street: `${Math.floor(Math.random() * 1000) + 1} ${streets[Math.floor(Math.random() * streets.length)]}`,
    city: cities[Math.floor(Math.random() * cities.length)],
    state: states[Math.floor(Math.random() * states.length)],
    zipCode: Math.floor(Math.random() * 90000) + 10000
  };
}

// Log request for debugging
function logRequest(requestParams, context, ee, next) {
  console.log(`Making request to: ${requestParams.url}`);
  return next();
}

// Process and validate response
function validateResponse(requestParams, response, context, ee, next) {
  const url = requestParams.url;
  
  if (response.statusCode >= 400) {
    console.error(`Error in request to ${url}: ${response.statusCode}`);
  }
  
  return next();
}

module.exports = {
  generateAddress,
  logRequest,
  validateResponse
};
                

Update your test script to use these functions:


config:
  target: "https://example-ecommerce-api.com"
  phases:
    # ... (phases remain the same)
  processor: "./custom-functions.js"
  # ... (rest of config remains the same)

scenarios:
  - name: "Browse and purchase flow"
    # ... (mostly the same as before)
    flow:
      # ... (earlier steps remain the same)
      
      # Updated Step 7: Checkout with custom function
      - think: 3
      - function: "generateAddress"
      - post:
          url: "/checkout"
          headers:
            Content-Type: "application/json"
            Accept: "application/json"
          beforeRequest: "logRequest"
          afterResponse: "validateResponse"
          json:
            payment:
              type: "credit_card"
              cardNumber: "4111111111111111"
              expiryMonth: "{{ $randomNumber(1, 12) }}"
              expiryYear: "{{ $randomNumber(2025, 2030) }}"
              cvv: "{{ $randomNumber(100, 999) }}"
            shipping: "{{ $generateAddress }}"
                

Running the Test

Now run your test and generate a report:


# Ensure your target API is accessible
# Then run the test
npx artillery run --output report.json ecommerce-test.yml

# Generate HTML report
npx artillery report report.json
                

Analysis and Optimization

After running the test, analyze the results and consider the following questions:

  1. What is the p95 response time for each endpoint?
  2. Are there any specific endpoints that are consistently slower?
  3. Did any requests fail? If so, why?
  4. How does the system perform under peak load compared to baseline?
  5. Are there any performance bottlenecks that need to be addressed?

Based on your findings, optimize your application and run the test again to verify improvements.

Key Takeaways

Further Resources

Homework Assignment

For your homework, you'll create and execute a comprehensive load test for a web application of your choice.

Assignment Details

  1. Choose a target application:
    • Your own application
    • A public API (ensure you have permission to load test it)
    • A local demo application we'll provide
  2. Create an Artillery test script that:
    • Tests at least 5 different endpoints or user actions
    • Includes at least 2 different scenarios with different weights
    • Implements custom JavaScript functions for dynamic data
    • Uses variable capture and reuse between requests
    • Defines realistic load phases (warm-up, ramp-up, sustained load)
  3. Execute your test and generate a report
  4. Analyze the results and document:
    • Overall performance metrics
    • Performance bottlenecks identified
    • Recommendations for improvement
    • How the application behaved under different load levels
  5. Optimize one aspect of the application based on your findings
  6. Re-run the test to verify improvements

Submission Requirements

Due Date

Submit your completed assignment before our next class session.