Introduction to Artillery
In our previous lecture, we explored various performance testing tools available in 2025. Today, we'll dive deep into Artillery, a modern, developer-friendly load testing platform that has gained significant popularity for its ease of use and powerful capabilities.
Artillery is a Node.js-based load testing tool designed to help developers and QA engineers test the performance and reliability of their applications under various load conditions. It supports testing HTTP APIs, WebSockets, Socket.IO, gRPC, and more, making it versatile for modern web applications.
Artillery in the Testing Ecosystem
Why Choose Artillery?
Artillery has gained popularity among developers and QA teams for several compelling reasons:
- Developer-Friendly: Uses YAML or JavaScript for test scripts, making it accessible to both developers and non-developers
- Protocol Support: Tests various protocols including HTTP, WebSockets, Socket.IO, and gRPC
- Realistic Scenarios: Creates complex, multi-step user journeys that mirror real-world usage patterns
- Extensibility: Extends functionality through plugins and custom JavaScript functions
- CI/CD Integration: Integrates seamlessly with continuous integration workflows
- Comprehensive Reporting: Provides detailed performance metrics and visualizations
- Resource Efficient: Generates substantial load with minimal hardware resources
- Browser Testing: Supports browser-based testing using Playwright for end-to-end scenarios
The Orchestra Analogy
Think of Artillery as the conductor of an orchestra. Just as a conductor directs various instruments to play in harmony, Artillery orchestrates virtual users to interact with your application in a coordinated manner. The test script is your musical score, detailing when and how each section (user) should perform. The resulting performance metrics are like the audience's reaction, telling you whether your application performed beautifully or hit some sour notes under pressure.
Installing Artillery
Before we can start load testing, we need to install Artillery. Since it's a Node.js package, we'll use npm for installation.
Prerequisites
- Node.js (version 14 or later recommended)
- npm (comes with Node.js)
- A target application to test (we'll use a sample API for demonstrations)
Installation Options
You can install Artillery either globally or as a project dependency.
Global Installation (For CLI access anywhere)
npm install -g artillery
Project Installation (Recommended for teams and CI/CD)
# Create a new project if needed
mkdir load-testing && cd load-testing
npm init -y
# Install Artillery as a dev dependency
npm install --save-dev artillery
Verifying Installation
After installation, verify that Artillery is working correctly:
# If installed globally
artillery -V
# If installed locally
npx artillery -V
You should see output similar to:
___ __ _ ____ _____/ | _____/ /_(_) / /__ _______ __ ___ /____/ /| | / ___/ __/ / / / _ \/ ___/ / / /____/ /____/ ___ |/ / / /_/ / / / __/ / / /_/ /____/ /_/ |_/_/ \__/_/_/_/\___/_/ \__ / /____/
VERSION INFO:
Artillery: 2.0.22
Node.js: v18.16.0
OS: darwin
Quick Start: Your First Artillery Test
Let's start with a simple test to get a feel for Artillery. The quick command allows you to run basic load tests without creating test scripts.
# Basic syntax
artillery quick --count 10 --num 20 https://example.com
# Breaking it down:
# --count 10: Create 10 virtual users
# --num 20: Each user makes 20 requests
# https://example.com: The target URL to test
While the quick command is useful for simple tests, real-world load testing requires more structured and comprehensive test scenarios. Let's dive into creating proper Artillery test scripts.
Creating Artillery Test Scripts
Artillery test scripts are typically written in YAML (though JavaScript is also supported) and consist of two main sections:
- config: Defines the test configuration, including target URLs and load phases
- scenarios: Describes the user journeys or scenarios to test
Basic Structure of a Test Script
# basic-test.yml
config:
target: "https://api.example.com"
phases:
- duration: 60
arrivalRate: 5
name: "Warm up phase"
- duration: 120
arrivalRate: 10
name: "Sustained load phase"
scenarios:
- name: "API testing scenario"
flow:
- get:
url: "/users"
- think: 2
- get:
url: "/products"
Understanding the Config Section
The config section sets up the test environment and load parameters:
- target: The base URL of the application under test
- phases: Defines how virtual users are created over time
- plugins: Optional plugins to extend Artillery's functionality
- http: HTTP-specific settings like timeouts and headers
Load Phases in Detail
Load phases control how Artillery generates traffic to your application:
Common Load Testing Patterns
arrivalRate: 10"] C --> G["duration: 300
arrivalRate: 5
rampTo: 50"] D --> H["Multiple phases with
rapid increase then decrease"] E --> I["Multiple phases with
increasing arrivalRate"]
Example Load Patterns
# Constant load - 10 new users per second for 5 minutes
phases:
- duration: 300
arrivalRate: 10
name: "Constant load"
# Ramp up - start with 5 new users per second, increase to 50 over 5 minutes
phases:
- duration: 300
arrivalRate: 5
rampTo: 50
name: "Ramp up load"
# Spike test - quickly ramp up to high load, then back down
phases:
- duration: 60
arrivalRate: 5
name: "Warm up"
- duration: 60
arrivalRate: 50
name: "Spike"
- duration: 120
arrivalRate: 5
name: "Recovery"
Understanding the Scenarios Section
The scenarios section defines the user journeys to simulate:
- name: A descriptive name for the scenario
- weight: Optional weighting to control scenario selection probability
- flow: A sequence of steps that define the user journey
Example Scenario with Multiple Steps
scenarios:
- name: "User browsing products"
flow:
# Step 1: Visit homepage
- get:
url: "/"
# Step 2: Wait a bit (simulating user reading)
- think: 3
# Step 3: Search for products
- get:
url: "/search?q=laptop"
headers:
Accept: "application/json"
# Step 4: View a specific product
- get:
url: "/products/{{ $randomNumber(1, 100) }}"
# Step 5: Add to cart (POST request with payload)
- post:
url: "/cart"
json:
productId: "{{ $randomNumber(1, 100) }}"
quantity: 1
# Step 6: Wait before checkout
- think: 5
# Step 7: Proceed to checkout
- get:
url: "/checkout"
Working with Dynamic Data
Real-world testing often requires dynamic data, such as random values, capturing values from responses, or using external data sets. Artillery provides several methods for handling dynamic data.
Built-in Variables and Functions
Artillery includes several built-in variables and functions you can use in your test scripts:
# Random number generation
- get:
url: "/products/{{ $randomNumber(1, 100) }}"
# Random string
- post:
url: "/users"
json:
username: "user_{{ $randomString(8) }}"
email: "user_{{ $randomString(8) }}@example.com"
# Random item from an array
- get:
url: "/search?category={{ $randomItem(['electronics', 'clothing', 'books', 'home']) }}"
# Current timestamp
- post:
url: "/events"
json:
timestamp: "{{ $timestamp }}"
Capturing and Reusing Values
You can capture values from responses and reuse them in subsequent requests:
# Capture a value from response
- get:
url: "/products"
capture:
- json: "$.products[0].id"
as: "productId"
# Use the captured value in a subsequent request
- get:
url: "/products/{{ productId }}/details"
Using Data from External Files
For more complex testing, you can load test data from external CSV or JSON files:
# In your test script
config:
target: "https://api.example.com"
phases:
- duration: 60
arrivalRate: 10
payload:
path: "users.csv"
fields:
- "username"
- "password"
scenarios:
- name: "Login with different users"
flow:
- post:
url: "/login"
json:
username: "{{ username }}"
password: "{{ password }}"
With a CSV file like this:
username,password
user1,pass1
user2,pass2
user3,pass3
...
Custom JavaScript Functions
Sometimes built-in functions aren't enough for complex scenarios. Artillery allows you to define custom JavaScript functions for more advanced test logic.
Processor Files
Create a JavaScript file with your custom functions:
custom-functions.js
// Generate a random valid credit card number
function generateCreditCardNumber() {
// Simple example - not a real credit card algorithm
const prefix = '4111'; // Visa prefix
const length = 16;
let number = prefix;
for (let i = prefix.length; i < length - 1; i++) {
number += Math.floor(Math.random() * 10);
}
// Add a valid checksum digit (simplified)
number += '0';
return number;
}
// Generate a formatted address
function generateAddress() {
const streets = ['Main St', 'Oak Ave', 'Maple Rd', 'Broadway'];
const cities = ['New York', 'Los Angeles', 'Chicago', 'Houston'];
const states = ['NY', 'CA', 'IL', 'TX'];
const streetNumber = Math.floor(Math.random() * 1000) + 1;
const street = streets[Math.floor(Math.random() * streets.length)];
const city = cities[Math.floor(Math.random() * cities.length)];
const state = states[Math.floor(Math.random() * states.length)];
const zip = Math.floor(Math.random() * 90000) + 10000;
return {
street: `${streetNumber} ${street}`,
city,
state,
zip: zip.toString()
};
}
// Make function available to Artillery
module.exports = {
generateCreditCardNumber,
generateAddress
};
Then reference this file in your test script:
Using Custom Functions in Test Script
config:
target: "https://api.example.com"
phases:
- duration: 60
arrivalRate: 5
processor: "./custom-functions.js"
scenarios:
- name: "Checkout process"
flow:
# Add product to cart
- post:
url: "/cart"
json:
productId: "{{ $randomNumber(1, 100) }}"
quantity: 1
# Checkout with custom data
- post:
url: "/checkout"
json:
creditCard: "{{ generateCreditCardNumber() }}"
shippingAddress: "{{ generateAddress() }}"
Hooks for Request Processing
Artillery provides hooks to execute custom logic at different stages of request processing:
hooks.js
// Add custom authentication header before each request
function addAuthHeader(requestParams, context, ee, next) {
// Generate or retrieve auth token
const authToken = `token-${Date.now()}`;
// Add to headers
requestParams.headers = requestParams.headers || {};
requestParams.headers['Authorization'] = `Bearer ${authToken}`;
return next();
}
// Process response data after request
function processResponse(requestParams, response, context, ee, next) {
// Extract useful information from response
if (response.statusCode === 200 && response.body) {
const body = JSON.parse(response.body);
// Store something from the response for later use
if (body.sessionId) {
context.vars.sessionId = body.sessionId;
}
}
return next();
}
module.exports = {
addAuthHeader,
processResponse
};
And in your test script:
config:
target: "https://api.example.com"
phases:
- duration: 60
arrivalRate: 5
processor: "./hooks.js"
scenarios:
- name: "API with auth"
flow:
- get:
url: "/secure-resource"
beforeRequest: "addAuthHeader"
afterResponse: "processResponse"
Running Artillery Tests
Once you've created your test script, it's time to run the test and analyze the results.
Basic Test Execution
# Run a test script
artillery run my-test.yml
# Run with verbose output
artillery run --verbose my-test.yml
# Run with a specific environment
artillery run --environment production my-test.yml
Generating Reports
Artillery can generate detailed HTML reports from your test runs:
# Generate a JSON report file
artillery run --output report.json my-test.yml
# Convert JSON report to HTML
artillery report report.json
Sample Artillery HTML Report
A visual representation of what an Artillery HTML report might look like
Understanding Test Results
Artillery reports include several key metrics to help you evaluate your application's performance:
- Response Times: Min, max, median, and percentile (p95, p99) response times
- Request Rates: Requests per second processed by your application
- HTTP Codes: Distribution of HTTP response codes
- Errors: Number and types of errors encountered
- Scenarios: Successfully completed and failed scenarios
Performance Metrics Interpretation
When analyzing load test results, pay special attention to:
- p95 and p99 Response Times: These show performance for the slowest 5% and 1% of requests, respectively. High values here often indicate performance problems that affect some users.
- Error Rates: Error rates above 1% typically indicate problems that need immediate attention.
- Response Time Stability: Look for sudden spikes in response times, which may indicate memory leaks or resource exhaustion.
Advanced Artillery Features
Testing with Multiple Environments
Artillery allows you to define multiple environments in a single test script:
config:
environments:
development:
target: "http://localhost:3000"
phases:
- duration: 60
arrivalRate: 2
staging:
target: "https://staging.example.com"
phases:
- duration: 120
arrivalRate: 5
production:
target: "https://api.example.com"
phases:
- duration: 300
arrivalRate: 10
rampTo: 50
# To run with a specific environment:
# artillery run --environment production my-test.yml
Artillery Plugins
Artillery's functionality can be extended with plugins:
# Install a plugin
npm install -D artillery-plugin-expect
# Use the plugin in your test script
config:
target: "https://api.example.com"
phases:
- duration: 60
arrivalRate: 5
plugins:
expect: {} # Load the 'expect' plugin for response validation
scenarios:
- name: "API with validation"
flow:
- get:
url: "/users/1"
expect:
- statusCode: 200
- contentType: "application/json"
- hasProperty: "data.email"
Browser-Based Load Testing with Playwright
Artillery can leverage Playwright for browser-based load testing:
# Install the Playwright engine
npm install -D @artillery/playwright
# Create a Playwright test script (playwright-test.yml)
config:
target: "https://example.com"
phases:
- duration: 60
arrivalRate: 2
engines:
playwright: {}
scenarios:
- engine: playwright
testFunction: browserTest
# Create the test function (browser-test.js)
async function browserTest(page) {
// Navigate to the home page
await page.goto('https://example.com');
// Click the login button
await page.click('#login-button');
// Fill the login form
await page.fill('#username', 'testuser');
await page.fill('#password', 'password123');
// Submit the form
await page.click('#submit-button');
// Wait for dashboard to load
await page.waitForSelector('#dashboard');
// Perform actions on the dashboard
await page.click('#profile-tab');
// Verify element is visible
await page.waitForSelector('#profile-info');
}
module.exports = { browserTest };
Distributed Load Testing
For large-scale tests, Artillery supports distributed testing using multiple machines:
Distributed Testing Architecture
Artillery provides multiple options for distributed testing, including AWS and Azure-based solutions, to generate larger loads than possible from a single machine.
Integrating with CI/CD Pipelines
Integrating Artillery into your CI/CD pipeline enables automated performance testing as part of your development workflow.
Setting Performance Thresholds
Define performance thresholds to automatically fail the build if performance degrades:
config:
target: "https://api.example.com"
phases:
- duration: 60
arrivalRate: 5
thresholds:
# 95th percentile response time must be below 200ms
http.response_time.p95: 200
# 99th percentile response time must be below 500ms
http.response_time.p99: 500
# Less than 1% of requests should fail
http.codes.404: 1
http.codes.500: 1
GitHub Actions Integration
Example GitHub Actions workflow for performance testing:
# .github/workflows/performance.yml
name: Performance Testing
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
performance-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: '18'
- name: Install dependencies
run: npm ci
- name: Start test server
run: npm run start:test-server &
env:
PORT: 3000
- name: Wait for server
run: npx wait-on http://localhost:3000
- name: Run performance tests
run: npx artillery run --output report.json performance-test.yml
- name: Generate HTML report
run: npx artillery report report.json
- name: Upload test report
uses: actions/upload-artifact@v3
with:
name: artillery-report
path: |
report.json
report.html
Best Practices for Artillery Load Testing
Test Design
- Start Small: Begin with small tests and gradually increase load to identify breaking points
- Focus on Critical Paths: Test the most important user journeys first
- Use Realistic Scenarios: Create test scenarios that mimic real user behavior
- Include Think Time: Add realistic pauses between actions to simulate human behavior
- Test with Variety: Vary parameters and data to avoid caching effects
Execution
- Isolate the Environment: Test in isolated environments to avoid affecting real users
- Monitor System Resources: Track CPU, memory, and network metrics during tests
- Run Progressive Tests: Start with baseline tests, then move to stress and spike tests
- Test from Multiple Locations: For public APIs, test from various geographic locations
- Run Tests Regularly: Integrate performance testing into your development cycle
Analysis
- Establish Baselines: Create performance baselines to track changes over time
- Focus on Trends: Look for patterns and trends rather than absolute numbers
- Analyze Percentiles: Pay special attention to p95 and p99 response times
- Correlate with System Metrics: Connect performance issues with system resource usage
- Document Findings: Keep detailed records of performance test results
Real-World Case Study: E-commerce Company
An e-commerce company implemented Artillery load testing in their CI/CD pipeline with the following results:
- Identified a database query that degraded performance under load before deployment
- Discovered that their CDN configuration was causing delays for certain regions
- Verified that their system could handle 3x the expected Black Friday traffic
- Reduced average response time by 45% by fixing issues identified in load tests
- Implemented automated performance regression testing for all major releases
By integrating Artillery into their development workflow, they prevented several potential outages and improved customer experience significantly.
Practical Activity: Building a Complete Load Test
Let's create a comprehensive load test for a sample e-commerce API. This activity will walk you through creating, running, and analyzing a realistic load test.
Setup
- Create a new directory for your load test project:
mkdir ecommerce-load-test && cd ecommerce-load-test - Initialize a new npm project and install Artillery:
npm init -y npm install --save-dev artillery - Create a basic test script file:
touch ecommerce-test.yml
Creating the Test Script
Open ecommerce-test.yml in your editor and add the following content:
config:
target: "https://example-ecommerce-api.com" # Replace with your target API
phases:
- duration: 60
arrivalRate: 5
name: "Warm up phase"
- duration: 120
arrivalRate: 10
rampTo: 30
name: "Ramp up phase"
- duration: 300
arrivalRate: 30
name: "Sustained load phase"
variables:
productIds:
- "1001"
- "1002"
- "1003"
- "1004"
- "1005"
categories:
- "electronics"
- "clothing"
- "books"
- "home"
- "sports"
scenarios:
- name: "Browse and purchase flow"
weight: 70
flow:
# Step 1: Visit homepage
- get:
url: "/"
headers:
Accept: "application/json"
# Step 2: Browse a category
- get:
url: "/categories/{{ $randomItem(categories) }}"
headers:
Accept: "application/json"
# Step 3: Search for products
- get:
url: "/search?q=popular&sort=price"
headers:
Accept: "application/json"
capture:
- json: "$.products[0].id"
as: "firstProductId"
# Step 4: View a product (either from search results or random)
- think: 2
- get:
url: "/products/{{ firstProductId || $randomItem(productIds) }}"
headers:
Accept: "application/json"
# Step 5: Add to cart
- think: 1
- post:
url: "/cart"
headers:
Content-Type: "application/json"
Accept: "application/json"
json:
productId: "{{ firstProductId || $randomItem(productIds) }}"
quantity: "{{ $randomNumber(1, 3) }}"
# Step 6: View cart
- think: 2
- get:
url: "/cart"
headers:
Accept: "application/json"
# Step 7: Checkout
- think: 3
- post:
url: "/checkout"
headers:
Content-Type: "application/json"
Accept: "application/json"
json:
payment:
type: "credit_card"
cardNumber: "4111111111111111"
expiryMonth: "{{ $randomNumber(1, 12) }}"
expiryYear: "{{ $randomNumber(2025, 2030) }}"
cvv: "{{ $randomNumber(100, 999) }}"
shipping:
name: "Test User"
address: "123 Test St"
city: "Test City"
zipCode: "12345"
country: "US"
- name: "Browse only flow"
weight: 30
flow:
# Step 1: Visit homepage
- get:
url: "/"
headers:
Accept: "application/json"
# Step 2: Browse categories
- get:
url: "/categories"
headers:
Accept: "application/json"
# Step 3: View a category
- think: 1
- get:
url: "/categories/{{ $randomItem(categories) }}"
headers:
Accept: "application/json"
# Step 4: View multiple products
- loop:
- think: 2
- get:
url: "/products/{{ $randomItem(productIds) }}"
headers:
Accept: "application/json"
count: 3
Adding Custom Functions
Create a file named custom-functions.js with helper functions:
// Generate a random address
function generateAddress() {
const streets = ['Main St', 'Oak Ave', 'Pine Rd', 'Maple Ln', 'Cedar Blvd'];
const cities = ['New York', 'Los Angeles', 'Chicago', 'Houston', 'Phoenix'];
const states = ['NY', 'CA', 'IL', 'TX', 'AZ'];
return {
street: `${Math.floor(Math.random() * 1000) + 1} ${streets[Math.floor(Math.random() * streets.length)]}`,
city: cities[Math.floor(Math.random() * cities.length)],
state: states[Math.floor(Math.random() * states.length)],
zipCode: Math.floor(Math.random() * 90000) + 10000
};
}
// Log request for debugging
function logRequest(requestParams, context, ee, next) {
console.log(`Making request to: ${requestParams.url}`);
return next();
}
// Process and validate response
function validateResponse(requestParams, response, context, ee, next) {
const url = requestParams.url;
if (response.statusCode >= 400) {
console.error(`Error in request to ${url}: ${response.statusCode}`);
}
return next();
}
module.exports = {
generateAddress,
logRequest,
validateResponse
};
Update your test script to use these functions:
config:
target: "https://example-ecommerce-api.com"
phases:
# ... (phases remain the same)
processor: "./custom-functions.js"
# ... (rest of config remains the same)
scenarios:
- name: "Browse and purchase flow"
# ... (mostly the same as before)
flow:
# ... (earlier steps remain the same)
# Updated Step 7: Checkout with custom function
- think: 3
- function: "generateAddress"
- post:
url: "/checkout"
headers:
Content-Type: "application/json"
Accept: "application/json"
beforeRequest: "logRequest"
afterResponse: "validateResponse"
json:
payment:
type: "credit_card"
cardNumber: "4111111111111111"
expiryMonth: "{{ $randomNumber(1, 12) }}"
expiryYear: "{{ $randomNumber(2025, 2030) }}"
cvv: "{{ $randomNumber(100, 999) }}"
shipping: "{{ $generateAddress }}"
Running the Test
Now run your test and generate a report:
# Ensure your target API is accessible
# Then run the test
npx artillery run --output report.json ecommerce-test.yml
# Generate HTML report
npx artillery report report.json
Analysis and Optimization
After running the test, analyze the results and consider the following questions:
- What is the p95 response time for each endpoint?
- Are there any specific endpoints that are consistently slower?
- Did any requests fail? If so, why?
- How does the system perform under peak load compared to baseline?
- Are there any performance bottlenecks that need to be addressed?
Based on your findings, optimize your application and run the test again to verify improvements.
Key Takeaways
- Artillery is a powerful, developer-friendly load testing tool for modern applications
- YAML-based test scripts make it accessible to both developers and QA engineers
- Custom JavaScript functions and hooks enable complex, realistic test scenarios
- Multiple protocols support (HTTP, WebSocket, Socket.IO, gRPC) covers most web applications
- Integration with CI/CD pipelines allows for automated performance testing
- Distributed testing capabilities enable large-scale load testing
- Regular load testing helps identify performance issues before they affect users
Further Resources
Homework Assignment
For your homework, you'll create and execute a comprehensive load test for a web application of your choice.
Assignment Details
- Choose a target application:
- Your own application
- A public API (ensure you have permission to load test it)
- A local demo application we'll provide
- Create an Artillery test script that:
- Tests at least 5 different endpoints or user actions
- Includes at least 2 different scenarios with different weights
- Implements custom JavaScript functions for dynamic data
- Uses variable capture and reuse between requests
- Defines realistic load phases (warm-up, ramp-up, sustained load)
- Execute your test and generate a report
- Analyze the results and document:
- Overall performance metrics
- Performance bottlenecks identified
- Recommendations for improvement
- How the application behaved under different load levels
- Optimize one aspect of the application based on your findings
- Re-run the test to verify improvements
Submission Requirements
- Your Artillery test script (.yml file)
- Any custom JavaScript files used
- HTML test reports (before and after optimization)
- A brief report (1-2 pages) documenting your findings and improvements
Due Date
Submit your completed assignment before our next class session.