tests/load/README.md

# Load Testing with Locust

This directory contains Locust load tests for validating Atom API performance under concurrent user load.

## Overview

Load testing simulates multiple concurrent users interacting with the API to identify performance bottlenecks, establish capacity limits, and validate that Phase 208 performance benchmarks hold under load.

**Key Differences from Phase 208 Benchmarks:**
- **Phase 208**: Single-user benchmarks establishing targets (<1ms cache, <100ms API)
- **Phase 209**: Multi-user load testing validating targets under concurrent load (100-1000 users)

## Prerequisites

1. **Locust installed** (already in requirements-testing.txt):
   ```bash

   pip install locust>=2.15.0

   ```

2. **Application running**:
   ```bash

   cd backend

   python -m uvicorn main:app --host 0.0.0.0 --port 8000

   ```

3. **Test database** (optional but recommended):
   ```bash

   export DATABASE_URL=sqlite:///./atom_load_test.db

   ```

## Running Locust

### Interactive Mode (Web UI)

Start Locust with web UI for real-time monitoring:

```bash

cd backend

locust -f tests/load/locustfile.py

```

Then open http://localhost:8089 in your browser.

**Web UI Features:**
- Real-time request rate (RPS) monitoring
- Response time percentiles (P50, P95, P99)
- Failure rate tracking
- User count adjustment during test
- Stop/start test control

**Default Settings:**
- Host: http://localhost:8000
- Web UI Port: 8089
- Default Users: Start with 100 users
- Spawn Rate: 10 users/second

### Headless Mode (CI/CD)

Run load tests without web UI for automation:

```bash

cd backend

locust -f tests/load/locustfile.py --headless \

  -u 100 \              # 100 concurrent users

  -r 10 \               # Spawn 10 users/second

  -t 5m \               # Run for 5 minutes

  --html tests/load/reports/load-test-report.html \

  --json tests/load/reports/load-test-results.json

```

**Recommended Load Test Profiles:**

| Profile | Users | Duration | Purpose |
|---------|-------|----------|---------|
| Smoke Test | 50 | 2m | Quick validation, CI/CD |
| Standard Load | 100 | 5m | Normal traffic simulation |
| Peak Load | 500 | 10m | Peak traffic validation |
| Stress Test | 1000 | 15m | Identify breaking point |

### Custom Port (Avoid Conflicts)

If port 8089 is in use:

```bash

locust -f tests/load/locustfile.py --web-port=8090

```

## User Scenarios

### 1. AtomAPIUser (Base User)

**Purpose**: Basic API interactions with health checks

**Tasks:**
- Health check (weight: 1) - GET /health/live

**Wait Time**: 1-3 seconds between tasks

**When to Use**: Baseline health check performance

### 2. AgentAPIUser

**Purpose**: Simulate agent management operations

**Tasks:**
- List agents (weight: 5) - GET /api/v1/agents
- Get agent (weight: 3) - GET /api/v1/agents/{id}
- Create agent (weight: 1) - POST /api/v1/agents

**Wait Time**: 1-3 seconds between tasks

**Phase 208 Targets:**
- List agents: <50ms
- Get agent: <50ms
- Create agent: <100ms

**When to Use**: Testing agent CRUD performance

### 3. WorkflowExecutionUser

**Purpose**: Simulate workflow execution operations

**Tasks:**
- Execute workflow (weight: 2) - POST /api/v1/workflows/{id}/execute
- List workflows (weight: 1) - GET /api/v1/workflows

**Wait Time**: 2-5 seconds (longer for workflow execution)

**Phase 208 Targets:**
- Execute workflow: <100ms
- List workflows: <50ms

**When to Use**: Testing workflow performance under load

### 4. GovernanceCheckUser

**Purpose**: Simulate governance permission checks

**Tasks:**
- Check permission (weight: 4) - POST /api/agent-governance/check-permission
- Get cache stats (weight: 2) - GET /api/agent-governance/cache-stats

**Wait Time**: 1-3 seconds between tasks

**Phase 208 Targets:**
- Cached permission check: <1ms
- Cache stats: <50ms

**When to Use**: Testing governance cache performance

### 5. EpisodeAPIUser

**Purpose**: Simulate episodic memory retrieval

**Tasks:**
- List episodes (weight: 3) - GET /api/v1/episodes
- Get episode (weight: 2) - GET /api/v1/episodes/{id}

**Wait Time**: 1-3 seconds between tasks

**Phase 208 Targets:**
- List episodes: <50ms
- Get episode: <50ms

**When to Use**: Testing episode retrieval performance

## Critical Endpoints

| Endpoint | Method | Purpose | Weight | Phase 208 Target |
|----------|--------|---------|--------|------------------|
| /health/live | GET | Health check | 1 | <10ms |
| /api/v1/agents | GET | List agents | 5 | <50ms |
| /api/v1/agents/{id} | GET | Get agent | 3 | <50ms |
| /api/v1/agents | POST | Create agent | 1 | <100ms |
| /api/v1/workflows/{id}/execute | POST | Execute workflow | 2 | <100ms |
| /api/v1/workflows | GET | List workflows | 1 | <50ms |
| /api/agent-governance/check-permission | POST | Check permission | 4 | <1ms (cached) |
| /api/agent-governance/cache-stats | GET | Cache stats | 2 | <50ms |
| /api/v1/episodes | GET | List episodes | 3 | <50ms |
| /api/v1/episodes/{id} | GET | Get episode | 2 | <50ms |

## Interpreting Results

### Key Metrics

**Requests Per Second (RPS):**
- Measure of throughput
- Higher is better (indicates capacity)
- Target: Maintain RPS as user count increases

**Response Time Percentiles:**
- **P50 (Median)**: 50% of requests complete faster than this
- **P95**: 95% of requests complete faster than this (SLA target)
- **P99**: 99% of requests complete faster than this (tail latency)

**Failure Rate:**
- Percentage of failed requests (non-2xx status codes)
- Target: <1% failure rate
- Common failures: 500 (server error), 503 (service unavailable)

### Example Output

```

Name                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s

---------------------------------------------------------------------------------------------------------------------------------------

GET /health/live                                                  150     0(0.00%) |      8       5      15       8 |   10.00        0.00

GET /api/v1/agents                                                 75     0(0.00%) |     45      20      80      42 |    5.00        0.00

POST /api/v1/agents                                                 5     0(0.00%) |     95      70     120      90 |    0.33        0.00

---------------------------------------------------------------------------------------------------------------------------------------

Aggregate                                                          230     0(0.00%) |     32       5     120      20 |   15.33        0.00



Response time percentiles (approximate)

  50%      20ms

  66%      25ms

  75%      30ms

  80%      35ms

  90%      50ms

  95%      70ms

  98%      90ms

  99%     110ms

 100%     120ms (longest request)

```

### Performance Indicators

**Good Performance:**
- P95 response time < Phase 208 targets (e.g., <50ms for agents)
- Failure rate <1%
- RPS scales linearly with user count
- No response time spikes as user count increases

**Performance Issues:**
- P95 response time increases with user count (bottleneck)
- Failure rate >5% (capacity limit)
- RPS plateaus or decreases (system overload)
- Response time spikes during test (resource exhaustion)

## Troubleshooting

### Server Not Running

**Symptom:** Connection refused errors

**Solution:**
```bash

# Check if server is running

curl http://localhost:8000/health/live



# Start server if not running

cd backend

python -m uvicorn main:app --host 0.0.0.0 --port 8000

```

### Authentication Failures

**Symptom:** 401 Unauthorized errors

**Solution:**
- Load tests use hardcoded credentials (`load_test@example.com`)
- Ensure test user exists in database
- Check authentication endpoint is working:
  ```bash

  curl -X POST http://localhost:8000/api/v1/auth/login \

    -H "Content-Type: application/json" \

    -d '{"email": "load_test@example.com", "password": "test_password_123"}'

  ```

### Port Conflicts

**Symptom:** "Address already in use" error

**Solution:**
```bash

# Use different port for Locust web UI

locust -f tests/load/locustfile.py --web-port=8090

```

### Database Connection Pool Exhaustion

**Symptom:** "Connection pool exhausted" errors, timeouts

**Solution:**
```bash

# Increase connection pool size

export SQLALCHEMY_POOL_SIZE=20

export SQLALCHEMY_MAX_OVERFLOW=40

export SQLALCHEMY_POOL_TIMEOUT=30



# Restart server with new settings

python -m uvicorn main:app --host 0.0.0.0 --port 8000

```

### High Failure Rate

**Symptom:** >5% failure rate with 500 errors

**Possible Causes:**
1. **Database overload**: Reduce concurrent users, increase pool size
2. **Memory pressure**: Check server memory usage, reduce user count
3. **Rate limiting**: Check if API has rate limits configured
4. **External service failures**: Check LLM provider, external APIs

**Debugging:**
```bash

# Check server logs

tail -f logs/atom.log | grep ERROR



# Monitor server resources

htop  # or top on macOS



# Check database connections

# For PostgreSQL:

psql -c "SELECT count(*) FROM pg_stat_activity;"

```

### Slow Response Times

**Symptom:** P95 > 2x Phase 208 targets

**Possible Causes:**
1. **Cache misses**: Cold cache, low hit rate
2. **Database queries**: Missing indexes, N+1 queries
3. **Network latency**: Local vs remote database
4. **Resource contention**: CPU, memory, disk I/O

**Debugging:**
```bash

# Check cache hit rate

curl http://localhost:8000/api/agent-governance/cache-stats



# Profile database queries

# Enable query logging in SQLAlchemy

export SQLALCHEMY_ECHO=true



# Run with profiling

python -m cProfile -o profile.stats -m uvicorn main:app

```

## CI Integration

### GitHub Actions Workflow

```yaml

name: Load Tests



on:

  schedule:

    # Run daily at 2 AM UTC

    - cron: '0 2 * * *'

  pull_request:

    paths:

      - 'backend/core/**'

      - 'backend/api/**'



jobs:

  load-test:

    name: Run Load Tests

    runs-on: ubuntu-large



    steps:

      - name: Checkout code

        uses: actions/checkout@v4



      - name: Set up Python

        uses: actions/setup-python@v5

        with:

          python-version: '3.11'



      - name: Install dependencies

        run: |

          cd backend

          pip install -r requirements.txt

          pip install -r requirements-testing.txt



      - name: Start application

        run: |

          cd backend

          python -m uvicorn main:app --host 0.0.0.0 --port 8000 &

          sleep 10



      - name: Run load tests

        run: |

          cd backend

          locust -f tests/load/locustfile.py --headless \

            -u 100 \

            -r 10 \

            -t 5m \

            --html tests/load/reports/load-test-report.html \

            --json tests/load/reports/load-test-results.json



      - name: Upload load test report

        uses: actions/upload-artifact@v4

        with:

          name: load-test-report

          path: backend/tests/load/reports/load-test-report.html

```

### Quick Smoke Test for CI

For faster CI feedback, run a 2-minute smoke test:

```bash

locust -f tests/load/locustfile.py --headless \

  -u 50 \              # Fewer users for speed

  -r 5 \               # Slower spawn rate

  -t 2m \              # Shorter duration

  --exit-code-on-error  # Fail CI if load test fails

```

## Best Practices

1. **Warm up the cache**: Run for 1-2 minutes before collecting metrics
2. **Start small**: Begin with 50 users, scale up gradually
3. **Monitor resources**: Watch CPU, memory, database connections during test
4. **Use realistic data**: Test with production-like data volumes
5. **Run multiple times**: Performance can vary due to system load
6. **Compare to baseline**: Track performance over time to detect regressions
7. **Test in staging**: Never run load tests in production without limits

## Next Steps

After running load tests:
1. Review HTML report for detailed metrics
2. Compare P95 times to Phase 208 targets
3. Identify bottlenecks (database, cache, network)
4. Optimize slow endpoints
5. Re-run load tests to validate improvements
6. Update baseline metrics if performance improved

## References

- **Phase 208 Benchmarks**: `.planning/phases/208-integration-performance-testing/208-07-PERFORMANCE-METRICS.md`
- **Locust Documentation**: https://docs.locust.io/
- **Load Testing Research**: `.planning/phases/209-load-stress-testing/209-RESEARCH.md`
- **Monitoring Setup**: `backend/docs/MONITORING_SETUP.md`