tests/performance_regression/README.md

# Performance Regression Tests

**Phase:** 243 - Memory & Performance Bug Discovery
**Location:** `backend/tests/performance_regression/`
**Last Updated:** March 25, 2026

## Purpose

Performance regression tests use **pytest-benchmark** to detect performance degradations in critical paths: API latency, database queries, and governance cache operations.

**20% Regression Threshold:** Detects significant performance degradations while allowing for minor measurement noise (±5-10%)

**Graceful Degradation:** Tests skip with pytest.skip if pytest-benchmark unavailable

## Key Features

- **Performance Regression Detection:** Detects >20% degradation vs baseline
- **Baseline Comparison:** Compare current performance against established baseline
- **Benchmark Statistics:** Mean, min, max, stddev for rigorous analysis
- **TestClient Pattern:** Use TestClient instead of httpx for faster benchmarks
- **Fixture Reuse:** Import db_session, authenticated_user from e2e_ui/fixtures



## Test Coverage



### API Latency Regression (`test_api_latency_regression.py`)

**Tests:**
- `test_api_get_agent_latency()` - GET /api/v1/agents/{id} latency
- `test_api_list_agents_latency()` - GET /api/v1/agents latency
- `test_api_create_agent_latency()` - POST /api/v1/agents latency
- `test_api_delete_agent_latency()` - DELETE /api/v1/agents/{id} latency

**Fixtures:**
- `check_regression` - Regression checker with 20% threshold
- `performance_baseline` - Baseline loader from JSON

**Focus Areas:**
- API endpoint response time
- Request validation overhead
- Database query performance
- JSON serialization overhead

### Database Query Regression (`test_database_query_regression.py`)



**Tests:**

- `test_query_agent_by_id_latency()` - Agent by ID query
- `test_query_agents_list_latency()` - Agent list query
- `test_query_executions_by_agent_latency()` - Executions by agent query
- `test_query_bulk_operations_latency()` - Bulk insert/update operations

**Fixtures:**
- `check_regression` - Regression checker
- `db_session` - Database session (imported from e2e_ui/fixtures)



**Focus Areas:**

- Database query execution time

- Index effectiveness

- N+1 query problems

- Bulk operation efficiency



### Governance Cache Regression (`test_governance_cache_regression.py`)

**Tests:**
- `test_cache_get_latency()` - Cache get latency
- `test_cache_set_latency()` - Cache set latency
- `test_cache_hit_rate()` - Cache hit rate regression
- `test_cache_bulk_operations_latency()` - Bulk cache operations

**Fixtures:**
- `check_regression` - Regression checker
- `governance_cache` - GovernanceCache instance

**Focus Areas:**
- Cache lookup performance
- Cache miss latency
- Cache hit rate (inverted logic: lower hit rate = regression)
- Bulk cache operation efficiency

## Fixtures

### check_regression



**Purpose:** Regression checker with 20% threshold



**Usage:**

```python

def test_api_latency(benchmark, check_regression):
    result = benchmark(api_call)

    check_regression(result, threshold=0.2)  # 20% threshold

```


**Parameters:**
- `result` - Benchmark result dict (mean, min, max, stddev)
- `threshold` - Regression threshold (default: 0.2 = 20%)

**Assertions:**
- Fails if `mean` increased >20% vs baseline
- Fails if `hit_rate` decreased >20% vs baseline (inverted logic)

### performance_baseline



**Purpose:** Load baseline from JSON file



**Usage:**

```python

def test_with_baseline(performance_baseline):
    baseline = performance_baseline["test_api_get_agent_latency"]

    print(f"Baseline mean: {baseline['mean']}")

```


**Baseline Location:** `backend/tests/performance_regression/.benchmarks/`

## Test Patterns

### API Latency Pattern

```python

@pytest.mark.benchmark

def test_api_get_agent_latency(benchmark, check_regression):

    """

    PROPERTY: GET /api/v1/agents/{id} should complete in <100ms (P50)



    STRATEGY: Use pytest-benchmark to measure API latency. Compare against

    baseline using 20% regression threshold.



    INVARIANT: mean_latency < 100ms AND regression < 20%



    RADII: 1000 benchmark iterations provides 99% confidence with 5ms

    measurement precision.



    BASELINE: Initial baseline established 2026-03-24

    """

    from fastapi.testclient import TestClient

    from main import app



    # Setup

    client = TestClient(app)



    # Benchmark function

    def get_agent():

        response = client.get("/api/v1/agents/test-agent-1")

        assert response.status_code == 200

        return response.json()



    # Run benchmark

    result = benchmark.pedantic(get_agent, iterations=1000, rounds=10)



    # Check regression (20% threshold)

    check_regression(result, threshold=0.2)



    # Assert baseline performance

    assert result["mean"] < 0.1  # 100ms

```

### Database Query Pattern

```python

@pytest.mark.benchmark

def test_query_agent_by_id_latency(benchmark, check_regression, db_session):

    """

    PROPERTY: Agent by ID query should complete in <50ms



    STRATEGY: Benchmark database query with 1000 iterations

    """

    from core.models import AgentRegistry



    # Setup

    agent_id = "test-agent-1"



    # Benchmark function

    def query_agent():

        agent = db_session.query(AgentRegistry).filter(AgentRegistry.id == agent_id).first()

        assert agent is not None

        return agent



    # Run benchmark

    result = benchmark(query_agent)



    # Check regression

    check_regression(result, threshold=0.2)



    # Assert baseline performance

    assert result["mean"] < 0.05  # 50ms

```

### Cache Hit Rate Pattern (Inverted Logic)

```python

@pytest.mark.benchmark

def test_cache_hit_rate(benchmark, check_regression):

    """

    PROPERTY: Cache hit rate should remain >90%



    STRATEGY: Measure cache hit rate over 1000 operations. Use inverted

    logic for regression: hit_rate DECREASE is regression.



    INVARIANT: hit_rate > 0.9 AND hit_rate decrease < 20%



    RADII: 1000 operations provides 99% confidence for hit rate estimation

    """

    from core.governance_cache import GovernanceCache



    # Setup

    cache = GovernanceCache()

    cache.set("agent:1", {"data": "value1"})



    # Benchmark function

    def cache_hit_rate():

        hits = 0

        total = 100

        for i in range(total):

            if cache.get("agent:1"):

                hits += 1

        return hits / total



    # Run benchmark

    result = benchmark(cache_hit_rate)



    # Check regression (inverted logic: hit_rate decrease is regression)

    check_regression(result, threshold=0.2, metric="hit_rate")



    # Assert baseline performance

    assert result["mean"] > 0.9  # 90% hit rate

```

## Running Tests

### Run All Performance Regression Tests

```bash

cd backend

pytest tests/performance_regression/ --benchmark-only



# Compare against baseline

pytest tests/performance_regression/ --benchmark-only --benchmark-compare=baseline



# Fail on regression (>20% degradation)

pytest tests/performance_regression/ --benchmark-only --benchmark-compare=baseline --benchmark-compare-fail=mean:20%



# Generate comparison table

pytest tests/performance_regression/ --benchmark-only --benchmark-compare=baseline --benchmark-compare-fail=mean:20% --benchmark-sort=name

```

### Run Specific Test File

```bash

# API latency regression

pytest tests/performance_regression/test_api_latency_regression.py --benchmark-only



# Database query regression

pytest tests/performance_regression/test_database_query_regression.py --benchmark-only



# Governance cache regression

pytest tests/performance_regression/test_governance_cache_regression.py --benchmark-only

```

### Run Single Test

```bash

pytest tests/performance_regression/test_api_latency_regression.py::test_api_get_agent_latency --benchmark-only -v

```

### Initialize Baseline (First Time)

```bash

# Generate initial baseline

pytest tests/performance_regression/ --benchmark-only --benchmark-autosave



# Commit baseline

git add backend/tests/performance_regression/.benchmarks/

git commit -m "perf: initialize performance regression baseline"

```

### Update Baseline (After Valid Improvements)

```bash

# Run tests and autosave new baseline

pytest tests/performance_regression/ --benchmark-only --benchmark-autosave



# Commit updated baseline

git add backend/tests/performance_regression/.benchmarks/

git commit -m "perf: update performance baseline (valid improvement)"

```

## Troubleshooting

### Common Issues

**1. pytest-benchmark not installed**
```bash

# Symptom: Tests fail with "pytest-benchmark not installed"

# Solution: Install pytest-benchmark

pip install pytest-benchmark

```

**2. Baseline missing**
```bash

# Symptom: Tests fail with "baseline not found"

# Solution: Generate initial baseline

pytest tests/performance_regression/ --benchmark-only --benchmark-autosave

```

**3. Performance regression false positives (<10% regression)**
```bash

# Symptom: Test fails with <10% regression

# Solution: Re-run test, adjust threshold, or mark as flaky

pytest tests/performance_regression/test_api_latency_regression.py::test_api_get_agent_latency --benchmark-only -v

```

**4. TestClient not available**
```bash

# Symptom: ImportError: TestClient not available

# Solution: Install FastAPI test dependencies

pip install fastapi[all]

```

**5. Database session fixture not found**
```bash

# Symptom: Fixture 'db_session' not found

# Solution: Import from e2e_ui/fixtures

# Add to conftest.py:

# from tests.e2e_ui.fixtures.database_fixtures import db_session

```

### Debugging Performance Regressions

**View Benchmark Comparison Output:**
```bash

# View benchmark comparison table

pytest tests/performance_regression/ --benchmark-only --benchmark-compare=baseline --benchmark-compare-fail=mean:20% --benchmark-sort=name



# Output columns:

# - name (benchmark name)

# - mean (current execution time)

# - min/max/stddev (execution time statistics)

# - rounds (number of iterations)

# - baseline (baseline execution time)

# - change (percentage change vs baseline)



# Regression example:

# name                            mean    baseline    change

# test_api_get_agent_latency     150ms    100ms    +50%  # REGRESSION

```

**Performance Regression Categories:**
1. **API Latency:** Increased response time (e.g., database query N+1 problem)
2. **Database Queries:** Slower queries (missing index, inefficient join)
3. **Cache Hit Rate:** Reduced cache effectiveness (cache invalidation issue)

**Common Performance Regression Patterns:**
```python

# Pattern 1: N+1 query problem

agents = db.query(Agent).all()

for agent in agents:  # N+1: N additional queries

    executions = db.query(Execution).filter_by(agent_id=agent.id).all()



# Solution: Eager loading

from sqlalchemy.orm import joinedload

agents = db.query(Agent).options(joinedload(Agent.executions)).all()



# Pattern 2: Inefficient database query

results = db.query(Agent).filter(Agent.status == "active").all()  # Full table scan



# Solution: Add index

CREATE INDEX idx_agent_status ON agents(status);



# Pattern 3: Cache miss storm

for agent_id in agent_ids:  # N cache misses

    agent = cache.get(f"agent:{agent_id}")



# Solution: Bulk cache get

agents = cache.get_many([f"agent:{id}" for id in agent_ids])

```

## Examples

### Writing Performance Regression Tests

**Example 1: API Latency Regression**
```python

import pytest

from tests.performance_regression.conftest import check_regression



@pytest.mark.benchmark

def test_api_get_agent_latency(benchmark, check_regression):

    """

    PROPERTY: GET /api/v1/agents/{id} should complete in <100ms (P50)



    STRATEGY: Use pytest-benchmark to measure API latency. Compare against

    baseline using 20% regression threshold.



    INVARIANT: mean_latency < 100ms AND regression < 20%



    RADII: 1000 benchmark iterations provides 99% confidence with 5ms

    measurement precision.



    BASELINE: Initial baseline established 2026-03-24

    """

    from fastapi.testclient import TestClient

    from main import app

    from core.models import AgentRegistry

    from sqlalchemy.orm import Session



    # Setup

    client = TestClient(app)

    with Session() as db:

        agent = db.query(AgentRegistry).first()



    # Benchmark function

    def get_agent():

        response = client.get(f"/api/v1/agents/{agent.id}")

        assert response.status_code == 200

        return response.json()



    # Run benchmark

    result = benchmark.pedantic(get_agent, iterations=1000, rounds=10)



    # Check regression (20% threshold)

    check_regression(result, threshold=0.2)



    # Assert baseline performance

    assert result["mean"] < 0.1  # 100ms

```

**Example 2: Database Query Regression**
```python

@pytest.mark.benchmark

def test_query_agent_by_id_latency(benchmark, check_regression, db_session):

    """

    PROPERTY: Agent by ID query should complete in <50ms



    STRATEGY: Benchmark database query with 1000 iterations

    """

    from core.models import AgentRegistry



    # Setup

    agent_id = "test-agent-1"



    # Benchmark function

    def query_agent():

        agent = db_session.query(AgentRegistry).filter(AgentRegistry.id == agent_id).first()

        assert agent is not None

        return agent



    # Run benchmark

    result = benchmark(query_agent)



    # Check regression

    check_regression(result, threshold=0.2)



    # Assert baseline performance

    assert result["mean"] < 0.05  # 50ms

```

**Example 3: Cache Hit Rate Regression (Inverted Logic)**
```python

@pytest.mark.benchmark

def test_cache_hit_rate(benchmark, check_regression):

    """

    PROPERTY: Cache hit rate should remain >90%



    STRATEGY: Measure cache hit rate over 1000 operations. Use inverted

    logic for regression: hit_rate DECREASE is regression.



    INVARIANT: hit_rate > 0.9 AND hit_rate decrease < 20%



    RADII: 1000 operations provides 99% confidence for hit rate estimation

    """

    from core.governance_cache import GovernanceCache



    # Setup

    cache = GovernanceCache()

    cache.set("agent:1", {"data": "value1"})



    # Benchmark function

    def cache_hit_rate():

        hits = 0

        total = 100

        for i in range(total):

            if cache.get("agent:1"):

                hits += 1

        return hits / total



    # Run benchmark

    result = benchmark(cache_hit_rate)



    # Check regression (inverted logic: hit_rate decrease is regression)

    check_regression(result, threshold=0.2, metric="hit_rate")



    # Assert baseline performance

    assert result["mean"] > 0.9  # 90% hit rate

```

## Best Practices

1. **Establish Baselines:** Generate baselines after valid performance improvements
2. **Use Realistic Thresholds:** 20% regression threshold balances noise sensitivity
3. **Benchmark Critical Paths:** Focus on user-facing operations (API latency, database queries)
4. **TestClient Pattern:** Use TestClient instead of httpx for faster benchmarks
5. **Fixture Reuse:** Import db_session, authenticated_user from e2e_ui/fixtures

6. **Document Invariants:** Use PROPERTY/STRATEGY/INVARIANT/RADII format

7. **Baseline Management:** Commit baselines to git for reproducible regression detection



## References



- **Phase 243 Documentation:** `docs/archive/implementation/MEMORY_PERFORMANCE_BUG_DISCOVERY.md`
- **pytest-benchmark Documentation:** https://pytest-benchmark.readthedocs.io/
- **Conftest:** `backend/tests/performance_regression/conftest.py`
- **Weekly CI:** `.github/workflows/memory-performance-weekly.yml`
- **Baseline Management:** `backend/tests/performance_regression/.benchmarks/`

---

*Last Updated: March 25, 2026*
*Phase 243 - Memory & Performance Bug Discovery*