tests/performance_regression/README.md

Performance Regression Tests

Phase: 243 - Memory & Performance Bug Discovery Location: backend/tests/performance_regression/ Last Updated: March 25, 2026

Purpose

Performance regression tests use pytest-benchmark to detect performance degradations in critical paths: API latency, database queries, and governance cache operations.

20% Regression Threshold: Detects significant performance degradations while allowing for minor measurement noise (±5-10%)

Graceful Degradation: Tests skip with pytest.skip if pytest-benchmark unavailable

Key Features

• Performance Regression Detection: Detects >20% degradation vs baseline
• Baseline Comparison: Compare current performance against established baseline
• Benchmark Statistics: Mean, min, max, stddev for rigorous analysis
• TestClient Pattern: Use TestClient instead of httpx for faster benchmarks
• Fixture Reuse: Import db_session, authenticated_user from e2e_ui/fixtures

Test Coverage

API Latency Regression (test_api_latency_regression.py)

Tests:

• test_api_get_agent_latency() - GET /api/v1/agents/{id} latency
• test_api_list_agents_latency() - GET /api/v1/agents latency
• test_api_create_agent_latency() - POST /api/v1/agents latency
• test_api_delete_agent_latency() - DELETE /api/v1/agents/{id} latency

Fixtures:

• check_regression - Regression checker with 20% threshold
• performance_baseline - Baseline loader from JSON

Focus Areas:

• API endpoint response time
• Request validation overhead
• Database query performance
• JSON serialization overhead

Database Query Regression (test_database_query_regression.py)

Tests:

• test_query_agent_by_id_latency() - Agent by ID query
• test_query_agents_list_latency() - Agent list query
• test_query_executions_by_agent_latency() - Executions by agent query
• test_query_bulk_operations_latency() - Bulk insert/update operations

Fixtures:

• check_regression - Regression checker
• db_session - Database session (imported from e2e_ui/fixtures)

Focus Areas:

• Database query execution time
• Index effectiveness
• N+1 query problems
• Bulk operation efficiency

Governance Cache Regression (test_governance_cache_regression.py)

Tests:

• test_cache_get_latency() - Cache get latency
• test_cache_set_latency() - Cache set latency
• test_cache_hit_rate() - Cache hit rate regression
• test_cache_bulk_operations_latency() - Bulk cache operations

Fixtures:

• check_regression - Regression checker
• governance_cache - GovernanceCache instance

Focus Areas:

• Cache lookup performance
• Cache miss latency
• Cache hit rate (inverted logic: lower hit rate = regression)
• Bulk cache operation efficiency

Fixtures

check_regression

Purpose: Regression checker with 20% threshold

Usage:

def test_api_latency(benchmark, check_regression):
    result = benchmark(api_call)
    check_regression(result, threshold=0.2)  # 20% threshold

Parameters:

• result - Benchmark result dict (mean, min, max, stddev)
• threshold - Regression threshold (default: 0.2 = 20%)

Assertions:

• Fails if mean increased >20% vs baseline
• Fails if hit_rate decreased >20% vs baseline (inverted logic)

performance_baseline

Purpose: Load baseline from JSON file

Usage:

def test_with_baseline(performance_baseline):
    baseline = performance_baseline["test_api_get_agent_latency"]
    print(f"Baseline mean: {baseline['mean']}")

Baseline Location: backend/tests/performance_regression/.benchmarks/

Test Patterns

API Latency Pattern

@pytest.mark.benchmark
def test_api_get_agent_latency(benchmark, check_regression):
    """
    PROPERTY: GET /api/v1/agents/{id} should complete in <100ms (P50)

    STRATEGY: Use pytest-benchmark to measure API latency. Compare against
    baseline using 20% regression threshold.

    INVARIANT: mean_latency < 100ms AND regression < 20%

    RADII: 1000 benchmark iterations provides 99% confidence with 5ms
    measurement precision.

    BASELINE: Initial baseline established 2026-03-24
    """
    from fastapi.testclient import TestClient
    from main import app

    # Setup
    client = TestClient(app)

    # Benchmark function
    def get_agent():
        response = client.get("/api/v1/agents/test-agent-1")
        assert response.status_code == 200
        return response.json()

    # Run benchmark
    result = benchmark.pedantic(get_agent, iterations=1000, rounds=10)

    # Check regression (20% threshold)
    check_regression(result, threshold=0.2)

    # Assert baseline performance
    assert result["mean"] < 0.1  # 100ms

Database Query Pattern

@pytest.mark.benchmark
def test_query_agent_by_id_latency(benchmark, check_regression, db_session):
    """
    PROPERTY: Agent by ID query should complete in <50ms

    STRATEGY: Benchmark database query with 1000 iterations
    """
    from core.models import AgentRegistry

    # Setup
    agent_id = "test-agent-1"

    # Benchmark function
    def query_agent():
        agent = db_session.query(AgentRegistry).filter(AgentRegistry.id == agent_id).first()
        assert agent is not None
        return agent

    # Run benchmark
    result = benchmark(query_agent)

    # Check regression
    check_regression(result, threshold=0.2)

    # Assert baseline performance
    assert result["mean"] < 0.05  # 50ms

Cache Hit Rate Pattern (Inverted Logic)

@pytest.mark.benchmark
def test_cache_hit_rate(benchmark, check_regression):
    """
    PROPERTY: Cache hit rate should remain >90%

    STRATEGY: Measure cache hit rate over 1000 operations. Use inverted
    logic for regression: hit_rate DECREASE is regression.

    INVARIANT: hit_rate > 0.9 AND hit_rate decrease < 20%

    RADII: 1000 operations provides 99% confidence for hit rate estimation
    """
    from core.governance_cache import GovernanceCache

    # Setup
    cache = GovernanceCache()
    cache.set("agent:1", {"data": "value1"})

    # Benchmark function
    def cache_hit_rate():
        hits = 0
        total = 100
        for i in range(total):
            if cache.get("agent:1"):
                hits += 1
        return hits / total

    # Run benchmark
    result = benchmark(cache_hit_rate)

    # Check regression (inverted logic: hit_rate decrease is regression)
    check_regression(result, threshold=0.2, metric="hit_rate")

    # Assert baseline performance
    assert result["mean"] > 0.9  # 90% hit rate

Running Tests

Run All Performance Regression Tests

cd backend
pytest tests/performance_regression/ --benchmark-only

# Compare against baseline
pytest tests/performance_regression/ --benchmark-only --benchmark-compare=baseline

# Fail on regression (>20% degradation)
pytest tests/performance_regression/ --benchmark-only --benchmark-compare=baseline --benchmark-compare-fail=mean:20%

# Generate comparison table
pytest tests/performance_regression/ --benchmark-only --benchmark-compare=baseline --benchmark-compare-fail=mean:20% --benchmark-sort=name

Run Specific Test File

# API latency regression
pytest tests/performance_regression/test_api_latency_regression.py --benchmark-only

# Database query regression
pytest tests/performance_regression/test_database_query_regression.py --benchmark-only

# Governance cache regression
pytest tests/performance_regression/test_governance_cache_regression.py --benchmark-only

Run Single Test

pytest tests/performance_regression/test_api_latency_regression.py::test_api_get_agent_latency --benchmark-only -v

Initialize Baseline (First Time)

# Generate initial baseline
pytest tests/performance_regression/ --benchmark-only --benchmark-autosave

# Commit baseline
git add backend/tests/performance_regression/.benchmarks/
git commit -m "perf: initialize performance regression baseline"

Update Baseline (After Valid Improvements)

# Run tests and autosave new baseline
pytest tests/performance_regression/ --benchmark-only --benchmark-autosave

# Commit updated baseline
git add backend/tests/performance_regression/.benchmarks/
git commit -m "perf: update performance baseline (valid improvement)"

Troubleshooting

Common Issues

1. pytest-benchmark not installed

# Symptom: Tests fail with "pytest-benchmark not installed"
# Solution: Install pytest-benchmark
pip install pytest-benchmark

2. Baseline missing

# Symptom: Tests fail with "baseline not found"
# Solution: Generate initial baseline
pytest tests/performance_regression/ --benchmark-only --benchmark-autosave

3. Performance regression false positives (<10% regression)

# Symptom: Test fails with <10% regression
# Solution: Re-run test, adjust threshold, or mark as flaky
pytest tests/performance_regression/test_api_latency_regression.py::test_api_get_agent_latency --benchmark-only -v

4. TestClient not available

# Symptom: ImportError: TestClient not available
# Solution: Install FastAPI test dependencies
pip install fastapi[all]

5. Database session fixture not found

# Symptom: Fixture 'db_session' not found
# Solution: Import from e2e_ui/fixtures
# Add to conftest.py:
# from tests.e2e_ui.fixtures.database_fixtures import db_session

Debugging Performance Regressions

View Benchmark Comparison Output:

# View benchmark comparison table
pytest tests/performance_regression/ --benchmark-only --benchmark-compare=baseline --benchmark-compare-fail=mean:20% --benchmark-sort=name

# Output columns:
# - name (benchmark name)
# - mean (current execution time)
# - min/max/stddev (execution time statistics)
# - rounds (number of iterations)
# - baseline (baseline execution time)
# - change (percentage change vs baseline)

# Regression example:
# name                            mean    baseline    change
# test_api_get_agent_latency     150ms    100ms    +50%  # REGRESSION

Performance Regression Categories:

1. API Latency: Increased response time (e.g., database query N+1 problem)
2. Database Queries: Slower queries (missing index, inefficient join)
3. Cache Hit Rate: Reduced cache effectiveness (cache invalidation issue)

Common Performance Regression Patterns:

# Pattern 1: N+1 query problem
agents = db.query(Agent).all()
for agent in agents:  # N+1: N additional queries
    executions = db.query(Execution).filter_by(agent_id=agent.id).all()

# Solution: Eager loading
from sqlalchemy.orm import joinedload
agents = db.query(Agent).options(joinedload(Agent.executions)).all()

# Pattern 2: Inefficient database query
results = db.query(Agent).filter(Agent.status == "active").all()  # Full table scan

# Solution: Add index
CREATE INDEX idx_agent_status ON agents(status);

# Pattern 3: Cache miss storm
for agent_id in agent_ids:  # N cache misses
    agent = cache.get(f"agent:{agent_id}")

# Solution: Bulk cache get
agents = cache.get_many([f"agent:{id}" for id in agent_ids])

Examples

Writing Performance Regression Tests

Example 1: API Latency Regression

import pytest
from tests.performance_regression.conftest import check_regression

@pytest.mark.benchmark
def test_api_get_agent_latency(benchmark, check_regression):
    """
    PROPERTY: GET /api/v1/agents/{id} should complete in <100ms (P50)

    STRATEGY: Use pytest-benchmark to measure API latency. Compare against
    baseline using 20% regression threshold.

    INVARIANT: mean_latency < 100ms AND regression < 20%

    RADII: 1000 benchmark iterations provides 99% confidence with 5ms
    measurement precision.

    BASELINE: Initial baseline established 2026-03-24
    """
    from fastapi.testclient import TestClient
    from main import app
    from core.models import AgentRegistry
    from sqlalchemy.orm import Session

    # Setup
    client = TestClient(app)
    with Session() as db:
        agent = db.query(AgentRegistry).first()

    # Benchmark function
    def get_agent():
        response = client.get(f"/api/v1/agents/{agent.id}")
        assert response.status_code == 200
        return response.json()

    # Run benchmark
    result = benchmark.pedantic(get_agent, iterations=1000, rounds=10)

    # Check regression (20% threshold)
    check_regression(result, threshold=0.2)

    # Assert baseline performance
    assert result["mean"] < 0.1  # 100ms

Example 2: Database Query Regression

@pytest.mark.benchmark
def test_query_agent_by_id_latency(benchmark, check_regression, db_session):
    """
    PROPERTY: Agent by ID query should complete in <50ms

    STRATEGY: Benchmark database query with 1000 iterations
    """
    from core.models import AgentRegistry

    # Setup
    agent_id = "test-agent-1"

    # Benchmark function
    def query_agent():
        agent = db_session.query(AgentRegistry).filter(AgentRegistry.id == agent_id).first()
        assert agent is not None
        return agent

    # Run benchmark
    result = benchmark(query_agent)

    # Check regression
    check_regression(result, threshold=0.2)

    # Assert baseline performance
    assert result["mean"] < 0.05  # 50ms

Example 3: Cache Hit Rate Regression (Inverted Logic)

@pytest.mark.benchmark
def test_cache_hit_rate(benchmark, check_regression):
    """
    PROPERTY: Cache hit rate should remain >90%

    STRATEGY: Measure cache hit rate over 1000 operations. Use inverted
    logic for regression: hit_rate DECREASE is regression.

    INVARIANT: hit_rate > 0.9 AND hit_rate decrease < 20%

    RADII: 1000 operations provides 99% confidence for hit rate estimation
    """
    from core.governance_cache import GovernanceCache

    # Setup
    cache = GovernanceCache()
    cache.set("agent:1", {"data": "value1"})

    # Benchmark function
    def cache_hit_rate():
        hits = 0
        total = 100
        for i in range(total):
            if cache.get("agent:1"):
                hits += 1
        return hits / total

    # Run benchmark
    result = benchmark(cache_hit_rate)

    # Check regression (inverted logic: hit_rate decrease is regression)
    check_regression(result, threshold=0.2, metric="hit_rate")

    # Assert baseline performance
    assert result["mean"] > 0.9  # 90% hit rate

Best Practices

1. Establish Baselines: Generate baselines after valid performance improvements
2. Use Realistic Thresholds: 20% regression threshold balances noise sensitivity
3. Benchmark Critical Paths: Focus on user-facing operations (API latency, database queries)
4. TestClient Pattern: Use TestClient instead of httpx for faster benchmarks
5. Fixture Reuse: Import db_session, authenticated_user from e2e_ui/fixtures
6. Document Invariants: Use PROPERTY/STRATEGY/INVARIANT/RADII format
7. Baseline Management: Commit baselines to git for reproducible regression detection

References

• Phase 243 Documentation: docs/archive/implementation/MEMORY_PERFORMANCE_BUG_DISCOVERY.md
• pytest-benchmark Documentation: https://pytest-benchmark.readthedocs.io/
• Conftest: backend/tests/performance_regression/conftest.py
• Weekly CI: .github/workflows/memory-performance-weekly.yml
• Baseline Management: backend/tests/performance_regression/.benchmarks/

---

Last Updated: March 25, 2026 Phase 243 - Memory & Performance Bug Discovery