tests/bug_discovery/feedback_loops/README.md

Feedback Loops Services

Complete feedback loop automation for bug discovery with regression test generation, fix verification, and ROI tracking.

Overview

The feedback loops package provides three services that close the bug discovery lifecycle:

1. RegressionTestGenerator: Convert BugReports to reproducible pytest test files
2. BugFixVerifier: Monitor GitHub Issues, re-run tests, auto-close verified fixes
3. ROITracker: Track ROI metrics and demonstrate business value

RegressionTestGenerator

Converts discovered bugs into permanent regression tests.

Usage

from tests.bug_discovery.feedback_loops import RegressionTestGenerator

generator = RegressionTestGenerator()

# Generate test from single bug
test_path = generator.generate_test_from_bug(bug_report)
print(f"Generated: {test_path}")

# Generate tests from bug list
test_paths = generator.generate_tests_from_bug_list(bug_reports)
print(f"Generated {len(test_paths)} tests")

# Archive test for verified fix
archived_path = generator.archive_test(test_path, reason="verified")

Templates

RegressionTestGenerator uses Jinja2 templates for each discovery method:

• pytest_regression_template.py.j2: Base template
• fuzzing_regression_template.py.j2: Fuzzing-specific tests
• chaos_regression_template.py.j2: Chaos engineering tests
• property_regression_template.py.j2: Property-based tests
• browser_regression_template.py.j2: Browser discovery tests

Archival Strategy

Tests are moved to archived/ subdirectory when:

• Bug fix is verified (2 consecutive test passes)
• GitHub issue is closed

Retention policy:

• Critical severity: Indefinite
• High severity: 1 year
• Medium/Low severity: 90 days

BugFixVerifier

Automatically verifies bug fixes by re-running regression tests.

Usage

from tests.bug_discovery.feedback_loops import BugFixVerifier

verifier = BugFixVerifier(
    github_token=os.getenv("GITHUB_TOKEN"),
    github_repository=os.getenv("GITHUB_REPOSITORY")
)

# Verify all fixes labeled in last 24 hours
results = verifier.verify_fixes(label="fix", hours_ago=24)

for result in results:
    if result["issue_closed"]:
        print(f"Issue #{result['issue_number']}: CLOSED ✅")
    elif result["test_passed"]:
        print(f"Issue #{result['issue_number']}: PENDING ({result['consecutive_passes']}/2 passes)")
    else:
        print(f"Issue #{result['issue_number']}: FAILED ❌")

Verification Workflow

1. Poll GitHub Issues for "fix" label
2. Extract bug_id from issue title/body
3. Find associated regression test file
4. Re-run test via subprocess pytest
5. If passes: Increment consecutive pass counter
6. If 2 consecutive passes: Add success comment, close issue
7. If fails: Reset counter, add failure comment

Consecutive Passes

Requires 2 consecutive test passes before closing to prevent flaky test false positives.

State tracked in .verification_state.json:

{
  "issue_123": {
    "bug_id": "abc123de",
    "consecutive_passes": 1,
    "last_passed": "2026-03-25T10:00:00Z"
  }
}

ROITracker

Tracks ROI metrics for bug discovery automation.

Usage

from tests.bug_discovery.feedback_loops import ROITracker

tracker = ROITracker()

# Record discovery run
tracker.record_discovery_run(
    bugs_found=42,
    unique_bugs=35,
    filed_bugs=30,
    duration_seconds=3600,
    by_method={"fuzzing": 20, "chaos": 10, "property": 8, "browser": 4},
    by_severity={"critical": 2, "high": 10, "medium": 15, "low": 15}
)

# Record bug fixes
tracker.record_fixes(
    bug_ids=["abc123", "def456"],
    issue_numbers=[123, 124],
    filed_dates=["2026-03-20T10:00:00Z", "2026-03-21T14:00:00Z"],
    fix_duration_hours=8.0
)

# Generate ROI report
roi_report = tracker.generate_roi_report(weeks=4)

print(f"Hours Saved: {roi_report['hours_saved']:.0f}h")
print(f"Cost Saved: ${roi_report['cost_saved']:,.0f}")
print(f"Bugs Prevented: {roi_report['bugs_prevented']}")
print(f"ROI: {roi_report['roi_ratio']:.1f}x")

# Save weekly summary
tracker.save_weekly_summary(roi_report)

# Get weekly trends for charts
trends = tracker.get_weekly_trends(weeks=12)
for week in trends:
    print(f"{week['week_start']}: {week['bugs_found']} bugs")

Cost Assumptions

Default cost assumptions (configurable via __init__):

Assumption	Default	Description
manual_qa_hourly_rate	$75/hour	Cost of manual QA labor
developer_hourly_rate	$100/hour	Cost of developer time
bug_production_cost	$10,000	Average cost of production bug
manual_qa_hours_per_bug	2 hours	Hours to manually find/report bug

ROI Calculation

Manual QA Cost = bugs_found × 2 hours × $75 = $150 × bugs_found
Automation Cost = (duration_seconds / 3600) × $100
Cost Saved = Manual QA Cost - Automation Cost
Bugs Prevented = bugs_found × 10% × $10,000
Total Savings = Cost Saved + Bugs Prevented
ROI Ratio = Total Savings / Automation Cost

Database Schema

-- Discovery runs
CREATE TABLE discovery_runs (
    id INTEGER PRIMARY KEY,
    timestamp TEXT,
    bugs_found INTEGER,
    unique_bugs INTEGER,
    filed_bugs INTEGER,
    duration_seconds REAL,
    by_method TEXT,              -- JSON
    by_severity TEXT,            -- JSON
    automation_cost REAL
);

-- Bug fixes
CREATE TABLE bug_fixes (
    id INTEGER PRIMARY KEY,
    bug_id TEXT,
    issue_number INTEGER,
    filed_at TEXT,
    fixed_at TEXT,
    fix_duration_hours REAL,
    severity TEXT,
    discovery_method TEXT
);

-- ROI summary (aggregated weekly)
CREATE TABLE roi_summary (
    id INTEGER PRIMARY KEY,
    week_start TEXT UNIQUE,
    bugs_found INTEGER,
    bugs_fixed INTEGER,
    hours_saved REAL,
    cost_saved REAL,
    automation_cost REAL,
    roi REAL,
    bugs_prevented INTEGER,
    cost_avoidance REAL,
    total_savings REAL,
    created_at TEXT
);

Integration Example

Complete feedback loop integration:

from tests.bug_discovery.core import DiscoveryCoordinator
from tests.bug_discovery.feedback_loops import BugFixVerifier, ROITracker
import os

# 1. Run discovery with feedback loops
coordinator = DiscoveryCoordinator(
    github_token=os.getenv("GITHUB_TOKEN"),
    github_repository=os.getenv("GITHUB_REPOSITORY"),
    enable_regression_tests=True,
    enable_roi_tracking=True
)

result = coordinator.run_full_discovery(duration_seconds=3600)

print(f"Bugs found: {result['bugs_found']}")
print(f"Regression tests: {len(result['regression_tests'])}")
print(f"ROI: {result['roi_data']['roi_ratio']:.1f}x")

# 2. Later, verify fixes
verifier = BugFixVerifier(
    github_token=os.getenv("GITHUB_TOKEN"),
    github_repository=os.getenv("GITHUB_REPOSITORY")
)

verification_results = verifier.verify_fixes()

# 3. Generate ROI report
roi_report = coordinator.get_roi_report(weeks=4)
weekly_trends = coordinator.get_weekly_trends(weeks=12)

Testing

Unit tests for all feedback loop services:

# RegressionTestGenerator tests
pytest backend/tests/bug_discovery/feedback_loops/tests/test_regression_test_generator.py -v

# BugFixVerifier tests
pytest backend/tests/bug_discovery/feedback_loops/tests/test_bug_fix_verifier.py -v

# ROITracker tests
pytest backend/tests/bug_discovery/feedback_loops/tests/test_roi_tracker.py -v

# Dashboard enhancement tests
pytest backend/tests/bug_discovery/feedback_loops/tests/test_dashboard_enhancements.py -v

Configuration

Environment Variables

# GitHub Integration (BugFixVerifier)
GITHUB_TOKEN=ghp_xxx
GITHUB_REPOSITORY=owner/repo

# Cost Assumptions (ROITracker)
MANUAL_QA_HOURLY_RATE=75
DEVELOPER_HOURLY_RATE=100
BUG_PRODUCTION_COST=10000
MANUAL_QA_HOURS_PER_BUG=2

File Locations

# Templates
backend/tests/bug_discovery/templates/*.j2

# Regression tests
backend/tests/bug_discovery/storage/regression_tests/test_regression_*.py
backend/tests/bug_discovery/storage/regression_tests/archived/

# Metrics database
backend/tests/bug_discovery/storage/metrics.db

# Verification state
backend/tests/bug_discovery/storage/regression_tests/.verification_state.json

Best Practices

1. Review generated tests: Auto-generated tests are minimal - review and enhance
2. Archive verified fixes: Keep regression tests directory clean by archiving
3. Validate ROI assumptions: Review cost assumptions with finance team quarterly
4. Monitor false positives: Track false positive rate, adjust verification threshold if >5%

Troubleshooting

Regression test generation fails

# Check templates directory
ls backend/tests/bug_discovery/templates/

# Verify BugReport has error_signature
python -c "from tests.bug_discovery.models import BugReport; b = BugReport(...); print(b.error_signature)"

Bug fix verification not running

# Check GitHub token has repo scope
gh auth status

# Verify "fix" label exists
gh label list

ROI metrics seem inflated

# Review cost assumptions
python -c "from tests.bug_discovery.feedback_loops import ROITracker; t = ROITracker(); print(t.manual_qa_hourly_rate)"

# Adjust based on actual project costs
tracker = ROITracker(manual_qa_hourly_rate=50, developer_hourly_rate=80)