tests/bug_discovery/README.md

Bug Discovery & Feedback Loops

Automated bug discovery through fuzzing, chaos engineering, property-based testing, and browser discovery with complete feedback loop integration.

Overview

Atom's bug discovery system automatically finds bugs through multiple discovery methods and closes the feedback loop with:

1. Automated Discovery: Fuzzing, chaos, property tests, browser exploration
2. Unified Pipeline: DiscoveryCoordinator orchestrates all methods
3. Automated Filing: BugFilingService creates GitHub Issues
4. Regression Tests: RegressionTestGenerator converts bugs to pytest tests
5. Fix Verification: BugFixVerifier re-runs tests and auto-closes issues
6. ROI Tracking: ROITracker demonstrates value of automation

Quick Start

Run Full Discovery

from tests.bug_discovery.core import run_discovery
import os

result = run_discovery(
    github_token=os.getenv("GITHUB_TOKEN"),
    github_repository=os.getenv("GITHUB_REPOSITORY"),
    duration_seconds=3600  # 1 hour
)

print(f"Bugs found: {result['bugs_found']}")
print(f"Unique bugs: {result['unique_bugs']}")
print(f"ROI: {result['roi_data']['roi_ratio']:.1f}x")

Generate Regression Tests

from tests.bug_discovery.feedback_loops import RegressionTestGenerator
from tests.bug_discovery.models.bug_report import BugReport, DiscoveryMethod, Severity
from datetime import datetime

generator = RegressionTestGenerator()

# Generate test from bug report
bug = BugReport(
    discovery_method=DiscoveryMethod.FUZZING,
    test_name="test_api_fuzzing",
    error_message="SQL injection in agent_id",
    error_signature="abc123def4567890",
    severity=Severity.CRITICAL
)

test_path = generator.generate_test_from_bug(bug)
print(f"Generated: {test_path}")

Verify Bug Fixes

from tests.bug_discovery.feedback_loops import BugFixVerifier
import os

verifier = BugFixVerifier(
    github_token=os.getenv("GITHUB_TOKEN"),
    github_repository=os.getenv("GITHUB_REPOSITORY")
)

results = verifier.verify_fixes(label="fix", hours_ago=24)

for r in results:
    if r["test_passed"]:
        print(f"Issue #{r['issue_number']}: VERIFIED ✅")
    else:
        print(f"Issue #{r['issue_number']}: FAILED ❌")

Track ROI

from tests.bug_discovery.feedback_loops import ROITracker

tracker = ROITracker()

# Record discovery run
tracker.record_discovery_run(
    bugs_found=42,
    unique_bugs=35,
    filed_bugs=30,
    duration_seconds=3600,
    by_method={"fuzzing": 20, "chaos": 10, "property": 8, "browser": 4},
    by_severity={"critical": 2, "high": 10, "medium": 15, "low": 15}
)

# Generate ROI report
roi_report = tracker.generate_roi_report(weeks=4)
print(f"Hours Saved: {roi_report['hours_saved']:.0f}h")
print(f"Cost Saved: ${roi_report['cost_saved']:,.0f}")
print(f"ROI: {roi_report['roi_ratio']:.1f}x")

Architecture

Discovery Pipeline

DiscoveryCoordinator
    ↓
1. Fuzzing Discovery (FuzzingOrchestrator)
    → BugReport objects
    ↓
2. Chaos Discovery (ChaosCoordinator)
    → BugReport objects
    ↓
3. Property Discovery (pytest)
    → BugReport objects
    ↓
4. Browser Discovery (Playwright)
    → BugReport objects
    ↓
5. Aggregation (ResultAggregator)
    → Normalized BugReport list
    ↓
6. Deduplication (BugDeduplicator)
    → Unique BugReport list
    ↓
7. Severity Classification (SeverityClassifier)
    → BugReport with severity
    ↓
8. Bug Filing (BugFilingService)
    → GitHub Issues
    ↓
9. Regression Tests (RegressionTestGenerator)
    → pytest test files
    ↓
10. ROI Tracking (ROITracker)
    → Metrics database
    ↓
11. Report Generation (DashboardGenerator)
    → HTML/JSON reports

Feedback Loops

Discovered Bug
    ↓
RegressionTestGenerator → test_regression_{method}_{bug_id}.py
    ↓
Developer fixes bug, labels issue "fix"
    ↓
BugFixVerifier detects label, re-runs test
    ↓
Test passes 2x consecutively? → Close issue ✅
Test fails? → Add failure comment, keep open ❌

ROI Calculation

Manual QA Cost:
  = bugs_found × manual_qa_hours_per_bug × manual_qa_hourly_rate
  = 50 × 2 × $75 = $7,500

Automation Cost:
  = duration_seconds / 3600 × developer_hourly_rate
  = 3600 / 3600 × $100 = $100

Cost Saved:
  = Manual QA Cost - Automation Cost
  = $7,500 - $100 = $7,400

Bugs Prevented:
  = bugs_found × 10% × bug_production_cost
  = 50 × 0.1 × $10,000 = $50,000

Total Savings:
  = Cost Saved + Bugs Prevented
  = $7,400 + $50,000 = $57,400

ROI Ratio:
  = Total Savings / Automation Cost
  = $57,400 / $100 = 574x

Directory Structure

backend/tests/bug_discovery/
├── bug_filing_service.py          # GitHub Issues automation
├── feedback_loops/                # Feedback loop services
│   ├── __init__.py
│   ├── regression_test_generator.py
│   ├── bug_fix_verifier.py
│   ├── roi_tracker.py
│   ├── tests/
│   │   ├── test_regression_test_generator.py
│   │   ├── test_bug_fix_verifier.py
│   │   ├── test_roi_tracker.py
│   │   └── test_dashboard_enhancements.py
│   └── README.md                  # Feedback loops documentation
├── core/                          # Core orchestration
│   ├── __init__.py
│   ├── discovery_coordinator.py   # Main orchestrator
│   ├── result_aggregator.py
│   ├── bug_deduplicator.py
│   ├── severity_classifier.py
│   └── dashboard_generator.py     # Report generation
├── models/
│   ├── __init__.py
│   └── bug_report.py              # Unified bug model
├── storage/                       # Data storage
│   ├── reports/                   # Weekly HTML/JSON reports
│   ├── regression_tests/          # Generated regression tests
│   │   ├── archived/              # Verified fixes (archived)
│   │   └── .gitkeep
│   ├── metrics.db                 # ROI metrics database
│   └── bug_reports.db             # Bug database
├── fuzzing/                       # Fuzzing tests
├── chaos/                         # Chaos engineering tests
├── property_tests/                # Property-based tests
├── browser_discovery/             # Browser discovery tests
└── README.md                      # This file

Discovery Methods

Fuzzing (FUZZING)

Coverage-guided fuzzing for FastAPI endpoints using Atheris.

What it finds: SQL injection, XSS, CSRF, buffer overflows, parsing errors

Example:

from tests.fuzzing.campaigns.fuzzing_orchestrator import FuzzingOrchestrator

orchestrator = FuzzingOrchestrator(github_token, github_repository)
result = orchestrator.run_campaign_with_bug_filing(
    target_endpoint="/api/v1/agents",
    test_file="tests/fuzzing/test_agent_api_fuzzing.py",
    duration_seconds=900
)

Chaos Engineering (CHAOS)

Controlled failure injection testing resilience.

What it finds: Connection timeouts, memory exhaustion, cascading failures

Example:

from tests.chaos.core.chaos_coordinator import ChaosCoordinator

coordinator = ChaosCoordinator(db_session=test_db, bug_filing_service=filing_service)
result = coordinator.run_experiment(
    experiment_name="network_latency_3g",
    failure_injection=LatencyInjection,
    verify_graceful_degradation=lambda: True,
    blast_radius_checks=[assert_blast_radius]
)

Property-Based Testing (PROPERTY)

Hypothesis-based invariant testing.

What it finds: State machine violations, contract violations, edge cases

Example:

pytest tests/property_tests/ -v

Browser Discovery (BROWSER)

Headless browser automation for UI bug detection.

What it finds: Console errors, accessibility violations, broken links

Example:

from tests.browser_discovery.conftest import authenticated_page

page.goto("http://localhost:3000/dashboard")
errors = page.evaluate("() => window.__consoleErrors || []")
assert len(errors) == 0

Configuration

Environment Variables

# GitHub Integration
GITHUB_TOKEN=ghp_xxx                    # GitHub Personal Access Token
GITHUB_REPOSITORY=owner/repo             # Repository for issue filing

# Discovery Configuration
DISCOVERY_DURATION_SECONDS=3600          # Duration per discovery method
ENABLE_REGRESSION_TESTS=true            # Generate regression tests
ENABLE_ROI_TRACKING=true                # Track ROI metrics

# ROI Cost Assumptions (optional, uses defaults if not set)
MANUAL_QA_HOURLY_RATE=75                # Cost per hour for manual QA
DEVELOPER_HOURLY_RATE=100               # Cost per hour for developer
BUG_PRODUCTION_COST=10000               # Average cost of production bug
MANUAL_QA_HOURS_PER_BUG=2               # Hours to manually find a bug

Pytest Configuration

# pytest.ini

[pytest]
markers =
    regression: Marks tests as regression tests
    fuzzing: Marks tests as fuzzing tests
    chaos: Marks tests as chaos engineering tests
    property: Marks tests as property-based tests
    browser: Marks tests as browser discovery tests
    slow: Marks tests as slow (run in weekly pipeline only)

CI/CD Integration

Weekly Bug Discovery Pipeline

Runs every Sunday at 2 AM UTC:

# .github/workflows/bug-discovery-weekly.yml
on:
  schedule:
    - cron: '0 2 * * 0'  # Sunday 2 AM UTC

jobs:
  bug-discovery:
    steps:
      - uses: actions/checkout@v4
      - run: |
          python -c "
          from tests.bug_discovery.core import run_discovery
          run_discovery(
              github_token=os.getenv('GITHUB_TOKEN'),
              github_repository=os.getenv('GITHUB_REPOSITORY'),
              duration_seconds=3600
          )
          "

Bug Fix Verification Pipeline

Runs every 6 hours to verify fixes:

# .github/workflows/bug-fix-verification.yml
on:
  schedule:
    - cron: '0 */6 * * *'  # Every 6 hours

jobs:
  verify-fixes:
    steps:
      - run: |
          python -c "
          from tests.bug_discovery.feedback_loops import BugFixVerifier
          verifier = BugFixVerifier(...)
          verifier.verify_fixes()
          "

Best Practices

Writing Regression Tests

Generated regression tests should be reviewed and enhanced:

# Auto-generated (minimal)
def test_regression_fuzzing_abc123de(api_client):
    # TODO: Implement test logic
    assert True  # Placeholder

# Enhanced (manual review)
def test_regression_fuzzing_abc123de(api_client):
    """Regression test for SQL injection in agent_id parameter."""
    # Test with malicious input
    response = api_client.post("/api/v1/agents", json={
        "agent_id": "1 OR 1=1--"
    })

    # Should reject malicious input
    assert response.status_code in [400, 422]

    # Test with valid input
    response = api_client.post("/api/v1/agents", json={
        "agent_id": "valid-uuid"
    })

    # Should accept valid input
    assert response.status_code == 200

Archival Strategy

Regression tests are archived when:

1. Bug fix is verified (test passes 2x consecutively)
2. GitHub issue is closed
3. Test moved to regression_tests/archived/

Restoration:

If a bug recurs (same error_signature detected):

1. Test moved back from archived/
2. Re-run to confirm bug exists
3. Re-open GitHub issue if confirmed

Metrics & Effectiveness

Key Metrics

Metric	Description	Target
Bugs found per hour	Discovery throughput	> 10/hour
Unique bug rate	Deduplication effectiveness	> 70%
False positive rate	Bugs not confirmed	< 5%
Fix verification rate	Bugs with verified fixes	> 80%
ROI ratio	Cost savings vs automation cost	> 10x

Weekly Report Sections

1. Bug Discovery Summary: Bugs found, unique, filed
2. ROI Metrics: Hours saved, cost saved, bugs prevented, ROI ratio
3. Fix Verification: Fixed, verified, pending counts
4. By Method: Breakdown by discovery method
5. By Severity: Breakdown by severity
6. Top Bugs: Highest priority bugs

Troubleshooting

Regression Test Generation Fails

# Check templates exist
ls backend/tests/bug_discovery/templates/*.j2

# Verify BugReport has required fields
python -c "from tests.bug_discovery.models import BugReport; print(BugReport.__fields__)"

Bug Fix Verification Not Running

# Check GitHub token has repo scope
# Check workflow is enabled in GitHub Actions
# Verify "fix" label exists in repository

ROI Metrics Not Appearing

# Check metrics.db exists
ls backend/tests/bug_discovery/storage/metrics.db

# Verify ROI tracking enabled
python -c "from tests.bug_discovery.core import DiscoveryCoordinator; c = DiscoveryCoordinator(...); print(c.enable_roi_tracking)"

Further Reading

• Bug Discovery Infrastructure
• Feedback Loops Documentation
• Property-Based Testing
• Chaos Engineering
• Fuzzing Guide

Contributing

When adding new discovery methods:

1. Create BugReport objects with required fields
2. Add method to DiscoveryMethod enum
3. Implement aggregation in ResultAggregator
4. Update DashboardGenerator templates
5. Add regression test template