tests/bug_discovery/README.md

# Bug Discovery & Feedback Loops

Automated bug discovery through fuzzing, chaos engineering, property-based testing, and browser discovery with complete feedback loop integration.

## Overview

Atom's bug discovery system automatically finds bugs through multiple discovery methods and closes the feedback loop with:

1. **Automated Discovery**: Fuzzing, chaos, property tests, browser exploration
2. **Unified Pipeline**: DiscoveryCoordinator orchestrates all methods
3. **Automated Filing**: BugFilingService creates GitHub Issues
4. **Regression Tests**: RegressionTestGenerator converts bugs to pytest tests
5. **Fix Verification**: BugFixVerifier re-runs tests and auto-closes issues
6. **ROI Tracking**: ROITracker demonstrates value of automation

## Quick Start

### Run Full Discovery

```python

from tests.bug_discovery.core import run_discovery

import os



result = run_discovery(

    github_token=os.getenv("GITHUB_TOKEN"),

    github_repository=os.getenv("GITHUB_REPOSITORY"),

    duration_seconds=3600  # 1 hour

)



print(f"Bugs found: {result['bugs_found']}")

print(f"Unique bugs: {result['unique_bugs']}")

print(f"ROI: {result['roi_data']['roi_ratio']:.1f}x")

```

### Generate Regression Tests

```python

from tests.bug_discovery.feedback_loops import RegressionTestGenerator

from tests.bug_discovery.models.bug_report import BugReport, DiscoveryMethod, Severity

from datetime import datetime



generator = RegressionTestGenerator()



# Generate test from bug report

bug = BugReport(

    discovery_method=DiscoveryMethod.FUZZING,

    test_name="test_api_fuzzing",

    error_message="SQL injection in agent_id",

    error_signature="abc123def4567890",

    severity=Severity.CRITICAL

)



test_path = generator.generate_test_from_bug(bug)

print(f"Generated: {test_path}")

```

### Verify Bug Fixes

```python

from tests.bug_discovery.feedback_loops import BugFixVerifier

import os



verifier = BugFixVerifier(

    github_token=os.getenv("GITHUB_TOKEN"),

    github_repository=os.getenv("GITHUB_REPOSITORY")

)



results = verifier.verify_fixes(label="fix", hours_ago=24)



for r in results:

    if r["test_passed"]:

        print(f"Issue #{r['issue_number']}: VERIFIED ✅")

    else:

        print(f"Issue #{r['issue_number']}: FAILED ❌")

```

### Track ROI

```python

from tests.bug_discovery.feedback_loops import ROITracker



tracker = ROITracker()



# Record discovery run

tracker.record_discovery_run(

    bugs_found=42,

    unique_bugs=35,

    filed_bugs=30,

    duration_seconds=3600,

    by_method={"fuzzing": 20, "chaos": 10, "property": 8, "browser": 4},

    by_severity={"critical": 2, "high": 10, "medium": 15, "low": 15}

)



# Generate ROI report

roi_report = tracker.generate_roi_report(weeks=4)

print(f"Hours Saved: {roi_report['hours_saved']:.0f}h")

print(f"Cost Saved: ${roi_report['cost_saved']:,.0f}")

print(f"ROI: {roi_report['roi_ratio']:.1f}x")

```

## Architecture

### Discovery Pipeline

```

DiscoveryCoordinator

    ↓

1. Fuzzing Discovery (FuzzingOrchestrator)

    → BugReport objects

    ↓

2. Chaos Discovery (ChaosCoordinator)

    → BugReport objects

    ↓

3. Property Discovery (pytest)

    → BugReport objects

    ↓

4. Browser Discovery (Playwright)

    → BugReport objects

    ↓

5. Aggregation (ResultAggregator)

    → Normalized BugReport list

    ↓

6. Deduplication (BugDeduplicator)

    → Unique BugReport list

    ↓

7. Severity Classification (SeverityClassifier)

    → BugReport with severity

    ↓

8. Bug Filing (BugFilingService)

    → GitHub Issues

    ↓

9. Regression Tests (RegressionTestGenerator)

    → pytest test files

    ↓

10. ROI Tracking (ROITracker)

    → Metrics database

    ↓

11. Report Generation (DashboardGenerator)

    → HTML/JSON reports

```

### Feedback Loops

```

Discovered Bug

    ↓

RegressionTestGenerator → test_regression_{method}_{bug_id}.py

    ↓

Developer fixes bug, labels issue "fix"

    ↓

BugFixVerifier detects label, re-runs test

    ↓

Test passes 2x consecutively? → Close issue ✅

Test fails? → Add failure comment, keep open ❌

```

### ROI Calculation

```

Manual QA Cost:

  = bugs_found × manual_qa_hours_per_bug × manual_qa_hourly_rate

  = 50 × 2 × $75 = $7,500



Automation Cost:

  = duration_seconds / 3600 × developer_hourly_rate

  = 3600 / 3600 × $100 = $100



Cost Saved:

  = Manual QA Cost - Automation Cost

  = $7,500 - $100 = $7,400



Bugs Prevented:

  = bugs_found × 10% × bug_production_cost

  = 50 × 0.1 × $10,000 = $50,000



Total Savings:

  = Cost Saved + Bugs Prevented

  = $7,400 + $50,000 = $57,400



ROI Ratio:

  = Total Savings / Automation Cost

  = $57,400 / $100 = 574x

```

## Directory Structure

```

backend/tests/bug_discovery/

├── bug_filing_service.py          # GitHub Issues automation

├── feedback_loops/                # Feedback loop services

│   ├── __init__.py

│   ├── regression_test_generator.py

│   ├── bug_fix_verifier.py

│   ├── roi_tracker.py

│   ├── tests/

│   │   ├── test_regression_test_generator.py

│   │   ├── test_bug_fix_verifier.py

│   │   ├── test_roi_tracker.py

│   │   └── test_dashboard_enhancements.py

│   └── README.md                  # Feedback loops documentation

├── core/                          # Core orchestration

│   ├── __init__.py

│   ├── discovery_coordinator.py   # Main orchestrator

│   ├── result_aggregator.py

│   ├── bug_deduplicator.py

│   ├── severity_classifier.py

│   └── dashboard_generator.py     # Report generation

├── models/

│   ├── __init__.py

│   └── bug_report.py              # Unified bug model

├── storage/                       # Data storage

│   ├── reports/                   # Weekly HTML/JSON reports

│   ├── regression_tests/          # Generated regression tests

│   │   ├── archived/              # Verified fixes (archived)

│   │   └── .gitkeep

│   ├── metrics.db                 # ROI metrics database

│   └── bug_reports.db             # Bug database

├── fuzzing/                       # Fuzzing tests

├── chaos/                         # Chaos engineering tests

├── property_tests/                # Property-based tests

├── browser_discovery/             # Browser discovery tests

└── README.md                      # This file

```

## Discovery Methods

### Fuzzing (FUZZING)

Coverage-guided fuzzing for FastAPI endpoints using Atheris.

**What it finds:** SQL injection, XSS, CSRF, buffer overflows, parsing errors

**Example:**
```python

from tests.fuzzing.campaigns.fuzzing_orchestrator import FuzzingOrchestrator



orchestrator = FuzzingOrchestrator(github_token, github_repository)

result = orchestrator.run_campaign_with_bug_filing(

    target_endpoint="/api/v1/agents",

    test_file="tests/fuzzing/test_agent_api_fuzzing.py",

    duration_seconds=900

)

```

### Chaos Engineering (CHAOS)

Controlled failure injection testing resilience.

**What it finds:** Connection timeouts, memory exhaustion, cascading failures

**Example:**
```python

from tests.chaos.core.chaos_coordinator import ChaosCoordinator



coordinator = ChaosCoordinator(db_session=test_db, bug_filing_service=filing_service)

result = coordinator.run_experiment(

    experiment_name="network_latency_3g",

    failure_injection=LatencyInjection,

    verify_graceful_degradation=lambda: True,

    blast_radius_checks=[assert_blast_radius]

)

```

### Property-Based Testing (PROPERTY)

Hypothesis-based invariant testing.

**What it finds:** State machine violations, contract violations, edge cases

**Example:**
```bash

pytest tests/property_tests/ -v

```

### Browser Discovery (BROWSER)

Headless browser automation for UI bug detection.

**What it finds:** Console errors, accessibility violations, broken links

**Example:**
```python

from tests.browser_discovery.conftest import authenticated_page



page.goto("http://localhost:3000/dashboard")

errors = page.evaluate("() => window.__consoleErrors || []")

assert len(errors) == 0

```

## Configuration

### Environment Variables

```bash

# GitHub Integration

GITHUB_TOKEN=ghp_xxx                    # GitHub Personal Access Token

GITHUB_REPOSITORY=owner/repo             # Repository for issue filing



# Discovery Configuration

DISCOVERY_DURATION_SECONDS=3600          # Duration per discovery method

ENABLE_REGRESSION_TESTS=true            # Generate regression tests

ENABLE_ROI_TRACKING=true                # Track ROI metrics



# ROI Cost Assumptions (optional, uses defaults if not set)

MANUAL_QA_HOURLY_RATE=75                # Cost per hour for manual QA

DEVELOPER_HOURLY_RATE=100               # Cost per hour for developer

BUG_PRODUCTION_COST=10000               # Average cost of production bug

MANUAL_QA_HOURS_PER_BUG=2               # Hours to manually find a bug

```

### Pytest Configuration

```ini

# pytest.ini



[pytest]

markers =

    regression: Marks tests as regression tests

    fuzzing: Marks tests as fuzzing tests

    chaos: Marks tests as chaos engineering tests

    property: Marks tests as property-based tests

    browser: Marks tests as browser discovery tests

    slow: Marks tests as slow (run in weekly pipeline only)

```

## CI/CD Integration

### Weekly Bug Discovery Pipeline

Runs every Sunday at 2 AM UTC:

```yaml

# .github/workflows/bug-discovery-weekly.yml

on:

  schedule:

    - cron: '0 2 * * 0'  # Sunday 2 AM UTC



jobs:

  bug-discovery:

    steps:

      - uses: actions/checkout@v4

      - run: |

          python -c "

          from tests.bug_discovery.core import run_discovery

          run_discovery(

              github_token=os.getenv('GITHUB_TOKEN'),

              github_repository=os.getenv('GITHUB_REPOSITORY'),

              duration_seconds=3600

          )

          "

```

### Bug Fix Verification Pipeline

Runs every 6 hours to verify fixes:

```yaml

# .github/workflows/bug-fix-verification.yml

on:

  schedule:

    - cron: '0 */6 * * *'  # Every 6 hours



jobs:

  verify-fixes:

    steps:

      - run: |

          python -c "

          from tests.bug_discovery.feedback_loops import BugFixVerifier

          verifier = BugFixVerifier(...)

          verifier.verify_fixes()

          "

```

## Best Practices

### Writing Regression Tests

Generated regression tests should be reviewed and enhanced:

```python

# Auto-generated (minimal)

def test_regression_fuzzing_abc123de(api_client):

    # TODO: Implement test logic

    assert True  # Placeholder



# Enhanced (manual review)

def test_regression_fuzzing_abc123de(api_client):

    """Regression test for SQL injection in agent_id parameter."""

    # Test with malicious input

    response = api_client.post("/api/v1/agents", json={

        "agent_id": "1 OR 1=1--"

    })



    # Should reject malicious input

    assert response.status_code in [400, 422]



    # Test with valid input

    response = api_client.post("/api/v1/agents", json={

        "agent_id": "valid-uuid"

    })



    # Should accept valid input

    assert response.status_code == 200

```

### Archival Strategy

Regression tests are archived when:

1. Bug fix is verified (test passes 2x consecutively)
2. GitHub issue is closed
3. Test moved to `regression_tests/archived/`

Restoration:

If a bug recurs (same error_signature detected):

1. Test moved back from `archived/`

2. Re-run to confirm bug exists

3. Re-open GitHub issue if confirmed



## Metrics & Effectiveness



### Key Metrics



| Metric | Description | Target |

|--------|-------------|--------|

| Bugs found per hour | Discovery throughput | > 10/hour |

| Unique bug rate | Deduplication effectiveness | > 70% |

| False positive rate | Bugs not confirmed | < 5% |

| Fix verification rate | Bugs with verified fixes | > 80% |

| ROI ratio | Cost savings vs automation cost | > 10x |



### Weekly Report Sections



1. **Bug Discovery Summary**: Bugs found, unique, filed

2. **ROI Metrics**: Hours saved, cost saved, bugs prevented, ROI ratio

3. **Fix Verification**: Fixed, verified, pending counts

4. **By Method**: Breakdown by discovery method

5. **By Severity**: Breakdown by severity

6. **Top Bugs**: Highest priority bugs



## Troubleshooting



### Regression Test Generation Fails



```bash

# Check templates exist

ls backend/tests/bug_discovery/templates/*.j2



# Verify BugReport has required fields

python -c "from tests.bug_discovery.models import BugReport; print(BugReport.__fields__)"

```



### Bug Fix Verification Not Running



```bash

# Check GitHub token has repo scope

# Check workflow is enabled in GitHub Actions

# Verify "fix" label exists in repository

```



### ROI Metrics Not Appearing



```bash

# Check metrics.db exists

ls backend/tests/bug_discovery/storage/metrics.db



# Verify ROI tracking enabled

python -c "from tests.bug_discovery.core import DiscoveryCoordinator; c = DiscoveryCoordinator(...); print(c.enable_roi_tracking)"

```



## Further Reading



- [Bug Discovery Infrastructure](./docs/BUG_DISCOVERY_INFRASTRUCTURE.md)

- [Feedback Loops Documentation](./feedback_loops/README.md)

- [Property-Based Testing](./property_tests/README.md)

- [Chaos Engineering](./chaos/README.md)

- [Fuzzing Guide](./fuzzing/README.md)



## Contributing



When adding new discovery methods:



1. Create BugReport objects with required fields

2. Add method to DiscoveryMethod enum

3. Implement aggregation in ResultAggregator

4. Update DashboardGenerator templates

5. Add regression test template