tests/mutation_tests/MUTATION_TESTING_GUIDE.md

# Mutation Testing Guide

**Last Updated:** February 7, 2026
**Version:** 1.0.0

---

## Table of Contents

1. [What is Mutation Testing?](#what-is-mutation-testing)
2. [Why Mutation Testing?](#why-mutation-testing)
3. [Quick Start](#quick-start)
4. [Configuration](#configuration)
5. [Running Mutation Tests](#running-mutation-tests)
6. [Interpreting Results](#interpreting-results)
7. [Quality Gates](#quality-gates)
8. [Best Practices](#best-practices)

---

## What is Mutation Testing?

Mutation testing is a technique for evaluating the quality of your test suite. It makes small changes (mutations) to your code and checks if your tests can detect them.

### How It Works

1. **Generate Mutants**: Mutmut introduces small bugs (mutations) into your code
   ```python

   # Original code

   if user.is_admin:

       grant_access()



   # Mutated code (conditional boundary mutation)

   if not user.is_admin:  # Changed: if → if not

       grant_access()

   ```

2. **Run Tests**: Mutmut runs your test suite against each mutant

3. **Classify Results**:
   - **Killed**: Tests detected the mutation (tests failed)
   - **Survived**: Tests didn't detect the mutation (tests still passed)
   - **Timeout**: Tests hung (potential performance issue)

4. **Calculate Score**:
   ```

   Mutation Score = (Killed Mutants / Total Mutants) × 100

   ```

### Common Mutation Types

| Type | Example | Description |
|------|---------|-------------|
| Arithmetic | `x + y` → `x - y` | Change operator |
| Boolean | `a and b` → `a or b` | Change logic |
| Comparison | `x < y` → `x <= y` | Change comparison |
| Conditional | `if condition:` → `if not condition:` | Negate condition |
| Statement | `return x` → `return None` | Remove statement |

---

## Why Mutation Testing?

### Problems with Traditional Coverage Metrics

**Code Coverage:**
```python

def is_admin(user):

    return True  # Bug: always returns True



# Test:

assert is_admin(user)  # 100% coverage, but bug not detected!

```

**Mutation Testing:**
- Changes `return True` to `return False`
- Test still passes → **Surviving mutant detected** → **Test gap found**

### Benefits

1. **Find Test Gaps**: Surviving mutants indicate missing tests
2. **Improve Test Quality**: Forces you to write better assertions
3. **Validate Edge Cases**: Mutations often test boundary conditions
4. **Measure Test Effectiveness**: More accurate than code coverage

### When to Use

- **Critical Code**: Financial operations, security, data validation
- **High Risk**: Authentication, authorization, payment processing
- **Complex Logic**: Algorithms, state machines, workflows
- **Refactoring**: Before refactoring, ensure tests are solid

---

## Quick Start

### Installation

```bash

# Install mutmut

pip install mutmut>=2.4.0



# Verify installation

mutmut --version

```

### Basic Usage

```bash

# Run mutation tests for a single module

mutmut run --paths-to-mutate core/security.py --runner "pytest tests/"



# Generate HTML report

mutmut html



# View results

mutmut results

```

### Using the Run Scripts

```bash

# Run specific target

python tests/mutation_tests/scripts/run_mutation_tests.py --target priority_p0_financial



# Run all targets

python tests/mutation_tests/scripts/run_mutation_tests.py --all



# Quick smoke test (fewer mutations)

python tests/mutation_tests/scripts/run_mutation_tests.py --all --quick



# Generate report

python tests/mutation_tests/scripts/generate_mutation_report.py

```

---

## Configuration

### Target Configuration (`TARGETS.ini`)

```ini

[priority_p0_financial]

modules = [

    "core/financial_ops_engine.py",

    "core/financial_forensics.py"

]

mutation_score_threshold = 95.0

description = "Financial operations"

risk_level = "CRITICAL"

```

**Parameters:**
- `modules`: List of Python files to mutate
- `mutation_score_threshold`: Minimum required score (0-100)
- `description`: Human-readable description
- `risk_level`: CRITICAL, HIGH, MEDIUM, or LOW

### Mutmut Configuration (`mutmut.ini`)

Already created in `backend/tests/mutation_tests/config/mutmut.ini` with:
- Test runner command
- Paths to mutate/exclude
- Mutation types
- Worker settings

---

## Running Mutation Tests

### Phase 5 Workflow

#### 1. Initial Run

```bash

# Run mutation tests for P0 (Financial & Security)

python tests/mutation_tests/scripts/run_mutation_tests.py --target priority_p0_financial

```

**Expected Output:**
```

============================================================

Running mutation tests for: priority_p0_financial

Modules: core/financial_ops_engine.py, core/financial_forensics.py

Score threshold: 95.0%

============================================================



- Mutation testing started ...

- 127 mutants generated

- 121 mutants killed

- 6 mutants survived

- 0 mutants timed out



Mutation Score: 95.28%

Threshold: 95.0%



============================================================

✅ PASSED: Score 95.28% >= 95.0%

============================================================

```

#### 2. Analyze Surviving Mutants

If score is below threshold:
```bash

# View surviving mutants

mutmut results | grep SURVIVED



# Apply surviving mutant for inspection

mutmut apply <mutant_id>



# View mutated code

cat core/financial_ops_engine.py | grep -A 10 -B 10 "MUTATION"



# Write test to kill mutant

# Add property test, re-run

```

#### 3. Generate Report

```bash

# Generate HTML report

python tests/mutation_tests/scripts/generate_mutation_report.py



# View report

open tests/mutation_tests/reports/mutation_report.html

```

#### 4. CI/CD Integration

**Weekly Workflow (`.github/workflows/mutation-tests.yml`)**:
- Runs every Sunday at 3 AM UTC
- Tests all priority targets
- Fails if any target below threshold
- Generates report artifact

---

## Interpreting Results

### Mutation Score Categories

| Score | Quality | Action |
|-------|---------|--------|
| 95-100% | Excellent | ✅ No action needed |
| 90-94% | Good | ⚠️ Review surviving mutants |
| 80-89% | Fair | ⚠️ Add tests to kill survivors |
| <80% | Poor | ❌ Significant test gaps |

### Surviving Mutant Analysis

**Example:**
```

⚠️  SURVIVED: core/security.py:42 (ARITHMETIC)

---

Original:

    if attempts >= 3:

        block_user()



Mutated:

    if attempts > 3:  # Changed: >= to >

        block_user()

```

**Why Survived?**
- Tests only check `attempts == 3` or `attempts > 3`
- No test for `attempts == 4` (boundary condition)

**Fix:**
```python

# Add property test

@given(attempts=st.integers(min_value=0, max_value=10))

def test_block_after_3_attempts(attempts):

    if attempts >= 3:

        assert user.is_blocked

```

### Common Test Gaps

1. **Boundary Conditions**: `x > 3` vs `x >= 3`
2. **Null/None Handling**: Missing null checks
3. **Empty Collections**: Empty list/dict handling
4. **Off-by-One**: Loop boundaries, array indices
5. **Logic Inversions**: `and` vs `or`, `not` missing

---

## Quality Gates

### Score Thresholds by Priority

| Priority | Target | Rationale |
|----------|--------|-----------|
| P0: Financial & Security | >95% | Highest risk (cost, security) |
| P1: Core Business Logic | >90% | High risk (data integrity) |
| P2: API & Tools | >85% | Medium risk (UX, functionality) |
| P3: Other | >80% | Lower risk (nice-to-have) |

### CI/CD Gates

**Weekly Mutation Test** (`.github/workflows/mutation-tests.yml`):
```yaml

name: Mutation Tests

on:

  schedule:

    - cron: '0 3 * * 0'  # Sunday 3 AM UTC



jobs:

  mutation-tests:

    runs-on: ubuntu-latest

    steps:

      - name: Run mutation tests

        run: |

          python tests/mutation_tests/scripts/run_mutation_tests.py --all



      - name: Upload mutation report

        uses: actions/upload-artifact@v4

        with:

          name: mutation-report

          path: html/

```

### Failing the Build

If mutation score below threshold:
```yaml

- name: Check mutation score

  run: |

    SCORE=$(mutmut results | grep "Mutation score" | awk '{print $3}' | sed 's/%//')



    if (( $(echo "$SCORE < 95" | bc -l) )); then

      echo "❌ Mutation score $SCORE% below threshold 95%"

      exit 1

    fi

```

---

## Best Practices

### 1. Start Small

Don't try to mutate everything at once. Start with:
- Critical paths (authentication, payments)
- High-risk code (security, financial)
- Recently changed code (regression testing)

### 2. Use Property-Based Tests

Property-based tests (Hypothesis) are great for killing mutants:
- Generate many random inputs
- Test edge cases automatically
- Find boundary conditions

```python

@given(x=st.integers(min_value=0, max_value=100))

def test_boundary(x):

    # Will test 0, 1, 2, ..., 100

    # Likely to catch off-by-one mutations

    assert is_valid(x)

```

### 3. Fix Surviving Mutants Immediately

When you find a surviving mutant:
1. ✅ Write a regression test **before** fixing code
2. ✅ Verify test fails on mutant
3. ✅ Fix test (if test bug) or code (if code bug)
4. ✅ Verify test now kills mutant
5. ✅ Commit with detailed message

### 4. Review Test Assertions

Surviving mutants often indicate weak assertions:
```python

# Weak assertion (won't catch mutations)

def test_add():

    result = add(2, 3)

    assert result is not None  # ❌ Too weak



# Strong assertion (will catch mutations)

def test_add():

    result = add(2, 3)

    assert result == 5  # ✅ Specific assertion

```

### 5. Balance Mutation Types

Not all mutations are equally useful:
- **High Value**: Arithmetic, boolean, comparison (business logic)
- **Medium Value**: Conditional, statement (control flow)
- **Low Value**: String literals, comments (cosmetic)

Focus on high-value mutations first.

### 6. Consider Performance

Mutation testing can be slow:
- **1000 mutants** × **10 min test suite** = **10,000 min** ≈ **7 days**

**Optimizations:**
- Use parallel workers (`mutmut run --workers=auto`)
- Reduce Hypothesis examples (`hypothesis_max_examples = 50`)
- Test only critical paths (P0, P1)
- Use incremental mutation testing (only new code)

### 7. Set Realistic Targets

100% mutation score is rarely practical:
- **90-95%**: Good for critical code
- **80-85%**: Acceptable for most code
- **<80%**: Needs improvement

Focus on critical code quality over perfect scores.

---

## Troubleshooting

### Issue: "ImportError: No module named 'mutmut'"

**Solution:**
```bash

pip install mutmut>=2.4.0

```

### Issue: "Mutations too slow"

**Solution:**
```bash

# Use quick mode (fewer mutations)

python run_mutation_tests.py --quick



# Limit number of mutants

mutmut run --max-mutations 100



# Use parallel workers

mutmut run --workers 4

```

### Issue: "Too many surviving mutants"

**Solution:**
1. Run with verbose output: `mutmut run --verbose`
2. Identify patterns in surviving mutants
3. Add property-based tests for those patterns
4. Re-run mutation testing

### Issue: "Mutant causes test to hang"

**Solution:**
```bash

# Kill process

pkill -9 mutmut



# Use timeout

mutmut run --timeout=30  # 30 second timeout per mutant

```

---

## Resources

- [Mutmut Documentation](https://mutmut.readthedocs.io/)
- [Mutation Testing on Wikipedia](https://en.wikipedia.org/wiki/Mutation_testing)
- [Property-Based Testing with Hypothesis](https://hypothesis.readthedocs.io/)
- [Testing Guide](../TESTING_GUIDE.md)

---

**Last Updated:** February 7, 2026
**Version:** 1.0.0