tests/docs/COVERAGE_TRENDING_GUIDE.md

# Coverage Trending Guide

**Last Updated:** March 7, 2026
**Purpose:** Track coverage changes over time, detect regressions, visualize trends
**Data Retention:** 30-day rolling history, 90-day regression log

---

## Overview

The Coverage Trending Infrastructure provides automated tracking of test coverage across all platforms (Backend, Frontend, Mobile, Desktop) with weighted overall coverage calculation. This system enables:

- **Historical Tracking:** Per-commit coverage data with platform breakdown
- **Regression Detection:** Automatic identification of coverage drops >1% threshold
- **Visual Dashboard:** Self-contained HTML dashboard with matplotlib charts
- **CI/CD Integration:** Automated trending on every push/PR to main/develop
- **External Analysis:** Export trending data to CSV/JSON/Excel formats

### Components

| Component | Purpose | Location |
|-----------|---------|----------|
| **Trend Analyzer** | Detect regressions, validate data integrity | `backend/tests/scripts/coverage_trend_analyzer.py` |
| **Dashboard Generator** | Create HTML visualization with matplotlib charts | `backend/tests/scripts/generate_coverage_dashboard.py` |
| **CI/CD Workflow** | Automated trending on every build | `.github/workflows/coverage-trending.yml` |
| **Export Utility** | Export data for external analysis | `backend/tests/scripts/coverage_trend_export.py` |

### Platform Coverage Weights

| Platform | Weight | Rationale |
|----------|--------|-----------|
| Backend | 40% | Core business logic, API contracts, data layer |
| Frontend | 30% | User interface, client-side validation |
| Mobile | 20% | React Native app, device integration |
| Desktop | 10% | Tauri desktop app, platform-specific features |

**Overall Coverage Formula:**
```
overall = (backend * 0.40) + (frontend * 0.30) + (mobile * 0.20) + (desktop * 0.10)
```

### Data Retention Policy

- **Trending History:** 30-day rolling window (auto-pruned via `--prune` flag)
- **Regression Log:** 90-day retention (persistent log for audit trails)
- **CI/CD Artifacts:** 30-day retention (GitHub Actions default)
- **Dashboard HTML:** Self-contained, no external dependencies, permanent storage

---

## Quick Start

### Local Development Setup

**1. Install Dependencies**

```bash
# Core dependencies (required)
pip install matplotlib pandas jsonschema

# Optional: Excel export support
pip install openpyxl
```

**2. Run Trend Analyzer**

```bash
# Basic regression detection
cd backend
python tests/scripts/coverage_trend_analyzer.py \
  --trending-file tests/coverage_reports/metrics/cross_platform_trend.json

# Custom regression threshold
python tests/scripts/coverage_trend_analyzer.py \
  --regression-threshold 2.0 \
  --format markdown

# JSON output for CI/CD integration
python tests/scripts/coverage_trend_analyzer.py \
  --format json \
  --output tests/coverage_reports/metrics/coverage_regressions.json
```

**3. Generate Dashboard**

```bash
# Generate 30-day trend dashboard
python tests/scripts/generate_coverage_dashboard.py \
  --trending-file tests/coverage_reports/metrics/cross_platform_trend.json \
  --output tests/coverage_reports/dashboards/coverage_trend_30d.html \
  --days 30

# Custom dimensions (larger charts)
python tests/scripts/generate_coverage_dashboard.py \
  --trending-file tests/coverage_reports/metrics/cross_platform_trend.json \
  --output tests/coverage_reports/dashboards/coverage_trend_30d.html \
  --days 30 \
  --width 1600 \
  --height 900
```

**4. Export Historical Data**

```bash
# Export last 30 days to CSV
python tests/scripts/coverage_trend_export.py \
  --format csv \
  --days 30 \
  --output coverage_export.csv

# Export all history to JSON
python tests/scripts/coverage_trend_export.py \
  --format json \
  --days 0 \
  --output coverage_export_all.json

# Export to Excel (requires openpyxl)
python tests/scripts/coverage_trend_export.py \
  --format excel \
  --days 90 \
  --output coverage_export_quarterly.xlsx
```

### CI/CD Setup

**Workflow Triggers:**
- Push to `main` or `develop` branches
- Pull requests to `main` or `develop` branches
- Manual dispatch via GitHub Actions UI

**Artifacts Available:**
- `coverage-trend-dashboard` (HTML) - 30-day retention
- `coverage-trending-data` (JSON files) - 30-day retention

**Job Summaries:**
- Automatic posting to GitHub Actions UI
- Trend report with +/- indicators per platform
- Regression alerts with severity levels (warning/critical)

**No Configuration Required:**
- Uses existing coverage files from `unified-tests-parallel.yml`
- Downloads artifacts automatically from 4 platform jobs
- Generates dashboard and uploads artifacts
- Posts trend report as job summary

---

## Script Reference

### coverage_trend_analyzer.py

**Purpose:** Detect significant coverage regressions and validate historical data integrity.

**Usage:**
```bash
python coverage_trend_analyzer.py [options]
```

**Options:**

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `--trending-file PATH` | string | `tests/coverage_reports/metrics/cross_platform_trend.json` | Path to trending data file |
| `--regression-threshold FLOAT` | float | `1.0` | Regression threshold in percentage points |
| `--output PATH` | string | `tests/coverage_reports/metrics/coverage_regressions.json` | Path to regression output file |
| `--periods INT` | int | `7` | Number of periods to compare for trend |
| `--format FORMAT` | string | `text` | Output format: `text`, `json`, `markdown` |

**Exit Codes:**
- `0` - Success (no critical regressions)
- `1` - Critical regressions detected (threshold >5%)

**Examples:**

```bash
# Text output (human-readable)
python coverage_trend_analyzer.py --format text

# JSON output (CI/CD integration)
python coverage_trend_analyzer.py --format json --output regressions.json

# Markdown output (PR comments)
python coverage_trend_analyzer.py --format markdown --regression-threshold 2.0

# Custom trending file
python coverage_trend_analyzer.py --trending-file /path/to/custom_trend.json
```

**Output Format (Text):**
```
Coverage Trend Analysis
========================

Current Coverage: 72.3% (overall)
Historical Average (7 periods): 71.8%
Trend: +0.5% (improving)

Platform Breakdown:
- Backend: 85.2% (+1.2%)
- Frontend: 68.4% (-0.8%)
- Mobile: 52.1% (+0.3%)
- Desktop: 78.9% (+0.5%)

Regressions Detected: 2
⚠️ WARNING: Frontend coverage dropped 0.8% (threshold: 1.0%)
⚠️ WARNING: Mobile coverage dropped 1.5% (threshold: 1.0%)
```

**Output Format (JSON):**
```json
{
  "current_coverage": 72.3,
  "historical_average": 71.8,
  "trend_delta": 0.5,
  "platforms": {
    "backend": {"current": 85.2, "delta": 1.2},
    "frontend": {"current": 68.4, "delta": -0.8},
    "mobile": {"current": 52.1, "delta": -1.5},
    "desktop": {"current": 78.9, "delta": 0.5}
  },
  "regressions": [
    {
      "platform": "mobile",
      "current_coverage": 52.1,
      "previous_coverage": 53.6,
      "delta": -1.5,
      "severity": "warning",
      "detected_at": "2026-03-07T19:30:00Z"
    }
  ]
}
```

---

### generate_coverage_dashboard.py

**Purpose:** Generate self-contained HTML dashboard with matplotlib charts for coverage visualization.

**Usage:**
```bash
python generate_coverage_dashboard.py [options]
```

**Options:**

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `--trending-file PATH` | string | `tests/coverage_reports/metrics/cross_platform_trend.json` | Path to trending data file |
| `--output PATH` | string | `coverage_trend_30d.html` | Path to output HTML file |
| `--days INT` | int | `30` | Number of days to include in dashboard |
| `--width INT` | int | `1200` | Chart width in pixels |
| `--height INT` | int | `600` | Chart height in pixels |

**Output:**
- Self-contained HTML file with embedded CSS/JavaScript
- Matplotlib charts encoded as base64 images
- No external dependencies (works offline)
- Responsive design for desktop/tablet/mobile viewing

**Examples:**

```bash
# Generate 30-day dashboard (default)
python generate_coverage_dashboard.py

# Generate 90-day dashboard
python generate_coverage_dashboard.py --days 90 --output coverage_trend_90d.html

# High-resolution dashboard (4K displays)
python generate_coverage_dashboard.py --width 2560 --height 1440

# Custom trending file
python generate_coverage_dashboard.py --trending-file /path/to/custom_trend.json
```

**Dashboard Sections:**

1. **Overall Coverage Trend** - Line chart showing weighted overall coverage over time
2. **Platform Breakdown** - Stacked area chart with per-platform coverage
3. **Regression Events** - Scatter plot highlighting regression points
4. **Summary Statistics** - Table with min/max/avg/current coverage per platform
5. **Commit History** - Timeline of coverage changes with commit SHAs

---

### coverage_trend_export.py

**Purpose:** Export trending data for external analysis (Excel, BI tools, custom scripts).

**Usage:**
```bash
python coverage_trend_export.py [options]
```

**Options:**

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `--trending-file PATH` | string | `tests/coverage_reports/metrics/cross_platform_trend.json` | Path to trending data file |
| `--output PATH` | string | `coverage_export.csv` | Path to output file |
| `--format FORMAT` | string | `csv` | Export format: `csv`, `json`, `excel` |
| `--days INT` | int | `30` | Number of days to export (0 for all history) |

**Export Formats:**

**CSV Format:**
```csv
timestamp,overall_coverage,backend,frontend,mobile,desktop,commit_sha,branch
2026-03-07T19:30:00Z,72.3,85.2,68.4,52.1,78.9,abc123,main
2026-03-06T14:15:00Z,71.8,84.0,69.2,51.8,78.4,def456,develop
...
```

**JSON Format:**
```json
{
  "export_time": "2026-03-07T19:30:00Z",
  "total_entries": 30,
  "filtered_entries": 30,
  "date_range": {
    "start": "2026-02-06T00:00:00Z",
    "end": "2026-03-07T19:30:00Z"
  },
  "history": [
    {
      "timestamp": "2026-03-07T19:30:00Z",
      "overall_coverage": 72.3,
      "platforms": {
        "backend": 85.2,
        "frontend": 68.4,
        "mobile": 52.1,
        "desktop": 78.9
      },
      "commit_sha": "abc123",
      "branch": "main"
    }
  ]
}
```

**Excel Format:**
- **Sheet 1: Summary** - Overall stats, platform breakdown, min/max/avg/current
- **Sheet 2: History** - Time series data with all columns
- Formatted with headers, column widths, number formats
- Requires `openpyxl` dependency (`pip install openpyxl`)

**Examples:**

```bash
# Export last 30 days to CSV
python coverage_trend_export.py --format csv --days 30

# Export all history to JSON
python coverage_trend_export.py --format json --days 0

# Export last 90 days to Excel
python coverage_trend_export.py --format excel --days 90 --output quarterly_report.xlsx

# Custom output path
python coverage_trend_export.py --output /tmp/coverage_data.csv
```

---

## CI/CD Integration

### Workflow: coverage-trending.yml

**Location:** `.github/workflows/coverage-trending.yml`

**Triggers:**
- `push` to `main` or `develop` branches
- `pull_request` to `main` or `develop` branches
- `workflow_dispatch` (manual trigger via GitHub Actions UI)

**Dependencies:**
- `unified-tests-parallel.yml` (test-platform job)
- Waits for all 4 platform tests to complete (backend, frontend, mobile, desktop)

**Workflow Steps:**

1. **Download Artifacts** - Downloads coverage artifacts from all 4 platform jobs
2. **Run Cross-Platform Coverage Gate** - Computes weighted overall coverage
3. **Update Trending Data** - Appends new entry with commit SHA and branch tracking
4. **Detect Regressions** - Analyzes trends for significant drops (>1% threshold)
5. **Generate Dashboard** - Creates 30-day trend HTML dashboard
6. **Upload Artifacts** - Uploads dashboard and trending data (30-day retention)
7. **Post Job Summary** - Creates GitHub Actions job summary with trend report
8. **Post Regression Alerts** - Comments on PR with regression details
9. **Fail Build** - Exits with error if critical regressions detected (>5% threshold)

**Artifacts:**

| Artifact | Contents | Retention |
|----------|----------|-----------|
| `coverage-trend-dashboard` | HTML dashboard (30-day trend) | 30 days |
| `coverage-trending-data` | `cross_platform_trend.json`, `coverage_regressions.json` | 30 days |

**Job Summaries:**
- Automatic posting to GitHub Actions UI
- Includes overall coverage trend, platform breakdown, regression events
- Accessible via "Summary" button in workflow run page

**PR Comments:**
- Automatic posting on pull_request events
- Includes coverage trend with +/- indicators per platform
- Links to dashboard artifact for detailed visualization
- Regression alerts with severity levels

**Regression Alerts:**
- **Warning:** Coverage drop >1% threshold (logged, doesn't fail build)
- **Critical:** Coverage drop >5% threshold (fails build on main branch)
- Includes platform name, current/previous coverage, delta percentage

---

## Data Schema

### cross_platform_trend.json

**Structure:**
```json
{
  "history": [
    {
      "timestamp": "2026-03-07T19:30:00Z",
      "overall_coverage": 72.3,
      "platforms": {
        "backend": 85.2,
        "frontend": 68.4,
        "mobile": 52.1,
        "desktop": 78.9
      },
      "thresholds": {
        "backend": 80.0,
        "frontend": 80.0,
        "mobile": 60.0,
        "desktop": 70.0
      },
      "commit_sha": "abc123def456",
      "branch": "main"
    }
  ],
  "latest": {
    "timestamp": "2026-03-07T19:30:00Z",
    "overall_coverage": 72.3,
    "platforms": {
      "backend": 85.2,
      "frontend": 68.4,
      "mobile": 52.1,
      "desktop": 78.9
    }
  },
  "platform_trends": {
    "backend": {
      "current": 85.2,
      "average": 84.5,
      "min": 82.1,
      "max": 86.0
    }
  },
  "computed_weights": {
    "backend": 0.4,
    "frontend": 0.3,
    "mobile": 0.2,
    "desktop": 0.1
  }
}
```

**Entry Structure:**

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `timestamp` | string | Yes | ISO 8601 timestamp with Z suffix (UTC) |
| `overall_coverage` | float | Yes | Weighted overall coverage percentage (0-100) |
| `platforms` | object | Yes | Per-platform coverage dict (backend, frontend, mobile, desktop) |
| `thresholds` | object | Yes | Platform thresholds at time of recording |
| `commit_sha` | string | No | Git commit SHA (optional, for CI/CD tracking) |
| `branch` | string | No | Git branch name (optional, for CI/CD tracking) |

---

### coverage_regressions.json

**Structure:**
```json
{
  "regressions": [
    {
      "platform": "mobile",
      "current_coverage": 52.1,
      "previous_coverage": 53.6,
      "delta": -1.5,
      "severity": "warning",
      "detected_at": "2026-03-07T19:30:00Z",
      "commit_sha": "abc123def456"
    }
  ],
  "metadata": {
    "regression_threshold": 1.0,
    "critical_threshold": 5.0,
    "retention_days": 90,
    "last_updated": "2026-03-07T19:30:00Z"
  }
}
```

**Regression Entry Structure:**

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `platform` | string | Yes | Affected platform name (backend, frontend, mobile, desktop) |
| `current_coverage` | float | Yes | Coverage after regression (percentage) |
| `previous_coverage` | float | Yes | Coverage before regression (percentage) |
| `delta` | float | Yes | Percentage change (negative for regression) |
| `severity` | string | Yes | Severity level: `warning` (>1%), `critical` (>5%) |
| `detected_at` | string | Yes | ISO 8601 timestamp when regression detected |
| `commit_sha` | string | No | Associated commit SHA (optional) |

---

## Troubleshooting

### Missing Coverage Files

**Symptoms:**
- Trend analyzer reports "No trending data found"
- Dashboard shows empty charts
- CI/CD workflow fails to download artifacts

**Solutions:**

1. **Check unified-tests-parallel.yml artifact uploads:**
   ```bash
   # Verify workflow uploaded artifacts
   gh run list --workflow=unified-tests-parallel.yml --limit 5
   gh run view <run-id> --log
   ```

2. **Verify platform test jobs completed successfully:**
   - Check GitHub Actions UI for platform job status
   - Look for "backend-coverage", "frontend-coverage", "mobile-coverage", "desktop-coverage" artifacts
   - Re-run failed jobs if needed

3. **Check artifact retention policy:**
   - GitHub Actions artifacts have 7-day default retention
   - coverage-trending.yml extends retention to 30 days
   - Artifacts >30 days old are automatically deleted

4. **Manual artifact download (debug):**
   ```bash
   # Download artifact via GitHub CLI
   gh run download <run-id> -n backend-coverage

   # Verify coverage file exists
   ls -la coverage/backend/coverage.json
   ```

---

### Dashboard Not Rendering

**Symptoms:**
- HTML file opens but shows blank page
- Charts display as broken images
- JavaScript errors in browser console

**Solutions:**

1. **Verify matplotlib installed:**
   ```bash
   pip show matplotlib
   # If not installed:
   pip install matplotlib
   ```

2. **Check HTML file size:**
   ```bash
   # HTML should be >50KB (contains base64-encoded charts)
   ls -lh coverage_trend_30d.html
   # If <10KB, charts may not have been generated
   ```

3. **Open browser console for JavaScript errors:**
   - Chrome/Edge: F12 → Console tab
   - Firefox: F12 → Console tab
   - Look for errors like "Uncaught ReferenceError" or "Failed to load resource"

4. **Regenerate dashboard with verbose output:**
   ```bash
   python generate_coverage_dashboard.py --days 30 --output test.html
   # Check console output for matplotlib errors
   ```

---

### False Positive Regressions

**Symptoms:**
- Regression alerts for normal coverage fluctuations
- Threshold too sensitive for platform with high variance
- Regression detected immediately after adding new tests

**Solutions:**

1. **Adjust regression threshold:**
   ```bash
   # Increase threshold from 1.0% to 2.0%
   python coverage_trend_analyzer.py --regression-threshold 2.0
   ```

2. **Check for flaky test coverage:**
   - Some tests have non-deterministic coverage (random data, time-based logic)
   - Exclude flaky tests from coverage measurement using `.coveragerc`
   - Stabilize tests with mocking/fixed data

3. **Verify 30-day pruning not removing valid data:**
   ```bash
   # Check trending data file size
   ls -lh cross_platform_trend.json
   # Should have 30+ entries (one per day)
   # If only 1-2 entries, pruning may be too aggressive
   ```

4. **Use moving average for smoothing:**
   - Trend analyzer uses 7-period moving average by default
   - Increase `--periods` to 14 or 30 for smoother trends
   ```bash
   python coverage_trend_analyzer.py --periods 14
   ```

---

### CI/CD Workflow Not Triggering

**Symptoms:**
- Workflow doesn't run on push/PR
- Manual dispatch option not available
- Workflow fails immediately with "Workflow not found"

**Solutions:**

1. **Check branch name:**
   - Workflow only triggers on `main` and `develop` branches
   - Feature branches (e.g., `feature/new-ui`) won't trigger trending
   - Update workflow triggers if needed:
     ```yaml
     on:
       push:
         branches: [main, develop, feature/*]
     ```

2. **Verify workflow file syntax:**
   ```bash
   # Validate YAML syntax
   yamllint .github/workflows/coverage-trending.yml
   # Or use GitHub CLI:
   gh workflow view coverage-trending.yml --yaml
   ```

3. **Check GitHub Actions permissions:**
   - Repository settings → Actions → General → Workflow permissions
   - Ensure "Read and write permissions" is enabled
   - Check "Allow GitHub Actions to create and approve pull requests"

4. **Manual workflow trigger (debug):**
   ```bash
   # Trigger workflow manually
   gh workflow run coverage-trending.yml
   # View workflow run logs
   gh run list --workflow=coverage-trending.yml --limit 1
   gh run view <run-id> --log
   ```

---

## Best Practices

### Running Trending Analysis

- **Run after every significant code change** - Catch regressions early
- **Review dashboard weekly** - Look for trend patterns (gradual decline, plateau)
- **Investigate regressions >2% immediately** - Small drops compound over time
- **Archive historical data monthly** - Download full export for long-term analysis

### Adjusting Platform Weights

- **Review quarterly** - Business priorities change (new platforms, deprecation)
- **Update in `cross_platform_coverage_gate.py`** - Modify `PLATFORM_WEIGHTS` dict
- **Document rationale** - Add comment explaining weight change
- **Communicate to team** - Update team on new overall coverage calculation

### Regression Thresholds

- **Keep at 1% for early warning** - Catches small drops before they compound
- **Use 5% for critical failures** - Fails main branch builds on significant regressions
- **Adjust per-platform if needed** - Some platforms have higher variance (mobile, desktop)
- **Document threshold changes** - Track in ROADMAP.md or project wiki

### Data Retention

- **30-day trending history** - Sufficient for short-term trend analysis
- **90-day regression log** - Audit trail for quality metrics
- **Monthly archives** - Export full history to JSON for permanent storage
- **CI/CD artifacts** - 30-day GitHub Actions retention (extendable to 90 days)

---

## Reference

### Platform-Specific Testing Guides

- **[Frontend Testing Guide](../../docs/FRONTEND_TESTING_GUIDE.md)** - Jest coverage targets and per-module thresholds
- **[Mobile Testing Guide](../../docs/MOBILE_TESTING_GUIDE.md)** - jest-expo coverage for React Native
- **[Desktop Testing Guide](../../docs/DESKTOP_TESTING_GUIDE.md)** - tarpaulin coverage for Rust/Tauri

### Related Documentation

- **PARALLEL_EXECUTION_GUIDE.md** - Test execution strategy and CI/CD workflow
- **CROSS_PLATFORM_COVERAGE.md** - Coverage thresholds and weighted calculation
- **ROADMAP.md** - Phase 150 quality infrastructure details

### Related Scripts

- **cross_platform_coverage_gate.py** - Coverage threshold enforcement
- **update_cross_platform_trending.py** - Data tracking and commit history
- **ci_status_aggregator.py** - Test results aggregation

### File Locations

| Type | Location |
|------|----------|
| **Scripts** | `backend/tests/scripts/` |
| **Data** | `backend/tests/coverage_reports/metrics/` |
| **Dashboards** | `backend/tests/coverage_reports/dashboards/` |
| **Docs** | `backend/tests/docs/` |
| **Workflows** | `.github/workflows/` |

### Quick Command Reference

```bash
# Run trend analyzer
python backend/tests/scripts/coverage_trend_analyzer.py

# Generate dashboard
python backend/tests/scripts/generate_coverage_dashboard.py --days 30

# Export data
python backend/tests/scripts/coverage_trend_export.py --format csv --days 30

# CI/CD workflow trigger
gh workflow run coverage-trending.yml

# View latest workflow run
gh run list --workflow=coverage-trending.yml --limit 1
gh run view $(gh run list --workflow=coverage-trending.yml --limit 1 --json databaseId --jq '.[0].databaseId') --log
```

---

**Last Updated:** March 7, 2026
**Maintained By:** Atom Quality Infrastructure Team
**Questions?** See ROADMAP.md Phase 150 or open GitHub issue