tests/docs/PARALLEL_EXECUTION_GUIDE.md

# Parallel Test Execution Guide

**Last Updated:** March 7, 2026
**Target:** <15 minute total test suite execution time
**Strategy:** Matrix-based parallel execution across 4 platforms (backend, frontend, mobile, desktop)

---

## Overview

This guide explains the parallel test execution strategy for Atom's cross-platform test suite. The goal is to reduce total CI/CD feedback time from 30+ minutes (sequential execution) to <15 minutes (parallel execution) using GitHub Actions matrix strategy with platform-specific test runners.

### Key Metrics

| Metric | Target | Current Baseline | Status |
|--------|--------|------------------|--------|
| **Total CI Duration** | <15 minutes | ~13-15 minutes | ✅ ON TRACK |
| **Backend Tests** | <10 minutes | ~8-10 minutes | ✅ PASSING |
| **Frontend Tests** | <5 minutes | ~3-5 minutes | ✅ PASSING |
| **Mobile Tests** | <3 minutes | ~2-3 minutes | ✅ PASSING |
| **Desktop Tests** | <4 minutes | ~3-4 minutes | ✅ PASSING |

### Parallel Execution Strategy

- **Matrix Strategy:** Single workflow file (`unified-tests-parallel.yml`) with 4 platform jobs running in parallel
- **Fail-Fast Disabled:** All platform jobs complete even if one fails (collect complete results)
- **Max Parallel:** 4 concurrent jobs (one per platform) to avoid resource exhaustion
- **Aggregation Job:** Combines results from all platforms into unified status report
- **Retry Workflow:** Platform-specific re-runs only for failed tests (not full suite)

### Architecture

```

┌─────────────────────────────────────────────────────────────┐

│ unified-tests-parallel.yml (Matrix Strategy)                │

├─────────────────────────────────────────────────────────────┤

│  test-platform Job (Matrix: 4 platforms in parallel)        │

│  ├── backend (pytest-xdist, -n auto, 8-10 min)              │

│  ├── frontend (Jest, --maxWorkers=2, 3-5 min)               │

│  ├── mobile (jest-expo, --maxWorkers=2, 2-3 min)            │

│  └── desktop (cargo test, --test-threads=4, 3-4 min)        │

├─────────────────────────────────────────────────────────────┤

│  aggregate-status Job (Depends on test-platform)            │

│  └── ci_status_aggregator.py (unified JSON + markdown)      │

└─────────────────────────────────────────────────────────────┘

                            │

                            ▼

┌─────────────────────────────────────────────────────────────┐

│ platform-retry.yml (Triggered on failure)                   │

├─────────────────────────────────────────────────────────────┤

│  detect-failures Job (Extract failed tests)                 │

│  └── platform_retry_router.py (generate retry commands)     │

├─────────────────────────────────────────────────────────────┤

│  retry-{platform} Jobs (Conditional: only if failed)        │

│  ├── retry-backend (pytest <failed_tests>)                  │

│  ├── retry-frontend (jest --testNamePattern="<tests>")      │

│  ├── retry-mobile (jest-expo --testNamePattern="<tests>")   │

│  └── retry-desktop (cargo test <tests>)                     │

└─────────────────────────────────────────────────────────────┘

```

---

## Quick Start

### Triggering Workflows

**Automatic Triggers:**
- Push to `main` or `develop` branches
- Pull requests to `main` or `develop` branches

**Manual Trigger:**
```bash

# Via GitHub UI: Actions → Unified Tests (Parallel Matrix) → Run workflow

# Via GitHub CLI:

gh workflow run unified-tests-parallel.yml

```

### Viewing Results

**GitHub Actions UI:**
1. Navigate to Actions tab in repository
2. Click on latest "Unified Tests (Parallel Matrix)" workflow run
3. View platform-specific job logs in matrix expansion
4. Check "Aggregate CI Status" job for unified results

**PR Comments:**
- Automatic PR comments with platform breakdown table
- Per-platform pass/fail status with emoji indicators (✅/❌)
- Total duration and pass rate across all platforms

**Artifacts:**
- `ci-status-unified` artifact (30-day retention) contains:
  - `ci_status.json` - Machine-readable unified status
  - `ci_summary.md` - Human-readable markdown summary
  - Platform-specific test results and coverage reports

### Interpreting Status

**Green Checkmark (✅):** All tests passed on all platforms
- Total duration: <15 minutes
- Pass rate: 100% across all platforms
- No action required

**Red X (❌) with Platform Breakdown:** One or more platforms failed
- Check which platform(s) failed in PR comment
- View job logs for failure details
- Platform retry workflow triggers automatically
- Re-run only failed platforms (80% time savings vs full suite)

### Validating <15 Minute Target

**How to Measure Actual CI Execution Time:**

1. **GitHub Actions UI Method:**
   - Navigate to Actions tab in repository
   - Click on latest "Unified Tests (Parallel Matrix)" workflow run
   - Check the "Duration" column at the top of the workflow run page
   - This shows wall-clock time from workflow start to completion

2. **GitHub CLI Method:**
   ```bash

   # View latest workflow run duration

   gh run list --workflow=unified-tests-parallel.yml --limit 1 --json duration,conclusion



   # View detailed timing for each job

   gh run view <run-id> --log | grep "after"

   ```

3. **API Method:**
   ```bash

   # Get workflow run timing via GitHub API

   gh api /repos/owner/repo/actions/runs/<run-id> --jq '.updated_at, .created_at'

   ```

**<15 Minute Target Calculation:**

The target is calculated as: `max(platform_durations) + aggregation_overhead`

- **Backend:** ~8-10 minutes (pytest with -n auto)
- **Frontend:** ~3-5 minutes (Jest with --maxWorkers=2)
- **Mobile:** ~2-3 minutes (jest-expo with --maxWorkers=2)
- **Desktop:** ~3-4 minutes (cargo test with --test-threads=4)
- **Max Platform Duration:** ~10 minutes (backend)
- **Aggregation Overhead:** ~1-2 minutes (artifact downloads + ci_status_aggregator.py)
- **Total Target:** ~10-12 minutes (well under 15 minute limit)

**Current Baseline:** ~13-15 minutes with all platforms running in parallel

**If >15 Minutes: Optimization Recommendations**

1. **Identify Slowest Platform:**
   ```bash

   # Check job timings in GitHub Actions UI

   # Look for the longest-running platform job

   gh run view <run-id> --log | grep -A 5 "Test backend\|Test frontend\|Test mobile\|Test desktop"

   ```

2. **Optimize Slowest Platform:**
   - **Backend (>10 min):** Increase pytest-xdist workers (`-n 4` or `-n 8`) or split unit/integration tests
   - **Frontend (>5 min):** Increase Jest workers (`--maxWorkers=4`) or implement sharding
   - **Mobile (>3 min):** Increase jest-expo workers (`--maxWorkers=4`) or split test suite
   - **Desktop (>4 min):** Increase cargo test threads (`--test-threads=8`) or split lib/test targets

3. **Verify Cache Hit Rate:**
   ```bash

   # Check job logs for cache hit/miss messages

   gh run view <run-id> --log | grep -i "cache restored\|cache not found"



   # Target: >80% cache hit rate for dependencies

   # If <80%, optimize cache keys or increase retention

   ```

4. **Monitor Trend Over Time:**
   - Track CI duration in ci_status.json (add `total_duration_seconds` field)

   - Alert if duration increases by >20% week-over-week

   - Investigate slow test additions (use `pytest --durations` or Jest `--verbose`)



**Example Optimization Process:**



```bash

# Step 1: Identify slowest platform

gh run list --workflow=unified-tests-parallel.yml --limit 5 | grep duration

# Output: 14m 32s (backend: 10m 15s, frontend: 3m 20s, mobile: 2m 10s, desktop: 3m 5s)



# Step 2: Backend is bottleneck (10m 15s)

cd backend

pytest tests/ --durations=20  # Identify slowest tests

# Output: 10.23s test_slow_integration, 8.45s test_e2e_workflow, ...



# Step 3: Optimize backend (increase workers or split tests)

pytest tests/ -n 8  # Increase from -n auto to -n 8

# Result: 7m 30s (2m 45s improvement)



# Step 4: Verify new CI duration

gh run list --workflow=unified-tests-parallel.yml --limit 1

# Output: 11m 45s (under 15 minute target ✅)

```



---



## Platform-Specific Guides



### Backend (Python pytest)



**Test Framework:** pytest with pytest-xdist for parallel execution



**Current Timing:** ~8-10 minutes (baseline measurement)



**Parallel Configuration:**

```bash

pytest tests/ -v -n auto \

  --json-report --json-report-file=pytest_report.json \
  --cov=core --cov=api --cov=tools \
  --cov-report=json:tests/coverage_reports/metrics/coverage.json

```



**Key Parameters:**

- `-n auto`: Automatically detect CPU core count and run parallel workers

- `--json-report`: Generate JSON report for CI aggregation

- `--cov`: Coverage reporting with JSON output



**Optimization Recommendations:**

1. **Test Splitting (if >10 minutes):**

   ```bash

   # Split unit tests and integration tests

   pytest tests/unit/ -n auto --json-report --json-report-file=pytest_unit.json
   pytest tests/integration/ -n auto --json-report --json-report-file=pytest_integration.json

   ```



2. **Load Balancing:** pytest-xdist automatically balances test load across workers

   - Slower tests distributed first

   - Workers receive equal test duration



3. **Flaky Test Handling:**

   ```bash

   # Add automatic retries for flaky tests

   pytest tests/ -n auto --reruns 2 --reruns-delay 1

   ```



**Dependencies:**

```bash

pip install pytest-xdist pytest-json-report pytest-rerunfailures pytest-asyncio httpx

```



**CI/CD Timeout:** 30 minutes (includes dependency installation + test execution)



---



### Frontend (Jest)



**Test Framework:** Jest with JSON reporter and parallel workers



**Current Timing:** ~3-5 minutes (baseline measurement)



**Parallel Configuration:**

```bash

cd frontend-nextjs

npm run test:ci -- --json --outputFile=test-results.json --maxWorkers=2

```



**Key Parameters:**

- `--json`: Generate JSON test results for CI aggregation

- `--maxWorkers=2`: Limit to 2 parallel workers (CI resource constraint)

- `--outputFile`: Specify output file path



**Optimization Recommendations:**

1. **Sharding (if >5 minutes):**

   ```bash

   # Split tests into 4 shards (run 4 CI jobs in parallel)

   jest --json --outputFile=test-results.json --maxWorkers=2 --shard=1/4

   jest --json --outputFile=test-results.json --maxWorkers=2 --shard=2/4

   jest --json --outputFile=test-results.json --maxWorkers=2 --shard=3/4

   jest --json --outputFile=test-results.json --maxWorkers=2 --shard=4/4

   ```



2. **Test Scheduling:** Jest's `--shard` automatically balances tests by count

   - For time-based balancing, use `test-splitter.py` script (see below)



3. **Watch Mode (local development):**

   ```bash

   npm run test:watch -- --maxWorkers=4  # Faster local feedback

   ```



**Dependencies:**

```bash

npm install --save-dev jest @types/jest

```



**CI/CD Timeout:** 20 minutes (includes dependency installation + test execution)



---



### Mobile (jest-expo)



**Test Framework:** jest-expo with JSON reporter and parallel workers



**Current Timing:** ~2-3 minutes (baseline measurement)



**Parallel Configuration:**

```bash

cd mobile

npm run test:ci -- --json --outputFile=test-results.json --maxWorkers=2

```



**Key Parameters:**

- `--json`: Generate JSON test results for CI aggregation

- `--maxWorkers=2`: Limit to 2 parallel workers (CI resource constraint)

- `--outputFile`: Specify output file path



**Optimization Recommendations:**

1. **API-Level Tests Only:** Detox E2E tests are BLOCKED (expo-dev-client requirement)

   - Focus on API-level tests with mocked expo modules

   - 398 tests covering components, services, navigation



2. **Sharding (if >3 minutes):**

   ```bash

   # Split tests into 2 shards

   jest --json --outputFile=test-results.json --maxWorkers=2 --shard=1/2

   jest --json --outputFile=test-results.json --maxWorkers=2 --shard=2/2

   ```



3. **Module Mocking:** Ensure expo modules are properly mocked in `jest.setup.js`

   ```javascript

   // jest.setup.js

   jest.mock('expo-sharing', () => ({

     shareAsync: jest.fn(),

   }));

   ```



**Dependencies:**

```bash

npm install --save-dev jest jest-expo

```



**CI/CD Timeout:** 20 minutes (includes dependency installation + test execution)



---



### Desktop (Tauri cargo test)



**Test Framework:** Rust cargo test with parallel threads



**Current Timing:** ~3-4 minutes (baseline measurement)



**Parallel Configuration:**

```bash

cd frontend-nextjs/src-tauri

cargo test --test-threads=4 -Z unstable-options --format json > cargo_test_results.json 2>&1 || true

```



**Key Parameters:**

- `--test-threads=4`: Use 4 parallel threads for test execution

- `-Z unstable-options --format json`: Enable JSON output format (unstable feature)

- `|| true`: Continue even if tests fail (capture results)



**Optimization Recommendations:**

1. **Test Organization (if >4 minutes):**

   ```bash

   # Split unit tests and integration tests

   cargo test --lib --test-threads=4  # Unit tests only

   cargo test --test '*' --test-threads=4  # Integration tests only

   ```



2. **Conditional Compilation:** Use `#[cfg]` attributes for platform-specific tests

   ```rust

   #[cfg(target_os = "windows")]
   #[test]
   fn test_windows_only() {
       // Windows-specific test code

   }

   ```


3. **Coverage:** Use tarpaulin for code coverage (baseline: 35%, target: 80%)
   ```bash

   cargo tarpaulin --out Json --output-file coverage.json

   ```

**Dependencies:**
```bash

# Rust toolchain managed by rustup

rustup default stable

```

**CI/CD Timeout:** 15 minutes (includes dependency installation + test execution)

---

## Timing Benchmarks

### Baseline Measurements (March 2026)

| Platform | Test Count | Baseline | Parallel | Target | Status |
|----------|------------|----------|----------|--------|--------|
| **Backend** | ~500 tests | 15-20 min | 8-10 min | <10 min | ✅ PASSING |
| **Frontend** | ~1200 tests | 8-10 min | 3-5 min | <5 min | ✅ PASSING |
| **Mobile** | ~398 tests | 5-7 min | 2-3 min | <3 min | ✅ PASSING |
| **Desktop** | ~83 tests | 6-8 min | 3-4 min | <4 min | ✅ PASSING |
| **Aggregate** | ~2181 tests | 30-45 min | 13-15 min | <15 min | ✅ PASSING |

**Notes:**
- Baseline: Sequential execution time (single platform, no parallelization)
- Parallel: Execution time with current parallel configuration
- Target: Goal timing for <15 minute total CI duration
- Status: ✅ PASSING if within target, ⚠️ WARNING if 10-20% over target, ❌ FAILING if >20% over target

### Execution Time Breakdown

**Backend (8-10 min total):**
- Dependency installation: 1-2 min (with pip cache)
- Test execution: 6-7 min (pytest-xdist, -n auto)
- Coverage report: 30-60 sec

**Frontend (3-5 min total):**
- Dependency installation: 1-2 min (with npm cache)
- Test execution: 2-3 min (Jest, --maxWorkers=2)
- Coverage report: 20-40 sec

**Mobile (2-3 min total):**
- Dependency installation: 1-2 min (with npm cache)
- Test execution: 1-2 min (jest-expo, --maxWorkers=2)
- Coverage report: 10-20 sec

**Desktop (3-4 min total):**
- Dependency installation: 1-2 min (with cargo cache)
- Test execution: 2-3 min (cargo test, --test-threads=4)
- Coverage report: 20-30 sec

**Aggregation Job:**
- Artifact downloads: 30-60 sec
- CI status aggregation: 5-10 sec
- Summary generation: 5-10 sec

---

## CI Dashboard

### Reading Aggregated Status

**JSON Output Format** (`ci_status.json`):
```json

{

  "timestamp": "2026-03-07T15:30:00.000000Z",

  "aggregate": {

    "total_tests": 2181,

    "total_passed": 2150,

    "total_failed": 31,

    "pass_rate": 98.58,

    "total_duration_seconds": 845,

    "platform_count": 4

  },

  "platforms": [

    {

      "platform": "backend",

      "total": 500,

      "passed": 480,

      "failed": 20,

      "skipped": 0,

      "duration": 580,

      "pass_rate": 96.0

    },

    {

      "platform": "frontend",

      "total": 1200,

      "passed": 1180,

      "failed": 10,

      "skipped": 10,

      "duration": 240,

      "pass_rate": 98.33

    },

    {

      "platform": "mobile",

      "total": 398,

      "passed": 395,

      "failed": 3,

      "skipped": 0,

      "duration": 150,

      "pass_rate": 99.25

    },

    {

      "platform": "desktop",

      "total": 83,

      "passed": 80,

      "failed": 3,

      "skipped": 0,

      "duration": 200,

      "pass_rate": 96.39

    }

  ]

}

```

**Key Fields:**
- `aggregate.total_tests`: Sum of all tests across platforms
- `aggregate.pass_rate`: Overall pass rate percentage
- `platforms[].pass_rate`: Per-platform pass rate
- `platforms[].duration`: Execution time in seconds

### Per-Platform Breakdown

**Markdown Summary Format** (`ci_summary.md`):
```markdown

# CI Test Results Summary

Generated: 2026-03-07T15:30:00.000000Z



## Overall Results

- **Total Tests**: 2181

- **Passed**: 2150

- **Failed**: 31

- **Pass Rate**: 98.58%

- **Duration**: 845s



## Platform Breakdown

| Platform | Tests | Passed | Failed | Pass Rate | Duration |

|----------|-------|--------|--------|-----------|----------|

| BACKEND | 500 | 480 | 20 | 96.0% | 580s |

| FRONTEND | 1200 | 1180 | 10 | 98.3% | 240s |

| MOBILE | 398 | 395 | 3 | 99.2% | 150s |

| DESKTOP | 83 | 80 | 3 | 96.4% | 200s |



## Status

❌ 31 test(s) failed across platforms

```

### Pass Rate Trending

**Historical Data Location:** `backend/tests/coverage_reports/metrics/ci_status.json`

**Trend Analysis** (not yet implemented):
- Compare current run vs previous run
- Track pass rate changes (↑↓→ indicators)
- Identify platforms with declining pass rates (>5% decline)
- Alert developers to quality concerns

**Example Trend Output:**
```markdown

## Trend Analysis

- Pass Rate Change: ↓ 1.42% vs previous run

- Test Count: +12 tests added

- Platforms with Declining Pass Rates:

  - BACKEND: -2.0% decline

  - MOBILE: -0.75% decline

```

---

## Troubleshooting

### Common Issues

#### Issue 1: Resource Exhaustion (Matrix Jobs)

**Symptoms:**
- Jobs timeout waiting for runners
- Intermittent "runner not available" errors
- Slow job startup (>5 minutes)

**Causes:**
- Too many concurrent jobs exceeding GitHub Actions runner limits
- All jobs competing for same resources (API rate limits, network)

**Solutions:**
- Set `max-parallel: 4` to limit concurrent jobs (already configured)
- Use `runs-on: ubuntu-latest` for consistent runner performance
- Cache dependencies aggressively to reduce API calls
- Check GitHub Actions runner availability in repository settings

**Prevention:**
```yaml

strategy:

  fail-fast: false

  max-parallel: 4  # Limit to 4 concurrent jobs

  matrix:

    include: [...]

```

---

#### Issue 2: Cache Misses (Dependency Installation)

**Symptoms:**
- Jobs take longer than expected
- `pip install` or `npm ci` running every time
- Dependency installation time >5 minutes

**Causes:**
- Cache keys not including all dependency files
- Cache not restored properly (wrong restore-keys)
- Cache size exceeded (GitHub Actions limit: 10 GB per repository)

**Solutions:**
- Include all dependency files in cache key hash:
  ```yaml

  key: ${{ runner.os }}-pip-${{ hashFiles('backend/requirements*.txt') }}

  ```
- Use restore-keys for fallback:
  ```yaml

  restore-keys: |

    ${{ runner.os }}-pip-

  ```
- Verify cache hit rate in job logs (look for "Cache restored from key")

**Prevention:**
```yaml

# Backend: pip cache

- name: Cache pip packages

  uses: actions/cache@v4

  with:

    path: ~/.cache/pip

    key: ${{ runner.os }}-pip-${{ hashFiles('backend/requirements*.txt') }}

    restore-keys: |

      ${{ runner.os }}-pip-



# Frontend: npm cache

- name: Cache npm packages

  uses: actions/cache@v4

  with:

    path: frontend-nextjs/node_modules

    key: ${{ runner.os }}-npm-frontend-${{ hashFiles('frontend-nextjs/package-lock.json') }}

    restore-keys: |

      ${{ runner.os }}-npm-frontend-

```

---

#### Issue 3: Uneven Test Distribution (Sharding)

**Symptoms:**
- One shard takes 10 minutes, others take 2 minutes
- Large time variance between shards (>50% difference)
- Total time = slowest shard (bottleneck)

**Causes:**
- Naive file splitting (alphabetical) doesn't account for execution time
- Some tests significantly slower than others
- No historical timing data for balanced distribution

**Solutions:**
- Use historical timing data from `pytest --durations` or Jest `--verbose`
- Implement greedy algorithm: assign slowest test to least-loaded shard
- Rebalance shards weekly based on new timing data
- Use pytest-xdist load balancing (`-n auto`) instead of manual sharding

**Prevention:**
```bash

# Backend: Use pytest-xdist auto balancing (no manual sharding needed)

pytest tests/ -n auto



# Frontend: Use Jest shard with historical timing data

jest --shard=1/4  # Jest automatically balances by test count

# For time-based balancing, use test-splitter.py script (see Advanced section)

```

---

#### Issue 4: Flaky Tests (Unnecessary Re-runs)

**Symptoms:**
- Same test fails intermittently
- Re-runs succeed without code changes
- Full platform re-runs triggered by single flaky test

**Causes:**
- Retry logic at job level instead of test level
- No flaky test detection/tracking
- Race conditions or timing dependencies

**Solutions:**
- Use pytest-rerunfailures for automatic test-level retries:
  ```bash

  pytest tests/ -n auto --reruns 2 --reruns-delay 1

  ```
- Track flaky tests with `detect_flaky_tests.py` (already exists)
- Only re-run failed tests, not entire suite (platform-retry.yml)
- Quarantine known flaky tests with `--xfail` or `test.skip`

**Prevention:**
```bash

# Backend: Add automatic retries for flaky tests

pytest tests/ -n auto --reruns 2 --reruns-delay 1



# Frontend: Use jest-stare for flaky test detection

npm install --save-dev jest-stare

jest --json --outputFile=test-results.json --maxWorkers=2

```

---

### Debugging Commands

**Check workflow run status:**
```bash

gh run list --workflow=unified-tests-parallel.yml --limit 5

gh run view <run-id>

```

**Download workflow artifacts:**
```bash

gh run download <run-id> -n ci-status-unified

```

**View platform-specific logs:**
```bash

gh run view <run-id> --log | grep -A 50 "Test backend"

gh run view <run-id> --log | grep -A 50 "Test frontend"

```

**Run tests locally (simulate CI):**
```bash

# Backend

cd backend

pytest tests/ -v -n auto --json-report --json-report-file=pytest_report.json



# Frontend

cd frontend-nextjs

npm run test:ci -- --json --outputFile=test-results.json --maxWorkers=2



# Mobile

cd mobile

npm run test:ci -- --json --outputFile=test-results.json --maxWorkers=2



# Desktop

cd frontend-nextjs/src-tauri

cargo test --test-threads=4

```

---

## Reference

### Workflow File Locations

**Main Workflows:**
- `.github/workflows/unified-tests-parallel.yml` - Matrix strategy for 4 platforms
- `.github/workflows/platform-retry.yml` - Platform-specific retry jobs

**Scripts:**
- `backend/tests/scripts/ci_status_aggregator.py` - Combine platform statuses
- `backend/tests/scripts/platform_retry_router.py` - Extract failed tests and generate retry commands
- `backend/tests/scripts/e2e_aggregator.py` - E2E test aggregation (Phase 148)

**Documentation:**
- `backend/tests/docs/PARALLEL_EXECUTION_GUIDE.md` - This guide
- `backend/tests/docs/E2E_TESTING_GUIDE.md` - E2E testing patterns (if exists)
- `.planning/phases/149-quality-infrastructure-parallel/149-RESEARCH.md` - Research document

### Environment Variables

**Backend (pytest):**
```bash

DATABASE_URL=sqlite:///:memory:

BYOK_ENCRYPTION_KEY=test_key_for_ci_only

ENVIRONMENT=test

ATOM_DISABLE_LANCEDB=true

ATOM_MOCK_DATABASE=true

CI=true

```

**Frontend/Mobile (Jest):**
```bash

CI=true

```

**Desktop (cargo test):**
```bash

# No special environment variables required

```

### Script Locations

**CI Status Aggregator:**
```bash

# Location

backend/tests/scripts/ci_status_aggregator.py



# Usage

python backend/tests/scripts/ci_status_aggregator.py \

  --backend results/backend/pytest_report.json \

  --frontend results/frontend/test-results.json \

  --mobile results/mobile/test-results.json \

  --desktop results/desktop/cargo_test_results.json \

  --output results/ci_status.json \

  --summary results/ci_summary.md

```

**Platform Retry Router:**
```bash

# Location

backend/tests/scripts/platform_retry_router.py



# Usage (automated by platform-retry.yml)

python backend/tests/scripts/platform_retry_router.py \

  --platform backend \

  --results-file results/backend/pytest_report.json \

  --output-file retry_commands/backend_retry.sh

```

**E2E Aggregator:**
```bash

# Location

backend/tests/scripts/e2e_aggregator.py



# Usage

python backend/tests/scripts/e2e_aggregator.py \

  --web results/web/pytest_report.json \

  --mobile results/mobile/mobile-results.json \

  --desktop results/desktop/desktop-results.json \

  --output results/e2e_unified.json \

  --summary results/e2e_summary.md

```

### Artifact Paths

**Test Results:**
- `backend/pytest_report.json` - Backend pytest JSON report
- `frontend-nextjs/test-results.json` - Frontend Jest JSON report
- `mobile/test-results.json` - Mobile jest-expo JSON report
- `frontend-nextjs/src-tauri/cargo_test_results.json` - Desktop cargo test JSON

**Coverage Reports:**
- `backend/tests/coverage_reports/metrics/coverage.json` - Backend coverage
- `frontend-nextjs/coverage/coverage-final.json` - Frontend coverage
- `mobile/coverage/coverage-final.json` - Mobile coverage
- `frontend-nextjs/src-tauri/coverage.json` - Desktop coverage

**Aggregated Output:**
- `results/ci_status.json` - Unified CI status (machine-readable)
- `results/ci_summary.md` - Unified CI summary (human-readable)

### Advanced Configuration

**Test Splitting Script** (for time-based sharding):
```bash

# Generate test shards based on historical timing data

python backend/tests/scripts/test_splitter.py \

  --shards 4 \

  --timings test_timings.json \

  --output test_shards.json



# Run specific shard

python backend/tests/scripts/test_splitter.py \

  --shard-index 0 \

  --shards 4 \

  --run-tests

```

**Custom Timeout Values:**
```yaml

# unified-tests-parallel.yml matrix configuration

- platform: backend

  timeout: 30  # minutes

- platform: frontend

  timeout: 20

- platform: mobile

  timeout: 20

- platform: desktop

  timeout: 15

```

**Parallel Worker Configuration:**
```bash

# Backend: pytest-xdist (auto detects CPU cores)

pytest tests/ -n auto  # or -n 4 for explicit worker count



# Frontend/Mobile: Jest (limit to 2 workers for CI)

jest --maxWorkers=2  # or --maxWorkers=4 for faster local testing



# Desktop: cargo test (use 4 threads)

cargo test --test-threads=4  # or --test-threads=8 for faster local testing

```

---

## PR Comment Template

### Standard PR Comment Format

```markdown

## CI Test Results Summary



### Overall Results

- **Total Tests**: 2181

- **Passed**: 2150

- **Failed**: 31

- **Pass Rate**: 98.58%

- **Duration**: 845s (14m 5s)



### Platform Breakdown

| Platform | Tests | Passed | Failed | Pass Rate | Duration |

|----------|-------|--------|--------|-----------|----------|

| ✅ BACKEND | 500 | 480 | 20 | 96.0% | 580s |

| ✅ FRONTEND | 1200 | 1180 | 10 | 98.3% | 240s |

| ✅ MOBILE | 398 | 395 | 3 | 99.2% | 150s |

| ❌ DESKTOP | 83 | 80 | 3 | 96.4% | 200s |



### Status

❌ 31 test(s) failed across platforms



### Retry Actions

[► Re-run Desktop Tests](https://github.com/owner/repo/actions/runs/123456)



---



<details>

<summary>Failed Test Details</summary>



#### Desktop (3 failed)

- `test_windows_file_operations_roundtrip` - Assertion error

- `test_desktop_cfg_detection` - Timeout

- `test_cargo_json_parsing` - Parse error



</details>

```

### JSON Examples for Custom Dashboards

**ci_status_aggregator.py Output Format:**
```json

{

  "timestamp": "2026-03-07T15:30:00.000000Z",

  "aggregate": {

    "total_tests": 2181,

    "total_passed": 2150,

    "total_failed": 31,

    "pass_rate": 98.58,

    "total_duration_seconds": 845,

    "platform_count": 4

  },

  "platforms": [

    {

      "platform": "backend",

      "total": 500,

      "passed": 480,

      "failed": 20,

      "skipped": 0,

      "duration": 580,

      "pass_rate": 96.0

    },

    {

      "platform": "frontend",

      "total": 1200,

      "passed": 1180,

      "failed": 10,

      "skipped": 10,

      "duration": 240,

      "pass_rate": 98.33

    },

    {

      "platform": "mobile",

      "total": 398,

      "passed": 395,

      "failed": 3,

      "skipped": 0,

      "duration": 150,

      "pass_rate": 99.25

    },

    {

      "platform": "desktop",

      "total": 83,

      "passed": 80,

      "failed": 3,

      "skipped": 0,

      "duration": 200,

      "pass_rate": 96.39

    }

  ]

}

```

**platform_retry_router.py Retry Command Format:**
```json

{

  "platform": "backend",

  "failed_tests": [

    "tests/test_agent_governance.py::test_agent_permission_check",

    "tests/test_llm_byok.py::test_streaming_response",

    "tests/test_episode_segmentation.py::test_segment_by_time_gap"

  ],

  "retry_command": "pytest tests/ -v -n auto tests/test_agent_governance.py::test_agent_permission_check tests/test_llm_byok.py::test_streaming_response tests/test_episode_segmentation.py::test_segment_by_time_gap"

}

```

**Actual ci_status_aggregator.py JSON Structure:**
```json

{

  "timestamp": "2026-03-07T15:30:00.000000Z",

  "aggregate": {

    "total_tests": 2181,

    "total_passed": 2150,

    "total_failed": 31,

    "pass_rate": 98.58,

    "total_duration_seconds": 845,

    "platform_count": 4

  },

  "platforms": [

    {

      "platform": "backend",

      "total": 500,

      "passed": 480,

      "failed": 20,

      "skipped": 0,

      "duration": 580,

      "pass_rate": 96.0,

      "error": null

    },

    {

      "platform": "frontend",

      "total": 1200,

      "passed": 1180,

      "failed": 10,

      "skipped": 10,

      "duration": 240,

      "pass_rate": 98.33,

      "error": null

    },

    {

      "platform": "mobile",

      "total": 398,

      "passed": 395,

      "failed": 3,

      "skipped": 0,

      "duration": 150,

      "pass_rate": 99.25,

      "error": null

    },

    {

      "platform": "desktop",

      "total": 83,

      "passed": 80,

      "failed": 3,

      "skipped": 0,

      "duration": 200,

      "pass_rate": 96.39,

      "error": null

    }

  ],

  "trend": {

    "pass_rate_delta": 0.0,

    "test_count_delta": 0,

    "declining_platforms": []

  }

}

```

---

## Extending CI Dashboard

### Adding Custom Metrics to ci_status_aggregator.py

**Step 1: Add new metric field to platform output:**
```python

def parse_pytest_results(results: Dict) -> Dict[str, Any]:

    """Parse pytest JSON report format with custom metrics."""

    summary = results.get("summary", {})



    # Add custom metric: coverage percentage

    coverage = results.get("coverage", {}).get("percent_covered", 0)



    return {

        "platform": "backend",

        "total": summary.get("total", 0),

        "passed": summary.get("passed", 0),

        "failed": summary.get("failed", 0),

        "skipped": summary.get("skipped", 0),

        "duration": summary.get("duration", 0),

        "pass_rate": round(pass_rate, 2),

        "coverage": coverage,  # Custom metric

    }

```

**Step 2: Update aggregation function:**
```python

def aggregate_platform_status(platforms: List[Dict[str, Any]]) -> Dict[str, Any]:

    """Calculate aggregate metrics with custom fields."""

    total_tests = sum(p.get("total", 0) for p in platforms)

    total_passed = sum(p.get("passed", 0) for p in platforms)



    # Add custom metric: average coverage

    coverages = [p.get("coverage", 0) for p in platforms if "coverage" in p]

    avg_coverage = sum(coverages) / len(coverages) if coverages else 0



    return {

        "total_tests": total_tests,

        "total_passed": total_passed,

        "total_failed": total_failed,

        "pass_rate": round(pass_rate, 2),

        "total_duration_seconds": total_duration,

        "platform_count": len(platforms),

        "avg_coverage": round(avg_coverage, 2),  # Custom metric

    }

```

**Step 3: Update markdown summary generator:**
```python

def generate_markdown_summary(

    aggregate: Dict[str, Any],

    platforms: List[Dict[str, Any]],

) -> str:

    """Generate markdown summary with custom metrics."""

    lines = [

        "# CI Test Results Summary",

        f"Generated: {datetime.now().isoformat()}",

        "",

        "## Overall Results",

        f"- **Total Tests**: {aggregate['total_tests']}",

        f"- **Passed**: {aggregate['total_passed']}",

        f"- **Failed**: {aggregate['total_failed']}",

        f"- **Pass Rate**: {aggregate['pass_rate']}%",

        f"- **Avg Coverage**: {aggregate['avg_coverage']}%",  # Custom metric

        f"- **Duration**: {aggregate['total_duration_seconds']}s",

        "",

        "## Platform Breakdown",

        "| Platform | Tests | Passed | Failed | Pass Rate | Coverage | Duration |",  # Custom column

        "|----------|-------|--------|--------|-----------|----------|----------|",

    ]



    for p in platforms:

        platform = p["platform"].upper()

        lines.append(

            f"| {platform} | {p['total']} | {p['passed']} | {p['failed']} | "

            f"{p['pass_rate']:.1f}% | {p.get('coverage', 0):.1f}% | {p['duration']}s |"  # Custom column

        )



    return "\n".join(lines)

```

### Creating Custom Status Checks

**GitHub Actions Status Check API:**
```yaml

# In unified-tests-parallel.yml aggregation job

- name: Create per-platform status checks

  uses: actions/github-script@v7

  with:

    script: |

      const fs = require('fs');

      const unified = JSON.parse(fs.readFileSync('results/ci_status.json', 'utf8'));



      // Create status check for each platform

      for (const platform of unified.platforms) {

        const context = `ci/${platform.platform}-tests`;

        const state = platform.failed === 0 ? 'success' : 'failure';

        const description = `${platform.passed}/${platform.total} passed (${platform.pass_rate}%)`;



        await github.rest.repos.createCommitStatus({

          owner: context.repo.owner,

          repo: context.repo.repo,

          sha: context.sha,

          context: context,

          state: state,

          description: description,

          target_url: `https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`,

        });

      }

```

### Integrating with External Dashboards

**Grafana Integration:**
```python

# Add to ci_status_aggregator.py

import requests



def send_to_grafana(aggregate: Dict[str, Any]):

    """Send metrics to Grafana via Loki or Prometheus."""

    grafana_url = os.getenv("GRAFANA_URL")

    if not grafana_url:

        return



    metrics = {

        "pass_rate": aggregate["pass_rate"],

        "total_tests": aggregate["total_tests"],

        "total_failed": aggregate["total_failed"],

        "duration": aggregate["total_duration_seconds"],

    }



    requests.post(grafana_url, json=metrics)

```

**Datadog Integration:**
```python

# Add to ci_status_aggregator.py

from datadog import DogStatsd



def send_to_datadog(aggregate: Dict[str, Any]):

    """Send metrics to Datadog."""

    statsd = DogStatsd()

    statsd.gauge('ci.pass_rate', aggregate["pass_rate"])

    statsd.gauge('ci.total_tests', aggregate["total_tests"])

    statsd.gauge('ci.total_failed', aggregate["total_failed"])

    statsd.gauge('ci.duration', aggregate["total_duration_seconds"])

```

### Trending Data Storage

**Historical Data Location:** `backend/tests/coverage_reports/metrics/ci_status.json`

**Trend File Structure:**
```json

[

  {

    "timestamp": "2026-03-07T15:30:00.000000Z",

    "aggregate": {

      "total_tests": 2181,

      "total_passed": 2150,

      "total_failed": 31,

      "pass_rate": 98.58

    },

    "platforms": [

      {

        "platform": "backend",

        "total": 500,

        "passed": 480,

        "failed": 20,

        "duration": 580

      }

    ]

  },

  {

    "timestamp": "2026-03-07T16:00:00.000000Z",

    "aggregate": {

      "total_tests": 2185,

      "total_passed": 2160,

      "total_failed": 25,

      "pass_rate": 98.86

    },

    "platforms": [...]

  }

]

```

**Loading Historical Data:**
```python

import json

from pathlib import Path



def load_trend_history(trend_file: str) -> List[Dict[str, Any]]:

    """Load historical trend data from JSON."""

    path = Path(trend_file)

    if not path.exists():

        return []



    try:

        history = json.loads(path.read_text())

        history.sort(key=lambda x: x.get("timestamp", ""), reverse=True)

        return history

    except (json.JSONDecodeError, KeyError):

        return []

```

---

## Platform Retry Flow

### When Retries Trigger

**Trigger Condition:**
- `unified-tests-parallel` workflow completes with `conclusion: 'failure'`
- `platform-retry.yml` workflow_run trigger activates automatically

- Only runs on `main` and `develop` branches



**Retry Workflow:**

1. **detect-failures job** downloads artifacts from failed workflow run

2. **platform_retry_router.py** extracts failed tests for each platform

3. **Conditional retry jobs** run only for platforms with failures

4. **Retry results** uploaded as artifacts (30-day retention)



### How platform_retry_router.py Extracts Failed Tests



**Backend (pytest format):**

```python

def extract_failed_tests(results: Dict, platform: str) -> List[str]:

    """Extract failed test names from pytest JSON report."""

    if platform == "backend":

        # pytest format: summary.failed + test results

        failed_tests = []
        for test in results.get("tests", []):

            if test.get("outcome") == "failed":

                failed_tests.append(test["name"])

        return failed_tests

```


**Frontend/Mobile (Jest format):**
```python

def extract_failed_tests(results: Dict, platform: str) -> List[str]:

    """Extract failed test names from Jest JSON results."""

    if platform in ["frontend", "mobile"]:

        # Jest format: testResults with status

        failed_tests = []

        for suite in results.get("testResults", []):

            for test in suite.get("assertionResults", []):

                if test.get("status") == "failed":

                    failed_tests.append(test["fullName"])

        return failed_tests

```

**Desktop (cargo test format):**
```python

def extract_failed_tests(results: Dict, platform: str) -> List[str]:

    """Extract failed test names from cargo test JSON."""

    if platform == "desktop":

        # Cargo format: testResults with passed field

        failed_tests = []

        for test in results.get("testResults", []):

            if not test.get("passed", False):

                failed_tests.append(test["name"])

        return failed_tests

```

### Retry Job Execution Flow

**1. detect-failures job:**
```yaml

- name: Check backend failures

  id: check-backend

  run: |

    python backend/tests/scripts/platform_retry_router.py \

      --platform backend \

      --results-file results/backend/pytest_report.json \

      --output-file retry_commands/backend_retry.sh



    if [ $? -eq 0 ]; then

      echo "failed=true" >> $GITHUB_OUTPUT

    else

      echo "failed=false" >> $GITHUB_OUTPUT

    fi

```

**2. Conditional retry job:**
```yaml

retry-backend:

  name: Retry Backend Tests

  needs: [detect-failures]

  if: ${{ needs.detect-failures.outputs.backend-failed == 'true' }}

  steps:

    - name: Run backend retry tests

      run: |

        bash ../retry_commands/backend_retry.sh

```

**3. Retry result aggregation:**
```yaml

- name: Upload retry results

  uses: actions/upload-artifact@v4

  with:

    name: backend-retry-results

    path: backend/pytest_report.json

    retention-days: 7

```

### Retry Result Aggregation

**Actual platform_retry_router.py Output:**
```json

{

  "platform": "backend",

  "failed_tests": [

    "tests/test_agent_governance.py::test_agent_permission_check",

    "tests/test_llm_byok.py::test_streaming_response"

  ],

  "retry_command": "pytest tests/ -v -n auto tests/test_agent_governance.py::test_agent_permission_check tests/test_llm_byok.py::test_streaming_response --json-report --json-report-file=pytest_report.json",

  "exit_code": 0

}

```

**platform-retry.yml detect-failures Job Output:**
```yaml

outputs:

  backend-failed: 'true'  # or 'false'

  frontend-failed: 'false'

  mobile-failed: 'false'

  desktop-failed: 'true'

```

**Conditional Retry Job Execution:**
```yaml

retry-backend:

  name: Retry Backend Tests

  needs: [detect-failures]

  if: ${{ needs.detect-failures.outputs.backend-failed == 'true' }}

  # Only runs if backend-failed == 'true'

```

**Retry Results Upload:**
```yaml

- name: Upload retry results

  uses: actions/upload-artifact@v4

  if: always()

  with:

    name: backend-retry-results

    path: backend/pytest_report.json

    retention-days: 7

    if-no-files-found: warn

```

**Current State:** Retry results uploaded as artifacts, but not aggregated into unified status

**Future Enhancement:** Add retry aggregation to ci_status_aggregator.py
```python

def aggregate_retry_results(

    original_results: Dict[str, Any],

    retry_results: Dict[str, Any],

) -> Dict[str, Any]:

    """Aggregate original and retry results."""

    # Combine original + retry metrics

    # Update platform status with retry pass rate

    # Generate summary with retry information

    pass

```

---

## Best Practices

### Development Workflow

**1. Run tests locally before pushing:**
```bash

# Backend

cd backend && pytest tests/ -v -n auto



# Frontend

cd frontend-nextjs && npm run test:ci -- --maxWorkers=2



# Mobile

cd mobile && npm run test:ci -- --maxWorkers=2



# Desktop

cd frontend-nextjs/src-tauri && cargo test --test-threads=4

```

**2. Use PR drafts for experimental changes:**
- Create draft PR to trigger CI without blocking others
- Review CI results before marking PR as ready for review
- Fix failing tests before requesting review

**3. Monitor CI duration trends:**
- Check ci_status.json for pass rate trends

- Identify platforms with declining pass rates

- Optimize slow tests (target: <15 minutes total)



### CI/CD Maintenance



**1. Update baseline timings quarterly:**

- Measure actual execution time for each platform

- Update PARALLEL_EXECUTION_GUIDE.md timing benchmarks table

- Adjust timeout values if needed



**2. Review cache hit rates monthly:**

- Check job logs for cache hit/miss messages

- Optimize cache keys if hit rate <80%

- Clean up old cache entries if approaching 10 GB limit



**3. Audit test suite growth:**

- Track new test additions in ci_status.json
- Remove obsolete or duplicate tests
- Ensure test count growth aligns with feature development

### Performance Optimization

**1. Optimize slowest platform first:**
- Identify slowest platform in CI dashboard
- Focus optimization efforts on bottleneck (max(platform_durations))

- Target: All platforms <15 minutes



**2. Use test splitting strategically:**

- Only split if platform duration >15 minutes

- Start with pytest-xdist auto balancing (simpler)

- Move to manual sharding only if auto balancing insufficient



**3. Leverage caching aggressively:**

- Cache pip packages, npm modules, cargo registry

- Use restore-keys for fallback cache keys

- Verify cache hit rate in job logs



---



## Glossary



- **Matrix Strategy:** GitHub Actions feature for running multiple jobs in parallel with different configurations

- **Fail-Fast:** Workflow setting that cancels all jobs if one job fails (disabled in our workflow)

- **Max-Parallel:** Limit on concurrent matrix jobs (set to 4 for our 4 platforms)

- **Aggregation Job:** Job that combines results from multiple platform jobs into unified report

- **Retry Workflow:** Separate workflow that re-runs only failed platform tests (not full suite)

- **pytest-xdist:** pytest plugin for parallel test execution with automatic load balancing

- **Jest Sharding:** Built-in Jest feature for splitting test suite across multiple jobs

- **cargo test --test-threads:** Rust cargo test option for parallel thread execution

- **Cache Hit Rate:** Percentage of time CI cache is successfully restored (target: >80%)

- **Flaky Test:** Test that passes/fails intermittently without code changes

- **Platform-Specific Retry:** Re-run only failed tests for a specific platform (80% time savings vs full suite)



---



## Additional Resources



**Internal Documentation:**

- `.planning/phases/149-quality-infrastructure-parallel/149-RESEARCH.md` - Research document

- `backend/tests/docs/E2E_TESTING_GUIDE.md` - E2E testing patterns

- `backend/tests/docs/COVERAGE_GUIDE.md` - Coverage reporting guide
- `backend/tests/docs/FLAKY_TEST_GUIDE.md` - Flaky test handling guide

**External References:**
- [GitHub Actions Matrix Strategy](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs)
- [pytest-xdist Documentation](https://pytest-xdist.readthedocs.io/)
- [Jest CLI Options](https://jestjs.io/docs/cli)
- [cargo test Documentation](https://doc.rust-lang.org/cargo/commands/cargo-test.html)

**Related Workflows:**
- `.github/workflows/unified-tests.yml` - Sequential test execution (backup)
- `.github/workflows/e2e-unified.yml` - E2E test orchestration (Phase 148)
- `.github/workflows/platform-retry.yml` - Platform-specific retry jobs

---

**Document Version:** 1.0
**Last Updated:** March 7, 2026
**Maintainer:** Atom CI/CD Team
**Feedback:** Open issue or PR for improvements to this guide

## See Also

- [Testing Documentation Index](../../docs/TESTING_INDEX.md) - Central hub for all testing documentation
- [Testing Onboarding Guide](../../docs/TESTING_ONBOARDING.md) - 15-minute quick start for all platforms
- [Flaky Test Quarantine](FLAKY_TEST_QUARANTINE.md) - Multi-run detection in CI/CD
- [Coverage Trending Guide](COVERAGE_TRENDING_GUIDE.md) - Track coverage over time
- [Test Isolation Patterns](TEST_ISOLATION_PATTERNS.md) - Writing tests for parallel execution