docs/TESTING.md

# Testing Guide — Color-UX-Access

> HF Space containers cannot run a browser. Playwright is used locally for
> screenshots but tests are source-inspection only — no live browser in CI.

For test setup (venv, uv install), see `docs/DEVELOPMENT.md`.

---

## Testing Philosophy

Tests use a layered approach, from fast to slow:

| Category | Marker | What it tests | Duration |
|----------|--------|---------------|----------|
| Smoke | `smoke` | Import, module load, CVD config, Gradio apps loadable | <10s |
| Pipeline | `pipeline` | Full image→CVD→WCAG report with mocked VLM | <30s |
| Slow | `slow` | Real VLM calls, needs HF_TOKEN | 60-90s |

**Source inspection tests** verify structural properties that mocks miss — e.g., `isinstance(file_obj, bytes)` check for GradioFile binary type, Modal app GPU config, Gradio 6 theme placement in `launch()` not `Blocks()`.

---

## Running Tests

```bash
uv run pytest -v                     # all tests (smoke + pipeline) — no network, no GPU
uv run pytest -m smoke               # fast: imports + smoke only (<10s)
uv run pytest -m pipeline            # full pipeline with mocked VLM (<30s)
uv run pytest tests/test_app_space.py -v  # single file
uv run pytest tests/test_app_space.py::test_cvd_gallery_generates_10_types -v  # single test
```

---

## Test Structure

```
tests/
├── conftest.py              Shared fixtures (img_factory, cvd_img_factory, mock_vlm_factory)
├── fixtures/                Synthetic UI screenshots — generated by generate_fixtures.py
│   ├── button_panel_accessible.png
│   ├── button_panel_color_only.png
│   ├── status_badges_accessible.png
│   ├── status_badges_color_only.png
│   ├── form_validation_accessible.png
│   ├── form_validation_color_only.png
│   └── *.png               Plus deuteranopia/protanopia/tritanopia variants per original
├── test_smoke.py            All core imports work, CVD variants correct, Gradio apps loadable
├── test_capture.py          capture module contract: take_screenshot exists + correct signature
├── test_app_space.py        Integration: CVD gallery, WCAG report, Gradio 6 theme launch
├── test_gradio_binary.py    Source inspection: Gradio File binary type handling
└── test_modal_app.py        Source inspection: Modal app structure (GPU, secrets, base URL)
```

---

## Shared Fixtures (from `conftest.py`)

All fixtures are auto-discovered by pytest — import them directly in tests.

### `img_factory`

```python
def test_something(img_factory):
    img = img_factory(640, 480, seed=42)  # deterministic random PIL Image
```

### `cvd_img_factory`

```python
def test_cvd_simulation(cvd_img_factory):
    img, simulate = cvd_img_factory(200, 100, seed=42)
    deut = simulate(img, 'deuteranopia')
    assert deut.size == img.size
```

### `mock_vlm_factory`

```python
def test_report_passes(mock_vlm_factory):
    result = mock_vlm_factory(passes=True)
    assert result['passes'] is True
    assert result['findings'] == []
```

### `sample_button_panel`

```python
def test_cvd_confuses_buttons(sample_button_panel, cvd_img_factory):
    img = sample_button_panel
    _, simulate = cvd_img_factory()
    deut = simulate(img, 'deuteranopia')
    # Green (Confirm) and red (Cancel) buttons — visually similar for CVD
```

### `ten_type_gallery`

```python
def test_gallery_types(ten_type_gallery):
    assert len(ten_type_gallery) == 7
```

---

## TDD Pattern: RED → GREEN → REFACTOR

```bash
# 1. Write a failing test
uv run pytest tests/test_capture.py::test_take_screenshot_signature -v
# → RED if not yet implemented

# 2. Implement the feature
# create color_ux_access/capture.py with take_screenshot(page, url, timeout=60000)

# 3. Run again — verify GREEN
uv run pytest tests/test_capture.py -v

# 4. Refactor if needed
```

For new features:
```python
# tests/test_new_feature.py
def test_feature_does_x():
    from color_ux_access.module import function
    result = function(input)
    assert expected_output in result
```

---

## Source Inspection Tests

Tests that verify source code structure rather than runtime behavior:

```python
# tests/test_gradio_binary.py — verify GradioFile bytes handling
def test_analyze_screenshot_handles_bytes():
    source = pathlib.Path("color_ux_access/modal_app.py").read_text()
    assert "isinstance(file_obj, bytes)" in source

# tests/test_modal_app.py — verify Modal app structure
def test_base_url_correct():
    source = pathlib.Path("color_ux_access/modal_app.py").read_text()
    assert "router.huggingface.co/v1" in source
```

---

## CI Integration

No CI configured yet. For now, run `uv run pytest` before every commit locally.

```yaml
# .github/workflows/test.yml (future)
name: Test
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v4
        with:
          python-version: "3.12"
      - run: uv sync --extra dev
      - run: uv run pytest -m smoke
```

---

## Fixture Generation

Generate synthetic UI screenshots for local testing:

```bash
PYTHONPATH=. python tests/generate_fixtures.py
```

Each original image gets CVD-simulated variants (deuteranopia, protanopia, tritanopia). Fixture pairs:

- `button_panel_color_only.png` vs `button_panel_accessible.png` — CVD-confusable buttons
- `status_badges_color_only.png` vs `status_badges_accessible.png` — CVD-invisible distinctions
- `form_validation_color_only.png` vs `form_validation_accessible.png` — CVD-invisible error states