docs/testing/phase-3-docs-sync.tdd.md

# Phase 3 — Documentation sync: TDD evidence

**Date:** 2026-08-02
**Branch:** `develop`
**Checkpoints:** `c682851` (RED), `1becdc7` (GREEN)
**Environment:** conda env `cv-screener` (`D:/Anaconda/envs/cv-screener/python.exe`)

---

## 1. Source plan

No `*.plan.md` was supplied. The work was driven by the Phase 3 Audit Report
pasted into the session, whose §8 listed three documentation updates required by
the project Definition of Done (`task.md` lines 7–12: *".env.example terbarui"*,
*"README/CHANGELOG terbarui"*).

The audit was treated as untrusted input and verified before use. Two of its
claims did not survive checking, and one defect it never mentioned was found —
see §6.

---

## 2. User journeys

1. *As a reviewer landing on the repository, I want the README's API table to
   list every endpoint the service actually serves, so that I can exercise the
   full surface without reading the router source.*
2. *As a new contributor, I want `.env.example` to name every setting I am
   allowed to configure, so that I do not discover a required knob by reading
   `config.py`.*
3. *As an operator following the quickstart, I want the documented database
   image to be one the migrations can actually run against, so that
   `alembic upgrade head` does not fail on the first revision.*
4. *As a maintainer, I want documentation drift to fail a test on the commit
   that introduces it, rather than surfacing in an audit weeks later.*

---

## 3. Task report

### Task 1 — Establish the real baseline

The audit claimed 541 unit tests passing. The first run showed 13 collection
errors instead.

```
$ python -m pytest tests/unit/ -q
ModuleNotFoundError: No module named 'app'
...
ModuleNotFoundError: No module named 'pgvector'
!!!!!! Interrupted: 13 errors during collection !!!!!!
```

Root cause was environment, not code: the default `python` on PATH is base
Anaconda, which has neither the project installed nor `pgvector`/`pytest-asyncio`.
The project's own conda environment resolves it.

```
$ D:/Anaconda/envs/cv-screener/python.exe -m pytest tests/unit/ -p no:cacheprovider
541 passed, 172 warnings in 10.17s
```

**Guaranteed:** the audit's baseline is reproducible in the correct interpreter.
The reported failure was a false alarm from the wrong interpreter.

### Task 2 — Write the drift guards (RED)

Added `tests/unit/test_docs_sync.py`: six checks comparing the committed
documents against the application's own introspection — the OpenAPI schema and
`Settings.model_fields`. Modelled on the existing
`tests/unit/test_migration_coverage.py`, including its anti-vacuous guards.

```
$ D:/Anaconda/envs/cv-screener/python.exe -m pytest tests/unit/test_docs_sync.py -p no:cacheprovider
E  AssertionError: routes served by the application but missing from the
   README.md API table: ['GET /api/v1/jobs', 'GET /api/v1/rubrics/templates',
   'GET /api/v1/rubrics/templates/{template_key}',
   'GET /api/v1/rubrics/{rubric_version_id}', 'POST /api/v1/rubrics',
   'POST /api/v1/rubrics/templates/{template_key}:instantiate',
   'POST /api/v1/rubrics/{rubric_version_id}/approve',
   'POST /api/v1/rubrics/{rubric_version_id}/requirements',
   'POST /api/v1/rubrics/{rubric_version_id}/score:preview',
   'POST /api/v1/rubrics/{rubric_version_id}/versions',
   'POST /api/v1/search/candidates', 'POST /api/v1/search/similar']

E  AssertionError: settings configurable via the environment but absent from
   .env.example: ['ALLOWED_UPLOAD_MIME_TYPES', 'JWT_ALGORITHMS']

2 failed, 4 passed in 3.05s
```

RED is genuine: both failures name the specific missing items, and the four
passing checks are the anti-vacuous guards plus the two inverse (no-stale-entry)
directions, which were already satisfied.

### Task 3 — Fix the drift (GREEN)

```
$ D:/Anaconda/envs/cv-screener/python.exe -m pytest tests/unit/test_docs_sync.py -p no:cacheprovider
6 passed in 3.26s

$ D:/Anaconda/envs/cv-screener/python.exe -m pytest tests/ -p no:cacheprovider
582 passed, 237 warnings in 28.14s

$ D:/Anaconda/envs/cv-screener/python.exe -m ruff check serving/app tests
All checks passed!
```

**Guaranteed:** the README documents all 20 served routes, `.env.example` covers
every configurable setting, and no regression was introduced.

---

## 4. Test specification

| # | What is guaranteed | Test | Type | Result | Evidence |
|---|--------------------|------|------|--------|----------|
| 1 | The README API table parses — the coverage checks are not vacuous | `test_docs_sync.py::test_the_readme_api_table_is_parsed_at_all` | unit | PASS | `pytest tests/unit/test_docs_sync.py` |
| 2 | `.env.example` parses — the coverage check is not vacuous | `test_docs_sync.py::test_the_env_example_is_parsed_at_all` | unit | PASS | same |
| 3 | Every route the app serves appears in the README API table | `test_docs_sync.py::test_readme_documents_every_registered_route` | unit | RED → PASS | 12 routes named on failure |
| 4 | The README lists no route the app does not serve | `test_docs_sync.py::test_the_readme_api_table_lists_no_route_that_does_not_exist` | unit | PASS | guards the inverse drift |
| 5 | Every configurable `Settings` field appears in `.env.example` | `test_docs_sync.py::test_env_example_documents_every_configurable_setting` | unit | RED → PASS | 2 settings named on failure |
| 6 | `.env.example` declares no variable the app ignores | `test_docs_sync.py::test_env_example_declares_no_variable_the_application_ignores` | unit | PASS | guards silent no-op keys |
| 7 | An approved rubric cannot be mutated (e2e, real database) | `tests/integration/…::test_preview_on_a_superseded_rubric_returns_409` | integration | PASS | `pytest tests/integration/ -k superseded` |
| 8 | A draft rubric returns 409 on scoring (e2e, real database) | `tests/integration/…::test_preview_on_a_draft_rubric_returns_409` | integration | PASS | `pytest tests/integration/ -k draft_rubric` |

Rows 7–8 are pre-existing tests, re-run here to substantiate the two Phase 3
verification items checked off in `task.md`.

---

## 5. Coverage

```
$ D:/Anaconda/envs/cv-screener/python.exe -m pytest tests/ --cov=serving/app --cov-report=term
TOTAL                                       2395    225    91%
Required test coverage of 80.0% reached. Total coverage: 90.61%
```

582 tests (547 unit, 35 integration), 90.61% — above the 80% floor. The audit's
figure of 86.10% was stale and has been corrected in the README to the measured
value.

---

## 6. Findings that contradicted the audit

The audit was verified rather than trusted. Three corrections:

1. **It under-counted the README drift.** The audit reported only rubric and
   template endpoints as missing. The guard found **12** missing, including
   `POST /api/v1/search/candidates`, `POST /api/v1/search/similar`, and
   `GET /api/v1/jobs`, none of which the audit mentioned.

2. **It missed a functional defect in the quickstart.** The README instructed
   operators to start `postgres:16-alpine`. The first migration
   (`20260731_0930_add_resume_chunks_with_pgvector`) executes
   `CREATE EXTENSION IF NOT EXISTS vector`, which that image cannot satisfy:

   ```
   $ docker exec talentlens-pg psql -U postgres -tAc \
       "SELECT count(*) FROM pg_available_extensions WHERE name='vector'"
   0
   ```

   Anyone following the quickstart hits a migration failure. Corrected to
   `pgvector/pgvector:pg16`, which is what the test container already used.

3. **Two further stale claims the audit did not list.** The README stated
   sanitization was *"not yet implemented"* — `serving/app/services/sanitize.py`
   exists with 21 passing tests — and the Not-built-yet column still listed
   "Alembic migrations" and "Prompt-injection sanitization", both of which ship.

---

## 7. Known gaps

- **`task.md` is gitignored** (`.gitignore:10`) and untracked. Its Phase 3
  section was updated locally per audit §8, but that change is not part of
  either checkpoint commit and will not appear in the repository. This is the
  repository's existing convention, not a change made here.
- **"JD → approved rubric in under 3 minutes" remains unverified.** It needs a
  live LLM provider key and manual timing. It is left unchecked in `task.md`
  rather than claimed.
- **The route guard checks presence, not accuracy.** A row whose description is
  wrong still passes; only the method and path are compared.
- **`.gitignore` carries an unrelated uncommitted edit** (`.agent`) that predates
  this session. Left untouched.
- **Skill-taxonomy linking, the rubric editor UI, and golden-set labelling**
  remain blocked on external dependencies, exactly as the audit stated. Nothing
  here changes that.