_test/PLAYBOOK.md

PLAYBOOK — sapptest procedures

Numbered procedures an agent follows verbatim. Shared conventions first — every procedure references them. Audience: an LLM operator with full repo access. All paths are relative to _test/ unless they start with _project/ or ./.

---

Conventions (read once, used by every procedure)

C1. Finding IDs — F-NNNN, stable across runs

IDs are content-hash derived so re-runs update findings instead of duplicating them, and body-wording tweaks never change the ID.

1. Title slug: lowercase the finding title; replace every run of non-alphanumerics with -; trim leading/trailing -; cut to 40 chars.
2. ID key: <primary file>|<slug> where primary file is the first entry in the finding's files list (repo-relative path). Example: src/auth/jwt.ts|login-rejects-valid-jwt
3. NNNN: decimal CRC of the key, mod 10000, zero-padded to 4 digits:

   printf '%s' 'src/auth/jwt.ts|login-rejects-valid-jwt' | cksum | awk '{printf "%04d\n", $1 % 10000}'
   # → 7204  ⇒  id F-7204, file findings/F-7204-login-rejects-valid-jwt.md
   

4. Dedup before minting. Grep findings/ for an existing finding with the same id_key, or any open finding on the same primary file with the same root cause. If found, update that file — never mint a second ID for the same issue, even if you would word the title differently today.
5. Collision: if the computed NNNN is already used by a finding with a different id_key, increment NNNN by 1 (mod 10000) until free. Keep id_key set to the original key so future dedup still matches.
6. Filename: findings/F-NNNN-<slug>.md. If a title is later reworded, keep the original id and id_key; ID stability beats slug accuracy.

C2. Writing a finding

1. Copy the matching template: templates/finding.template.md (general), templates/bug.template.md (confirmed bug), or templates/refactor.template.md (refactor proposal).
2. Fill every front-matter field. Long fields (repro, expected, actual, suggested_fix) use YAML | block scalars.
3. Severity per config/severity.yml definitions — judge impact, not effort.
4. Validate against schemas/finding.schema.json: every required field present, enum values exact, files non-empty. Fix the file, never the schema.
5. A finding must be self-sufficient: an agent reading only that file can reproduce and fix the issue.
6. Findings are live state: when a later pass verifies an issue is fixed, delete the finding file (and its synced lines, C4). Never edit reports/ history.

C3. Computing the verdict

1. Count open findings in findings/F-*.md by severity.
2. Apply config/ship-rules.yml top-down (first match wins):
   • any sev1 count > block_thresholds.sev1 → BLOCK, reason sev1_open
   • sev2 count > block_thresholds.sev2 → BLOCK, reason sev2_over_threshold
   • sev2 count ≥ caveat_thresholds.sev2 or sev3 count ≥ caveat_thresholds.sev3 → SHIP WITH CAVEATS, reason open_findings_below_threshold
   • otherwise → SHIP, reason clean (or minor_findings_only if any sev3/sev4 remain open)
3. audit_date = today (YYYY-MM-DD). Every procedure that examined code and reached this step refreshes it. Staleness (audit_staleness_days, and verdict older than the newest commit) is enforced by the reader — /ship — not pre-computed here.
4. blocking = the findings that triggered a BLOCK (all open sev1s; the sev2 overflow). Empty array otherwise.
5. Write verdict/latest.json (shape: templates/verdict.example.json, must validate against schemas/verdict.schema.json) and verdict/latest.md from templates/verdict.template.md. Overwrite both.

C4. TODO / CURRENT_STATE sync (one-way, tagged, idempotent)

sapptest owns exactly the lines it tagged <!-- F-NNNN -->. Never touch untagged lines; never let user edits to tagged lines survive (they are overwritten).

1. For each open finding, upsert one line into _project/TODO.md under the section matching todo_priority in config/severity.yml ("Next up" / "Backlog" / "Ideas / maybe"):

   - [sev2] login rejects valid JWT — _test/findings/F-7204-login-rejects-valid-jwt.md <!-- F-7204 -->
   

2. Upsert = if a line containing <!-- F-NNNN --> exists, replace it in place; move it only if its section no longer matches the severity's todo_priority. Otherwise append to the right section.
3. Delete any tagged line whose F-NNNN no longer has a findings/ file.
4. For open sev1/sev2 findings, also upsert into _project/CURRENT_STATE.md → "known broken / flaky" section, same line format, same tag. Remove on close.

C5. Runners — normalized exit codes

config/adapters.yml maps detection files → runner. Run every adapter that matches (mixed stacks run all). From the project root: bash _test/runners/run-<lang>.sh . Exit codes: 0 pass · 1 test failures · 2 not applicable / no tests · 3 environment error. Read the native output — it names the failing tests.

---

full-audit

Non-destructive. Writes only to findings/, reports/, verdict/, and tagged _project/ lines.

1. Read prompts/audit.md, config/coverage-goals.yml, config/ship-rules.yml. Note the depth each category demands.
2. Detect stacks via config/adapters.yml; run each matching runner (C5). Record exit codes and failing test names. A failing native test is automatically a finding (sev2 unless impact says otherwise).
3. Map the codebase: entry points, core modules, data writes, git log --oneline -30 for churn hot spots.
4. Audit per prompts/audit.md, applying the specialist lenses it names (prompts/security.md, prompts/a11y.md, prompts/perf.md) where coverage-goals require them. Confirm every issue by reading the code — no speculative findings.
5. For each confirmed issue: dedup (C1.4), then write a finding (C2).
6. Sweep existing findings/: any that the codebase shows are fixed → verify, then delete the file.
7. Write reports/audit-YYYY-MM-DD.md: scope covered vs coverage-goals, runner results, table of findings (ID, sev, title), closed findings, verdict line. Append-only — never rewrite old reports.
8. Compute the verdict (C3). Sync (C4).
9. Do not commit. Show the user what changed and the verdict.

bug-hunt

Targeted. Input: a symptom, error message, or suspect file list.

1. Read prompts/bug-hunt.md. Restate the symptom as expected vs actual.
2. Trace from the symptom to candidates: entry point → handler → data. Read the implicated code paths fully. Run the relevant runner (C5) or a single native test if it shortens the hunt.
3. Confirm the root cause — name file and line. If you cannot confirm, report what you ruled out; write no speculative finding.
4. Write one finding per confirmed root cause using templates/bug.template.md (C2), with a minimal repro.
5. Findings changed → recompute the verdict (C3) and sync (C4).
6. Report root cause + finding ID to the user. Do not fix it in this pass — fixes go through refactor or normal feature work.

regression-check

Input: a diff, branch, or commit range ("did this change break anything?").

1. Read the diff. List behaviors the change could plausibly affect — callers of changed functions, shared state, changed contracts/schemas.
2. Run runners (C5) for every affected stack.
3. For each at-risk behavior, read the post-change code and confirm it still holds. Check reports/ for previously closed issues in the touched files — regressions of past findings get priority.
4. New breakage → findings via templates/bug.template.md (C2), noting the offending commit in repro.
5. Findings changed → recompute verdict (C3), sync (C4). Nothing found → say so; refresh the verdict only if you audited deeply enough to vouch for it (C3 sets audit_date).

refactor

Two phases. Phase A proposes; phase B opens PRs. /refactor runs phase B.

A — propose (code untouched):

1. Read prompts/refactor.md. Identify improvements with concrete payoff (duplication, dead code, complexity hot spots, API awkwardness).
2. Write each as a finding via templates/refactor.template.md with refactor: true and a suggested_fix precise enough to apply without re-deriving it (C2). Recompute verdict (C3), sync (C4).

B — execute (PR-per-finding):

1. Read refactor_mode from config/ship-rules.yml (per-finding default, batched opt-in).
2. Filter findings/F-*.md where refactor: true. None → say so, stop.
3. Per finding (per-finding mode): a. git switch -c refactor/F-NNNN-<slug> from the working branch. b. Apply only the finding's suggested_fix. Don't improvise beyond it. c. Run the host project's ./verify.sh. Fails → abort this PR, switch back, continue with the next finding. d. Commit, push, open a PR titled [F-NNNN] <summary>, body linking _test/findings/F-NNNN-<slug>.md. Switch back to the working branch.
4. Batched mode: one branch, one commit, one PR listing every F-NNNN.
5. Do not mark findings fixed or delete them — the next audit verifies merged fixes and closes them. Never merge your own PRs.

pre-ship

Deeper gate than ./verify.sh, run before /ship commits.

1. Read prompts/pre-ship.md. Identify what is shipping: git status --short + git diff against the last shipped state.
2. Run the host ./verify.sh. Fails → stop, report, no-go.
3. Run runners (C5) for affected stacks.
4. Check verdict/latest.json: if audit_date predates the newest commit or exceeds audit_staleness_days, the verdict is stale — run regression-check (small change) or full-audit (large) first.
5. Cross-check the diff against open findings: if shipped files appear in any open sev1/sev2 finding, verify the change doesn't worsen it.
6. Smoke the changed behavior per prompts/pre-ship.md.
7. New issues → findings (C2), recompute verdict (C3), sync (C4).
8. Tell the user: go / no-go, the verdict, and any caveats needing acknowledgment. /ship enforces the verdict; you never bypass it.

docs-drift

Compares _project/*.md claims against code. sappcode's /sync delegates here. Cheap first, thorough second.

1. Read prompts/docs-drift.md. Sample reality before opening docs: git log --oneline -30, git status --short, ls at root.
2. Follow the prompt's claim checklist (env vars, commands, dependencies, architecture claims, CURRENT_STATE staleness). Open only the docs the git evidence implicates, plus _project/CURRENT_STATE.md always.
3. Each confirmed divergence → finding, category: docs-drift, severity sev3 (misleading runbook/env claims) or sev4 (cosmetic staleness), with suggested_fix containing the corrected doc text (C2).
4. Findings changed → recompute verdict (C3), sync (C4).
5. Propose the doc edits to the user. Apply them only with approval — and then delete the corresponding findings, since the drift is gone.