Phase 5 prep: harden eval CI against the HF-429 judge flake

Deploy-readiness audit + CI hygiene ahead of the Hugging Face Docker Space.

The live LLM-judge step was blocking and twice reddened the workflow on a
transient HF-Hub HTTP 429 while pulling the all-MiniLM-L6-v2 embedding model
during build_index. The deterministic safety gate (the real blocking gate)
always passed; only the advisory judge flaked on an external dependency.

- Make the judge step continue-on-error (advisory). Per this repo's own design,
only the deterministic gate may fail the build; the nondeterministic live
model and its external deps (HF Hub, Anthropic API) must never gate.
- Cache ~/.cache/huggingface (actions/cache) so the embedding model isn't
re-downloaded every run — removes the 429 root cause on warm caches.

Same model download happens in the Space's Docker build (build_index), so a
transient 429 there is a one-shot rebuild, not a code bug — documented in STATUS.

Build readiness verified: README HF frontmatter (sdk: docker, port 7860),
multi-stage Dockerfile, frontend/package-lock.json (npm ci), api.py imports no
eval/ dep, requirements has fastapi+uvicorn. React stage builds clean locally.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.github/workflows/eval.yml

@@ -36,10 +36,25 @@ jobs:
       - name: Deterministic safety gate (free, blocking)
         run: python -m eval.run_eval --deterministic-only
 
+      # Cache the local embedding model so the judge step's build_index doesn't re-download it from
+      # the HF Hub on every run (that download has rate-limited to HTTP 429 and is the only thing
+      # that has ever reddened this workflow). Cold cache still downloads; a warm cache skips it.
+      - name: Cache embedding model (all-MiniLM-L6-v2)
+        if: ${{ env.ANTHROPIC_API_KEY != '' }}
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/huggingface
+          key: hf-${{ runner.os }}-all-MiniLM-L6-v2
+
       # Live layers run only if the secret is present. Builds the policy/precedent index first
       # (local embeddings, no key needed), then triages a 5-case subset with the LLM-as-judge.
+      # ADVISORY, not blocking (continue-on-error): per this repo's design the *deterministic* gate
+      # above is the only thing that may fail the build. The live model is nondeterministic and its
+      # external deps (HF Hub, Anthropic API) can flake, so it must never gate. Results are still
+      # uploaded below; a real judge regression surfaces there, not as a red required check.
       - name: LLM-judge on a subset (only if ANTHROPIC_API_KEY is set)
         if: ${{ env.ANTHROPIC_API_KEY != '' }}
+        continue-on-error: true
         run: |
           python -m scripts.build_index
           python -m eval.run_eval --judge --limit 5 --strict

STATUS.md

@@ -252,12 +252,16 @@ gate's invariants become a CI gate. New/changed:
   if `ANTHROPIC_API_KEY` is set** (so forks/unconfigured repos stay green and the model never gates
   the build). Results uploaded as an artifact.
 
-**Verified:** `eval.run_eval --deterministic-only` → **10/10**, exit 0, fully offline (no key, no
-index, agent import kept lazy so the free gate stays light); `build_testset --check` green; live
-harness exercised end-to-end (Anthropic 2-case + judge, and a full **free local Ollama** run, zero
-spend); `scripts.smoke_test` still 12/12. **CI is fully green on GitHub Actions** — the free
-deterministic gate blocks every push, and the `ANTHROPIC_API_KEY` secret is now set, so the **live
-judge step ran green in CI** (`build_index` + 5-case triage + judge, `--strict`, 2m31s).
+**Verified:** `eval.run_eval --deterministic-only` → **10/10** (now **12/12** after Phase 4.1),
+exit 0, fully offline (no key, no index, agent import kept lazy so the free gate stays light);
+`build_testset --check` green; live harness exercised end-to-end (Anthropic 2-case + judge, and a
+full **free local Ollama** run, zero spend); `scripts.smoke_test` still 12/12. **CI: the free
+deterministic gate blocks every push and is reliably green**; the `ANTHROPIC_API_KEY` secret is set
+so the **live judge step runs** (`build_index` + 5-case triage + judge, `--strict`). *Caveat (fixed
+in Phase 5 prep):* the judge step was **blocking** and twice reddened the run on a transient HF-Hub
+**HTTP 429** while downloading the embedding model — never a code/safety failure. The judge is now
+**advisory (`continue-on-error`)** + the model is **cached**, matching the design intent that only
+the deterministic gate gates the build. See the Phase 5 prep note below.
 
 **Repo is on GitHub:** `m-misbahuddin/amana-triage-copilot` (**private**), default branch `main`
 carries the full project; work continues on `feature/ts-triage-copilot`. CI: `.github/workflows/eval.yml`.
@@ -314,6 +318,30 @@ substance.
 debt-principal showcase APPROVE is preserved. **Not yet re-run:** the full `--judge` Anthropic eval
 for the headline video numbers (user action).
 
+## What's IN PROGRESS — Phase 5: Deploy prep 🛠️ (2026-06-06)
+
+**Build-readiness audited before handing the image to Hugging Face.** No new app code — verifying the
+container will actually build, and hardening the one thing that has ever flaked.
+- **Verified ready:** README HF frontmatter (`sdk: docker`, `app_port: 7860`) ✓; multi-stage
+  `Dockerfile` + `.dockerignore` ✓; `frontend/package-lock.json` present so `npm ci` works ✓;
+  `api.py` imports only `fastapi`/`pydantic`/`src.*` (no `eval/` dep — Dockerfile correctly omits
+  `eval/`) ✓; `requirements.txt` has fastapi + uvicorn[standard] + anthropic + chromadb ✓.
+- **Stage 1 (React) builds clean locally** (`npm run build` → 1535 modules, dist emitted).
+- **Stage 2 (`build_index`)** proven by the smoke suite (local embeddings, no key).
+- **CI hardening (`.github/workflows/eval.yml`):** the live judge step was *blocking* and twice
+  reddened the workflow on a transient **HF-Hub HTTP 429** while pulling `all-MiniLM-L6-v2` (the
+  deterministic gate always passed). Fixed: judge is now **`continue-on-error`** (advisory — only the
+  deterministic gate blocks, matching the workflow's own stated design) and the model is **cached**
+  (`actions/cache` on `~/.cache/huggingface`) to avoid the 429 across runs.
+- **⚠ Deploy-build risk (same root cause):** the Space's Docker build runs the *same* `build_index`
+  → *same* `all-MiniLM-L6-v2` download. A 429 during the HF Space build would fail the build the same
+  way — it's a one-shot build, so **just rebuild the Space** if it hits a transient 429 (or build the
+  image locally first, where the model layer caches). Not a code bug.
+
+**Remaining (needs Docker Desktop running / an HF account — user actions):** local `docker build` +
+`docker run` smoke; create the HF **Docker** Space; set `ANTHROPIC_API_KEY` + `LLM_PROVIDER=anthropic`
+as Space secrets; add the Space git remote and push. Recipe in `PLAN.md` §5b.
+
 ## AWAITING — user action
 - **Build/run the container** (start Docker Desktop): `docker build -t amana .` then
   `docker run -p 7860:7860 -e ANTHROPIC_API_KEY=sk-... amana` → open http://localhost:7860.
@@ -334,21 +362,22 @@ for the headline video numbers (user action).
 FastAPI console), 3.7 (policy drawer), 4 (eval + CI), and **4.1 (gate hardening)** are **done and
 verified** (see the DONE sections above; build log in `docs/DEVLOG.md`). **On GitHub** at
 `m-misbahuddin/amana-triage-copilot` (private); work branch `feature/ts-triage-copilot`, full project
-also on `main`. **CI is green incl. the live judge step** (secret is set). The two eval findings are
-**closed** (gate corroborates hard citations + enforces ELIG-4). **Not yet committed** — Phase 4.1
-changes are in the working tree (gate.py, run_eval.py, smoke_test.py, STATUS/DEVLOG); commit + push
-to re-run CI.
+also on `main`. Phase 4.1 gate hardening is **committed + pushed** (`438e8e8`) and **CI passed**
+(12-check deterministic gate + judge both green on that run). The two eval findings are **closed**
+(gate corroborates hard citations + enforces ELIG-4). **Phase 5 deploy prep is underway** — see the
+"Phase 5: Deploy prep" section above; CI judge step hardened against the HF-429 flake (advisory +
+cached). The eval.yml/STATUS/DEVLOG deploy-prep edits may be uncommitted in the working tree.
 
 **Pick up here:**
-1. *(commit first)* Commit the Phase 4.1 gate-hardening changes on `feature/ts-triage-copilot` and
-   push so the CI deterministic gate (now 12 checks) + live judge re-run green.
-2. *(then) **Phase 5 — Deploy.** Create the Hugging Face **Docker** Space; push the repo to its git
-   remote; set `ANTHROPIC_API_KEY` + `LLM_PROVIDER=anthropic` as Space secrets. Verify
-   `docker build -t amana .` + `docker run` locally first (start Docker Desktop). See `PLAN.md` §5.
-3. *(user, for the video)* Run a **full Anthropic eval** (`python -m eval.run_eval --judge`) for the
+1. *(Phase 5 — Deploy, user actions)* Start Docker Desktop and run `docker build -t amana .` +
+   `docker run -p 7860:7860 -e ANTHROPIC_API_KEY=sk-... amana` → open http://localhost:7860 to smoke
+   the container. Then create the Hugging Face **Docker** Space, set `ANTHROPIC_API_KEY` +
+   `LLM_PROVIDER=anthropic` as Space secrets, add the Space git remote, and push. See `PLAN.md` §5b.
+   *(If the Space build hits a transient HF-Hub 429 on the embedding-model pull, just rebuild it.)*
+2. *(user, for the video)* Run a **full Anthropic eval** (`python -m eval.run_eval --judge`) for the
    demo's headline numbers — escalation recall / reject precision should now improve on camp-011/012/
    018 — and the in-browser click-through; switch to **Ollama** to show the gate firing on a weak model.
-4. *(optional hardening)* the Chroma RustBindings flake noted under Phase 3.5 (the eval harness
+3. *(optional hardening)* the Chroma RustBindings flake noted under Phase 3.5 (the eval harness
    already isolates it as an `ERROR` row so it can't tank a run).
 
 **Progress:** ✅ Phase 1 data · ✅ Phase 2 agent+tools · ✅ Phase 2.5 local-LLM toggle · ✅ Phase 3