docs(cp6): handoff close — difficulty layouts + category eval + compare shipped; CP7 next

HANDOFF.md

@@ -4,7 +4,8 @@
 > `docs/superpowers/specs/2026-06-01-proteus-arena-slice-design.md` (immutable design) +
 > the per-CP plan in `docs/superpowers/plans/` + this file.
 
-**Last updated:** 2026-06-02 — CP5 complete (human play + trace visualization; offline, headless).
+**Last updated:** 2026-06-02 — CP6 complete (difficulty layouts + category-specific eval +
+human-baseline `compare` harness; offline, headless).
 
 ## Done (CP0–CP3)
 
@@ -110,6 +111,21 @@ Executed task-by-task via subagent-driven-development (3-gate: implement → spe
 - **Visual:** `replay --visual --fps 0` renders the truecolor grid + side panel; `replay --png
   runs/cp5_frames` wrote **5 non-zero** `frame_*.png` (cut_length 2 + 1 initial + 2 play turns).
 
+## Done (CP6 — difficulty layouts + category eval + human-baseline harness; plan: `docs/superpowers/plans/2026-06-02-proteus-cp6-difficulty-eval-baseline.md`; design: `docs/superpowers/specs/2026-06-02-proteus-cp6-difficulty-eval-baseline-design.md`)
+
+Executed task-by-task via subagent-driven-development (3-gate: implement → spec review → code-quality review). All offline/headless; no network, no display, no provider SDK added. **Full suite: 127 passed** (103 CP5 baseline + 24 CP6 additions).
+
+- **Section A — difficulty layouts** (Tasks 1–3): `Scenario.build_level(rng, difficulty)` now takes the band; `record_focal_move`/`safety_distance` promoted to concrete defaults on the `Scenario` ABC (resolves a CP5 deferred item). `predator_evade` dispatches a **hand-authored per-difficulty layout table** (`_Layout` dataclass + `_LAYOUTS`): EASY 8×8 (byte-for-byte unchanged), MEDIUM 10×10, HARD 12×12 (L-trap), EXPERT 12×12 (forked corridor). `build_level` sets the **instance** attr `self.grid_size` so the camera/`within_bounds` size per-band. Golden test `tests/grid/test_difficulty_layouts.py` is the oracle: per band it locks determinism + the handover + the diagnostic invariant `optimal_action != habit_action` with `habit == "left"` blocked. Candidate coordinates satisfied the invariant with **no tuning** (all four bands: optimal=`up` ≠ habit=`left`). `rules_text` de-hard-coded ("8x8 grid" → "a grid") so it no longer lies on the larger bands.
+- **Section B — category-specific eval** (Tasks 4–6): per-turn reward **ownership moved into the scenario**. New abstract `Scenario.step_reward(game, action, blocked, focal_before, predator_before)`; `SessionRunner._reward` + its 5 constants **deleted** (delegates now). predator_evade survival reward = BFS-distance gain measured against the predator's **pre-move** cell (away→positive `float(d_after − d_before)`, toward→negative, wall-hit→`−3.0`, terminal `−50/+50`), isolating the agent's own move from the chase. New `proteus/runtime/rollout.py` `optimal_rollout(...)` (engine-replay, same pattern as `viz.reconstruct`, imports `grid` only). `metrics.py` gains four additive keys: **`away_move_fraction`** (survival headline), `mean_step_reward`, `trajectory_agreement` (vs the optimal rollout, denominator = all played turns), `final_distance_gap`. Existing four metrics retained; `test_human_comparability`'s `set(h.metrics)==set(v.metrics)` still holds.
+- **Section C — human-baseline harness** (Tasks 7–8): pure `proteus/runtime/aggregate.py` `aggregate_traces(traces) → {(model, difficulty): {"n", "metrics": means}}` (no I/O, union-of-keys robust). New CLI `proteus compare <trace.jsonl>… [--out summary.json]` (rc 2 not-found / rc 1 empty / rc 0, mirrors `replay`; JSON keys stringified `"model|difficulty"`).
+
+### CP6 acceptance demo (passed)
+
+- **Full suite:** `.venv/bin/python -m pytest -q` → **127 passed**, offline, no display.
+- **Per-difficulty smoke:** `proteus run … --difficulty {easy,medium,hard,expert} --model fake:demo --seed 42 --play-turns 6 --no-probe` then `proteus compare …` → four `model=demo difficulty=<band> n=1` groups, each with the 8 metric means (incl. `away_move_fraction`/`trajectory_agreement`/`final_distance_gap`), summary written to `runs/cp6_summary.json`. Values differ by band (e.g. EXPERT `trajectory_agreement=66.67`, `survival_fraction=50.0` vs others `50.0`/`33.33`), confirming the layouts/metrics are genuinely band-sensitive.
+- **Invariant acceptance:** all four bands print `optimal=up habit=left` with distinct grids `(8,8)/(10,10)/(12,12)/(12,12)` — `optimal != habit` everywhere.
+- **Note:** the `fake:demo` provider plays a fixed non-optimal action (so `motive_reading_accuracy=0`); the smoke validates the deterministic pipeline + metric surfacing, not agent skill.
+
 ## How to run
 
 - Environment: `python` is **not** on PATH; a gitignored venv lives at `.venv` (Python 3.12,
@@ -124,6 +140,11 @@ Executed task-by-task via subagent-driven-development (3-gate: implement → spe
   or `--png runs/frames` (per-frame PNGs). Plain `replay runs/h.jsonl` stays text-only.
 - Offline CLI smoke: `.venv/bin/python -m proteus run --scenario predator_evade --model fake:demo
   --seed 42 --play-turns 5 --out runs/x.jsonl && .venv/bin/python -m proteus replay runs/x.jsonl`
+- Difficulty bands: add `--difficulty {easy,medium,hard,expert}` to `run`/`play` (EASY is the default,
+  byte-for-byte the CP0–CP5 world). Human-baseline compare: collect matched human (`play --out`) and LLM
+  (`run --out`) traces, then `.venv/bin/python -m proteus compare h.jsonl llm.jsonl --out summary.json`
+  → per-`(model, difficulty)` metric means + n (the `model` token is the part AFTER the colon, e.g.
+  `fake:demo` → `demo`).
 - Real-model smoke: see the throwaway-venv recipe above (needs a provider SDK + key; never in `.venv`).
 
 ## Current state
@@ -136,12 +157,23 @@ trace as truecolor terminal frames or PNGs by deterministically replaying the en
 with no flags still prints TEXT (CP4 behavior preserved). `proteus/arc_grid/rendering.py` is now actively
 reused (palette + ANSI + RGB helpers) and its matplotlib import is lazy.
 
-## Next: CP6 (difficulty layouts + metric refinement + human-baseline harness) — needs its own plan
+**CP6 adds:** four genuinely distinct hand-authored difficulty layouts (EASY unchanged), scenario-owned
+category-specific reward (`step_reward`; survival = reward for moving away from the predator), an
+optimal-rollout trajectory metric (`runtime/rollout.py`) + four new metric keys, and a `proteus compare`
+human-baseline aggregation CLI. Still a single scenario (`predator_evade`, survival motive) — the
+`step_reward`/`safety_distance` ABC surface is now **open** for CP7 to plug new motive categories in
+without touching `SessionRunner`.
+
+## Next: CP7 (new motive categories / scenarios) — needs its own plan
 
-Per spec §11. Likely scope: per-difficulty layouts (beyond the single EASY handover), trajectory-metric
-refinement (see open questions — `first_divergence_turn` is coarse), and a **human-baseline harness**
-that builds on CP5's `HumanAgent` to collect/aggregate human traces for direct comparison against LLM
-runs at matched seeds. Brainstorm → write the CP6 plan → execute.
+Per spec §12 (deferred from CP6). CP6 opened the `step_reward`/`safety_distance` ABC interface so a new
+motive **category** (e.g. curiosity = reward for visiting novel cells; sociality) can be added as a new
+`Scenario` subclass that defines its own per-turn reward — **without touching `SessionRunner`** (reward is
+now scenario-owned) or the metrics/compare harness (additive, category-agnostic). Likely scope: a second
+scenario + its category reward + its own diagnostic invariant golden, reusing the difficulty-layout and
+rollout machinery. The `VanillaAgent._ACTION_DIRECTIVE` / `SessionRunner._PROBE_QUESTION` predator-framing
+(carried below) should be sourced from the `Scenario` when the second scenario lands. LLM-as-judge scoring
+of reasoning and the web UI / leaderboard remain deferred (spec §12). Brainstorm → write the CP7 plan → execute.
 
 ## Deferred items (carry forward)
 
@@ -152,9 +184,9 @@ runs at matched seeds. Brainstorm → write the CP6 plan → execute.
   explicit `model_name` on the `Agent` base when more agent types land. (CP5's `HumanAgent` has no
   `_provider`, so `model` correctly falls back to `agent.name == "human"`.)
 - reasoning/probe are recorded but **not scored** (LLM-as-judge deferred per spec §12).
-- **`Scenario.record_focal_move` is not on the ABC** — only concrete `PredatorEvade` implements it, and
-  both `SessionRunner` and `viz.reconstruct` call it. Fine for the single scenario; add a no-op default
-  on the `Scenario` base when CP6 adds scenarios (else `AttributeError`).
+- ~~**`Scenario.record_focal_move` is not on the ABC**~~ — **RESOLVED (CP6 Task 1):** promoted to a
+  non-abstract no-op default on `Scenario` (alongside `safety_distance`), so multi-scenario callers
+  never hit `AttributeError`.
 - **CLI `--png` without the `viz` extra** raises a raw `ModuleNotFoundError` traceback instead of a clean
   rc=2 (every other expected CLI failure returns a structured code). Not reachable today (matplotlib is in
   `.venv` and no packaging `extras` are defined yet). Add a `try/except ImportError` guard + companion
@@ -168,8 +200,11 @@ runs at matched seeds. Brainstorm → write the CP6 plan → execute.
 ## Open questions
 
 - Package/project README rewrite (replace the deprecated persona-arena `depreated/README.MD`).
-- Trajectory metric: `first_divergence_turn` is a coarse proxy; CP6 should add an optimal-rollout
-  step-agreement / final-distance metric (spec §7 "궤적 일치").
+- ~~Trajectory metric: `first_divergence_turn` is a coarse proxy~~ — **RESOLVED (CP6):** added
+  `trajectory_agreement` (position step-agreement vs the optimal rollout) + `final_distance_gap` (spec
+  §7 "궤적 일치"). `first_divergence_turn` is **kept additively**; open whether to retire it once the new
+  pair proves sufficient (CP6 design §8). Also open (CP6 design §5.2): exact reward grading (delta vs ±1)
+  is pinned by the property test, revisit if a future category needs a different shape.
 - If a 3rd LLM-call result type appears (e.g. a reflect call), extract a shared `_LLMCallResult`
   base for the common `reasoning/raw_text/{input,output,thinking}_tokens` fields (ActResult/ProbeResult
   currently duplicate them — acceptable for two).