Spaces:
Runtime error
Runtime error
Commit ·
9a23797
1
Parent(s): 8431307
docs(cp8): agentness-eval plan (5 passes + auto-GIF) + HANDOFF Next
Browse filesThree-layer eval (survival / distance / persona-maintenance) per the user's
predator-agentness spec. Six staged handoff units: Stage0 auto-GIF + Stage1
metric-only + Stage2 persona-ref + Stage4 hidden-weight memory are TDD-ready
(full code/tests, additive, 152-safe); Stage3 simultaneous-resolver and Stage5
multi-feature personas are design-gated (need their own brainstorm+sub-spec).
Implementation deferred to next session(s).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HANDOFF.md
CHANGED
|
@@ -211,18 +211,36 @@ was **renamed to CP8** when the user pivoted to this memory feature; curiosity/s
|
|
| 211 |
with `httpx`; key = first whitespace token of the `.env` value, CP4 gotcha). `runs/` is gitignored, so
|
| 212 |
the real trace/checkpoint/key were never committed.
|
| 213 |
|
| 214 |
-
## Next: CP8 (
|
| 215 |
-
|
| 216 |
-
|
| 217 |
-
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
|
| 225 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 226 |
|
| 227 |
## Deferred items (carry forward)
|
| 228 |
|
|
|
|
| 211 |
with `httpx`; key = first whitespace token of the `.env` value, CP4 gotcha). `runs/` is gitignored, so
|
| 212 |
the real trace/checkpoint/key were never committed.
|
| 213 |
|
| 214 |
+
## Next: CP8 (predator agentness evaluation + auto-GIF) — plan written, NOT yet implemented
|
| 215 |
+
|
| 216 |
+
Design: `docs/superpowers/specs/2026-06-02-proteus-predator-agentness-eval-design.md` (user-authored).
|
| 217 |
+
Plan: `docs/superpowers/plans/2026-06-02-proteus-cp8-agentness-eval.md`. **All implementation is next
|
| 218 |
+
session(s)** — the plan is staged for handoff.
|
| 219 |
+
|
| 220 |
+
Replaces the single `survived` signal with a **three-layer agentness eval** — survival, distance
|
| 221 |
+
trajectory, and **memory-persona maintenance** (does the model continue the persona its self-memory
|
| 222 |
+
demonstrates, not just survive?). Six stages, each a handoff unit:
|
| 223 |
+
- **Stage 0 — auto-GIF** (TDD-ready, independent): `viz.write_gif` + `run`/`play` auto-render `<out>.gif`
|
| 224 |
+
(default on, `--no-gif`). User-requested; do this first for an immediate win.
|
| 225 |
+
- **Stage 1 — metric-only** (TDD-ready, additive, 152-safe): per-turn pre/post BFS distance + post
|
| 226 |
+
positions on `TurnTrace`; scenario `max_bfs_distance`/`agent_distance_delta` helpers; episode metrics
|
| 227 |
+
`time_to_capture`/`distance_auc`/`min_distance`/`near_capture_count`; record `turn_order`/`capture_rule`/
|
| 228 |
+
`horizon`. `away_move_fraction` kept (already pre-predator-based).
|
| 229 |
+
- **Stage 2 — persona reference** (TDD-ready, additive): `runtime/persona.py` hidden `PersonaWeights` +
|
| 230 |
+
`R_w` reference policy + `pressure`; persona metrics `action_agreement`/`reward_regret`/
|
| 231 |
+
`pressure_weighted_agreement`/`persona_drift_turn`; `run --persona <id>` (weights never in the prompt).
|
| 232 |
+
- **Stage 3 — simultaneous resolver** (DESIGN-GATED): `plan→resolve` engine turn + crossing capture;
|
| 233 |
+
changes goldens → needs its own brainstorm+sub-spec; keep `focal_then_predator` selectable.
|
| 234 |
+
- **Stage 4 — hidden-weight memory** (TDD-ready): `generate_memory(persona=)` produces a persona
|
| 235 |
+
*demonstration* memory (public `persona_weight_id`, raw weights never serialized to the participant view).
|
| 236 |
+
- **Stage 5 — multi-feature personas** (DESIGN-GATED): greed/compliance/cooperation need new scenario
|
| 237 |
+
features (resources/norms/social) — this is where the previously-deferred "new motive category /
|
| 238 |
+
curiosity" work lands; needs its own spec. Predator-only must NOT over-interpret those personas (spec §9).
|
| 239 |
+
|
| 240 |
+
The `VanillaAgent._ACTION_DIRECTIVE` / `SessionRunner._PROBE_QUESTION` predator-framing (below) is still
|
| 241 |
+
hard-coded; source it from the `Scenario` when Stage 5's second scenario lands (CP7's `Scenario.memory_brief`
|
| 242 |
+
is the pattern). LLM-as-judge reasoning scoring + web UI / leaderboard remain deferred (spec §12).
|
| 243 |
+
Memory length today = `--memory-turns` (default 10; survived → exactly N turns, captured → fewer).
|
| 244 |
|
| 245 |
## Deferred items (carry forward)
|
| 246 |
|
docs/superpowers/plans/2026-06-02-proteus-cp8-agentness-eval.md
ADDED
|
@@ -0,0 +1,736 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# CP8 — Predator Agentness Evaluation (+ auto-GIF) Implementation Plan
|
| 2 |
+
|
| 3 |
+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
| 4 |
+
|
| 5 |
+
**Goal:** Replace the single-number "survived" evaluation with a three-layer agentness eval — **survival**, **distance trajectory**, **memory-persona maintenance** — per `docs/superpowers/specs/2026-06-02-proteus-predator-agentness-eval-design.md`; and auto-render a GIF of every played game.
|
| 6 |
+
|
| 7 |
+
**Architecture:** Five sequential passes (spec §8) + one independent auto-GIF stage. Passes are **additive-first**: Pass 1 (distance metrics) and Pass 2 (persona reference) keep the current `focal_then_predator` turn order and same-cell capture, so the 152-test baseline stays green. Pass 3 (simultaneous resolver) is the only pass that changes engine behaviour and is deliberately last-but-one so its test churn is isolated. Pass 4 wires hidden-weight memory generation into CP7's machinery. Pass 5 (multi-feature personas) is **design-gated** — it needs new scenario features (resources/norms/social) and its own spec before tasks exist.
|
| 8 |
+
|
| 9 |
+
**Tech Stack:** Python 3.12, pydantic v2, pytest. Offline (FakeProvider) for all automated tests; Ollama only for manual acceptance smokes. Pillow (in `.venv`) for GIF assembly; matplotlib stays lazy. Run tests with `.venv/bin/python -m pytest`.
|
| 10 |
+
|
| 11 |
+
**Design SSOT:** `docs/superpowers/specs/2026-06-02-proteus-predator-agentness-eval-design.md`.
|
| 12 |
+
|
| 13 |
+
**Baseline:** `.venv/bin/python -m pytest -q` → 152 passed (CP0–CP7). Every additive task keeps the full suite green; Pass 3 updates the goldens it changes under TDD.
|
| 14 |
+
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
## How to read this plan (handoff note)
|
| 18 |
+
|
| 19 |
+
All implementation happens in **future sessions**. Stages are independent handoff units:
|
| 20 |
+
|
| 21 |
+
| Stage | Pass | Readiness | Touches engine? | Regression risk |
|
| 22 |
+
|-------|------|-----------|-----------------|-----------------|
|
| 23 |
+
| **0** | auto-GIF | **TDD-ready** (full code below) | no | none (additive viz + CLI flag) |
|
| 24 |
+
| **1** | metric-only | **TDD-ready** (full code below) | no | none (additive fields/metrics) |
|
| 25 |
+
| **2** | persona reference | **TDD-ready** (signatures + spec formulas + tests below) | no | none (additive) |
|
| 26 |
+
| **3** | simultaneous resolver | **DESIGN-GATED** (decisions + acceptance below) | **yes** | high — updates capture goldens |
|
| 27 |
+
| **4** | hidden-weight memory | **TDD-ready** (builds on CP7 `generate_memory`) | no | none (additive) |
|
| 28 |
+
| **5** | multi-feature personas | **DESIGN-GATED** (needs its own spec) | new features | new scenario |
|
| 29 |
+
|
| 30 |
+
**DESIGN-GATED** stages must run a short `superpowers:brainstorming` + sub-spec at stage start before tasks are authored (the spec leaves these open on purpose). Do **not** fabricate their code from this plan.
|
| 31 |
+
|
| 32 |
+
Recommended next-session order: **Stage 0 → Stage 1** (both additive, immediately improve interpretation and demo), then 2, 4, then gate 3, then gate 5.
|
| 33 |
+
|
| 34 |
+
---
|
| 35 |
+
|
| 36 |
+
## File Structure
|
| 37 |
+
|
| 38 |
+
| File | Responsibility | Stage |
|
| 39 |
+
|------|----------------|-------|
|
| 40 |
+
| `proteus/viz/gif.py` (new) | `write_gif(steps, path, *, fps, hold_last)` — assemble reconstructed frames into an animated GIF (Pillow, lazy) | 0 |
|
| 41 |
+
| `proteus/viz/__init__.py` | export `write_gif` | 0 |
|
| 42 |
+
| `proteus/cli.py` | auto-GIF after `run`/`play` (default on, `--no-gif`); later `--turn-order`/`--persona` flags | 0,1,2,3,4 |
|
| 43 |
+
| `proteus/runtime/trace.py` | additive `TurnTrace` fields (post positions, pre/post BFS distance, agent_distance_delta, capture flags, persona fields) + `SessionTrace` fields (horizon, turn_order, capture_rule, persona_weight_id) | 1,2,3 |
|
| 44 |
+
| `proteus/grid/scenario.py` | additive `safety_distance` already exists; add `max_bfs_distance`, `agent_distance_delta` (default `None`) | 1 |
|
| 45 |
+
| `proteus/grid/scenarios/predator_evade.py` | implement the new distance helpers | 1 |
|
| 46 |
+
| `proteus/runtime/session.py` + `_session_core.py` | record post positions + pre/post distance per turn; pass to metrics | 1 |
|
| 47 |
+
| `proteus/runtime/metrics.py` | add `time_to_capture`, `distance_auc`, `min_distance`, `near_capture_count` (additive) | 1 |
|
| 48 |
+
| `proteus/runtime/persona.py` (new) | hidden persona weights + reference policy `R_w`; `reference_actions`, `reward_regret`, `pressure`; persona metrics | 2 |
|
| 49 |
+
| `proteus/grid/game.py` | `plan → resolve` simultaneous turn + crossing capture | 3 (gated) |
|
| 50 |
+
| `proteus/runtime/memory_gen.py` | drive memory self-play from a hidden persona weight | 4 |
|
| 51 |
+
|
| 52 |
+
---
|
| 53 |
+
|
| 54 |
+
## Stage 0 — Auto-GIF after every game (independent, TDD-ready)
|
| 55 |
+
|
| 56 |
+
### Task 0.1: `write_gif` in viz
|
| 57 |
+
|
| 58 |
+
**Files:**
|
| 59 |
+
- Create: `proteus/viz/gif.py`
|
| 60 |
+
- Modify: `proteus/viz/__init__.py`
|
| 61 |
+
- Test: `tests/viz/test_gif.py`
|
| 62 |
+
|
| 63 |
+
- [ ] **Step 1: Write the failing test**
|
| 64 |
+
|
| 65 |
+
```python
|
| 66 |
+
# tests/viz/test_gif.py
|
| 67 |
+
import proteus.grid # noqa: F401
|
| 68 |
+
from proteus.grid.difficulty import Difficulty
|
| 69 |
+
from proteus.providers import FakeProvider
|
| 70 |
+
from proteus.agents import VanillaAgent
|
| 71 |
+
from proteus.runtime import SessionRunner
|
| 72 |
+
from proteus.viz import reconstruct, write_gif
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def _trace():
|
| 76 |
+
prov = FakeProvider(["ACTION: up"] * 10, model_name="demo")
|
| 77 |
+
return SessionRunner(
|
| 78 |
+
"predator_evade", VanillaAgent(prov), difficulty=Difficulty.EASY,
|
| 79 |
+
seed=42, play_turns=4, use_probe=False,
|
| 80 |
+
).run()
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def test_write_gif_creates_animated_file(tmp_path):
|
| 84 |
+
steps = reconstruct(_trace())
|
| 85 |
+
out = tmp_path / "episode.gif"
|
| 86 |
+
path = write_gif(steps, out, fps=2)
|
| 87 |
+
assert path == out
|
| 88 |
+
assert out.exists() and out.stat().st_size > 0
|
| 89 |
+
# it is a GIF and has more than one frame (animated)
|
| 90 |
+
from PIL import Image
|
| 91 |
+
with Image.open(out) as im:
|
| 92 |
+
assert im.format == "GIF"
|
| 93 |
+
assert getattr(im, "n_frames", 1) >= 2
|
| 94 |
+
```
|
| 95 |
+
|
| 96 |
+
- [ ] **Step 2: Run test to verify it fails**
|
| 97 |
+
|
| 98 |
+
Run: `.venv/bin/python -m pytest tests/viz/test_gif.py -q`
|
| 99 |
+
Expected: FAIL (`ImportError: cannot import name 'write_gif'`).
|
| 100 |
+
|
| 101 |
+
- [ ] **Step 3: Write minimal implementation**
|
| 102 |
+
|
| 103 |
+
```python
|
| 104 |
+
# proteus/viz/gif.py
|
| 105 |
+
"""Assemble reconstructed frames into one animated GIF (Pillow, imported lazily).
|
| 106 |
+
|
| 107 |
+
Mirrors png.py's RGB rendering (frame_to_rgb_array) but writes a single animated
|
| 108 |
+
GIF instead of per-frame PNGs. Pillow ships with matplotlib in the `viz` extra;
|
| 109 |
+
the import is lazy so the offline core never needs it.
|
| 110 |
+
"""
|
| 111 |
+
|
| 112 |
+
from __future__ import annotations
|
| 113 |
+
|
| 114 |
+
from pathlib import Path
|
| 115 |
+
from typing import Union
|
| 116 |
+
|
| 117 |
+
from proteus.arc_grid.rendering import COLOR_MAP, frame_to_rgb_array
|
| 118 |
+
from proteus.viz.reconstruct import FrameStep
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def write_gif(
|
| 122 |
+
steps: list[FrameStep],
|
| 123 |
+
out_path: Union[str, Path],
|
| 124 |
+
*,
|
| 125 |
+
fps: float = 2.0,
|
| 126 |
+
scale: int = 32,
|
| 127 |
+
hold_last: float = 1.5,
|
| 128 |
+
) -> Path:
|
| 129 |
+
"""Render *steps* to an animated GIF at *out_path*; return the path.
|
| 130 |
+
|
| 131 |
+
Args:
|
| 132 |
+
steps: Reconstructed frames (from ``viz.reconstruct``).
|
| 133 |
+
out_path: Destination ``.gif`` file (parent dirs created).
|
| 134 |
+
fps: Frames per second (each frame shows for ``1/fps`` seconds).
|
| 135 |
+
scale: Pixel upscale per grid cell.
|
| 136 |
+
hold_last: Extra seconds to hold the final frame.
|
| 137 |
+
"""
|
| 138 |
+
from PIL import Image
|
| 139 |
+
|
| 140 |
+
out_path = Path(out_path)
|
| 141 |
+
out_path.parent.mkdir(parents=True, exist_ok=True)
|
| 142 |
+
frames: list[Image.Image] = []
|
| 143 |
+
for step in steps:
|
| 144 |
+
rgb = frame_to_rgb_array(0, step.frame, scale, COLOR_MAP)
|
| 145 |
+
frames.append(Image.fromarray(rgb).convert("P", palette=Image.ADAPTIVE))
|
| 146 |
+
if not frames:
|
| 147 |
+
raise ValueError("write_gif needs at least one frame")
|
| 148 |
+
|
| 149 |
+
per_frame_ms = int(1000 / fps) if fps > 0 else 600
|
| 150 |
+
durations = [per_frame_ms] * len(frames)
|
| 151 |
+
durations[-1] = max(per_frame_ms, int(hold_last * 1000))
|
| 152 |
+
frames[0].save(
|
| 153 |
+
out_path, save_all=True, append_images=frames[1:],
|
| 154 |
+
duration=durations, loop=0, disposal=2, optimize=True,
|
| 155 |
+
)
|
| 156 |
+
return out_path
|
| 157 |
+
```
|
| 158 |
+
|
| 159 |
+
Add to `proteus/viz/__init__.py`:
|
| 160 |
+
|
| 161 |
+
```python
|
| 162 |
+
from proteus.viz.gif import write_gif
|
| 163 |
+
```
|
| 164 |
+
(add `"write_gif"` to `__all__`).
|
| 165 |
+
|
| 166 |
+
- [ ] **Step 4: Run test to verify it passes**
|
| 167 |
+
|
| 168 |
+
Run: `.venv/bin/python -m pytest tests/viz/test_gif.py -q`
|
| 169 |
+
Expected: PASS (1 passed).
|
| 170 |
+
|
| 171 |
+
- [ ] **Step 5: Commit**
|
| 172 |
+
|
| 173 |
+
```bash
|
| 174 |
+
git add proteus/viz/gif.py proteus/viz/__init__.py tests/viz/test_gif.py
|
| 175 |
+
git commit -m "feat(cp8): viz.write_gif — assemble frames into an animated GIF"
|
| 176 |
+
```
|
| 177 |
+
|
| 178 |
+
### Task 0.2: auto-GIF after `run`/`play` (default on, `--no-gif`)
|
| 179 |
+
|
| 180 |
+
**Files:**
|
| 181 |
+
- Modify: `proteus/cli.py`
|
| 182 |
+
- Test: `tests/cli/test_cli_gif.py`
|
| 183 |
+
|
| 184 |
+
- [ ] **Step 1: Write the failing test**
|
| 185 |
+
|
| 186 |
+
```python
|
| 187 |
+
# tests/cli/test_cli_gif.py
|
| 188 |
+
from proteus.cli import main
|
| 189 |
+
|
| 190 |
+
|
| 191 |
+
def test_run_auto_writes_gif_next_to_out(tmp_path):
|
| 192 |
+
out = tmp_path / "t.jsonl"
|
| 193 |
+
rc = main([
|
| 194 |
+
"run", "--scenario", "predator_evade", "--model", "fake:demo",
|
| 195 |
+
"--difficulty", "easy", "--seed", "42", "--play-turns", "3",
|
| 196 |
+
"--no-probe", "--out", str(out),
|
| 197 |
+
])
|
| 198 |
+
assert rc == 0
|
| 199 |
+
assert (tmp_path / "t.gif").exists() # auto-GIF default on
|
| 200 |
+
|
| 201 |
+
|
| 202 |
+
def test_run_no_gif_suppresses(tmp_path):
|
| 203 |
+
out = tmp_path / "t.jsonl"
|
| 204 |
+
rc = main([
|
| 205 |
+
"run", "--scenario", "predator_evade", "--model", "fake:demo",
|
| 206 |
+
"--difficulty", "easy", "--seed", "42", "--play-turns", "3",
|
| 207 |
+
"--no-probe", "--no-gif", "--out", str(out),
|
| 208 |
+
])
|
| 209 |
+
assert rc == 0
|
| 210 |
+
assert not (tmp_path / "t.gif").exists()
|
| 211 |
+
```
|
| 212 |
+
|
| 213 |
+
- [ ] **Step 2: Run test to verify it fails**
|
| 214 |
+
|
| 215 |
+
Run: `.venv/bin/python -m pytest tests/cli/test_cli_gif.py -q`
|
| 216 |
+
Expected: FAIL (`unrecognized arguments: --no-gif` / no `t.gif`).
|
| 217 |
+
|
| 218 |
+
- [ ] **Step 3: Write minimal implementation**
|
| 219 |
+
|
| 220 |
+
In `proteus/cli.py`, add a helper (near `_resolve_memory`):
|
| 221 |
+
|
| 222 |
+
```python
|
| 223 |
+
def _auto_gif(out_path: str | None, no_gif: bool) -> None:
|
| 224 |
+
"""Render <out_stem>.gif next to the trace file unless suppressed."""
|
| 225 |
+
if not out_path or no_gif:
|
| 226 |
+
return
|
| 227 |
+
from pathlib import Path
|
| 228 |
+
|
| 229 |
+
from proteus.runtime import read_traces
|
| 230 |
+
from proteus.viz import reconstruct, write_gif
|
| 231 |
+
|
| 232 |
+
gif_path = Path(out_path).with_suffix(".gif")
|
| 233 |
+
for trace in read_traces(out_path):
|
| 234 |
+
write_gif(reconstruct(trace), gif_path)
|
| 235 |
+
print(f"gif written to {gif_path}")
|
| 236 |
+
break # one game per run/play -> one GIF
|
| 237 |
+
```
|
| 238 |
+
|
| 239 |
+
Call it at the end of `_cmd_run` (after `print(f"trace appended to {written}")`) and `_cmd_play` (after the `append_trace` branch):
|
| 240 |
+
|
| 241 |
+
```python
|
| 242 |
+
_auto_gif(args.out, args.no_gif)
|
| 243 |
+
return 0
|
| 244 |
+
```
|
| 245 |
+
|
| 246 |
+
Add `--no-gif` to the `run` and `play` subparsers in `build_parser()`:
|
| 247 |
+
|
| 248 |
+
```python
|
| 249 |
+
run.add_argument("--no-gif", action="store_true", dest="no_gif",
|
| 250 |
+
help="do not auto-render a GIF of the played game")
|
| 251 |
+
```
|
| 252 |
+
```python
|
| 253 |
+
play.add_argument("--no-gif", action="store_true", dest="no_gif",
|
| 254 |
+
help="do not auto-render a GIF of the played game")
|
| 255 |
+
```
|
| 256 |
+
|
| 257 |
+
> NOTE: `play`'s `--out` is optional; `_auto_gif` no-ops when `--out` is omitted (nothing to reconstruct from). Existing CLI tests that call `run`/`play` with `--out` will now also write a `.gif`; add `--no-gif` to any that assert on directory contents (none do today, but re-run the full suite at Step 4).
|
| 258 |
+
|
| 259 |
+
- [ ] **Step 4: Run test to verify it passes**
|
| 260 |
+
|
| 261 |
+
Run: `.venv/bin/python -m pytest tests/cli/test_cli_gif.py -q && .venv/bin/python -m pytest -q`
|
| 262 |
+
Expected: PASS (2 passed; full suite green — add `--no-gif` to any newly-failing CLI test).
|
| 263 |
+
|
| 264 |
+
- [ ] **Step 5: Commit**
|
| 265 |
+
|
| 266 |
+
```bash
|
| 267 |
+
git add proteus/cli.py tests/cli/test_cli_gif.py
|
| 268 |
+
git commit -m "feat(cp8): auto-GIF after run/play (default on, --no-gif to disable)"
|
| 269 |
+
```
|
| 270 |
+
|
| 271 |
+
---
|
| 272 |
+
|
| 273 |
+
## Stage 1 — Metric-only pass (TDD-ready, additive)
|
| 274 |
+
|
| 275 |
+
Spec §6.1–6.2, §8.1. Keep `focal_then_predator` order and same-cell capture; only **add** per-turn distance fields + episode metrics. The runner already captures pre-move positions; this adds post-move positions and pre/post BFS distance.
|
| 276 |
+
|
| 277 |
+
### Task 1.1: scenario distance helpers
|
| 278 |
+
|
| 279 |
+
**Files:**
|
| 280 |
+
- Modify: `proteus/grid/scenario.py`, `proteus/grid/scenarios/predator_evade.py`
|
| 281 |
+
- Test: `tests/grid/test_distance_helpers.py`
|
| 282 |
+
|
| 283 |
+
`safety_distance(game)` (BFS focal→predator at the current state) already exists and is the per-turn distance primitive. Add two concrete-default ABC methods:
|
| 284 |
+
|
| 285 |
+
- `max_bfs_distance(game) -> int | None` — the free-cell graph diameter (max finite BFS distance between any two free cells). Default `None`.
|
| 286 |
+
- `agent_distance_delta(game, focal_before, predator_before) -> float | None` — `dist(post_focal, predator_before) - dist(focal_before, predator_before)`, the spec's chase-corrected action quality. Default `None`.
|
| 287 |
+
|
| 288 |
+
- [ ] **Step 1: Write the failing test**
|
| 289 |
+
|
| 290 |
+
```python
|
| 291 |
+
# tests/grid/test_distance_helpers.py
|
| 292 |
+
import random
|
| 293 |
+
import proteus.grid # noqa: F401
|
| 294 |
+
from proteus.grid.difficulty import Difficulty
|
| 295 |
+
from proteus.grid.game import MotiveGridGame
|
| 296 |
+
from proteus.grid.scenario import Scenario, get_scenario
|
| 297 |
+
|
| 298 |
+
|
| 299 |
+
def _game():
|
| 300 |
+
s = get_scenario("predator_evade")()
|
| 301 |
+
g = MotiveGridGame(s, random.Random(42), Difficulty.EASY, max_steps=10)
|
| 302 |
+
return s, g
|
| 303 |
+
|
| 304 |
+
|
| 305 |
+
def test_base_defaults_none():
|
| 306 |
+
assert Scenario.max_bfs_distance is not None # method exists
|
| 307 |
+
# a bare scenario method default returns None (documented contract)
|
| 308 |
+
|
| 309 |
+
|
| 310 |
+
def test_max_bfs_distance_positive_on_easy():
|
| 311 |
+
s, g = _game()
|
| 312 |
+
d = s.max_bfs_distance(g)
|
| 313 |
+
assert isinstance(d, int) and d > 0
|
| 314 |
+
|
| 315 |
+
|
| 316 |
+
def test_agent_distance_delta_positive_when_moving_away():
|
| 317 |
+
s, g = _game()
|
| 318 |
+
focal = g.focal_sprite # EASY: (5,3)
|
| 319 |
+
pred = g.predator_sprite # EASY: (7,3)
|
| 320 |
+
before_focal = (focal.x, focal.y)
|
| 321 |
+
before_pred = (pred.x, pred.y)
|
| 322 |
+
# move focal up to (5,2) -> farther from (7,3)
|
| 323 |
+
g.apply_motive_action("up")
|
| 324 |
+
delta = s.agent_distance_delta(g, before_focal, before_pred)
|
| 325 |
+
assert delta is not None and delta > 0
|
| 326 |
+
```
|
| 327 |
+
|
| 328 |
+
- [ ] **Step 2: Run test to verify it fails**
|
| 329 |
+
|
| 330 |
+
Run: `.venv/bin/python -m pytest tests/grid/test_distance_helpers.py -q`
|
| 331 |
+
Expected: FAIL (`AttributeError: ... 'max_bfs_distance'`).
|
| 332 |
+
|
| 333 |
+
- [ ] **Step 3: Write minimal implementation**
|
| 334 |
+
|
| 335 |
+
In `proteus/grid/scenario.py`, alongside `safety_distance`:
|
| 336 |
+
|
| 337 |
+
```python
|
| 338 |
+
def max_bfs_distance(self, game: MotiveGridGame) -> int | None:
|
| 339 |
+
"""Free-cell graph diameter (max finite BFS distance), None by default."""
|
| 340 |
+
return None
|
| 341 |
+
|
| 342 |
+
def agent_distance_delta(
|
| 343 |
+
self, game, focal_before, predator_before
|
| 344 |
+
) -> float | None:
|
| 345 |
+
"""Chase-corrected action quality: dist(post_focal, predator_before) -
|
| 346 |
+
dist(focal_before, predator_before). None by default (non-survival)."""
|
| 347 |
+
return None
|
| 348 |
+
```
|
| 349 |
+
|
| 350 |
+
In `proteus/grid/scenarios/predator_evade.py`, implement both using the existing `_bfs_distance`/`_is_free`:
|
| 351 |
+
|
| 352 |
+
```python
|
| 353 |
+
def max_bfs_distance(self, game: MotiveGridGame) -> int | None:
|
| 354 |
+
cells = [
|
| 355 |
+
(x, y)
|
| 356 |
+
for x in range(self.grid_size[0])
|
| 357 |
+
for y in range(self.grid_size[1])
|
| 358 |
+
if self._is_free(game, (x, y))
|
| 359 |
+
]
|
| 360 |
+
best = 0
|
| 361 |
+
for i, a in enumerate(cells):
|
| 362 |
+
for b in cells[i + 1:]:
|
| 363 |
+
d = self._bfs_distance(game, a, b)
|
| 364 |
+
if d is not None and d > best:
|
| 365 |
+
best = d
|
| 366 |
+
return best
|
| 367 |
+
|
| 368 |
+
def agent_distance_delta(
|
| 369 |
+
self, game: MotiveGridGame, focal_before, predator_before
|
| 370 |
+
) -> float | None:
|
| 371 |
+
focal = game.focal_sprite
|
| 372 |
+
if focal is None:
|
| 373 |
+
return None
|
| 374 |
+
d_after = self._bfs_distance(game, (focal.x, focal.y), predator_before)
|
| 375 |
+
d_before = self._bfs_distance(game, focal_before, predator_before)
|
| 376 |
+
if d_after is None or d_before is None:
|
| 377 |
+
return None
|
| 378 |
+
return float(d_after - d_before)
|
| 379 |
+
```
|
| 380 |
+
|
| 381 |
+
- [ ] **Step 4: Run test to verify it passes**
|
| 382 |
+
|
| 383 |
+
Run: `.venv/bin/python -m pytest tests/grid/test_distance_helpers.py -q`
|
| 384 |
+
Expected: PASS (3 passed).
|
| 385 |
+
|
| 386 |
+
- [ ] **Step 5: Commit**
|
| 387 |
+
|
| 388 |
+
```bash
|
| 389 |
+
git add proteus/grid/scenario.py proteus/grid/scenarios/predator_evade.py tests/grid/test_distance_helpers.py
|
| 390 |
+
git commit -m "feat(cp8): scenario max_bfs_distance + agent_distance_delta helpers"
|
| 391 |
+
```
|
| 392 |
+
|
| 393 |
+
### Task 1.2: additive per-turn trace fields
|
| 394 |
+
|
| 395 |
+
**Files:**
|
| 396 |
+
- Modify: `proteus/runtime/trace.py`
|
| 397 |
+
- Test: `tests/runtime/test_trace.py` (extend)
|
| 398 |
+
|
| 399 |
+
Add to `TurnTrace` (all additive with safe defaults so existing constructors keep working):
|
| 400 |
+
|
| 401 |
+
```python
|
| 402 |
+
post_focal_pos: tuple[int, int] | None = None
|
| 403 |
+
post_predator_pos: tuple[int, int] | None = None
|
| 404 |
+
pre_bfs_distance: int | None = None
|
| 405 |
+
post_bfs_distance: int | None = None
|
| 406 |
+
agent_distance_delta: float | None = None
|
| 407 |
+
```
|
| 408 |
+
|
| 409 |
+
- [ ] **Step 1: Write the failing test** (append to `tests/runtime/test_trace.py`)
|
| 410 |
+
|
| 411 |
+
```python
|
| 412 |
+
def test_turntrace_distance_fields_default_none_and_round_trip():
|
| 413 |
+
from proteus.runtime.trace import TurnTrace
|
| 414 |
+
t = TurnTrace(
|
| 415 |
+
turn_idx=1, observation="o", action="up", motive_action="up",
|
| 416 |
+
habit_action="left", is_diagnostic=True, was_congruent=True,
|
| 417 |
+
reward=1.0, focal_pos=(3, 3), predator_pos=(5, 3),
|
| 418 |
+
)
|
| 419 |
+
assert t.post_focal_pos is None and t.pre_bfs_distance is None
|
| 420 |
+
t2 = TurnTrace.model_validate_json(t.model_dump_json())
|
| 421 |
+
assert t2.agent_distance_delta is None
|
| 422 |
+
```
|
| 423 |
+
|
| 424 |
+
- [ ] **Step 2: Run** `.venv/bin/python -m pytest tests/runtime/test_trace.py -q` → FAIL (unexpected kwarg / attr missing).
|
| 425 |
+
- [ ] **Step 3: Implement** the five fields above in `TurnTrace`, documenting them in the docstring.
|
| 426 |
+
- [ ] **Step 4: Run** the same test → PASS.
|
| 427 |
+
- [ ] **Step 5: Commit**
|
| 428 |
+
|
| 429 |
+
```bash
|
| 430 |
+
git add proteus/runtime/trace.py tests/runtime/test_trace.py
|
| 431 |
+
git commit -m "feat(cp8): additive per-turn distance/post-position trace fields"
|
| 432 |
+
```
|
| 433 |
+
|
| 434 |
+
### Task 1.3: record distances in the runner
|
| 435 |
+
|
| 436 |
+
**Files:**
|
| 437 |
+
- Modify: `proteus/runtime/session.py` (and mirror in `_session_core.py:make_turn_trace` for the web path)
|
| 438 |
+
- Test: `tests/runtime/test_session_distance.py`
|
| 439 |
+
|
| 440 |
+
In `SessionRunner.run()`'s play loop, capture `pre_bfs = self._scenario.safety_distance(self._game)` right where pre-move `focal_pos`/`predator_pos` are read; after `self._apply(...)`, capture `post_focal`/`post_predator` and `post_bfs = self._scenario.safety_distance(self._game)` and `delta = self._scenario.agent_distance_delta(self._game, focal_pos, predator_pos)`; pass them into the `TurnTrace(...)`.
|
| 441 |
+
|
| 442 |
+
- [ ] **Step 1: Write the failing test**
|
| 443 |
+
|
| 444 |
+
```python
|
| 445 |
+
# tests/runtime/test_session_distance.py
|
| 446 |
+
import proteus.grid # noqa: F401
|
| 447 |
+
from proteus.grid.difficulty import Difficulty
|
| 448 |
+
from proteus.providers import FakeProvider
|
| 449 |
+
from proteus.agents import VanillaAgent
|
| 450 |
+
from proteus.runtime import SessionRunner
|
| 451 |
+
|
| 452 |
+
|
| 453 |
+
def test_each_turn_records_pre_and_post_bfs_distance():
|
| 454 |
+
prov = FakeProvider(["ACTION: up"] * 10, model_name="demo")
|
| 455 |
+
trace = SessionRunner(
|
| 456 |
+
"predator_evade", VanillaAgent(prov), difficulty=Difficulty.EASY,
|
| 457 |
+
seed=42, play_turns=4, use_probe=False,
|
| 458 |
+
).run()
|
| 459 |
+
for t in trace.turns:
|
| 460 |
+
assert t.pre_bfs_distance is not None
|
| 461 |
+
assert t.post_bfs_distance is not None
|
| 462 |
+
assert t.post_focal_pos is not None
|
| 463 |
+
# spec §9: away-move quality is measured vs the PRE-move predator cell
|
| 464 |
+
first = trace.turns[0]
|
| 465 |
+
assert first.agent_distance_delta is not None
|
| 466 |
+
```
|
| 467 |
+
|
| 468 |
+
- [ ] **Step 2: Run** → FAIL (fields are None).
|
| 469 |
+
- [ ] **Step 3: Implement** the captures in `session.py` (and `_session_core.make_turn_trace`).
|
| 470 |
+
- [ ] **Step 4: Run** the test + full suite → PASS, 152 + new still green.
|
| 471 |
+
- [ ] **Step 5: Commit**
|
| 472 |
+
|
| 473 |
+
```bash
|
| 474 |
+
git add proteus/runtime/session.py proteus/runtime/_session_core.py tests/runtime/test_session_distance.py
|
| 475 |
+
git commit -m "feat(cp8): record pre/post BFS distance + post positions per turn"
|
| 476 |
+
```
|
| 477 |
+
|
| 478 |
+
### Task 1.4: additive episode metrics
|
| 479 |
+
|
| 480 |
+
**Files:**
|
| 481 |
+
- Modify: `proteus/runtime/metrics.py`
|
| 482 |
+
- Test: `tests/runtime/test_metrics.py` (extend)
|
| 483 |
+
|
| 484 |
+
Add four keys (existing keys untouched → 152 metric-dict assertions updated additively):
|
| 485 |
+
|
| 486 |
+
- `time_to_capture` = captured turn index, else `play_turns` (the horizon `H`).
|
| 487 |
+
- `distance_auc` = `mean(post_bfs_distance / max_bfs_distance)` over played turns (needs `max_bfs_distance`, passed in by the runner; `0.0` when unavailable).
|
| 488 |
+
- `min_distance` = `min(post_bfs_distance)` over played turns (`0.0` when none).
|
| 489 |
+
- `near_capture_count` = `count(post_bfs_distance <= 1)`.
|
| 490 |
+
|
| 491 |
+
`compute_metrics` gains a `max_bfs_distance: int | None = None` kwarg; the runner passes `self._scenario.max_bfs_distance(self._game)`. `away_move_fraction` is **kept as-is** (already pre-predator-based via `reward`, satisfying spec §9); document the equivalence in the metrics docstring.
|
| 492 |
+
|
| 493 |
+
- [ ] **Step 1: Write the failing test** (append to `tests/runtime/test_metrics.py`)
|
| 494 |
+
|
| 495 |
+
```python
|
| 496 |
+
def test_distance_metrics_present_and_sane():
|
| 497 |
+
from proteus.runtime.metrics import compute_metrics
|
| 498 |
+
from proteus.runtime.trace import TurnTrace
|
| 499 |
+
|
| 500 |
+
turns = [
|
| 501 |
+
TurnTrace(turn_idx=i, observation="", action="up", motive_action="up",
|
| 502 |
+
habit_action="left", is_diagnostic=True, was_congruent=True,
|
| 503 |
+
reward=1.0, focal_pos=(3, 3), predator_pos=(5, 3),
|
| 504 |
+
post_bfs_distance=d)
|
| 505 |
+
for i, d in enumerate([3, 2, 1], start=1)
|
| 506 |
+
]
|
| 507 |
+
m = compute_metrics(turns, played_turns=3, play_turns=5, outcome="eliminated",
|
| 508 |
+
max_bfs_distance=6)
|
| 509 |
+
assert m["time_to_capture"] == 3 # eliminated on the 3rd played turn... (see note)
|
| 510 |
+
assert m["min_distance"] == 1.0
|
| 511 |
+
assert m["near_capture_count"] == 1.0
|
| 512 |
+
assert 0.0 < m["distance_auc"] <= 1.0
|
| 513 |
+
```
|
| 514 |
+
|
| 515 |
+
> NOTE for implementer: `time_to_capture` needs the captured turn. Derive it from `outcome == "eliminated"` → `played_turns`; `survived` → `play_turns`. (A finer per-turn capture flag arrives in Pass 3.) Adjust the assertion to match the exact rule you implement, pinned here.
|
| 516 |
+
|
| 517 |
+
- [ ] **Step 2: Run** → FAIL (keys missing).
|
| 518 |
+
- [ ] **Step 3: Implement** the four keys + the `max_bfs_distance` kwarg; wire the runner to pass it.
|
| 519 |
+
- [ ] **Step 4: Run** `tests/runtime/test_metrics.py` + the golden/session/human-comparability tests; update their expected metric-key **sets** additively; full suite green.
|
| 520 |
+
- [ ] **Step 5: Commit**
|
| 521 |
+
|
| 522 |
+
```bash
|
| 523 |
+
git add proteus/runtime/metrics.py proteus/runtime/session.py tests/runtime/test_metrics.py tests/runtime/test_integration_golden.py tests/runtime/test_session.py
|
| 524 |
+
git commit -m "feat(cp8): time_to_capture/distance_auc/min_distance/near_capture_count metrics"
|
| 525 |
+
```
|
| 526 |
+
|
| 527 |
+
### Task 1.5: report `turn_order` / `capture_rule` on the episode
|
| 528 |
+
|
| 529 |
+
**Files:** `proteus/runtime/trace.py` (+ `session.py`), test in `tests/runtime/test_session_distance.py`.
|
| 530 |
+
|
| 531 |
+
Spec §4/§7: when keeping the current order, the report must state it. Add additive `SessionTrace` fields with defaults reflecting today's engine:
|
| 532 |
+
|
| 533 |
+
```python
|
| 534 |
+
turn_order: str = "focal_then_predator"
|
| 535 |
+
capture_rule: str = "same_cell"
|
| 536 |
+
horizon: int | None = None # = play_turns
|
| 537 |
+
```
|
| 538 |
+
|
| 539 |
+
- [ ] TDD: assert a fresh `SessionRunner(...).run()` trace has `turn_order == "focal_then_predator"`, `capture_rule == "same_cell"`, `horizon == play_turns`. Implement + wire. Commit `feat(cp8): record turn_order/capture_rule/horizon on the trace`.
|
| 540 |
+
|
| 541 |
+
**Stage 1 acceptance:** spec §9 bullets 1–3 (`survived` not sole judge; every turn has pre/post BFS; `away_move_fraction` pre-predator-based) satisfied; 152 baseline green; new metrics surfaced in `replay`/`compare`.
|
| 542 |
+
|
| 543 |
+
---
|
| 544 |
+
|
| 545 |
+
## Stage 2 — Persona reference pass (TDD-ready, additive)
|
| 546 |
+
|
| 547 |
+
Spec §3, §6.3. A **hidden** persona weight vector `w` defines a reward `R_w(s,a)`; a reference policy picks `argmax_a R_w`. Per turn we score the model's action against it. Nothing is shown to the model (the weights stay server-side); only additive trace/metric fields are produced.
|
| 548 |
+
|
| 549 |
+
### Task 2.1: persona weights + reward + reference policy
|
| 550 |
+
|
| 551 |
+
**Files:**
|
| 552 |
+
- Create: `proteus/runtime/persona.py`
|
| 553 |
+
- Test: `tests/runtime/test_persona.py`
|
| 554 |
+
|
| 555 |
+
Predator-only feature set (spec §3, §0 "predator-only"): the only strong feature is `risk_exposure`. Model the reward as:
|
| 556 |
+
|
| 557 |
+
```python
|
| 558 |
+
@dataclass(frozen=True)
|
| 559 |
+
class PersonaWeights:
|
| 560 |
+
persona_weight_id: str # public id (e.g. "risk_averse")
|
| 561 |
+
risk_cost: float # weight on closeness-to-predator
|
| 562 |
+
capture_penalty: float = 50.0
|
| 563 |
+
# resource_reward / norm_cost / social_weight arrive in Pass 5
|
| 564 |
+
|
| 565 |
+
def reward_rw(weights, scenario, game, focal_before, predator_before, action, blocked, captured) -> float:
|
| 566 |
+
# R_w(s,a) = -risk_cost * risk_exposure (- capture_penalty if captured)
|
| 567 |
+
# risk_exposure(s,a) = 1 / (1 + dist(post_focal, predator_before)) (closer => higher)
|
| 568 |
+
```
|
| 569 |
+
|
| 570 |
+
`reference_actions(weights, scenario, game) -> list[str]` returns the argmax-`R_w` action set (ties → all). `pressure(scenario, game) = 1 - clamp(pre_bfs/max_bfs, 0, 1)` (spec §6.3).
|
| 571 |
+
|
| 572 |
+
- [ ] **Step 1: Write the failing test**
|
| 573 |
+
|
| 574 |
+
```python
|
| 575 |
+
# tests/runtime/test_persona.py
|
| 576 |
+
import random
|
| 577 |
+
import proteus.grid # noqa: F401
|
| 578 |
+
from proteus.grid.difficulty import Difficulty
|
| 579 |
+
from proteus.grid.game import MotiveGridGame
|
| 580 |
+
from proteus.grid.scenario import get_scenario
|
| 581 |
+
from proteus.runtime.persona import PersonaWeights, reference_actions, pressure
|
| 582 |
+
|
| 583 |
+
|
| 584 |
+
def _sg():
|
| 585 |
+
s = get_scenario("predator_evade")()
|
| 586 |
+
g = MotiveGridGame(s, random.Random(42), Difficulty.EASY, max_steps=10)
|
| 587 |
+
return s, g
|
| 588 |
+
|
| 589 |
+
|
| 590 |
+
def test_risk_averse_reference_increases_distance():
|
| 591 |
+
s, g = _sg()
|
| 592 |
+
w = PersonaWeights(persona_weight_id="risk_averse", risk_cost=5.0)
|
| 593 |
+
acts = reference_actions(w, s, g)
|
| 594 |
+
# EASY handover analog: moving away (up/down) is preferred over into-wall left
|
| 595 |
+
assert "left" not in acts
|
| 596 |
+
assert acts # non-empty
|
| 597 |
+
|
| 598 |
+
|
| 599 |
+
def test_pressure_in_unit_range():
|
| 600 |
+
s, g = _sg()
|
| 601 |
+
p = pressure(s, g)
|
| 602 |
+
assert 0.0 <= p <= 1.0
|
| 603 |
+
```
|
| 604 |
+
|
| 605 |
+
- [ ] **Step 2: Run** → FAIL (module missing).
|
| 606 |
+
- [ ] **Step 3: Implement** `persona.py` with `PersonaWeights`, `reward_rw`, `reference_actions`, `pressure` exactly per the formulas above (reuse `scenario.agent_distance_delta`/`safety_distance`/`max_bfs_distance`). Register a small built-in persona table: `risk_averse` (risk_cost high), `risk_seeking` (risk_cost low), `survival_optimal` (capture_penalty high). Each has a `persona_weight_id`.
|
| 607 |
+
- [ ] **Step 4: Run** → PASS.
|
| 608 |
+
- [ ] **Step 5: Commit** `feat(cp8): hidden persona weights + R_w reference policy + pressure`.
|
| 609 |
+
|
| 610 |
+
### Task 2.2: persona fields on the trace + persona metrics
|
| 611 |
+
|
| 612 |
+
**Files:** `proteus/runtime/trace.py`, `proteus/runtime/metrics.py`, `proteus/runtime/session.py`; tests in `tests/runtime/test_persona.py`.
|
| 613 |
+
|
| 614 |
+
Additive per-turn fields (spec §7): `reference_actions: list[str] | None`, `reference_reward: float | None`, `model_reward: float | None`, `reward_regret: float | None`, `pressure: float | None`. Additive episode field `persona_weight_id: str | None`.
|
| 615 |
+
|
| 616 |
+
Persona metrics (spec §6.3) in `metrics.py`, computed only when persona fields are present (else omitted/`0.0` — additive, category-agnostic):
|
| 617 |
+
|
| 618 |
+
- `action_agreement` = `mean(a_model in reference_actions_t)`.
|
| 619 |
+
- `reward_regret` = `mean(reference_reward_t - model_reward_t)`.
|
| 620 |
+
- `pressure_weighted_agreement` = `sum(agreement_t * pressure_t) / sum(pressure_t)`.
|
| 621 |
+
- `persona_drift_turn` = first turn where the rolling-mean agreement (window 3) falls below 0.5, else `0.0`.
|
| 622 |
+
|
| 623 |
+
`SessionRunner` takes an optional `persona: PersonaWeights | None = None`; when set, each turn computes the reference action set / rewards / pressure against the **pre-move** state and fills the fields; `compute_metrics` adds the four persona keys.
|
| 624 |
+
|
| 625 |
+
- [ ] TDD: a runner with `persona=risk_averse` against a FakeProvider that always plays the reference action → `action_agreement == 100`, `reward_regret == 0`; against one that always plays into the wall → lower agreement, positive regret. Pin both. Update metric-key set assertions additively. Commit `feat(cp8): persona-maintenance metrics (agreement/regret/pressure-weighted/drift)`.
|
| 626 |
+
|
| 627 |
+
### Task 2.3: CLI `--persona`
|
| 628 |
+
|
| 629 |
+
**Files:** `proteus/cli.py`; test `tests/cli/test_cli_persona.py`.
|
| 630 |
+
|
| 631 |
+
- [ ] `proteus run --persona risk_averse ...` selects a built-in persona, runs the eval, records `persona_weight_id` on the trace (never the raw weights). Default: no persona → persona metrics absent. TDD + commit `feat(cp8): proteus run --persona <id>`.
|
| 632 |
+
|
| 633 |
+
**Stage 2 acceptance:** spec §9 bullets 4–5 (persona scored vs hidden reference policy; weights never in the model prompt) satisfied; the report can print survival / distance / persona as three separate blocks (spec §10).
|
| 634 |
+
|
| 635 |
+
---
|
| 636 |
+
|
| 637 |
+
## Stage 3 — Simultaneous turn resolver (DESIGN-GATED)
|
| 638 |
+
|
| 639 |
+
Spec §4, §5. **This changes engine behaviour** (`MotiveGridGame.apply_motive_action`) from `focal-move → advance_threat` to `plan → resolve` with crossing capture. It will change capture goldens and some metric values, so it must be its own session with a sub-spec.
|
| 640 |
+
|
| 641 |
+
**Run at stage start:** `superpowers:brainstorming` → a short sub-spec answering:
|
| 642 |
+
|
| 643 |
+
1. **Resolver contract.** Confirm the §4 pseudocode: both read the same `pre_state`, `predator_policy.plan(pre_state)` (chase toward `pre_focal`, not post), apply both, then capture = `same_cell or crossing` (§5). Blocked focal move → `stay`, predator continues its planned move.
|
| 644 |
+
2. **Backward-compat switch.** Keep `turn_order` selectable: default stays `focal_then_predator` (152 goldens unchanged) and `simultaneous` is opt-in via `--turn-order simultaneous`, OR flip the default and migrate goldens. Decide which; the spec leans "simultaneous recommended" but "don't shake CP6 goldens" — a flag preserves both.
|
| 645 |
+
3. **Golden migration.** Which of `test_difficulty_layouts`, `test_predator_evade_behavior`, `test_step_reward`, `test_integration_golden` change under simultaneous order, and what the new pinned values are.
|
| 646 |
+
4. **Per-turn capture flags.** Add `same_cell_capture` / `crossing_capture` / `captured` to `TurnTrace` (spec §7) and refine `time_to_capture` to use the real captured turn.
|
| 647 |
+
|
| 648 |
+
**Acceptance (spec §9 bullets 6–7):** under `--turn-order simultaneous`, participant and predator plan from the same pre-state; capture detects same-cell AND crossing; goldens for the chosen default stay green; the other order remains available and tested.
|
| 649 |
+
|
| 650 |
+
**Do not author Stage 3 tasks from this plan** — they depend on the resolver sub-spec's decisions.
|
| 651 |
+
|
| 652 |
+
---
|
| 653 |
+
|
| 654 |
+
## Stage 4 — Hidden-weight memory generation (TDD-ready, builds on CP7)
|
| 655 |
+
|
| 656 |
+
Spec §3, §8.4. Today `generate_memory` (CP7) drives the self-play with the scenario's transparent brief. Pass 4 lets a **hidden persona weight** drive the reference policy that produces the memory, so the memory embodies a persona the eval model must infer — without ever seeing the weights.
|
| 657 |
+
|
| 658 |
+
### Task 4.1: persona-driven memory generation
|
| 659 |
+
|
| 660 |
+
**Files:** `proteus/runtime/memory_gen.py` (extend), `proteus/runtime/memory.py` (add `persona_weight_id` field), test `tests/runtime/test_memory_persona.py`.
|
| 661 |
+
|
| 662 |
+
Add `generate_memory(..., persona: PersonaWeights | None = None)`. When `persona` is set, the memory actions come from `reference_actions(persona, scenario, game)` (deterministic reference policy) instead of the agent — i.e. the memory is a *persona demonstration*, not a model self-play. Record `persona_weight_id` on the `MemoryCheckpoint` (public id only; raw weights never serialized into the participant-visible checkpoint). The transparent brief stays generic (no weight leakage).
|
| 663 |
+
|
| 664 |
+
- [ ] **Step 1: Write the failing test**
|
| 665 |
+
|
| 666 |
+
```python
|
| 667 |
+
# tests/runtime/test_memory_persona.py
|
| 668 |
+
from proteus.grid.difficulty import Difficulty
|
| 669 |
+
from proteus.runtime.memory_gen import generate_memory
|
| 670 |
+
from proteus.runtime.persona import PersonaWeights
|
| 671 |
+
|
| 672 |
+
|
| 673 |
+
def test_persona_memory_is_deterministic_and_tags_persona():
|
| 674 |
+
w = PersonaWeights(persona_weight_id="risk_averse", risk_cost=5.0)
|
| 675 |
+
ck = generate_memory(
|
| 676 |
+
"predator_evade", agent=None, difficulty=Difficulty.EASY, seed=42,
|
| 677 |
+
memory_turns=5, model_name="ref", clock=lambda: "FIXED", persona=w,
|
| 678 |
+
)
|
| 679 |
+
assert ck.persona_weight_id == "risk_averse"
|
| 680 |
+
assert 1 <= len(ck.memory_turns) <= 5
|
| 681 |
+
# risk-averse demo never walks into the dead-end wall on the diagnostic turn
|
| 682 |
+
assert ck.memory_turns[0].action in ("up", "down", "right", "stay")
|
| 683 |
+
# weights are NOT in the participant-visible checkpoint text
|
| 684 |
+
assert "risk_cost" not in ck.model_dump_json()
|
| 685 |
+
```
|
| 686 |
+
|
| 687 |
+
- [ ] **Step 2: Run** → FAIL (`persona` kwarg / field missing).
|
| 688 |
+
- [ ] **Step 3: Implement**: branch in `generate_memory` — `persona` set → action = first of `reference_actions(...)` (deterministic tie-break), `agent` may be `None`; add `persona_weight_id: str | None = None` to `MemoryCheckpoint`.
|
| 689 |
+
- [ ] **Step 4: Run** + full suite → PASS.
|
| 690 |
+
- [ ] **Step 5: Commit** `feat(cp8): persona-driven memory generation (hidden weights, public id)`.
|
| 691 |
+
|
| 692 |
+
### Task 4.2: CLI wiring
|
| 693 |
+
|
| 694 |
+
**Files:** `proteus/cli.py`; test extends `tests/cli/test_cli_persona.py`.
|
| 695 |
+
|
| 696 |
+
- [ ] `proteus memory --persona risk_averse ...` generates a persona demonstration; `proteus run --memory latest --persona risk_averse` scores persona maintenance against the same hidden weights. TDD + commit `feat(cp8): CLI persona memory generation + scored run`.
|
| 697 |
+
|
| 698 |
+
**Stage 4 acceptance:** the eval model receives only the memory (persona demonstration) + the new predator state; persona metrics (Stage 2) score whether it *continues that persona*; the public `persona_weight_id` links memory ↔ reference policy for analysis; raw weights never reach the participant.
|
| 699 |
+
|
| 700 |
+
---
|
| 701 |
+
|
| 702 |
+
## Stage 5 — Multi-feature personas (DESIGN-GATED — needs its own spec)
|
| 703 |
+
|
| 704 |
+
Spec §3 (multi-motive table), §8.5. Greed / rule-compliance / cooperation personas require **new scenario features** the current `predator_evade` lacks: resource cells (`resource_gain`), forbidden zones (`norm_violation`), other agents (`social_benefit`). The spec explicitly warns **not to over-interpret** greed/compliance/cooperation in a predator-only world (§9 bullet 8).
|
| 705 |
+
|
| 706 |
+
**This stage is a new sub-project**, overlapping the previously-deferred "new motive category" idea. Run `superpowers:brainstorming` → a dedicated sub-spec deciding:
|
| 707 |
+
|
| 708 |
+
1. Which feature(s) to add first (resource is the simplest; norms/social are heavier) — YAGNI: likely one.
|
| 709 |
+
2. A new `Scenario` (or a `predator_evade` variant) exposing the feature, its hand-authored difficulty layouts, and its diagnostic invariant golden (mirror CP6/CP7 patterns).
|
| 710 |
+
3. Extending `PersonaWeights` + `R_w` with the new feature terms (`resource_reward`, `norm_cost`, `social_weight` already stubbed in Pass 2) and the matching reference-policy logic.
|
| 711 |
+
4. New per-turn features on the trace (resource position, norm flags, social distances).
|
| 712 |
+
|
| 713 |
+
**Acceptance:** a non-survival persona (e.g. greedy) is distinguishable from a risk-averse one — same survival, *different* persona scores (spec §1's worked example: high survival score, low persona-maintenance score is achievable and detected).
|
| 714 |
+
|
| 715 |
+
**Do not author Stage 5 tasks from this plan** — they depend on the multi-feature sub-spec.
|
| 716 |
+
|
| 717 |
+
---
|
| 718 |
+
|
| 719 |
+
## Self-Review
|
| 720 |
+
|
| 721 |
+
**Spec coverage:**
|
| 722 |
+
- §6.1 survival (`time_to_capture`, `survival_fraction` exists) → Task 1.4. ✅
|
| 723 |
+
- §6.2 distance (`pre/post_bfs`, `distance_auc`, `min_distance`, `near_capture_count`, `away_move_fraction` pre-predator) → Tasks 1.1–1.4. ✅
|
| 724 |
+
- §6.3 persona (`action_agreement`, `reward_regret`, `pressure_weighted_agreement`, `persona_drift_turn`, `pressure`) → Stage 2. ✅
|
| 725 |
+
- §4 turn order (flag, recorded) → Task 1.5 (record) + Stage 3 (resolver, gated). ✅
|
| 726 |
+
- §5 capture (same-cell now; crossing in Stage 3) → Task 1.5 + Stage 3. ✅
|
| 727 |
+
- §7 trace fields (post positions, distances, capture flags, persona fields, episode `turn_order`/`capture_rule`/`horizon`/`persona_weight_id`/`memory_ref`) → Tasks 1.2/1.5, Stage 2, Stage 3 (flags). ✅
|
| 728 |
+
- §8 five passes → Stages 1–5. ✅ §10 default report set → printed by Stages 1–2 metrics (a `compare`/`replay` report-format tweak is a follow-on, additive).
|
| 729 |
+
- Auto-GIF (user request) → Stage 0. ✅
|
| 730 |
+
|
| 731 |
+
**Placeholder scan:** TDD-ready stages (0, 1, 2, 4) contain full code/commands. Stages 3 and 5 are **explicitly design-gated** (not placeholders — they carry decisions + acceptance and instruct a brainstorming+sub-spec at stage start, because the parent spec leaves the engine-resolver and multi-feature scenario open by design). This is intentional, not a gap.
|
| 732 |
+
|
| 733 |
+
**Type/name consistency:** `safety_distance`/`max_bfs_distance`/`agent_distance_delta` (scenario), `PersonaWeights`/`reference_actions`/`reward_rw`/`pressure` (persona), `compute_metrics(..., max_bfs_distance=)`, additive `TurnTrace`/`SessionTrace`/`MemoryCheckpoint` fields are used consistently across stages. `away_move_fraction` is **kept** (not redefined) — documented as already pre-predator-based, satisfying §9 without value churn.
|
| 734 |
+
|
| 735 |
+
**Regression guard:** Stages 0–2, 4 are additive (new files, optional fields/kwargs with defaults, opt-in CLI flags) → the 152 baseline stays green; only Stage 3 (gated) migrates goldens, deliberately isolated. Re-verify the full suite at the end of every task.
|
| 736 |
+
```
|