Spaces:
Runtime error
Runtime error
Commit ·
8431307
1
Parent(s): 51ee471
docs(cp7): HANDOFF — CP7 memory pre-roll complete + Ollama smoke; Next=CP8
Browse files152 passed offline; Ollama acceptance (gpt-oss:120b-cloud) survived with a
6-turn self-played memory shown at the handover, memory_ref recorded, .venv
SDK-free. Curiosity/new-category renumbered CP8.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- HANDOFF.md +59 -10
HANDOFF.md
CHANGED
|
@@ -4,8 +4,9 @@
|
|
| 4 |
> `docs/superpowers/specs/2026-06-01-proteus-arena-slice-design.md` (immutable design) +
|
| 5 |
> the per-CP plan in `docs/superpowers/plans/` + this file.
|
| 6 |
|
| 7 |
-
**Last updated:** 2026-06-02 —
|
| 8 |
-
|
|
|
|
| 9 |
|
| 10 |
## Done (CP0–CP3)
|
| 11 |
|
|
@@ -145,6 +146,11 @@ Executed task-by-task via subagent-driven-development (3-gate: implement → spe
|
|
| 145 |
(`run --out`) traces, then `.venv/bin/python -m proteus compare h.jsonl llm.jsonl --out summary.json`
|
| 146 |
→ per-`(model, difficulty)` metric means + n (the `model` token is the part AFTER the colon, e.g.
|
| 147 |
`fake:demo` → `demo`).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 148 |
- Real-model smoke: see the throwaway-venv recipe above (needs a provider SDK + key; never in `.venv`).
|
| 149 |
|
| 150 |
## Current state
|
|
@@ -164,16 +170,59 @@ human-baseline aggregation CLI. Still a single scenario (`predator_evade`, survi
|
|
| 164 |
`step_reward`/`safety_distance` ABC surface is now **open** for CP7 to plug new motive categories in
|
| 165 |
without touching `SessionRunner`.
|
| 166 |
|
| 167 |
-
##
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 173 |
scenario + its category reward + its own diagnostic invariant golden, reusing the difficulty-layout and
|
| 174 |
rollout machinery. The `VanillaAgent._ACTION_DIRECTIVE` / `SessionRunner._PROBE_QUESTION` predator-framing
|
| 175 |
-
(carried below) should be sourced from the `Scenario` when the second scenario lands
|
| 176 |
-
|
|
|
|
|
|
|
| 177 |
|
| 178 |
## Deferred items (carry forward)
|
| 179 |
|
|
|
|
| 4 |
> `docs/superpowers/specs/2026-06-01-proteus-arena-slice-design.md` (immutable design) +
|
| 5 |
> the per-CP plan in `docs/superpowers/plans/` + this file.
|
| 6 |
|
| 7 |
+
**Last updated:** 2026-06-02 — CP7 complete (LLM-generated memory pre-roll; hybrid handover; offline
|
| 8 |
+
suite 152 passed + Ollama acceptance smoke passed). CP6 before it (difficulty layouts + category eval +
|
| 9 |
+
`compare`).
|
| 10 |
|
| 11 |
## Done (CP0–CP3)
|
| 12 |
|
|
|
|
| 146 |
(`run --out`) traces, then `.venv/bin/python -m proteus compare h.jsonl llm.jsonl --out summary.json`
|
| 147 |
→ per-`(model, difficulty)` metric means + n (the `model` token is the part AFTER the colon, e.g.
|
| 148 |
`fake:demo` → `demo`).
|
| 149 |
+
- Memory pre-roll (CP7): generate a checkpoint then reuse it —
|
| 150 |
+
`.venv/bin/python -m proteus memory --scenario predator_evade --model fake:demo --seed 42 --memory-turns 8`
|
| 151 |
+
writes `runs/memory/<model>/<stamp>.json`; `proteus run … --memory latest` (or `--memory generate` to do
|
| 152 |
+
both in one shot, `--memory <path>` for a specific file, default `none`) injects it at the handover.
|
| 153 |
+
`runs/memory/` is gitignored. Korean usage guide: `docs/USAGE-ko.md`.
|
| 154 |
- Real-model smoke: see the throwaway-venv recipe above (needs a provider SDK + key; never in `.venv`).
|
| 155 |
|
| 156 |
## Current state
|
|
|
|
| 170 |
`step_reward`/`safety_distance` ABC surface is now **open** for CP7 to plug new motive categories in
|
| 171 |
without touching `SessionRunner`.
|
| 172 |
|
| 173 |
+
## Done (CP7 — LLM-generated memory pre-roll; design: `docs/superpowers/specs/2026-06-02-proteus-cp7-memory-preroll-design.md`; plan: `docs/superpowers/plans/2026-06-02-proteus-cp7-memory-preroll.md`)
|
| 174 |
+
|
| 175 |
+
Executed inline (executing-plans, TDD, commit-per-task). **Full suite: 152 passed** (127 baseline + 25
|
| 176 |
+
CP7), offline/headless; no provider SDK added to `.venv`. The original "CP7 = new motive category" idea
|
| 177 |
+
was **renamed to CP8** when the user pivoted to this memory feature; curiosity/sociality stay deferred.
|
| 178 |
+
|
| 179 |
+
- **Stage 1 — `runtime/memory.py`** (pure pydantic + stdlib, no intra-runtime imports, like `trace.py`):
|
| 180 |
+
`MemoryTurn`/`MemoryCheckpoint` models; `save_checkpoint`/`load_checkpoint` (single-file JSON at
|
| 181 |
+
`runs/memory/<safe(model)>/<stamp>.json`); `latest_for_model` (lexical stamp = chronological);
|
| 182 |
+
`render_memory_block` (pure observation renderer). Exported from `runtime/__init__`.
|
| 183 |
+
- **Stage 2 — `runtime/memory_gen.py`** (engine-coupled, like `rollout.py`): `generate_memory(scenario,
|
| 184 |
+
agent, *, difficulty, seed, memory_turns, model_name, clock)` runs a **full-info self-play** episode
|
| 185 |
+
(`max_steps=memory_turns`, no scripted Cut, no answer-key leakage), capturing frames/actions/reasoning
|
| 186 |
+
→ `MemoryCheckpoint`. Deterministic under an injected `clock`. New scenario-sourced `Scenario.memory_brief`
|
| 187 |
+
(concrete attr default `""`); `predator_evade` sets a transparent brief (discloses the BFS chase). This
|
| 188 |
+
resolves the **memory-prompt** half of the old "source prompts from Scenario" item; the
|
| 189 |
+
`_PROBE_QUESTION`/`_ACTION_DIRECTIVE` predator-framing is still hard-coded (carried to CP8).
|
| 190 |
+
- **Stage 3 — hybrid handover** (`trace.py` + `session.py`): additive `SessionTrace.memory_ref: str | None`;
|
| 191 |
+
`SessionRunner(..., memory=None, memory_ref=None)` prepends the memory block (+ `NOW — this run so far:`)
|
| 192 |
+
to the **turn-1** observation only, **before** the unchanged scripted Cut. `step_reward`, answer keys,
|
| 193 |
+
metrics, and `_session_core.py` untouched → diagnostic invariant + 127 preserved (pinned by
|
| 194 |
+
`test_session_memory.py`: with/without memory the per-turn `motive_action`/`habit_action`/`is_diagnostic`
|
| 195 |
+
and `metrics` are identical; only the turn-1 observation grows).
|
| 196 |
+
- **Stage 4 — CLI** (`cli.py`): `proteus memory` (generate+save a checkpoint) and `proteus run --memory
|
| 197 |
+
MODE` where `MODE ∈ {none(default), generate, latest, <path>}` via a `_resolve_memory` helper. `--memory-turns`
|
| 198 |
+
(default 10) and `--memory-root` (default `runs/memory`) on both. Exit codes mirror `run` (unknown
|
| 199 |
+
model/scenario → 2; `latest` with no checkpoint → 2).
|
| 200 |
+
|
| 201 |
+
### CP7 acceptance smoke (passed)
|
| 202 |
+
|
| 203 |
+
- **Offline end-to-end:** `proteus memory … fake:demo` → `proteus run … --memory latest` → the turn-1
|
| 204 |
+
observation carries the `MEMORY` block + `NOW` separator + scripted `Cut 1/2:` in order; `memory_ref` set.
|
| 205 |
+
- **Ollama (networked, manual):** `ollama:gpt-oss:120b-cloud`, seed 42 / easy. `proteus memory
|
| 206 |
+
--memory-turns 6` → the **real model self-played 6 turns and survived** (vs the fake `stay` agent which
|
| 207 |
+
is eliminated in 2), checkpoint written to `runs/memory/gpt-oss_120b-cloud/<stamp>.json`. Then `run
|
| 208 |
+
--play-turns 8 --memory latest` → **survived | motive_reading_accuracy=50% | reactivity_index=40%**;
|
| 209 |
+
turn-1 observation = 962 chars containing the 6-turn memory block + the scripted Cut;
|
| 210 |
+
`memory_ref="gpt-oss:120b-cloud@<stamp>"`. `.venv` stayed SDK-free (throwaway `/tmp/proteus-smoke-venv`
|
| 211 |
+
with `httpx`; key = first whitespace token of the `.env` value, CP4 gotcha). `runs/` is gitignored, so
|
| 212 |
+
the real trace/checkpoint/key were never committed.
|
| 213 |
+
|
| 214 |
+
## Next: CP8 (new motive categories / scenarios) — needs its own plan
|
| 215 |
+
|
| 216 |
+
Per spec §12 (the work formerly numbered CP7). CP6 opened the `step_reward`/`safety_distance` ABC interface
|
| 217 |
+
so a new motive **category** (e.g. curiosity = reward for visiting novel cells; sociality) can be added as a
|
| 218 |
+
new `Scenario` subclass with its own per-turn reward — **without touching `SessionRunner`** (reward is
|
| 219 |
+
scenario-owned) or the metrics/compare harness (additive, category-agnostic). Likely scope: a second
|
| 220 |
scenario + its category reward + its own diagnostic invariant golden, reusing the difficulty-layout and
|
| 221 |
rollout machinery. The `VanillaAgent._ACTION_DIRECTIVE` / `SessionRunner._PROBE_QUESTION` predator-framing
|
| 222 |
+
(carried below) should be sourced from the `Scenario` when the second scenario lands (CP7 already added
|
| 223 |
+
`Scenario.memory_brief` as the pattern to follow). The curiosity brainstorming explored two diagnostic
|
| 224 |
+
geometries (A: blocked-dead-end analog with novelty reward; B: explored-pocket) — pick one in CP8. LLM-as-judge
|
| 225 |
+
scoring of reasoning and the web UI / leaderboard remain deferred (spec §12). Brainstorm → write the CP8 plan → execute.
|
| 226 |
|
| 227 |
## Deferred items (carry forward)
|
| 228 |
|