memory_selection/HOW_TO_READ.md

# How to read the memory-selection heatmaps

Each PNG is one rollout episode.

- **X axis** = policy call (time →). One call ≈ 16 env-steps (the model acts on a 16-step chunk,
  then re-plans). The first few calls are the demo-watch/priming phase.
- **Y axis** = the 8 memory slots of the log-spaced window, labelled by how far back they look:
  `current` (0), `-16`, `-32`, `-64`, `-128`, `-256`, `-512`, `-1024` env-steps.
  Early in an episode the deep slots don't exist yet, so they clamp to the oldest available frame.
- **Color** = the gate weight on that slot at that moment. **Black = discarded (~0), bright =
  kept.** This is literally "how much the model trusts that past observation right now."

## What to look for — the core finding

**Permanence (`VideoUnmask_episode_5_success`) — recall WORKS.**
The `current` row stays warm (always anchor on now), and as the episode proceeds the gate
progressively **lights up specific deeper slots**, reaching all the way back to `-1024` (the
demo / pre-occlusion region) by late episode. The task is "remember which container hides the
green cube" — and you can see the gate reach back to the observation from *before* the cube was
covered. This is the mechanism behind Permanence ≈ HAMLET (20.0 vs 19.5).

**Counting (`BinFill_episode_*`) — integration FAILS.**
Compare the same layout: the gate is almost entirely stuck on the `current` row, with only
faint, sparse deep activation. Counting ("put N cubes in the bin, then stop") needs the model to
**retain every placement event and add them up** — but the gate's job is *selective* recall, so
it throws most of the history away. The result is below even no-memory (5.0 vs vanilla 14): the
answer is partly visible in the current frame, and the sparse gated memory adds noise instead of
a reliable count.

**Imitation (`MoveCube_episode_*`).**
The demo is ~15 snapshots (n_demo≈246 env-steps). At exec time the log-window can only sample a
few of them, front-loaded — so the fine demo trajectory is under-represented. The gate primes and
attends, but the window resolution over a long demo is the bottleneck (5.0 vs HAMLET 14).

## One-line summary
Selective gating **helps "recall one past thing" and hurts "integrate all past things."**
That trade-off is the whole story of this checkpoint, and it is visible directly in these images.