memory_selection/HOW_TO_READ.md

How to read the memory-selection heatmaps

Each PNG is one rollout episode.

• X axis = policy call (time →). One call ≈ 16 env-steps (the model acts on a 16-step chunk, then re-plans). The first few calls are the demo-watch/priming phase.
• Y axis = the 8 memory slots of the log-spaced window, labelled by how far back they look: current (0), -16, -32, -64, -128, -256, -512, -1024 env-steps. Early in an episode the deep slots don't exist yet, so they clamp to the oldest available frame.
• Color = the gate weight on that slot at that moment. Black = discarded (~0), bright = kept. This is literally "how much the model trusts that past observation right now."

What to look for — the core finding

Permanence (VideoUnmask_episode_5_success) — recall WORKS. The current row stays warm (always anchor on now), and as the episode proceeds the gate progressively lights up specific deeper slots, reaching all the way back to -1024 (the demo / pre-occlusion region) by late episode. The task is "remember which container hides the green cube" — and you can see the gate reach back to the observation from before the cube was covered. This is the mechanism behind Permanence ≈ HAMLET (20.0 vs 19.5).

Counting (BinFill_episode_*) — integration FAILS. Compare the same layout: the gate is almost entirely stuck on the current row, with only faint, sparse deep activation. Counting ("put N cubes in the bin, then stop") needs the model to retain every placement event and add them up — but the gate's job is selective recall, so it throws most of the history away. The result is below even no-memory (5.0 vs vanilla 14): the answer is partly visible in the current frame, and the sparse gated memory adds noise instead of a reliable count.

Imitation (MoveCube_episode_*). The demo is ~15 snapshots (n_demo≈246 env-steps). At exec time the log-window can only sample a few of them, front-loaded — so the fine demo trajectory is under-represented. The gate primes and attends, but the window resolution over a long demo is the bottleneck (5.0 vs HAMLET 14).

One-line summary

Selective gating helps "recall one past thing" and hurts "integrate all past things." That trade-off is the whole story of this checkpoint, and it is visible directly in these images.