README.md

# Temporal-VLA — HAMLET-AAM on GR00T N1.6 / RoboMME

**AAM = Action-aware Adaptive Memory.** A memory module on top of the known-good HAMLET
window architecture (GR00T-N1.6 backbone). It keeps a **log-spaced 8-frame window** of the
past (`[-1024,-512,-256,-128,-64,-32,-16,0]` env-steps) and a **learned gate** that assigns a
soft weight to each of the 8 memory slots every policy call — "how much to trust each past
observation right now" — before they enter the memory transformer that conditions the action
head. A second head predicts how the current observation follows from the (weighted) memory
plus the last action chunk (action-conditioned forward prediction).

This bucket is everything you need to **observe, debug, and trace** the model at checkpoint-60000.

---

## TL;DR result (this checkpoint, our eval harness, 10 eps/task)

| Suite | AAM-60k | vanilla N1.6 | HAMLET | read |
|---|---|---|---|---|
| Counting | **5.0** | 14 | 17.5 | ✗ gate can't integrate a running count |
| Permanence | **20.0** | 9 | 19.5 | ✓ **matches HAMLET** — memory recall works |
| Reference | **12.5** | 11 | 15 | ~ between vanilla and HAMLET |
| Imitation | **5.0** | 6.5 | 14 | ✗ log-window samples the long demo too sparsely |
| **Total** | **10.6** | 10.12 | 16.5 | ≈ vanilla overall; a real, interpretable trade-off |

The headline finding: **the adaptive gate is genuinely good at selective recall (Permanence,
where it ties HAMLET and doubles vanilla) but bad at temporal integration (Counting, where it
falls below even no-memory).** See `memory_selection/HOW_TO_READ.md` and the `showcase/` clips.

> A severe inference bug was found and fixed first (memory history was reset every policy call,
> collapsing the 8-frame window to 8 copies of the current frame). All results here are AFTER the
> fix. See `THE_FIX.md`.

---

## How to navigate / trace

- **`showcase/`** — start here. 5 curated episodes, each folder pairs:
  - `rollout.mp4` — what the robot actually did (front + wrist cams; the red-bordered frames at
    the start are the demo the model "watched")
  - `memory_heatmap.png` — what the memory gate selected, aligned to the same episode
  - `gate_weights_full_episode.jsonl` — raw per-call gate weights for that task (trace it yourself)
  Folders: `01_PERMANENCE_success`, `02_PERMANENCE_fail`, `03_COUNTING_success`,
  `04_COUNTING_fail`, `05_IMITATION_fail`.

- **`memory_selection/`** — all 13 gate heatmaps (VideoUnmask, BinFill, MoveCube episodes) +
  `HOW_TO_READ.md` explaining the visualization and the Permanence-vs-Counting contrast.

- **`all_rollouts/<Suite>/<Task>/`** — every one of the 160 rollout videos from the full 16-task
  eval, named `..._episode_N-{success|failure}.mp4`. Browse any task's successes/failures.

- **`results_csv/`** — the raw `simulation_results.csv` for all 16 tasks (col 3 = success bool).

- **`raw_memdumps/`** — the full per-call gate-weight logs (JSONL) for the 3 instrumented tasks.
  Each line: `{reset, t_hist, steps_back[8], weight[8]}`. `reset:true` marks an episode start.

- **`THE_FIX.md`** — the inference bug and the one-line-ish fix, for reproducibility.

## Reproduce a heatmap yourself

```
python scripts/viz_aam_memory.py <task_video_dir> <task_memdump.jsonl> <out_dir>
```
(script included as `viz_aam_memory.py`). Set `AAM_MEMDUMP=/path/file.jsonl` when running the
eval to regenerate the raw gate logs.