code/doc/memory_mechanisms.md

# Memory Mechanisms

This note maps the paper's memory rows to the repository implementation and explains the modeling role of each family. Echo-Memory treats memory as a controlled intervention on what information from chunk 1 is stored and how chunk 2 reads it back during denoising.

## Modeling View

All rows use the same action-conditioned Wan DiT backbone and the same two-chunk training/evaluation setup:

1. **Context chunk:** clean history frames are encoded into latent/context tokens, optionally with matched camera RT actions.
2. **Target chunk:** noisy target latents are denoised while the selected memory mechanism exposes information from the context chunk.
3. **Read-out:** memory is injected through raw context concatenation, compressed context tokens, spatial memory tokens, or recurrent state-space modules attached to DiT blocks.

The ablations are designed to change only the memory pathway while keeping the backbone, action conditioning, resolution, chunk length, and training schedule aligned.

## Paper Rows

| Paper family | Paper row / repo name | What is stored or read | Main code path | Training entry |
| --- | --- | --- | --- | --- |
| Raw context | `context_k1`, `context_k5`, `context_k20` | Uncompressed retrieved context frames. `K=1` is the anchor/I2V floor; `K=5/20` are context-learning capacity rows. | `diffsynth/pipelines/wan_video_new.py` context latent path | `train/context_learning/run_pre_qkv_ctx{1,5,20}.sh` |
| Compression | `framepack_weight` | Context tokens are kept at the same length but temporally reweighted. | `diffsynth/models/memory/framepack_weight.py` | `train/memory_baselines_basic/run_ablation_framepack_weight_two_chunk.sh` |
| Compression | `framepack_len_r2`, `framepack_len_r4` | Context latents and matched RT actions are pooled along time. | `diffsynth/models/memory/framepack_length.py` | `train/memory_baselines_basic/run_ablation_framepack_len_r{2,4}_two_chunk.sh` |
| Compression | `framepack_hybrid_r2`, `framepack_hybrid_r4` | Length compression plus token reweighting. | `wan_video_new.py` + FramePack helpers | `train/memory_baselines_basic/run_ablation_framepack_hybrid_r*_weight_two_chunk.sh` |
| Token-grid | `spatial_mem` | Context tokens are time-averaged and summarized into learned grid tokens. This is the implementation behind the currently reported `spatial_mem` row; it does **not** reconstruct depth or 3D geometry. | `diffsynth/models/memory/spatial_grid_memory.py` | `train/memory_baselines_basic/run_spatial_memory_baseline.sh` |
| Token-grid | `spatial_inject_none`, `spatial_concat_text`, `spatial_cross_attn_readout` | Same token-grid storage, different read-out: withheld, text-KV concat, or dedicated cross-attention. | `spatial_grid_memory.py` read-out helpers | matching `run_ablation_spatial_*_two_chunk.sh` scripts |
| Geometry-grounded spatial | `geometry_spatial_mem` | A static scene is reconstructed outside the DiT using depth, intrinsics, extrinsics, and TSDF fusion. The fused point cloud is rendered along the target trajectory, VAE-encoded, and converted into conditioning tokens. | `diffsynth/models/memory/geometry_spatial_memory.py` | `train/memory_baselines_basic/run_geometry_spatial_memory_baseline.sh` |
| State-space | `block_wise_ssm` | Paper-aligned recurrent state attached to selected DiT blocks. Checkpoint keys contain `block_wise_ssm.*`. | `diffsynth/models/memory/block_wise_ssm.py` | `train/memory_baselines_basic/run_ablation_block_wise_ssm_two_chunk.sh` |
| State-space | `videossm_hybrid` | Legacy VideoSSM hybrid baseline: depthwise temporal-conv state-space-like module. Checkpoint keys contain `videossm_hybrid.*`. | `diffsynth/models/memory/videossm_hybrid.py` | `train/memory_baselines_basic/run_videossm_hybrid_baseline.sh` |

## Naming Rules

- Do not describe `SpatialGridMemory` or the existing `spatial_mem` results as the
  geometry-grounded method from arXiv:2506.05284. It is a token-grid baseline.
- Use **Geometry-grounded Spatial Memory** only when the metadata supplies
  rendered static geometry through `geometry_memory` (or a configured column).
  The geometry extractor is the external reconstruction pipeline: depth and
  cameras → TSDF-fused static point cloud → target-view renders. The model-side
  encoder does not estimate depth itself.
- Use **Block-wise SSM** only for `--use_block_wise_ssm` / `BlockWiseStateSpaceMemory`.
- Use **VideoSSM hybrid** only for the legacy `--use_videossm_hybrid` / `HybridStateSpaceMemory` baseline.
- Use **Context learning** for raw-context capacity rows (`K=1/5/20`), not for compact memory modules.
- Keep checkpoint folder names stable; `env/memory_baseline_runtime.py` and `inference/unified_inference.py` infer memory profiles from those names.