code/doc/dynamic_dataset_preprocessing.md

Dynamic training pool — SpatialVID subset

Echo-Memory’s dynamic training pool uses a motion-filtered subset of SpatialVID/SpatialVID: ego-centric clips with camera poses and captions, exported into the same sample format used by the static pool.

This guide covers download → export → training/inference settings only. Dynamic eval is TODO; current public support is training and inference.

License: SpatialVID is CC-BY-NC-SA 4.0 (non-commercial). Static and dynamic pools may have different licenses — check before mixing runs.

---

1. Download (subset)

Hugging Face: SpatialVID/SpatialVID

• Accept the dataset terms on Hugging Face before download.
• Full corpus is large (~7 TB+). For Echo-Memory dynamic training, download selected groups only — you do not need the full 545 groups.

pip install -U "huggingface_hub[cli]"
huggingface-cli login

export SPATIALVID_ROOT=/path/to/SpatialVID
hf download SpatialVID/SpatialVID --repo-type dataset --local-dir "${SPATIALVID_ROOT}"

To fetch specific groups, use include patterns or the helper script linked from the dataset card (download_SpatialVID.py on the SpatialVID GitHub).

Extract downloaded .tar.gz groups:

cd "${SPATIALVID_ROOT}"
tar -xzvf annotations/group_0001.tar.gz
tar -xzvf videos/group_0001.tar.gz

Raw layout (per clip)

SPATIALVID_ROOT/
├── annotations/group_0001/{clip_id}/
│   ├── poses.npy          # (N, 7) = tx,ty,tz,qx,qy,qz,qw
│   ├── indexes.txt        # pose index → source frame index
│   ├── caption.json       # scene / motion text
│   └── dyn_masks.npz      # optional dynamic-region masks
├── videos/group_0001/{clip_id}.mp4
└── data/train/SpatialVID_metadata.csv

Use SpatialVID_metadata.csv to filter clips (e.g. motion score, dynamicRatio, sceneType) when building your subset.

---

2. Export to Echo layout (dynamic training pool)

Use data/dynamic-spatialvid-motion60/mixed/ as the public training root and set DATASET_BASE_PATH to it:

data/dynamic-spatialvid-motion60/
├── L1/                                    # single-level exports are also valid roots
├── L2/
├── L3/
└── mixed/
    ├── frames/L{1,2,3}/{clip_id}/0000.png ... 0080.png
    ├── jsons/L{1,2,3}/{clip_id}.json
    ├── overlap_labels/L{1,2,3}/{clip_id}/
    ├── captions.txt
    ├── metadata_train.csv
    ├── metadata_train_sample.csv
    ├── metadata_train_sample_L1.csv
    ├── metadata_eval.csv
    └── metadata_eval_2chunk.csv

Per-clip steps:

Step	Setting
Frame sample	81 PNGs per clip, 640×352
Pose	Interpolate poses.npy + indexes.txt → jsons/{clip_id}.json (Euler CineCameraActor format, same as static data)
Prompt	Short caption from caption.json (SceneSummary or SceneDescription)
Overlap	Build overlap_labels/ for FOV-based context retrieval
Metadata row	video, prompt, video_name, start_frame, end_frame, optional level

metadata_train.csv is written at export time. Use metadata_train_sample.csv or metadata_train_sample_L1.csv for local step checks. Do not re-run run_generate_metadata.sh unless you regenerate from raw frames only.

---

3. Training settings

Same env vars and on-disk layout as the static in-domain pool — only DATASET_BASE_PATH changes.

export WAN_BASE_MODEL=/path/to/Wan2.1-T2V-1.3B
export DATASET_BASE_PATH=data/dynamic-spatialvid-motion60/mixed
export PYTHONPATH=$PWD:${PYTHONPATH:-}

Recommended settings for the dynamic training pool (match memory baseline scripts):

Parameter	Typical value
Resolution	640 × 352
Frames / chunk	81
Context frames	1–20 (recipe-dependent)
--use_rt_relative	on
--enable_fov_retrieval	on (when overlap_labels/ present)
--enable_context_memory	on for context / spatial / SSM rows
--timestep_shift	15
Learning rate	5e-5 (adjust per row)

Example — run a dynamic row:

METADATA_NAME=metadata_train.csv bash train/dynamic_spatialvid/run_dyn_spatial_mem.sh

For local one-step validation:

METADATA_NAME=metadata_train_sample_L1.csv \
MAX_TRAIN_STEPS=1 \
PROGRESS_TOTAL_STEPS=30000 \
NUM_WORKERS=0 \
bash train/dynamic_spatialvid/run_dyn_block_wise_ssm.sh

Inference wrappers live under inference/dynamic_spatialvid/.

---

4. Demo selection

Dynamic demos are selected from training-scene replay rather than from fixed eval scripts:

1. Randomly sample candidate scenes from metadata_train.csv or metadata_train_sample.csv.
2. Use the same prompt, first frame, and GT action trajectory for all six dynamic rows.
3. Run inference/unified_inference.py or inference/dynamic_spatialvid/*.sh for each checkpoint.
4. Manually pick a representative scene where all rows are viewable.

The checked-in README previews are compressed GIFs under assets/readme_previews/.

---

5. Checklist

• Hugging Face access approved for SpatialVID/SpatialVID
• Subset of group_**** archives downloaded and extracted
• Clips filtered (poses + caption present; optional motion / dynamic filters)
• frames/, jsons/, metadata_train.csv under one root
• (Recommended) overlap_labels/ for FOV retrieval
• DATASET_BASE_PATH exported before training/inference

---

Reference

• SpatialVID: SpatialVID/SpatialVID · arXiv:2509.09676
• Static in-domain pool: dataset_preprocessing.md
• Dynamic training pool: dynamic_dataset_preprocessing.md