Static in-domain pool — download & preprocessing

Echo-Memory’s static in-domain pool is released through Echo-Team/Echo-Memory-Data as tar parts under static_pool_tar_parts/. The underlying pool is sourced from KlingTeam/Context-as-Memory-Dataset on Hugging Face (Kling Team, SIGGRAPH Asia 2025; arXiv:2506.03141). Total size is about 340 GB — plan disk space before downloading and unpacking.

1. Download

Option A — Echo-Team packaged release

pip install -U "huggingface_hub[cli]"

mkdir -p data

huggingface-cli download Echo-Team/Echo-Memory-Data \
  --repo-type dataset \
  --include "static_pool_tar_parts/*" \
  --local-dir ./data/echo-memory-data-release

cat ./data/echo-memory-data-release/static_pool_tar_parts/echo-memory-data.tar.part-* | tar -xf - -C ./data

You should end up with data/Context-as-Memory-Dataset/.

Option B — original KlingTeam source

If you prefer the upstream release, download or merge the original parts from the KlingTeam dataset card:

mkdir -p data
cd data

# after all Context-as-Memory-Dataset_* parts are downloaded into this directory:
cat Context-as-Memory-Dataset_* > Context-as-Memory-Dataset.zip
unzip Context-as-Memory-Dataset.zip -d .

You should end up with a directory named Context-as-Memory-Dataset/ (adjust the path below if your folder name differs).

2. Expected layout (static in-domain pool)

After extraction, point DATASET_BASE_PATH at the pool root (default: data/Context-as-Memory-Dataset/):

data/Context-as-Memory-Dataset/
├── frames/              # 100 scene folders, ~7601 PNGs each
│   ├── AncientTempleEnv_0/
│   │   ├── 0000.png
│   │   └── ...
│   └── ...
├── jsons/               # per-scene camera pose JSON (one file per scene)
│   ├── AncientTempleEnv_0.json
│   └── ...
├── overlap_labels/      # per-frame overlap indices (used by context retrieval / latent precompute)
│   ├── AncientTempleEnv_0/
│   │   ├── 0.json
│   │   └── ...
│   └── ...
├── captions.txt         # segment captions (optional for some workflows)
└── metadata_full.csv    # released Echo-Memory segment metadata

Quick sanity check:

export DATASET_BASE_PATH=data/Context-as-Memory-Dataset

test -d "${DATASET_BASE_PATH}/frames" && echo "frames OK"
test -d "${DATASET_BASE_PATH}/jsons" && echo "jsons OK"
test -d "${DATASET_BASE_PATH}/overlap_labels" && echo "overlap_labels OK"
ls "${DATASET_BASE_PATH}/frames" | head
ls "${DATASET_BASE_PATH}/jsons" | head

3. Point Echo-Memory at the static in-domain pool

export DATASET_BASE_PATH=data/Context-as-Memory-Dataset
export WAN_BASE_MODEL=/path/to/Wan2.1-T2V-1.3B
export PYTHONPATH=$PWD:${PYTHONPATH:-}

Training scripts also accept data/Context-as-Memory-Dataset under the repo root if DATASET_BASE_PATH is unset.

4. Metadata (required)

metadata_full.csv is included in the Echo-Team packaged release. If you downloaded the upstream KlingTeam source instead, fetch the released metadata into the pool root:

cd /path/to/Echo-Memory
export DATASET_BASE_PATH=data/Context-as-Memory-Dataset

huggingface-cli download Echo-Team/Echo-Memory-Data metadata_full.csv \
  --repo-type dataset \
  --local-dir "${DATASET_BASE_PATH}"

If you modify the pool or need to rebuild metadata locally, regenerate it from frames/ and captions.txt:

bash scripts/run_generate_metadata.sh

You can also generate a smaller custom index for ablations or reduced-size training:

OUTPUT_CSV="${DATASET_BASE_PATH}/metadata_1000.csv" \
METADATA_MAX_ROWS=1000 \
bash scripts/run_generate_metadata.sh

Pass the custom CSV to training/evaluation with --dataset_metadata_path "${DATASET_BASE_PATH}/metadata_1000.csv".

Defaults (override via env vars):

Variable	Default	Meaning
`OUTPUT_CSV`	`${DATASET_BASE_PATH}/metadata_full.csv`	Output metadata path
`SEGMENT_LENGTH`	`81`	Frames per training segment
`CONTEXT_FRAMES`	`5`	Context window used when building metadata
`NUM_WORKERS`	CPU count − 2	Parallel workers
`METADATA_MAX_ROWS` / `DATASET_SIZE_ROWS`	`0`	Keep only the first N metadata rows after generation; `0` keeps the full CSV

Verify:

wc -l "${DATASET_BASE_PATH}/metadata_full.csv"
head -n 3 "${DATASET_BASE_PATH}/metadata_full.csv"

5. Precompute latents (optional, speeds training)

If you train with precomputed VAE latents:

export WAN_BASE_MODEL=/path/to/Wan2.1-T2V-1.3B
export DATASET_BASE_PATH=data/Context-as-Memory-Dataset
NUM_PROCESSES=8 bash scripts/run_precompute_ctx_target_latents.sh

Latents are written under ${DATASET_BASE_PATH}/latents/. The script can use overlap_labels/ when --use_overlap_labels is enabled (see scripts/run_precompute_ctx_target_latents.sh).

6. Training pools vs. open-domain assets

Echo pool / asset	Location	Purpose
Static in-domain pool	`DATASET_BASE_PATH` → `data/Context-as-Memory-Dataset`	Training, in-domain replay/revisit, metadata
Dynamic training pool	`DATASET_BASE_PATH` → `data/dynamic-memory-dataset`	Training on the dynamic pool (guide)
Open-domain first frames	`assets/opendomain_revisit/`	Held-out OOD revisit probes (already in repo)

You do not need to rebuild open-domain anchors for the released revisit suite.

7. Troubleshooting

DATASET_BASE_PATH is not set — export the variable or place data at data/Context-as-Memory-Dataset relative to the repo root.

Missing frames/ or jsons/ — re-check unzip path; the root folder name must match what you pass to DATASET_BASE_PATH.

Metadata script missing — ensure you are on the latest Echo-Memory main branch; metadata generation is invoked via scripts/run_generate_metadata.sh.

Disk space — keep ~340 GB for raw frames plus extra space for metadata_full.csv, latents/, and training outputs.

Reference

Static in-domain pool: dataset_preprocessing.md
Dynamic training pool: dynamic_dataset_preprocessing.md