Data scripts (static in-domain pool)

Scripts for the static in-domain pool. Complete download and layout verification first — see doc/dataset_preprocessing.md.

Both training pools share the Echo-Memory layout (frames/, jsons/, overlap_labels/, metadata_full.csv). Set DATASET_BASE_PATH to the pool root before running these scripts.

Metadata CSV

The Echo-Team packaged static pool already includes metadata_full.csv. If you downloaded the upstream pool without metadata, fetch the released metadata:

export DATASET_BASE_PATH=data/Context-as-Memory-Dataset
huggingface-cli download Echo-Team/Echo-Memory-Data metadata_full.csv \
  --repo-type dataset \
  --local-dir "${DATASET_BASE_PATH}"

Or regenerate it locally after changing the pool:

bash scripts/run_generate_metadata.sh

For a smaller custom metadata file, set an output path and row limit:

OUTPUT_CSV="${DATASET_BASE_PATH}/metadata_1000.csv" \
METADATA_MAX_ROWS=1000 \
bash scripts/run_generate_metadata.sh

Use that file with --dataset_metadata_path "${DATASET_BASE_PATH}/metadata_1000.csv".

Optional variables:

OUTPUT_CSV: output CSV path, defaults to ${DATASET_BASE_PATH}/metadata_full.csv.
SEGMENT_LENGTH: frames per segment, default 81.
CONTEXT_FRAMES: context frames in metadata construction, default 5.
METADATA_MAX_ROWS / DATASET_SIZE_ROWS: optional row cap for custom-size metadata; 0 keeps the full CSV.

Latent precompute

export WAN_BASE_MODEL=/path/to/Wan2.1-T2V-1.3B
export DATASET_BASE_PATH=data/Context-as-Memory-Dataset
NUM_PROCESSES=8 bash scripts/run_precompute_ctx_target_latents.sh

Optional variables:

MODEL_PATHS: JSON list of model weight paths.
CONTEXT_FRAMES: number of context frames, default 20.
NUM_PROCESSES: accelerate processes, default 1.

Open-domain revisit first-frame assets are already included under assets/opendomain_revisit; no dataset construction step is needed for those probes.