# Data scripts (static in-domain pool)

Scripts for the **static in-domain pool**. Complete download and layout verification first — see **[doc/dataset_preprocessing.md](../doc/dataset_preprocessing.md)**.

Both training pools share the Echo-Memory layout (`frames/`, `jsons/`, `overlap_labels/`, `metadata_full.csv`). Set `DATASET_BASE_PATH` to the pool root before running these scripts.

## Metadata CSV

The Echo-Team packaged static pool already includes `metadata_full.csv`. If you downloaded the upstream pool without metadata, fetch the released metadata:

```bash
export DATASET_BASE_PATH=data/Context-as-Memory-Dataset
huggingface-cli download Echo-Team/Echo-Memory-Data metadata_full.csv \
  --repo-type dataset \
  --local-dir "${DATASET_BASE_PATH}"
```

Or regenerate it locally after changing the pool:

```bash
bash scripts/run_generate_metadata.sh
```

For a smaller custom metadata file, set an output path and row limit:

```bash
OUTPUT_CSV="${DATASET_BASE_PATH}/metadata_1000.csv" \
METADATA_MAX_ROWS=1000 \
bash scripts/run_generate_metadata.sh
```

Use that file with `--dataset_metadata_path "${DATASET_BASE_PATH}/metadata_1000.csv"`.

Optional variables:

- `OUTPUT_CSV`: output CSV path, defaults to `${DATASET_BASE_PATH}/metadata_full.csv`.
- `SEGMENT_LENGTH`: frames per segment, default `81`.
- `CONTEXT_FRAMES`: context frames in metadata construction, default `5`.
- `METADATA_MAX_ROWS` / `DATASET_SIZE_ROWS`: optional row cap for custom-size metadata; `0` keeps the full CSV.

## Latent precompute

```bash
export WAN_BASE_MODEL=/path/to/Wan2.1-T2V-1.3B
export DATASET_BASE_PATH=data/Context-as-Memory-Dataset
NUM_PROCESSES=8 bash scripts/run_precompute_ctx_target_latents.sh
```

Optional variables:

- `MODEL_PATHS`: JSON list of model weight paths.
- `CONTEXT_FRAMES`: number of context frames, default `20`.
- `NUM_PROCESSES`: accelerate processes, default `1`.

Open-domain revisit first-frame assets are already included under `assets/opendomain_revisit`; no dataset construction step is needed for those probes.