File size: 2,051 Bytes
c335050 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | # Data scripts (static in-domain pool)
Scripts for the **static in-domain pool**. Complete download and layout verification first — see **[doc/dataset_preprocessing.md](../doc/dataset_preprocessing.md)**.
Both training pools share the Echo-Memory layout (`frames/`, `jsons/`, `overlap_labels/`, `metadata_full.csv`). Set `DATASET_BASE_PATH` to the pool root before running these scripts.
## Metadata CSV
The Echo-Team packaged static pool already includes `metadata_full.csv`. If you downloaded the upstream pool without metadata, fetch the released metadata:
```bash
export DATASET_BASE_PATH=data/Context-as-Memory-Dataset
huggingface-cli download Echo-Team/Echo-Memory-Data metadata_full.csv \
--repo-type dataset \
--local-dir "${DATASET_BASE_PATH}"
```
Or regenerate it locally after changing the pool:
```bash
bash scripts/run_generate_metadata.sh
```
For a smaller custom metadata file, set an output path and row limit:
```bash
OUTPUT_CSV="${DATASET_BASE_PATH}/metadata_1000.csv" \
METADATA_MAX_ROWS=1000 \
bash scripts/run_generate_metadata.sh
```
Use that file with `--dataset_metadata_path "${DATASET_BASE_PATH}/metadata_1000.csv"`.
Optional variables:
- `OUTPUT_CSV`: output CSV path, defaults to `${DATASET_BASE_PATH}/metadata_full.csv`.
- `SEGMENT_LENGTH`: frames per segment, default `81`.
- `CONTEXT_FRAMES`: context frames in metadata construction, default `5`.
- `METADATA_MAX_ROWS` / `DATASET_SIZE_ROWS`: optional row cap for custom-size metadata; `0` keeps the full CSV.
## Latent precompute
```bash
export WAN_BASE_MODEL=/path/to/Wan2.1-T2V-1.3B
export DATASET_BASE_PATH=data/Context-as-Memory-Dataset
NUM_PROCESSES=8 bash scripts/run_precompute_ctx_target_latents.sh
```
Optional variables:
- `MODEL_PATHS`: JSON list of model weight paths.
- `CONTEXT_FRAMES`: number of context frames, default `20`.
- `NUM_PROCESSES`: accelerate processes, default `1`.
Open-domain revisit first-frame assets are already included under `assets/opendomain_revisit`; no dataset construction step is needed for those probes.
|