File size: 6,575 Bytes
eafbe80 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 | # Static in-domain pool — download & preprocessing
Echo-Memory’s **static in-domain pool** is released through [Echo-Team/Echo-Memory-Data](https://huggingface.co/datasets/Echo-Team/Echo-Memory-Data) as tar parts under `static_pool_tar_parts/`. The underlying pool is sourced from [KlingTeam/Context-as-Memory-Dataset](https://huggingface.co/datasets/KlingTeam/Context-as-Memory-Dataset) on Hugging Face (Kling Team, SIGGRAPH Asia 2025; [arXiv:2506.03141](https://arxiv.org/abs/2506.03141)). Total size is about **340 GB** — plan disk space before downloading and unpacking.
---
## 1. Download
### Option A — Echo-Team packaged release
```bash
pip install -U "huggingface_hub[cli]"
mkdir -p data
huggingface-cli download Echo-Team/Echo-Memory-Data \
--repo-type dataset \
--include "static_pool_tar_parts/*" \
--local-dir ./data/echo-memory-data-release
cat ./data/echo-memory-data-release/static_pool_tar_parts/echo-memory-data.tar.part-* | tar -xf - -C ./data
```
You should end up with `data/Context-as-Memory-Dataset/`.
### Option B — original KlingTeam source
If you prefer the upstream release, download or merge the original parts from the [KlingTeam dataset card](https://huggingface.co/datasets/KlingTeam/Context-as-Memory-Dataset):
```bash
mkdir -p data
cd data
# after all Context-as-Memory-Dataset_* parts are downloaded into this directory:
cat Context-as-Memory-Dataset_* > Context-as-Memory-Dataset.zip
unzip Context-as-Memory-Dataset.zip -d .
```
You should end up with a directory named `Context-as-Memory-Dataset/` (adjust the path below if your folder name differs).
---
## 2. Expected layout (static in-domain pool)
After extraction, point `DATASET_BASE_PATH` at the pool root (default: `data/Context-as-Memory-Dataset/`):
```text
data/Context-as-Memory-Dataset/
├── frames/ # 100 scene folders, ~7601 PNGs each
│ ├── AncientTempleEnv_0/
│ │ ├── 0000.png
│ │ └── ...
│ └── ...
├── jsons/ # per-scene camera pose JSON (one file per scene)
│ ├── AncientTempleEnv_0.json
│ └── ...
├── overlap_labels/ # per-frame overlap indices (used by context retrieval / latent precompute)
│ ├── AncientTempleEnv_0/
│ │ ├── 0.json
│ │ └── ...
│ └── ...
├── captions.txt # segment captions (optional for some workflows)
└── metadata_full.csv # released Echo-Memory segment metadata
```
Quick sanity check:
```bash
export DATASET_BASE_PATH=data/Context-as-Memory-Dataset
test -d "${DATASET_BASE_PATH}/frames" && echo "frames OK"
test -d "${DATASET_BASE_PATH}/jsons" && echo "jsons OK"
test -d "${DATASET_BASE_PATH}/overlap_labels" && echo "overlap_labels OK"
ls "${DATASET_BASE_PATH}/frames" | head
ls "${DATASET_BASE_PATH}/jsons" | head
```
---
## 3. Point Echo-Memory at the static in-domain pool
```bash
export DATASET_BASE_PATH=data/Context-as-Memory-Dataset
export WAN_BASE_MODEL=/path/to/Wan2.1-T2V-1.3B
export PYTHONPATH=$PWD:${PYTHONPATH:-}
```
Training scripts also accept `data/Context-as-Memory-Dataset` under the repo root if `DATASET_BASE_PATH` is unset.
---
## 4. Metadata (required)
`metadata_full.csv` is included in the Echo-Team packaged release. If you downloaded the upstream KlingTeam source instead, fetch the released metadata into the pool root:
```bash
cd /path/to/Echo-Memory
export DATASET_BASE_PATH=data/Context-as-Memory-Dataset
huggingface-cli download Echo-Team/Echo-Memory-Data metadata_full.csv \
--repo-type dataset \
--local-dir "${DATASET_BASE_PATH}"
```
If you modify the pool or need to rebuild metadata locally, regenerate it from `frames/` and `captions.txt`:
```bash
bash scripts/run_generate_metadata.sh
```
You can also generate a smaller custom index for ablations or reduced-size training:
```bash
OUTPUT_CSV="${DATASET_BASE_PATH}/metadata_1000.csv" \
METADATA_MAX_ROWS=1000 \
bash scripts/run_generate_metadata.sh
```
Pass the custom CSV to training/evaluation with `--dataset_metadata_path "${DATASET_BASE_PATH}/metadata_1000.csv"`.
Defaults (override via env vars):
| Variable | Default | Meaning |
| --- | --- | --- |
| `OUTPUT_CSV` | `${DATASET_BASE_PATH}/metadata_full.csv` | Output metadata path |
| `SEGMENT_LENGTH` | `81` | Frames per training segment |
| `CONTEXT_FRAMES` | `5` | Context window used when building metadata |
| `NUM_WORKERS` | CPU count − 2 | Parallel workers |
| `METADATA_MAX_ROWS` / `DATASET_SIZE_ROWS` | `0` | Keep only the first N metadata rows after generation; `0` keeps the full CSV |
Verify:
```bash
wc -l "${DATASET_BASE_PATH}/metadata_full.csv"
head -n 3 "${DATASET_BASE_PATH}/metadata_full.csv"
```
---
## 5. Precompute latents (optional, speeds training)
If you train with precomputed VAE latents:
```bash
export WAN_BASE_MODEL=/path/to/Wan2.1-T2V-1.3B
export DATASET_BASE_PATH=data/Context-as-Memory-Dataset
NUM_PROCESSES=8 bash scripts/run_precompute_ctx_target_latents.sh
```
Latents are written under `${DATASET_BASE_PATH}/latents/`. The script can use `overlap_labels/` when `--use_overlap_labels` is enabled (see `scripts/run_precompute_ctx_target_latents.sh`).
---
## 6. Training pools vs. open-domain assets
| Echo pool / asset | Location | Purpose |
| --- | --- | --- |
| Static in-domain pool | `DATASET_BASE_PATH` → `data/Context-as-Memory-Dataset` | Training, in-domain replay/revisit, metadata |
| Dynamic training pool | `DATASET_BASE_PATH` → `data/dynamic-memory-dataset` | Training on the dynamic pool ([guide](dynamic_dataset_preprocessing.md)) |
| Open-domain first frames | `assets/opendomain_revisit/` | Held-out OOD revisit probes (already in repo) |
You do **not** need to rebuild open-domain anchors for the released revisit suite.
---
## 7. Troubleshooting
**`DATASET_BASE_PATH is not set`** — export the variable or place data at `data/Context-as-Memory-Dataset` relative to the repo root.
**Missing `frames/` or `jsons/`** — re-check unzip path; the root folder name must match what you pass to `DATASET_BASE_PATH`.
**Metadata script missing** — ensure you are on the latest Echo-Memory `main` branch; metadata generation is invoked via `scripts/run_generate_metadata.sh`.
**Disk space** — keep ~340 GB for raw frames plus extra space for `metadata_full.csv`, `latents/`, and training outputs.
---
## Reference
- Static in-domain pool: [dataset_preprocessing.md](dataset_preprocessing.md)
- Dynamic training pool: [dynamic_dataset_preprocessing.md](dynamic_dataset_preprocessing.md)
|