File size: 3,136 Bytes
eafbe80 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | # Eval v2
This folder keeps the public replay, in-domain loop/revisit, and open-domain revisit evaluations.
## Static Consistency
Run multi-chunk loop/revisit evaluation:
```bash
cd /data/work/run_codes/Echo-Memory/
source /tmp/uv-venv/bin/activate
export WAN_BASE_MODEL=/local-ssd/echo-memory/Wan2.1-T2V-1.3B/
export DATASET_BASE_PATH=/threed-code/yorenchen/data/echo-memory/Context-as-Memory-Dataset/
export CKPT=/local-ssd/echo-memory/ckpts/spatial_mem/epoch-0.safetensors
bash eval/v2/run_static_consistency_loop_and_revisit.sh
```
Useful optional variables:
- `EVALS_ROOT`: output directory, defaults to `${CKPT_DIR}/evals_v2/static_consistency`.
- `NUM_SAMPLES_LOOP`: number of loop-closure samples.
- `NUM_SAMPLES_STATIC`: number of in-domain revisit samples.
- `MULTIVIEW_FIRSTFRAME_LIST`: text/jsonl/csv list for open-domain first frames.
- `MULTIVIEW_FIRSTFRAME_DIR`: image directory used to generate the list automatically, for example `assets/opendomain_revisit`.
- `RUN_GEOMETRY_DIAG=1`: enable optional geometry diagnostics.
The script writes in-domain outputs under `${EVALS_ROOT}/in_domain` and open-domain outputs under `${EVALS_ROOT}/open_domain`.
## Basic Capability
Run a single-video GT trajectory replay check:
```bash
export WAN_BASE_MODEL=/path/to/Wan2.1-T2V-1.3B
export DATASET_BASE_PATH=/path/to/Context-as-Memory-Dataset
export CKPT=/path/to/epoch-0.safetensors
bash eval/v2/run_basic_replay_gt.sh
```
Outputs include generated videos and `replay_gt_metrics.json` with MSE, PSNR, SSIM, and optional LPIPS.
## Open-Domain Revisit + VLM
For the paper-style open-domain return probe, use the one-click revisit suite:
```bash
export WAN_BASE_MODEL=/path/to/Wan2.1-T2V-1.3B
export DATASET_BASE_PATH=/path/to/Context-as-Memory-Dataset
PHASE=stage1 OOD_DIR=assets/opendomain_revisit \
bash eval/v2/revisit_suite/run_one_click_revisit_eval.sh
```
Set `PHASE=vlm` with `EVAL_ROOT=/path/to/eval_root` to score an existing stage-1 run. The scorer expects an OpenAI-compatible endpoint via `VLM_API_BASE` and `VLM_MODEL`; use `VLM_DRY_RUN=1` for a wiring check.
## Visual Outputs
Every evaluation path keeps human-readable artifacts:
- GT replay writes `replay_gt_gen_only.mp4`, `replay_gt_metrics.json`, and optional per-frame CSV files.
- Static loop/revisit writes generated MP4 files under `in_domain/loop_closure` and `in_domain/combo_revisit_in_domain`.
- Open-domain multiview writes MP4 files under `open_domain/multiview_revisit`.
- The one-click revisit suite writes `stage1_frames/first_00.png`, `stage1_frames/revisit_tail_*.png`, optional first-vs-last chunk change maps, and `revisit_gen_only.mp4` for each case.
For paper figures, start from the one-click revisit suite outputs:
```bash
python eval/v2/revisit_suite/export_revisit_materials.py \
--eval-root eval_outputs/revisit_suite_<timestamp> \
--out-dir paper_case_materials \
--prefix echo_memory_revisit
```
The exported material directory contains flattened case metadata plus image references that can be used to build qualitative grids.
Dynamic/private evaluation code is intentionally not included in this release.
|