Memory Eval Metrics

This folder contains two evaluation layers:

Numeric post-processing for evals_v2 or other generated-video folders.
Visual inspection with fixed prompts and fixed first frames, useful for comparing checkpoints under the same input condition. See VISUAL_EVAL_DESIGN.md and visual_eval_config.yaml.

Visual Inspection

This route is intentionally human-readable. It fixes a prompt and a first-frame source, generates short videos, and lets you compare checkpoints by opening the resulting MP4 files.

Config: visual_eval_config.yaml defines prompt sets and first-frame presets.
Design note: VISUAL_EVAL_DESIGN.md explains recommended case groups and output layout.

Run:

python3 eval/metrics/run_visual_eval.py --ckpt /path/to/epoch-0.safetensors --output_root /path/to/ckpt_dir/evals_visual

Outputs are written under evals_visual/prompt_<id>_first_<id>/; each case folder contains 2-chunk or 4-chunk MP4 files.

Numeric Metrics

Usage

export EVALS_ROOT=/path/to/ckpt_dir/evals_v2/static_consistency

# Run all six dimensions.
python eval/metrics/run_all_metrics.py --evals_root "$EVALS_ROOT"

# Run specific dimensions.
python eval/metrics/run_all_metrics.py --evals_root "$EVALS_ROOT" --dims 1 2 5

# Optional: dataset for loop-closure trajectory reference; CLIP for identity.
python eval/metrics/run_all_metrics.py --evals_root "$EVALS_ROOT" --dataset /path/to/Context-as-Memory-Dataset --use_clip --write_csv

Results are written to evals_root/metrics/ by default (or --output_dir): per-dimension *.json and all_metrics_summary.json. Use --write_csv to also write aggregate_summary.csv.

Dimensions

Dim	Name	Metrics (Phase 1)	Optional
1	Long-Horizon Consistency	Stable sequence length, frame-to-frame drift rate	User Study: see below
2	Loop Closure / Revisit	View Recall PSNR, View Recall SSIM	Trajectory ref error (when dataset provided)
3	Identity Preservation	CLIP consistency (or simple embedding)	Face Embedding, character ID (insightface/torchreid)
4	State Tracking	Consecutive displacement, large-jump fraction	Detection+tracking, VLM state accuracy
5	Temporal Coherence	Frame-to-frame PSNR	Optical flow consistency, FVD
6	Semantic/Logic Consistency	Rule-based physics violation rate	VLM common-sense, WorldModelBench

Paper Case and Video Access

For paper figures, prefer outputs from eval/v2/revisit_suite because each case stores the input frame, revisit-tail evidence frames, and the generated video in one directory:

eval_outputs/revisit_suite_<timestamp>/stage1/<run_id>/<domain>/<sample_id>/<mode>/

Useful files:

revisit_gen_only.mp4: generated return trajectory.
stage1_frames/first_00.png: source view.
stage1_frames/revisit_tail_*.png: final return frames.
stage1_frames/first_last_chunk_changes/*.png: optional visual change maps.
stage1_metrics.json and vlm_score.json: case-level metrics and VLM scores.

Serve the output folder when reviewing videos remotely:

python -m http.server 8000 --directory eval_outputs

User Study (Long-Horizon Consistency)

To collect User Study consistency scores (1–5) for long sequences:

Export list: From evals_root, list all *_gen_only.mp4 files, for example find "$EVALS_ROOT" -name "*_gen_only.mp4" > video_list.txt.
Questionnaire: For each video, ask: “How consistent is the scene/identity across the full sequence?” (1 = very inconsistent, 5 = very consistent).
Summary: Store responses in a CSV with columns e.g. video_path,score. Aggregate: mean and std of score per run or per model.

No automatic scoring is implemented; the pipeline only provides the list and this procedure.

Optional Dependencies

Phase 1 (no extra deps): numpy, opencv-python, PIL; skimage for PSNR/SSIM (recommended).
Optional:
- scikit-image — PSNR/SSIM in loop_closure and temporal_coherence.
- CLIP (diffsynth ImageQualityMetric) — --use_clip in identity_preservation (requires model weights under models/QualityMetric/).
- Face / ReID: insightface, torchreid — for identity_preservation Face Embedding and character ID (placeholders in code).
- Optical flow: RAFT or torchvision.optical_flow — for temporal_coherence flow consistency (placeholder).
- FVD: pytorch-fvd or I3D — for temporal_coherence FVD (placeholder).
- VLM: local or API — for semantic_consistency common-sense/physics (placeholder).

Save optional deps to a separate file if needed, e.g. requirements-optional.txt:

scikit-image
# insightface
# torchreid

WorldModelBench

For WorldModelBench or similar benchmarks: use their official data and evaluation protocol. This repo does not implement their scoring. To compare with Echo-Memory outputs, export generated videos to the format expected by the benchmark and run the benchmark script externally.

Running a Single Dimension

Each module can be run standalone:

python eval/metrics/long_horizon_consistency.py --evals_root "$EVALS_ROOT" --output metrics/dim1.json
python eval/metrics/loop_closure.py --evals_root "$EVALS_ROOT" --output metrics/dim2.json
python eval/metrics/identity_preservation.py --evals_root "$EVALS_ROOT" --output metrics/dim3.json
python eval/metrics/state_tracking.py --evals_root "$EVALS_ROOT" --output metrics/dim4.json
python eval/metrics/temporal_coherence.py --evals_root "$EVALS_ROOT" --output metrics/dim5.json
python eval/metrics/semantic_consistency.py --evals_root "$EVALS_ROOT" --output metrics/dim6.json