|
Download code/evaluation/README.md from luyu1021/turbot2va-ltx2: direct link, hf CLI and curl.
- Browser
- Download file 2.31 kB
-
https://huggingface.co/luyu1021/turbot2va-ltx2/resolve/main/code/evaluation/README.md
- Command line
-
hf download hf://luyu1021/turbot2va-ltx2/code/evaluation/README.md
-
curl -L -o README.md https://huggingface.co/luyu1021/turbot2va-ltx2/resolve/main/code/evaluation/README.md
2.31 kB
| # turbot2av-eval | |
| Evaluation utilities for TurboT2AV quality and diversity experiments. | |
| The repository intentionally stores code, configs, and documentation only. | |
| Large assets such as videos, audio, checkpoints, ImageBind weights, and | |
| JavisBench outputs should stay on local storage or Hugging Face datasets. | |
| ## What This Runs | |
| - Quality: calls JavisBench from `JavisDiT` on one generated sample per prompt. | |
| - Diversity generation: calls the TurboT2AV/LTX inference script to produce | |
| `num_prompts x num_seeds` samples. | |
| - Diversity metrics: computes same-prompt pairwise ImageBind distances over | |
| video, audio, and joint audio-video embeddings. | |
| ## Expected Layout | |
| The default config points to the current lab paths: | |
| - `JavisDiT`: `/home/jovyan/codes/turbodiff/new_Turbo/JavisDiT` | |
| - `LTX-2`: `/home/jovyan/codes/turbodiff/new_Turbo/TurboT2AV/LTX-2` | |
| - data root: `/data/datasets/turbodiff_datasets_and_ckpt` | |
| - diversity outputs: `/data/datasets/turbodiff_datasets_and_ckpt/main_experiment/diversity_200x8` | |
| Edit `configs/turbot2av_main.yaml` when moving to another machine or adding a | |
| new model. | |
| ## Quick Commands | |
| Install the lightweight launcher environment: | |
| ```bash | |
| pixi install | |
| ``` | |
| Run JavisBench quality on existing diversity seed0000 outputs: | |
| ```bash | |
| pixi run quality --gpus 0,1,2,3 --only 05_scm1_dmd1_pgmean05_ckpt5000 | |
| ``` | |
| Generate 200 prompts x 8 seeds, one model per GPU: | |
| ```bash | |
| pixi run diversity-generate --gpus 0,1,2,3 --only 05_scm1_dmd1_pgmean05_ckpt5000 | |
| ``` | |
| Compute ImageBind pairwise diversity: | |
| ```bash | |
| pixi run diversity-metrics --gpu 0 --only 05_scm1_dmd1_pgmean05_ckpt5000 | |
| ``` | |
| Merge quality and diversity outputs into a compact CSV/Markdown table: | |
| ```bash | |
| pixi run summary | |
| ``` | |
| All scripts remove proxy variables from child processes by default. | |
| ## Outputs | |
| Quality: | |
| ```text | |
| <quality_output_root>/<model>.json | |
| <quality_output_root>/batch_summary.json | |
| ``` | |
| Diversity generation: | |
| ```text | |
| <diversity_root>/<model>/sample_0000_seed0000.mp4 | |
| <diversity_root>/<model>/sample_0000_seed0000.wav | |
| <diversity_root>/generation_summary.json | |
| ``` | |
| Diversity metrics: | |
| ```text | |
| <diversity_metric_root>/summary.csv | |
| <diversity_metric_root>/summary.json | |
| <diversity_metric_root>/embeddings/<model>.pt | |
| ``` | |
| Merged summary: | |
| ```text | |
| results/metrics_summary.csv | |
| results/metrics_summary.md | |
| ``` | |