# turbot2av-eval Evaluation utilities for TurboT2AV quality and diversity experiments. The repository intentionally stores code, configs, and documentation only. Large assets such as videos, audio, checkpoints, ImageBind weights, and JavisBench outputs should stay on local storage or Hugging Face datasets. ## What This Runs - Quality: calls JavisBench from `JavisDiT` on one generated sample per prompt. - Diversity generation: calls the TurboT2AV/LTX inference script to produce `num_prompts x num_seeds` samples. - Diversity metrics: computes same-prompt pairwise ImageBind distances over video, audio, and joint audio-video embeddings. ## Expected Layout The default config points to the current lab paths: - `JavisDiT`: `/home/jovyan/codes/turbodiff/new_Turbo/JavisDiT` - `LTX-2`: `/home/jovyan/codes/turbodiff/new_Turbo/TurboT2AV/LTX-2` - data root: `/data/datasets/turbodiff_datasets_and_ckpt` - diversity outputs: `/data/datasets/turbodiff_datasets_and_ckpt/main_experiment/diversity_200x8` Edit `configs/turbot2av_main.yaml` when moving to another machine or adding a new model. ## Quick Commands Install the lightweight launcher environment: ```bash pixi install ``` Run JavisBench quality on existing diversity seed0000 outputs: ```bash pixi run quality --gpus 0,1,2,3 --only 05_scm1_dmd1_pgmean05_ckpt5000 ``` Generate 200 prompts x 8 seeds, one model per GPU: ```bash pixi run diversity-generate --gpus 0,1,2,3 --only 05_scm1_dmd1_pgmean05_ckpt5000 ``` Compute ImageBind pairwise diversity: ```bash pixi run diversity-metrics --gpu 0 --only 05_scm1_dmd1_pgmean05_ckpt5000 ``` Merge quality and diversity outputs into a compact CSV/Markdown table: ```bash pixi run summary ``` All scripts remove proxy variables from child processes by default. ## Outputs Quality: ```text /.json /batch_summary.json ``` Diversity generation: ```text //sample_0000_seed0000.mp4 //sample_0000_seed0000.wav /generation_summary.json ``` Diversity metrics: ```text /summary.csv /summary.json /embeddings/.pt ``` Merged summary: ```text results/metrics_summary.csv results/metrics_summary.md ```