luyu1021's picture
Add documented LTX-2 training source, recipes and checkpoint release plan
76b7b25 verified
|
Raw History Blame Contribute Delete
2.31 kB

turbot2av-eval

Evaluation utilities for TurboT2AV quality and diversity experiments.

The repository intentionally stores code, configs, and documentation only. Large assets such as videos, audio, checkpoints, ImageBind weights, and JavisBench outputs should stay on local storage or Hugging Face datasets.

What This Runs

  • Quality: calls JavisBench from JavisDiT on one generated sample per prompt.
  • Diversity generation: calls the TurboT2AV/LTX inference script to produce num_prompts x num_seeds samples.
  • Diversity metrics: computes same-prompt pairwise ImageBind distances over video, audio, and joint audio-video embeddings.

Expected Layout

The default config points to the current lab paths:

  • JavisDiT: /home/jovyan/codes/turbodiff/new_Turbo/JavisDiT
  • LTX-2: /home/jovyan/codes/turbodiff/new_Turbo/TurboT2AV/LTX-2
  • data root: /data/datasets/turbodiff_datasets_and_ckpt
  • diversity outputs: /data/datasets/turbodiff_datasets_and_ckpt/main_experiment/diversity_200x8

Edit configs/turbot2av_main.yaml when moving to another machine or adding a new model.

Quick Commands

Install the lightweight launcher environment:

pixi install

Run JavisBench quality on existing diversity seed0000 outputs:

pixi run quality --gpus 0,1,2,3 --only 05_scm1_dmd1_pgmean05_ckpt5000

Generate 200 prompts x 8 seeds, one model per GPU:

pixi run diversity-generate --gpus 0,1,2,3 --only 05_scm1_dmd1_pgmean05_ckpt5000

Compute ImageBind pairwise diversity:

pixi run diversity-metrics --gpu 0 --only 05_scm1_dmd1_pgmean05_ckpt5000

Merge quality and diversity outputs into a compact CSV/Markdown table:

pixi run summary

All scripts remove proxy variables from child processes by default.

Outputs

Quality:

<quality_output_root>/<model>.json
<quality_output_root>/batch_summary.json

Diversity generation:

<diversity_root>/<model>/sample_0000_seed0000.mp4
<diversity_root>/<model>/sample_0000_seed0000.wav
<diversity_root>/generation_summary.json

Diversity metrics:

<diversity_metric_root>/summary.csv
<diversity_metric_root>/summary.json
<diversity_metric_root>/embeddings/<model>.pt

Merged summary:

results/metrics_summary.csv
results/metrics_summary.md