File size: 2,307 Bytes
76b7b25
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
# turbot2av-eval

Evaluation utilities for TurboT2AV quality and diversity experiments.

The repository intentionally stores code, configs, and documentation only.
Large assets such as videos, audio, checkpoints, ImageBind weights, and
JavisBench outputs should stay on local storage or Hugging Face datasets.

## What This Runs

- Quality: calls JavisBench from `JavisDiT` on one generated sample per prompt.
- Diversity generation: calls the TurboT2AV/LTX inference script to produce
  `num_prompts x num_seeds` samples.
- Diversity metrics: computes same-prompt pairwise ImageBind distances over
  video, audio, and joint audio-video embeddings.

## Expected Layout

The default config points to the current lab paths:

- `JavisDiT`: `/home/jovyan/codes/turbodiff/new_Turbo/JavisDiT`
- `LTX-2`: `/home/jovyan/codes/turbodiff/new_Turbo/TurboT2AV/LTX-2`
- data root: `/data/datasets/turbodiff_datasets_and_ckpt`
- diversity outputs: `/data/datasets/turbodiff_datasets_and_ckpt/main_experiment/diversity_200x8`

Edit `configs/turbot2av_main.yaml` when moving to another machine or adding a
new model.

## Quick Commands

Install the lightweight launcher environment:

```bash
pixi install
```

Run JavisBench quality on existing diversity seed0000 outputs:

```bash
pixi run quality --gpus 0,1,2,3 --only 05_scm1_dmd1_pgmean05_ckpt5000
```

Generate 200 prompts x 8 seeds, one model per GPU:

```bash
pixi run diversity-generate --gpus 0,1,2,3 --only 05_scm1_dmd1_pgmean05_ckpt5000
```

Compute ImageBind pairwise diversity:

```bash
pixi run diversity-metrics --gpu 0 --only 05_scm1_dmd1_pgmean05_ckpt5000
```

Merge quality and diversity outputs into a compact CSV/Markdown table:

```bash
pixi run summary
```

All scripts remove proxy variables from child processes by default.

## Outputs

Quality:

```text
<quality_output_root>/<model>.json
<quality_output_root>/batch_summary.json
```

Diversity generation:

```text
<diversity_root>/<model>/sample_0000_seed0000.mp4
<diversity_root>/<model>/sample_0000_seed0000.wav
<diversity_root>/generation_summary.json
```

Diversity metrics:

```text
<diversity_metric_root>/summary.csv
<diversity_metric_root>/summary.json
<diversity_metric_root>/embeddings/<model>.pt
```

Merged summary:

```text
results/metrics_summary.csv
results/metrics_summary.md
```