File size: 2,912 Bytes
a9e3126 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | # Unified-Bench old protocol reproduction bundle
This folder contains the assets used to reproduce the historical raw BAGEL
Unified-Bench score `0.8345940754981711` under the old TorchUMM protocol.
HF path prefix:
```text
HuggingSelf/test0519/unified_bench_old_protocol_0p8346
```
## Contents
- `inputs/unified_bench/Image/`: the 100 Unified-Bench reference input images.
- `outputs/raw_bagel/images/`: raw BAGEL round-trip generated images.
- `outputs/raw_bagel/captions.jsonl`: image-to-text captions produced by raw BAGEL.
- `outputs/raw_bagel/summary.json`: original old-protocol score JSON.
- `scorers/dinov3-vitl16_old_protocol/`: the old DINOv3 scorer checkpoint used by the historical run.
- `configs/unified_bench_bagel.yaml`: old TorchUMM Unified-Bench config.
- `scripts/reproduce_apr4_unified_bagel_score.py`: Modal-side reproduction helper used to verify the score.
- `scripts/run_scoring.py` and `scripts/scorers/`: scoring code snapshot from TorchUMM.
- `manifest.json`: hashes, counts, paths, and expected scores.
## Expected Score
| scorer | average |
| --- | ---: |
| CLIP | 0.894650644659996 |
| DINOv2 | 0.787667881194502 |
| DINOv3 old protocol | 0.723970373794436 |
| LongCLIP | 0.932087402343750 |
| Overall | 0.834594075498171 |
Overall is the mean of the four scorer averages.
## DINOv3
Old-protocol DINOv3 file:
```text
scorers/dinov3-vitl16_old_protocol/model.safetensors
```
Expected SHA256:
```text
dcb2e45127cccbf1601e5f42fef165eea275c8e5213197e8dcf3f48822718179
```
The newer cached DINOv3 weight we tested separately produced very high
reference-to-reference similarity and should not be mixed with this old
protocol score.
## Reproduction Notes
The historical run was verified on Modal workspace `aifrontierwm` using:
```text
ref_dir = /datasets/unified_bench/Image
gen_dir = /output-apr4/unified_bench/bagel/images
```
Scoring model paths:
```text
clip = /model_cache/unified_bench/clip-vit-large-patch14
dinov2 = /model_cache/unified_bench/dinov2-large
dinov3 = /model_cache/unified_bench/dinov3-vitl16
longclip = /model_cache/unified_bench/longclip-L.pt
```
The bundle includes the old DINOv3 directory because this is the piece that made
the old protocol reproducible. CLIP, DINOv2, and LongCLIP are public/downloadable
from the sources listed in `manifest.json`.
## Minimal Download Example
```python
from huggingface_hub import snapshot_download
root = snapshot_download(
repo_id="HuggingSelf/test0519",
repo_type="model",
allow_patterns=["unified_bench_old_protocol_0p8346/**"],
)
print(root)
```
After downloading, point TorchUMM's Unified-Bench scorer at:
```text
ref_dir: <download_root>/unified_bench_old_protocol_0p8346/inputs/unified_bench/Image
out_dir: <download_root>/unified_bench_old_protocol_0p8346/outputs/raw_bagel
dinov3: <download_root>/unified_bench_old_protocol_0p8346/scorers/dinov3-vitl16_old_protocol
```
|