unified_bench_old_protocol_0p8346/README.md

# Unified-Bench old protocol reproduction bundle

This folder contains the assets used to reproduce the historical raw BAGEL
Unified-Bench score `0.8345940754981711` under the old TorchUMM protocol.

HF path prefix:

```text
HuggingSelf/test0519/unified_bench_old_protocol_0p8346
```

## Contents

- `inputs/unified_bench/Image/`: the 100 Unified-Bench reference input images.
- `outputs/raw_bagel/images/`: raw BAGEL round-trip generated images.
- `outputs/raw_bagel/captions.jsonl`: image-to-text captions produced by raw BAGEL.
- `outputs/raw_bagel/summary.json`: original old-protocol score JSON.
- `scorers/dinov3-vitl16_old_protocol/`: the old DINOv3 scorer checkpoint used by the historical run.
- `configs/unified_bench_bagel.yaml`: old TorchUMM Unified-Bench config.
- `scripts/reproduce_apr4_unified_bagel_score.py`: Modal-side reproduction helper used to verify the score.
- `scripts/run_scoring.py` and `scripts/scorers/`: scoring code snapshot from TorchUMM.
- `manifest.json`: hashes, counts, paths, and expected scores.

## Expected Score

| scorer | average |
| --- | ---: |
| CLIP | 0.894650644659996 |
| DINOv2 | 0.787667881194502 |
| DINOv3 old protocol | 0.723970373794436 |
| LongCLIP | 0.932087402343750 |
| Overall | 0.834594075498171 |

Overall is the mean of the four scorer averages.

## DINOv3

Old-protocol DINOv3 file:

```text
scorers/dinov3-vitl16_old_protocol/model.safetensors
```

Expected SHA256:

```text
dcb2e45127cccbf1601e5f42fef165eea275c8e5213197e8dcf3f48822718179
```

The newer cached DINOv3 weight we tested separately produced very high
reference-to-reference similarity and should not be mixed with this old
protocol score.

## Reproduction Notes

The historical run was verified on Modal workspace `aifrontierwm` using:

```text
ref_dir = /datasets/unified_bench/Image
gen_dir = /output-apr4/unified_bench/bagel/images
```

Scoring model paths:

```text
clip     = /model_cache/unified_bench/clip-vit-large-patch14
dinov2   = /model_cache/unified_bench/dinov2-large
dinov3   = /model_cache/unified_bench/dinov3-vitl16
longclip = /model_cache/unified_bench/longclip-L.pt
```

The bundle includes the old DINOv3 directory because this is the piece that made
the old protocol reproducible. CLIP, DINOv2, and LongCLIP are public/downloadable
from the sources listed in `manifest.json`.

## Minimal Download Example

```python
from huggingface_hub import snapshot_download

root = snapshot_download(
    repo_id="HuggingSelf/test0519",
    repo_type="model",
    allow_patterns=["unified_bench_old_protocol_0p8346/**"],
)
print(root)
```

After downloading, point TorchUMM's Unified-Bench scorer at:

```text
ref_dir: <download_root>/unified_bench_old_protocol_0p8346/inputs/unified_bench/Image
out_dir: <download_root>/unified_bench_old_protocol_0p8346/outputs/raw_bagel
dinov3:  <download_root>/unified_bench_old_protocol_0p8346/scorers/dinov3-vitl16_old_protocol
```