unified_bench_old_protocol_0p8346/README.md

Unified-Bench old protocol reproduction bundle

This folder contains the assets used to reproduce the historical raw BAGEL Unified-Bench score 0.8345940754981711 under the old TorchUMM protocol.

HF path prefix:

HuggingSelf/test0519/unified_bench_old_protocol_0p8346

Contents

• inputs/unified_bench/Image/: the 100 Unified-Bench reference input images.
• outputs/raw_bagel/images/: raw BAGEL round-trip generated images.
• outputs/raw_bagel/captions.jsonl: image-to-text captions produced by raw BAGEL.
• outputs/raw_bagel/summary.json: original old-protocol score JSON.
• scorers/dinov3-vitl16_old_protocol/: the old DINOv3 scorer checkpoint used by the historical run.
• configs/unified_bench_bagel.yaml: old TorchUMM Unified-Bench config.
• scripts/reproduce_apr4_unified_bagel_score.py: Modal-side reproduction helper used to verify the score.
• scripts/run_scoring.py and scripts/scorers/: scoring code snapshot from TorchUMM.
• manifest.json: hashes, counts, paths, and expected scores.

Expected Score

scorer	average
CLIP	0.894650644659996
DINOv2	0.787667881194502
DINOv3 old protocol	0.723970373794436
LongCLIP	0.932087402343750
Overall	0.834594075498171

Overall is the mean of the four scorer averages.

DINOv3

Old-protocol DINOv3 file:

scorers/dinov3-vitl16_old_protocol/model.safetensors

Expected SHA256:

dcb2e45127cccbf1601e5f42fef165eea275c8e5213197e8dcf3f48822718179

The newer cached DINOv3 weight we tested separately produced very high reference-to-reference similarity and should not be mixed with this old protocol score.

Reproduction Notes

The historical run was verified on Modal workspace aifrontierwm using:

ref_dir = /datasets/unified_bench/Image
gen_dir = /output-apr4/unified_bench/bagel/images

Scoring model paths:

clip     = /model_cache/unified_bench/clip-vit-large-patch14
dinov2   = /model_cache/unified_bench/dinov2-large
dinov3   = /model_cache/unified_bench/dinov3-vitl16
longclip = /model_cache/unified_bench/longclip-L.pt

The bundle includes the old DINOv3 directory because this is the piece that made the old protocol reproducible. CLIP, DINOv2, and LongCLIP are public/downloadable from the sources listed in manifest.json.

Minimal Download Example

from huggingface_hub import snapshot_download

root = snapshot_download(
    repo_id="HuggingSelf/test0519",
    repo_type="model",
    allow_patterns=["unified_bench_old_protocol_0p8346/**"],
)
print(root)

After downloading, point TorchUMM's Unified-Bench scorer at:

ref_dir: <download_root>/unified_bench_old_protocol_0p8346/inputs/unified_bench/Image
out_dir: <download_root>/unified_bench_old_protocol_0p8346/outputs/raw_bagel
dinov3:  <download_root>/unified_bench_old_protocol_0p8346/scorers/dinov3-vitl16_old_protocol