Vanguard — Moderation Checkpoints
Trained heads for multimodal-content-moderation.
Each is a small classifier over a frozen CLIP ViT-B/32 backbone, which is not
included here — it is loaded from openai/clip-vit-base-patch32 at runtime.
Six checkpoints: three arms on each of two benchmarks.
| Arm | Benchmark | Test macro-F1 | Temperature |
|---|---|---|---|
cv_only |
Hateful Memes | 0.6217 | 1.57 |
nlp_only |
Hateful Memes | 0.6283 | 2.38 |
cross_attention |
Hateful Memes | 0.7035 | 8.60 |
cv_only |
Fakeddit | 0.6863 | 1.62 |
nlp_only |
Fakeddit | 0.7031 | 3.08 |
cross_attention |
Fakeddit | 0.7705 | 4.47 |
Each benchmark keeps its own trio. A Hateful Memes model's misinformation head never saw a misinformation label, so serving it would produce a confident number with nothing behind it.
Calibration
Each file carries a temperature fitted on validation. Logits must be divided
by it before softmax. The models are severely overconfident without it —
expected calibration error was 0.28 for the Hateful Memes fusion arm — and
temperature scaling reduces that to 0.036 without changing a single prediction,
since dividing by a positive scalar cannot move an argmax.
The size of the temperature is itself the finding: 8.60 is a large correction, and training loss reached 0.0008 while validation plateaued.
Result, stated honestly
Cross-attention beats late fusion on Hateful Memes by 0.0125 macro-F1 at p = 0.051, which is not significant at the conventional threshold. On Fakeddit the two are indistinguishable (p = 0.596). The split is consistent with Hateful Memes being constructed so neither modality alone is offensive, while Fakeddit's text frequently carries the label by itself.
Loading
import torch
from huggingface_hub import hf_hub_download
path = hf_hub_download("Aadithya1122/vanguard-moderation-checkpoints",
"cross_attention__hateful_memes.pt")
blob = torch.load(path, map_location="cpu", weights_only=False)
blob["state_dict"], blob["config"], blob["temperature"]
Trained on a MacBook Air M4 (MPS). No CUDA anywhere in the pipeline.