---
license: mit
tags:
  - content-moderation
  - multimodal
  - cross-attention
  - clip
library_name: pytorch
---

# Vanguard — Moderation Checkpoints

Trained heads for [multimodal-content-moderation](https://github.com/Aadithyaar22/multimodal-content-moderation).
Each is a small classifier over a **frozen** CLIP ViT-B/32 backbone, which is not
included here — it is loaded from `openai/clip-vit-base-patch32` at runtime.

Six checkpoints: three arms on each of two benchmarks.

| Arm | Benchmark | Test macro-F1 | Temperature |
|---|---|---|---|
| `cv_only` | Hateful Memes | 0.6217 | 1.57 |
| `nlp_only` | Hateful Memes | 0.6283 | 2.38 |
| `cross_attention` | Hateful Memes | 0.7035 | 8.60 |
| `cv_only` | Fakeddit | 0.6863 | 1.62 |
| `nlp_only` | Fakeddit | 0.7031 | 3.08 |
| `cross_attention` | Fakeddit | 0.7705 | 4.47 |

Each benchmark keeps its own trio. A Hateful Memes model's misinformation head
never saw a misinformation label, so serving it would produce a confident number
with nothing behind it.

## Calibration

Each file carries a `temperature` fitted on validation. Logits must be divided
by it before softmax. The models are severely overconfident without it —
expected calibration error was 0.28 for the Hateful Memes fusion arm — and
temperature scaling reduces that to 0.036 without changing a single prediction,
since dividing by a positive scalar cannot move an argmax.

The size of the temperature is itself the finding: 8.60 is a large correction,
and training loss reached 0.0008 while validation plateaued.

## Result, stated honestly

Cross-attention beats late fusion on Hateful Memes by 0.0125 macro-F1 at
**p = 0.051**, which is not significant at the conventional threshold. On
Fakeddit the two are indistinguishable (p = 0.596). The split is consistent with
Hateful Memes being constructed so neither modality alone is offensive, while
Fakeddit's text frequently carries the label by itself.

## Loading

```python
import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download("Aadithya1122/vanguard-moderation-checkpoints",
                       "cross_attention__hateful_memes.pt")
blob = torch.load(path, map_location="cpu", weights_only=False)
blob["state_dict"], blob["config"], blob["temperature"]
```

Trained on a MacBook Air M4 (MPS). No CUDA anywhere in the pipeline.