CLIP-GmP-Reword-ViT-L-14

This model is an improved version of zer0int/CLIP-GmP-ViT-L-14, trained with a re-worded label dataset.

ℹ️ CLIP's 'text-reading obsession' aka typographic attack vulnerability emerges due to shortcut learning; InfoNCE contrastive loss can be reduced via matching text-in-image == text-in-label.
💡 We mitigate further encouraging this shortcut by avoiding labels that mention any word present in a given image.
👉 You can find the code to fine-tune CLIP on github.com/zer0int/CLIP-fine-tune, including the OCR + local LLM re-labeling pipeline and the final *_gpt_oss.json dataset labels used to train this model.

A fine-tune of CLIP-L. Original model: openai/clip-vit-large-patch14

Love ❤️ this CLIP?

ᐅ Buy me a coffee on Ko-Fi ☕

Or click here for address to send 🪙₿ BTC

3PscBrWYvrutXedLmvpcnQbE12Py8qLqMK

---

Evaluation Results

TA = Typographic Attack ZS dataset

Section	Measurement / Task	Pre-Trained	GmP-CLIP	GmP-Reword
zer0int / RTA-100 (TA)	NoRTA	0.9880	0.9920	0.9920
HF Datasets	RTA	0.4310	0.6020	0.7810
	SynthRTA	0.3890	0.5990	0.7620
BLISS-e-V / SCAM (TA)	NoSCAM	0.9905	0.9897	0.9880
HF Datasets	SCAM	0.4191	0.6523	0.7470
	SynthSCAM	0.3227	0.6136	0.7263
ILSVRC2012 Linear Probe	Top-1	72.35%	72.35%	71.57%
git/zer0int	Top-5	93.42%	94.20%	93.74%
ObjectNet MVT (ZS)	Accuracy	0.8652	0.8868	0.8664
git/zer0int
ImageNet 1k (ZS)	acc1	0.32696	0.4787	0.4043
LAION/CLIP-Benchmark	acc5	0.52997	0.6872	0.6357
	mean_per_class_recall	0.32609	0.4767	0.4028
VoC-2007 (ZS)	mAP	0.7615	0.8574	0.8530
LAION/CLIP-Benchmark
mscoco ZS Retrieval	image_retrieval_recall@5	0.2196	0.3415	0.3427
LAION/CLIP-Benchmark	text_retrieval_recall@5	0.3032	0.4991	0.4934
xm3600 ZS Retrieval	image_retrieval_recall@5	0.30593	0.4160	0.4156
LAION/CLIP-Benchmark	text_retrieval_recall@5	0.24293	0.4105	0.4026
Sugar_Crepe (PT)	add_obj: acc	0.7842	0.9263	0.9685
git/zer0int	add_att: acc	0.7168	0.8280	0.9220
	replace_obj: acc	0.9407	0.9673	0.9782
	replace_att: acc	0.7919	0.8655	0.8693
	replace_rel: acc	0.6529	0.7674	0.7617
	swap_obj: acc	0.6041	0.7184	0.6816
	swap_att: acc	0.6261	0.6937	0.7252
Flickr-8k Cross-modal	Euclidean Gap (center)	0.8299	0.8038	0.6717
git/zer0int	Geometry:Image cone_R	0.7481	0.7285	0.4370
	Geometry:Text cone_R	0.5908	0.5023	0.4636
	Image-Text Cos Sim (mean)	0.2754	0.3583	0.3379
	Text-Text Cos Sim (mean)	0.6762	0.6973	0.6628
	Image-Image Cos Sim (mean)	0.5594	0.5306	0.1899

---