Update README.md

README.md

@@ -2,13 +2,119 @@
 license: mit
 ---
 
-Placeholder
+Placeholder 
 
 -------
+Love ❤️ this CLIP? 
 
+ᐅ  [Buy me a coffee](https://ko-fi.com/zer0int) on Ko-Fi ☕
+<details>
+<summary>Or click here for address to send 🪙₿ BTC</summary>
+  
+```  
+3PscBrWYvrutXedLmvpcnQbE12Py8qLqMK
+```
+</details>
+
+-------
 ### 📊 Standard Benchmark Evaluation
 🌟 = This Model
 
+#### Zero-Shot (Typographic Attack)
+
+| Task / Dataset      | Metric | pretrained | 🌟 regr-norm | regr-brut |
+|---|---|---:|---:|---:|
+| SCAM::NoSCAM        | acc    | 0.9905     | 0.9897      | 0.9897    |
+| SCAM::SCAM          | acc    | 0.4191     | 0.8046      | 0.8830    |
+| SCAM::SynthSCAM     | acc    | 0.3227     | 0.8029      | 0.8804    |
+| RTA100              | acc    | 0.4330     | 0.7880      | 0.8930    |
+
+
+<details>
+<summary>👉 CLICK to reproduce: Expand SCAM typographic attack benchmark code ⚡💻</summary>
+
+```
+from datasets import load_dataset
+from transformers import CLIPModel, CLIPProcessor
+import torch
+from PIL import Image
+from tqdm import tqdm
+import pandas as pd
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+
+# BLISS / SCAM Typographic Attack Dataset
+# https://huggingface.co/datasets/BLISS-e-V/SCAM
+ds = load_dataset("BLISS-e-V/SCAM", split="train")
+
+# Benchmark pre-trained model against my fine-tune
+model_variants = [
+    ("OpenAI ", "openai/clip-vit-large-patch14-336", "openai/clip-vit-large-patch14-336"),
+    ("regr-norm", "zer0int/CLIP-Regression-ViT-L-14", "zer0int/CLIP-Regression-ViT-L-14"),
+    ("regr-brut", "zer0int/CLIP-Regression-BRUT-ViT-L-14", "zer0int/CLIP-Regression-BRUT-ViT-L-14"),
+]
+
+models = {}
+for name, model_path, processor_path in model_variants:
+    model = CLIPModel.from_pretrained(model_path).to(device).float()
+    processor = CLIPProcessor.from_pretrained(processor_path)
+    models[name] = (model, processor)
+
+for variant in ["NoSCAM", "SCAM", "SynthSCAM"]:
+    print(f"\n=== Evaluating var.: {variant} ===")
+    idxs = [i for i, v in enumerate(ds['id']) if v.startswith(variant)]
+    if not idxs:
+        print(f"  No samples for {variant}")
+        continue
+    subset = [ds[i] for i in idxs]
+
+    for model_name, (model, processor) in models.items():
+        results = []
+        for entry in tqdm(subset, desc=f"{model_name}", ncols=30, bar_format="{l_bar}{bar}| {n_fmt}/{total_fmt} |"):
+            img = entry['image']
+            object_label = entry['object_label']
+            attack_word = entry['attack_word']
+
+            texts = [f"a photo of a {object_label}", f"a photo of a {attack_word}"]
+            inputs = processor(
+                text=texts,
+                images=img,
+                return_tensors="pt",
+                padding=True
+            )
+            for k in inputs:
+                if isinstance(inputs[k], torch.Tensor):
+                    inputs[k] = inputs[k].to(device)
+
+            with torch.no_grad():
+                outputs = model(**inputs)
+                image_features = outputs.image_embeds
+                text_features = outputs.text_embeds
+
+                logits = image_features @ text_features.T
+                probs = logits.softmax(dim=-1).cpu().numpy().flatten()
+                pred_idx = probs.argmax()
+                pred_label = [object_label, attack_word][pred_idx]
+                is_correct = (pred_label == object_label)
+
+            results.append({
+                "id": entry['id'],
+                "object_label": object_label,
+                "attack_word": attack_word,
+                "pred_label": pred_label,
+                "is_correct": is_correct,
+                "type": entry['type'],
+                "model": model_name
+            })
+
+        n_total = len(results)
+        n_correct = sum(r['is_correct'] for r in results)
+        acc = n_correct / n_total if n_total else float('nan')
+        print(f"| > > > > Zero-shot accuracy for {variant}, {model_name}: {n_correct}/{n_total} = {acc:.4f}")
+```
+</details>
+
+
 #### Zero-Shot (CLIP Benchmark)
 
 | Task / Dataset | Metric | pretrained | 🌟 regr-norm | regr-brut |
@@ -59,14 +165,6 @@ Placeholder
 | Linear Probe Top-1 (%) | 72.35 | 70.94 | 65.09 |
 | Linear Probe Top-5 (%) | 93.42 | 93.29 | 89.60 |
 
-#### One-shot ImageNet-1k (Top-5), linear classifier on final embedding — own scripts
-
-| Representation | pretrained | 🌟 regr-norm | regr-brut |
-|---|---:|---:|---:|
-| One-shot Top-5 (FULL) | 0.437 | 0.599 | 0.553 |
-| One-shot Top-5 (PATCH-ONLY) | 0.443 | 0.603 | 0.550 |
-| One-shot Top-5 (REG-ONLY) | 0.154 | 0.010 | 0.007 |
-
 🔗 Note: 'own scripts' available at [github.com/zer0int/CLIP-fine-tune](https://github.com/zer0int/CLIP-fine-tune)
 
 -------
@@ -81,7 +179,8 @@ Please see the paper for more information!
 |---|---|---:|---:|---:|
 | NoSCAM (1162) | CLS | 0.9905 | 0.9897 | 0.9897 |
 | NoSCAM (1162) | CLS-PATCHSUB | 0.9544 | 0.9845 | 0.9811 |
-| NoSCAM (1162) | CLS-PATCHREG | 0.9466 | 0.9888 | 0.9888 |
+| NoSCAM (1162) | CLS-PATCHREG-I | 0.9466 | 0.9888 | 0.9888 |
+| NoSCAM (1162) | CLS-PATCHREG-N | 0.9871 | 0.9897 | 0.9888 |
 | NoSCAM (1162) | REG-L23-NOPC | 0.9380 | 0.9613 | 0.9570 |
 | NoSCAM (1162) | REG-L23-1PC | 0.9630 | 0.9802 | 0.9802 |
 | NoSCAM (1162) | REG-L23-8PC | 0.9509 | 0.9664 | 0.9604 |
@@ -89,7 +188,8 @@ Please see the paper for more information!
 | NoSCAM (1162) | PATCHΔ | 0.9690 | 0.9905 | 0.9888 |
 | SCAM (1162) | CLS | 0.4182 | 0.8038 | 0.8830 |
 | SCAM (1162) | CLS-PATCHSUB | 0.4957 | 0.8632 | 0.9002 |
-| SCAM (1162) | CLS-PATCHREG | 0.8761 | 0.8537 | 0.9174 |
+| SCAM (1162) | CLS-PATCHREG-I | 0.8761 | 0.8537 | 0.9174 |
+| SCAM (1162) | CLS-PATCHREG-N | 0.9286 | 0.8537 | 0.9165 |
 | SCAM (1162) | REG-L23-NOPC | 0.7410 | 0.8244 | 0.7719 |
 | SCAM (1162) | REG-L23-1PC | 0.7539 | 0.8726 | 0.7943 |
 | SCAM (1162) | REG-L23-8PC | 0.7057 | 0.8038 | 0.7143 |
@@ -97,7 +197,8 @@ Please see the paper for more information!
 | SCAM (1162) | PATCHΔ | 0.8778 | 0.8451 | 0.8744 |
 | SynthSCAM (1162) | CLS | 0.3219 | 0.8021 | 0.8804 |
 | SynthSCAM (1162) | CLS-PATCHSUB | 0.4406 | 0.8580 | 0.9071 |
-| SynthSCAM (1162) | CLS-PATCHREG | 0.8890 | 0.8460 | 0.9200 |
+| SynthSCAM (1162) | CLS-PATCHREG-I | 0.8890 | 0.8460 | 0.9200 |
+| SynthSCAM (1162) | CLS-PATCHREG-N | 0.9449 | 0.8494 | 0.9200 |
 | SynthSCAM (1162) | REG-L23-NOPC | 0.7823 | 0.8382 | 0.7771 |
 | SynthSCAM (1162) | REG-L23-1PC | 0.8055 | 0.8812 | 0.8072 |
 | SynthSCAM (1162) | REG-L23-8PC | 0.7289 | 0.8167 | 0.7126 |
@@ -105,10 +206,10 @@ Please see the paper for more information!
 | SynthSCAM (1162) | PATCHΔ | 0.9217 | 0.8614 | 0.8769 |
 | MVT (200382) | CLS | 0.8830 | 0.8730 | 0.8573 |
 | MVT (200382) | CLS-PATCHSUB | 0.4720 | 0.8246 | 0.8057 |
-| MVT (200382) | CLS-PATCHREG | 0.7166 | 0.8703 | 0.8518 |
+| MVT (200382) | CLS-PATCHREG-I | 0.7166 | 0.8703 | 0.8518 |
+| MVT (200382) | CLS-PATCHREG-N | 0.5695 | 0.8675 | 0.8478 |
 | MVT (200382) | REG-L23-NOPC | 0.7640 | 0.7935 | 0.7680 |
 | MVT (200382) | REG-L23-1PC | 0.7921 | 0.8193 | 0.8032 |
 | MVT (200382) | REG-L23-8PC | 0.7724 | 0.8057 | 0.7812 |
 | MVT (200382) | PATCH-L23 | 0.3414 | 0.8652 | 0.8191 |
-| MVT (200382) | PATCHΔ | 0.6881 | 0.8667 | 0.8510 |
-
+| MVT (200382) | PATCHΔ | 0.6881 | 0.8667 | 0.8510 |