CorrectedTabaccoDatasetwithOCR benchmark

README.md

@@ -3,9 +3,11 @@
 Benchmarks several vision encoder families as **frozen embedding extractors** for document image classification on the [Tobacco-3482](https://huggingface.co/datasets/maveriq/tobacco3482) dataset.
 
 ---
-
 ## Dataset
 
+### Corrected Tabacco Dataset with OCR Benchmark
+ → [`README_tobacco3482_ocr.md`](tabacco3482_benchmarks/README_tobacco3482_ocr.md)
+
 | Property | Value |
 |---|---|
 | Dataset | `maveriq/tobacco3482` |

tabacco3482_benchmarks/README_tobacco3482_ocr.md

@@ -0,0 +1,170 @@
+# Tobacco-3482 OCR Dataset — Embedding Benchmark Results
+
+## Overview
+
+This benchmark evaluates three vision/text embedding models on the **Tobacco-3482 OCR dataset** for document image classification and retrieval. The dataset contains scanned document images across 10 real-world office document categories.
+
+- **Gallery (train + val):** 2,458 images  
+- **Query (test):** 279 images  
+- **Classes:** ADVE, Email, Form, Letter, Memo, News, Note, Report, Resume, Scientific
+
+---
+
+## Dataset Split
+
+| Class      | Gallery | Test |
+|------------|--------:|-----:|
+| ADVE       |     177 |   20 |
+| Email      |     508 |   57 |
+| Form       |     280 |   32 |
+| Letter     |     342 |   38 |
+| Memo       |     423 |   48 |
+| News       |     166 |   19 |
+| Note       |       148 |   17 |
+| Report     |     225 |   26 |
+| Resume     |     108 |   13 |
+| Scientific |      81 |    9 |
+
+![Class Distribution](figures/ocr_class_distribution.png)
+
+---
+
+## Sample Images per Class
+
+![Sample Images per Class](figures/ocr_class_samples.png)
+
+---
+
+## Benchmark Results
+
+| Model                          | Clf Acc | kNN@1  | kNN@5  |   Time  |
+|-------------------------------|--------:|-------:|-------:|--------:|
+| SigLIP2 Base (patch16-224)    |  85.30% | 78.49% | 77.06% |  262.9s |
+| SigLIP2 Large (patch16-256)   |  90.68% | 86.38% | 87.46% |  351.8s |
+| ModernVBERT Embed             |  **91.76%** | **86.38%** | 86.02% | 2698.2s |
+
+- **Best Clf Accuracy:** ModernVBERT Embed (91.76%)  
+- **Best kNN@1:** SigLIP2 Large & ModernVBERT Embed (tied at 86.38%)  
+- **Best kNN@5:** SigLIP2 Large (87.46%)
+
+### Evaluation Protocol
+- **Clf Accuracy / F1:** Logistic regression classifier trained on gallery embeddings, evaluated on query embeddings.  
+- **kNN@1 / kNN@5:** Cosine-similarity nearest-neighbour retrieval; majority-vote label from top-k gallery neighbours.
+
+![Results Bar Chart](figures/ocr_results_bar.png)
+
+---
+
+## Cross-Dataset Comparison (OCR vs. Raw Tobacco-3482)
+
+| Model                        | Dataset      | Clf Acc | kNN@1  | kNN@5  |
+|-----------------------------|--------------|--------:|-------:|-------:|
+| SigLIP2 Base (patch16-224)  | OCR          |  85.30% | 78.49% | 77.06% |
+|                              | Tobacco-3482 |  82.07% | 74.32% | 74.75% |
+| SigLIP2 Large (patch16-256) | OCR          |  90.68% | 86.38% | 87.46% |
+|                              | Tobacco-3482 |  86.80% | 79.91% | 82.35% |
+| ModernVBERT Embed           | OCR          |  91.76% | 86.38% | 86.02% |
+|                              | Tobacco-3482 |  83.79% | 80.06% | 80.20% |
+
+> **Key observation:** All models improve on the OCR-processed version of the dataset. ModernVBERT Embed shows the largest OCR gain for Clf Accuracy (+7.97 pp). Per the notebook, ModernVBERT is a **multimodal encoder** (images processed via a visual tokeniser fed into a language-model backbone), so it can directly leverage OCR-readable text content in addition to visual features — explaining why it responds more strongly to text clarity than the vision-only SigLIP models. SigLIP2 Large gains most consistently across all three metrics.
+
+---
+
+## Per-Label Error Analysis (Top-1 kNN)
+
+![Per-Label Error Rate](figures/ocr_error_rate_bar.png)
+
+### Bad-Case Summary
+
+#### SigLIP2 Base (patch16-224)
+| Label      | Bad / Total | Error  | Top Confusion Targets              |
+|------------|------------:|-------:|------------------------------------|
+| Report     |    12 / 26  |  46.2% ⚠ | Memo ×4, Letter ×4, Email ×3     |
+| ADVE       |     7 / 20  |  35.0% ⚠ | News ×6, Report ×1               |
+| Memo       |    16 / 48  |  33.3% ⚠ | Letter ×12, Email ×4             |
+| Note       |     5 / 17  |  29.4% ⚠ | Report ×2, Email ×2, Memo ×1    |
+| Letter     |    11 / 38  |  28.9% ⚠ | Memo ×10, Report ×1             |
+| News       |     3 / 19  |  15.8%   | Scientific ×2, Note ×1           |
+| Form       |     5 / 32  |  15.6%   | Memo ×3, Note ×1, Resume ×1     |
+| Resume     |     1 / 13  |   7.7%   | Report ×1                         |
+| Email      |     0 / 57  |   0.0%   | —                                  |
+| Scientific |      0 / 9  |   0.0%   | —                                  |
+
+#### SigLIP2 Large (patch16-256)
+| Label      | Bad / Total | Error  | Top Confusion Targets              |
+|------------|------------:|-------:|------------------------------------|
+| Report     |     8 / 26  |  30.8% ⚠ | Memo ×3, Letter ×2, Email ×2     |
+| Note       |     5 / 17  |  29.4% ⚠ | Email ×3, Report ×2              |
+| Memo       |    10 / 48  |  20.8%   | Letter ×5, Email ×2, Report ×2  |
+| Form       |     5 / 32  |  15.6%   | Report ×3, Letter ×1, Resume ×1 |
+| Letter     |     5 / 38  |  13.2%   | Memo ×3, Report ×2              |
+| News       |     2 / 19  |  10.5%   | Email ×1, Scientific ×1          |
+| ADVE       |     2 / 20  |  10.0%   | News ×2                           |
+| Resume     |     1 / 13  |   7.7%   | Report ×1                         |
+| Email      |     0 / 57  |   0.0%   | —                                  |
+| Scientific |      0 / 9  |   0.0%   | —                                  |
+
+#### ModernVBERT Embed
+| Label      | Bad / Total | Error  | Top Confusion Targets              |
+|------------|------------:|-------:|------------------------------------|
+| Report     |    14 / 26  |  57.7% ‼ | Letter ×4, Form ×3, News ×3      |
+| Memo       |     8 / 48  |  16.7%   | Letter ×6, Email ×1, Form ×1    |
+| Form       |     5 / 32  |  15.6%   | Letter ×2, Note ×2, Memo ×1    |
+| Note       |     2 / 17  |  11.8%   | Email ×1, Letter ×1              |
+| News       |     2 / 19  |  10.5%   | Memo ×1, ADVE ×1                |
+| Letter     |     3 / 38  |   7.9%   | Memo ×3                           |
+| ADVE       |     1 / 20  |   5.0%   | News ×1                           |
+| Email      |     2 / 57  |   3.5%   | Letter ×2                         |
+| Resume     |     0 / 13  |   0.0%   | —                                  |
+| Scientific |      0 / 9  |   0.0%   | —                                  |
+
+> **Notable:** ModernVBERT Embed achieves the highest overall accuracy but has a severe **Report** error rate (14/26, 57.7%). ModernVBERT processes images through a visual tokeniser into a language-model backbone — the embedding for each image is mean-pooled over image token positions. For Report documents, which vary widely in layout and textual content, this approach appears to produce embeddings that overlap with Letter, Form, News, Memo, Note, and Scientific — a potential blind spot where visual layout cues matter more than token-level semantics.
+
+---
+
+## Confusion Heatmaps
+
+### SigLIP2 Base (patch16-224)
+![Confusion Matrix — SigLIP2 Base](figures/ocr_confusion_SigLIP2_Base__OCR.png)
+
+### SigLIP2 Large (patch16-256)
+![Confusion Matrix — SigLIP2 Large](figures/ocr_confusion_SigLIP2_Large__OCR.png)
+
+### ModernVBERT Embed
+![Confusion Matrix — ModernVBERT Embed](figures/ocr_confusion_ModernVBERT_Embed__OCR.png)
+
+---
+
+## t-SNE Embeddings
+
+t-SNE projects the high-dimensional test query embeddings down to 2D for visual inspection of how well each model separates the 10 classes in embedding space. Tight, well-separated clusters indicate the model has learned discriminative representations; overlapping clusters explain where retrieval errors come from.
+
+### SigLIP2 Base (patch16-224)
+![t-SNE — SigLIP2 Base](figures/ocr_tsne_SigLIP2_Base__OCR.png)
+
+The embedding space shows partial structure but with significant mixing in the centre. Email and ADVE/News form loosely separated peripheral clusters, consistent with their low error rates. However, Letter, Memo, Report, and Note points are heavily interleaved in the central region — directly explaining the high mutual confusion between these classes. Resume and Scientific are relatively isolated, matching their near-zero error rates.
+
+### SigLIP2 Large (patch16-256)
+![t-SNE — SigLIP2 Large](figures/ocr_tsne_SigLIP2_Large__OCR.png)
+
+The larger model shows noticeably tighter and more separated clusters overall. Email forms a distinct dense cluster, and Scientific and Resume are cleanly isolated. Letter and Memo still overlap substantially, consistent with their remaining confusion, but the Report scatter is more contained compared to Base. The improved separation correlates directly with the jump in kNN@1 from 78.49% to 86.38%.
+
+### ModernVBERT Embed
+![t-SNE — ModernVBERT Embed](figures/ocr_tsne_ModernVBERT_Embed__OCR.png)
+
+ModernVBERT shows a distinctly different embedding geometry. Email is tightly clustered and well-separated at the bottom, and Resume/Scientific form clean isolated groups — consistent with their 0% error rates. However, Report points (grey) are scattered broadly across the plot with no coherent cluster, mixing into Letter, Form, Memo, and News regions. This directly visualises why Report has a 57.7% retrieval error rate: the model has not learned a stable representation for this class, likely because Report documents span highly varied layouts and content that the image-token mean-pool does not capture consistently.
+
+---
+
+## Key Takeaways
+
+1. **OCR pre-processing consistently helps** — all three models score higher on the OCR split vs. raw Tobacco-3482.
+2. **SigLIP2 Large** is the most balanced model: best kNN@5 (87.46%), strong Clf Acc, and competitive speed vs. ModernVBERT.
+3. **ModernVBERT Embed** edges out on overall Clf Acc (91.76%) but is ~7.7× slower and struggles severely with **Report** retrieval (57.7% error). Unlike SigLIP, ModernVBERT uses a visual tokeniser feeding into a language-model backbone — per the notebook, the benchmark embeds image tokens only (no OCR text input), so the embedding depends entirely on patch-level visual token representations.
+4. **Persistent hard classes:** Report, Memo, and Letter confuse each other across all models, reflecting genuine visual and textual overlap in the dataset.
+5. **Easy classes:** Email, Scientific, and Resume are reliably classified by all models.
+
+---
+
+Checkpointed results are saved to `results/results_checkpoint_tobacco3482ocr.json` after each model completes — safe to re-run after a crash.  
+Figures are saved to `figures/`.

tabacco3482_benchmarks/embeddings/test_emb_ModernVBERT_Embed_OCR.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27f34fab425907c3d9fd04e4157c4fdad70c1fc091cc8c1ea8dd238547ce704b
+size 857216

tabacco3482_benchmarks/embeddings/test_emb_SigLIP2_Base_patch16-224_OCR.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:107a005e2d7de90759c33b266408e5e05c70afc53283cc3ee0e4570bdab513d5
+size 857216

tabacco3482_benchmarks/embeddings/test_emb_SigLIP2_Large_patch16-256_OCR.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:263b5b80fb2007f3f3e26bb1067bf93989876ce3696c16cf463a0465d397b084
+size 1142912

tabacco3482_benchmarks/embeddings/train_emb_ModernVBERT_Embed_OCR.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3b4ae6b25e4734a628719b647513b76ea7761fa22af5450282ffc926d4a3662
+size 7551104

tabacco3482_benchmarks/embeddings/train_emb_SigLIP2_Base_patch16-224_OCR.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d7d830cb3620853e325cf3b2e868fd3de86497a179e11aeb193bd74599d0798
+size 7551104

tabacco3482_benchmarks/embeddings/train_emb_SigLIP2_Large_patch16-256_OCR.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f37bb9e72327ab21238758885dfd70c618729b751269ad1caec46ca8b2cb7a7
+size 10068096

tabacco3482_benchmarks/results/results_checkpoint_tobacco3482ocr.json

@@ -0,0 +1,44 @@
+{
+  "SigLIP2 Base (patch16-224) \u2014 OCR": {
+    "family": "siglip",
+    "model_id": "google/siglip2-base-patch16-224",
+    "clf_acc": 0.8530465949820788,
+    "f1": 0.850913760997383,
+    "knn1": 0.7849462365591398,
+    "knn5": 0.7706093189964157,
+    "time": 262.86674404144287,
+    "postprocess": "none",
+    "embed_dim": 768,
+    "compressed_dim": null,
+    "num_params_M": 375.2,
+    "batch_size": 64
+  },
+  "SigLIP2 Large (patch16-256) \u2014 OCR": {
+    "family": "siglip",
+    "model_id": "google/siglip2-large-patch16-256",
+    "clf_acc": 0.9068100358422939,
+    "f1": 0.9053795569607039,
+    "knn1": 0.8637992831541219,
+    "knn5": 0.8745519713261649,
+    "time": 351.804461479187,
+    "postprocess": "none",
+    "embed_dim": 1024,
+    "compressed_dim": null,
+    "num_params_M": 881.5,
+    "batch_size": 32
+  },
+  "ModernVBERT Embed \u2014 OCR": {
+    "family": "modernvbert",
+    "model_id": "ModernVBERT/modernvbert-embed",
+    "clf_acc": 0.9175627240143369,
+    "f1": 0.9163334021627294,
+    "knn1": 0.8637992831541219,
+    "knn5": 0.8602150537634409,
+    "time": 2698.196870326996,
+    "postprocess": "none",
+    "embed_dim": 768,
+    "compressed_dim": null,
+    "num_params_M": 252.0,
+    "batch_size": 8
+  }
+}