update readme and added result analysis.md

README.md

@@ -148,7 +148,15 @@ Three strategies tested per pair:
 
 ![Fusion Results](tabacco3482_benchmarks/figures/fusion_results.png)
 
-Results saved to [`fusion_summary.txt`](results/fusion_summary.txt) and [`fusion_summary.json`](results/fusion_summary.json) on completion.
+Results saved to [`fusion_summary.txt`](tabacco3482_benchmarks/results/fusion_summary.txt) and [`fusion_summary.json`](results/fusion_summary.json) on completion.
+
+---
+
+## Bad Cases Analysis
+
+Per-label error rates, retrieval confusion heatmaps, top confusion pairs, failure analysis, and improvement experiments for SigLIP2 SO400M and ModernVBERT Bi.
+
+→ [`result_analysis_tobacco3482.md`](tabacco3482_benchmarks/figures/bad_cases/result_analysis_tobacco3482.md)
 
 ---
 

tabacco3482_benchmarks/figures/bad_cases/result_analysis.md

@@ -0,0 +1,347 @@
+# Result Analysis — Tobacco-3482 Embedding Benchmark
+
+**Dataset:** `maveriq/tobacco3482` — 10-class document image classification, 3,482 images  
+**Models evaluated:** SigLIP2 SO400M · ModernVBERT Bi  
+**Metrics:** Logistic Regression Accuracy (Clf Acc) · kNN@1 · kNN@5  
+**Split:** 80/20 train/test, seed=42
+
+---
+
+## 1. Baseline Results
+
+Raw pretrained embeddings, no post-processing.
+
+| Model | Clf Acc | kNN@1 | kNN@5 |
+|---|---|---|---|
+| SigLIP2 SO400M (CLS) | 88.38% | 84.36% | 86.37% |
+| ModernVBERT Bi | 84.79% | 81.78% | 80.20% |
+
+SigLIP2 leads on all metrics at baseline. ModernVBERT's lower kNN@5 vs kNN@1 suggests its embedding space has less consistent neighbourhood structure.
+
+---
+
+## 2. Bad Case Analysis
+
+### Definition
+
+A bad case is a test image whose top-1 nearest neighbour by cosine distance has a different label. A small distance (e.g. 0.05) at error time is particularly concerning — the model was confident but wrong.
+
+### Per-Label Error Rates (kNN@1 baseline)
+
+#### Summary Bar Chart
+
+![Summary Bar Chart](summary_bar_chart.png)
+
+**SigLIP2 SO400M**
+
+| Label | Bad Cases | Total | Error % | Top Confusion |
+|---|---|---|---|---|
+| ADVE | 5 | 46 | 10.9% | News x2, Memo x1, Report x1 |
+| Email | 10 | 115 | 8.7% | Letter x4, Memo x3, Note x2 |
+| Form | 10 | 79 | 12.7% | Letter x3, Report x3, Note x2 |
+| Letter | 13 | 114 | 11.4% | Memo x8, Form x4, Scientific x1 |
+| Memo | 12 | 130 | 9.2% | Letter x6, Report x2, Email x1 |
+| News | 3 | 44 | 6.8% | ADVE x2, Scientific x1 |
+| Note | 11 | 32 | **34.4%** | Letter x3, Email x3, News x3 |
+| Report | 24 | 54 | **44.4%** | Scientific x10, Letter x9, ADVE x2 |
+| Resume | 0 | 24 | 0.0% | — |
+| Scientific | 21 | 59 | **35.6%** | Report x9, Form x5, Memo x4 |
+
+**ModernVBERT Bi**
+
+| Label | Bad Cases | Total | Error % | Top Confusion |
+|---|---|---|---|---|
+| ADVE | 3 | 46 | 6.5% | Letter x1, Form x1, News x1 |
+| Email | 10 | 115 | 8.7% | Memo x4, Letter x4, Note x1 |
+| Form | 7 | 79 | 8.9% | Scientific x3, Letter x2, ADVE x1 |
+| Letter | 14 | 114 | 12.3% | Memo x5, Report x3, Scientific x2 |
+| Memo | 15 | 130 | 11.5% | Letter x11, Form x1, News x1 |
+| News | 8 | 44 | 18.2% | ADVE x3, Letter x3, Scientific x2 |
+| Note | 11 | 32 | **34.4%** | Letter x5, Email x3, News x1 |
+| Report | 38 | 54 | **70.4%** | Letter x12, Scientific x7, News x6 |
+| Resume | 0 | 24 | 0.0% | — |
+| Scientific | 21 | 59 | **35.6%** | Report x8, Letter x6, News x4 |
+
+### Retrieval Confusion Heatmaps
+
+Row = true label, Column = predicted top-1 label. Diagonal = correct retrievals.
+
+**SigLIP2 SO400M**
+
+![Confusion Heatmap SigLIP2 SO400M](confusion_heatmap_SigLIP2_SO400M.png)
+
+**ModernVBERT Bi**
+
+![Confusion Heatmap ModernVBERT Bi](confusion_heatmap_ModernVBERT_Bi.png)
+
+### Top Confusion Pairs
+
+**SigLIP2 SO400M**
+
+| True Label | Predicted As | Count | Rate |
+|---|---|---|---|
+| Report | Scientific | 10 | 18.5% |
+| Report | Letter | 9 | 16.7% |
+| Scientific | Report | 9 | 15.3% |
+| Letter | Memo | 8 | 7.0% |
+| Memo | Letter | 6 | 4.6% |
+| Scientific | Form | 5 | 8.5% |
+| Letter | Form | 4 | 3.5% |
+| Email | Letter | 4 | 3.5% |
+
+**ModernVBERT Bi**
+
+| True Label | Predicted As | Count | Rate |
+|---|---|---|---|
+| Report | Letter | 12 | 22.2% |
+| Memo | Letter | 11 | 8.5% |
+| Scientific | Report | 8 | 13.6% |
+| Report | Scientific | 7 | 13.0% |
+| Report | News | 6 | 11.1% |
+| Report | Memo | 6 | 11.1% |
+| Scientific | Letter | 6 | 10.2% |
+| Letter | Memo | 5 | 4.4% |
+
+### Why It Fails
+
+**Report — SigLIP2 44.4% / ModernVBERT 70.4%**
+
+Report is the single worst class for both models. Tobacco-era reports share nearly all visual attributes with scientific documents — dense multi-column text, inline figures, data tables, and no strong layout signal in any single region. The confusion is bidirectional: reports get predicted as scientific papers and vice versa, meaning the two classes sit in overlapping regions of the embedding space rather than one being a subset of the other. ModernVBERT's 70.4% error is particularly severe — it lacks the high-resolution patch encoding that SigLIP2 uses, so fine layout differences (e.g. column width, figure placement) are lost entirely.
+
+**Scientific — SigLIP2 35.6% / ModernVBERT 35.6%**
+
+Scientific documents suffer from the same Report↔Scientific overlap described above, plus a secondary confusion into Form (5 cases in SigLIP2). Some scientific documents in this dataset are structured like tables or filled forms — data-heavy pages with grid layouts — which pushes their embedding closer to Form than to typical prose-heavy scientific papers.
+
+**Note — SigLIP2 34.4% / ModernVBERT 34.4%**
+
+Note is the smallest class by far (32 test samples) and is visually heterogeneous — handwritten sticky notes, informal typed memos, and margin annotations all carry the same label. The model cannot learn a tight cluster for a class that has no consistent visual signature. The errors scatter widely into Letter, Email, and News rather than concentrating on one target, which confirms the within-class variance is the root cause rather than a specific visual overlap with a single other class.
+
+**Letter / Memo — mutual confusion**
+
+Letter and Memo are visually nearly identical in this dataset: both are single-column typed text on plain paper with a salutation, date, and body. The only discriminating cues are semantic (subject lines, signatures, formal language) rather than visual. SigLIP2 misclassifies 8 letters as memos and 6 memos as letters. ModernVBERT is even more affected — 11 memos are predicted as letters — suggesting its image-token pooling captures less of the fine header-region detail that would distinguish these classes.
+
+**News — ModernVBERT 18.2% vs SigLIP2 6.8%**
+
+News pages are generally distinctive (multi-column layout, large headlines, photographs) and SigLIP2 handles them well. ModernVBERT's higher error rate and the centering-induced degradation to 27.3% points to a representation issue specific to how ModernVBERT encodes layout structure: the headline and photograph zones that make news visually unique may not be well-captured by image-token mean-pooling, making news embeddings drift toward ADVE and Letter after mean-centering shifts the embedding space.
+
+**Resume — 0% error in both models**
+
+Resume is the only class with perfect retrieval. The two-column structured layout with section headers (Education, Experience, Skills) is visually unique in this dataset and produces a tight, well-separated cluster for both models without any fine-tuning.
+
+### Bad Case Grids
+
+Query image (orange border) + top-10 retrieved neighbours. Green border = correct label, Red border = wrong label.
+
+[SigLIP2 SO400M — Bad Case Grids](bad_case_grids.html)
+
+---
+
+## 3. Recommendations
+
+### A — Input / Pre-processing
+
+**Tiled high-res inference (SigLIP2)** — Divide each page into overlapping 384px crops and average-pool patch embeddings. Preserves fine text detail lost at global resize. Critical for Report vs Scientific where layout cues are fine-grained.
+
+**Document-aware augmentation** — RandomPerspective +-3 degrees (scan skew), elastic distortion (paper warp), random text-block crop-and-resize, random binarisation. Improves robustness to real-world scan variation.
+
+**Grayscale-aware channel normalisation** — Recalculate per-channel mean/std across the three RGB channels. Prevents wasted model capacity on redundant channels from the greyscale→RGB conversion.
+
+### B — Architecture / Pooling
+
+**Ensemble SigLIP2 + ModernVBERT score fusion** — Average (or apply per-class weights to) both model scores. The two models have complementary confusion patterns — fusion improves Report/Scientific without any retraining cost.
+
+**Multi-scale pooling (SigLIP2)** — Concatenate CLS token + mean of last-layer patch tokens to add spatial layout signal (already validated in experiment B, +2.73% kNN@1).
+
+**Layout-zone pooling** — Split each image into fixed header/body/footer zones and concatenate zone-level mean embeddings. Directly targets Letter vs Memo confusion where the header zone is the discriminating region.
+
+**OCR-text fusion (ModernVBERT Bi)** — Pass extracted OCR text as language input alongside image patch tokens. Salutations, subject lines, and dates discriminate Letter/Memo/Email/Note, which are visually indistinguishable.
+
+### C — Training Objective
+
+**Domain fine-tuning with SupCon loss** *(validated — experiment E)* — Even 5–10 epochs tightens within-class clusters for confusable pairs. Highest ROI intervention.
+
+**Hard-negative mining (Report↔Scientific priority)** *(validated — experiment G, best overall)* — Online mining builds mini-batches where negatives are visually similar but wrong class, forcing the model to separate currently-overlapping clusters.
+
+**Oversample the Note class** — Note has only 32 test samples — far smaller than any other class — yet 11 bad cases. Oversample or synthesise Note examples and apply stronger augmentation specifically to this class before fine-tuning.
+
+**Auxiliary OCR bag-of-words loss** — Jointly predict token distribution from the embedding to force the model to encode lexical cues that pure vision models ignore. Addresses Letter/Memo/Note confusion without a full multimodal architecture.
+
+### D — Post-processing / Retrieval (zero training cost)
+
+**Query Expansion** — Replace query embedding with mean(query, top-3 neighbours). Smooths outlier queries. Fastest available win with no model changes (+0.43–0.57% kNN@1 on centered variants).
+
+**Two-stage classifier for Report↔Scientific** *(validated — experiments D2, F)* — SVC RBF second-pass classifier for this specific pair gives +0.86% Acc / +0.96% F1 for SigLIP2.
+
+**Cross-encoder re-ranking** — Fine-tuned DiT re-scores top-20 kNN candidates. Especially valuable for Report/Scientific where initial retrieval is close but a re-ranker can use richer pairwise features.
+
+**DiT-weighted fusion** — Upweight DiT for Letter/Memo/Form pairs (DiT pre-trained on 42M scanned pages is especially strong on formal document layouts).
+
+### Priority Roadmap
+
+| Timeline | Actions |
+|---|---|
+| **Short-term** (no training) | Ensemble score fusion · Query Expansion · Two-stage Report↔Scientific classifier |
+| **Medium-term** | SupCon fine-tune + hard-negative mining · Note class oversampling · Layout-zone pooling |
+| **Long-term** | OCR-text fusion · Cross-encoder re-ranking · Tiled high-res inference |
+
+---
+
+## 4. Experiments
+
+### A — Mean-Centering
+
+Subtract training-set mean from all embeddings, then re-normalise. Zero training cost.
+
+| Model | Metric | Baseline | Result | Delta |
+|---|---|---|---|---|
+| SigLIP2 SO400M | Clf Acc | 88.38% | 88.95% | +0.57% |
+| SigLIP2 SO400M | kNN@1 | 84.36% | 85.51% | +1.15% |
+| SigLIP2 SO400M | kNN@5 | 86.37% | 84.79% | -1.58% |
+| ModernVBERT Bi | Clf Acc | 84.79% | 85.65% | +0.86% |
+| ModernVBERT Bi | kNN@1 | 81.78% | 83.36% | +1.58% |
+| ModernVBERT Bi | kNN@5 | 80.20% | 83.79% | +3.59% |
+
+Centering consistently improves kNN@1 and Clf Acc. The one regression (SigLIP2 kNN@5) is minor. Especially impactful for ModernVBERT — a significant +3.59% on kNN@5 suggests the raw embeddings have a large mean bias.
+
+---
+
+### B — CLS + Patch Pooling (SigLIP2)
+
+Concatenate the CLS token with the mean of all last-layer patch tokens to add spatial layout signal.
+
+| Pooling | Clf Acc | kNN@1 | kNN@5 |
+|---|---|---|---|
+| CLS only (baseline) | 88.38% | 84.36% | 86.37% |
+| CLS + mean-patch | 86.94% | 87.09% | 87.52% |
+| CLS + patch + centering | 90.53% | 87.80% | 88.95% |
+
+**Deltas vs CLS baseline (CLS+patch):** Clf Acc -1.43% · kNN@1 +2.73% · kNN@5 +1.15%
+
+The pure CLS+patch combination trades linear probe accuracy for substantially better kNN. Adding centering on top recovers the clf loss and pushes Clf Acc to 90.53% — the best non-fine-tuned result for SigLIP2.
+
+---
+
+### C — Per-Label Error Breakdown (kNN@1)
+
+Error rates for labels with >=10% error in at least one variant. Bold = best per row.
+
+| Label | SigLIP2 CLS | SigLIP2 CLS+ctr | SigLIP2 CLS+patch | SigLIP2 CLS+patch+ctr | ModernVBERT Bi | ModernVBERT Bi+ctr |
+|---|---|---|---|---|---|---|
+| ADVE | 10.9% | 6.5% | 6.5% | **2.2%** | 6.5% | 4.3% |
+| Email | 8.7% | 11.3% | **4.3%** | 7.0% | 8.7% | 7.8% |
+| Form | 12.7% | 11.4% | **8.9%** | **8.9%** | **8.9%** | **8.9%** |
+| Letter | **11.4%** | 13.2% | **11.4%** | 13.2% | 12.3% | 14.9% |
+| Memo | 9.2% | **6.9%** | **6.9%** | **6.9%** | 11.5% | 10.0% |
+| News | **6.8%** | 11.4% | 11.4% | 9.1% | 18.2% | 27.3% |
+| Note | 34.4% | 21.9% | 21.9% | **15.6%** | 34.4% | 34.4% |
+| Report | 44.4% | 44.4% | 48.1% | **35.2%** | 70.4% | 55.6% |
+| Scientific | 35.6% | 27.1% | **25.4%** | 28.8% | 35.6% | **25.4%** |
+
+Key observations:
+
+- **Report** is the hardest class across the board — even the best variant (SigLIP2 CLS+patch+ctr) misclassifies 35.2% of reports.
+- **Scientific** and **Note** are the other persistent pain points. Scientific / Report mutual confusion is the dominant error pattern.
+- ModernVBERT Bi's Report error at 70.4% is severe — the model essentially cannot distinguish reports from other formal document types.
+- **News** is an interesting reversal: SigLIP2 CLS has only 6.8% error but ModernVBERT Bi+ctr reaches 27.3% — centering appears to degrade News for ModernVBERT specifically.
+
+---
+
+### D — Query Expansion Sweep
+
+Replace each test query with `mean(query, top-n neighbours)` before kNN evaluation. Sweep n in {1, 3, 5, 10}.
+
+| Variant | Base@1 | QE@1 | Delta@1 | Base@5 | QE@5 | Delta@5 | Best n |
+|---|---|---|---|---|---|---|---|
+| SigLIP2 CLS | 84.36% | 84.36% | 0.00% | 86.37% | 83.21% | -3.16% | 1 |
+| SigLIP2 CLS+center | 85.51% | 85.94% | +0.43% | 84.79% | 84.07% | -0.72% | 3 |
+| ModernVBERT Bi | 81.78% | 81.78% | 0.00% | 80.20% | 81.49% | +1.29% | 1 |
+| ModernVBERT Bi+center | 83.36% | 83.36% | 0.00% | 83.79% | 85.08% | +1.29% | 1 |
+| SigLIP2 CLS+patch | 87.09% | 87.09% | 0.00% | 87.52% | 85.65% | -1.87% | 1 |
+| SigLIP2 CLS+patch+center | 87.80% | 88.38% | +0.57% | 88.95% | 87.66% | -1.29% | 3 |
+
+Query expansion yields modest gains on kNN@1 for already-centered embeddings but tends to hurt kNN@5. Not a high-ROI intervention on its own for this dataset.
+
+---
+
+### D2 — Two-Stage Report↔Scientific Classifier
+
+Stage 1: standard kNN top-1. Stage 2: when stage 1 predicts Report or Scientific, override with a dedicated binary classifier.
+
+**SVC RBF kernel (experiment D2) vs Logistic Regression (experiment F):**
+
+| Model | Classifier | Base Acc | Stage2 Acc | Delta Acc | Delta F1 | Overrides |
+|---|---|---|---|---|---|---|
+| SigLIP2 SO400M | LogReg | 84.36% | 85.08% | +0.72% | +0.80% | 97 |
+| SigLIP2 SO400M | SVC RBF | 84.36% | 85.22% | +0.86% | +0.96% | 97 |
+| ModernVBERT Bi | LogReg | 81.78% | 82.07% | +0.29% | +0.13% | 80 |
+| ModernVBERT Bi | SVC RBF | 81.78% | 82.64% | +0.86% | +0.92% | 80 |
+
+Per-class improvement (SVC RBF):
+
+| Model | Class | Stage 1 | Stage 2 | Delta |
+|---|---|---|---|---|
+| SigLIP2 | Report | 30/54 (55.6%) | 34/54 (63.0%) | +4 |
+| SigLIP2 | Scientific | 38/59 (64.4%) | 40/59 (67.8%) | +2 |
+| ModernVBERT | Report | 16/54 (29.6%) | 17/54 (31.5%) | +1 |
+| ModernVBERT | Scientific | 38/59 (64.4%) | 43/59 (72.9%) | +5 |
+
+SVC RBF consistently outperforms LogReg for this binary task. Even so, gains are modest — the underlying Report embedding problem is too large to fix with a lightweight second stage alone.
+
+---
+
+### E — SupCon Fine-tuning
+
+Fine-tune SigLIP2 SO400M with Supervised Contrastive loss on 50% of training data (stratified).
+
+**Config:** 5 epochs · batch size 4 · lr 1e-5 · temp 0.07 · 2 unfrozen blocks  
+**Loss:** 0.2981 → 0.1817 → 0.2000 → 0.1627 → 0.1146
+
+| Variant | kNN@1 | kNN@5 | Clf Acc |
+|---|---|---|---|
+| Baseline (CLS+center) | 85.51% | 85.51% | 88.95% |
+| Fine-tuned (SupCon 50% FT) | 90.67% | 90.53% | 91.68% |
+| **Delta** | **+5.16%** | **+5.02%** | **+2.73%** |
+
+The largest single-step improvement of any experiment. SupCon training substantially tightens within-class clusters and separates confusable pairs — particularly impactful for Report/Scientific given the contrastive objective.
+
+---
+
+### G — Hard-Negative Mining (Best Overall)
+
+SupCon fine-tuning with online hard-negative mining, prioritising Report↔Scientific pairs.
+
+**Config:** 5 epochs · batch size 8 · lr 5e-6 · temp 0.07 · `MultiSimilarityMiner(epsilon=0.1)` · 2 unfrozen blocks · pretrained weights  
+**Loss:** 1.1411 → 0.8039 → 0.6990 → 0.4402 → 0.3783  
+**Mined pairs per epoch:** 927 → 635 → 471 → 327 → 270 *(declining = model improving)*
+
+| Variant | kNN@1 | kNN@5 | Clf Acc |
+|---|---|---|---|
+| Baseline (CLS+center) | 85.51% | 85.51% | 88.95% |
+| SupCon FT (E) | 90.67% | 90.53% | 91.68% |
+| **SupCon + Hard-Neg Mining (G)** | **92.68%** | **91.68%** | **92.54%** |
+| Delta vs baseline | +7.17% | +6.17% | +3.59% |
+| Delta vs SupCon (E) | +2.01% | +1.15% | +0.86% |
+
+Hard-negative mining adds a consistent +2% on top of SupCon alone. The declining mined pair count across epochs confirms the model is actively separating previously-confused pairs during training.
+
+---
+
+## 5. Best Results Summary
+
+### SigLIP2 SO400M
+
+| Metric | Best Score | Source |
+|---|---|---|
+| Clf Acc | **92.54%** | G — Hard-Neg Mining |
+| kNN@1 | **92.68%** | G — Hard-Neg Mining |
+| kNN@5 | **91.68%** | G — Hard-Neg Mining |
+
+### ModernVBERT Bi
+
+| Metric | Best Score | Source |
+|---|---|---|
+| Clf Acc | **85.65%** | A — Mean-Centering |
+| kNN@1 | **83.36%** | A — Mean-Centering |
+| kNN@5 | **85.08%** | D — Query Expansion (ModernVBERT Bi+center) |
+
+