Updates/Week_06_Update.md

Week 06 Update — Embedding Fusion, Bad Case Analysis, Post-Processing & Fine-tuning

Image embedding benchmark · Tobacco-3482 · 20 Apr 2026 – 24 Apr 2026

Summary

This week covered embedding fusion experiments, bad case analysis and reporting, post-processing and pooling experiments, SupCon fine-tuning, and hard-negative mining — culminating in a new best result of 92.68% kNN@1 on Tobacco-3482.

Completed

Embedding Fusion - Ran embedding fusion experiments across 7 model pairs (SigLIP, SigLIP2, DiT, DINOv2 combinations), testing raw concat (α=0.5), PCA-whitened concat, and alpha sweep at α ∈ {0.6, 0.7, 0.8}. - Results saved to results/fusion_summary.txt and results/fusion_summary.json. README updated with full fusion section. Bad Case Analysis - Fixed result logging issue where post-processed results were overwriting raw embedding results — separated into independent entries. - Ran bad case analysis for SigLIP2 SO400M and ModernVBERT Bi using top-1 kNN criterion; generated per-class error breakdowns across all 10 document classes. - Sampled misclassified cases per class with top-10 nearest neighbour retrieval for visual inspection. - Generated retrieval confusion heatmaps and error rate bar charts; saved bad case visualisations to figures/bad_cases. Bad Case Report & Initial Experiments - Improved bad case PNG visualisations for readability; added bad_cases_grids.html for grid-layout viewing. - Created figures/bad_cases/bad_case_report.html with per-label summary tables, confusion heatmaps, colour-coded error severity, root cause analysis, and a full improvement roadmap. - Benchmarked mean-centering — consistent gains across both models, notably ModernVBERT Bi kNN@5 +4.73%. - Benchmarked CLS + mean-patch pooling for SigLIP2 SO400M — kNN@1 +2.73% over CLS-only; further gains combined with centering (87.80% kNN@1, 90.53% Clf Acc). - Ran per-label kNN@1 error breakdown across 6 variants — CLS+patch+center best overall. - Swept query expansion across k ∈ {1, 3, 5, 10} — marginal gains at best; does not reliably improve retrieval on this dataset. HuggingFace Fix & SupCon Fine-tuning - Diagnosed and resolved HuggingFace push failure — root cause was bad_cases_grids.html embedding images directly. Refactored to path references and split PNG grids by row to stay within XET size limits. - Ran SupCon fine-tuning on SigLIP2 SO400M (5 epochs, batch 4, lr 1e-5, temp 0.07, 2 unfrozen blocks, 50% stratified data). Loss: 0.2981 → 0.1146. Results: kNN@1 +5.16% (85.51% → 90.67%), kNN@5 +5.02%, Clf Acc +2.73% — largest single-step gain of the week. Hard-Negative Mining & Results Consolidation - Ran SupCon fine-tuning with online hard-negative mining (5 epochs, batch 8, lr 5e-6, MultiSimilarityMiner(epsilon=0.1), 2 unfrozen blocks). Mined pairs: 927 → 270 across epochs. Results: kNN@1 92.68%, kNN@5 91.68%, Clf Acc 92.54% — best overall result, +2.01% kNN@1 over SupCon alone. - Consolidated all results (experiments A–G) into result_analysis.md, linked from README alongside bad_case_report.html. - Successfully pushed all outstanding assets to HuggingFace. - Weekly meeting held — action items carried into next week.

Best Results This Week

Variant	kNN@1	kNN@5	Clf Acc
SigLIP2 SO400M baseline (CLS)	84.36%	86.37%	88.38%
+ Mean-centering	85.51%	84.79%	88.95%
+ CLS + patch + centering	87.80%	88.95%	90.53%
+ SupCon fine-tuning	90.67%	90.53%	91.68%
+ Hard-negative mining (best)	92.68%	91.68%	92.54%

---

Carry-Over / Next Week

• Add image-level labels (filename/index, true label, predicted label, cosine distance) to bad case PNG grids for easier sample referencing.
• Benchmark LayoutLMv3 as an additional model variant and compare results against SigLIP2 and ModernVBERT Bi.

---

Key Learnings

• Embedding fusion was explored but post-processing and fine-tuning interventions on single models proved to be the higher-ROI path this week.
• Mean-centering and CLS+patch pooling are complementary — combining both consistently outperforms either alone.
• Query expansion does not help on Tobacco-3482 — the neighbourhood structure is noisy enough that averaging in neighbours introduces more error than it corrects.
• SupCon fine-tuning is the single biggest lever, with hard-negative mining adding a further consistent ~+2% on top by forcing separation of visually similar but label-distinct pairs.
• Report and Scientific remain the hardest classes across all variants — even the best model still misclassifies 35.2% of Report documents. Structural embedding space overlap between these two classes is the dominant unresolved challenge.
• Directly embedding images in HTML is not viable for HuggingFace at this dataset scale — path references and per-row PNG splitting are necessary.