# Week 06 Update — Embedding Fusion, Bad Case Analysis, Post-Processing & Fine-tuning
*Image embedding benchmark · Tobacco-3482* · 20 Apr 2026 – 24 Apr 2026
 
---
 
## Summary
 
This week covered embedding fusion experiments, bad case analysis and reporting, post-processing and pooling experiments, SupCon fine-tuning, and hard-negative mining — culminating in a new best result of 92.68% kNN@1 on Tobacco-3482.
 
---
 
## Completed
 
**Embedding Fusion**
- Ran embedding fusion experiments across 7 model pairs (SigLIP, SigLIP2, DiT, DINOv2 combinations), testing raw concat (α=0.5), PCA-whitened concat, and alpha sweep at α ∈ {0.6, 0.7, 0.8}.
- Results saved to `results/fusion_summary.txt` and `results/fusion_summary.json`. README updated with full fusion section.
**Bad Case Analysis**
- Fixed result logging issue where post-processed results were overwriting raw embedding results — separated into independent entries.
- Ran bad case analysis for SigLIP2 SO400M and ModernVBERT Bi using top-1 kNN criterion; generated per-class error breakdowns across all 10 document classes.
- Sampled misclassified cases per class with top-10 nearest neighbour retrieval for visual inspection.
- Generated retrieval confusion heatmaps and error rate bar charts; saved bad case visualisations to `figures/bad_cases`.
**Bad Case Report & Initial Experiments**
- Improved bad case PNG visualisations for readability; added `bad_cases_grids.html` for grid-layout viewing.
- Created `figures/bad_cases/bad_case_report.html` with per-label summary tables, confusion heatmaps, colour-coded error severity, root cause analysis, and a full improvement roadmap.
- Benchmarked **mean-centering** — consistent gains across both models, notably ModernVBERT Bi kNN@5 +4.73%.
- Benchmarked **CLS + mean-patch pooling** for SigLIP2 SO400M — kNN@1 +2.73% over CLS-only; further gains combined with centering (87.80% kNN@1, 90.53% Clf Acc).
- Ran **per-label kNN@1 error breakdown** across 6 variants — CLS+patch+center best overall.
- Swept **query expansion** across k ∈ {1, 3, 5, 10} — marginal gains at best; does not reliably improve retrieval on this dataset.
**HuggingFace Fix & SupCon Fine-tuning**
- Diagnosed and resolved HuggingFace push failure — root cause was `bad_cases_grids.html` embedding images directly. Refactored to path references and split PNG grids by row to stay within XET size limits.
- Ran **SupCon fine-tuning** on SigLIP2 SO400M (5 epochs, batch 4, lr 1e-5, temp 0.07, 2 unfrozen blocks, 50% stratified data). Loss: 0.2981 → 0.1146. Results: kNN@1 +5.16% (85.51% → 90.67%), kNN@5 +5.02%, Clf Acc +2.73% — largest single-step gain of the week.
**Hard-Negative Mining & Results Consolidation**
- Ran **SupCon fine-tuning with online hard-negative mining** (5 epochs, batch 8, lr 5e-6, `MultiSimilarityMiner(epsilon=0.1)`, 2 unfrozen blocks). Mined pairs: 927 → 270 across epochs. Results: kNN@1 92.68%, kNN@5 91.68%, Clf Acc 92.54% — best overall result, +2.01% kNN@1 over SupCon alone.
- Consolidated all results (experiments A–G) into `result_analysis.md`, linked from README alongside `bad_case_report.html`.
- Successfully pushed all outstanding assets to HuggingFace.
- Weekly meeting held — action items carried into next week.
---
 
## Best Results This Week
 
| Variant | kNN@1 | kNN@5 | Clf Acc |
|---|---|---|---|
| SigLIP2 SO400M baseline (CLS) | 84.36% | 86.37% | 88.38% |
| + Mean-centering | 85.51% | 84.79% | 88.95% |
| + CLS + patch + centering | 87.80% | 88.95% | 90.53% |
| + SupCon fine-tuning | 90.67% | 90.53% | 91.68% |
| **+ Hard-negative mining (best)** | **92.68%** | **91.68%** | **92.54%** |
 
---
 
## Carry-Over / Next Week
 
- Add image-level labels (filename/index, true label, predicted label, cosine distance) to bad case PNG grids for easier sample referencing.
- Benchmark **LayoutLMv3** as an additional model variant and compare results against SigLIP2 and ModernVBERT Bi.
---
 
## Key Learnings
 
- Embedding fusion was explored but post-processing and fine-tuning interventions on single models proved to be the higher-ROI path this week.
- Mean-centering and CLS+patch pooling are complementary — combining both consistently outperforms either alone.
- Query expansion does not help on Tobacco-3482 — the neighbourhood structure is noisy enough that averaging in neighbours introduces more error than it corrects.
- SupCon fine-tuning is the single biggest lever, with hard-negative mining adding a further consistent ~+2% on top by forcing separation of visually similar but label-distinct pairs.
- Report and Scientific remain the hardest classes across all variants — even the best model still misclassifies 35.2% of Report documents. Structural embedding space overlap between these two classes is the dominant unresolved challenge.
- Directly embedding images in HTML is not viable for HuggingFace at this dataset scale — path references and per-row PNG splitting are necessary.