Daily/Weekly Update

Updates/Week_06_Update.md

@@ -0,0 +1,64 @@
+# Week 06 Update — Embedding Fusion, Bad Case Analysis, Post-Processing & Fine-tuning
+*Image embedding benchmark · Tobacco-3482* · 20 Apr 2026 – 24 Apr 2026
+ 
+---
+ 
+## Summary
+ 
+This week covered embedding fusion experiments, bad case analysis and reporting, post-processing and pooling experiments, SupCon fine-tuning, and hard-negative mining — culminating in a new best result of 92.68% kNN@1 on Tobacco-3482.
+ 
+---
+ 
+## Completed
+ 
+**Embedding Fusion**
+- Ran embedding fusion experiments across 7 model pairs (SigLIP, SigLIP2, DiT, DINOv2 combinations), testing raw concat (α=0.5), PCA-whitened concat, and alpha sweep at α ∈ {0.6, 0.7, 0.8}.
+- Results saved to `results/fusion_summary.txt` and `results/fusion_summary.json`. README updated with full fusion section.
+**Bad Case Analysis**
+- Fixed result logging issue where post-processed results were overwriting raw embedding results — separated into independent entries.
+- Ran bad case analysis for SigLIP2 SO400M and ModernVBERT Bi using top-1 kNN criterion; generated per-class error breakdowns across all 10 document classes.
+- Sampled misclassified cases per class with top-10 nearest neighbour retrieval for visual inspection.
+- Generated retrieval confusion heatmaps and error rate bar charts; saved bad case visualisations to `figures/bad_cases`.
+**Bad Case Report & Initial Experiments**
+- Improved bad case PNG visualisations for readability; added `bad_cases_grids.html` for grid-layout viewing.
+- Created `figures/bad_cases/bad_case_report.html` with per-label summary tables, confusion heatmaps, colour-coded error severity, root cause analysis, and a full improvement roadmap.
+- Benchmarked **mean-centering** — consistent gains across both models, notably ModernVBERT Bi kNN@5 +4.73%.
+- Benchmarked **CLS + mean-patch pooling** for SigLIP2 SO400M — kNN@1 +2.73% over CLS-only; further gains combined with centering (87.80% kNN@1, 90.53% Clf Acc).
+- Ran **per-label kNN@1 error breakdown** across 6 variants — CLS+patch+center best overall.
+- Swept **query expansion** across k ∈ {1, 3, 5, 10} — marginal gains at best; does not reliably improve retrieval on this dataset.
+**HuggingFace Fix & SupCon Fine-tuning**
+- Diagnosed and resolved HuggingFace push failure — root cause was `bad_cases_grids.html` embedding images directly. Refactored to path references and split PNG grids by row to stay within XET size limits.
+- Ran **SupCon fine-tuning** on SigLIP2 SO400M (5 epochs, batch 4, lr 1e-5, temp 0.07, 2 unfrozen blocks, 50% stratified data). Loss: 0.2981 → 0.1146. Results: kNN@1 +5.16% (85.51% → 90.67%), kNN@5 +5.02%, Clf Acc +2.73% — largest single-step gain of the week.
+**Hard-Negative Mining & Results Consolidation**
+- Ran **SupCon fine-tuning with online hard-negative mining** (5 epochs, batch 8, lr 5e-6, `MultiSimilarityMiner(epsilon=0.1)`, 2 unfrozen blocks). Mined pairs: 927 → 270 across epochs. Results: kNN@1 92.68%, kNN@5 91.68%, Clf Acc 92.54% — best overall result, +2.01% kNN@1 over SupCon alone.
+- Consolidated all results (experiments A–G) into `result_analysis.md`, linked from README alongside `bad_case_report.html`.
+- Successfully pushed all outstanding assets to HuggingFace.
+- Weekly meeting held — action items carried into next week.
+---
+ 
+## Best Results This Week
+ 
+| Variant | kNN@1 | kNN@5 | Clf Acc |
+|---|---|---|---|
+| SigLIP2 SO400M baseline (CLS) | 84.36% | 86.37% | 88.38% |
+| + Mean-centering | 85.51% | 84.79% | 88.95% |
+| + CLS + patch + centering | 87.80% | 88.95% | 90.53% |
+| + SupCon fine-tuning | 90.67% | 90.53% | 91.68% |
+| **+ Hard-negative mining (best)** | **92.68%** | **91.68%** | **92.54%** |
+ 
+---
+ 
+## Carry-Over / Next Week
+ 
+- Add image-level labels (filename/index, true label, predicted label, cosine distance) to bad case PNG grids for easier sample referencing.
+- Benchmark **LayoutLMv3** as an additional model variant and compare results against SigLIP2 and ModernVBERT Bi.
+---
+ 
+## Key Learnings
+ 
+- Embedding fusion was explored but post-processing and fine-tuning interventions on single models proved to be the higher-ROI path this week.
+- Mean-centering and CLS+patch pooling are complementary — combining both consistently outperforms either alone.
+- Query expansion does not help on Tobacco-3482 — the neighbourhood structure is noisy enough that averaging in neighbours introduces more error than it corrects.
+- SupCon fine-tuning is the single biggest lever, with hard-negative mining adding a further consistent ~+2% on top by forcing separation of visually similar but label-distinct pairs.
+- Report and Scientific remain the hardest classes across all variants — even the best model still misclassifies 35.2% of Report documents. Structural embedding space overlap between these two classes is the dominant unresolved challenge.
+- Directly embedding images in HTML is not viable for HuggingFace at this dataset scale — path references and per-row PNG splitting are necessary.

Updates/daily_updates/20042026-DailyUpdate.md

@@ -1,4 +1,4 @@
-**Daily Update — Embedding Fusion Experiments**
+# Daily Update — Embedding Fusion Experiments
 *Image embedding benchmark · Tobacco-3482*
 
 ---

Updates/daily_updates/21042026-DailyUpdate.md

@@ -1,4 +1,4 @@
-**Daily Update — Bad Case Analysis**
+# Daily Update — Bad Case Analysis
 *Image embedding benchmark · Tobacco-3482*
 
 ---

Updates/daily_updates/22042026-DailyUpdate.md

@@ -1,7 +1,9 @@
-**Daily Update — Bad Case Analysis, Improvement Report & Initial Experiments**
+# Daily Update — Bad Case Analysis, Improvement Report & Initial Experiments
 *Image embedding benchmark · Tobacco-3482*
 
-**Completed**
+--- 
+
+## Completed
 
 - Improved the bad case `.png` visualisations for better readability and repushed to HuggingFace. Added `bad_cases_grids.html` for improved grid-layout readability alongside the per-case PNGs.
 - Created `figures/bad_cases/bad_case_report.html` with per-label summary tables, confusion heatmaps, retrieval error rate bar charts, colour-coded error severity (green <25%, amber 25–50%, red >50%), root cause analysis, and a full improvement roadmap across four categories (preprocessing, architecture/pooling, training objectives, post-processing/retrieval).
@@ -10,15 +12,21 @@
 - Ran **per-label kNN@1 error breakdown** across all six variants — CLS+patch+center is best overall and wins on 4 of 8 labels including Report (44.4% → 35.2%) and Note (35.6% → 25.4% tied with ModernVBERT Bi+ctr).
 - Implemented **query expansion** swept across k ∈ {1, 3, 5, 10} for all variants — marginal gains at best; CLS+patch+center with k=3 gives +0.57% kNN@1 (87.80% → 88.38%) but kNN@5 drops 2.15%. Query expansion does not reliably improve retrieval on this dataset.
 
-**In Progress**
+---
+
+## In Progress
 
 - Consolidating results — to be pushed to HuggingFace tomorrow.
 
-**Blockers**
+---
+
+## Blockers
 
 - Couldn't sync commit to huggingface
 
-**Key Learnings**
+--- 
+
+## Key Learnings
 
 - Centering consistently helps both models but ModernVBERT Bi benefits more than SigLIP2, suggesting its embedding space is less well-centred out of the box.
 - CLS+patch pooling trades classifier accuracy for kNN@1 accuracy on its own, but combining both with centering gives the best results across all three metrics — the two post-processing steps are complementary.

Updates/daily_updates/23042026-DailyUpdate.md

@@ -0,0 +1,32 @@
+# Daily Update — HuggingFace Push Fix, File Size Optimisation & SupCon Fine-tuning
+*Image embedding benchmark · Tobacco-3482*
+ 
+---
+ 
+## Completed
+ 
+- Diagnosed and resolved the HuggingFace push failure from 22/04. Root cause: `bad_cases_grids.html` was loading images directly, resulting in a file size too large to sync. Some of the per-case PNG grids were also too large to be rendered on HuggingFace's XET storage.
+- Refactored `bad_cases_grids.html` to reference image paths instead of embedding them inline, significantly reducing file size.
+- Split the per-case PNG grids by row so individual files stay within HuggingFace's renderable size limits while preserving readability.
+- Conducted trial-and-error optimisation across both file types, balancing compression against visual clarity before re-pushing assets.
+- Ran **SupCon fine-tuning** on SigLIP2 SO400M (5 epochs, batch size 4, lr 1e-5, temp 0.07, 2 unfrozen blocks, 50% of training data stratified). Loss: 0.2981 → 0.1146. Results: kNN@1 +5.16% (85.51% → 90.67%), kNN@5 +5.02%, Clf Acc +2.73% — the largest single-step gain across all experiments to date.
+
+---
+ 
+## In Progress
+ 
+- Hard-negative mining fine-tuning experiment scheduled for next day.
+
+---
+
+## Blockers
+
+- ...
+
+---
+ 
+## Key Learnings
+ 
+- Directly embedding images in HTML is not viable for HuggingFace hosting at this scale — path references are necessary. File size constraints also apply to PNGs, requiring per-row splitting rather than full grids.
+- SupCon fine-tuning is the highest-ROI intervention so far — even with only 50% of training data and 5 epochs, it substantially tightens within-class clusters and separates the Report/Scientific confusion pair.
+ 

Updates/daily_updates/24042026-DailyUpdate.md

@@ -0,0 +1,29 @@
+# Daily Update — Hard-Negative Mining, Results Consolidation & LayoutLMv3 Kickoff
+*Image embedding benchmark · Tobacco-3482*
+ 
+---
+ 
+## Completed
+ 
+- Ran **SupCon fine-tuning with online hard-negative mining** on SigLIP2 SO400M (5 epochs, batch size 8, lr 5e-6, temp 0.07, `MultiSimilarityMiner(epsilon=0.1)`, 2 unfrozen blocks). Mined pairs per epoch: 927 → 635 → 471 → 327 → 270 (declining count confirms the model is actively separating confused pairs). Results: kNN@1 92.68%, kNN@5 91.68%, Clf Acc 92.54% — best overall result across all experiments, +2.01% kNN@1 over SupCon alone.
+- Consolidated all experiment results (A through G) into `result_analysis.md` and linked it from the README as the canonical results reference, alongside the existing `bad_case_report.html`.
+- Successfully pushed all outstanding assets to HuggingFace (previously blocked).
+- Weekly meeting held. Action items identified (see In Progress).
+---
+ 
+## In Progress
+ 
+- Add image-level labels (filename/index, true label, predicted label, cosine distance) to bad case PNG grids for easier sample referencing.
+- Benchmark **LayoutLMv3** as an additional model variant and compare results against SigLIP2 and ModernVBERT Bi — to be ready for next week's review.
+
+## Blockers
+
+- ...
+
+---
+ 
+## Key Learnings
+ 
+- Hard-negative mining adds a consistent ~+2% on top of vanilla SupCon. Declining mined pair count across epochs confirms the model is actively separating previously-confused pairs.
+- Report and Scientific remain the hardest classes even after fine-tuning (best variant still at 35.2% Report error), but narrowed significantly from the 44.4% baseline.
+- Maintaining results in both `.md` and `.html` formats is worth the duplication — `.md` is easier to link and diff on HuggingFace, while `.html` retains the visual heatmaps and bar charts.

tabacco3482_benchmarks/figures/bad_cases/result_analysis.md

@@ -132,7 +132,7 @@ Resume is the only class with perfect retrieval. The two-column structured layou
 
 Query image (orange border) + top-10 retrieved neighbours. Green border = correct label, Red border = wrong label.
 
-[SigLIP2 SO400M — Bad Case Grids](bad_case_grids.html)
+[Bad Case Grids Visualisations](bad_case_grids.html)
 
 ---