...

Updates/daily_updates/210042026-DailyUpdate.md

@@ -1,35 +0,0 @@
-**Daily Update — Bad Case Analysis**
-*Image embedding benchmark · Tobacco-3482*
-
----
-
-## Completed
-
-- Fixed a result logging issue where post-processed results were overwriting the raw embedding results in the checkpoint. Separated raw and post-processed entries so both are retained independently for comparison.
-- Ran bad case analysis for both SigLIP2 SO400M and ModernVBERT Bi using top-1 kNN criterion, loading embeddings directly from cached `.npy` files.
-- Generated per-class error breakdowns for all 10 document classes across both models.
-- Sampled misclassified cases per class and retrieved top-10 nearest neighbours for each query for visual inspection.
-- Saved full error breakdown to `results/bad_case_analysis.json` and bad case report to `.txt`.
-- Created retrieval confusion heatmap and retrieval error rate summary bar chart saved to `figures\bad_cases`
-- Visualised sampled bad cases with top-10 retrieval results per query, saved to `.png` at `figures\bad_cases`.
-
----
-
-## In Progress
-
-- Improving the visual retrieval report resolution or switching to a more readable format (HTML or PDF) for better presentation — planned for tomorrow.
-- Writing up a bad case summary covering the key failure patterns identified and potential approaches to improve embedding quality for the affected classes.
-
----
-
-## Blockers
-
-- The visualised bad cases is non-readable.
-
----
-
-## Key Learnings
-
-- **Per-class errors reveal more than aggregate accuracy:** SigLIP2 and ModernVBERT Bi are only ~2.6 percentage points apart in overall error rate but ModernVBERT Bi is dramatically worse on Report specifically (70.4% vs 44.4%), which aggregate accuracy alone hides. Full breakdown in the bad case report.
-- **Systematic vs random errors:** The Memo/Letter confusion is consistent across both models and almost all Memo error cases, pointing to a structural embedding space overlap rather than noise — a more actionable signal than scattered errors.
-- **Low distance does not guarantee correct retrieval:** Some Scientific and Note cases in SigLIP2 show very low cosine distances to wrong-class neighbours, meaning the model is highly confident but wrong — suggesting genuine class overlap in embedding space rather than general uncertainty.