AsyiraFitri commited on
Commit
4e1a4e7
·
1 Parent(s): 4dd03df

fix the date naming:)

Browse files
Updates/daily_updates/21042026-DailyUpdate.md ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ **Daily Update — Bad Case Analysis**
2
+ *Image embedding benchmark · Tobacco-3482*
3
+
4
+ ---
5
+
6
+ ## Completed
7
+
8
+ - Fixed a result logging issue where post-processed results were overwriting the raw embedding results in the checkpoint. Separated raw and post-processed entries so both are retained independently for comparison.
9
+ - Ran bad case analysis for both SigLIP2 SO400M and ModernVBERT Bi using top-1 kNN criterion, loading embeddings directly from cached `.npy` files.
10
+ - Generated per-class error breakdowns for all 10 document classes across both models.
11
+ - Sampled misclassified cases per class and retrieved top-10 nearest neighbours for each query for visual inspection.
12
+ - Saved full error breakdown to `results/bad_case_analysis.json` and bad case report to `.txt`.
13
+ - Created retrieval confusion heatmap and retrieval error rate summary bar chart saved to `figures\bad_cases`
14
+ - Visualised sampled bad cases with top-10 retrieval results per query, saved to `.png` at `figures\bad_cases`.
15
+
16
+ ---
17
+
18
+ ## In Progress
19
+
20
+ - Improving the visual retrieval report resolution or switching to a more readable format (HTML or PDF) for better presentation — planned for tomorrow.
21
+ - Writing up a bad case summary covering the key failure patterns identified and potential approaches to improve embedding quality for the affected classes.
22
+
23
+ ---
24
+
25
+ ## Blockers
26
+
27
+ - The visualised bad cases is non-readable.
28
+
29
+ ---
30
+
31
+ ## Key Learnings
32
+
33
+ - **Per-class errors reveal more than aggregate accuracy:** SigLIP2 and ModernVBERT Bi are only ~2.6 percentage points apart in overall error rate but ModernVBERT Bi is dramatically worse on Report specifically (70.4% vs 44.4%), which aggregate accuracy alone hides. Full breakdown in the bad case report.
34
+ - **Systematic vs random errors:** The Memo/Letter confusion is consistent across both models and almost all Memo error cases, pointing to a structural embedding space overlap rather than noise — a more actionable signal than scattered errors.
35
+ - **Low distance does not guarantee correct retrieval:** Some Scientific and Note cases in SigLIP2 show very low cosine distances to wrong-class neighbours, meaning the model is highly confident but wrong — suggesting genuine class overlap in embedding space rather than general uncertainty.