AsyiraFitri commited on
Commit ·
4e1a4e7
1
Parent(s): 4dd03df
fix the date naming:)
Browse files
Updates/daily_updates/21042026-DailyUpdate.md
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
**Daily Update — Bad Case Analysis**
|
| 2 |
+
*Image embedding benchmark · Tobacco-3482*
|
| 3 |
+
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
## Completed
|
| 7 |
+
|
| 8 |
+
- Fixed a result logging issue where post-processed results were overwriting the raw embedding results in the checkpoint. Separated raw and post-processed entries so both are retained independently for comparison.
|
| 9 |
+
- Ran bad case analysis for both SigLIP2 SO400M and ModernVBERT Bi using top-1 kNN criterion, loading embeddings directly from cached `.npy` files.
|
| 10 |
+
- Generated per-class error breakdowns for all 10 document classes across both models.
|
| 11 |
+
- Sampled misclassified cases per class and retrieved top-10 nearest neighbours for each query for visual inspection.
|
| 12 |
+
- Saved full error breakdown to `results/bad_case_analysis.json` and bad case report to `.txt`.
|
| 13 |
+
- Created retrieval confusion heatmap and retrieval error rate summary bar chart saved to `figures\bad_cases`
|
| 14 |
+
- Visualised sampled bad cases with top-10 retrieval results per query, saved to `.png` at `figures\bad_cases`.
|
| 15 |
+
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
## In Progress
|
| 19 |
+
|
| 20 |
+
- Improving the visual retrieval report resolution or switching to a more readable format (HTML or PDF) for better presentation — planned for tomorrow.
|
| 21 |
+
- Writing up a bad case summary covering the key failure patterns identified and potential approaches to improve embedding quality for the affected classes.
|
| 22 |
+
|
| 23 |
+
---
|
| 24 |
+
|
| 25 |
+
## Blockers
|
| 26 |
+
|
| 27 |
+
- The visualised bad cases is non-readable.
|
| 28 |
+
|
| 29 |
+
---
|
| 30 |
+
|
| 31 |
+
## Key Learnings
|
| 32 |
+
|
| 33 |
+
- **Per-class errors reveal more than aggregate accuracy:** SigLIP2 and ModernVBERT Bi are only ~2.6 percentage points apart in overall error rate but ModernVBERT Bi is dramatically worse on Report specifically (70.4% vs 44.4%), which aggregate accuracy alone hides. Full breakdown in the bad case report.
|
| 34 |
+
- **Systematic vs random errors:** The Memo/Letter confusion is consistent across both models and almost all Memo error cases, pointing to a structural embedding space overlap rather than noise — a more actionable signal than scattered errors.
|
| 35 |
+
- **Low distance does not guarantee correct retrieval:** Some Scientific and Note cases in SigLIP2 show very low cosine distances to wrong-class neighbours, meaning the model is highly confident but wrong — suggesting genuine class overlap in embedding space rather than general uncertainty.
|