Updates/Week_05_Update.md

Week 5 — Progress Update Image embedding benchmark · Tobacco-3482

---

Completed

• Extended PCA post-processing (mean-centering, whitening, L2 re-normalisation) to all remaining model families: SigLIP SO400M, SigLIP2 SO400M, DINOv2, DiT Base, and DiT Large.
• SigLIP SO400M + postprocess reached 90.24% clf accuracy. SigLIP2 SO400M + postprocess reached 89.10%, confirming PCA whitening at n=128 is the sweet spot for SO400M variants.
• Completed a full clean rerun of all 19 models with results serialised to JSON checkpoints for reproducibility.
• Created and committed a fully updated README.md to HuggingFace covering model architecture explanations, HuggingFace IDs, batch sizes, dataset preprocessing steps, full results table, fusion experiment plans, and the new repo folder structure (embeddings/, results/, metadata/, figures/).
• Analysed misclassified samples on Tobacco-3482 using SigLIP2 SO400M embeddings and identified common failure modes across the 10 document classes.
• Corrected a silent bug where ModernVBERT Embed/Bi were not being rerun after code changes. Duplicate cells were adding post-processing on top of stale embeddings. After a proper full rerun, ModernVBERT Embed accuracy recovered to 80%+.
• Explored centering-only post-processing (no PCA) as a lighter alternative. Preliminary result on SigLIP2 SO400M: clf 89%, kNN@1/5 ~86%, suggesting the whitening step is responsible for the kNN trade-off.

---

In Progress

• Multi-layer embedding fusion, investigating whether combining intermediate transformer layers improves representation quality, starting with ModernVBERT variants where the gap vs SigLIP is most pronounced.

---

Blockers

• ColModernVBERT remains blocked. It depends on a private base model (ettin-encoder-150m) and an unmerged colpali_engine branch with no path forward without access.
• Layer fusion experiments not yet executed. Full model rerun consumed most available compute time remaining of this week and fusion runs are queued for next week.

---

Key Learnings

• ModernVBERT architecture: Embed and Bi variants require Base to first extract vision patch embeddings and are not standalone vision encoders. Misunderstanding this caused the near-random results in Run 1.
• PCA whitening trade-offs: Whitening compresses embeddings into a lower-dimensional isotropic space that benefits a linear probe but can hurt kNN retrieval by distorting the original cosine geometry. The two metrics optimise for different things.
• Reproducibility discipline: Silently duplicating cells without rerunning upstream code can produce misleading results. The ModernVBERT jump flagged was traced back to this exact issue. Learned to always do a clean end-to-end rerun before recording numbers.
• HuggingFace workflow: Gained hands-on experience committing experiment documentation and navigating the HuggingFace Hub for dataset access, and repo management.