Updates/Week_05_Update.md

**Week 5 — Progress Update** _Image embedding benchmark · Tobacco-3482_

---

## Completed

- Extended PCA post-processing (mean-centering, whitening, L2 re-normalisation) to all remaining model families: SigLIP SO400M, SigLIP2 SO400M, DINOv2, DiT Base, and DiT Large.
- SigLIP SO400M + postprocess reached **90.24%** clf accuracy. SigLIP2 SO400M + postprocess reached **89.10%**, confirming PCA whitening at n=128 is the sweet spot for SO400M variants.
- Completed a full clean rerun of all 19 models with results serialised to JSON checkpoints for reproducibility.
- Created and committed a fully updated `README.md` to HuggingFace covering model architecture explanations, HuggingFace IDs, batch sizes, dataset preprocessing steps, full results table, fusion experiment plans, and the new repo folder structure (`embeddings/`, `results/`, `metadata/`, `figures/`).
- Analysed misclassified samples on Tobacco-3482 using SigLIP2 SO400M embeddings and identified common failure modes across the 10 document classes.
- Corrected a silent bug where ModernVBERT Embed/Bi were not being rerun after code changes. Duplicate cells were adding post-processing on top of stale embeddings. After a proper full rerun, ModernVBERT Embed accuracy recovered to 80%+.
- Explored centering-only post-processing (no PCA) as a lighter alternative. Preliminary result on SigLIP2 SO400M: clf 89%, kNN@1/5 ~86%, suggesting the whitening step is responsible for the kNN trade-off.

---

## In Progress

- Multi-layer embedding fusion, investigating whether combining intermediate transformer layers improves representation quality, starting with ModernVBERT variants where the gap vs SigLIP is most pronounced.

---

## Blockers
- ColModernVBERT remains blocked. It depends on a private base model (`ettin-encoder-150m`) and an unmerged `colpali_engine` branch with no path forward without access.
- Layer fusion experiments not yet executed. Full model rerun consumed most available compute time remaining of this week and fusion runs are queued for next week.

---

## Key Learnings

-  **ModernVBERT architecture:** Embed and Bi variants require Base to first extract vision patch embeddings and are not standalone vision encoders. Misunderstanding this caused the near-random results in Run 1.
- **PCA whitening trade-offs:** Whitening compresses embeddings into a lower-dimensional isotropic space that benefits a linear probe but can hurt kNN retrieval by distorting the original cosine geometry. The two metrics optimise for different things.
- **Reproducibility discipline:** Silently duplicating cells without rerunning upstream code can produce misleading results. The ModernVBERT jump flagged was traced back to this exact issue. Learned to always do a clean end-to-end rerun before recording numbers.
- **HuggingFace workflow:** Gained hands-on experience committing experiment documentation and navigating the HuggingFace Hub for dataset access, and repo management.