| **Week 5 — Progress Update** _Image embedding benchmark · Tobacco-3482_ |
|
|
| --- |
|
|
| ## Completed |
|
|
| - Extended PCA post-processing (mean-centering, whitening, L2 re-normalisation) to all remaining model families: SigLIP SO400M, SigLIP2 SO400M, DINOv2, DiT Base, and DiT Large. |
| - SigLIP SO400M + postprocess reached **90.24%** clf accuracy. SigLIP2 SO400M + postprocess reached **89.10%**, confirming PCA whitening at n=128 is the sweet spot for SO400M variants. |
| - Completed a full clean rerun of all 19 models with results serialised to JSON checkpoints for reproducibility. |
| - Created and committed a fully updated `README.md` to HuggingFace covering model architecture explanations, HuggingFace IDs, batch sizes, dataset preprocessing steps, full results table, fusion experiment plans, and the new repo folder structure (`embeddings/`, `results/`, `metadata/`, `figures/`). |
| - Analysed misclassified samples on Tobacco-3482 using SigLIP2 SO400M embeddings and identified common failure modes across the 10 document classes. |
| - Corrected a silent bug where ModernVBERT Embed/Bi were not being rerun after code changes. Duplicate cells were adding post-processing on top of stale embeddings. After a proper full rerun, ModernVBERT Embed accuracy recovered to 80%+. |
| - Explored centering-only post-processing (no PCA) as a lighter alternative. Preliminary result on SigLIP2 SO400M: clf 89%, kNN@1/5 ~86%, suggesting the whitening step is responsible for the kNN trade-off. |
|
|
| --- |
|
|
| ## In Progress |
|
|
| - Multi-layer embedding fusion, investigating whether combining intermediate transformer layers improves representation quality, starting with ModernVBERT variants where the gap vs SigLIP is most pronounced. |
|
|
| --- |
|
|
| ## Blockers |
| - ColModernVBERT remains blocked. It depends on a private base model (`ettin-encoder-150m`) and an unmerged `colpali_engine` branch with no path forward without access. |
| - Layer fusion experiments not yet executed. Full model rerun consumed most available compute time remaining of this week and fusion runs are queued for next week. |
|
|
| --- |
|
|
| ## Key Learnings |
|
|
| - **ModernVBERT architecture:** Embed and Bi variants require Base to first extract vision patch embeddings and are not standalone vision encoders. Misunderstanding this caused the near-random results in Run 1. |
| - **PCA whitening trade-offs:** Whitening compresses embeddings into a lower-dimensional isotropic space that benefits a linear probe but can hurt kNN retrieval by distorting the original cosine geometry. The two metrics optimise for different things. |
| - **Reproducibility discipline:** Silently duplicating cells without rerunning upstream code can produce misleading results. The ModernVBERT jump flagged was traced back to this exact issue. Learned to always do a clean end-to-end rerun before recording numbers. |
| - **HuggingFace workflow:** Gained hands-on experience committing experiment documentation and navigating the HuggingFace Hub for dataset access, and repo management. |
|
|
