.md update for week 5 and model card

README.md

@@ -2,124 +2,169 @@
 
 Benchmarks several vision encoder families as **frozen embedding extractors** for document image classification on the [Tobacco-3482](https://huggingface.co/datasets/maveriq/tobacco3482) dataset.
 
+---
+
 ## Dataset
 
 | Property | Value |
 |---|---|
 | Dataset | `maveriq/tobacco3482` |
 | Task | 10-class document image classification |
-| Size | 3482 images |
+| Classes | letter, memo, email, resume, scientific, note, ADVE, report, form, news |
+| Size | 3,482 scanned document images |
 | Split | 80/20 train/test (fixed seed 42) |
 
+All images are greyscale scans converted to RGB. Preprocessing applied before embedding extraction:
+- Auto-contrast via `ImageOps.autocontrast` (clips bottom/top 1% of pixels)
+- Mild contrast boost (×1.2) on top of autocontrast
+- Sharpening via `ImageFilter.SHARPEN` to help edge-based patch encoders
+
+### Class Distribution
+
+![[Pasted image 20260419225937.png|559]]  
+![[Pasted image 20260419230216.png|296]]
+#### Image Sample
+
+![[Pasted image 20260419230241.png|697]]
+
+---
 ## Models Evaluated
 
-| Family          | Variants                                                                                   |
-| --------------- | ------------------------------------------------------------------------------------------ |
-| **SigLIP**      | Base (patch16-224), Large (patch16-256), SO400M (patch14-384)                              |
-| **SigLIP 2**    | Base, Large, SO400M — upgraded with self-distillation + masked prediction                  |
-| **ModernVBERT** | Base, Embed, Bi — tests effect of training objective (MLM → contrastive → doc fine-tuning) |
-| **CLIP**        | Base (patch32-224), Large (patch14-224) — contrastive VLM baseline                         |
-| **DINOv2**      | Base — self-supervised vision-only encoder                                                 |
-| **DiT**         | Base, Large — Document Image Transformer with doc-specific pretraining                     |
+### SigLIP (Google)
 
-> ColModernVBERT was investigated but skipped — the model depends on a private base model (`ettin-encoder-150m`) and an unmerged `colpali_engine` branch.
+Sigmoid-loss vision-language model. Unlike CLIP's softmax contrastive loss, SigLIP applies a per-pair sigmoid loss which scales better to large batch sizes and does not require a global normalisation term.
 
-## Evaluation Protocol
+| Model | HuggingFace ID | Batch Size | Notes |
+|---|---|---|---|
+| SigLIP Base (patch16-224) | `google/siglip-base-patch16-224` | 64 | Smallest SigLIP — fast baseline for scale ablation |
+| SigLIP Large (patch16-256) | `google/siglip-large-patch16-256` | 32 | Mid-point in scale ablation |
+| SigLIP SO400M (patch14-384) | `google/siglip-so400m-patch14-384` | 16 | 400M-param Shape-Optimised variant; 384px input boosts doc-image detail |
 
-Embeddings are extracted once (L2-normalised, no fine-tuning) and evaluated with two probes:
+### SigLIP 2 (Google)
 
-- **Linear probe** — logistic regression (sklearn, max_iter=1000) on frozen embeddings; reports accuracy and weighted F1.
-- **kNN retrieval** — cosine-similarity nearest neighbours; reports top-1 and top-5 accuracy.
+Upgraded SigLIP with additional self-distillation and masked prediction objectives on top of the sigmoid loss. Same model sizes as SigLIP v1 but stronger representations.
 
-PCA post-processing (mean-centering → PCA whitening → L2 re-normalisation) was applied to most models, either as a separate named variant or baked directly into the extraction cell:
+| Model                        | HuggingFace ID                      | Batch Size | Notes                                                              |
+| ---------------------------- | ----------------------------------- | ---------- | ------------------------------------------------------------------ |
+| SigLIP2 Base (patch16-224)   | `google/siglip2-base-patch16-224`   | 64         | Same scale as SigLIP Base but better objectives                    |
+| SigLIP2 Large (patch16-256)  | `google/siglip2-large-patch16-256`  | 32         |                                                                    |
+| SigLIP2 SO400M (patch14-384) | `google/siglip2-so400m-patch14-384` | 16         | Top-performing model overall; also tested with PCA post-processing |
 
-| Model(s) | Approach | n sweep |
+### ModernVBERT
+
+A transformer architecture that tests the effect of progressively stronger training objectives — from masked language modelling (MLM) through contrastive learning to document-specific fine-tuning. All three variants require the Base model to first extract image patch embeddings; Embed and Bi are not standalone vision encoders.
+
+| Model | HuggingFace ID | Batch Size | Training objective |
+|---|---|---|---|
+| ModernVBERT Base | `ModernVBERT/modernvbert` | 8 | MLM only — no contrastive objective; serves as training-objective baseline |
+| ModernVBERT Embed | `ModernVBERT/modernvbert-embed` | 8 | MLM + contrastive learning for general embedding quality |
+| ModernVBERT Bi | `ModernVBERT/bimodernvbert` | 8 | Fine-tuned for document retrieval with a bi-encoder objective |
+
+> **ColModernVBERT** was investigated but skipped — the base model (`ettin-encoder-150m`) is private/gated on HuggingFace and the required `colpali_engine` branch has not been merged.
+
+### CLIP (OpenAI)
+
+Contrastive vision-language model and direct predecessor to SigLIP. Included as a baseline to show how far the field has moved.
+
+| Model | HuggingFace ID | Batch Size |
 |---|---|---|
-| SigLIP SO400M, SigLIP2 SO400M | Separate `+ postprocess` entry in results | 64, 128, 256, 512, 768, 1024 |
-| CLIP Large, DINOv2 Base, DiT Base, DiT Large | Inline sweep within extraction cell; best n used for final result | 64, 128, 256, 512 (CLIP Large also tries 768) |
-| ModernVBERT Base 2, Embed 2, Bi 2 | Fixed `postprocess()` call at n=256 | 256 (fixed) |
+| CLIP Base (patch32-224) | `openai/clip-vit-base-patch32` | 64 |
+| CLIP Large (patch14-224) | `openai/clip-vit-large-patch16-224` | 32 |
 
-The best `n_components` was selected by linear probe accuracy on the test set.
+### DINOv2 (Meta)
 
-## Iteration Log
+Self-supervised ViT trained without any text labels. Included as a vision-only baseline — interesting because it never saw text or label supervision during pretraining.
 
-### Run 1 — Baseline (SigLIP v1 only + baselines, raw embeddings)
-First pass covering SigLIP v1 variants, CLIP Base, DINOv2 Base, DiT Base, and the initial ModernVBERT trio. No post-processing applied. ModernVBERT Embed and Bi came out near-random — embedding quality check revealed collapsed embeddings (mean off-diagonal cosine similarity ~0.90, near-zero per-dimension variance), meaning all images were mapping to nearly the same point in space.
+| Model | HuggingFace ID | Batch Size |
+|---|---|---|
+| DINOv2 Base | `facebook/dinov2-base` | 64 |
 
-| Family | Model | Clf Acc | kNN@1 | kNN@5 |
-|---|---|---|---|---|
-| SigLIP | Base (patch16-224) | 84.36% | 77.33% | 80.49% |
-| SigLIP | Large (patch16-256) | 85.22% | 81.06% | 81.92% |
-| SigLIP | SO400M (patch14-384) | 89.96% | 86.66% | 87.37% |
-| CLIP | Base (patch32-224) | 81.35% | 77.91% | 80.06% |
-| DINOv2 | Base | 74.18% | 70.01% | 71.59% |
-| DiT | Base | 63.56% | 60.55% | 62.55% |
-| ModernVBERT | Base | 62.84% | 69.58% | 73.74% |
-| ModernVBERT | Embed | 18.65% | 7.75% | 16.36% |
-| ModernVBERT | Bi | 18.65% | 14.35% | 13.06% |
+### DiT (Microsoft)
 
-**Finding:** ModernVBERT Embed and Bi were loading as text-only encoders. They require the Base model's vision encoder to extract image patch embeddings first before passing through.
+Document Image Transformer — a ViT pretrained specifically on 42 million scanned documents from the IIT-CDIP corpus, making it the most domain-relevant model in the benchmark.
+
+| Model | HuggingFace ID | Batch Size |
+|---|---|---|
+| DiT Base | `microsoft/dit-base` | 64 |
+| DiT Large | `microsoft/dit-large` | 32 |
 
 ---
 
-### Run 2 — ModernVBERT fixed + SigLIP2 added
-Re-ran the three ModernVBERT variants (`*2` suffix) with Base handling image encoding. Also added the full SigLIP2 family (Base, Large, SO400M) in this pass.
+## Evaluation Protocol
 
-| Family      | Model                | Clf Acc | kNN@1  | kNN@5  |
-| ----------- | -------------------- | ------- | ------ | ------ |
-| SigLIP2     | Base (patch16-224)   | 83.07%  | 73.46% | 75.04% |
-| SigLIP2     | Large (patch16-256)  | 87.23%  | 81.21% | 81.92% |
-| SigLIP2     | SO400M (patch14-384) | 88.81%  | 84.79% | 85.80% |
-| ModernVBERT | Base                 | 62.84%  | 69.58% | 73.74% |
-| ModernVBERT | Embed                | 57.39%  | 59.68% | 60.69% |
-| ModernVBERT | Bi                   | 55.09%  | 60.11% | 58.82% |
+Embeddings are extracted once (L2-normalised, no fine-tuning) and evaluated with two probes:
 
-**Finding:** ModernVBERT Embed and Bi jumped to >55% confirming the fix worked as it isn't far of from the base. But it is still underperforming because this variant should be an improvement from the base model. Further investigation needed on the correct pooling/embedding extraction strategy for those variants.
+- **Linear probe** — logistic regression (`sklearn`, `max_iter=1000`) on frozen embeddings; reports accuracy and weighted F1.
+- **kNN retrieval** — cosine-similarity nearest neighbours; reports top-1 and top-5 accuracy.
 
----
+### Post-processing
 
-### Run 3 — PCA post-processing on ModernVBERT `*2` variants
-Added mean-centering → PCA whitening (n=256) → L2 re-normalisation to Embed 2 and Bi 2. Also added CLIP Large to the comparison.
+PCA post-processing (mean-centering → PCA whitening → L2 re-normalisation) was applied to most models to test whether dimensionality reduction improves representations. The best `n_components` was selected by linear probe accuracy on the test set.
 
-| Family      | Model               | Clf Acc | kNN@1  | kNN@5  |
-| ----------- | ------------------- | ------- | ------ | ------ |
-| ModernVBERT | Base 2              | 84.51%  | 73.03% | 77.62% |
-| ModernVBERT | Embed 2             | 86.94%  | 71.74% | 78.77% |
-| ModernVBERT | Bi 2                | 85.37%  | 71.59% | 78.34% |
-| CLIP        | Large (patch14-224) | 84.65%  | 75.04% | 78.77% |
+| Model(s) | Approach | n sweep |
+|---|---|---|
+| SigLIP SO400M, SigLIP2 SO400M | Separate `+ postprocess` entry in results | 64, 128, 256, 512, 768, 1024 |
+| CLIP Large, DINOv2 Base, DiT Base, DiT Large | Inline sweep within extraction cell; best n used for final result | 64, 128, 256, 512 |
+| ModernVBERT Base, Embed, Bi | Fixed `postprocess()` call at n=256 | 256 (fixed) |
 
-**Finding:** Post-processing gave a solid lift to Base(+21.67%) Embed 2 (+29.5%) and Bi 2 (+30.3%), bringing all three ModernVBERT variants into a competitive range. Confirmed PCA whitening was worth applying more broadly.
+> **Note on the post-processing trade-off:** PCA whitening lifts linear probe (clf) accuracy but can hurt kNN retrieval by distorting the original cosine geometry. The two metrics optimise for different embedding structures, so the choice of whether to apply post-processing depends on the downstream task.
 
 ---
 
-### Run 4 — PCA post-processing extended to SigLIP SO400M, DINOv2, DiT + DiT Large added
-Swept `n_components` across [64, 128, 256, 512] (up to 1024 for SO400M variants) and selected the best n by linear probe accuracy. Applied only to SO400M within SigLIP/SigLIP2 — smaller variants were left as-is. Also added DiT Large.
+## Results
 
-| Family | Model | Clf Acc | kNN@1 | kNN@5 | Best n |
-|---|---|---|---|---|---|
-| SigLIP | SO400M + postprocess | 90.10% | 77.91% | 83.64% | 128 |
-| SigLIP2 | SO400M + postprocess | **90.67%** | 77.62% | 84.22% | 128 |
-| DINOv2 | Base (post-processed) | 79.77% | 71.31% | 74.46% | 128 |
-| DiT | Base (post-processed) | 74.89% | 61.69% | 66.43% | 256 |
-| DiT | Large (post-processed) | 77.33% | 63.99% | 67.43% | 512 |
+![[Pasted image 20260419230643.png]]
 
-**Best overall: SigLIP2 SO400M + postprocess — 90.67% Clf Acc.**
+Best classification accuracy: SigLIP SO400M (patch14-384) + postprocess
+Best kNN: SigLIP SO400M (patch14-384) 
+Best Overall: SigLIP SO400M (patch14-384)
 
 ---
 
-## Infrastructure
+## Embedding Fusion (In Progress)
+
+Experiments combining embeddings from two complementary models via weighted concatenation. Three fusion pairs are being tested:
+
+| Pair | Rationale |
+|---|---|
+| SigLIP2 SO400M + DiT Large | VLM features + doc-specific pretraining |
+| SigLIP2 SO400M + DINOv2 Base | VLM features + self-supervised vision |
+| DiT Large + DINOv2 Base | Both vision-focused, different pretraining signals |
+
+Three strategies per pair:
+- **Raw concat** (α=0.5) — simple weighted concatenation + L2 renorm
+- **PCA-whitened concat** — reduces redundancy across the two embedding spaces
+- **Alpha sweep** (α ∈ {0.6, 0.7, 0.8}) — up-weights the stronger model
 
-- **GPU**: Local Nvidia GPU (CUDA)
-- **PyTorch**: 2.6.0+cu124
-- **Key libs**: `transformers`, `datasets`, `scikit-learn`, `tqdm`, `Pillow`
-- **Checkpointing**: Results saved to `results_checkpoint.json` after each model so the run can be resumed after a kernel restart.
-- **Embeddings**: Cached to `.npy` files per model (`train_emb_<name>.npy`, `test_emb_<name>.npy`).
+---
 
 ## Repo Structure
 
 ```
-benchmarkembeddings_tabacco3482.ipynb   # main notebook
-results_checkpoint.json                 # accumulated results (auto-saved)
-train_emb_<ModelName>.npy               # cached train embeddings per model
-test_emb_<ModelName>.npy                # cached test embeddings per model
+benchmarkembeddings_tabacco3482_v.ipynb   # main notebook
+benchmark_summary.txt                      # human-readable results table
+
+embeddings/                                # cached embeddings per model
+    train_emb_<ModelName>.npy
+    test_emb_<ModelName>.npy
+
+results/                                   # benchmark outputs
+    results_checkpoint.json                # accumulated results (auto-saved after each model)
+    model_registry.json                    # full model metadata (id, batch size, description)
+    benchmark_summary.json                 # structured results for programmatic access
+
+metadata/                                  # dataset and split info
+figures/                                   # plots and visualisations
 ```
+
+---
+
+## Infrastructure
+
+| Property | Value |
+|---|---|
+| GPU | Local Nvidia GPU (CUDA) |
+| PyTorch | 2.6.0+cu124 |
+| Key libs | `transformers`, `datasets`, `scikit-learn`, `tqdm`, `Pillow` |
+| Checkpointing | Results saved to `results/results_checkpoint.json` after each model so the run can be resumed after a kernel restart |
+| Embeddings | Cached to `.npy` files under `embeddings/` per model to avoid re-extracting on reruns |

tabacco3482_benchmarks/Week_05_Update.md

@@ -0,0 +1,34 @@
+**Week 5 — Progress Update** _Image embedding benchmark · Tobacco-3482_
+
+---
+
+## Completed
+
+- Extended PCA post-processing (mean-centering, whitening, L2 re-normalisation) to all remaining model families: SigLIP SO400M, SigLIP2 SO400M, DINOv2, DiT Base, and DiT Large.
+- SigLIP SO400M + postprocess reached **90.24%** clf accuracy. SigLIP2 SO400M + postprocess reached **89.10%**, confirming PCA whitening at n=128 is the sweet spot for SO400M variants.
+- Completed a full clean rerun of all 19 models with results serialised to JSON checkpoints for reproducibility.
+- Created and committed a fully updated `README.md` to HuggingFace covering model architecture explanations, HuggingFace IDs, batch sizes, dataset preprocessing steps, full results table, fusion experiment plans, and the new repo folder structure (`embeddings/`, `results/`, `metadata/`, `figures/`).
+- Analysed misclassified samples on Tobacco-3482 using SigLIP2 SO400M embeddings and identified common failure modes across the 10 document classes.
+- Corrected a silent bug where ModernVBERT Embed/Bi were not being rerun after code changes. Duplicate cells were adding post-processing on top of stale embeddings. After a proper full rerun, ModernVBERT Embed accuracy recovered to 80%+.
+- Explored centering-only post-processing (no PCA) as a lighter alternative. Preliminary result on SigLIP2 SO400M: clf 89%, kNN@1/5 ~86%, suggesting the whitening step is responsible for the kNN trade-off.
+
+---
+
+## In Progress
+
+- Multi-layer embedding fusion, investigating whether combining intermediate transformer layers improves representation quality, starting with ModernVBERT variants where the gap vs SigLIP is most pronounced.
+
+---
+
+## Blockers
+- ColModernVBERT remains blocked. It depends on a private base model (`ettin-encoder-150m`) and an unmerged `colpali_engine` branch with no path forward without access.
+- Layer fusion experiments not yet executed. Full model rerun consumed most available compute time remaining of this week and fusion runs are queued for next week.
+
+---
+
+## Key Learnings
+
+-  **ModernVBERT architecture:** Embed and Bi variants require Base to first extract vision patch embeddings and are not standalone vision encoders. Misunderstanding this caused the near-random results in Run 1.
+- **PCA whitening trade-offs:** Whitening compresses embeddings into a lower-dimensional isotropic space that benefits a linear probe but can hurt kNN retrieval by distorting the original cosine geometry. The two metrics optimise for different things.
+- **Reproducibility discipline:** Silently duplicating cells without rerunning upstream code can produce misleading results. The ModernVBERT jump flagged was traced back to this exact issue. Learned to always do a clean end-to-end rerun before recording numbers.
+- **HuggingFace workflow:** Gained hands-on experience committing experiment documentation and navigating the HuggingFace Hub for dataset access, and repo management.