File size: 10,030 Bytes

# Image Embedding Benchmark — SigLIP vs ModernVBERT on Tobacco-3482

Benchmarks several vision encoder families as **frozen embedding extractors** for document image classification on the [Tobacco-3482](https://huggingface.co/datasets/maveriq/tobacco3482) dataset.

---
## Dataset

### Corrected Tabacco Dataset with OCR Benchmark
 → [`README_tobacco3482_ocr.md`](tabacco3482_benchmarks/README_tobacco3482_ocr.md)

| Property | Value |
|---|---|
| Dataset | `maveriq/tobacco3482` |
| Task | 10-class document image classification |
| Classes | letter, memo, email, resume, scientific, note, ADVE, report, form, news |
| Size | 3,482 scanned document images |
| Split | 80/20 train/test (fixed seed 42) |

All images are greyscale scans converted to RGB. Preprocessing applied before embedding extraction:
- Auto-contrast via `ImageOps.autocontrast` (clips bottom/top 1% of pixels)
- Mild contrast boost (×1.2) on top of autocontrast
- Sharpening via `ImageFilter.SHARPEN` to help edge-based patch encoders

### Class Distribution

![Class Distribution Graph](tabacco3482_benchmarks/figures/class_distribution_graph.png) 
![Class Distribution](tabacco3482_benchmarks/figures/class_distribution.png)
#### Image Sample

![Class Samples](tabacco3482_benchmarks/figures/class_samples.png)

---
## Models Evaluated

### SigLIP (Google)

Sigmoid-loss vision-language model. Unlike CLIP's softmax contrastive loss, SigLIP applies a per-pair sigmoid loss which scales better to large batch sizes and does not require a global normalisation term.

| Model | HuggingFace ID | Batch Size | Notes |
|---|---|---|---|
| SigLIP Base (patch16-224) | `google/siglip-base-patch16-224` | 64 | Smallest SigLIP — fast baseline for scale ablation |
| SigLIP Large (patch16-256) | `google/siglip-large-patch16-256` | 32 | Mid-point in scale ablation |
| SigLIP SO400M (patch14-384) | `google/siglip-so400m-patch14-384` | 16 | 400M-param Shape-Optimised variant; 384px input boosts doc-image detail |

### SigLIP 2 (Google)

Upgraded SigLIP with additional self-distillation and masked prediction objectives on top of the sigmoid loss. Same model sizes as SigLIP v1 but stronger representations.

| Model                        | HuggingFace ID                      | Batch Size | Notes                                                              |
| ---------------------------- | ----------------------------------- | ---------- | ------------------------------------------------------------------ |
| SigLIP2 Base (patch16-224)   | `google/siglip2-base-patch16-224`   | 64         | Same scale as SigLIP Base but better objectives                    |
| SigLIP2 Large (patch16-256)  | `google/siglip2-large-patch16-256`  | 32         |                                                                    |
| SigLIP2 SO400M (patch14-384) | `google/siglip2-so400m-patch14-384` | 16         | Top-performing model overall; also tested with PCA post-processing |

### ModernVBERT

A transformer architecture that tests the effect of progressively stronger training objectives — from masked language modelling (MLM) through contrastive learning to document-specific fine-tuning. All three variants require the Base model to first extract image patch embeddings; Embed and Bi are not standalone vision encoders.

| Model | HuggingFace ID | Batch Size | Training objective |
|---|---|---|---|
| ModernVBERT Base | `ModernVBERT/modernvbert` | 8 | MLM only — no contrastive objective; serves as training-objective baseline |
| ModernVBERT Embed | `ModernVBERT/modernvbert-embed` | 8 | MLM + contrastive learning for general embedding quality |
| ModernVBERT Bi | `ModernVBERT/bimodernvbert` | 8 | Fine-tuned for document retrieval with a bi-encoder objective |

> **ColModernVBERT** was investigated but skipped — the base model (`ettin-encoder-150m`) is private/gated on HuggingFace and the required `colpali_engine` branch has not been merged.

### CLIP (OpenAI)

Contrastive vision-language model and direct predecessor to SigLIP. Included as a baseline to show how far the field has moved.

| Model | HuggingFace ID | Batch Size |
|---|---|---|
| CLIP Base (patch32-224) | `openai/clip-vit-base-patch32` | 64 |
| CLIP Large (patch14-224) | `openai/clip-vit-large-patch16-224` | 32 |

### DINOv2 (Meta)

Self-supervised ViT trained without any text labels. Included as a vision-only baseline — interesting because it never saw text or label supervision during pretraining.

| Model | HuggingFace ID | Batch Size |
|---|---|---|
| DINOv2 Base | `facebook/dinov2-base` | 64 |

### DiT (Microsoft)

Document Image Transformer — a ViT pretrained specifically on 42 million scanned documents from the IIT-CDIP corpus, making it the most domain-relevant model in the benchmark.

| Model | HuggingFace ID | Batch Size |
|---|---|---|
| DiT Base | `microsoft/dit-base` | 64 |
| DiT Large | `microsoft/dit-large` | 32 |

Saved to [model_registry.txt](tabacco3482_benchmarks/model_registry.txt) and [model_registry.json](tabacco3482_benchmarks/results/model_registry.json)

---

## Evaluation Protocol

Embeddings are extracted once (L2-normalised, no fine-tuning) and evaluated with two probes:

- **Linear probe** — logistic regression (`sklearn`, `max_iter=1000`) on frozen embeddings; reports accuracy and weighted F1.
- **kNN retrieval** — cosine-similarity nearest neighbours; reports top-1 and top-5 accuracy.

### Post-processing

PCA post-processing (mean-centering → PCA whitening → L2 re-normalisation) was applied to most models to test whether dimensionality reduction improves representations. The best `n_components` was selected by linear probe accuracy on the test set.

| Model(s) | Approach | n sweep |
|---|---|---|
| SigLIP SO400M, SigLIP2 SO400M | Separate `+ postprocess` entry in results | 64, 128, 256, 512, 768, 1024 |
| CLIP Large, DINOv2 Base, DiT Base, DiT Large | Inline sweep within extraction cell; best n used for final result | 64, 128, 256, 512 |
| ModernVBERT Base, Embed, Bi | Fixed `postprocess()` call at n=256 | 256 (fixed) |

> **Note on the post-processing trade-off:** PCA whitening lifts linear probe (clf) accuracy but can hurt kNN retrieval by distorting the original cosine geometry. The two metrics optimise for different embedding structures, so the choice of whether to apply post-processing depends on the downstream task.

---

## Results

![Results](tabacco3482_benchmarks/figures/results.png) <br>
Results are checkpointed during execution at [`results_checkpoint.txt`](tabacco3482_benchmarks/results_checkpoint.txt) and [`results_checkpoint.json`](tabacco3482_benchmarks/results/results_checkpoint.json), while the final aggregated results are saved at [`benchmark_summary.txt`](tabacco3482_benchmarks/benchmark_summary.txt) and [`benchmark_summary.json`](tabacco3482_benchmarks/results/benchmark_summary.json).

- Best classification accuracy: SigLIP SO400M (patch14-384) + postprocess
- Best kNN: SigLIP SO400M (patch14-384) 
- Best Overall: SigLIP SO400M (patch14-384)

---

## Embedding Fusion (In Progress)

Experiments combining embeddings from two complementary models via weighted concatenation.

| Pair | Rationale |
|---|---|
| SigLIP2 SO400M + DiT Large | VLM features + doc-specific pretraining |
| SigLIP2 SO400M + DINOv2 Base | VLM features + self-supervised vision |
| SigLIP SO400M + DiT Large | VLM features + doc-specific pretraining |
| SigLIP SO400M + DINOv2 Base | VLM features + self-supervised vision |
| ModernVBERT Bi + DiT Large | Doc-retrieval features + doc-specific pretraining |
| ModernVBERT Bi + DINOv2 Base | Doc-retrieval features + self-supervised vision |
| DiT Large + DINOv2 Base | Both vision-focused, different pretraining signals |

Three strategies tested per pair:

- **Raw concat** (α=0.5) — simple weighted concatenation + L2 renorm
- **PCA-whitened concat** — reduces redundancy across the two embedding spaces
- **Alpha sweep** (α ∈ {0.6, 0.7, 0.8}) — up-weights the stronger model

### Results

![Fusion Results](tabacco3482_benchmarks/figures/fusion_results.png)

Results saved to [`fusion_summary.txt`](tabacco3482_benchmarks/results/fusion_summary.txt) and [`fusion_summary.json`](results/fusion_summary.json) on completion.

---

## Bad Cases Analysis

Per-label error rates, retrieval confusion heatmaps, top confusion pairs, failure analysis, and improvement experiments for SigLIP2 SO400M and ModernVBERT Bi.

→ [`result_analysis_tobacco3482.md`](tabacco3482_benchmarks/figures/bad_cases/result_analysis.md)

---
## LayoutLMv3 Result Analysis
Per-label error rates, retrieval confusion heatmaps, top confusion pairs, failure analysis, and improvement experiments for the different layoutLMv3 variants.

→ [`result_analysis_layoutlmv3.md`](tabacco3482_benchmarks/figures/layoutlmv3/result_analysis_layoutlmv3.md)

## Repo Structure

```
benchmarkembeddings_tabacco3482_v.ipynb   # main notebook
benchmark_summary.txt                      # human-readable results table

embeddings/                                # cached embeddings per model
    train_emb_<ModelName>.npy
    test_emb_<ModelName>.npy

results/                                   # benchmark outputs
    results_checkpoint.json                # accumulated results (auto-saved after each model)
    model_registry.json                    # full model metadata (id, batch size, description)
    benchmark_summary.json                 # structured results for programmatic access

metadata/                                  # dataset and split info
figures/                                   # plots and visualisations
```

---

## Infrastructure

| Property | Value |
|---|---|
| GPU | Local Nvidia GPU (CUDA) |
| PyTorch | 2.6.0+cu124 |
| Key libs | `transformers`, `datasets`, `scikit-learn`, `tqdm`, `Pillow` |
| Checkpointing | Results saved to `results/results_checkpoint.json` after each model so the run can be resumed after a kernel restart |
| Embeddings | Cached to `.npy` files under `embeddings/` per model to avoid re-extracting on reruns |