Image Embedding Benchmark — SigLIP vs ModernVBERT on Tobacco-3482

Benchmarks several vision encoder families as frozen embedding extractors for document image classification on the Tobacco-3482 dataset.

---

Dataset

Corrected Tabacco Dataset with OCR Benchmark

→ README_tobacco3482_ocr.md

All images are greyscale scans converted to RGB. Preprocessing applied before embedding extraction:

• Auto-contrast via ImageOps.autocontrast (clips bottom/top 1% of pixels)
• Mild contrast boost (×1.2) on top of autocontrast
• Sharpening via ImageFilter.SHARPEN to help edge-based patch encoders

Class Distribution

Image Sample

---

Models Evaluated

SigLIP (Google)

Sigmoid-loss vision-language model. Unlike CLIP's softmax contrastive loss, SigLIP applies a per-pair sigmoid loss which scales better to large batch sizes and does not require a global normalisation term.

SigLIP 2 (Google)

Upgraded SigLIP with additional self-distillation and masked prediction objectives on top of the sigmoid loss. Same model sizes as SigLIP v1 but stronger representations.

ModernVBERT

A transformer architecture that tests the effect of progressively stronger training objectives — from masked language modelling (MLM) through contrastive learning to document-specific fine-tuning. All three variants require the Base model to first extract image patch embeddings; Embed and Bi are not standalone vision encoders.

ColModernVBERT was investigated but skipped — the base model (ettin-encoder-150m) is private/gated on HuggingFace and the required colpali_engine branch has not been merged.

CLIP (OpenAI)

Contrastive vision-language model and direct predecessor to SigLIP. Included as a baseline to show how far the field has moved.

DINOv2 (Meta)

Self-supervised ViT trained without any text labels. Included as a vision-only baseline — interesting because it never saw text or label supervision during pretraining.

DiT (Microsoft)

Document Image Transformer — a ViT pretrained specifically on 42 million scanned documents from the IIT-CDIP corpus, making it the most domain-relevant model in the benchmark.

Saved to model_registry.txt and model_registry.json

---

Evaluation Protocol

Embeddings are extracted once (L2-normalised, no fine-tuning) and evaluated with two probes:

• Linear probe — logistic regression (sklearn, max_iter=1000) on frozen embeddings; reports accuracy and weighted F1.
• kNN retrieval — cosine-similarity nearest neighbours; reports top-1 and top-5 accuracy.

Post-processing

PCA post-processing (mean-centering → PCA whitening → L2 re-normalisation) was applied to most models to test whether dimensionality reduction improves representations. The best n_components was selected by linear probe accuracy on the test set.

Note on the post-processing trade-off: PCA whitening lifts linear probe (clf) accuracy but can hurt kNN retrieval by distorting the original cosine geometry. The two metrics optimise for different embedding structures, so the choice of whether to apply post-processing depends on the downstream task.

---

Results

Results are checkpointed during execution at results_checkpoint.txt and results_checkpoint.json, while the final aggregated results are saved at benchmark_summary.txt and benchmark_summary.json.

• Best classification accuracy: SigLIP SO400M (patch14-384) + postprocess
• Best kNN: SigLIP SO400M (patch14-384)
• Best Overall: SigLIP SO400M (patch14-384)

---

Embedding Fusion (In Progress)

Experiments combining embeddings from two complementary models via weighted concatenation.

Three strategies tested per pair:

• Raw concat (α=0.5) — simple weighted concatenation + L2 renorm
• PCA-whitened concat — reduces redundancy across the two embedding spaces
• Alpha sweep (α ∈ {0.6, 0.7, 0.8}) — up-weights the stronger model

Results

Results saved to fusion_summary.txt and fusion_summary.json on completion.

---

Bad Cases Analysis

Per-label error rates, retrieval confusion heatmaps, top confusion pairs, failure analysis, and improvement experiments for SigLIP2 SO400M and ModernVBERT Bi.

→ result_analysis_tobacco3482.md

---

LayoutLMv3 Result Analysis

Per-label error rates, retrieval confusion heatmaps, top confusion pairs, failure analysis, and improvement experiments for the different layoutLMv3 variants.

→ result_analysis_layoutlmv3.md

Repo Structure

benchmarkembeddings_tabacco3482_v.ipynb   # main notebook
benchmark_summary.txt                      # human-readable results table

embeddings/                                # cached embeddings per model
    train_emb_<ModelName>.npy
    test_emb_<ModelName>.npy

results/                                   # benchmark outputs
    results_checkpoint.json                # accumulated results (auto-saved after each model)
    model_registry.json                    # full model metadata (id, batch size, description)
    benchmark_summary.json                 # structured results for programmatic access

metadata/                                  # dataset and split info
figures/                                   # plots and visualisations

---

Infrastructure