AIStudyHi/asyira

Image Embedding Benchmark — SigLIP vs ModernVBERT on Tobacco-3482

Benchmarks several vision encoder families as frozen embedding extractors for document image classification on the Tobacco-3482 dataset.

---

Dataset

Corrected Tabacco Dataset with OCR Benchmark

→ README_tobacco3482_ocr.md

Property	Value
Dataset	maveriq/tobacco3482
Task	10-class document image classification
Classes	letter, memo, email, resume, scientific, note, ADVE, report, form, news
Size	3,482 scanned document images
Split	80/20 train/test (fixed seed 42)

All images are greyscale scans converted to RGB. Preprocessing applied before embedding extraction:

• Auto-contrast via ImageOps.autocontrast (clips bottom/top 1% of pixels)
• Mild contrast boost (×1.2) on top of autocontrast
• Sharpening via ImageFilter.SHARPEN to help edge-based patch encoders

Class Distribution

Image Sample

---

Models Evaluated

SigLIP (Google)

Sigmoid-loss vision-language model. Unlike CLIP's softmax contrastive loss, SigLIP applies a per-pair sigmoid loss which scales better to large batch sizes and does not require a global normalisation term.

Model	HuggingFace ID	Batch Size	Notes
SigLIP Base (patch16-224)	google/siglip-base-patch16-224	64	Smallest SigLIP — fast baseline for scale ablation
SigLIP Large (patch16-256)	google/siglip-large-patch16-256	32	Mid-point in scale ablation
SigLIP SO400M (patch14-384)	google/siglip-so400m-patch14-384	16	400M-param Shape-Optimised variant; 384px input boosts doc-image detail

SigLIP 2 (Google)

Upgraded SigLIP with additional self-distillation and masked prediction objectives on top of the sigmoid loss. Same model sizes as SigLIP v1 but stronger representations.

Model	HuggingFace ID	Batch Size	Notes
SigLIP2 Base (patch16-224)	google/siglip2-base-patch16-224	64	Same scale as SigLIP Base but better objectives
SigLIP2 Large (patch16-256)	google/siglip2-large-patch16-256	32
SigLIP2 SO400M (patch14-384)	google/siglip2-so400m-patch14-384	16	Top-performing model overall; also tested with PCA post-processing

ModernVBERT

A transformer architecture that tests the effect of progressively stronger training objectives — from masked language modelling (MLM) through contrastive learning to document-specific fine-tuning. All three variants require the Base model to first extract image patch embeddings; Embed and Bi are not standalone vision encoders.

Model	HuggingFace ID	Batch Size	Training objective
ModernVBERT Base	ModernVBERT/modernvbert	8	MLM only — no contrastive objective; serves as training-objective baseline
ModernVBERT Embed	ModernVBERT/modernvbert-embed	8	MLM + contrastive learning for general embedding quality
ModernVBERT Bi	ModernVBERT/bimodernvbert	8	Fine-tuned for document retrieval with a bi-encoder objective

ColModernVBERT was investigated but skipped — the base model (ettin-encoder-150m) is private/gated on HuggingFace and the required colpali_engine branch has not been merged.

CLIP (OpenAI)

Contrastive vision-language model and direct predecessor to SigLIP. Included as a baseline to show how far the field has moved.

Model	HuggingFace ID	Batch Size
CLIP Base (patch32-224)	openai/clip-vit-base-patch32	64
CLIP Large (patch14-224)	openai/clip-vit-large-patch16-224	32

DINOv2 (Meta)

Self-supervised ViT trained without any text labels. Included as a vision-only baseline — interesting because it never saw text or label supervision during pretraining.

Model	HuggingFace ID	Batch Size
DINOv2 Base	facebook/dinov2-base	64

DiT (Microsoft)

Document Image Transformer — a ViT pretrained specifically on 42 million scanned documents from the IIT-CDIP corpus, making it the most domain-relevant model in the benchmark.

Model	HuggingFace ID	Batch Size
DiT Base	microsoft/dit-base	64
DiT Large	microsoft/dit-large	32

Saved to model_registry.txt and model_registry.json

---

Evaluation Protocol

Embeddings are extracted once (L2-normalised, no fine-tuning) and evaluated with two probes:

• Linear probe — logistic regression (sklearn, max_iter=1000) on frozen embeddings; reports accuracy and weighted F1.
• kNN retrieval — cosine-similarity nearest neighbours; reports top-1 and top-5 accuracy.

Post-processing

PCA post-processing (mean-centering → PCA whitening → L2 re-normalisation) was applied to most models to test whether dimensionality reduction improves representations. The best n_components was selected by linear probe accuracy on the test set.

Model(s)	Approach	n sweep
SigLIP SO400M, SigLIP2 SO400M	Separate + postprocess entry in results	64, 128, 256, 512, 768, 1024
CLIP Large, DINOv2 Base, DiT Base, DiT Large	Inline sweep within extraction cell; best n used for final result	64, 128, 256, 512
ModernVBERT Base, Embed, Bi	Fixed postprocess() call at n=256	256 (fixed)

Note on the post-processing trade-off: PCA whitening lifts linear probe (clf) accuracy but can hurt kNN retrieval by distorting the original cosine geometry. The two metrics optimise for different embedding structures, so the choice of whether to apply post-processing depends on the downstream task.

---

Results

Results are checkpointed during execution at results_checkpoint.txt and results_checkpoint.json, while the final aggregated results are saved at benchmark_summary.txt and benchmark_summary.json.

• Best classification accuracy: SigLIP SO400M (patch14-384) + postprocess
• Best kNN: SigLIP SO400M (patch14-384)
• Best Overall: SigLIP SO400M (patch14-384)

---

Embedding Fusion (In Progress)

Experiments combining embeddings from two complementary models via weighted concatenation.

Pair	Rationale
SigLIP2 SO400M + DiT Large	VLM features + doc-specific pretraining
SigLIP2 SO400M + DINOv2 Base	VLM features + self-supervised vision
SigLIP SO400M + DiT Large	VLM features + doc-specific pretraining
SigLIP SO400M + DINOv2 Base	VLM features + self-supervised vision
ModernVBERT Bi + DiT Large	Doc-retrieval features + doc-specific pretraining
ModernVBERT Bi + DINOv2 Base	Doc-retrieval features + self-supervised vision
DiT Large + DINOv2 Base	Both vision-focused, different pretraining signals

Three strategies tested per pair:

• Raw concat (α=0.5) — simple weighted concatenation + L2 renorm
• PCA-whitened concat — reduces redundancy across the two embedding spaces
• Alpha sweep (α ∈ {0.6, 0.7, 0.8}) — up-weights the stronger model

Results

Results saved to fusion_summary.txt and fusion_summary.json on completion.

---

Bad Cases Analysis

Per-label error rates, retrieval confusion heatmaps, top confusion pairs, failure analysis, and improvement experiments for SigLIP2 SO400M and ModernVBERT Bi.

→ result_analysis_tobacco3482.md

---

LayoutLMv3 Result Analysis

Per-label error rates, retrieval confusion heatmaps, top confusion pairs, failure analysis, and improvement experiments for the different layoutLMv3 variants.

→ result_analysis_layoutlmv3.md

Repo Structure

benchmarkembeddings_tabacco3482_v.ipynb   # main notebook
benchmark_summary.txt                      # human-readable results table

embeddings/                                # cached embeddings per model
    train_emb_<ModelName>.npy
    test_emb_<ModelName>.npy

results/                                   # benchmark outputs
    results_checkpoint.json                # accumulated results (auto-saved after each model)
    model_registry.json                    # full model metadata (id, batch size, description)
    benchmark_summary.json                 # structured results for programmatic access

metadata/                                  # dataset and split info
figures/                                   # plots and visualisations

---

Infrastructure

Property	Value
GPU	Local Nvidia GPU (CUDA)
PyTorch	2.6.0+cu124
Key libs	transformers, datasets, scikit-learn, tqdm, Pillow
Checkpointing	Results saved to results/results_checkpoint.json after each model so the run can be resumed after a kernel restart
Embeddings	Cached to .npy files under embeddings/ per model to avoid re-extracting on reruns