ColUltraVBERT

ColUltraVBERT is a 255M-parameter late-interaction retriever for visual documents, fine-tuned from UltraVBERT. It encodes each query token and each page token as a 768-d vector and ranks pages by MaxSim. Truncating the vectors to 128, 256, 320 or 512 dimensions gives a smaller index.

Models

Model	Role	Status
UltraVBERT	Base encoder aligned to page images	Available
BiUltraVBERT	Single-vector retriever for visual documents	Available
ColUltraVBERT	Late-interaction retriever for visual documents	Available · Demo

Data

Dataset	Content
VDR Train 500k	499,704 query-page pairs; queries in English (73%), Spanish, German, French, Italian and Portuguese

Usage

Requires sentence-transformers>=6.1 and transformers>=5.20.

from PIL import Image
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("athrael-soju/ColUltraVBERT", trust_remote_code=True)

pages = [Image.open("page1.png"), Image.open("page2.png")]
queries = ["What was the 2023 revenue?", "Which table lists regional margins?"]

page_embeddings = model.encode_document(pages)
query_embeddings = model.encode_query(queries)

# One row per query, one column per page.
scores = model.similarity(query_embeddings, page_embeddings)

For a smaller index, keep the first 128, 256, 320 or 512 dimensions of each vector and L2-normalize it again.

Model

A linear layer projects each of UltraVBERT's 768-d token states to a 768-d retrieval vector. Vectors are L2-normalized and padding tokens are dropped. Pages are resized to at most 3,072 px on the longest edge and split into up to 25 tiles of 512 px, plus a downscaled view of the whole page: at most 1,664 image vectors per page, plus the prompt and tile-tag vectors.

Training

The released weights are a per-block merge of four fine-tunes of UltraVBERT on VDR Train 500k that differ in hard negatives, distillation and seed. Each fine-tune trains rank-32 LoRA adapters on the text encoder and the output projection for one epoch at batch size 256, with Matryoshka losses at 128, 256, 320, 512 and 768 dimensions; the vision encoder and connector stay frozen.

ViDoRe

ColUltraVBERT's scores are its official MTEB results (results#783): MTEB 2.26.3, 768-d vectors, bf16 and SDPA attention. The MTEB maintainers scored the private ViDoRe V3 tasks, Nuclear and Telecom (results#792). ColModernVBERT's V1 and V2 scores are our MTEB 2.26.3 runs of ModernVBERT/colmodernvbert at revision 9052fc0e; its V3 scores are its MTEB results.

ViDoRe V1 (nDCG@5)	ColModernVBERT	ColUltraVBERT
ArxivQA	75.0	80.1
DocVQA	56.2	57.7
InfoVQA	86.5	89.5
TabFQuAD	76.1	92.5
TAT-DQA	73.9	75.1
ShiftProject	64.8	81.2
SyntheticDocQA AI	97.5	99.6
SyntheticDocQA Energy	94.2	95.7
SyntheticDocQA Government Reports	93.3	95.1
SyntheticDocQA Healthcare Industry	96.9	97.9
Mean	81.4	86.4

ViDoRe V2 (nDCG@5)	ColModernVBERT	ColUltraVBERT
ESG Reports	40.5	49.8
Economics Reports	32.6	51.2
BioMedical Lectures	34.8	54.3
ESG Reports HL	54.8	64.0
Mean	40.7	54.8

ViDoRe V3 (nDCG@10)	ColModernVBERT	ColUltraVBERT
Computer Science	37.6	66.5
Energy	32.5	51.9
Finance EN	28.8	53.3
Finance FR	19.8	33.3
HR	19.8	51.1
Industrial	15.7	37.6
Pharmaceuticals	32.2	57.6
Physics	22.6	37.7
Nuclear (private)	21.5	42.0
Telecom (private)	30.3	62.0
Mean, public	26.1	48.6
Mean, private	25.9	52.0
Mean, all	26.1	49.3

Smaller indexes

Scores come from cached page features; at 768 dimensions and 3,072 px they are within 0.1 of the official means. For 2,048 px pages with 16 tiles, set size.longest_edge to 2048 and max_image_tiles to 16 in processor_config.json; a page then produces at most 1,088 image vectors.

Vector size	Page resolution	V1	V2	V3 public
768	3,072 px, 25 tiles	86.4	54.8	48.7
512	3,072 px, 25 tiles	86.3	54.3	48.4
320	3,072 px, 25 tiles	86.0	54.2	48.2
256	3,072 px, 25 tiles	86.0	54.3	48.0
128	3,072 px, 25 tiles	85.7	54.3	46.8
768	2,048 px, 16 tiles	86.3	53.3	47.6
512	2,048 px, 16 tiles	86.3	52.9	47.5
320	2,048 px, 16 tiles	86.4	52.4	47.2
256	2,048 px, 16 tiles	86.4	52.1	46.9
128	2,048 px, 16 tiles	85.6	52.6	45.9