--- license: mit language: - en - es - de - fr - it - pt base_model: - athrael-soju/UltraVBERT base_model_relation: finetune datasets: - athrael-soju/vdr-train-500k tags: - ultravbert - sentence-transformers - vision-language - multimodal - document-retrieval - late-interaction - multi-vector - colbert - colpali - vidore pipeline_tag: visual-document-retrieval library_name: sentence-transformers --- > [!NOTE] > This is a copy of [athrael-soju/ColUltraVBERT](https://huggingface.co/athrael-soju/ColUltraVBERT) at commit `486d2f2`, made on 2026-10-30. vLLM CI uses this checkpoint and the original is on a personal account, so we keep a copy here to reduce supply-chain risk.

# ColUltraVBERT ColUltraVBERT is a 255M-parameter late-interaction retriever for visual documents, fine-tuned from [UltraVBERT](https://huggingface.co/athrael-soju/UltraVBERT). It encodes each query token and each page token as a 768-d vector and ranks pages by MaxSim. Truncating the vectors to 128, 256, 320 or 512 dimensions gives a smaller index. ## Models

Model	Role	Status
UltraVBERT	Base encoder aligned to page images	Available
BiUltraVBERT	Single-vector retriever for visual documents	Available
ColUltraVBERT	Late-interaction retriever for visual documents	Available · Demo

## Data

Dataset	Content
VDR Train 500k	499,704 query-page pairs; queries in English (73%), Spanish, German, French, Italian and Portuguese

## Usage Requires `sentence-transformers>=6.1` and `transformers>=5.21`, which supports UltraVBERT natively ([docs](https://huggingface.co/docs/transformers/model_doc/ultravbert)). On 5.20.x, pass `trust_remote_code=True` to load the code in this repository. ```python from PIL import Image from sentence_transformers import MultiVectorEncoder model = MultiVectorEncoder("athrael-soju/ColUltraVBERT") pages = [Image.open("page1.png"), Image.open("page2.png")] queries = ["What was the 2023 revenue?", "Which table lists regional margins?"] page_embeddings = model.encode_document(pages) query_embeddings = model.encode_query(queries) # One row per query, one column per page. scores = model.similarity(query_embeddings, page_embeddings) ``` For a smaller index, keep the first 128, 256, 320 or 512 dimensions of each vector and L2-normalize it again. ## Model A linear layer projects each of UltraVBERT's 768-d token states to a 768-d retrieval vector. Vectors are L2-normalized and padding tokens are dropped. Pages are resized to at most 3,072 px on the longest edge and split into up to 25 tiles of 512 px, plus a downscaled view of the whole page: at most 1,664 image vectors per page, plus the prompt and tile-tag vectors. ## Training The released weights are a per-block merge of four fine-tunes of UltraVBERT on VDR Train 500k that differ in hard negatives, distillation and seed. Each fine-tune trains rank-32 LoRA adapters on the text encoder and the output projection for one epoch at batch size 256, with Matryoshka losses at 128, 256, 320, 512 and 768 dimensions; the vision encoder and connector stay frozen. ## ViDoRe ColUltraVBERT's scores are its official MTEB results ([results#783](https://github.com/embeddings-benchmark/results/pull/783)): MTEB 2.26.3, 768-d vectors, bf16 and SDPA attention. The MTEB maintainers scored the private ViDoRe V3 tasks, Nuclear and Telecom ([results#792](https://github.com/embeddings-benchmark/results/pull/792)). ColModernVBERT's V1 and V2 scores are our MTEB 2.26.3 runs of [`ModernVBERT/colmodernvbert`](https://huggingface.co/ModernVBERT/colmodernvbert) at revision `9052fc0e`; its V3 scores are its MTEB results. | ViDoRe V1 (nDCG@5) | ColModernVBERT | ColUltraVBERT | | --- | ---: | ---: | | ArxivQA | 75.0 | 80.1 | | DocVQA | 56.2 | 57.7 | | InfoVQA | 86.5 | 89.5 | | TabFQuAD | 76.1 | 92.5 | | TAT-DQA | 73.9 | 75.1 | | ShiftProject | 64.8 | 81.2 | | SyntheticDocQA AI | 97.5 | 99.6 | | SyntheticDocQA Energy | 94.2 | 95.7 | | SyntheticDocQA Government Reports | 93.3 | 95.1 | | SyntheticDocQA Healthcare Industry | 96.9 | 97.9 | | **Mean** | **81.4** | **86.4** | | ViDoRe V2 (nDCG@5) | ColModernVBERT | ColUltraVBERT | | --- | ---: | ---: | | ESG Reports | 40.5 | 49.8 | | Economics Reports | 32.6 | 51.2 | | BioMedical Lectures | 34.8 | 54.3 | | ESG Reports HL | 54.8 | 64.0 | | **Mean** | **40.7** | **54.8** | | ViDoRe V3 (nDCG@10) | ColModernVBERT | ColUltraVBERT | | --- | ---: | ---: | | Computer Science | 37.6 | 66.5 | | Energy | 32.5 | 51.9 | | Finance EN | 28.8 | 53.3 | | Finance FR | 19.8 | 33.3 | | HR | 19.8 | 51.1 | | Industrial | 15.7 | 37.6 | | Pharmaceuticals | 32.2 | 57.6 | | Physics | 22.6 | 37.7 | | Nuclear (private) | 21.5 | 42.0 | | Telecom (private) | 30.3 | 62.0 | | **Mean, public** | **26.1** | **48.6** | | **Mean, private** | **25.9** | **52.0** | | **Mean, all** | **26.1** | **49.3** | ### Smaller indexes Scores come from cached page features; at 768 dimensions and 3,072 px they are within 0.1 of the official means. For 2,048 px pages with 16 tiles, set `size.longest_edge` to 2048 and `max_image_tiles` to 16 in `processor_config.json`; a page then produces at most 1,088 image vectors. | Vector size | Page resolution | V1 | V2 | V3 public | | ---: | --- | ---: | ---: | ---: | | 768 | 3,072 px, 25 tiles | 86.4 | 54.8 | 48.7 | | 512 | 3,072 px, 25 tiles | 86.3 | 54.3 | 48.4 | | 320 | 3,072 px, 25 tiles | 86.0 | 54.2 | 48.2 | | 256 | 3,072 px, 25 tiles | 86.0 | 54.3 | 48.0 | | 128 | 3,072 px, 25 tiles | 85.7 | 54.3 | 46.8 | | 768 | 2,048 px, 16 tiles | 86.3 | 53.3 | 47.6 | | 512 | 2,048 px, 16 tiles | 86.3 | 52.9 | 47.5 | | 320 | 2,048 px, 16 tiles | 86.4 | 52.4 | 47.2 | | 256 | 2,048 px, 16 tiles | 86.4 | 52.1 | 46.9 | | 128 | 2,048 px, 16 tiles | 85.6 | 52.6 | 45.9 | ## Limits - A page produces up to 1,664 image vectors, 2.4 MiB at 768 dimensions in bf16, plus its prompt and tile-tag vectors. ## License `MIT`. The UltraBERT-150M and SigLIP2-B/16-512 components are Apache-2.0, with their license text in `LICENSE-APACHE`.