| --- |
| license: mit |
| language: |
| - en |
| - es |
| - de |
| - fr |
| - it |
| - pt |
| base_model: |
| - athrael-soju/UltraVBERT |
| base_model_relation: finetune |
| datasets: |
| - athrael-soju/vdr-train-500k |
| tags: |
| - ultravbert |
| - sentence-transformers |
| - vision-language |
| - multimodal |
| - document-retrieval |
| - late-interaction |
| - multi-vector |
| - colbert |
| - colpali |
| - vidore |
| pipeline_tag: visual-document-retrieval |
| library_name: sentence-transformers |
| --- |
| |
| <!-- vllm-ci-copy --> |
| > [!NOTE] |
| > This is a copy of [athrael-soju/ColUltraVBERT](https://huggingface.co/athrael-soju/ColUltraVBERT) at commit `486d2f2`, made on 2026-10-30. vLLM CI uses this checkpoint and the original is on a personal account, so we keep a copy here to reduce supply-chain risk. |
|
|
|
|
| <img src="colultravbert-hero.png" width="100%" alt="ColUltraVBERT"> |
|
|
| # ColUltraVBERT |
|
|
| ColUltraVBERT is a 255M-parameter late-interaction retriever for visual documents, fine-tuned from |
| [UltraVBERT](https://huggingface.co/athrael-soju/UltraVBERT). It encodes each query token and each page token as a |
| 768-d vector and ranks pages by MaxSim. Truncating the vectors to 128, 256, 320 or 512 dimensions gives a smaller |
| index. |
|
|
| ## Models |
|
|
| <table style="display: table; width: 100%;"> |
| <thead> |
| <tr><th align="left">Model</th><th align="left">Role</th><th align="left">Status</th></tr> |
| </thead> |
| <tbody> |
| <tr><td><a href="https://huggingface.co/athrael-soju/UltraVBERT">UltraVBERT</a></td><td>Base encoder aligned to page images</td><td>Available</td></tr> |
| <tr><td><a href="https://huggingface.co/athrael-soju/BiUltraVBERT">BiUltraVBERT</a></td><td>Single-vector retriever for visual documents</td><td>Available</td></tr> |
| <tr><td><a href="https://huggingface.co/athrael-soju/ColUltraVBERT">ColUltraVBERT</a></td><td>Late-interaction retriever for visual documents</td><td>Available · <a href="https://huggingface.co/spaces/athrael-soju/ColUltraVBERT">Demo</a></td></tr> |
| </tbody> |
| </table> |
|
|
| ## Data |
|
|
| <table style="display: table; width: 100%;"> |
| <thead> |
| <tr><th align="left">Dataset</th><th align="left">Content</th></tr> |
| </thead> |
| <tbody> |
| <tr><td><a href="https://huggingface.co/datasets/athrael-soju/vdr-train-500k">VDR Train 500k</a></td><td>499,704 query-page pairs; queries in English (73%), Spanish, German, French, Italian and Portuguese</td></tr> |
| </tbody> |
| </table> |
|
|
| ## Usage |
|
|
| Requires `sentence-transformers>=6.1` and `transformers>=5.21`, which supports UltraVBERT natively |
| ([docs](https://huggingface.co/docs/transformers/model_doc/ultravbert)). On 5.20.x, pass `trust_remote_code=True` to load the code in this repository. |
|
|
| ```python |
| from PIL import Image |
| from sentence_transformers import MultiVectorEncoder |
| |
| model = MultiVectorEncoder("athrael-soju/ColUltraVBERT") |
| |
| pages = [Image.open("page1.png"), Image.open("page2.png")] |
| queries = ["What was the 2023 revenue?", "Which table lists regional margins?"] |
| |
| page_embeddings = model.encode_document(pages) |
| query_embeddings = model.encode_query(queries) |
| |
| # One row per query, one column per page. |
| scores = model.similarity(query_embeddings, page_embeddings) |
| ``` |
|
|
| For a smaller index, keep the first 128, 256, 320 or 512 dimensions of each vector and L2-normalize it again. |
|
|
| ## Model |
|
|
| A linear layer projects each of UltraVBERT's 768-d token states to a 768-d retrieval vector. Vectors are L2-normalized |
| and padding tokens are dropped. Pages are resized to at most 3,072 px on the longest edge and split into up to 25 tiles |
| of 512 px, plus a downscaled view of the whole page: at most 1,664 image vectors per page, plus the prompt and |
| tile-tag vectors. |
|
|
| ## Training |
|
|
| The released weights are a per-block merge of four fine-tunes of UltraVBERT on VDR Train 500k that differ in |
| hard negatives, distillation and seed. Each fine-tune trains rank-32 LoRA adapters on the text encoder and the output |
| projection for one epoch at batch size 256, with Matryoshka losses at 128, 256, 320, 512 and 768 dimensions; the vision |
| encoder and connector stay frozen. |
|
|
| ## ViDoRe |
|
|
| ColUltraVBERT's scores are its official MTEB results |
| ([results#783](https://github.com/embeddings-benchmark/results/pull/783)): MTEB 2.26.3, 768-d vectors, bf16 and SDPA |
| attention. The MTEB maintainers scored the private ViDoRe V3 tasks, Nuclear and Telecom |
| ([results#792](https://github.com/embeddings-benchmark/results/pull/792)). ColModernVBERT's V1 and V2 scores are our |
| MTEB 2.26.3 runs of [`ModernVBERT/colmodernvbert`](https://huggingface.co/ModernVBERT/colmodernvbert) at revision |
| `9052fc0e`; its V3 scores are its MTEB results. |
|
|
| | ViDoRe V1 (nDCG@5) | ColModernVBERT | ColUltraVBERT | |
| | --- | ---: | ---: | |
| | ArxivQA | 75.0 | 80.1 | |
| | DocVQA | 56.2 | 57.7 | |
| | InfoVQA | 86.5 | 89.5 | |
| | TabFQuAD | 76.1 | 92.5 | |
| | TAT-DQA | 73.9 | 75.1 | |
| | ShiftProject | 64.8 | 81.2 | |
| | SyntheticDocQA AI | 97.5 | 99.6 | |
| | SyntheticDocQA Energy | 94.2 | 95.7 | |
| | SyntheticDocQA Government Reports | 93.3 | 95.1 | |
| | SyntheticDocQA Healthcare Industry | 96.9 | 97.9 | |
| | **Mean** | **81.4** | **86.4** | |
|
|
| | ViDoRe V2 (nDCG@5) | ColModernVBERT | ColUltraVBERT | |
| | --- | ---: | ---: | |
| | ESG Reports | 40.5 | 49.8 | |
| | Economics Reports | 32.6 | 51.2 | |
| | BioMedical Lectures | 34.8 | 54.3 | |
| | ESG Reports HL | 54.8 | 64.0 | |
| | **Mean** | **40.7** | **54.8** | |
|
|
| | ViDoRe V3 (nDCG@10) | ColModernVBERT | ColUltraVBERT | |
| | --- | ---: | ---: | |
| | Computer Science | 37.6 | 66.5 | |
| | Energy | 32.5 | 51.9 | |
| | Finance EN | 28.8 | 53.3 | |
| | Finance FR | 19.8 | 33.3 | |
| | HR | 19.8 | 51.1 | |
| | Industrial | 15.7 | 37.6 | |
| | Pharmaceuticals | 32.2 | 57.6 | |
| | Physics | 22.6 | 37.7 | |
| | Nuclear (private) | 21.5 | 42.0 | |
| | Telecom (private) | 30.3 | 62.0 | |
| | **Mean, public** | **26.1** | **48.6** | |
| | **Mean, private** | **25.9** | **52.0** | |
| | **Mean, all** | **26.1** | **49.3** | |
|
|
| ### Smaller indexes |
|
|
| Scores come from cached page features; at 768 dimensions and 3,072 px they are within 0.1 of the official means. For |
| 2,048 px pages with 16 tiles, set `size.longest_edge` to 2048 and `max_image_tiles` to 16 in `processor_config.json`; a |
| page then produces at most 1,088 image vectors. |
|
|
| | Vector size | Page resolution | V1 | V2 | V3 public | |
| | ---: | --- | ---: | ---: | ---: | |
| | 768 | 3,072 px, 25 tiles | 86.4 | 54.8 | 48.7 | |
| | 512 | 3,072 px, 25 tiles | 86.3 | 54.3 | 48.4 | |
| | 320 | 3,072 px, 25 tiles | 86.0 | 54.2 | 48.2 | |
| | 256 | 3,072 px, 25 tiles | 86.0 | 54.3 | 48.0 | |
| | 128 | 3,072 px, 25 tiles | 85.7 | 54.3 | 46.8 | |
| | 768 | 2,048 px, 16 tiles | 86.3 | 53.3 | 47.6 | |
| | 512 | 2,048 px, 16 tiles | 86.3 | 52.9 | 47.5 | |
| | 320 | 2,048 px, 16 tiles | 86.4 | 52.4 | 47.2 | |
| | 256 | 2,048 px, 16 tiles | 86.4 | 52.1 | 46.9 | |
| | 128 | 2,048 px, 16 tiles | 85.6 | 52.6 | 45.9 | |
|
|
| ## Limits |
|
|
| - A page produces up to 1,664 image vectors, 2.4 MiB at 768 dimensions in bf16, plus its prompt and tile-tag vectors. |
|
|
| ## License |
|
|
| `MIT`. The UltraBERT-150M and SigLIP2-B/16-512 components are Apache-2.0, with their license text in |
| `LICENSE-APACHE`. |
|
|
