Integrate with Sentence Transformers via MultiVectorEncoder

1_Dense/config.json

@@ -0,0 +1,8 @@
+{
+    "in_features": 2560,
+    "out_features": 128,
+    "bias": true,
+    "activation_function": "torch.nn.modules.linear.Identity",
+    "module_input_name": "token_embeddings",
+    "module_output_name": "token_embeddings"
+}

1_Dense/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2260e6ae0b1e8c26611963073d6f9c39e14f59897f943eb2bf6762d8baea26d4
+size 1311392

2_Normalize/config.json

@@ -0,0 +1,4 @@
+{
+    "module_input_name": "token_embeddings",
+    "module_output_name": "token_embeddings"
+}

3_MultiVectorMask/config.json

@@ -0,0 +1,3 @@
+{
+    "skiplist_words": []
+}

README.md

@@ -22,6 +22,7 @@ tags:
 - vidore
 - document-retrieval
 - multimodal
+- sentence-transformers
 base_model:
 - Qwen/Qwen3.5-4B
 datasets:
@@ -143,13 +144,64 @@ Evaluated across 8 domains with queries spanning 6 languages (EN, FR, DE, IT, PT
 
 ## Quick Start
 
-### Installation
+### Using Sentence Transformers
+
+EVIE loads as a [Sentence Transformers](https://www.sbert.net/) `MultiVectorEncoder`, which exposes
+the familiar `encode_query` / `encode_document` / `similarity` API and computes MaxSim for you:
 
 ```bash
-pip install -r requirements.txt
+pip install "sentence-transformers[image]>=6.0.0"
+```
+
+```python
+from sentence_transformers import MultiVectorEncoder
+
+model = MultiVectorEncoder("tencent/EVIE-Preview-4.5B")
+
+queries = [
+    "What is the variable represented on the y-axis of the graph?",
+    "Total outlay is maximum in which year?",
+]
+documents = [
+    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
+    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
+    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc3.jpg",
+    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc4.jpg",
+]
+
+query_embeddings = model.encode_query(queries)
+document_embeddings = model.encode_document(documents)
+print(query_embeddings[0].shape, document_embeddings[0].shape)
+# torch.Size([23, 128]) torch.Size([755, 128])
+
+scores = model.similarity(query_embeddings, document_embeddings)
+print(scores)
+# tensor([[17.2949, 10.7598,  7.9268,  7.3613],
+#         [ 6.5547, 13.3711,  6.2627,  6.1104]])
+print("Best document per query:", scores.argmax(dim=1))
+# Best document per query: tensor([0, 1])
+```
+
+Documents may be given as URLs, file paths or `PIL.Image` objects. Bidirectional attention is part of
+the configuration, so no extra call is needed. The scores above come from the plain load, which uses
+the checkpoint's `bfloat16` weights and SDPA. Loading options are forwarded through `model_kwargs`,
+and the 768-visual-token budget is a `processor_kwargs` setting:
+
+```python
+model = MultiVectorEncoder(
+    "tencent/EVIE-Preview-4.5B",
+    model_kwargs={"attn_implementation": "flash_attention_2", "device_map": "cuda:0"},
+    processor_kwargs={"size": {"longest_edge": 1536 * 32 * 32, "shortest_edge": 65536}},
+)
 ```
 
-### Python Inference
+Text passed to `encode_document` is rendered as a query, since the model has no text-document format.
+
+### Using ColPali Engine
+
+```bash
+pip install -r requirements.txt
+```
 
 ```python
 import torch

chat_template.jinja

@@ -0,0 +1,19 @@
+{%- for message in messages -%}
+{%- set ns = namespace(has_image=false, text='') -%}
+{%- if message['content'] is string -%}
+{%- set ns.text = message['content'] -%}
+{%- else -%}
+{%- for item in message['content'] -%}
+{%- if 'image' in item or 'image_url' in item or item.type == 'image' -%}
+{%- set ns.has_image = true -%}
+{%- elif 'text' in item -%}
+{%- set ns.text = ns.text + item.text -%}
+{%- endif -%}
+{%- endfor -%}
+{%- endif -%}
+{%- if ns.has_image -%}
+{{- '<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Describe the image.<|im_end|><|endoftext|>' -}}
+{%- else -%}
+{{- ns.text + '<|endoftext|>' * 10 -}}
+{%- endif -%}
+{%- endfor -%}

config_sentence_transformers.json

@@ -0,0 +1,9 @@
+{
+    "__version__": {
+        "sentence_transformers": "6.0.0"
+    },
+    "model_type": "MultiVectorEncoder",
+    "similarity_fn_name": "maxsim",
+    "prompts": {},
+    "default_prompt_name": null
+}

modules.json

@@ -0,0 +1,26 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.base.modules.transformer.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Dense",
+    "type": "sentence_transformers.base.modules.dense.Dense"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.sentence_transformer.modules.normalize.Normalize"
+  },
+  {
+    "idx": 3,
+    "name": "3",
+    "path": "3_MultiVectorMask",
+    "type": "sentence_transformers.multi_vector_encoder.modules.multi_vector_mask.MultiVectorMask"
+  }
+]

processor_config.json

@@ -25,7 +25,7 @@
     },
     "temporal_patch_size": 2
   },
-  "processor_class": "ColQwen3_5Processor",
+  "processor_class": "Qwen3VLProcessor",
   "video_processor": {
     "do_convert_rgb": true,
     "do_normalize": true,

sentence_bert_config.json

@@ -0,0 +1,25 @@
+{
+    "transformer_task": "feature-extraction",
+    "modality_config": {
+        "text": {
+            "method": "forward",
+            "method_output_name": "last_hidden_state"
+        },
+        "image": {
+            "method": "forward",
+            "method_output_name": "last_hidden_state"
+        },
+        "message": {
+            "method": "forward",
+            "method_output_name": "last_hidden_state",
+            "format": "structured"
+        }
+    },
+    "module_output_name": "token_embeddings",
+    "unpad_inputs": false,
+    "config_kwargs": {
+        "text_config": {
+            "is_causal": false
+        }
+    }
+}

tokenizer_config.json

@@ -21,8 +21,9 @@
     "vision_eos_token": "<|vision_end|>"
   },
   "pad_token": "<|endoftext|>",
+  "padding_side": "left",
   "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
-  "processor_class": "ColQwen3_5Processor",
+  "processor_class": "Qwen3VLProcessor",
   "split_special_tokens": false,
   "tokenizer_class": "Qwen2Tokenizer",
   "unk_token": null,