Feature Extraction
sentence-transformers
Safetensors
English
fusion-embedding-connector
embeddings
multimodal
audio
retrieval
matryoshka
qwen3-vl
adapters
custom_code
Instructions to use EximiusLabs/fusion-embedding-2-2b-preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use EximiusLabs/fusion-embedding-2-2b-preview with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("EximiusLabs/fusion-embedding-2-2b-preview", trust_remote_code=True) sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] - Notebooks
- Google Colab
- Kaggle
Model card: add Sentence Transformers usage section
Browse files
README.md
CHANGED
|
@@ -3,6 +3,7 @@ license: cc-by-nc-4.0
|
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
pipeline_tag: feature-extraction
|
|
|
|
| 6 |
tags:
|
| 7 |
- embeddings
|
| 8 |
- multimodal
|
|
@@ -209,7 +210,8 @@ default.
|
|
| 209 |
<details>
|
| 210 |
<summary>Requirements</summary>
|
| 211 |
|
| 212 |
-
- `fusion_embedding` package: `pip install fusion-embedding[hf]`
|
|
|
|
| 213 |
- `transformers>=4.46`, `torch` (CUDA), `torchvision`, `pillow`, `soundfile`, `librosa`
|
| 214 |
- ~14 GB GPU memory at bf16
|
| 215 |
|
|
@@ -255,6 +257,110 @@ gallery = FusionEmbedder.center(gallery_embeddings)
|
|
| 255 |
|
| 256 |
</details>
|
| 257 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 258 |
## Changelog
|
| 259 |
|
| 260 |
- **v0.3-preview** (default): retrieval-tuned flagship. AudioCaps a2t R@10 0.785
|
|
|
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
pipeline_tag: feature-extraction
|
| 6 |
+
library_name: sentence-transformers
|
| 7 |
tags:
|
| 8 |
- embeddings
|
| 9 |
- multimodal
|
|
|
|
| 210 |
<details>
|
| 211 |
<summary>Requirements</summary>
|
| 212 |
|
| 213 |
+
- `fusion_embedding` package: `pip install fusion-embedding[hf]` (the
|
| 214 |
+
Sentence Transformers section below uses the lighter `[sense]` extra)
|
| 215 |
- `transformers>=4.46`, `torch` (CUDA), `torchvision`, `pillow`, `soundfile`, `librosa`
|
| 216 |
- ~14 GB GPU memory at bf16
|
| 217 |
|
|
|
|
| 257 |
|
| 258 |
</details>
|
| 259 |
|
| 260 |
+
## Sentence Transformers
|
| 261 |
+
|
| 262 |
+
fusion-embedding-2 works directly with [Sentence Transformers](https://sbert.net) (v5.5.1+,
|
| 263 |
+
multimodal `encode`). The custom module on this repository is a thin adapter over the
|
| 264 |
+
`fusion-embedding` package, so embeddings are identical to the native
|
| 265 |
+
`fusion_embedding.UnifiedEmbedder.from_pretrained` path.
|
| 266 |
+
|
| 267 |
+
```bash
|
| 268 |
+
pip install -U "sentence-transformers>=5.5.1" "fusion-embedding[sense]>=0.3.0" torchvision
|
| 269 |
+
|
| 270 |
+
# only to embed a video by file path; torchcodec also needs FFmpeg installed
|
| 271 |
+
pip install -U torchcodec
|
| 272 |
+
```
|
| 273 |
+
|
| 274 |
+
```python
|
| 275 |
+
import numpy as np
|
| 276 |
+
from sentence_transformers import SentenceTransformer
|
| 277 |
+
|
| 278 |
+
model = SentenceTransformer(
|
| 279 |
+
"EximiusLabs/fusion-embedding-2-2b-preview",
|
| 280 |
+
trust_remote_code=True,
|
| 281 |
+
)
|
| 282 |
+
|
| 283 |
+
# Text
|
| 284 |
+
text_embs = model.encode([
|
| 285 |
+
"a dog barks in the distance while rain falls on a tin roof",
|
| 286 |
+
"an orchestra tuning up before a performance",
|
| 287 |
+
])
|
| 288 |
+
|
| 289 |
+
# Images: local path, PIL image, or HxWxC uint8 array
|
| 290 |
+
image_embs = model.encode(["photo_of_a_bicycle.jpg"])
|
| 291 |
+
|
| 292 |
+
# Audio: local file path, or a waveform with its sampling rate
|
| 293 |
+
waveform = np.sin(2 * np.pi * 440.0 * np.arange(44100) / 44100).astype(np.float32)
|
| 294 |
+
audio_embs = model.encode([
|
| 295 |
+
"dog_bark.wav",
|
| 296 |
+
{"audio": {"array": waveform, "sampling_rate": 44100}},
|
| 297 |
+
])
|
| 298 |
+
|
| 299 |
+
# Video: local file path (requires torchcodec) or a [T, C, H, W] uint8 frame
|
| 300 |
+
# tensor
|
| 301 |
+
video_embs = model.encode(["clip.mp4"])
|
| 302 |
+
|
| 303 |
+
# One shared space: rank any modality against any other
|
| 304 |
+
scores = model.similarity(text_embs, audio_embs)
|
| 305 |
+
```
|
| 306 |
+
|
| 307 |
+
All vectors are L2-normalized at the full interoperability dimension (2048).
|
| 308 |
+
|
| 309 |
+
### Queries, documents, and instructions
|
| 310 |
+
|
| 311 |
+
The text side carries a retrieval instruction through the base model's chat template.
|
| 312 |
+
`encode` uses the query instruction by default; `encode_query` / `encode_document`
|
| 313 |
+
select the released query/document instructions, and a custom instruction can be
|
| 314 |
+
passed as `prompt=` (it replaces the instruction inside the chat template rather than
|
| 315 |
+
being prepended to the text):
|
| 316 |
+
|
| 317 |
+
```python
|
| 318 |
+
q = model.encode_query(["engine failure sounds"])
|
| 319 |
+
d = model.encode_document(["The compressor stalls with a loud metallic bang."])
|
| 320 |
+
custom = model.encode(["dog"], prompt="Retrieve audio by sound description.")
|
| 321 |
+
```
|
| 322 |
+
|
| 323 |
+
Image, video, and audio inputs use the model's fixed native formats; instructions and
|
| 324 |
+
prompts only affect text.
|
| 325 |
+
|
| 326 |
+
### Matryoshka dimensions
|
| 327 |
+
|
| 328 |
+
The checkpoint was trained with Matryoshka rungs (2048, 1536, 1024, 512, 256, 128, 64).
|
| 329 |
+
Truncate and renormalize to use a shorter rung (this matches the native MRL readout):
|
| 330 |
+
|
| 331 |
+
```python
|
| 332 |
+
vecs = model.encode(
|
| 333 |
+
["a dog barks in the distance"],
|
| 334 |
+
truncate_dim=1024,
|
| 335 |
+
normalize_embeddings=True,
|
| 336 |
+
)
|
| 337 |
+
```
|
| 338 |
+
|
| 339 |
+
### Notes
|
| 340 |
+
|
| 341 |
+
- Revisions: the integration files are on `main`, which is what loads when no
|
| 342 |
+
`revision=` is given. The immutable release tags, `v0.3-preview` among them, predate
|
| 343 |
+
the integration and do not carry these files, so the `revision=` pin shown in the
|
| 344 |
+
`inference.py` example above does not apply here. `main` carries the same flagship
|
| 345 |
+
checkpoint as `v0.3-preview`. For a reproducible load, pin the commit hash of a
|
| 346 |
+
revision that carries the integration files.
|
| 347 |
+
- Each input item is a single modality; embed each modality in its own `encode` call
|
| 348 |
+
(the shared space makes the vectors directly comparable). Fused multi-modality items
|
| 349 |
+
(for example `{"text": ..., "audio": ...}` in one dict) are not part of the released
|
| 350 |
+
model contract and raise an error.
|
| 351 |
+
- Audio arrays must carry a sampling rate (`{"array": ..., "sampling_rate": ...}`);
|
| 352 |
+
any rate is accepted and resampled to 16 kHz exactly as the native path does.
|
| 353 |
+
- Cross-modal ranking of a full gallery benefits from per-modality mean-centering, worth
|
| 354 |
+
a couple of points of R@1; see the "Cross-modal ranking tip" above. The package
|
| 355 |
+
equivalent of that section's `FusionEmbedder.center` is
|
| 356 |
+
`fusion_embedding.UnifiedEmbedder.center`.
|
| 357 |
+
- The first load downloads the frozen base (`Qwen/Qwen3-VL-Embedding-2B`) and the
|
| 358 |
+
frozen audio tower (`Qwen/Qwen2.5-Omni-7B` audio encoder) plus this repository's
|
| 359 |
+
trained connector checkpoint, exactly like the native loader.
|
| 360 |
+
- `model.save(...)` writes configuration only; the 2B weight stack always loads from
|
| 361 |
+
the Hugging Face repositories above, so saved directories stay small and reloads
|
| 362 |
+
fetch weights from the hub cache.
|
| 363 |
+
|
| 364 |
## Changelog
|
| 365 |
|
| 366 |
- **v0.3-preview** (default): retrieval-tuned flagship. AudioCaps a2t R@10 0.785
|