abtonmoy commited on
Commit
f496b0e
·
verified ·
1 Parent(s): f18df08

Model card: add Sentence Transformers usage section

Browse files
Files changed (1) hide show
  1. README.md +107 -1
README.md CHANGED
@@ -3,6 +3,7 @@ license: cc-by-nc-4.0
3
  language:
4
  - en
5
  pipeline_tag: feature-extraction
 
6
  tags:
7
  - embeddings
8
  - multimodal
@@ -209,7 +210,8 @@ default.
209
  <details>
210
  <summary>Requirements</summary>
211
 
212
- - `fusion_embedding` package: `pip install fusion-embedding[hf]`
 
213
  - `transformers>=4.46`, `torch` (CUDA), `torchvision`, `pillow`, `soundfile`, `librosa`
214
  - ~14 GB GPU memory at bf16
215
 
@@ -255,6 +257,110 @@ gallery = FusionEmbedder.center(gallery_embeddings)
255
 
256
  </details>
257
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
258
  ## Changelog
259
 
260
  - **v0.3-preview** (default): retrieval-tuned flagship. AudioCaps a2t R@10 0.785
 
3
  language:
4
  - en
5
  pipeline_tag: feature-extraction
6
+ library_name: sentence-transformers
7
  tags:
8
  - embeddings
9
  - multimodal
 
210
  <details>
211
  <summary>Requirements</summary>
212
 
213
+ - `fusion_embedding` package: `pip install fusion-embedding[hf]` (the
214
+ Sentence Transformers section below uses the lighter `[sense]` extra)
215
  - `transformers>=4.46`, `torch` (CUDA), `torchvision`, `pillow`, `soundfile`, `librosa`
216
  - ~14 GB GPU memory at bf16
217
 
 
257
 
258
  </details>
259
 
260
+ ## Sentence Transformers
261
+
262
+ fusion-embedding-2 works directly with [Sentence Transformers](https://sbert.net) (v5.5.1+,
263
+ multimodal `encode`). The custom module on this repository is a thin adapter over the
264
+ `fusion-embedding` package, so embeddings are identical to the native
265
+ `fusion_embedding.UnifiedEmbedder.from_pretrained` path.
266
+
267
+ ```bash
268
+ pip install -U "sentence-transformers>=5.5.1" "fusion-embedding[sense]>=0.3.0" torchvision
269
+
270
+ # only to embed a video by file path; torchcodec also needs FFmpeg installed
271
+ pip install -U torchcodec
272
+ ```
273
+
274
+ ```python
275
+ import numpy as np
276
+ from sentence_transformers import SentenceTransformer
277
+
278
+ model = SentenceTransformer(
279
+ "EximiusLabs/fusion-embedding-2-2b-preview",
280
+ trust_remote_code=True,
281
+ )
282
+
283
+ # Text
284
+ text_embs = model.encode([
285
+ "a dog barks in the distance while rain falls on a tin roof",
286
+ "an orchestra tuning up before a performance",
287
+ ])
288
+
289
+ # Images: local path, PIL image, or HxWxC uint8 array
290
+ image_embs = model.encode(["photo_of_a_bicycle.jpg"])
291
+
292
+ # Audio: local file path, or a waveform with its sampling rate
293
+ waveform = np.sin(2 * np.pi * 440.0 * np.arange(44100) / 44100).astype(np.float32)
294
+ audio_embs = model.encode([
295
+ "dog_bark.wav",
296
+ {"audio": {"array": waveform, "sampling_rate": 44100}},
297
+ ])
298
+
299
+ # Video: local file path (requires torchcodec) or a [T, C, H, W] uint8 frame
300
+ # tensor
301
+ video_embs = model.encode(["clip.mp4"])
302
+
303
+ # One shared space: rank any modality against any other
304
+ scores = model.similarity(text_embs, audio_embs)
305
+ ```
306
+
307
+ All vectors are L2-normalized at the full interoperability dimension (2048).
308
+
309
+ ### Queries, documents, and instructions
310
+
311
+ The text side carries a retrieval instruction through the base model's chat template.
312
+ `encode` uses the query instruction by default; `encode_query` / `encode_document`
313
+ select the released query/document instructions, and a custom instruction can be
314
+ passed as `prompt=` (it replaces the instruction inside the chat template rather than
315
+ being prepended to the text):
316
+
317
+ ```python
318
+ q = model.encode_query(["engine failure sounds"])
319
+ d = model.encode_document(["The compressor stalls with a loud metallic bang."])
320
+ custom = model.encode(["dog"], prompt="Retrieve audio by sound description.")
321
+ ```
322
+
323
+ Image, video, and audio inputs use the model's fixed native formats; instructions and
324
+ prompts only affect text.
325
+
326
+ ### Matryoshka dimensions
327
+
328
+ The checkpoint was trained with Matryoshka rungs (2048, 1536, 1024, 512, 256, 128, 64).
329
+ Truncate and renormalize to use a shorter rung (this matches the native MRL readout):
330
+
331
+ ```python
332
+ vecs = model.encode(
333
+ ["a dog barks in the distance"],
334
+ truncate_dim=1024,
335
+ normalize_embeddings=True,
336
+ )
337
+ ```
338
+
339
+ ### Notes
340
+
341
+ - Revisions: the integration files are on `main`, which is what loads when no
342
+ `revision=` is given. The immutable release tags, `v0.3-preview` among them, predate
343
+ the integration and do not carry these files, so the `revision=` pin shown in the
344
+ `inference.py` example above does not apply here. `main` carries the same flagship
345
+ checkpoint as `v0.3-preview`. For a reproducible load, pin the commit hash of a
346
+ revision that carries the integration files.
347
+ - Each input item is a single modality; embed each modality in its own `encode` call
348
+ (the shared space makes the vectors directly comparable). Fused multi-modality items
349
+ (for example `{"text": ..., "audio": ...}` in one dict) are not part of the released
350
+ model contract and raise an error.
351
+ - Audio arrays must carry a sampling rate (`{"array": ..., "sampling_rate": ...}`);
352
+ any rate is accepted and resampled to 16 kHz exactly as the native path does.
353
+ - Cross-modal ranking of a full gallery benefits from per-modality mean-centering, worth
354
+ a couple of points of R@1; see the "Cross-modal ranking tip" above. The package
355
+ equivalent of that section's `FusionEmbedder.center` is
356
+ `fusion_embedding.UnifiedEmbedder.center`.
357
+ - The first load downloads the frozen base (`Qwen/Qwen3-VL-Embedding-2B`) and the
358
+ frozen audio tower (`Qwen/Qwen2.5-Omni-7B` audio encoder) plus this repository's
359
+ trained connector checkpoint, exactly like the native loader.
360
+ - `model.save(...)` writes configuration only; the 2B weight stack always loads from
361
+ the Hugging Face repositories above, so saved directories stay small and reloads
362
+ fetch weights from the hub cache.
363
+
364
  ## Changelog
365
 
366
  - **v0.3-preview** (default): retrieval-tuned flagship. AudioCaps a2t R@10 0.785