Spaces:
Running
Running
Update org card: add motion/IMU (Tremor), k3-vision, cookbook, and Fusion Perception
Browse files
README.md
CHANGED
|
@@ -9,73 +9,28 @@ pinned: false
|
|
| 9 |
|
| 10 |

|
| 11 |
|
| 12 |
-
|
| 13 |
-
research. Everything here gives developers and researchers a production-ready
|
| 14 |
-
starting point for cross-modal retrieval and semantic search, backed by the
|
| 15 |
-
research behind the Fusion Embedding stack. The technical report is on
|
| 16 |
-
arXiv: https://arxiv.org/abs/2607.18666.
|
| 17 |
|
| 18 |
-
|
| 19 |
|
| 20 |
-
|
| 21 |
-
sense pack, thermal β built on a single design constraint: the base model
|
| 22 |
-
is never retrained. Both generations are
|
| 23 |
-
live on the public MTEB leaderboards for audio (MAEB) and video (MVEB),
|
| 24 |
-
evaluated through the official `mteb` harness anyone can rerun.
|
| 25 |
|
| 26 |
-
|
| 27 |
-
generation 1 trains a 16.4M-parameter connector between a frozen
|
| 28 |
-
Qwen2.5-Omni audio tower and a byte-frozen Qwen3-VL-Embedding-2B base:
|
| 29 |
-
under 1% of the model. AudioCaps audio-to-text R@10 0.741, at the frontier
|
| 30 |
-
of models that keep their base frozen. Aligned on audio-text only, it
|
| 31 |
-
retrieves images from sound at 29x chance with zero paired audio-visual
|
| 32 |
-
training data.
|
| 33 |
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
(
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
text-to-audio results on every benchmark: AudioCaps R@10 0.785, Clotho
|
| 40 |
-
zero-shot 0.485.
|
| 41 |
|
| 42 |
-
|
| 43 |
-
Ember is the thermal sense for fusion-embedding-2 and the first *sense
|
| 44 |
-
pack*: a separately loadable adapter pack that adds a modality while every
|
| 45 |
-
existing vector stays valid. Text-to-thermal retrieval R@10 0.785 on a
|
| 46 |
-
2,000-image gallery against a 0.224 frozen baseline. Sense packs are named
|
| 47 |
-
for the physical trace their sensor reads; Ember reads heat.
|
| 48 |
|
| 49 |
-
|
| 50 |
|
| 51 |
-
|
| 52 |
-
base's input space; gated bottleneck adapters add in-layer capacity per
|
| 53 |
-
modality. Gates are mutually exclusive by input modality, so non-gated
|
| 54 |
-
modalities execute the base's exact computation. A single forward pass
|
| 55 |
-
yields nested Matryoshka embeddings truncatable to
|
| 56 |
-
{2048, 1536, 1024, 512, 256, 128, 64} dimensions. Training runs in hours on a single
|
| 57 |
-
GPU; connectors and packs ship as small artifacts with the frozen towers
|
| 58 |
-
downloaded separately.
|
| 59 |
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
```
|
| 64 |
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
The audio corpus is ~500K open audio-caption pairs (AudioCaps and
|
| 68 |
-
AudioCaps 2.0, FSD50K, WavCaps/AudioSet-SL, a LAION-FreeSound subset);
|
| 69 |
-
Ember trains on IR-TD real thermal imagery with restricted sources removed.
|
| 70 |
-
Evaluation sets are excluded from training by clip id at ingestion, and
|
| 71 |
-
every reported number traces to a result record in the open repository.
|
| 72 |
-
|
| 73 |
-
## Status
|
| 74 |
-
|
| 75 |
-
English-language research previews. Model weights are released under
|
| 76 |
-
CC-BY-NC-4.0 (Ember under its own research license; see each model card).
|
| 77 |
-
Sound-event retrieval is the strongest axis; speech and music are
|
| 78 |
-
undertrained in the current corpus. We release early to invite research and
|
| 79 |
-
feedback on unified multimodal embedding spaces.
|
| 80 |
-
|
| 81 |
-
[eximiuslabs.com](https://eximiuslabs.com/) Β· [github.com/Eximius-Labs](https://github.com/Eximius-Labs)
|
|
|
|
| 9 |
|
| 10 |

|
| 11 |
|
| 12 |
+
Eximius Labs builds open-weight multimodal models that run on your own hardware: the memory and perception layer for physical AI. The technical report is on arXiv: https://arxiv.org/abs/2607.18666.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
+
We freeze a strong base model and add new senses from the outside, so existing embeddings stay valid and gaining a modality never forces a re-index. One shared vector space spans text, image, video, audio, thermal, and motion (IMU).
|
| 15 |
|
| 16 |
+
## Fusion Embedding, the memory layer
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
+
One vector space for cross-modal retrieval, RAG, clustering, and semantic search.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
+
- [fusion-embedding-2](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview) β text, image, video, and audio in one space (connector plus modality-gated adapters).
|
| 21 |
+
- [fusion-embedding-1](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview) β the connector-only first generation.
|
| 22 |
+
- [Ember](https://huggingface.co/EximiusLabs/fusion-embedding-2-ember) β thermal and infrared, as a loadable sense pack.
|
| 23 |
+
- [Tremor](https://huggingface.co/EximiusLabs/fusion-embedding-2-tremor) β body-worn and robot motion (IMU and accelerometer), read as language, with a [Unitree-G1 head](https://huggingface.co/EximiusLabs/fusion-embedding-2-tremor-g1).
|
| 24 |
+
- [fusion-embedding-2-k3-vision](https://huggingface.co/EximiusLabs/fusion-embedding-2-k3-vision) β Kimi K3's vision encoder, projected into the shared space.
|
|
|
|
|
|
|
| 25 |
|
| 26 |
+
## Fusion Perception, the perception layer
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
+
Dense scene understanding and geometric place recognition on a frozen vision backbone, projected into the Fusion Embedding space. Code: https://github.com/Eximius-Labs/fusion-perception.
|
| 29 |
|
| 30 |
+
## Build with it
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
+
- Cookbook: https://github.com/Eximius-Labs/cookbook
|
| 33 |
+
- Code: https://github.com/Eximius-Labs
|
| 34 |
+
- Paper: https://arxiv.org/abs/2607.18666
|
|
|
|
| 35 |
|
| 36 |
+
Everything is open weight and self-hostable. Core models are research preview; the sensor packs are CC-BY-NC-4.0.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|