abtonmoy commited on
Commit
304ae78
Β·
verified Β·
1 Parent(s): 5adec67

Update org card: add motion/IMU (Tremor), k3-vision, cookbook, and Fusion Perception

Browse files
Files changed (1) hide show
  1. README.md +16 -61
README.md CHANGED
@@ -9,73 +9,28 @@ pinned: false
9
 
10
  ![Eximius Labs](assets/banner.png)
11
 
12
- This is Eximius Labs' home for open model weights and multimodal embedding
13
- research. Everything here gives developers and researchers a production-ready
14
- starting point for cross-modal retrieval and semantic search, backed by the
15
- research behind the Fusion Embedding stack. The technical report is on
16
- arXiv: https://arxiv.org/abs/2607.18666.
17
 
18
- ## The Fusion Embedding family
19
 
20
- One embedding space for text, images, video, audio β€” and, with its first
21
- sense pack, thermal β€” built on a single design constraint: the base model
22
- is never retrained. Both generations are
23
- live on the public MTEB leaderboards for audio (MAEB) and video (MVEB),
24
- evaluated through the official `mteb` harness anyone can rerun.
25
 
26
- **[fusion-embedding-1-2b-preview](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview)** (v0.3) β€”
27
- generation 1 trains a 16.4M-parameter connector between a frozen
28
- Qwen2.5-Omni audio tower and a byte-frozen Qwen3-VL-Embedding-2B base:
29
- under 1% of the model. AudioCaps audio-to-text R@10 0.741, at the frontier
30
- of models that keep their base frozen. Aligned on audio-text only, it
31
- retrieves images from sound at 29x chance with zero paired audio-visual
32
- training data.
33
 
34
- **[fusion-embedding-2-2b-preview](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview)** (v0.2) β€”
35
- generation 2 adds modality-gated adapters inside the frozen decoder layers
36
- (+44.2M parameters), opened only for audio inputs. For text, images, and
37
- video the adapters never execute and outputs are bit-for-bit identical to
38
- the base model, verified after every training run. The family's best
39
- text-to-audio results on every benchmark: AudioCaps R@10 0.785, Clotho
40
- zero-shot 0.485.
41
 
42
- **[fusion-embedding-2-ember](https://huggingface.co/EximiusLabs/fusion-embedding-2-ember)** (v0.1) β€”
43
- Ember is the thermal sense for fusion-embedding-2 and the first *sense
44
- pack*: a separately loadable adapter pack that adds a modality while every
45
- existing vector stays valid. Text-to-thermal retrieval R@10 0.785 on a
46
- 2,000-image gallery against a 0.224 frozen baseline. Sense packs are named
47
- for the physical trace their sensor reads; Ember reads heat.
48
 
49
- ## How it works
50
 
51
- A perceiver-resampler connector maps frozen audio-tower features into the
52
- base's input space; gated bottleneck adapters add in-layer capacity per
53
- modality. Gates are mutually exclusive by input modality, so non-gated
54
- modalities execute the base's exact computation. A single forward pass
55
- yields nested Matryoshka embeddings truncatable to
56
- {2048, 1536, 1024, 512, 256, 128, 64} dimensions. Training runs in hours on a single
57
- GPU; connectors and packs ship as small artifacts with the frozen towers
58
- downloaded separately.
59
 
60
- ```python
61
- import mteb
62
- model = mteb.get_model("EximiusLabs/fusion-embedding-2-2b-preview")
63
- ```
64
 
65
- ## Training data
66
-
67
- The audio corpus is ~500K open audio-caption pairs (AudioCaps and
68
- AudioCaps 2.0, FSD50K, WavCaps/AudioSet-SL, a LAION-FreeSound subset);
69
- Ember trains on IR-TD real thermal imagery with restricted sources removed.
70
- Evaluation sets are excluded from training by clip id at ingestion, and
71
- every reported number traces to a result record in the open repository.
72
-
73
- ## Status
74
-
75
- English-language research previews. Model weights are released under
76
- CC-BY-NC-4.0 (Ember under its own research license; see each model card).
77
- Sound-event retrieval is the strongest axis; speech and music are
78
- undertrained in the current corpus. We release early to invite research and
79
- feedback on unified multimodal embedding spaces.
80
-
81
- [eximiuslabs.com](https://eximiuslabs.com/) Β· [github.com/Eximius-Labs](https://github.com/Eximius-Labs)
 
9
 
10
  ![Eximius Labs](assets/banner.png)
11
 
12
+ Eximius Labs builds open-weight multimodal models that run on your own hardware: the memory and perception layer for physical AI. The technical report is on arXiv: https://arxiv.org/abs/2607.18666.
 
 
 
 
13
 
14
+ We freeze a strong base model and add new senses from the outside, so existing embeddings stay valid and gaining a modality never forces a re-index. One shared vector space spans text, image, video, audio, thermal, and motion (IMU).
15
 
16
+ ## Fusion Embedding, the memory layer
 
 
 
 
17
 
18
+ One vector space for cross-modal retrieval, RAG, clustering, and semantic search.
 
 
 
 
 
 
19
 
20
+ - [fusion-embedding-2](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview) β€” text, image, video, and audio in one space (connector plus modality-gated adapters).
21
+ - [fusion-embedding-1](https://huggingface.co/EximiusLabs/fusion-embedding-1-2b-preview) β€” the connector-only first generation.
22
+ - [Ember](https://huggingface.co/EximiusLabs/fusion-embedding-2-ember) β€” thermal and infrared, as a loadable sense pack.
23
+ - [Tremor](https://huggingface.co/EximiusLabs/fusion-embedding-2-tremor) β€” body-worn and robot motion (IMU and accelerometer), read as language, with a [Unitree-G1 head](https://huggingface.co/EximiusLabs/fusion-embedding-2-tremor-g1).
24
+ - [fusion-embedding-2-k3-vision](https://huggingface.co/EximiusLabs/fusion-embedding-2-k3-vision) β€” Kimi K3's vision encoder, projected into the shared space.
 
 
25
 
26
+ ## Fusion Perception, the perception layer
 
 
 
 
 
27
 
28
+ Dense scene understanding and geometric place recognition on a frozen vision backbone, projected into the Fusion Embedding space. Code: https://github.com/Eximius-Labs/fusion-perception.
29
 
30
+ ## Build with it
 
 
 
 
 
 
 
31
 
32
+ - Cookbook: https://github.com/Eximius-Labs/cookbook
33
+ - Code: https://github.com/Eximius-Labs
34
+ - Paper: https://arxiv.org/abs/2607.18666
 
35
 
36
+ Everything is open weight and self-hostable. Core models are research preview; the sensor packs are CC-BY-NC-4.0.