README.md

---
license: cc-by-nc-4.0
base_model: moonshotai/Kimi-K3
base_model_relation: adapter
library_name: fusion-embedding
pipeline_tag: feature-extraction
tags:
  - multimodal
  - embeddings
  - retrieval
  - vision
  - kimi-k3
  - moonvit
---

<img src="assets/k3-vision-banner.png" alt="fusion-embedding-2-k3-vision — Eximius Labs" width="100%">

# fusion-embedding-2-k3-vision

Kimi K3's vision encoder, made language-searchable.

<div align="center">

[![Base model](https://img.shields.io/badge/base%20model-Kimi%20K3-8A2BE2)](https://huggingface.co/moonshotai/Kimi-K3) [![Family](https://img.shields.io/badge/%F0%9F%A4%97%20family-fusion--embedding--2-yellow)](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview) [![License](https://img.shields.io/badge/license-CC--BY--NC--4.0-blue)](https://creativecommons.org/licenses/by-nc/4.0/) [![Cookbook](https://img.shields.io/badge/GitHub-cookbook-181717?logo=github)](https://github.com/Eximius-Labs/cookbook) [![arXiv](https://img.shields.io/badge/arXiv-2607.18666-b31b1b)](https://arxiv.org/abs/2607.18666)

[![Deploy on RunPod](https://api.runpod.io/badge/Eximius-Labs/k3-vision)](https://www.runpod.io/console/hub/Eximius-Labs/k3-vision)

</div>

This is a small trained projector that maps Kimi K3's frozen vision encoder
(**MoonViT-V2**, about 401M parameters) into the frozen
[fusion-embedding-2](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview) text
space. K3's own visual features become searchable in plain language, and land in the same
space as fusion-embedding's text, image, video, and audio, plus its thermal (Ember) and
motion (Tremor) sensor packs.

It builds on **only the ~401M vision tower of Kimi K3, never the 2.8T language model**. The
tower runs on a single GPU. The projector is the only trained weight, about 3.15M
parameters.

## Results

Text-to-image retrieval on Flickr30k (test), 1000-image gallery, querying in the
fusion-embedding-2 text space:

| Metric | fusion-embedding-2-k3-vision | same recipe, DINOv2 backbone |
| --- | --- | --- |
| R@1 | 0.432 | 0.318 |
| R@5 | 0.740 | 0.613 |
| **R@10** | **0.836** | 0.749 |

MoonViT-V2 was trained alongside a language model, so its features align to text more
readily than a self-supervised vision backbone. On the identical projector recipe it beats
the DINOv2 version by nearly 9 points at R@10.

## Usage

The projector ships here; the vision tower is pulled from `moonshotai/Kimi-K3` on first use
(only the `vision_tower.*` tensors, never the 2.8T language model).

```python
from inference import K3VisionEmbedder
m = K3VisionEmbedder.from_pretrained("EximiusLabs/fusion-embedding-2-k3-vision")
v = m.embed_image("photo.jpg")             # 2048-d, in the fusion-embedding-2 text space
```

Query it with text using fusion-embedding-2 itself:

```python
from inference import FusionEmbedder      # from the fusion-embedding-2 repo
fe = FusionEmbedder.from_pretrained("EximiusLabs/fusion-embedding-2-2b-preview")
q = fe.embed_text("a dog on a beach")
score = float(q @ v)                       # cosine, both unit-norm
```

## How it works

1. Kimi K3's MoonViT-V2 vision tower (frozen) encodes an image into patch tokens (1024-d).
2. The tokens are pooled into one image descriptor.
3. A trained projector (`LayerNorm -> Linear -> GELU -> Dropout -> Linear`) maps that
   descriptor into the 2048-d fusion-embedding-2 text space.
4. Trained with image-caption InfoNCE on Flickr30k. Same recipe as the Fusion Perception
   DINO to fusion-embedding projector, with K3's vision tower as the backbone.

## Composability

K3's projected vision does not sit alone. It shares the fusion-embedding-2 space with the Ember
thermal pack and the Tremor motion pack, so one text query reaches across senses in a single index.
Below, each query returns its top match from vision, thermal, and motion at once. Vision and thermal
retrieve cleanly (ask for "a running dog" and it returns a photo of a running dog and a thermal image
of a dog); motion (Tremor) is in the same space, though its text-to-motion retrieval is research
preview.

![Composability demo: one query, the top match per sense](assets/k3-vision-composability-demo.png)

Real data throughout: Flickr photos (vision, via this model), thermal infrared (Ember), recorded IMU
(Tremor).

## What this is and is not

- It is a projector on a frozen vision encoder, not a new foundation model, and not a
  fine-tune of Kimi K3's language model.
- The `base_model` tag points at `moonshotai/Kimi-K3` because the vision tower is a component
  of that model. We use only MoonViT-V2, not the 2.8T language model.
- It is a research preview.

## Deploy on RunPod

One-click from the [RunPod Hub](https://www.runpod.io/console/hub/Eximius-Labs/k3-vision).

```bash
curl -s https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync \n  -H "Authorization: Bearer $RUNPOD_API_KEY" \n  -H "Content-Type: application/json" \n  -d '{"input": {"image": "<https url | data-uri | base64>"}}'
```

Kimi-K3 is gated: set `HF_TOKEN` (with the K3 license accepted) on the endpoint. Returns 2048-d in the fusion-embedding-2 text space; query with text via the fusion-embedding-2 endpoint.

## License and attribution

The projector weights and code in this repository are released under **CC-BY-NC-4.0**.

The vision tower is part of **Kimi K3** (Moonshot AI), used under the **Kimi K3 License**,
which permits derivative works with its notice retained. This model is a derivative that
uses K3's frozen MoonViT-V2 vision encoder. Copyright 2026 Moonshot AI.

## Family

- [fusion-embedding-2](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview) (the base multimodal embedding model)
- [Ember](https://huggingface.co/EximiusLabs/fusion-embedding-2-ember) (thermal) and [Tremor](https://huggingface.co/EximiusLabs/fusion-embedding-2-tremor) (motion) sensor packs
- Cookbook: [give a Kimi K3 agent a multimodal memory](https://github.com/Eximius-Labs/cookbook)
- Code and paper: [github.com/Eximius-Labs](https://github.com/Eximius-Labs), [arXiv:2607.18666](https://arxiv.org/abs/2607.18666)