README.md

---
license: other
license_name: kimi-k3
license_link: LICENSE
pipeline_tag: image-feature-extraction
library_name: transformers
tags:
- vision
base_model:
- moonshotai/Kimi-K3
---

# MoonViT-V2

**MoonViT-V2** is the native-resolution vision backbone of [Kimi K3](https://huggingface.co/moonshotai/Kimi-K3).

| Attribute | Specification |
| --- | --- |
| **Parameters** | ~0.4B |
| **Architecture** | 27-layer ViT · 1024 hidden · 12 heads · 1536 QKV dim |
| **Patch Size** | 14 × 14 |
| **Token Compression** | 2×2 pixel-shuffle + temporal pooling |
| **Max Resolution** | ~3584 × 3584 |
| **Initialization** | From scratch (not SigLIP) |
| **Source Model** | [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) |
| **Technical Report** | [Kimi K3 (arXiv:2607.24653)](https://arxiv.org/abs/2607.24653) |

## Overview

MoonViT-V2 encodes images and videos into visual tokens for Kimi K3’s shared multimodal embedding space (via a lightweight MLP projector in the full checkpoint).

Unlike [MoonViT-SO-400M](https://huggingface.co/moonshotai/MoonViT-SO-400M) (SigLIP-initialized), **MoonViT-V2 is randomly initialized and jointly trained** with the language model under next-token prediction.

- **Unified image/video path** — shared parameters with factorized spatial (intra-frame) and temporal (inter-frame) attention.
- **Token compression** — 2×2 pixel-shuffle (~4× fewer spatial tokens) plus temporal pooling, for high-res images and long videos within Kimi K3’s up-to-1M-token context.
- **Training stability** — RMSNorm and bias-free linear/attention projections; from-scratch training avoided gradient spikes seen with contrastive vision initializations at this scale.

## Usage

```python
import torch
from transformers import AutoModel, AutoImageProcessor

model_id = "AI4Industry/MoonViT-V2"

model = AutoModel.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    trust_remote_code=True,
)
processor = AutoImageProcessor.from_pretrained(model_id, trust_remote_code=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
```

### Image input

```python
from PIL import Image

image = Image.open("your_image.png").convert("RGB")
inputs = processor(image, return_tensors="pt")  # T=1
pixel_values = inputs["pixel_values"].to(device=device, dtype=model.dtype)
grid_thws = inputs["grid_thws"].to(device=device)

with torch.no_grad():
    image_features = model(pixel_values, grid_thws)

print(image_features[0].dtype, image_features[0].shape)
# e.g. torch.bfloat16, torch.Size([N, 4, 1024])
```

### Video input

Pass an ordered list of frames. All frames share the first frame’s resize/pad so spatial grids align. `T` must be ≤ `max_num_frames` (default **4**, matching temporal position embedding).

```python
from PIL import Image

frames = [
    Image.open("frame_0.png").convert("RGB"),
    Image.open("frame_1.png").convert("RGB"),
    Image.open("frame_2.png").convert("RGB"),
    Image.open("frame_3.png").convert("RGB"),
]
inputs = processor.preprocess_video(frames, return_tensors="pt")  # T=len(frames)
pixel_values = inputs["pixel_values"].to(device=device, dtype=model.dtype)
grid_thws = inputs["grid_thws"].to(device=device)  # [[T, H_patches, W_patches]]

with torch.no_grad():
    video_features = model(pixel_values, grid_thws)

print(grid_thws.tolist(), video_features[0].shape)
# e.g. [[4, H, W]], torch.Size([N, 4, 1024])  # N after temporal pool + 2×2 merge
```

### Notes

- `pixel_values`: packed patches `(num_patches, 3, 14, 14)`
- `grid_thws`: `(batch, 3)` = `(T, H_patches, W_patches)`; images use `T=1`
- Normalization: mean/std `(0.5, 0.5, 0.5)`
- Optional: `attn_implementation="flash_attention_2"` if [FlashAttention-2](https://github.com/Dao-AILab/flash-attention) is installed (default is SDPA)

## Weight Provenance

Weights (~401M params, `bfloat16`) are taken from `model-00096-of-000096.safetensors` in [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3).

| Original Kimi K3 Prefix | This Repo |
| --- | --- |
| `vision_tower.*` | `*` (prefix stripped) |
| `mm_projector.*` | omitted |
| `language_model.*` | omitted |

## Training Data & Infrastructure

Per the [technical report](https://arxiv.org/abs/2607.24653), pretraining covers captions, interleaved documents, OCR, video, visual code, grounding (absolute + normalized coords), SVG, 3D assets, web pages, games, and CAD. Training uses context-parallel KV gather, image sharding across devices, and hiding ViT compute in pipeline bubbles. Typical use cases: OCR, visual UI iteration, and video understanding.

## License

Inherits the [Kimi K3 License](https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE)