File size: 3,552 Bytes
08c77f0 4c88f31 08c77f0 4c88f31 08c77f0 4c88f31 08c77f0 4c88f31 08c77f0 4c88f31 08c77f0 4c88f31 08c77f0 4c88f31 08c77f0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | ---
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
- thinkingmachines/Inkling-2-Small
---
This is a randomly initialized tiny model based on [thinkingmachines/Inkling-2-Small](https://huggingface.co/thinkingmachines/Inkling-2-Small) for testing and debugging.
| Feature | Source | Tiny |
|---|---|---|
| Checkpoint tensor elements (including packed weights/scales) | 8.69B BF16, 565.33K F32, 128.42B F8_E4M3, 247.67B U8 | 16.40M BF16, 3.21K F32, 1.88M F8_E4M3, 940.03K U8 |
| Safetensors weight files (including MTP) | 393.48 GB (25 files) | 35.68 MB (2 files) |
| Decoder layers / attention residual block | 48 / 6 | 12 / 6 |
| Attention cycle | 5 SWA + 1 global | Same; two complete cycles |
| Routed experts / experts per token | 384 / 8 | 96 / 8 |
| Expert precision | 40 NVFP4 + 8 MXFP8 layers | 6 NVFP4 + 6 MXFP8 layers |
| Vocabulary / tied embeddings | 201,024 / false | 201,024 / false |
| Hidden / expert width | 4,608 / 2,304 | 32 / 32 |
| Q heads / KV heads / head dimension | 48 / 8 / 128 | 24 / 4 / 32 |
| Vision width / layers | 1,024 / 23 | 32 / 4 |
| MTP layers / block size / Markov rank | 5 / 8 / 256 | 5 / 8 / 8 |
| MTP feature taps | 5,11,17,23,29,35,41,48 | 5,12 (final norm sentinel) |
| File path | Size |
|------|------|
| model.safetensors | 31.7MB |
| mtp.safetensors | 4.0MB |
### Example usage
#### Transformers (unverified)
Install [Transformers with Inkling 2 support](https://github.com/huggingface/transformers/tree/c748111b3bb4d97df452f0fe60e63c480beb6fe6). The bundled loader expands packed expert weights to BF16.
```python
def run_transformers(model_id: str):
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
model_id, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
)
image = Image.new("RGB", (80, 80), (80, 120, 160))
inputs = processor(text=f"{processor.image_token}Describe this image.", images=image, return_tensors="pt").to(
model.device
)
with torch.inference_mode():
output = model.generate(**inputs, max_new_tokens=10, min_new_tokens=10, do_sample=False)
print(processor.batch_decode(output[:, inputs.input_ids.shape[1] :], skip_special_tokens=True))
if __name__ == "__main__":
run_transformers(model_id="tiny-random/inkling-2")
```
#### SGLang serving (unverified)
Use the pinned SGLang version and the bundled small-matrix kernel plugin.
```bash
SGLANG_BUILD_RUST_EXTS=none pip install 'git+https://github.com/sgl-project/sglang.git@f726172d760dea737c7f1e797c9f32efe14d1685#subdirectory=python'
hf download tiny-random/inkling-2 --local-dir inkling-2-tiny
pip install --no-deps ./inkling-2-tiny
sglang serve --model-path ./inkling-2-tiny --tp 4 --ep 4 \
--dtype bfloat16 --attention-backend fa4 --enable-multimodal \
--context-length 2048 --mem-fraction-static 0.2 \
--disable-custom-all-reduce \
--cuda-graph-config '{"decode": {"bs": [1, 2, 4]}, "prefill": {"bs": [128, 2048]}}' \
--reasoning-parser inkling --tool-call-parser inkling2
```
### Creating this repo
- Code: [.meta/create_model.py](.meta/create_model.py)
- Model structure: [.meta/model.log](.meta/model.log)
- Environment: [.meta/env.json](.meta/env.json)
### Change log
- 2026-11-14: Updated tiny Inkling 2 with 96 mixed NVFP4/MXFP8 experts, independent token embeddings/output head, and BF16 MTP.
*This repo is generated by GPT-6.1-Sol* |