File size: 2,408 Bytes
06b2c86 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | ---
license: apache-2.0
base_model:
- thinkingmachines/Inkling-Small
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- amd-quark
- mxfp4
- rocm
- vllm
- inkling
---
# Inkling-Small-MXFP4
## Model Overview
- **Model architecture:** [Thinking Machines Lab Inkling-Small](https://huggingface.co/thinkingmachines/Inkling-Small)
- **Parameters:** 276B total / 12B active
- **Input:** Text, image, audio
- **Output:** Text
- **Validated inference engine:** [vLLM](https://github.com/vllm-project/vllm)
- **Model optimizer:** [AMD Quark](https://github.com/amd/quark) (`0.12.post1+rocm72.torch2.11`)
- **Quantized layers:** MoE routed experts in transformer layers 3 through 41
- **Weight quantization:** OCP MXFP4, static, group size 32, E8M0 scales
- **Activation quantization:** OCP MXFP4, dynamic, group size 32, E8M0 scales
This checkpoint was built from `thinkingmachines/Inkling-Small` revision
`b2d4f225a02032c5d154bff748ab5a00c5ca26e4` by applying AMD Quark OCP MXFP4
quantization to the BF16 routed experts. Routed-expert weights are stored as
packed MXFP4 weights with E8M0 scales. Dense layers 0 through 2, attention,
shared experts, embeddings, norms, the audio and vision towers, MTP, and other
non-routed components remain in their source formats.
## Environment
The file-to-file conversion and validation targeted AMD `gfx950` and used:
- **Container:** `docker.io/rocm/vllm-dev:nightly_main_20260714`
- **Python:** 3.12
- **ROCm/HIP:** 7.2
- **PyTorch:** `2.11.0+gitd0c8b1f`
- **AMD Quark:** `0.12.post1+rocm72.torch2.11`
- **Expert chunk size:** 8
The paired quality evaluation used the same GPU type and TP8 topology, with
Transformers `5.14.1` and vLLM commit
`846e2d01a0be00acf31f1a354059c7c302c93042`
(`0.23.1rc1.dev1212+g846e2d01a`).
## Evaluation
| Benchmark | BF16 Reference | MXFP4 | MXFP4 − BF16 |
| --- | ---: | ---: | ---: |
| BFCL exact calls | 76.54% (1,034/1,351) | 76.76% (1,037/1,351) | +0.22 pp |
| BFCL all-live macro | 76.56% | 71.01% | −5.55 pp |
| MMAU (official string match) | 75.5% (755/1,000) | 76.3% (763/1,000) | +0.80 pp |
| GPQA Diamond | 89.19% (883/990) | 87.98% (871/990) | −1.21 pp |
| AIME 2026 | 94.58% (908/960) | 94.58% (908/960) | 0.00 pp |
Refer to the [Inkling-Small model card](https://huggingface.co/thinkingmachines/Inkling-Small)
for architecture, training, intended-use, safety, and acceptable-use details. |