---
license: apache-2.0
base_model:
- thinkingmachines/Inkling-Small
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- amd-quark
- mxfp4
- rocm
- vllm
- inkling
---

# Inkling-Small-MXFP4

## Model Overview

- **Model architecture:** [Thinking Machines Lab Inkling-Small](https://huggingface.co/thinkingmachines/Inkling-Small)
  - **Parameters:** 276B total / 12B active
  - **Input:** Text, image, audio
  - **Output:** Text
- **Validated inference engine:** [vLLM](https://github.com/vllm-project/vllm)
- **Model optimizer:** [AMD Quark](https://github.com/amd/quark) (`0.12.post1+rocm72.torch2.11`)
  - **Quantized layers:** MoE routed experts in transformer layers 3 through 41
  - **Weight quantization:** OCP MXFP4, static, group size 32, E8M0 scales
  - **Activation quantization:** OCP MXFP4, dynamic, group size 32, E8M0 scales

This checkpoint was built from `thinkingmachines/Inkling-Small` revision
`b2d4f225a02032c5d154bff748ab5a00c5ca26e4` by applying AMD Quark OCP MXFP4
quantization to the BF16 routed experts. Routed-expert weights are stored as
packed MXFP4 weights with E8M0 scales. Dense layers 0 through 2, attention,
shared experts, embeddings, norms, the audio and vision towers, MTP, and other
non-routed components remain in their source formats.

## Environment

The file-to-file conversion and validation targeted AMD `gfx950` and used:
- **Container:** `docker.io/rocm/vllm-dev:nightly_main_20260714`
- **Python:** 3.12
- **ROCm/HIP:** 7.2
- **PyTorch:** `2.11.0+gitd0c8b1f`
- **AMD Quark:** `0.12.post1+rocm72.torch2.11`
- **Expert chunk size:** 8

The paired quality evaluation used the same GPU type and TP8 topology, with
Transformers `5.14.1` and vLLM commit
`846e2d01a0be00acf31f1a354059c7c302c93042`
(`0.23.1rc1.dev1212+g846e2d01a`).

## Evaluation

| Benchmark | BF16 Reference | MXFP4 | MXFP4 − BF16 |
| --- | ---: | ---: | ---: |
| BFCL exact calls | 76.54% (1,034/1,351) | 76.76% (1,037/1,351) | +0.22 pp |
| BFCL all-live macro | 76.56% | 71.01% | −5.55 pp |
| MMAU (official string match) | 75.5% (755/1,000) | 76.3% (763/1,000) | +0.80 pp |
| GPQA Diamond | 89.19% (883/990) | 87.98% (871/990) | −1.21 pp |
| AIME 2026 | 94.58% (908/960) | 94.58% (908/960) | 0.00 pp |

Refer to the [Inkling-Small model card](https://huggingface.co/thinkingmachines/Inkling-Small)
for architecture, training, intended-use, safety, and acceptable-use details.