---
license: apache-2.0
base_model:
- thinkingmachines/Inkling-Small
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- amd-quark
- mxfp4
- rocm
- tokenspeed
---

# Inkling-Small-MXFP4

## Model Overview

- **Base model:** [thinkingmachines/Inkling-Small](https://huggingface.co/thinkingmachines/Inkling-Small)
- **Architecture:** 42-layer multimodal sparse Mixture-of-Experts transformer
  - **Parameters:** 276B total, 12B active
  - **Input:** Text, image, and audio
  - **Output:** Text
- **Inference engine:** [TokenSpeed](https://github.com/lightseekorg/tokenspeed)
- **Model optimizer:** [AMD Quark](https://github.com/amd/quark) (`0.12.post1+rocm72.torch2.11`)
  - **Quantized layers:** MoE routed experts in transformer layers 3-41
  - **Weight quantization:** OCP MXFP4, static, group size 32
  - **Activation quantization:** OCP MXFP4, dynamic, group size 32

This checkpoint was produced by applying AMD Quark MXFP4 file-to-file quantization to the BF16 Inkling-Small checkpoint. Attention layers, shared experts, router weights, embeddings, normalization layers, multimodal components, MTP weights, and transformer layers 0-2 remain in BF16.

## Environment

The quantization was performed on an AMD gfx950 system with the following software:

- **GPU:** AMD MI350/MI355
- **Target graphics version:** gfx950
- **ROCm:** 7.2.1
- **Python:** 3.12.3
- **PyTorch:** 2.13.0+rocm7.1
- **AMD Quark:** 0.12.post1+rocm72.torch2.11
- **Safetensors:** 0.8.0

Create and activate a dedicated Quark environment:

```bash
python3 -m venv ~/.venv-quark
source ~/.venv-quark/bin/activate
```

Install the required packages:

```bash
python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.1
python -m pip install amd-quark --extra-index-url https://pypi.amd.com/quark/rocm72/simple
python -m pip install safetensors transformers accelerate tqdm
```

## Model Quantization

The included `quantize_quark.py` uses Quark's file-to-file flow to process safetensor shards without loading the full BF16 checkpoint into GPU memory. It automatically applies the Inkling-Small exclusion policy and quantizes only routed expert weights in layers 3-41.

```bash
python quantize_quark.py \
  --model_dir /path/to/Inkling-Small \
  --output_dir /path/to/Inkling-Small-MXFP4 \
  --quant_scheme mxfp4 \
  --file2file_quantization
```

## Deployment

This model can be served with [TokenSpeed](https://github.com/lightseekorg/tokenspeed):

```bash
tokenspeed serve \
  --model lightseekorg/Inkling-Small-MXFP4 \
  --attn-tp-size 1 \
  --moe-tp-size 1 \
  --max-model-len 81920 \
  --max-num-seqs 16 \
  --max-prefill-tokens 8192 \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.95 \
  --disable-cuda-graph-padding \
  --trust-remote-code \
  --dtype bfloat16 \
  --disable-kvstore \
  --kvstore-ratio 0 \
  --block-size 128 \
  --speculative-algorithm MTP \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --host 127.0.0.1 \
  --port 22015
```