Inkling-Small-MXFP4

Model Overview

• Base model: thinkingmachines/Inkling-Small
• Architecture: 42-layer multimodal sparse Mixture-of-Experts transformer
  • Parameters: 276B total, 12B active
  • Input: Text, image, and audio
  • Output: Text
• Inference engine: TokenSpeed
• Model optimizer: AMD Quark (0.12.post1+rocm72.torch2.11)
  • Quantized layers: MoE routed experts in transformer layers 3-41
  • Weight quantization: OCP MXFP4, static, group size 32
  • Activation quantization: OCP MXFP4, dynamic, group size 32

This checkpoint was produced by applying AMD Quark MXFP4 file-to-file quantization to the BF16 Inkling-Small checkpoint. Attention layers, shared experts, router weights, embeddings, normalization layers, multimodal components, MTP weights, and transformer layers 0-2 remain in BF16.

Environment

The quantization was performed on an AMD gfx950 system with the following software:

• GPU: AMD MI350/MI355
• Target graphics version: gfx950
• ROCm: 7.2.1
• Python: 3.12.3
• PyTorch: 2.13.0+rocm7.1
• AMD Quark: 0.12.post1+rocm72.torch2.11
• Safetensors: 0.8.0

Create and activate a dedicated Quark environment:

python3 -m venv ~/.venv-quark
source ~/.venv-quark/bin/activate

Install the required packages:

python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.1
python -m pip install amd-quark --extra-index-url https://pypi.amd.com/quark/rocm72/simple
python -m pip install safetensors transformers accelerate tqdm

Model Quantization

The included quantize_quark.py uses Quark's file-to-file flow to process safetensor shards without loading the full BF16 checkpoint into GPU memory. It automatically applies the Inkling-Small exclusion policy and quantizes only routed expert weights in layers 3-41.

python quantize_quark.py \
  --model_dir /path/to/Inkling-Small \
  --output_dir /path/to/Inkling-Small-MXFP4 \
  --quant_scheme mxfp4 \
  --file2file_quantization

Deployment

This model can be served with TokenSpeed:

tokenspeed serve \
  --model lightseekorg/Inkling-Small-MXFP4 \
  --attn-tp-size 1 \
  --moe-tp-size 1 \
  --max-model-len 81920 \
  --max-num-seqs 16 \
  --max-prefill-tokens 8192 \
  --chunked-prefill-size 8192 \
  --gpu-memory-utilization 0.95 \
  --disable-cuda-graph-padding \
  --trust-remote-code \
  --dtype bfloat16 \
  --disable-kvstore \
  --kvstore-ratio 0 \
  --block-size 128 \
  --speculative-algorithm MTP \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --host 127.0.0.1 \
  --port 22015