seekr-org/Inkling-2-Small-FP8-ROCm

Inkling 2 Small FP8 v2 for AMD MI300X

A row-scaled E4M3FNUZ conversion of Thinking Machines' Inkling 2 Small MXFP8, revision 3febc62259ee92ea36e2fb6e8303371c4eb3922c. The original model was developed by Thinking Machines Lab. This repository contains a precision conversion. The upstream model card is preserved in UPSTREAM_README.md.

V2 selects power-of-two expert weight and activation scales by default. The previous v1 checkpoint remains available at revision 78a4761861425cf44fdcd688d3bcc0ecc10d90e2.

Precision and files

The 96 expert projection tensors use native AMD E4M3FNUZ FP8 with one FP32 scale per expert output channel. Activations use dynamic per-token E4M3FNUZ. Both scale methods use the smallest normal FP32 power of two at least amax / 240, with a minimum of float32.tiny. Power-of-two weight scales preserve the source MXFP8 mantissas when their exponents fit the row format. Very small values can still underflow or round in the subnormal range.

The full scan of 587,068,342,272 expert values found:

• 99.99881% reconstruct to exactly the same FP32 numeric values as the source MXFP8 checkpoint.
• Added relative RMS error: 5.21049e-8 (0.00000521%).
• Maximum absolute reconstruction error: 1.90735e-6.
• 870,110 nonzero source values underflow to zero.

These statistics measure the additional conversion error against an already quantized checkpoint, not error against original full-precision weights. The scan uses FP32 reconstruction and FP64 error accumulation. Weight fidelity does not guarantee identical model outputs; row activation quantization and matrix accumulation also differ from the original block-scaled path.

All 1,418 nonexpert tensors and the separate MTP checkpoint were verified bitwise unchanged. Attention, dense projections, vision, and the DSpark draft retain upstream precision. This is mixed precision inference.

All 48 target weight shards, tokenizer/configuration files, vision weights, and mtp.safetensors are included, approximately 605 GB in total. conversion_manifest.json records conversion settings, full expert-weight error statistics, and per-shard serialization fingerprints.

hf_quant_config.json marks converted experts with compute_dtype: fp8. config.json selects expert_fp8_scale_mode: row, expert_fp8_weight_scale_method: power_of_two, and expert_fp8_activation_scale_method: power_of_two. Expert scale tensors retain the _weight_scale suffix and have shape [384, output_channels], dtype FP32. The loader reads the converted weights directly without requantizing them.

Serving

Use the ROCm vLLM image built by the Seekr GitLab pipeline from source containing the v2 power-of-two scale controls, Inkling 2, and prequantized row-FP8 support. The image must honor expert_fp8_activation_scale_method; earlier implementations can load the row weight format but do not select the tested v2 activation path. The Docker build checks that the wheel and installed implementation match the source. No separate vLLM patch is distributed in this model repository.

Validated hardware: 8 AMD MI300X GPUs (gfx942), 192 GiB per GPU. Other GPU architectures have not been validated.

Inside the updated deployment image:

export OMP_NUM_THREADS=1
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_CUSTOM_AR=1

vllm serve seekr-org/Inkling-2-Small-FP8-ROCm \
  --quantization inkling2 --tensor-parallel-size 8 \
  --max-model-len 262144 --max-num-batched-tokens 2048 --max-num-seqs 8 \
  --no-enable-prefix-caching \
  --reasoning-parser inkling --limit-mm-per-prompt '{"image":2}' \
  --gpu-memory-utilization 0.9 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","decode_cudagraph_capture_request_sizes":[1,2,4,8],"use_breakable_cudagraph":false}' \
  --kernel-config '{"enable_jit_warmup":false}'

The checkpoint configuration already selects both power-of-two scale methods; no --hf-overrides is needed for this published v2 checkpoint. The image entrypoint is vllm serve. With Docker, expose /dev/kfd and /dev/dri, use host IPC, mount a persistent Hugging Face cache, and pass the model name and flags above as container arguments.

Images use standard image_url chat content parts. Set chat_template_kwargs.reasoning_effort to 0 to request direct answers. This is a hint and does not prevent the model from emitting reasoning.

Measurements and accuracy

A fresh comparison used 24 distinct math/programming/prose throughput tasks, 2,016 input tokens and 512 output tokens per request, temperature 1, default reasoning, prefix caching disabled, and the seven-token probabilistic DSpark configuration above. Throughput includes prefill, the final partially occupied batch, and reasoning tokens. These are short-prompt measurements.

Configuration	C1 output tok/s	C8 aggregate output tok/s	GSM8K	HumanEval	Prose constraints
V1: amax weights and activations	244.62	820.75	48/50	20/20	6/6
V2: power-of-two weights, amax activations	237.73	814.39	48/50	20/20	5/6
Published v2: power-of-two weights and activations	244.06	807.66	48/50	20/20	6/6

Accuracy used fixed samples of 50 GSM8K and 20 HumanEval problems, temperature zero, reasoning effort zero, a 2,048-token output limit, and concurrency eight. All 20 sampled coding tasks passed execution checks. The same two math examples missed in each configuration; one official reference answer is arithmetically incorrect. Raw scores retain that reference. These small samples do not establish full benchmark parity, and greedy continuations can vary with batching.

Prose constraints cover one story: 180–220 words, required characters/objects, dialogue, and an exact ending. V1 produced 192 words, weight-only v2 produced 175 and missed the length constraint, and the published v2 configuration produced 219. These checks do not measure literary quality in general.

Both v2 configurations passed arithmetic, factual text, image colors, OCR, two-image ordering, and streaming checks. All three configurations recalled the correct code from a 261,040-token input with a 128-token output budget. The initial weight-only v2 trial at 32 output tokens recalled the code in reasoning but exhausted the budget before a final answer. Published v2 used 109 output tokens, including 100 reasoning tokens; v1 returned the code in six. This is a capacity and simple recall check, not broad long-context accuracy.

The published v2 C1 domain rates were 331.49 tok/s for math, 247.24 for programming, and 191.23 for prose. The mixed C1 average exceeds 200 tok/s; the prose subset does not. The C8 goal of 1,000 output tok/s was not met. V2 improves weight fidelity without demonstrating a throughput improvement.

validation_results.json contains the paired results, numerical verification, and test summary. GPU kernel, renderer, CI-helper, pre-commit, wheel, and isolated wheel installation checks passed. A full Docker build was not run locally; the GitLab image pipeline performs the build checks.

License and attribution

Apache License 2.0, inherited from the upstream checkpoint. See LICENSE. Original model: Thinking Machines Lab. This repository converts expert precision for AMD MI300X deployment and retains upstream attribution. Model purpose and upstream information are in UPSTREAM_README.md.