|
Download README.md from seekr-org/Inkling-2-Small-FP8-ROCm: direct link, hf CLI and curl.
- Browser
- Download file 8.06 kB
-
https://huggingface.co/seekr-org/Inkling-2-Small-FP8-ROCm/resolve/main/README.md
- Command line
-
hf download hf://seekr-org/Inkling-2-Small-FP8-ROCm/README.md
-
curl -L -o README.md https://huggingface.co/seekr-org/Inkling-2-Small-FP8-ROCm/resolve/main/README.md
8.06 kB
| license: apache-2.0 | |
| base_model: thinkingmachines/Inkling-2-Small-MXFP8 | |
| base_model_relation: quantized | |
| library_name: vllm | |
| pipeline_tag: image-text-to-text | |
| tags: | |
| - fp8 | |
| - rocm | |
| - mi300x | |
| - inkling2 | |
| - moe | |
| - dspark | |
| - vision | |
| # Inkling 2 Small FP8 v2 for AMD MI300X | |
| A row-scaled E4M3FNUZ conversion of | |
| [Thinking Machines' Inkling 2 Small MXFP8](https://huggingface.co/thinkingmachines/Inkling-2-Small-MXFP8), | |
| revision `3febc62259ee92ea36e2fb6e8303371c4eb3922c`. | |
| The original model was developed by Thinking Machines Lab. This repository | |
| contains a precision conversion. The upstream model card is preserved in | |
| [UPSTREAM_README.md](UPSTREAM_README.md). | |
| **V2 selects power-of-two expert weight and activation scales by default.** | |
| The previous v1 checkpoint remains available at | |
| [revision 78a4761861425cf44fdcd688d3bcc0ecc10d90e2](https://huggingface.co/seekr-org/Inkling-2-Small-FP8-ROCm/tree/78a4761861425cf44fdcd688d3bcc0ecc10d90e2). | |
| ## Precision and files | |
| The 96 expert projection tensors use native AMD E4M3FNUZ FP8 with one FP32 | |
| scale per expert output channel. Activations use dynamic per-token E4M3FNUZ. | |
| Both scale methods use the smallest normal FP32 power of two at least | |
| `amax / 240`, with a minimum of `float32.tiny`. Power-of-two weight scales | |
| preserve the source MXFP8 mantissas when their exponents fit the row format. | |
| Very small values can still underflow or round in the subnormal range. | |
| The full scan of **587,068,342,272 expert values** found: | |
| - **99.99881%** reconstruct to exactly the same FP32 numeric values as the source MXFP8 checkpoint. | |
| - Added relative RMS error: **5.21049e-8** (0.00000521%). | |
| - Maximum absolute reconstruction error: **1.90735e-6**. | |
| - 870,110 nonzero source values underflow to zero. | |
| These statistics measure the additional conversion error against an already | |
| quantized checkpoint, not error against original full-precision weights. | |
| The scan uses FP32 reconstruction and FP64 error accumulation. Weight fidelity | |
| does not guarantee identical model outputs; row activation quantization and | |
| matrix accumulation also differ from the original block-scaled path. | |
| All 1,418 nonexpert tensors and the separate MTP checkpoint were verified | |
| bitwise unchanged. Attention, dense projections, vision, and the DSpark draft | |
| retain upstream precision. This is mixed precision inference. | |
| All 48 target weight shards, tokenizer/configuration files, vision weights, | |
| and `mtp.safetensors` are included, approximately 605 GB in total. | |
| [conversion_manifest.json](conversion_manifest.json) records conversion settings, | |
| full expert-weight error statistics, and per-shard serialization fingerprints. | |
| `hf_quant_config.json` marks converted experts with `compute_dtype: fp8`. | |
| `config.json` selects `expert_fp8_scale_mode: row`, | |
| `expert_fp8_weight_scale_method: power_of_two`, and | |
| `expert_fp8_activation_scale_method: power_of_two`. | |
| Expert scale tensors retain the `_weight_scale` suffix and have shape | |
| `[384, output_channels]`, dtype FP32. The loader reads the converted weights | |
| directly without requantizing them. | |
| ## Serving | |
| Use the ROCm vLLM image built by the Seekr GitLab pipeline from source containing | |
| the **v2 power-of-two scale controls**, Inkling 2, and prequantized row-FP8 support. | |
| The image must honor `expert_fp8_activation_scale_method`; earlier implementations | |
| can load the row weight format but do not select the tested v2 activation path. | |
| The Docker build checks that the wheel and installed implementation match the | |
| source. No separate vLLM patch is distributed in this model repository. | |
| Validated hardware: **8 AMD MI300X GPUs (gfx942), 192 GiB per GPU**. | |
| Other GPU architectures have not been validated. | |
| Inside the updated deployment image: | |
| ```bash | |
| export OMP_NUM_THREADS=1 | |
| export VLLM_ROCM_USE_AITER=1 | |
| export VLLM_ROCM_USE_AITER_CUSTOM_AR=1 | |
| vllm serve seekr-org/Inkling-2-Small-FP8-ROCm \ | |
| --quantization inkling2 --tensor-parallel-size 8 \ | |
| --max-model-len 262144 --max-num-batched-tokens 2048 --max-num-seqs 8 \ | |
| --no-enable-prefix-caching \ | |
| --reasoning-parser inkling --limit-mm-per-prompt '{"image":2}' \ | |
| --gpu-memory-utilization 0.9 \ | |
| --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' \ | |
| --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","decode_cudagraph_capture_request_sizes":[1,2,4,8],"use_breakable_cudagraph":false}' \ | |
| --kernel-config '{"enable_jit_warmup":false}' | |
| ``` | |
| The checkpoint configuration already selects both power-of-two scale methods; | |
| no `--hf-overrides` is needed for this published v2 checkpoint. | |
| The image entrypoint is `vllm serve`. With Docker, expose `/dev/kfd` and | |
| `/dev/dri`, use host IPC, mount a persistent Hugging Face cache, and pass the | |
| model name and flags above as container arguments. | |
| Images use standard `image_url` chat content parts. Set | |
| `chat_template_kwargs.reasoning_effort` to `0` to request direct answers. | |
| This is a hint and does not prevent the model from emitting reasoning. | |
| ## Measurements and accuracy | |
| A fresh comparison used 24 distinct math/programming/prose throughput tasks, | |
| 2,016 input tokens and 512 output tokens per request, temperature 1, default | |
| reasoning, prefix caching disabled, and the seven-token probabilistic DSpark | |
| configuration above. Throughput includes prefill, the final partially occupied | |
| batch, and reasoning tokens. These are short-prompt measurements. | |
| | Configuration | C1 output tok/s | C8 aggregate output tok/s | GSM8K | HumanEval | Prose constraints | | |
| | --- | ---: | ---: | ---: | ---: | ---: | | |
| | V1: amax weights and activations | 244.62 | 820.75 | 48/50 | 20/20 | 6/6 | | |
| | V2: power-of-two weights, amax activations | 237.73 | 814.39 | 48/50 | 20/20 | 5/6 | | |
| | **Published v2: power-of-two weights and activations** | **244.06** | **807.66** | **48/50** | **20/20** | **6/6** | | |
| Accuracy used fixed samples of 50 GSM8K and 20 HumanEval problems, temperature | |
| zero, reasoning effort zero, a 2,048-token output limit, and concurrency eight. | |
| All 20 sampled coding tasks passed execution checks. The same two math examples | |
| missed in each configuration; one official reference answer is arithmetically | |
| incorrect. Raw scores retain that reference. These small samples do not establish | |
| full benchmark parity, and greedy continuations can vary with batching. | |
| Prose constraints cover one story: 180–220 words, required characters/objects, | |
| dialogue, and an exact ending. V1 produced 192 words, weight-only v2 produced | |
| 175 and missed the length constraint, and the published v2 configuration | |
| produced 219. These checks do not measure literary quality in general. | |
| Both v2 configurations passed arithmetic, factual text, image colors, OCR, | |
| two-image ordering, and streaming checks. All three configurations recalled | |
| the correct code from a **261,040-token input with a 128-token output budget**. | |
| The initial weight-only v2 trial at 32 output tokens recalled the code in | |
| reasoning but exhausted the budget before a final answer. Published v2 used | |
| 109 output tokens, including 100 reasoning tokens; v1 returned the code in six. | |
| This is a capacity and simple recall check, not broad long-context accuracy. | |
| The published v2 C1 domain rates were 331.49 tok/s for math, 247.24 for | |
| programming, and 191.23 for prose. The mixed C1 average exceeds 200 tok/s; | |
| the prose subset does not. The **C8 goal of 1,000 output tok/s was not met**. | |
| V2 improves weight fidelity without demonstrating a throughput improvement. | |
| [validation_results.json](validation_results.json) contains the paired results, | |
| numerical verification, and test summary. GPU kernel, renderer, CI-helper, | |
| pre-commit, wheel, and isolated wheel installation checks passed. A full Docker | |
| build was not run locally; the GitLab image pipeline performs the build checks. | |
| ## License and attribution | |
| Apache License 2.0, inherited from the upstream checkpoint. See [LICENSE](LICENSE). | |
| Original model: Thinking Machines Lab. This repository converts expert precision | |
| for AMD MI300X deployment and retains upstream attribution. Model purpose and | |
| upstream information are in UPSTREAM_README.md. | |