Inkling 2 Mini UD-Q4_K_M h6-lbf16 (GGUF for a patched llama.cpp on AMD GPUs)

One file, Inkling-2-Mini-h6-lbf16-UD-Q4_K_M.gguf (16.442 GB): Unsloth's Inkling-2-Mini-UD-Q4_K_M.gguf with two small changes that reduce the bytes read per generated token. It is a 4-bit-class file like the one it derives from, about 5.0 bits per weight on average (the "bf16" in its name is the type of the small loop-gate tensors only). It is meant for a patched llama.cpp that makes Inkling 2 Mini fast on AMD RDNA3 GPUs under ROCm/HIP: the inkling2-amd branch of https://github.com/Makdok/llama.cpp-inkling2-amd. Most of the speed-up comes from that engine; this file adds a few percent on top of it. The engine's GPU code paths switch on for RDNA3.0 GPUs (gfx1100-gfx1103) only; on other GPUs the file runs at the speed of the stock engine. Everything below was measured on one GPU model, a Radeon RX 7900 XTX (gfx1100, 24 GB). With the Unsloth file on both engines, where the stock release fits in VRAM: prompt reading 829 -> 2103 tok/s and writing 42 -> 78 tok/s at the context of Unsloth's guide (32768) without a drafter, and 984 -> 1884 tok/s and 60 -> 94 tok/s with the DSpark drafter at 8192, the largest context where the release with the drafter stays in VRAM on this card.

What changed compared with the Unsloth file, and why

tensors Unsloth UD-Q4_K_M this file why
144 loop-gate weights, blk.6 to blk.17: loop_{attn,k,v,mlp}_{down,gain,forget}.weight F32 BF16 the 12 looped layers run three times per token and their gates are read on every pass after the first; every value is exactly representable in BF16, so the conversion loses nothing (see below) and halves these reads (113 MB to 57 MB stored)
output.weight (201024 x 2304) Q8_0 Q6_K the output head is read once per verify batch and once per drafting step by the DSpark drafter
the other 589 tensors (NVFP4 and Q5_K/Q8_0 experts, Q8_0 attention, BF16 and F32 small tensors) unchanged same type, copied by llama-quantize

Lossless check, done on 2026-11-21 with scripts/bf16_exact.py (in this repository): all 144 loop-gate tensors of the Unsloth file, 28 311 552 F32 values, have zero low 16 bits, and the BF16 tensors of this file are bit-identical to their high 16 bits. The same script lists the type of every tensor of both files: 734 tensors in each, the 145 above changed, none other.

How the file was made (llama-quantize built from the patched engine, whose patch does not touch quantization, so it behaves as the one of the Unsloth release; --pure with a base type of Q8_0, and overrides that restate the existing type of every family that must not change, so that llama-quantize copies them):

llama-quantize --allow-requantize --pure \
  --tensor-type "blk\.([0-9]|1[0-9])\.ffn_(gate_up|down)_exps\.weight=nvfp4" \
  --tensor-type "ffn_gate_up_exps\.weight=q5_K" --tensor-type "ffn_down_exps\.weight=q8_0" \
  --tensor-type "(attn_sink_proj|conv_proj)\.weight=bf16" \
  --tensor-type "loop_(attn|k|v|mlp)_(down|gain|forget)\.weight=bf16" \
  --output-tensor-type q6_K \
  Inkling-2-Mini-UD-Q4_K_M.gguf Inkling-2-Mini-h6-lbf16-UD-Q4_K_M.gguf Q8_0 16

Run

Build the engine for your GPU (GPU_TARGETS, see its README: Windows script scripts\inkling2-amd\build-hip.cmd, or the Linux cmake lines), download this file and the drafter dspark-Inkling-2-Mini-Q8_0.gguf from unsloth/Inkling-2-Mini-GGUF, then:

MODEL=<path to Inkling-2-Mini-h6-lbf16-UD-Q4_K_M.gguf>
DRAFT=<path to dspark-Inkling-2-Mini-Q8_0.gguf>
llama-server -m "$MODEL" -ngl 99 -fa 1 -c 65536 -np 1 -b 2048 -ub 1024 --jinja \
  -md "$DRAFT" -ngld 99 --spec-type draft-dspark --spec-draft-n-max 3 --spec-draft-n-ubatch 512 \
  --backend-sampling --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \
  --chat-template-kwargs '{"reasoning_effort":"high"}' --host 127.0.0.1 --port 8080

Stop it with Ctrl+C. In the Windows command prompt, end the lines with ^ instead of \ and write the JSON as "{\"reasoning_effort\":\"high\"}". The sampling settings are the ones recommended for the model (temperature 1.0, top-p 1.0, top-k 0, min-p 0); --backend-sampling samples on the GPU and is turned off by the server for requests with a grammar (tool calls). reasoning_effort takes none, minimal, low, medium, high or xhigh, or a number from 0 to 0.99 (default of the template: 0.9). Flash attention and the default F16 KV cache are required for the fast path.

Memory: with the command line above the server process peaked at 19.90 GB dedicated + 0.25 GB shared over a long run (measured on RX 7900 XTX), so a 24 GB GPU is the practical minimum. -ub 512 saves about 0.9 GB and reads prompts about 20 % more slowly; every further 65536 tokens of context adds about 2.1 GB of F16 KV cache (estimate from the cache size, not measured).

Measurements

Measured on Radeon RX 7900 XTX (24 GB), Windows, HIP SDK 7.2.

Quality (2026-11-21, final engine)

Perplexity: llama-perplexity -c 2048 --chunks 4 -ngl 99 -fa 1 -b 2048 -ub 512 on a 224 KB excerpt from the start of the WikiText-2 raw test set.

engine file perplexity against (a)
(a) release b12568-mix-8eebad4, HIP build Unsloth UD-Q4_K_M 6.2387 (+/- 0.229)
(b) inkling2-amd Unsloth UD-Q4_K_M 6.2364 -0.04 %
(c) inkling2-amd this file 6.2420 +0.05 %

The three values are within the statistical error of the estimate and within the 0.1 to 0.3 % that changes of rounding alone move on this model: no measurable difference.

Smoke prompts, (a) against (c), same server settings without a drafter (temperature 1.0, top-p 1.0, top-k 0, seed 1234 per request): a one-sentence factual question, an arithmetic question with a boxed answer, a short coding task with a test. Both engines answered all three correctly; the first two answers (reasoning and answer) were identical word for word, the two coding answers differ in form and are both correct. Writing speed in this short test: 44 tok/s for (a), 80 tok/s for (c).

Speed where the release fits in VRAM (2026-11-21, Unsloth file on both engines)

llama-server, one slot. A prompt of source code is read once, then six answers of up to 300 tokens are generated after it (three questions, temperature 1.0, top-p 0.95, top-k 20, a fixed seed per answer). Reading = prompt tokens per second over the whole prompt, as reported by the server; writing = generated tokens per second over the six answers; VRAM = peak of the server process over the run, dedicated + shared system memory (about 0.1-0.2 GB of shared memory from load on in every run: pinned host buffers, not a spill). The release runs with the llama-server line of Unsloth's "How to Run" guide (all layers and the drafter on the GPU, default flash-attention, batch and ubatch settings, the recommended sampling, without its vision projector), the patched engine with the command line of "Run" (without its drafter options in the rows without drafter).

engine, context, prompt reading writing tokens per drafter step VRAM peak
release, no drafter, context 32768 (Unsloth's guide), 24 829 tokens 829 tok/s 42.2 tok/s 19.95 + 0.22 GB
patched engine, no drafter, context 32768, 24 829 tokens 2103 tok/s 77.6 tok/s 19.09 + 0.14 GB
release, DSpark drafter, context 8192, 6 494 tokens 984 tok/s 60.2 tok/s 2.65 (44.0 ms per step) 19.07 + 0.37 GB
patched engine, DSpark drafter, context 8192, 6 494 tokens 1884 tok/s 93.5 tok/s 2.53 (27.1 ms per step) 17.78 + 0.19 GB

The release with the drafter does not stay in dedicated memory above 8192 on this card: at 32768 it keeps 1.20 GB in shared memory from load on, at 24576 2.72 GB, and at 16384 it fits at load but grows to 2.12 GB of shared memory while it reads the prompt (reading 54 tok/s on 14 356 tokens). The patched engine with the drafter: 2050 tok/s and 87.7 tok/s at 32768 (18.77 + 0.22 GB), 2053 tok/s and 90.0 tok/s at 16384 (18.13 + 0.20 GB).

At context 65536 the release does not fit on a 24 GB card (same text, 52 804 tokens)

Same method; a second identical client run reads only the follow-up after the cached prefix. At this context the release warns that it failed to fit its parameters to the free device memory, keeps every layer on the GPU as asked, and 4 to 6 GB of its buffers sit in shared system memory from the start: its rows measure that spill as much as the engine.

engine, file, server options reading writing tokens per drafter step follow-up VRAM peak
release b12568-mix-8eebad4 (HIP build), Unsloth UD-Q4_K_M, the options of Unsloth's guide, no drafter 265 tok/s 35.3 tok/s 514 tokens at 112 tok/s 20.31 + 4.05 GB
same, with the DSpark drafter (n-max 3) 261 tok/s 25.3 / 24.9 tok/s 2.46 / 2.43 514 tokens at 110 tok/s 20.41 + 5.77 GB
patched engine, Unsloth UD-Q4_K_M, the command line of "Run" 1898 / 1901 tok/s 90.3 / 90.2 tok/s 2.66 (29.5 ms per step) 20.06 + 0.25 GB
patched engine, this file, the command line of "Run" 1943 / 1943 tok/s 87.6 / 87.4 tok/s 2.55 (29.1 ms per step) 1026 tokens at 1223 tok/s 19.90 + 0.25 GB
  • Later engine revision (2026-11-21, after the rows above: 3-head attention tiles for GQA ratio 3, NVFP4 expert products of prompt batches on FP16 WMMA), h6-lbf16 file, the command line of "Run", same 52 804-token prompt: reading 1952 -> 2381 tok/s (2442 in a second session), writing unchanged within noise (83-87 tok/s), VRAM peak 19.92 GB dedicated + 0.25 GB shared; perplexity 6.2610 against 6.2584 with the new paths off (GGML_CUDA_INK2_FA_GQA3=0 GGML_CUDA_INK2_MOE_F16=0; same text and settings for both, not those of the other perplexity figures here).
  • Next engine revision (2026-11-21, after the line above: drafted tokens picked on the device, input uploads staged and batched on the GPU stream, no GPU wait before the drafter's CPU split), h6-lbf16 file, the command line of "Run", same 52 804-token prompt: reading 2439-2467 tok/s (was 2381-2442), writing 100-103 tok/s (was 83-87), VRAM peak unchanged at 19.92 GB dedicated + 0.25 GB shared.
  • Engine revision of 2026-11-22 (after the line above: SITU activation written by the FP16 expert kernel, Q8_0 / Q5_K experts of the last 4 layers on that kernel, the relative-bias table as one flat product, one shared-memory buffer in the NVFP4 expert kernel), h6-lbf16 file, the command line of "Run", same 52 804-token prompt: reading 2593-2596 tok/s (was 2439-2467), writing 98-103 tok/s after the prompt (unchanged within noise), VRAM peak 19.62 GB dedicated + 0.25 GB shared (was 19.92 + 0.25); perplexity 6.2608 (6.2610 before, same text and settings as in the first line).
  • This file against the Unsloth file on the patched engine: two alternated runs per file. Prompt reading +2.3 %, VRAM peak -0.16 GB, time per drafter step -1.2 % (29.1 against 29.5 ms). With the fixed seeds each file gave the same answers in both of its runs, and the Unsloth file's answers happened to accept more drafts (2.66 tokens per step against 2.55), so its tokens per second are higher in this test: the time per step is the fair comparison.
  • The follow-up of the last row comes from a separate run of the same setting (reading 1959 tok/s, writing 88.2 / 90.3 tok/s in that run). Without the drafter the patched engine wrote at 71.1 tok/s with --backend-sampling (one engine revision earlier).

Limitations

  • Quality was checked by perplexity on one text and three smoke questions only (table above). There is no KL-divergence measurement against a reference and no task benchmark. The KLD figures of the Unsloth model card describe their file, not this one.
  • One GPU model (RX 7900 XTX) under Windows with HIP SDK 7.2. Other GPUs, Linux, NVIDIA, Vulkan, Metal and CPU were not tested; on them the engine runs its stock paths.
  • The gain of this file depends on the patched engine: its fused loop-gate kernels read the BF16 weights directly. The file uses only standard GGUF types but was not run on the stock release engine (the quality check above compares the release with the Unsloth file against the patched engine with this file). Upstream llama.cpp could not load Inkling 2 Mini when this was measured (2026-11-20).
  • Text only: the vision projector (mmproj) of the Unsloth repository was not tested with this file.
  • Contexts from 8192 to 65536 tokens were measured; longer ones were not.

Attribution

  • Model: Inkling 2 Mini by Thinking Machines, Apache-2.0.
  • GGUF conversion, the UD-Q4_K_M quantization this file is derived from, and the DSpark drafter: Unsloth, unsloth/Inkling-2-Mini-GGUF.
  • Engine: llama.cpp (MIT, ggml-org and contributors), in Unsloth's release b12568-mix-8eebad4, with two Unsloth pull requests by Daniel Han ported (unslothai/llama.cpp#385 and #386) and the AMD changes of the inkling2-amd branch.

AI assistance

The engine changes, the scripts and this model card were written with AI assistance (Anthropic's Claude models, in Claude Code). The measurements were run on one Radeon RX 7900 XTX.

Downloads last month
-
GGUF
Model size
26B params
Architecture
inkling2
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Makdok/Inkling-2-Mini-AMD-GGUF

Quantized
(137)
this model