Instructions to use Makdok/Inkling-2-Mini-AMD-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Makdok/Inkling-2-Mini-AMD-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M # Run inference directly in the terminal: llama cli -hf Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M # Run inference directly in the terminal: llama cli -hf Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M
Use Docker
docker model run hf.co/Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use Makdok/Inkling-2-Mini-AMD-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Makdok/Inkling-2-Mini-AMD-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Makdok/Inkling-2-Mini-AMD-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M
- Ollama
How to use Makdok/Inkling-2-Mini-AMD-GGUF with Ollama:
ollama run hf.co/Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M
- Unsloth Desktop
- Pi
How to use Makdok/Inkling-2-Mini-AMD-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use Makdok/Inkling-2-Mini-AMD-GGUF with Docker Model Runner:
docker model run hf.co/Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M
- Lemonade
How to use Makdok/Inkling-2-Mini-AMD-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M
Run and chat with the model
lemonade run user.Inkling-2-Mini-AMD-GGUF-UD-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use Makdok/Inkling-2-Mini-AMD-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Makdok/Inkling-2-Mini-AMD-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "Makdok/Inkling-2-Mini-AMD-GGUF:UD-Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Inkling 2 Mini UD-Q4_K_M h6-lbf16 (GGUF for a patched llama.cpp on AMD GPUs)
One file, Inkling-2-Mini-h6-lbf16-UD-Q4_K_M.gguf (16.442 GB): Unsloth's Inkling-2-Mini-UD-Q4_K_M.gguf with two small
changes that reduce the bytes read per generated token. It is a 4-bit-class file like the one it derives from, about
5.0 bits per weight on average (the "bf16" in its name is the type of the small loop-gate tensors only). It is meant for a patched llama.cpp that makes Inkling 2 Mini
fast on AMD RDNA3 GPUs under ROCm/HIP: the inkling2-amd branch of https://github.com/Makdok/llama.cpp-inkling2-amd. Most of
the speed-up comes from that engine; this file adds a few percent on top of it. The engine's GPU code paths switch on
for RDNA3.0 GPUs (gfx1100-gfx1103) only; on other GPUs the file runs at the speed of the stock engine. Everything below
was measured on one GPU model, a Radeon RX 7900 XTX (gfx1100, 24 GB). With the Unsloth file on both engines, where the stock release fits in
VRAM: prompt reading 829 -> 2103 tok/s and writing 42 -> 78 tok/s at the context of Unsloth's guide (32768) without a
drafter, and 984 -> 1884 tok/s and 60 -> 94 tok/s with the DSpark drafter at 8192, the largest context where the release
with the drafter stays in VRAM on this card.
What changed compared with the Unsloth file, and why
| tensors | Unsloth UD-Q4_K_M | this file | why |
|---|---|---|---|
144 loop-gate weights, blk.6 to blk.17: loop_{attn,k,v,mlp}_{down,gain,forget}.weight |
F32 | BF16 | the 12 looped layers run three times per token and their gates are read on every pass after the first; every value is exactly representable in BF16, so the conversion loses nothing (see below) and halves these reads (113 MB to 57 MB stored) |
output.weight (201024 x 2304) |
Q8_0 | Q6_K | the output head is read once per verify batch and once per drafting step by the DSpark drafter |
| the other 589 tensors (NVFP4 and Q5_K/Q8_0 experts, Q8_0 attention, BF16 and F32 small tensors) | unchanged | same type, copied by llama-quantize |
Lossless check, done on 2026-11-21 with scripts/bf16_exact.py (in this repository): all 144 loop-gate tensors of the
Unsloth file, 28 311 552 F32 values, have zero low 16 bits, and the BF16 tensors of this file are bit-identical to their
high 16 bits. The same script lists the type of every tensor of both files: 734 tensors in each, the 145 above changed,
none other.
How the file was made (llama-quantize built from the patched engine, whose patch does not touch quantization, so it
behaves as the one of the Unsloth release; --pure with a base type of Q8_0, and overrides that restate the existing
type of every family that must not change, so that llama-quantize copies them):
llama-quantize --allow-requantize --pure \
--tensor-type "blk\.([0-9]|1[0-9])\.ffn_(gate_up|down)_exps\.weight=nvfp4" \
--tensor-type "ffn_gate_up_exps\.weight=q5_K" --tensor-type "ffn_down_exps\.weight=q8_0" \
--tensor-type "(attn_sink_proj|conv_proj)\.weight=bf16" \
--tensor-type "loop_(attn|k|v|mlp)_(down|gain|forget)\.weight=bf16" \
--output-tensor-type q6_K \
Inkling-2-Mini-UD-Q4_K_M.gguf Inkling-2-Mini-h6-lbf16-UD-Q4_K_M.gguf Q8_0 16
Run
Build the engine for your GPU (GPU_TARGETS, see its README: Windows script scripts\inkling2-amd\build-hip.cmd, or
the Linux cmake lines), download this file and the drafter dspark-Inkling-2-Mini-Q8_0.gguf from
unsloth/Inkling-2-Mini-GGUF, then:
MODEL=<path to Inkling-2-Mini-h6-lbf16-UD-Q4_K_M.gguf>
DRAFT=<path to dspark-Inkling-2-Mini-Q8_0.gguf>
llama-server -m "$MODEL" -ngl 99 -fa 1 -c 65536 -np 1 -b 2048 -ub 1024 --jinja \
-md "$DRAFT" -ngld 99 --spec-type draft-dspark --spec-draft-n-max 3 --spec-draft-n-ubatch 512 \
--backend-sampling --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \
--chat-template-kwargs '{"reasoning_effort":"high"}' --host 127.0.0.1 --port 8080
Stop it with Ctrl+C. In the Windows command prompt, end the lines with ^ instead of \ and write the JSON as
"{\"reasoning_effort\":\"high\"}". The sampling settings are the ones recommended for the model (temperature 1.0,
top-p 1.0, top-k 0, min-p 0); --backend-sampling samples on the GPU and is turned off by the server for requests with
a grammar (tool calls). reasoning_effort takes none, minimal, low, medium, high or xhigh, or a number from 0 to 0.99
(default of the template: 0.9). Flash attention and the default F16 KV cache are required for the fast path.
Memory: with the command line above the server process peaked at 19.90 GB dedicated + 0.25 GB shared over a long run
(measured on RX 7900 XTX), so a 24 GB GPU is the practical minimum. -ub 512 saves about 0.9 GB and reads prompts
about 20 % more slowly; every further 65536 tokens of context adds about 2.1 GB of F16 KV cache (estimate from the
cache size, not measured).
Measurements
Measured on Radeon RX 7900 XTX (24 GB), Windows, HIP SDK 7.2.
Quality (2026-11-21, final engine)
Perplexity: llama-perplexity -c 2048 --chunks 4 -ngl 99 -fa 1 -b 2048 -ub 512 on a 224 KB excerpt from the start of
the WikiText-2 raw test set.
| engine | file | perplexity | against (a) | |
|---|---|---|---|---|
| (a) | release b12568-mix-8eebad4, HIP build | Unsloth UD-Q4_K_M | 6.2387 (+/- 0.229) | |
| (b) | inkling2-amd | Unsloth UD-Q4_K_M | 6.2364 | -0.04 % |
| (c) | inkling2-amd | this file | 6.2420 | +0.05 % |
The three values are within the statistical error of the estimate and within the 0.1 to 0.3 % that changes of rounding alone move on this model: no measurable difference.
Smoke prompts, (a) against (c), same server settings without a drafter (temperature 1.0, top-p 1.0, top-k 0, seed 1234 per request): a one-sentence factual question, an arithmetic question with a boxed answer, a short coding task with a test. Both engines answered all three correctly; the first two answers (reasoning and answer) were identical word for word, the two coding answers differ in form and are both correct. Writing speed in this short test: 44 tok/s for (a), 80 tok/s for (c).
Speed where the release fits in VRAM (2026-11-21, Unsloth file on both engines)
llama-server, one slot. A prompt of source code is read once, then six answers of up to 300 tokens are generated after
it (three questions, temperature 1.0, top-p 0.95, top-k 20, a fixed seed per answer). Reading = prompt tokens per second
over the whole prompt, as reported by the server; writing = generated tokens per second over the six answers; VRAM =
peak of the server process over the run, dedicated + shared system memory (about 0.1-0.2 GB of shared memory from load
on in every run: pinned host buffers, not a spill). The release runs with the llama-server line of Unsloth's "How to
Run" guide (all layers and the drafter on the GPU, default flash-attention, batch and ubatch settings, the recommended
sampling, without its vision projector), the patched engine with the command line of "Run" (without its drafter options
in the rows without drafter).
| engine, context, prompt | reading | writing | tokens per drafter step | VRAM peak |
|---|---|---|---|---|
| release, no drafter, context 32768 (Unsloth's guide), 24 829 tokens | 829 tok/s | 42.2 tok/s | 19.95 + 0.22 GB | |
| patched engine, no drafter, context 32768, 24 829 tokens | 2103 tok/s | 77.6 tok/s | 19.09 + 0.14 GB | |
| release, DSpark drafter, context 8192, 6 494 tokens | 984 tok/s | 60.2 tok/s | 2.65 (44.0 ms per step) | 19.07 + 0.37 GB |
| patched engine, DSpark drafter, context 8192, 6 494 tokens | 1884 tok/s | 93.5 tok/s | 2.53 (27.1 ms per step) | 17.78 + 0.19 GB |
The release with the drafter does not stay in dedicated memory above 8192 on this card: at 32768 it keeps 1.20 GB in shared memory from load on, at 24576 2.72 GB, and at 16384 it fits at load but grows to 2.12 GB of shared memory while it reads the prompt (reading 54 tok/s on 14 356 tokens). The patched engine with the drafter: 2050 tok/s and 87.7 tok/s at 32768 (18.77 + 0.22 GB), 2053 tok/s and 90.0 tok/s at 16384 (18.13 + 0.20 GB).
At context 65536 the release does not fit on a 24 GB card (same text, 52 804 tokens)
Same method; a second identical client run reads only the follow-up after the cached prefix. At this context the release warns that it failed to fit its parameters to the free device memory, keeps every layer on the GPU as asked, and 4 to 6 GB of its buffers sit in shared system memory from the start: its rows measure that spill as much as the engine.
| engine, file, server options | reading | writing | tokens per drafter step | follow-up | VRAM peak |
|---|---|---|---|---|---|
| release b12568-mix-8eebad4 (HIP build), Unsloth UD-Q4_K_M, the options of Unsloth's guide, no drafter | 265 tok/s | 35.3 tok/s | 514 tokens at 112 tok/s | 20.31 + 4.05 GB | |
| same, with the DSpark drafter (n-max 3) | 261 tok/s | 25.3 / 24.9 tok/s | 2.46 / 2.43 | 514 tokens at 110 tok/s | 20.41 + 5.77 GB |
| patched engine, Unsloth UD-Q4_K_M, the command line of "Run" | 1898 / 1901 tok/s | 90.3 / 90.2 tok/s | 2.66 (29.5 ms per step) | 20.06 + 0.25 GB | |
| patched engine, this file, the command line of "Run" | 1943 / 1943 tok/s | 87.6 / 87.4 tok/s | 2.55 (29.1 ms per step) | 1026 tokens at 1223 tok/s | 19.90 + 0.25 GB |
- Later engine revision (2026-11-21, after the rows above: 3-head attention tiles for GQA ratio 3, NVFP4 expert products of prompt batches on FP16 WMMA), h6-lbf16 file, the command line of "Run", same 52 804-token prompt: reading 1952 -> 2381 tok/s (2442 in a second session), writing unchanged within noise (83-87 tok/s), VRAM peak 19.92 GB dedicated + 0.25 GB shared; perplexity 6.2610 against 6.2584 with the new paths off (
GGML_CUDA_INK2_FA_GQA3=0 GGML_CUDA_INK2_MOE_F16=0; same text and settings for both, not those of the other perplexity figures here). - Next engine revision (2026-11-21, after the line above: drafted tokens picked on the device, input uploads staged and batched on the GPU stream, no GPU wait before the drafter's CPU split), h6-lbf16 file, the command line of "Run", same 52 804-token prompt: reading 2439-2467 tok/s (was 2381-2442), writing 100-103 tok/s (was 83-87), VRAM peak unchanged at 19.92 GB dedicated + 0.25 GB shared.
- Engine revision of 2026-11-22 (after the line above: SITU activation written by the FP16 expert kernel, Q8_0 / Q5_K experts of the last 4 layers on that kernel, the relative-bias table as one flat product, one shared-memory buffer in the NVFP4 expert kernel), h6-lbf16 file, the command line of "Run", same 52 804-token prompt: reading 2593-2596 tok/s (was 2439-2467), writing 98-103 tok/s after the prompt (unchanged within noise), VRAM peak 19.62 GB dedicated + 0.25 GB shared (was 19.92 + 0.25); perplexity 6.2608 (6.2610 before, same text and settings as in the first line).
- This file against the Unsloth file on the patched engine: two alternated runs per file. Prompt reading +2.3 %, VRAM peak -0.16 GB, time per drafter step -1.2 % (29.1 against 29.5 ms). With the fixed seeds each file gave the same answers in both of its runs, and the Unsloth file's answers happened to accept more drafts (2.66 tokens per step against 2.55), so its tokens per second are higher in this test: the time per step is the fair comparison.
- The follow-up of the last row comes from a separate run of the same setting (reading 1959 tok/s, writing 88.2 / 90.3
tok/s in that run). Without the drafter the patched engine wrote at 71.1 tok/s with
--backend-sampling(one engine revision earlier).
Limitations
- Quality was checked by perplexity on one text and three smoke questions only (table above). There is no KL-divergence measurement against a reference and no task benchmark. The KLD figures of the Unsloth model card describe their file, not this one.
- One GPU model (RX 7900 XTX) under Windows with HIP SDK 7.2. Other GPUs, Linux, NVIDIA, Vulkan, Metal and CPU were not tested; on them the engine runs its stock paths.
- The gain of this file depends on the patched engine: its fused loop-gate kernels read the BF16 weights directly. The file uses only standard GGUF types but was not run on the stock release engine (the quality check above compares the release with the Unsloth file against the patched engine with this file). Upstream llama.cpp could not load Inkling 2 Mini when this was measured (2026-11-20).
- Text only: the vision projector (mmproj) of the Unsloth repository was not tested with this file.
- Contexts from 8192 to 65536 tokens were measured; longer ones were not.
Attribution
- Model: Inkling 2 Mini by Thinking Machines, Apache-2.0.
- GGUF conversion, the UD-Q4_K_M quantization this file is derived from, and the DSpark drafter: Unsloth, unsloth/Inkling-2-Mini-GGUF.
- Engine: llama.cpp (MIT, ggml-org and contributors), in Unsloth's release b12568-mix-8eebad4, with two Unsloth pull requests by Daniel Han ported (unslothai/llama.cpp#385 and #386) and the AMD changes of the inkling2-amd branch.
AI assistance
The engine changes, the scripts and this model card were written with AI assistance (Anthropic's Claude models, in Claude Code). The measurements were run on one Radeon RX 7900 XTX.
- Downloads last month
- -
4-bit
Model tree for Makdok/Inkling-2-Mini-AMD-GGUF
Base model
thinkingmachines/Inkling-2-Mini