Inkling-2-Mini GGUF

Atomic Chat Discord GitHub X
  • Built by the team behind Atomic Chat, the app that runs open-weight models locally and offline.

GGUF files of thinkingmachines/Inkling-2-Mini by AtomicChat: a 26.4B mixture-of-experts model (3.4B active, 128 experts with 8 active) with a looped core, where the middle 12 of its 24 layers run three times. It takes text and image input. License: Apache-2.0, the same as the base model (LICENSE in this repo).

Runtime: llama.cpp with ggml-org/llama.cpp#32695, a draft PR at the time of writing. The files are built at its commit 6196d98. llama.cpp releases without that PR do not load the looped Inkling 2.

Files

AD (Atomic Dynamic) files give each tensor its own type. The experts hold 94% of the weights and carry the compression; attention and the loop controllers stay Q8_0. Quality is measured against our BF16: top-1 is how often the file picks the same next token as the original, and mean KLD is the divergence of the whole next-token distribution (lower is better).

file GB on disk top-1 mean KLD notes
BF16/ (2 parts) 52.2 โ€” โ€” the reference
Inkling-2-Mini-Q8_0.gguf 27.74 97.7% 0.0033
Inkling-2-Mini-AD-Q6_K.gguf 23.79 97.6% 0.0037 Q8_0 quality, 4 GB smaller
Inkling-2-Mini-Q6_K.gguf 23.40 97.3% 0.0047 llama.cpp preset, no imatrix
Inkling-2-Mini-Q5_K_M.gguf 19.81 96.3% 0.0079 llama.cpp preset, no imatrix
Inkling-2-Mini-AD-Q5_K.gguf 19.07 97.1% 0.0052
Inkling-2-Mini-Q4_K_M.gguf 17.38 94.7% 0.0165 llama.cpp preset, no imatrix
Inkling-2-Mini-AD-Q4_K.gguf 16.53 96.3% 0.0080 Q5_K_M quality, 3.3 GB smaller
Inkling-2-Mini-AD-IQ4_XS.gguf 15.00 95.8% 0.0101
Inkling-2-Mini-AD-IQ3_S.gguf 13.34 94.2% 0.0205
Inkling-2-Mini-AD-IQ3_XXS.gguf 12.58 93.3% 0.0271
Inkling-2-Mini-AD-IQ2_S.gguf 11.56 91.3% 0.047
Inkling-2-Mini-AD-IQ2_XXS.gguf 10.43 88.5% 0.084
Inkling-2-Mini-AD-IQ2_M.gguf 8.67 85.7% 0.130 expert down Q2_0 except the 8 busiest layers
Inkling-2-Mini-AD-IQ2_XXS-Q2.gguf 7.90 84.3% 0.158
Inkling-2-Mini-AD-IQ1_M.gguf 7.26 81.1% 0.236 1-bit
Inkling-2-Mini-AD-IQ1_S.gguf 6.88 79.5% 0.278 1-bit
mmproj-Inkling-2-Mini-F16.gguf / -BF16.gguf 0.64 vision projector (image input)

Even Q8_0 matches the BF16's top token only 97.7% of the time on this model, so 97โ€“98% is the ceiling here, not 99%+. Our guess is that rounding adds up over the 48 layer passes of the loop; we have not tested it.

  • Measured: our neutral corpus in 30 languages, 87 chunks of 4096 tokens, llama.cpp llama-perplexity --kl-divergence against the BF16 logits, one H100 build of PR #32695.
  • How to choose: the memory a file needs is its size plus the KV cache plus about 1 GB of buffers. The KV cache is small here: only 4 of the 24 layers use full attention, the rest a 1024-token window. Good 4-bit files reach 96โ€“97% top-1. The 1- and 2-bit files are for machines where nothing larger fits.
  • The presets' names don't tell the whole story. The expert down projection has rows of 1152, which k-quants cannot take, so llama.cpp swaps the type there: Q8_0 in all 24 layers of Q6_K; Q8_0 in half of the layers and Q5_1 in the rest for Q5_K_M; Q8_0 and Q5_0 for Q4_K_M. The AD files use block-32 types on that tensor by design. The 1- and 2-bit files put Q2_0 there, quantized with the importance matrix (patches/q2_0-imatrix.patch; llama.cpp's own Q2_0 quantizer ignores the matrix).
  • Importance matrix: 4.76M tokens of our calibration corpus rendered through the model's chat template, in 6 shards. Two shards ran on the BF16. Four ran on the Q8_0, because no GPU with room for the 52 GB BF16 was free at release time.
  • Base: the vendor ships the weights in NVFP4 and MXFP8; our BF16 is that checkpoint dequantized.

Run

Build llama.cpp from the PR:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git fetch origin pull/32695/head:inkling2-loop && git checkout inkling2-loop
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --config Release   # -DGGML_METAL=ON on a Mac
./build/bin/llama-server -m Inkling-2-Mini-AD-Q4_K.gguf --mmproj mmproj-Inkling-2-Mini-F16.gguf --jinja -c 32768 -ngl 99
  • Thinking is set by the chat template's reasoning_effort: none, minimal, low, medium, high, xhigh (default 0.9). Pass it as "chat_template_kwargs": {"reasoning_effort": "low"} on /v1/chat/completions. The vendor notes that none does not switch reasoning off completely.
  • Sampling: the vendor's generation config uses temperature 1.0.
  • Checked in llama-server (BF16, AD-IQ1_M, AD-IQ1_S, with the F16 projector): plain chat, reasoning at none and high, a tool call, an image question, and a reply in Chinese.

Related

Downloads last month
-
GGUF
Model size
26B params
Architecture
inkling2
Hardware compatibility
Log In to add your hardware

1-bit

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for AtomicChat/Inkling-2-Mini-GGUF

Quantized
(22)
this model

Collection including AtomicChat/Inkling-2-Mini-GGUF