card: README.md

README.md

@@ -1,134 +1,120 @@
 ---
 license: apache-2.0
-license_link: https://huggingface.co/thinkingmachines/Inkling/blob/main/LICENSE
-base_model:
-- thinkingmachines/Inkling
+base_model: thinkingmachines/Inkling
 base_model_relation: quantized
 quantized_by: AtomicChat
 pipeline_tag: text-generation
-library_name: gguf
 tags:
 - atomic-chat
 - inkling
 - thinking-machines
 - moe
-- multimodal
-- gguf
 - imatrix
 - quantized
 - llama.cpp
 ---
 
-<center>
+[![Atomic Chat](https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/pill_atomic_v3.png)](https://atomic.chat)
+[![Join Discord](https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/pill_discord_v3.png)](https://discord.gg/8wGSsvmg4V)
+[![GitHub](https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/pill_github_v3.png)](https://github.com/AtomicBot-ai/Atomic-Chat)
 
-<div style="display:flex; justify-content:center; align-items:center; gap:2%; max-width:560px; margin:0 auto;">
-<a href="https://atomic.chat" style="flex:0 1 auto; min-width:0;"><img src="https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/pill_atomic_v3.png" alt="Atomic Chat" style="width:100%; height:auto; max-width:186px;"></a>
-<a href="https://discord.gg/8wGSsvmg4V" style="flex:0 1 auto; min-width:0;"><img src="https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/pill_discord_v3.png" alt="Join Discord" style="width:100%; height:auto; max-width:184px;"></a>
-<a href="https://github.com/AtomicBot-ai/Atomic-Chat" style="flex:0 1 auto; min-width:0;"><img src="https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/pill_github_v3.png" alt="GitHub" style="width:100%; height:auto; max-width:141px;"></a>
-</div>
+![Inkling](https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/hero.png)
 
-<br/>
+[**Base model: thinkingmachines/Inkling**](https://huggingface.co/thinkingmachines/Inkling)
 
-<img src="https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/hero.png" alt="Inkling" style="width:420px; max-width:100%; height:auto; margin-bottom:0.6em;"/>
-
-<div style="display:flex; justify-content:center; gap:0.5em;">
-<a href="https://huggingface.co/thinkingmachines/Inkling"><strong>Base model: thinkingmachines/Inkling</strong></a>
-</div>
-</center>
-
-**Inkling**, self-quantized to GGUF by [Atomic Chat](https://atomic.chat). Built straight from Thinking Machines' original weights with a per-tensor importance matrix, which we publish alongside the quants. Runs fully offline, including a 1-bit build that brings this 975B model down to 226GB.
+**Inkling** (Thinking Machines Lab), self-quantized to GGUF by [Atomic Chat](https://atomic.chat). Built straight from Thinking Machines' original BF16 weights with a per-tensor importance matrix. Runs fully offline, including a 1-bit build that brings this 975B model down to 226 GB.
 
 ## Highlights
 
-- **975B parameters with 41B active**: a sparse Mixture-of-Experts backbone with 256 experts, top-6 routed per token plus 2 shared experts always on.
-- **1M context** across 66 decoder layers that alternate sliding-window and global attention in a 5:1 pattern.
-- **Relative attention instead of RoPE**: each attention layer learns position directly in the attention logits via a per-token, per-head relative feature.
-- **Natively multimodal** (image, text and audio in, text out) with a hierarchical MLP patchifier for vision and a discretized mel spectrogram for audio. These GGUF quants cover the text path.
-- **Frontier scores** (Thinking Machines-reported): AIME 2026 97.1, VoiceBench 91.4, GPQA Diamond 87.2, SWEBench Verified 77.6, MMMU Pro 73.3.
-- **Built for reasoning across modalities**, agentic and tool-use systems, and domain adaptation via fine-tuning.
-- **Full imatrix quantization** over [`calibration_datav3`](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8), down to a 1-bit `IQ1_M`.
+- **975B-parameter MoE with 41B active**: each token is routed to 6 of 256 experts, plus 2 shared experts active on every token, across a 66-layer decoder.
+- **Context up to 1M tokens** with a hybrid of local and global attention layers.
+- **Natively multimodal base**: the original model reasons over text, images and audio in a shared hidden space. This repo ships the text path.
+- **Strong reasoning scores** (Thinking Machines-reported, effort=0.99): AIME 2026 97.1, GPQA Diamond 87.2, SWEBench Verified 77.6.
+- **Built to be fine-tuned**: Thinking Machines positions Inkling as a base for domain adaptation, released under Apache 2.0.
+- **Full imatrix quantization over a code corpus**, including a 1-bit `IQ1_M` and an `MXFP4` build with Q8 attention and routing.
 
-> [!NOTE]
 > These GGUFs are **self-quantized from the original weights**, not a repack. The importance matrix keeps low-bit quants closer to the full-precision model.
+> Always pass `--jinja` so the **Inkling chat template** (interleaved thinking and tool calls) is applied. Without it the model can emit malformed turns.
+
+> **Support status:** the `inkling` architecture is not yet in a mainline llama.cpp release. Until [PR #25731](https://github.com/ggml-org/llama.cpp/pull/25731) is merged, build llama.cpp from that PR (instructions below). Standard Ollama / LM Studio flows will work once support lands upstream.
 
-> [!IMPORTANT]
-> Always pass `--jinja` so the **Inkling chat template** is applied. Without it the model can emit malformed turns.
+> This repo ships the **text path only**: no vision or audio projector (`mmproj`) is included. For multimodal use, run the original weights.
 
 ## Model Overview
 
 | Property | Value |
-|---|---|
+| --- | --- |
 | Base model | `thinkingmachines/Inkling` |
 | Total / active parameters | 975B total / 41B active |
-| Layers | 66, alternating sliding-window and global attention (5:1) |
-| Experts | 256 experts, top-6 routed + 2 shared always active |
-| Context length | 1M |
-| Architecture | Decoder-only sparse MoE transformer; relative attention in the logits (no RoPE); natively multimodal (hierarchical patchifier for vision, discretized mel for audio) |
-| This repo | GGUF quants (imatrix), text path: `IQ1_M` at 1-bit, `MXFP4`, and `Q8_0` for reference. Files are multi-part; the importance matrix we built is published here too. |
+| Layers | 66 |
+| Experts | 256 routed (top-6) + 2 shared, active on every token |
+| Context length | up to 1M tokens |
+| Architecture | Decoder-only Mixture-of-Experts, hybrid local/global attention, natively multimodal (text, image, audio in; text out) |
+| This repo | GGUF quants (imatrix), text path: `Q8_0` reference, `MXFP4` with Q8 attention/routing, and a 1-bit `IQ1_M` (226 GB) |
 
-<img src="https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/benchmark.png" alt="Inkling benchmark scores" style="width:100%; max-width:900px;"/>
-
-Scores are Thinking Machines's published results for the base `thinkingmachines/Inkling`. Quantization preserves the large majority of this; `Q4_K_M` and up sit within a point or two of full precision.
+![Inkling benchmark scores](https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/benchmark.png)
 
+Scores are Thinking Machines' published results for the base `thinkingmachines/Inkling`, reported at thinking effort 0.99. Quantization preserves the large majority of this; low-bit builds trade some quality for size.
 
 ## Choosing a quant
 
 | Quant | Size | Notes |
-|---|---|---|
-| **`IQ1_M`** | 226 GB | **Smallest. 1-bit imatrix build that brings the 975B model down to 226GB. Expect quality tradeoffs at this bitrate.** |
-| **`MXFP4`** | 514 GB | **4-bit MXFP4. The balanced pick: far closer to reference than 1-bit, at half the size of Q8_0.** |
-| `Q8_0` | 1006 GB | Effectively lossless, reference quality. For rigs that can hold the full thing. |
+| --- | --- | --- |
+| **`IQ1_M`** | 226 GB | **Smallest. 1-bit imatrix build that makes a 975B model runnable on a single big-RAM server (about 226 GB plus context). Expect quality tradeoffs; reasoning still works.** |
+| **`MXFP4`** | 479 GB | **Recommended for quality. 4.34 bits per weight: expert FFN weights in MXFP4, with attention, expert router and shared experts held at Q8_0.** |
+| `Q8_0` | 1.01 TB | Reference quality, near-lossless. Also the substrate our importance matrix was computed on. For large multi-node or big-RAM setups. |
 
 ## Get started
 
-Inkling is a 975B model, so each quant ships as a **multi-part GGUF** inside its own folder (`IQ1_M-final/`, `MXFP4/`, `Q8_0/`). Pull the folder you want, then point llama.cpp at the **first shard** and it loads the rest automatically.
-
-- **[Atomic Chat](https://atomic.chat):** the easiest path. Open the app, search `AtomicChat/Inkling-GGUF`, pick a quant, hit **Use this model**.
-
-Download one quant:
+Inkling needs a llama.cpp build with the `inkling` architecture (see **Run in llama.cpp** below). Then:
 
 ```bash
-hf download AtomicChat/Inkling-GGUF --include "MXFP4/*" --local-dir Inkling
+./build/bin/llama-server -hf AtomicChat/Inkling-GGUF:IQ1_M --jinja -c 8192
 ```
 
-Run it:
+Or download a quant explicitly:
 
 ```bash
-llama-server -m Inkling/MXFP4/Inkling-Atomic-MXFP4-00001-of-00013.gguf \
-    --jinja -ngl 99 -c 8192 -fa on
+hf download AtomicChat/Inkling-GGUF --include "IQ1_M-final/*" --local-dir Inkling-GGUF
+# "MXFP4/*" for 4-bit, "Q8_0/*" for the reference build
 ```
 
+<!-- Atomic Chat: раскомментируй, когда билд приложения включает поддержку inkling (PR #25731)
+- **[Atomic Chat](https://atomic.chat):** the easiest path. Open the app, search `AtomicChat/Inkling-GGUF`, pick a quant, hit **Use this model**.
+-->
+
 ## Best practices
 
-| Parameter | Value |
-|---|---|
-| temperature | 1.0 |
-| top_p | 1.0 |
+Thinking Machines does not publish recommended sampler settings for local inference. Two things do matter:
 
-Thinking Machines does not publish recommended sampling settings for Inkling. These are neutral starting points, so tune them for your task.
+- Always pass `--jinja`. The Inkling template carries the model's interleaved thinking and tool-call blocks; without it output breaks.
+- Benchmark numbers above are reported at **thinking effort 0.99**. Inkling's thinking effort is adjustable, so shorter-thinking runs will score below the chart.
 
 ## Run in llama.cpp
 
+Inkling support lives in [PR #25731](https://github.com/ggml-org/llama.cpp/pull/25731) until it is merged upstream:
+
 ```bash
-git clone https://github.com/ggerganov/llama.cpp
-cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
-cmake --build llama.cpp/build --config Release -j --target llama-cli llama-server
+git clone https://github.com/ggml-org/llama.cpp
+cd llama.cpp
+gh pr checkout 25731
+cmake -B build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
+cmake --build build --config Release -j --target llama-cli llama-server
 ```
 
 ```bash
-./llama.cpp/build/bin/llama-server \
-    -hf AtomicChat/Inkling-GGUF:IQ1_M \
+./build/bin/llama-server \
+    -hf AtomicChat/Inkling-GGUF:MXFP4 \
     --jinja -ngl 99 -c 8192 -fa on
 ```
 
 ## How these were made
 
-1. Download `thinkingmachines/Inkling` (original weights).
-2. Convert to GGUF with a [llama.cpp](https://github.com/ggerganov/llama.cpp) build that supports the Inkling architecture (banded relative attention, hybrid sliding-window and global layers).
-3. Build an importance matrix over `calibration_datav3`. We publish it in this repo under `imatrix/`.
-4. Quantize with `--imatrix`: `Q8_0` as the reference, `MXFP4` for balance, and `IQ1_M` for the smallest footprint that keeps this 975B model coherent.
+1. Download `thinkingmachines/Inkling` (original BF16 weights, about 2 TB).
+2. Convert to GGUF with llama.cpp built from [PR #25731](https://github.com/ggml-org/llama.cpp/pull/25731), which adds the `inkling` architecture.
+3. Produce a `Q8_0` reference and compute an importance matrix over an 18 MB code corpus (Linux, CPython, Rust and llama.cpp sources), 7,040 chunks of 512 tokens, with 93-99% expert activation coverage. The imatrix files are published in [`imatrix/`](https://huggingface.co/AtomicChat/Inkling-GGUF/tree/main/imatrix).
+4. Quantize with `--imatrix`: `MXFP4` for expert FFNs with attention, router and shared experts at Q8_0, and `IQ1_M` with the same Q8 overlay for the smallest coherent build.
 
 ## License
 
 Released by Thinking Machines Lab under the Apache 2.0 license. Quantized by Atomic Chat.
-