card: README.md

README.md

@@ -0,0 +1,134 @@
+---
+license: apache-2.0
+license_link: https://huggingface.co/thinkingmachines/Inkling/blob/main/LICENSE
+base_model:
+- thinkingmachines/Inkling
+base_model_relation: quantized
+quantized_by: AtomicChat
+pipeline_tag: text-generation
+library_name: gguf
+tags:
+- atomic-chat
+- inkling
+- thinking-machines
+- moe
+- multimodal
+- gguf
+- imatrix
+- quantized
+- llama.cpp
+---
+
+<center>
+
+<div style="display:flex; justify-content:center; align-items:center; gap:2%; max-width:560px; margin:0 auto;">
+<a href="https://atomic.chat" style="flex:0 1 auto; min-width:0;"><img src="https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/pill_atomic_v3.png" alt="Atomic Chat" style="width:100%; height:auto; max-width:186px;"></a>
+<a href="https://discord.gg/8wGSsvmg4V" style="flex:0 1 auto; min-width:0;"><img src="https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/pill_discord_v3.png" alt="Join Discord" style="width:100%; height:auto; max-width:184px;"></a>
+<a href="https://github.com/AtomicBot-ai/Atomic-Chat" style="flex:0 1 auto; min-width:0;"><img src="https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/pill_github_v3.png" alt="GitHub" style="width:100%; height:auto; max-width:141px;"></a>
+</div>
+
+<br/>
+
+<img src="https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/hero.png" alt="Inkling" style="width:420px; max-width:100%; height:auto; margin-bottom:0.6em;"/>
+
+<div style="display:flex; justify-content:center; gap:0.5em;">
+<a href="https://huggingface.co/thinkingmachines/Inkling"><strong>Base model: thinkingmachines/Inkling</strong></a>
+</div>
+</center>
+
+**Inkling**, self-quantized to GGUF by [Atomic Chat](https://atomic.chat). Built straight from Thinking Machines' original weights with a per-tensor importance matrix, which we publish alongside the quants. Runs fully offline, including a 1-bit build that brings this 975B model down to 226GB.
+
+## Highlights
+
+- **975B parameters with 41B active**: a sparse Mixture-of-Experts backbone with 256 experts, top-6 routed per token plus 2 shared experts always on.
+- **1M context** across 66 decoder layers that alternate sliding-window and global attention in a 5:1 pattern.
+- **Relative attention instead of RoPE**: each attention layer learns position directly in the attention logits via a per-token, per-head relative feature.
+- **Natively multimodal** (image, text and audio in, text out) with a hierarchical MLP patchifier for vision and a discretized mel spectrogram for audio. These GGUF quants cover the text path.
+- **Frontier scores** (Thinking Machines-reported): AIME 2026 97.1, VoiceBench 91.4, GPQA Diamond 87.2, SWEBench Verified 77.6, MMMU Pro 73.3.
+- **Built for reasoning across modalities**, agentic and tool-use systems, and domain adaptation via fine-tuning.
+- **Full imatrix quantization** over [`calibration_datav3`](https://gist.github.com/bartowski1182/eb213dccb3571f863da82e99418f81e8), down to a 1-bit `IQ1_M`.
+
+> [!NOTE]
+> These GGUFs are **self-quantized from the original weights**, not a repack. The importance matrix keeps low-bit quants closer to the full-precision model.
+
+> [!IMPORTANT]
+> Always pass `--jinja` so the **Inkling chat template** is applied. Without it the model can emit malformed turns.
+
+## Model Overview
+
+| Property | Value |
+|---|---|
+| Base model | `thinkingmachines/Inkling` |
+| Total / active parameters | 975B total / 41B active |
+| Layers | 66, alternating sliding-window and global attention (5:1) |
+| Experts | 256 experts, top-6 routed + 2 shared always active |
+| Context length | 1M |
+| Architecture | Decoder-only sparse MoE transformer; relative attention in the logits (no RoPE); natively multimodal (hierarchical patchifier for vision, discretized mel for audio) |
+| This repo | GGUF quants (imatrix), text path: `IQ1_M` at 1-bit, `MXFP4`, and `Q8_0` for reference. Files are multi-part; the importance matrix we built is published here too. |
+
+<img src="https://huggingface.co/AtomicChat/Inkling-GGUF/resolve/main/benchmark.png" alt="Inkling benchmark scores" style="width:100%; max-width:900px;"/>
+
+Scores are Thinking Machines's published results for the base `thinkingmachines/Inkling`. Quantization preserves the large majority of this; `Q4_K_M` and up sit within a point or two of full precision.
+
+
+## Choosing a quant
+
+| Quant | Size | Notes |
+|---|---|---|
+| **`IQ1_M`** | 226 GB | **Smallest. 1-bit imatrix build that brings the 975B model down to 226GB. Expect quality tradeoffs at this bitrate.** |
+| **`MXFP4`** | 514 GB | **4-bit MXFP4. The balanced pick: far closer to reference than 1-bit, at half the size of Q8_0.** |
+| `Q8_0` | 1006 GB | Effectively lossless, reference quality. For rigs that can hold the full thing. |
+
+## Get started
+
+Inkling is a 975B model, so each quant ships as a **multi-part GGUF** inside its own folder (`IQ1_M-final/`, `MXFP4/`, `Q8_0/`). Pull the folder you want, then point llama.cpp at the **first shard** and it loads the rest automatically.
+
+- **[Atomic Chat](https://atomic.chat):** the easiest path. Open the app, search `AtomicChat/Inkling-GGUF`, pick a quant, hit **Use this model**.
+
+Download one quant:
+
+```bash
+hf download AtomicChat/Inkling-GGUF --include "MXFP4/*" --local-dir Inkling
+```
+
+Run it:
+
+```bash
+llama-server -m Inkling/MXFP4/Inkling-Atomic-MXFP4-00001-of-00013.gguf \
+    --jinja -ngl 99 -c 8192 -fa on
+```
+
+## Best practices
+
+| Parameter | Value |
+|---|---|
+| temperature | 1.0 |
+| top_p | 1.0 |
+
+Thinking Machines does not publish recommended sampling settings for Inkling. These are neutral starting points, so tune them for your task.
+
+## Run in llama.cpp
+
+```bash
+git clone https://github.com/ggerganov/llama.cpp
+cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
+cmake --build llama.cpp/build --config Release -j --target llama-cli llama-server
+```
+
+```bash
+./llama.cpp/build/bin/llama-server \
+    -hf AtomicChat/Inkling-GGUF:IQ1_M \
+    --jinja -ngl 99 -c 8192 -fa on
+```
+
+## How these were made
+
+1. Download `thinkingmachines/Inkling` (original weights).
+2. Convert to GGUF with a [llama.cpp](https://github.com/ggerganov/llama.cpp) build that supports the Inkling architecture (banded relative attention, hybrid sliding-window and global layers).
+3. Build an importance matrix over `calibration_datav3`. We publish it in this repo under `imatrix/`.
+4. Quantize with `--imatrix`: `Q8_0` as the reference, `MXFP4` for balance, and `IQ1_M` for the smallest footprint that keeps this 975B model coherent.
+
+## License
+
+Released by Thinking Machines Lab under the Apache 2.0 license. Quantized by Atomic Chat.
+