---
license: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
base_model: thinkingmachines/Inkling-2-Mini
base_model_relation: quantized
pipeline_tag: image-text-to-text
tags:
- strata
- inkling-2-mini
- moe
- quantization
- looped-transformer
- dspark
---

Inkling 2 Mini for Strata
Run Thinking Machines Lab's Inkling 2 Mini on your own computer. A 4 GiB graphics card is enough to start, with packs made for 12, 16 and 24 GiB cards. Free and open source. No account needed.
Pick your pack · Get started · Quality · Speed · Support Strata
**Inkling 2 Mini** is Thinking Machines Lab's open-weights model made to run on local hardware. It is a looped transformer: the middle layers run three times for every token. It has 26.4B parameters and uses 3.4B per token (5.4B when the repeated passes are counted). It reads text and images, writes text, and supports up to 1M tokens of context (1,048,576).
## Pick your pack
**Pick by your graphics card's memory.** Each pack has a recommended card size. Smaller cards work too, from 4 GiB.
| Recommended GPU | Pack file | Download |
| ---: | --- | ---: |
| **12 GiB** | `Inkling-2-Mini-SQ-10.strata` | 10.47 GB |
| **16 GiB** | `Inkling-2-Mini-SQ-15.strata` | 14.35 GB |
| **24 GiB** | `Inkling-2-Mini-SQ-20.strata` | 19.35 GB |
"Recommended GPU" is the graphics card memory the pack is made for. The number in a pack's name is its approximate size in GB, not the card it needs. To read images, add `Inkling-2-Mini-Vision.strata` (0.64 GB). [All files](#files) are listed below.
- **A 4 GiB graphics card is enough to start.** Strata keeps the part of the model that does not fit on the card in system RAM. This works because Inkling 2 Mini is a Mixture-of-Experts model: it uses only 3.4B of its 26.4B parameters for each token.
- **12 GiB?** SQ-10 fits entirely in the card's own memory.
- **16 GiB?** SQ-15 fits entirely in the card's own memory, and picks the same next word as the original 96.68% of the time.
- **24 GiB?** SQ-20 matched the original exactly in every test.
## Get started
1. **Install Strata.** [Download the app for Windows, Linux or Mac](https://github.com/Niko1221/Strata/releases/latest). It needs no account.
2. **Choose Inkling 2 Mini** and the pack that matches your graphics card.
3. **Start chatting.**
Prefer the terminal?
Linux or Mac:
```sh
curl -fsSL --connect-timeout 15 --max-time 120 https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
```
Windows PowerShell:
```powershell
irm -TimeoutSec 120 https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex
```
On Windows, open a new terminal afterwards. Then run `strata launch` and follow the prompts.
## How close is it to the original?
**SQ-20 is identical to the original model in every test.** SQ-15 and SQ-10 give up a little closeness to fit on smaller cards.
**How to read this.** Each pack's next-word predictions are compared with the original model's, on a set of test prompts and on WikiText-2. "Same word" is how often the pack's first choice for the next word is the same as the original's, so higher is better. "Drift" is the average gap between the two sets of predictions, called KL divergence. Zero means identical, so lower is better.
| Pack | Same word, test | Same word, WikiText-2 | Drift, test | Drift, WikiText-2 |
| --- | ---: | ---: | ---: | ---: |
| **SQ-20** | **100.00%** | **100.00%** | **0.00000** | **0.00000** |
| SQ-15 | 96.68% | 95.97% | 0.00889 | 0.01071 |
| SQ-10 | 92.35% | 89.79% | 0.05017 | 0.07199 |
How this was measured
The reference is the original model with its own weights. The test prompts were held out from tuning, and are also scored in two parts: short prompts and long prompts of 2K tokens. "Dev" is the set of development prompts. WikiText-2 was run at 2K tokens of context, where the original scores a perplexity of 6.141 (lower is better).
| Pack | Drift, dev | Drift, short test | Drift, 2K test | WikiText-2 perplexity | DSpark AR α |
| --- | ---: | ---: | ---: | ---: | ---: |
| **SQ-20** | 0.00000 | 0.00000 | 0.00000 | 6.141 (+0.000) | 2.701 |
| SQ-15 | 0.00880 | 0.00968 | 0.00575 | 6.177 (+0.036) | 2.583 |
| SQ-10 | 0.04340 | 0.05645 | 0.02508 | 6.461 (+0.320) | 2.701 |
Every pack has a DSpark draft model built in. The last column is its AR α. The part that reads images, `Inkling-2-Mini-Vision.strata`, is stored without loss in BF16: 422 tensors, 318.3M parameters.
## Performance with Strata
**Up to 250 tokens a second for one request, and over 4,000 in total with 64 requests at once.** Both on an RTX 5090. Speeds measured with Strata on 20 machines, from integrated graphics and Macs to an RTX 5090.
Each bar is one machine, from its lowest to its highest measured speed across packs and kinds of prompt. The RTX 5090's top figure is with SQ-10. With SQ-20 it writes 176.5 to 198.9 tokens a second. Prompt speed is the highest measured on that machine.
How speeds were measured, with every figure
Measured with thinking off. All speeds are in tokens per second.
- **Prompt speed** is how fast the input is read.
- **One request** is how fast the answer is written when one request runs alone, with Strata's built-in draft decoding on (W=4 Beam / DSpark) unless a figure is marked plain.
- **Many at once** is the total written across that many requests running together.
- **Hybrid** means part of the model stays in system RAM because the pack does not fit in the card's memory.
- Where a cell has several figures separated by a slash, they follow the order of the packs in that row. "Code", "math" and "chat" are kinds of prompt. A plus sign means at least that much.
| Machine | Pack | Prompt speed | One request | Many at once |
| --- | --- | ---: | ---: | ---: |
| RTX 5090 32 GB | SQ-20 | 11,306 (8,722 at 2K) | 198.9 code, 176.5 math | 2,094.1 with 32, 4,089.7 with 64 |
| RTX 5090 32 GB | SQ-10 | 10,452 (7,645 at 2K) | 250.3 code, 238.7 math | 1,492.9 with 32, 3,010.4 with 64 |
| 2x RTX PRO 6000 96 GB | SQ-20 on two cards | 7,980 | 131.6 | 2,081.1 with 32 |
| RTX PRO 5000 72 GB | SQ-10 | 6,840+ | 192.2 code, 214.5 math, 154.4 chat | 1,850+ with 64 |
| Olares One (RTX 5090 Laptop 24 GB) | SQ-20 | 4,367 (3,951 at 2K) | 80.5 plain, 115+ with Beam | 502.0 with 32 |
| RTX 5070 Ti 16 GB | SQ-15 / SQ-10 / SQ-20 hybrid | 4,237 / 4,136 / 2,205 | 88.0 / 155.5 / 54.2 | 980+ with 32 |
| RTX 5070 12 GB (Windows 11) | SQ-10 | 2,216 | 93.9 code, 101.4 math | 120.0 with 64, hybrid |
| DGX Spark GB10 128 GB | SQ-20 / SQ-10 | 2,197 / 2,070 | 46.1 / 74.3 | 1,145.1 / 805.8 with 64 |
| 2x RTX A4000 16 GB | Official safetensors and GGUF files, on two cards | 1,890+ | 68.4 | 410+ with 32 |
| Tesla V100 32 GB | SQ-20 / SQ-10 | 2,287 / 2,265 | 70.5 | 990.1 / 676.0 with 64 |
| RTX 2080 Ti 11 GB | SQ-10 hybrid | 1,304 | 42.8 | 145+ with 16 |
| GTX 1080 Ti 11 GB | SQ-10 hybrid | 244 | 24.6 | 78+ with 8 |
| Tesla P100 16 GB | SQ-10 | 223 | 35.0 | 136.9 with 32 |
| Radeon AI PRO R9700 32 GB | SQ-10 | 3,982 (4,308 with 64 at once) | 99.0 code, 106.5 math | 291.5 with 8, 587.8 with 64 |
| Radeon RX 7900 XT 20 GB (over OCuLink) | SQ-10 | 2,685 (2,873 with 8 at once) | 54.7+, about 78+ with Beam | 352.3 with 64 |
| Ryzen AI Max+ Strix Halo 128 GB | SQ-20 / SQ-10 | 1,466 (1,654 with 64 at once) | 47.3 / 48.0 | 172.8 with 8, 374.9 with 64 |
| Radeon 780M integrated graphics (60 GB shared memory) | SQ-10 | 344.2 | 11.1 | 51.1 with 32 |
| Radeon RX 6700 XT 12 GB | SQ-10 hybrid | 68.5 | 17.8 | 33.4 with 4 |
| Apple M5 Max 128 GB | SQ-20 / SQ-15 / SQ-10 | 3,063 / 3,023 / 2,997 | 108.9 / 100.0 / 114.5 | 597.2 / 542.5 with 64 |
| Apple M4 Mac mini 32 GB | SQ-10 | 312.3 | 27.6 code | 73.1 with 32 |
| Intel Arc A750 8 GB | SQ-10 hybrid | 67.9 | 6.1 | 10.8 with 8 |
Left out of the chart and the headline: two peak readings (222.3 with SQ-20 on the RTX 5090, 28.3 on the M4 Mac mini) and one micro-benchmark reading (306.2 with SQ-10 on the RTX 5090).
## Recommended settings
| Setting | Value |
| --- | --- |
| `temperature` | 1.0 |
This is the default in the original model's own settings file. Two more points from Thinking Machines Lab's model card:
- **Set a limit on output tokens**, and on agent turns if you run agents. The model can get stuck in long reasoning loops.
- **Reasoning cannot be switched off fully.** The lowest reasoning effort still reasons a little.
## Files
| File | Size |
| --- | ---: |
| `Inkling-2-Mini-SQ-10.strata` | 10.47 GB (9.75 GiB) |
| `Inkling-2-Mini-SQ-15.strata` | 14.35 GB (13.36 GiB) |
| `Inkling-2-Mini-SQ-20.strata` | 19.35 GB (18.02 GiB) |
| `Inkling-2-Mini-Vision.strata`, for reading images | 0.64 GB (0.59 GiB) |
## Support Strata
Help keep local AI fast, independent, and open to everyone.
Strata is built independently and released free under the MIT license. Sponsoring on GitHub or buying a coffee directly funds new GPU kernels, model quantizations, and hardware testing.
## Licence
- **The model:** Inkling 2 Mini is made by Thinking Machines Lab and released under the [Apache 2.0 licence](https://www.apache.org/licenses/LICENSE-2.0), with a [Model Acceptable Use Policy](https://thinkingmachines.ai/model-acceptable-use-policy/). The files here are builds of [thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) packed for Strata.
- **The Strata engine:** [MIT](https://github.com/Niko1221/Strata/blob/main/LICENSE).