strata-engine's picture
Update model card README.md and SVG assets
04996a1 verified
|
Raw History Blame Contribute Delete
18.5 kB
metadata
license: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
base_model: thinkingmachines/Inkling-2-Mini
base_model_relation: quantized
pipeline_tag: image-text-to-text
tags:
  - strata
  - inkling-2-mini
  - moe
  - quantization
  - looped-transformer
  - dspark

Strata on GitHub

Inkling 2 Mini for Strata

Run Thinking Machines Lab's Inkling 2 Mini on your own computer. A 4 GiB graphics card is enough to start, with packs made for 12, 16 and 24 GiB cards. Free and open source. No account needed.

Pick your pack 路 Get started 路 Quality 路 Speed 路 Support Strata

26.4B parameters, 3.4B active per token. A 4 GiB graphics card is enough to start. Up to 1M tokens of context, with text and images in.

Inkling 2 Mini is Thinking Machines Lab's open-weights model made to run on local hardware. It is a looped transformer: the middle layers run three times for every token. It has 26.4B parameters and uses 3.4B per token (5.4B when the repeated passes are counted). It reads text and images, writes text, and supports up to 1M tokens of context (1,048,576).

Pick your pack

Pick by your graphics card's memory. Each pack has a recommended card size. Smaller cards work too, from 4 GiB.

Three packs. Recommended GPU 12 GiB: SQ-10, a 10.47 GB download, matches the original 92.35% of the time, the smallest download. Recommended GPU 16 GiB: SQ-15, a 14.35 GB download, matches the original 96.68% of the time. Recommended GPU 24 GiB: SQ-20, a 19.35 GB download, matches the original 100% of the time, identical to the original.

Recommended GPU Pack file Download
12 GiB Inkling-2-Mini-SQ-10.strata 10.47 GB
16 GiB Inkling-2-Mini-SQ-15.strata 14.35 GB
24 GiB Inkling-2-Mini-SQ-20.strata 19.35 GB

"Recommended GPU" is the graphics card memory the pack is made for. The number in a pack's name is its approximate size in GB, not the card it needs. To read images, add Inkling-2-Mini-Vision.strata (0.64 GB). All files are listed below.

  • A 4 GiB graphics card is enough to start. Strata keeps the part of the model that does not fit on the card in system RAM. This works because Inkling 2 Mini is a Mixture-of-Experts model: it uses only 3.4B of its 26.4B parameters for each token.
  • 12 GiB? SQ-10 fits entirely in the card's own memory.
  • 16 GiB? SQ-15 fits entirely in the card's own memory, and picks the same next word as the original 96.68% of the time.
  • 24 GiB? SQ-20 matched the original exactly in every test.

Get started

  1. Install Strata. Download the app for Windows, Linux or Mac. It needs no account.
  2. Choose Inkling 2 Mini and the pack that matches your graphics card.
  3. Start chatting.
Prefer the terminal?

Linux or Mac:

curl -fsSL --connect-timeout 15 --max-time 120 https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh

Windows PowerShell:

irm -TimeoutSec 120 https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex

On Windows, open a new terminal afterwards. Then run strata launch and follow the prompts.

How close is it to the original?

SQ-20 is identical to the original model in every test. SQ-15 and SQ-10 give up a little closeness to fit on smaller cards.

How often each pack picks the same next word as the original model, higher is better. Test prompts: SQ-20 100%, SQ-15 96.68%, SQ-10 92.35%. WikiText-2: SQ-20 100%, SQ-15 95.97%, SQ-10 89.79%. SQ-20 is identical to the original on both.

How to read this. Each pack's next-word predictions are compared with the original model's, on a set of test prompts and on WikiText-2. "Same word" is how often the pack's first choice for the next word is the same as the original's, so higher is better. "Drift" is the average gap between the two sets of predictions, called KL divergence. Zero means identical, so lower is better.

Pack Same word, test Same word, WikiText-2 Drift, test Drift, WikiText-2
SQ-20 100.00% 100.00% 0.00000 0.00000
SQ-15 96.68% 95.97% 0.00889 0.01071
SQ-10 92.35% 89.79% 0.05017 0.07199
How this was measured

The reference is the original model with its own weights. The test prompts were held out from tuning, and are also scored in two parts: short prompts and long prompts of 2K tokens. "Dev" is the set of development prompts. WikiText-2 was run at 2K tokens of context, where the original scores a perplexity of 6.141 (lower is better).

Pack Drift, dev Drift, short test Drift, 2K test WikiText-2 perplexity DSpark AR 伪
SQ-20 0.00000 0.00000 0.00000 6.141 (+0.000) 2.701
SQ-15 0.00880 0.00968 0.00575 6.177 (+0.036) 2.583
SQ-10 0.04340 0.05645 0.02508 6.461 (+0.320) 2.701

Every pack has a DSpark draft model built in. The last column is its AR 伪. The part that reads images, Inkling-2-Mini-Vision.strata, is stored without loss in BF16: 422 tensors, 318.3M parameters.

Performance with Strata

Up to 250 tokens a second for one request, and over 4,000 in total with 64 requests at once. Both on an RTX 5090. Speeds measured with Strata on 20 machines, from integrated graphics and Macs to an RTX 5090.

Output speed with Strata for one request, in tokens per second, with prompt speed beside it, fastest first. RTX 5090: 176.5 to 250.3, prompt 11,306. RTX PRO 5000: 154.4 to 214.5, prompt 6,840+. RTX 5070 Ti: 54.2 to 155.5, prompt 4,237. 2x RTX PRO 6000: 131.6, prompt 7,980. Olares One: 80.5 to 115+, prompt 4,367. Apple M5 Max: 100.0 to 114.5, prompt 3,063. Radeon AI PRO R9700: 99.0 to 106.5, prompt 3,982. RTX 5070: 93.9 to 101.4, prompt 2,216. DGX Spark: 46.1 to 74.3, prompt 2,197. Tesla V100: 70.5, prompt 2,287. 2x RTX A4000: 68.4, prompt 1,890+. Radeon RX 7900 XT: 54.7+, prompt 2,685. Strix Halo 128 GB: 47.3 to 48.0, prompt 1,466. RTX 2080 Ti: 42.8, prompt 1,304. Tesla P100: 35.0, prompt 223. Apple M4 Mac mini: 27.6, prompt 312.3. GTX 1080 Ti: 24.6, prompt 244. Radeon RX 6700 XT: 17.8, prompt 68.5. Radeon 780M: 11.1, prompt 344.2. Intel Arc A750: 6.1, prompt 67.9. A plus sign means at least that much.

Each bar is one machine, from its lowest to its highest measured speed across packs and kinds of prompt. The RTX 5090's top figure is with SQ-10. With SQ-20 it writes 176.5 to 198.9 tokens a second. Prompt speed is the highest measured on that machine.

How speeds were measured, with every figure

Measured with thinking off. All speeds are in tokens per second.

  • Prompt speed is how fast the input is read.
  • One request is how fast the answer is written when one request runs alone, with Strata's built-in draft decoding on (W=4 Beam / DSpark) unless a figure is marked plain.
  • Many at once is the total written across that many requests running together.
  • Hybrid means part of the model stays in system RAM because the pack does not fit in the card's memory.
  • Where a cell has several figures separated by a slash, they follow the order of the packs in that row. "Code", "math" and "chat" are kinds of prompt. A plus sign means at least that much.
Machine Pack Prompt speed One request Many at once
RTX 5090 32 GB SQ-20 11,306 (8,722 at 2K) 198.9 code, 176.5 math 2,094.1 with 32, 4,089.7 with 64
RTX 5090 32 GB SQ-10 10,452 (7,645 at 2K) 250.3 code, 238.7 math 1,492.9 with 32, 3,010.4 with 64
2x RTX PRO 6000 96 GB SQ-20 on two cards 7,980 131.6 2,081.1 with 32
RTX PRO 5000 72 GB SQ-10 6,840+ 192.2 code, 214.5 math, 154.4 chat 1,850+ with 64
Olares One (RTX 5090 Laptop 24 GB) SQ-20 4,367 (3,951 at 2K) 80.5 plain, 115+ with Beam 502.0 with 32
RTX 5070 Ti 16 GB SQ-15 / SQ-10 / SQ-20 hybrid 4,237 / 4,136 / 2,205 88.0 / 155.5 / 54.2 980+ with 32
RTX 5070 12 GB (Windows 11) SQ-10 2,216 93.9 code, 101.4 math 120.0 with 64, hybrid
DGX Spark GB10 128 GB SQ-20 / SQ-10 2,197 / 2,070 46.1 / 74.3 1,145.1 / 805.8 with 64
2x RTX A4000 16 GB Official safetensors and GGUF files, on two cards 1,890+ 68.4 410+ with 32
Tesla V100 32 GB SQ-20 / SQ-10 2,287 / 2,265 70.5 990.1 / 676.0 with 64
RTX 2080 Ti 11 GB SQ-10 hybrid 1,304 42.8 145+ with 16
GTX 1080 Ti 11 GB SQ-10 hybrid 244 24.6 78+ with 8
Tesla P100 16 GB SQ-10 223 35.0 136.9 with 32
Radeon AI PRO R9700 32 GB SQ-10 3,982 (4,308 with 64 at once) 99.0 code, 106.5 math 291.5 with 8, 587.8 with 64
Radeon RX 7900 XT 20 GB (over OCuLink) SQ-10 2,685 (2,873 with 8 at once) 54.7+, about 78+ with Beam 352.3 with 64
Ryzen AI Max+ Strix Halo 128 GB SQ-20 / SQ-10 1,466 (1,654 with 64 at once) 47.3 / 48.0 172.8 with 8, 374.9 with 64
Radeon 780M integrated graphics (60 GB shared memory) SQ-10 344.2 11.1 51.1 with 32
Radeon RX 6700 XT 12 GB SQ-10 hybrid 68.5 17.8 33.4 with 4
Apple M5 Max 128 GB SQ-20 / SQ-15 / SQ-10 3,063 / 3,023 / 2,997 108.9 / 100.0 / 114.5 597.2 / 542.5 with 64
Apple M4 Mac mini 32 GB SQ-10 312.3 27.6 code 73.1 with 32
Intel Arc A750 8 GB SQ-10 hybrid 67.9 6.1 10.8 with 8

Left out of the chart and the headline: two peak readings (222.3 with SQ-20 on the RTX 5090, 28.3 on the M4 Mac mini) and one micro-benchmark reading (306.2 with SQ-10 on the RTX 5090).

Recommended settings

Setting Value
temperature 1.0

This is the default in the original model's own settings file. Two more points from Thinking Machines Lab's model card:

  • Set a limit on output tokens, and on agent turns if you run agents. The model can get stuck in long reasoning loops.
  • Reasoning cannot be switched off fully. The lowest reasoning effort still reasons a little.

Files

File Size
Inkling-2-Mini-SQ-10.strata 10.47 GB (9.75 GiB)
Inkling-2-Mini-SQ-15.strata 14.35 GB (13.36 GiB)
Inkling-2-Mini-SQ-20.strata 19.35 GB (18.02 GiB)
Inkling-2-Mini-Vision.strata, for reading images 0.64 GB (0.59 GiB)

Support Strata

Help keep local AI fast, independent, and open to everyone.
Strata is built independently and released free under the MIT license. Sponsoring on GitHub or buying a coffee directly funds new GPU kernels, model quantizations, and hardware testing.

Sponsor Strata on GitHub   Buy Strata a coffee

Licence