Paris99's picture
Card: builds table and memory guidance
2f36859 verified
|
Raw History Blame Contribute Delete
7.47 kB
---
license: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
base_model: thinkingmachines/Inkling-2-Mini
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: mlx
tags:
- mlx
- apple-silicon
- inkling2_loop
- moe
- dspark
- speculative-decoding
---
# Inkling 2 Mini for Apple Silicon (MLX, 4/8-bit, lossless)
[thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) (revision `6e7c824`) for MLX on
Apple Silicon, with its DSpark speculative-decoding draft bundled (`mtp.safetensors`) and its vision encoder included.
The routed experts are kept exactly as Thinking Machines released them: 4-bit NVFP4 in layers 0–19 and 8-bit MXFP8 in layers 20–23, repacked into MLX's native nvfp4 / mxfp8 formats with every value unchanged. Attention, embeddings, the output layer and the vision encoder stay in the original bf16. This is the release itself, in MLX. Size: 20.2 GB plus the 0.64 GB draft.
Inkling 2 Mini is Thinking Machines' looped mixture-of-experts model: 26.4B parameters, about 3.4B active per token,
24 layers of which the middle twelve run three times (48 layer passes per token), text and image input, up to 1M tokens of context.
## Builds
| Build | Size | Same next token as Thinking Machines' code | KL vs their code | GSM8K (50 questions) | Peak memory, 8K-token prompt |
|---|---|---|---|---|---|
| [netprojects/Inkling-2-Mini-MLX-4-8bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-4-8bit) (recommended) | 20.2 GB + 0.64 GB draft | 98.6% | 0.0018 | 47/50 | 27.8 GB |
| [netprojects/Inkling-2-Mini-MLX-3bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-3bit) | 14.7 GB + 0.64 GB draft | 95.7% | 0.0421 | 48/50 | 21.2 GB |
| [netprojects/Inkling-2-Mini-MLX-2.5bit](https://huggingface.co/netprojects/Inkling-2-Mini-MLX-2.5bit) | 11.5 GB + 0.64 GB draft | 87.3% | 0.1904 | 45/50 | 18.2 GB |
"Thinking Machines' code" is their Transformers implementation of Inkling 2 Mini, run on the released weights; the
columns are explained under "What was checked".
## Install
Inkling 2 Mini support is not in a released MLX-VLM yet. Install MLX-VLM with it from [internet-projects/inkling2-mlx](https://github.com/internet-projects/inkling2-mlx)
(MLX-VLM plus Inkling 2 Mini support; Python 3.10 or newer):
```bash
pip install "git+https://github.com/internet-projects/inkling2-mlx@v0.1.0" torch torchvision
```
PyTorch and torchvision are only used on the CPU by the image processor.
## Use
```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
model, processor = load("netprojects/Inkling-2-Mini-MLX-4-8bit")
prompt = apply_chat_template(processor, model.config, "Explain in two sentences why the sky is blue.",
reasoning_effort="low")
print(generate(model, processor, prompt, max_tokens=2048, temperature=0.0).text)
```
`reasoning_effort` is part of Inkling's chat template: `none`, `minimal`, `low`, `medium`, `high` (the default), `xhigh`,
`max`, or a number from 0.0 to 0.99. Thinking Machines notes that the model can think at length even at `none` and can get
stuck in long reasoning loops, so set `max_tokens`.
With an image:
```python
prompt = apply_chat_template(processor, model.config, "What error does this terminal show?", num_images=1, reasoning_effort="low")
print(generate(model, processor, prompt, image=["screenshot.png"], max_tokens=2048).text)
```
With the DSpark draft (greedy output stays the model's own, up to near-ties):
```python
from mlx_vlm.speculative.drafters import load_drafter
draft, kind = load_drafter("netprojects/Inkling-2-Mini-MLX-4-8bit")
print(generate(model, processor, prompt, max_tokens=2048, draft_model=draft, draft_kind=kind).text)
```
OpenAI-compatible server, with tool calls and the thinking returned separately (`reasoning_content`):
```bash
mlx_vlm.server --model netprojects/Inkling-2-Mini-MLX-4-8bit --port 8000
```
## Thinking Machines' results
From the [Inkling 2 Mini card](https://huggingface.co/thinkingmachines/Inkling-2-Mini), for the original model (their
harnesses and settings; a selection of their table):
| Benchmark (Thinking Machines' figures) | Inkling 2 Mini | Qwen3.8-27B | Gemma 4 31B |
|---|---|---|---|
| Artificial Analysis Intelligence Index v5.0 | 20.1 | 20.1 | 4.4 |
| SWE-bench Verified (bash only) | 79.2% | 77.8% | 46.1% |
| SWE-bench Pro v2 Hard (mini-swe-agent) | 77.9% | 73.8% | 14.2% |
| Terminal-Bench 2.1 (pi) | 71.6% | 73.0% | 43.4% |
| BrowseComp (with compaction) | 79.0% | 61.6% | 37.0% |
| MCP Atlas | 82.7% | 77.3% | 54.2% |
| MMMU Pro | 74.9% | 76.3% | 73.4% |
| OmniDocBench v1.6 | 92.8% | 91.5% | 82.5% |
| LongBench v2 | 66.9% | 52.3% | 52.9% |
| ARC-AGI v1 / v2 | 84.5 / 49.3% | 87.5 / 42.4% | – |
## What was checked
On a MacBook Pro M5 Max (128 GB), 21 November 2026, with MLX 0.32.4 and [inkling2-mlx](https://github.com/internet-projects/inkling2-mlx) v0.1.0.
- **Against Thinking Machines' code.** Their Transformers implementation (pull request #51050) ran on the CPU with the
released weights, on 12 short chat texts (488 answer positions). This build gave the same next token at
98.6% of positions (mean KL 0.0018), and its greedy answers to 3 prompts matched theirs token for token (24 tokens each). Every tensor of this build is bit-identical to the release as MLX reads it.
- **GSM8K, 50 fixed test questions** (reasoning `none`, greedy): 47/50.
- **Tool calling** (reasoning `none`): an agent prompt with `bash` and `read` tools produced a well-formed call in Inkling's
format, which `mlx_vlm.server` returns as OpenAI `tool_calls`.
- **Coding:** 3/3 short Python tasks passed their asserts. **Thinking** (reasoning `low`): 2/2
questions answered correctly after one closed thinking block. **Vision:** named the red square and blue circle of a test image.
- **Memory:** peak 26.9 GB with a 4K-token prompt and 27.8 GB with an 8K-token prompt (mlx-vlm defaults). With a smaller prefill step the peak drops: 22.3 / 22.6 GB at `prefill_step_size=512` and 21.4 / 21.6 GB at `prefill_step_size=256` (4K / 8K-token prompts, plus 0.64 GB if the DSpark draft is loaded), which fits the default GPU memory limit of a 36 GB Mac: pass `prefill_step_size=256` to `generate`, or `--prefill-step-size 256` to `mlx_vlm.server`.
- **DSpark:** the bundled draft loads with this build and drafts about 7 tokens per round.
Precision note: Thinking Machines released the routed experts in 4-bit NVFP4 (8-bit MXFP8 in the last four layers) and no higher-precision weights; MLX builds of this model keep the experts at that precision, so this 4/8-bit build already carries the full released precision.
Known limitation of the current server: when the model calls a tool, `content` can also carry a short sentence of its
reasoning and the tool name.
## License and credits
Apache-2.0, as Inkling 2 Mini. Use of the model is also governed by Thinking Machines'
[Model Acceptable Use Policy](https://thinkingmachines.ai/model-acceptable-use-policy/). `LICENSE` is the Apache-2.0
text and `NOTICE` describes the conversion. Inkling 2 Mini is by Thinking Machines Lab; please cite their work.
Published by [NetProjects](https://netprojects.com) (Internet Projects Ltd). Not affiliated with Thinking Machines.
More MLX builds from NetProjects: [huggingface.co/netprojects](https://huggingface.co/netprojects) · [netprojects.com](https://netprojects.com)