README.md

---
license: apache-2.0
base_model: thinkingmachines/Inkling-Small
library_name: mlx
tags:
- mlx
- vision
- moe
pipeline_tag: image-text-to-text
---

# ToPo-ToPo/Inkling-Small-mlx-4bit

MLX **4bit** conversion of [`thinkingmachines/Inkling-Small`](https://huggingface.co/thinkingmachines/Inkling-Small)
for Apple Silicon (mlx-vlm). 276B total / 12B active sparse MoE (42 layers, 256 routed experts top-6 + 2 shared),
text + image + audio in, text out.

## Requires mlx-vlm >= 0.6.9

0.6.9 is the first release whose `models/inkling` can load an official Inkling checkpoint through the
public loader, and the first that implements the MoE `global_scale` / `gate.bias` tensors. On 0.6.7 / 0.6.8
this repo will not load.

```python
from mlx_vlm import load, generate
model, processor = load("ToPo-ToPo/Inkling-Small-mlx-4bit")
```

The config is the **official schema, unmodified** — no key translation and no loader patches are needed.

## Provenance (self-converted from official weights)
- Source: [`thinkingmachines/Inkling-Small`](https://huggingface.co/thinkingmachines/Inkling-Small) (license: apache-2.0, bf16, 531.9 GB)
- Tool: `mlx-vlm 0.6.9` — `mlx_vlm.convert --hf-path thinkingmachines/Inkling-Small --mlx-path . -q --q-bits 4 --q-group-size 64`
- Effective: **4.506 bits/weight** (138 GiB on disk, ~148.7 GB peak RSS at inference)
- Only edit on top of the conversion: `pad_token` / `eos_token` added to `tokenizer_config.json`
  (the official `TokenizersBackend` config sets neither, so transformers raises on any padded call).
  Both point at existing ids — the vocabulary is unchanged.
- Verified end-to-end through an OpenAI-compatible gateway: text generation, and image input
  (a 640×480 test image expands to 204 vision tokens and is described correctly).

## Reasoning effort

The chat template always injects a `Thinking effort level:` system message (default **0.9**). Control it
with the OpenAI-compatible `reasoning_effort` — `"none"` / `"minimal"` / `"low"` / `"medium"` / `"high"` /
`"max"`, or a float in `[0.0, 0.99]`. `"none"` disables thinking entirely.

When serving over `mlx_vlm.server`, note that Inkling wraps its answer in structural tokens
(`<|message_model|>`, `<|content_text|>`, `<|end_message|>`) which the server's fixed
`_CONTENT_MARKERS` list does not strip, and that its reasoning channel is
`<|content_thinking|>` … `<|end_message|><|message_model|>` rather than one of the built-in marker pairs.
Set `MLX_VLM_THINKING_START_TOKEN` / `MLX_VLM_THINKING_END_TOKEN` accordingly and strip the structural
tokens, or the reasoning and those markers end up in `content`.

## MTP (speculative decoding)

The conversion drops the built-in `model.mtp.*` weights (160 keys in the official bf16), as every
quantized Inkling repo does. Split a drafter from the **official bf16** instead:

```bash
python -m mlx_vlm.speculative.drafters.inkling_mtp.split \
    --model thinkingmachines/Inkling-Small --output Inkling-Small-MTP-bf16
```

As of mlx-vlm 0.6.9 the resulting drafter still cannot be used: the first draft block snapshots an empty
cache and `models/cache.py` dereferences `self.keys` while it is `None`.

## Revision history

- **2026-08-04** — reconverted with mlx-vlm 0.6.9. The previous upload had been converted with 0.6.7,
  whose `models/inkling` did not implement the MoE `mlp.global_scale` (50 keys) and `mlp.gate.bias`
  (40 keys) present in the official checkpoint, so **those tensors were silently dropped**. It also
  shipped a translated config (renamed `intermediate_size` / `dense_intermediate_size`, etc.) that 0.6.9
  rejects. If you pulled this repo before this date, re-download it.