README.md

---
license: apache-2.0
library_name: mlx
tags: [mlx, inkling, moe, multimodal, text-generation, image-text-to-text, apple-silicon]
base_model: thinkingmachines/Inkling-Small
pipeline_tag: image-text-to-text
---

# Inkling-Small-mlx-3bit

An **MLX 3bit** build of [`thinkingmachines/Inkling-Small`](https://huggingface.co/thinkingmachines/Inkling-Small)
— 264 B total parameters, **~12 B active**; 42 layers, 256 routed experts (top-6) + 2 shared;
natively multimodal (image + audio in, text out).

Sized to run **fully resident on a 128 GB Mac** — no expert-offload, no expert pruning, no
layer streaming. The whole model sits in unified memory and decodes at conversational speed.

> Why this model is the interesting one for Apple Silicon: speed follows the **active**
> parameter count, not how hard you compress. Inkling-Small activates ~12 B params per token,
> so this tier reads roughly 9.8 GB per token. The 975 B Inkling, by contrast, needs SSD
> expert-offload even at 2-bit and manages ~0.2–0.4 tok/s.

## Tiers

Pick the tier that fits your machine. Sizes are on-disk and **measured from the built
artifacts**; peak/speed columns are filled in only where a benchmark has actually run.

| tier | target Mac | recipe | on-disk | peak | load | prefill tok/s |
|---|---|---|---|---|---|---|
| [4bit](https://huggingface.co/mlx-community/Inkling-Small-mlx-4bit) | 192 GB | experts 4-bit, non-expert 8-bit | 153.5 GB | — | — | — |
| **3bit** ← **this repo** | 128 GB | experts 3-bit, non-expert 8-bit | 120.9 GB | — | — | — |
| [2bit](https://huggingface.co/mlx-community/Inkling-Small-mlx-2bit) | 96 GB | experts 2-bit, non-expert 8-bit | 88.4 GB | — | — | — |

> **Speeds are not yet measured.** On-disk sizes above are real (taken from the built artifacts), but load/peak/tok-s columns stay empty until `serving/bench_mlx.py` has run on the target hardware — we would rather ship a blank column than a guess. Run it yourself with the command below and please open a discussion with your numbers.

## Why MLX on Apple Silicon

MLX is Apple's array framework for Apple Silicon. For a big sparse MoE like this one, the
practical differences from a CPU/GPU-split runtime are:

| | what it means here |
|---|---|
| **Unified memory** | The GPU reads the same DRAM as the CPU, so a 120 GB build needs 120 GB of *system* memory — not VRAM plus a host copy. There is no PCIe transfer per layer, which is what makes a >100 GB model practical on a desktop at all. |
| **Lazy, mmap'd loading** | `mx.load` memory-maps safetensors, so weights page in on demand instead of being read and copied up front. |
| **Native quantized matmul** | Quantized weights are multiplied in their packed form (`quantized_matmul`, and `gather_qmm` for MoE expert gathers) rather than dequantized to fp16 first — so low-bit tiers save bandwidth at *runtime*, not just on disk. |
| **Per-module precision** | `nn.quantize(..., class_predicate=...)` lets one checkpoint mix bit-widths per tensor class, which is exactly how these tiers keep attention and the router high-precision while the experts go low. |
| **Small dependency surface** | `pip install mlx mlx-lm` and a single Python model file. No compile step, no separate server binary. |

Honest limits: MLX's ecosystem is younger than llama.cpp's, it has no importance-matrix
("i-quant") style calibrated quantization yet, and it runs on Apple Silicon only. If you want a
GGUF build instead, one exists at
[`unsloth/Inkling-Small-GGUF`](https://huggingface.co/unsloth/Inkling-Small-GGUF).

## Memory notes

`peak` (once measured) is peak unified-memory use, not file size — leave headroom for the KV
cache and the OS. Two things matter on a machine near its limit:

* **Raise the Metal wired limit.** It defaults to ~75% of RAM, which is below this tier's
  footprint on a 128 GB machine:
  ```bash
  sudo sysctl iogpu.wired_limit_mb=180000   # ~180 GB on a 192 GB Mac; scale to your RAM
  ```
* **macOS will kill a process that stays too large for too long** (jetsam), even while memory
  pressure looks fine. If long generations die silently, move down a tier.

## Quantization recipe

`experts_only`: the routed experts (and the vision/audio matmuls) carry the tier's bit-width,
while **attention, token/output embeddings, RMSNorms, the router gate, the per-layer
short-convolutions and the relative-position bias stay high-precision (8-bit)**.

That asymmetry is the point, and it is nearly free: non-expert weights are only **5.9 B of the
264 B params (~6 GB at 8-bit)**, yet they are read **in full on every token**, while just 6 of
256 experts are. Bits spent there cost ~2% of the file and protect the paths every token
depends on. Expert precision degrades *gracefully*; attention and router precision degrade
*globally*.

## Usage

```python
# pip install mlx mlx-lm transformers        (+ scipy for audio, pillow for images)
# this repo bundles the inkling_mlx/ loader, so there is nothing else to install
from inkling_mlx.load import load
from inkling_mlx.generate import greedy_generate
from transformers import AutoTokenizer

path = "./Inkling-Small-mlx-3bit"
model, config = load(path)
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
ids = tok("The capital of France is")["input_ids"]
print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))
```

Images are cut into 40 px patches (one soft-token each) and audio is d-mel encoded at 20
tokens/s; both go through `InklingProcessor`. See the source repo for a multimodal runner.

### Benchmark it yourself

```bash
python serving/bench_mlx.py --model ./Inkling-Small-mlx-3bit --tier 3bit
```

Reports load time, prefill tok/s, decode tok/s at several context lengths, and peak memory.

## Attribution

- Base model: [`thinkingmachines/Inkling-Small`](https://huggingface.co/thinkingmachines/Inkling-Small), Apache-2.0.
- The bundled `inkling_mlx/` loader is vendored from
  [PipeNetwork/inkling-mlx](https://github.com/PipeNetwork/inkling-mlx), Apache-2.0.
- Quantized with `mlx`; see the recipe above for exactly which tensors were touched.