---
license: apache-2.0
library_name: mlx
tags:
  - mlx
  - inkling
  - moe
  - text-generation
base_model: thinkingmachines/Inkling
pipeline_tag: text-generation
---

# Inkling-mlx (4-bit, text backbone, BF16-sourced)

An **MLX 4-bit** build of the **text backbone** of Thinking Machines' **Inkling**
(975B-total / 41B-active MoE), quantized **directly from the BF16 checkpoint** (no
NVFP4->INT4 double-quantization), for running natively on Apple Silicon with
[`mlx-lm`](https://github.com/ml-explore/mlx-lm).

> **Community note.** Early, **not fully numerically-verified**
> conversion, shared to see whether anyone can load/run a model this large on Apple
> Silicon. Expect rough edges; please open a discussion with results (or failures).

## Heads up

- **Memory:** ~**560 GB** on disk (4-bit routed experts + bf16 attention / shared
  experts / embeddings). Loading needs roughly that much **unified memory** - beyond any
  single Mac today (max 512 GB), so realistically distributed/multi-device MLX. Largely a
  **research artifact**.
- **Not verified yet:** the custom Inkling forward (factorized attention + short-conv +
  sigmoid MoE) is a from-reference reimplementation; logits have **not** been checked
  against the original.
- **Scope:** **text decoder only.** Vision (image/video) and audio encoders are not included.

## Provenance

- **Source:** `thinkingmachines/Inkling` (BF16) -> MLX affine **4-bit** (group size 64).
  Only routed MoE experts are quantized; everything else is bf16. Quantizing straight from
  BF16 avoids the NVFP4->INT4 step, so this should be slightly higher quality than the
  NVFP4-sourced sibling ([`huckiyang/Inkling-NVFP4-mlx`](https://huggingface.co/huckiyang/Inkling-NVFP4-mlx)).

## Usage (once a loader is available)

```python
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Inkling-mlx-4bit")
print(generate(model, tokenizer, prompt="The capital of France is", max_tokens=64))
```

> The custom model class lives in the conversion repo (`models/inkling_mlx.py`); until it's
> registered in `mlx-lm`, load via that module's `load()`.
>
> Blog: https://huckiyang.github.io/blog/inkling-audio-design.html