mlx-community/Inkling-mlx-4bit

Inkling-mlx (4-bit, text backbone, BF16-sourced)

An MLX 4-bit build of the text backbone of Thinking Machines' Inkling (975B-total / 41B-active MoE), quantized directly from the BF16 checkpoint (no NVFP4->INT4 double-quantization), for running natively on Apple Silicon with mlx-lm.

Community note. Early, not fully numerically-verified conversion, shared to see whether anyone can load/run a model this large on Apple Silicon. Expect rough edges; please open a discussion with results (or failures).

Heads up

• Memory: ~560 GB on disk (4-bit routed experts + bf16 attention / shared experts / embeddings). Loading needs roughly that much unified memory - beyond any single Mac today (max 512 GB), so realistically distributed/multi-device MLX. Largely a research artifact.
• Not verified yet: the custom Inkling forward (factorized attention + short-conv + sigmoid MoE) is a from-reference reimplementation; logits have not been checked against the original.
• Scope: text decoder only. Vision (image/video) and audio encoders are not included.

Provenance

• Source: thinkingmachines/Inkling (BF16) -> MLX affine 4-bit (group size 64). Only routed MoE experts are quantized; everything else is bf16. Quantizing straight from BF16 avoids the NVFP4->INT4 step, so this should be slightly higher quality than the NVFP4-sourced sibling (huckiyang/Inkling-NVFP4-mlx).

Usage (once a loader is available)

from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Inkling-mlx-4bit")
print(generate(model, tokenizer, prompt="The capital of France is", max_tokens=64))

The custom model class lives in the conversion repo (models/inkling_mlx.py); until it's registered in mlx-lm, load via that module's load().

Blog: https://huckiyang.github.io/blog/inkling-audio-design.html