Inkling-MLX-8bit

Built with Inkling (Thinking Machines Lab).

MLX (Apple Silicon) conversion of thinkingmachines/Inkling, quantized to 8-bit (affine group quant, group size 64).

Code / loader: github.com/PipeNetwork/inkling-mlx

Inkling is a 975B-total / 41B-active sparse-MoE, natively multimodal model (text + image/video + audio → text). This is the full multimodal conversion: all three towers (text backbone, HMLP vision, dMel audio) are ported; the multi-token-prediction head is dropped (inference-irrelevant).

Quantizations

Variant	Size	Notes
8bit	~937 GB	near-lossless
6bit	~717 GB	high quality
4bit	~490 GB	balanced default

⚠️ Loading requires the bundled inkling_mlx loader

The inkling_mm_model architecture is not in stock mlx-lm / mlx-vlm, so this repo bundles a minimal, numerically-validated MLX implementation under inkling_mlx/.

pip install mlx mlx-lm transformers

from inkling_mlx.load import load
from inkling_mlx.generate import greedy_generate
from transformers import AutoTokenizer

model, config = load("/path/to/this/repo")
tok = AutoTokenizer.from_pretrained("/path/to/this/repo", trust_remote_code=True)
ids = tok("The capital of France is")["input_ids"]
print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))

Needs an Apple-Silicon Mac with enough unified memory to hold the weights (≈ the size above).

Status & caveats

• Text generation works end-to-end via an incremental KV + short-convolution cache.
• Multimodal is supported end-to-end: the vision/audio towers and their preprocessing (InklingProcessor — image patchify/normalize, audio log-mel→dMel, validated ~1e-7 vs the reference) are included. Pass images/audio via the processor.
• Quantized: attention / MLP / expert projections, token embed+unembed, and the vision/audio matmuls. Kept in higher precision: the MoE router, RMSNorms, the four short-convolutions per layer, and the relative-position bias.

Conversion is streaming (tensor-by-tensor; the ~1.9 TB bf16 model never fully loads into RAM) and was validated with fp32 numerical parity against transformers PR #47347. License: Apache-2.0 (inherits the base model).