File size: 2,182 Bytes
bbb09f8 7d963e1 bbb09f8 7d963e1 c96bb63 4976c2f 7d963e1 4976c2f 7d963e1 4976c2f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | ---
license: apache-2.0
library_name: mlx
tags:
- mlx
- inkling
- moe
- text-generation
base_model: thinkingmachines/Inkling
pipeline_tag: text-generation
---
# Inkling-mlx (4-bit, text backbone, BF16-sourced)
An **MLX 4-bit** build of the **text backbone** of Thinking Machines' **Inkling**
(975B-total / 41B-active MoE), quantized **directly from the BF16 checkpoint** (no
NVFP4->INT4 double-quantization), for running natively on Apple Silicon with
[`mlx-lm`](https://github.com/ml-explore/mlx-lm).
> **Community note.** Early, **not fully numerically-verified**
> conversion, shared to see whether anyone can load/run a model this large on Apple
> Silicon. Expect rough edges; please open a discussion with results (or failures).
## Heads up
- **Memory:** ~**560 GB** on disk (4-bit routed experts + bf16 attention / shared
experts / embeddings). Loading needs roughly that much **unified memory** - beyond any
single Mac today (max 512 GB), so realistically distributed/multi-device MLX. Largely a
**research artifact**.
- **Not verified yet:** the custom Inkling forward (factorized attention + short-conv +
sigmoid MoE) is a from-reference reimplementation; logits have **not** been checked
against the original.
- **Scope:** **text decoder only.** Vision (image/video) and audio encoders are not included.
## Provenance
- **Source:** `thinkingmachines/Inkling` (BF16) -> MLX affine **4-bit** (group size 64).
Only routed MoE experts are quantized; everything else is bf16. Quantizing straight from
BF16 avoids the NVFP4->INT4 step, so this should be slightly higher quality than the
NVFP4-sourced sibling ([`huckiyang/Inkling-NVFP4-mlx`](https://huggingface.co/huckiyang/Inkling-NVFP4-mlx)).
## Usage (once a loader is available)
```python
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Inkling-mlx-4bit")
print(generate(model, tokenizer, prompt="The capital of France is", max_tokens=64))
```
> The custom model class lives in the conversion repo (`models/inkling_mlx.py`); until it's
> registered in `mlx-lm`, load via that module's `load()`.
>
> Blog: https://huckiyang.github.io/blog/inkling-audio-design.html |