README.md

---
license: apache-2.0
library_name: mlx
tags:
  - mlx
  - inkling
  - moe
  - text-generation
base_model: thinkingmachines/Inkling
pipeline_tag: text-generation
---

# Inkling-mlx-2bit (2-bit, text backbone, BF16-sourced)

An **MLX 2-bit** build of the **text backbone** of Thinking Machines' **Inkling**
(975B-total / 41B-active MoE), quantized directly from the BF16 checkpoint. The most
compact build in the ladder - for **multi-Mac distributed** experiments.

This is created for community using a one Apple Mac Studio M3 Ultra with 512 GB.


## Heads up

- **Memory:** ~**329 GB** on disk (routed experts at 2-bit, group size 64; attention /
  shared experts / embeddings / norms kept bf16). Loading needs roughly that much
  **unified memory** -> fits a **2x 192 GB Mac Studio distributed** setup; does **not** fit
  a single Mac.
- **2-bit quality:** experts are quantized hard; this is the lowest-quality rung. For better
  quality see the **3-bit** / **4-bit** siblings.
- **Not verified yet:** custom Inkling forward (factorized attention + short-conv + sigmoid
  MoE) is a from-reference reimplementation; logits not yet checked vs the original.
- **Scope:** text decoder only (no vision/audio).

## Ladder

| variant | bits | ~size | fits |
|---|---|---|---|
| this | 2 | 329 GB | 2 Macs |
| [Inkling-mlx-3bit](https://huggingface.co/huckiyang/Inkling-mlx-3bit) | 3 | ~454 GB | 3 Macs |
| [Inkling-mlx](https://huggingface.co/huckiyang/Inkling-mlx) | 4 (bf16 src) | ~560 GB | 3-4 Macs |
| [Inkling-NVFP4-mlx](https://huggingface.co/huckiyang/Inkling-NVFP4-mlx) | 4 (nvfp4 src) | ~581 GB | 3-4 Macs |

## Usage (once a loader is available)

```python
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Inkling-mlx-2bit")
print(generate(model, tokenizer, prompt="The capital of France is", max_tokens=64))
```