README.md

---
license: apache-2.0
base_model: thinkingmachines/Inkling
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: mlx
tags:
- mlx
- moe
- multimodal
- inkling
- thinking-machines
---

# Inkling-MLX-4bit

**Built with Inkling (Thinking Machines Lab).**

MLX (Apple Silicon) conversion of
[thinkingmachines/Inkling](https://huggingface.co/thinkingmachines/Inkling),
quantized to **4-bit** (affine group quant, group size 64).

**Code / loader:** [github.com/PipeNetwork/inkling-mlx](https://github.com/PipeNetwork/inkling-mlx)

Inkling is a **975B-total / 41B-active** sparse-MoE, natively multimodal model
(text + image/video + audio → text). This is the **full multimodal** conversion:
all three towers (text backbone, HMLP vision, dMel audio) are ported; the
multi-token-prediction head is dropped (inference-irrelevant).

## Quantizations

| Variant | Size | Notes |
|---|---|---|
| [8bit](https://huggingface.co/pipenetwork/Inkling-MLX-8bit) | ~937 GB | near-lossless |
| [6bit](https://huggingface.co/pipenetwork/Inkling-MLX-6bit) | ~717 GB | high quality |
| [4bit](https://huggingface.co/pipenetwork/Inkling-MLX-4bit) | ~490 GB | balanced default |

## Quantization scheme: affine int4 (not NVFP4 / MXFP4)

MLX supports FP4 modes and Thinking Machines ships an
[Inkling-NVFP4](https://huggingface.co/thinkingmachines/Inkling-NVFP4) checkpoint — so for
the record, we benchmarked round-trip reconstruction error (‖W − Ŵ‖ / ‖W‖ vs bf16) on real
Inkling expert weights:

| Scheme | bits/weight | reconstruction error |
|---|---:|---:|
| **affine int4** (group 64) | 4.50 | **~9.1%** |
| nvfp4 (group 16) | 4.50 | ~10.2% |
| mxfp4 (group 32) | 4.25 | ~12.3% |

Affine int4 is the most faithful: it is *asymmetric* (per-group scale **and** zero-point, 16
uniform levels), which centers on Inkling's near-Gaussian expert weights better than
symmetric FP4's fixed non-uniform levels. FP4's real payoff is heavy-tailed *activations* and
native Blackwell FP4 tensor cores — neither helps weight fidelity on Apple Silicon, where MLX
would dequantize FP4 anyway. So these builds use affine int4.

## ⚠️ Loading requires the bundled `inkling_mlx` loader

The `inkling_mm_model` architecture is **not** in stock `mlx-lm` / `mlx-vlm`, so this
repo bundles a minimal, numerically-validated MLX implementation under `inkling_mlx/`.

```bash
pip install mlx mlx-lm transformers
```
```python
from inkling_mlx.load import load
from inkling_mlx.generate import greedy_generate
from transformers import AutoTokenizer

model, config = load("/path/to/this/repo")
tok = AutoTokenizer.from_pretrained("/path/to/this/repo", trust_remote_code=True)
ids = tok("The capital of France is")["input_ids"]
print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))
```

Needs an Apple-Silicon Mac with enough unified memory to hold the weights (≈ the
size above).

## Status & caveats

- **Text generation** works end-to-end via an incremental KV + short-convolution cache.
- **Multimodal** is supported end-to-end: the vision/audio towers and their
  preprocessing (`InklingProcessor` — image patchify/normalize, audio log-mel→dMel,
  validated ~1e-7 vs the reference) are included. Pass images/audio via the processor.
- Quantized: attention / MLP / expert projections, token embed+unembed, and the
  vision/audio matmuls. Kept in higher precision: the MoE router, RMSNorms, the four
  short-convolutions per layer, and the relative-position bias.

Conversion is streaming (tensor-by-tensor; the ~1.9 TB bf16 model never fully loads
into RAM) and was validated with fp32 numerical parity against transformers PR #47347.
License: Apache-2.0 (inherits the base model).