Hygaard's picture
Link preferred mixed 2/3-bit quant for 256 GB hardware
d51692b verified
|
Raw History Blame Contribute Delete
7.7 kB
---
license: apache-2.0
base_model: thinkingmachines/Inkling-2-Small-MXFP8
base_model_relation: quantized
library_name: mlx
pipeline_tag: image-text-to-text
tags:
- mlx
- omlx
- inkling2
- oq2e
- quantized
- image-text-to-text
---
# Inkling-2-Small oQ2e
**oQ2e mixed-precision MLX conversion of [Thinking Machines’ Inkling-2-Small-MXFP8](https://huggingface.co/thinkingmachines/Inkling-2-Small-MXFP8) for Apple Silicon.**
**Effective weight storage: 2.61 BPW Β· 194.72 GB Β· group size 64.**
The full expert set, vision encoder/projector, original tokenizer and chat template are retained, along with the original BF16 multi-token-prediction (MTP) sidecar. All 48 text layers retain 384 routed experts and top-8 routing.
Other precision: [mixed 2/3-bit oQe (3.07 BPW)](https://huggingface.co/Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX), **preferred for 256 GB Apple Silicon hardware**.
## Conversion
- **Source:** published MXFP8 checkpoint, `thinkingmachines/Inkling-2-Small-MXFP8@086a047e8b7ff2ac902dd52355328ddb66b16914`.
- **Format:** MLX safetensors; oQ2e mixed precision, affine packing, group size 64.
- **Tooling:** oMLX native Inkling 2/oQe converter, [PR #5229](https://github.com/jundot/omlx/pull/5229).
- **Calibration:** 384 samples of 512 tokens with text/image inputs; full expert coverage.
- **Weights:** 33 main shards plus the MTP sidecar, **194.72 GB / 181.34 GiB**.
Conversion records are in [`CONVERSION.json`](./CONVERSION.json).
## Mixed precision
The packed quantized matrices average **2.066 bits per weight**. The **2.61 effective BPW** includes quantization metadata, unquantized tensors and safetensors headers.
| Component | Weight precision | Original scalar parameters | Stored GB |
|---|---:|---:|---:|
| Attention-residual controls | FP32 | 516,096 | 0.002064 |
| MTP / draft | BF16 | 1,711,710,384 | 3.423421 |
| Output head | 8-bit | 926,318,592 | 0.984214 |
| Text attention | 8-bit, BF16 | 4,713,811,968 | 4.862202 |
| Text convolutions | BF16 | 2,162,688 | 0.004325 |
| Text normalization | BF16 | 525,312 | 0.001051 |
| Text routed experts | 2-bit | 587,068,342,272 | 183.458857 |
| Text routers | FP32 | 84,953,088 | 0.339812 |
| Token embeddings | 8-bit | 926,318,592 | 0.984214 |
| Vision encoder | BF16 | 292,041,728 | 0.584083 |
| Vision merger / projector | BF16 | 35,651,584 | 0.071303 |
| Safetensors headers | β€” | β€” | 0.000201 |
| **Total** | **2.61 effective BPW** | **595,762,352,304** | **194.715747** |
Stored GB uses decimal units. MTP is included; no tensors are counted twice.
**Calculation:** `8 Γ— 194,715,747,050 stored bytes Γ· 595,762,352,304 original scalar parameters = 2.614677 BPW`. Tokenizer/config files, KV cache and runtime workspace are excluded.
<details>
<summary>Per-block precision map</summary>
| Block | Routed expert matrices | Other quantized matrices | Unquantized tensors | Stored GB |
|---|---:|---:|---|---:|
| Text global | β€” | 8-bit | BF16, FP32 | 1.968464 |
| Text 0 | 2-bit | 8-bit | BF16, FP32 | 3.930511 |
| Text 1 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 2 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 3 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 4 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 5 | 2-bit | 8-bit | BF16, FP32 | 3.930889 |
| Text 6 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 7 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 8 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 9 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 10 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 11 | 2-bit | 8-bit | BF16, FP32 | 3.930889 |
| Text 12 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 13 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 14 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 15 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 16 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 17 | 2-bit | 8-bit | BF16, FP32 | 3.930889 |
| Text 18 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 19 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 20 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 21 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 22 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 23 | 2-bit | 8-bit | BF16, FP32 | 3.930889 |
| Text 24 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 25 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 26 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 27 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 28 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 29 | 2-bit | 8-bit | BF16, FP32 | 3.930889 |
| Text 30 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 31 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 32 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 33 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 34 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 35 | 2-bit | 8-bit | BF16, FP32 | 3.930889 |
| Text 36 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 37 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 38 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 39 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 40 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 41 | 2-bit | 8-bit | BF16, FP32 | 3.930889 |
| Text 42 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 43 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 44 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 45 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 46 | 2-bit | 8-bit | BF16, FP32 | 3.930530 |
| Text 47 | 2-bit | 8-bit | BF16, FP32 | 3.930889 |
| Vision global | β€” | β€” | BF16 | 0.075866 |
| Vision 0 | β€” | β€” | BF16 | 0.025197 |
| Vision 1 | β€” | β€” | BF16 | 0.025197 |
| Vision 2 | β€” | β€” | BF16 | 0.025197 |
| Vision 3 | β€” | β€” | BF16 | 0.025197 |
| Vision 4 | β€” | β€” | BF16 | 0.025197 |
| Vision 5 | β€” | β€” | BF16 | 0.025197 |
| Vision 6 | β€” | β€” | BF16 | 0.025197 |
| Vision 7 | β€” | β€” | BF16 | 0.025197 |
| Vision 8 | β€” | β€” | BF16 | 0.025197 |
| Vision 9 | β€” | β€” | BF16 | 0.025197 |
| Vision 10 | β€” | β€” | BF16 | 0.025197 |
| Vision 11 | β€” | β€” | BF16 | 0.025197 |
| Vision 12 | β€” | β€” | BF16 | 0.025197 |
| Vision 13 | β€” | β€” | BF16 | 0.025197 |
| Vision 14 | β€” | β€” | BF16 | 0.025197 |
| Vision 15 | β€” | β€” | BF16 | 0.025197 |
| Vision 16 | β€” | β€” | BF16 | 0.025197 |
| Vision 17 | β€” | β€” | BF16 | 0.025197 |
| Vision 18 | β€” | β€” | BF16 | 0.025197 |
| Vision 19 | β€” | β€” | BF16 | 0.025197 |
| Vision 20 | β€” | β€” | BF16 | 0.025197 |
| Vision 21 | β€” | β€” | BF16 | 0.025197 |
| Vision 22 | β€” | β€” | BF16 | 0.025197 |
| MTP global | β€” | β€” | BF16 | 0.120264 |
| MTP 0 | β€” | β€” | BF16 | 0.660631 |
| MTP 1 | β€” | β€” | BF16 | 0.660631 |
| MTP 2 | β€” | β€” | BF16 | 0.660631 |
| MTP 3 | β€” | β€” | BF16 | 0.660631 |
| MTP 4 | β€” | β€” | BF16 | 0.660631 |
</details>
Complete per-module/tensor accounting: [CSV](./WEIGHT_PRECISION.csv) and [JSON](./WEIGHT_PRECISION.json).
## Running
Requires oMLX’s native Inkling 2 implementation ([PR #5229](https://github.com/jundot/omlx/pull/5229), tested at `2a18263b91e14ee92ed984f8c5a56d0053178cea`). Native loader: `omlx.patches.inkling2.vlm.load`. Keep the included processor, tokenizer, chat template and complete `tiktoken/` directory; generic `mlx-lm` loading is not supported.
**Cold loading, text and image generation tested on a 256 GB M5 Ultra.** Main-model MLX allocator peak: **179.32 GiB**. Runtime and context state need additional memory. MTP decoding, broader quality and full-context capacity remain untested.
## Credits and license
Model: **Thinking Machines Lab**. Conversion tooling: **oMLX / oQe**. Community conversion: **Hygaard**.
Licensed under [Apache-2.0](./LICENSE). See [`NOTICE`](./NOTICE) for attribution and modification notes.