--- license: apache-2.0 base_model: thinkingmachines/Inkling-2-Small-MXFP8 base_model_relation: quantized library_name: mlx pipeline_tag: image-text-to-text tags: - mlx - omlx - inkling2 - oq2e - quantized - image-text-to-text --- # Inkling-2-Small oQ2e **oQ2e mixed-precision MLX conversion of [Thinking Machines’ Inkling-2-Small-MXFP8](https://huggingface.co/thinkingmachines/Inkling-2-Small-MXFP8) for Apple Silicon.** **Effective weight storage: 2.61 BPW · 194.72 GB · group size 64.** The full expert set, vision encoder/projector, original tokenizer and chat template are retained, along with the original BF16 multi-token-prediction (MTP) sidecar. All 48 text layers retain 384 routed experts and top-8 routing. Other precision: [mixed 2/3-bit oQe (3.07 BPW)](https://huggingface.co/Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX), **preferred for 256 GB Apple Silicon hardware**. ## Conversion - **Source:** published MXFP8 checkpoint, `thinkingmachines/Inkling-2-Small-MXFP8@086a047e8b7ff2ac902dd52355328ddb66b16914`. - **Format:** MLX safetensors; oQ2e mixed precision, affine packing, group size 64. - **Tooling:** oMLX native Inkling 2/oQe converter, [PR #5229](https://github.com/jundot/omlx/pull/5229). - **Calibration:** 384 samples of 512 tokens with text/image inputs; full expert coverage. - **Weights:** 33 main shards plus the MTP sidecar, **194.72 GB / 181.34 GiB**. Conversion records are in [`CONVERSION.json`](./CONVERSION.json). ## Mixed precision The packed quantized matrices average **2.066 bits per weight**. The **2.61 effective BPW** includes quantization metadata, unquantized tensors and safetensors headers. | Component | Weight precision | Original scalar parameters | Stored GB | |---|---:|---:|---:| | Attention-residual controls | FP32 | 516,096 | 0.002064 | | MTP / draft | BF16 | 1,711,710,384 | 3.423421 | | Output head | 8-bit | 926,318,592 | 0.984214 | | Text attention | 8-bit, BF16 | 4,713,811,968 | 4.862202 | | Text convolutions | BF16 | 2,162,688 | 0.004325 | | Text normalization | BF16 | 525,312 | 0.001051 | | Text routed experts | 2-bit | 587,068,342,272 | 183.458857 | | Text routers | FP32 | 84,953,088 | 0.339812 | | Token embeddings | 8-bit | 926,318,592 | 0.984214 | | Vision encoder | BF16 | 292,041,728 | 0.584083 | | Vision merger / projector | BF16 | 35,651,584 | 0.071303 | | Safetensors headers | — | — | 0.000201 | | **Total** | **2.61 effective BPW** | **595,762,352,304** | **194.715747** | Stored GB uses decimal units. MTP is included; no tensors are counted twice. **Calculation:** `8 × 194,715,747,050 stored bytes ÷ 595,762,352,304 original scalar parameters = 2.614677 BPW`. Tokenizer/config files, KV cache and runtime workspace are excluded.
Per-block precision map | Block | Routed expert matrices | Other quantized matrices | Unquantized tensors | Stored GB | |---|---:|---:|---|---:| | Text global | — | 8-bit | BF16, FP32 | 1.968464 | | Text 0 | 2-bit | 8-bit | BF16, FP32 | 3.930511 | | Text 1 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 2 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 3 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 4 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 5 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | Text 6 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 7 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 8 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 9 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 10 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 11 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | Text 12 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 13 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 14 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 15 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 16 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 17 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | Text 18 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 19 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 20 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 21 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 22 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 23 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | Text 24 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 25 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 26 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 27 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 28 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 29 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | Text 30 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 31 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 32 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 33 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 34 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 35 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | Text 36 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 37 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 38 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 39 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 40 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 41 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | Text 42 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 43 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 44 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 45 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 46 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | Text 47 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | Vision global | — | — | BF16 | 0.075866 | | Vision 0 | — | — | BF16 | 0.025197 | | Vision 1 | — | — | BF16 | 0.025197 | | Vision 2 | — | — | BF16 | 0.025197 | | Vision 3 | — | — | BF16 | 0.025197 | | Vision 4 | — | — | BF16 | 0.025197 | | Vision 5 | — | — | BF16 | 0.025197 | | Vision 6 | — | — | BF16 | 0.025197 | | Vision 7 | — | — | BF16 | 0.025197 | | Vision 8 | — | — | BF16 | 0.025197 | | Vision 9 | — | — | BF16 | 0.025197 | | Vision 10 | — | — | BF16 | 0.025197 | | Vision 11 | — | — | BF16 | 0.025197 | | Vision 12 | — | — | BF16 | 0.025197 | | Vision 13 | — | — | BF16 | 0.025197 | | Vision 14 | — | — | BF16 | 0.025197 | | Vision 15 | — | — | BF16 | 0.025197 | | Vision 16 | — | — | BF16 | 0.025197 | | Vision 17 | — | — | BF16 | 0.025197 | | Vision 18 | — | — | BF16 | 0.025197 | | Vision 19 | — | — | BF16 | 0.025197 | | Vision 20 | — | — | BF16 | 0.025197 | | Vision 21 | — | — | BF16 | 0.025197 | | Vision 22 | — | — | BF16 | 0.025197 | | MTP global | — | — | BF16 | 0.120264 | | MTP 0 | — | — | BF16 | 0.660631 | | MTP 1 | — | — | BF16 | 0.660631 | | MTP 2 | — | — | BF16 | 0.660631 | | MTP 3 | — | — | BF16 | 0.660631 | | MTP 4 | — | — | BF16 | 0.660631 |
Complete per-module/tensor accounting: [CSV](./WEIGHT_PRECISION.csv) and [JSON](./WEIGHT_PRECISION.json). ## Running Requires oMLX’s native Inkling 2 implementation ([PR #5229](https://github.com/jundot/omlx/pull/5229), tested at `2a18263b91e14ee92ed984f8c5a56d0053178cea`). Native loader: `omlx.patches.inkling2.vlm.load`. Keep the included processor, tokenizer, chat template and complete `tiktoken/` directory; generic `mlx-lm` loading is not supported. **Cold loading, text and image generation tested on a 256 GB M5 Ultra.** Main-model MLX allocator peak: **179.32 GiB**. Runtime and context state need additional memory. MTP decoding, broader quality and full-context capacity remain untested. ## Credits and license Model: **Thinking Machines Lab**. Conversion tooling: **oMLX / oQe**. Community conversion: **Hygaard**. Licensed under [Apache-2.0](./LICENSE). See [`NOTICE`](./NOTICE) for attribution and modification notes.