Inkling 2 Mini — MLX mixed 2bit

Experimental low-bit conversion. Expect greater quality loss than the 4-bit and 5-bit versions.

Converted from the official Thinking Machines checkpoint, revision 6e7c8248f75265bdd4fbab5273304243b5d8c2cb. The source is the authors' trained NVFP4/MXFP8 release, not an original BF16 checkpoint. Conversion to affine MLX adds reconstruction error; 5-bit conversion does not recover higher-precision training weights. This is an independent conversion, not an official Thinking Machines release.

Quantization

  • 2-bit expert gate/up and 3-bit expert down matrices in layers 0–19; 4-bit expert matrices in the original MXFP8 tail layers 20–23; 4-bit attention, group size 32.
  • Token embeddings and output head: 8-bit.
  • Vision encoder, expert routing and loop-control gates: retained at their source floating-point precision. Vision is included.
  • All 24 stored layers and 48 layer applications per token are retained. The twelve middle layers still run three times.
  • Weight-only per-group clipping search: factors 1, .995, .985, .97, .94, .9, .85, .8, selected by reconstruction MSE. This is not activation-calibrated GPTQ or AWQ.
  • Sum of per-group MSE decreased 11.74% versus min/max round-to-nearest for the matrices included in the clipping search. This is a weight statistic, not a benchmark score.
  • Target plus vision weights: 13.24 GB / 12.33 GiB. Scale/bias metadata and protected tensors mean the actual average exceeds the nominal bit label.

The architecture has about 26.4B logical parameters, per the upstream model card. A Hub counter that counts packed integer storage may show a different number. The original auxiliary mtp.safetensors is not included or qualified by this conversion; no speculative-decoding speedup is claimed.

Required runtime

This model needs the pinned Inkling MLX-LM port. Stock released MLX-LM did not support this architecture when this artifact was prepared.

python3 -m venv inkling-env
source inkling-env/bin/activate
pip install 'mlx-lm @ git+https://github.com/ml-explore/mlx-lm.git@0c4c99192230f1943dd468d297f0b8a64ae7b7c2'
MLX_ENABLE_TF32=0 mlx_lm generate --model araysuter/Inkling-2-Mini-MLX-mixed-2bit --prompt 'Explain a binary search in two sentences.' -m 256

Validated with MLX 0.32.4, Python 3.14, and an Apple M5 Max with 64 GiB unified memory. The port's eight model tests pass with MLX_ENABLE_TF32=0. The default M5 reduced-precision float32 path fails three of those tests; MLX documents the precision switch. The recommended flag retains full float32 execution for those operations. This does not require converting all weights to float32.

For images, the pinned port includes the Inkling image processor and embed_inputs; the included validate_inkling.py shows the tested path with image embeddings. The full upstream context setting is retained in config; 128K and 1M contexts have not been qualified here.

Small validation sample

All artifacts were reloaded from disk before these probes. Same tokenizer and text were used for the source and conversion; the native trained source ran with the same MLX port and precision setting.

Check Observed converted output
arithmetic 391
coding python def larger(a, b): return max(a, b)
vision There is a red square on the left and a blue circle on the right.
Synthetic text probe Native source This conversion
Mean negative log likelihood 2.26608 2.15405
Perplexity 9.64151 8.61969

Delta NLL: -0.11203. This is one short, synthetic text passage plus three smoke prompts, not a standard benchmark or evidence of general benchmark accuracy. Full prompt text, responses, timing and checks are in VALIDATION.json. Reasoning tokens count toward the recorded generation lengths and speeds.

Reproduce

convert_inkling.py converts one profile at a time with bounded tensor tiles; it does not materialize a full BF16 checkpoint. Put the pinned original checkpoint in source/, then run MLX_ENABLE_TF32=0 python convert_inkling.py mixed-2bit. Use INKLING_WORKDIR to select the working directory. The included scripts use the runtime pinned above. QUANTIZATION_MANIFEST.json records every quantized matrix and its selected precision.

Attribution

Original model: Thinking Machines Lab, Inc. Original license: Apache 2.0; see LICENSE, the upstream model card, and its linked Model Acceptable Use Policy. MLX runtime: the MLX contributors. Published quantized weights retain the upstream model license.

Downloads last month
-
Safetensors
Model size
26B params
Tensor type
U32
·
F32
·
BF16
·
MLX
Hardware compatibility
Log In to add your hardware

2-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for araysuter/Inkling-2-Mini-MLX-mixed-2bit

Quantized
(22)
this model

Collection including araysuter/Inkling-2-Mini-MLX-mixed-2bit