Inkling-Small MLX mixed 2-bit
Mixed-precision MLX conversion of
thinkingmachines/Inkling-Small at revision
b2d4f225a02032c5d154bff748ab5a00c5ca26e4. The conversion uses MLX 0.32.0 and mlx-vlm
0.6.7. See conversion.json for the machine-readable provenance
and artifact inventory.
Only the 256 routed experts in each sparse MoE layer use affine 2-bit, group-128 weights. Attention, embeddings, routers, shared experts, dense MLPs, normalization, vision, and audio weights remain at the released BF16 precision. This conservative split targets 128 GB unified-memory Macs while protecting the non-routed paths that blanket 2-bit quantization damaged.
Validation status
The conversion gate checks the affine 2-bit/group-128 routed-expert metadata, all 120 quantized expert projections, every shard, and representative BF16 and packed tensor shapes.
On 2026-07-31, a full CUDA load and greedy generation smoke with MLX
0.32.0 and mlx-vlm 0.6.7 used 84,530,593,368 bytes of
active MLX memory and peaked at 84,766,898,240 bytes on an NVIDIA H200. The
model correctly answered 2 + 2 with 4, the "all but 9" sheep question with
9, and produced a correct Swift isEven function. This proves CUDA loading
and token quality for the generated artifact; Apple Silicon runtime validation
is tracked separately by mere.run.
mlx-vlm 0.6.7 also required the compatibility shims in the
conversion script for its Inkling config exports and CUDA mask fallback; the
managed mere.run lane uses its own native Swift/MLX loader.
mere.run text chat --model text-chat-inkling-small --prompt "Who are you?"
- Downloads last month
- 20
2-bit
Model tree for Sawfwair/Inkling-Small-MLX-Mixed-2bit
Base model
thinkingmachines/Inkling-Small