Inkling-2-Mini-MLX-Serve-4bit

4-bit MLX-Serve affine quantization (bits=4, group_size=64) of thinkingmachines/Inkling-2-Mini (model_type: inkling2_loop, 26.4B total / 3.4B–5.4B active looped MoE).

Conversion details

  • Routed experts (w13_weight, w2_weight): dequantized from source modelopt NVFP4 (group_size=16, global_scale = amax * 6 * 448 / outer_scale_denominator), split into switch_mlp.gate_proj, switch_mlp.up_proj, and switch_mlp.down_proj, and quantized to 4-bit MLX affine (group_size=64).
  • Attention & embed/unembed (wq_du, wk_dv, wv_dv, wr_du, wo_ud, gate_attn, embed, unembed): quantized to 4-bit MLX affine (group_size=64).
  • Routers, controllers, attention residuals, short-conv weights, norms, attn_sink_proj, rel_logits_proj: kept in full/bfloat16 precision (sconv transposed [C, 1, K] -> [C, K, 1]).
Downloads last month
-
Safetensors
Model size
26B params
Tensor type
F32
·
U32
·
BF16
·
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for pgmharikrishnan/Inkling-2-Mini-MLX-Serve-4bit

Quantized
(22)
this model