Ling-3.0-Flash-4.6bpw-MLX

A mixed-precision MLX quantization of inclusionAI/Ling-3.0-flash, sized for local inference on Apple Silicon. ~67GB, down from the ~237GB BF16 original, at an effective 4.616 bits/weight average.

Quantization scheme

Not a uniform bit-width — the name reflects the real average, not a round number:

Component	Bits	Notes
Routed MoE experts (switch_mlp)	4-bit	~97% of total parameters
Attention (KDA + MLA), MoE router, shared experts, lm_head, embed_tokens	8-bit	~3% of total parameters

Affine (round-to-nearest) quantization, group_size=64, no calibration data required. Built with a custom streaming pipeline (lazy-loads the BF16 teacher and quantizes shard-by-shard, so the full 237GB model is never resident) — tooling published at True2456/streaming-dwq-mlx.

Why 4-bit routed experts specifically

Tested routed-expert bit widths from 2 to 6-bit against the BF16 teacher, using realistic chunked prefill (not a single-mega-chunk test, which this specific model's hybrid KDA/MLA architecture is unusually sensitive to — see the linked repo's findings doc if you're building your own quantization of this model family). Results, teacher-vs-quantized KL divergence sampled at 64 positions across a 16K-token document:

Routed bits	Size	Max bucket KL	Catastrophic positions
5-bit	81GB	0.013	0
4-bit (this model)	67GB	0.054	0
3-bit	53GB	0.137	0
2-bit	39GB	0.521	2 (real degradation)

4-bit was the chosen tradeoff: meaningfully smaller than a straightforward 5-bit or 8-bit quant, with no measured degradation.

Real benchmark validation

KL divergence against a teacher is a distributional proxy, not a task score — so this was also run through real accuracy benchmarks (mmlu, gsm8k, humaneval) against the original unquantized-routed 5-bit-average build:

Benchmark	This model (4-bit routed)	5-bit routed	Diff
MMLU	83.0% (166/200)	84.0% (168/200)	−1.0pp
GSM8K	95.5% (191/200)	93.5% (187/200)	+2.0pp
HumanEval	89.0% (146/164)	87.8% (144/164)	+1.2pp

At these sample sizes, a 2-4 question swing is within normal binomial noise — statistically indistinguishable from parity.

Important: two upstream mlx-lm bugs this model needs patched

mlx-lm's bailing_hybrid implementation (the architecture family Ling-3.0-flash belongs to) is missing two pieces of Ling-specific logic that its own config.json specifies. Neither is optional — running without them costs real accuracy, silently, with no error or crash to signal it:

1. SwiGLU clamp (expert_swiglu_limit_list / share_expert_swiglu_limit_list, layers 34-41). Measured cost of running unclamped: 88.41% → 71.34% HumanEval (17 points).
2. KDA safe-gate clamp (kda_safe_gate: true, kda_lower_bound: -5.0). mlx-lm never reads these config fields; the HF reference implementation does. Impact not fully characterized, but it's a real, config-specified fix, same category as (1).

Both patches: bailing_swiglu_clamp.py, kda_safe_gate_patch.py. If you're loading this model through a runtime that doesn't already apply equivalent fixes, apply these first or expect degraded output.

Usage

Requires an mlx-lm build with bailing_hybrid support (not yet upstream in ml-explore/mlx-lm as of this writing — see the linked repo for details) and the two patches above applied.

from mlx_lm import load, generate

model, tokenizer = load("True2456/Ling-3.0-Flash-4.6bpw-MLX")
# apply the SwiGLU clamp + KDA safe-gate patches before/at load time -- see
# https://github.com/True2456/streaming-dwq-mlx for the patch modules

response = generate(model, tokenizer, prompt="Hello!", max_tokens=256)
print(response)

License

MIT, inherited from the base model.