Ling-3.0-flash-GGUF — sm_120-safe quants

Quants of inclusionAI/Ling-3.0-flash, a 127.5B-total / 5.1B-active hybrid MoE (bailing_hybrid / BailingMoeV3).

Every file here contains only sm_120-safe tensor types — no iq1_s, iq2_s or iq3_s, which are broken on NVIDIA Blackwell (RTX PRO 6000, RTX 5090, RTX 5070 Ti). All were quantized from the BF16 master, never from another quant, and declare a verified 256K context window.

⚠️ Requires a patched llama.cpp

Ling 3.0 is not yet supported upstream. This repo ships bailing-hybrid-llama.cpp.patch against ggml-org/llama.cpp commit 6ea215d17 — see Building llama.cpp.

These files declare general.architecture = bailing-hybrid, which is our name, not an official one. Upstream PR #26608 proposes bailingmoe3, and we expect that to be what lands. When it does, these files will not load on a stock build — this port also introduces tensor names (ssm_f/ssm_g, a g_full_proj rename) and KV keys (kda.lower_bound) that upstream may define differently. We will republish against the upstream conventions once they are final. Until then, use the patch.

Which one do I want?

	Q4_K_M	Q5_K_M	Q5_K_XL	Q5_K_XXL	Q8_0
Weights	69.70 GiB	84.18 GiB	88.92 GiB	92.78 GiB	126.32 GiB
+ KV at full 256K	+1.97 GiB	+1.97 GiB	+1.97 GiB	+1.97 GiB	+1.97 GiB
Mean KLD vs BF16	0.02154	0.01159	0.01044	0.00919	0.00668
99th pct KLD	0.17519	0.07896	0.06725	0.05934	0.03848
Top-1 agreement	94.92 %	96.59 %	97.02 %	97.22 %	97.84 %
PPL ratio vs BF16	1.0111	1.0045	1.0036	1.0018	0.9986

• Q5_K_XXL — the most accurate build that fits a 96 GB card at the full 256K window. Recommended for a dedicated RTX Pro 6000; it leaves ~1.4 GiB spare.
• Q5_K_M — the balanced pick, and the right one if anything else shares the GPU (~10.5 GiB spare).
• Q5_K_XL — in between, ~5.9 GiB spare.
• Q4_K_M — smallest. The Q4→Q5 step is about twice the Q5→Q8 step on both mean KLD and top-1, so if you are deciding where to spend memory, that is the one that matters.
• Q8_0 — reference fidelity. Needs expert offload or two GPUs.

KV cache is unusually cheap here — only 7 of 42 layers are MLA, and those are compressed — so the full 256K window costs under 2 GiB. Weights are what limit you on this model, not context.

Running it

Point llama.cpp at shard 00001; it finds the rest. Omitting -c gives the full 262,144-token window.

./build/bin/llama-server -m Ling-3.0-flash-Q5_K_XXL-00001-of-00003.gguf -ngl 99

Reasoning is on by default — allow enough completion tokens or you will get truncated or empty content. Sampling per the upstream model card: temperature 0.6, top_p 0.95, top_k 20.

If -ngl 99 does not fit your card you have two dials, and context is the cheaper one to keep:

• Offload experts to CPU — -ncmoe N keeps the experts of N blocks in system RAM. Experts are 97% of the weights, so this is the big lever. Costs speed, not context.
• Reduce context — saves comparatively little, since the whole 256K window is under 2 GiB of KV. Reach for this last.

Rough sizing, weights + KV at full 256K: Q4_K_M ≈ 72 GiB, Q5_K_M ≈ 86 GiB, Q5_K_XL ≈ 91 GiB, Q5_K_XXL ≈ 95 GiB, Q8_0 ≈ 128 GiB, plus a couple of GiB of compute buffers.

Config for a single 96 GB card (RTX PRO 6000 Blackwell)

Specific to a 96 GB card with nothing else resident

./build/bin/llama-server \
  -m Ling-3.0-flash-Q5_K_XXL-00001-of-00003.gguf \
  -ngl 99 -c 262144 -ub 512 --no-mmap

Measured: 96,363 MiB resident at the full 256K window with no CPU offload, 134.5 tok/s decode, ~2,040 tok/s prefill on a 235K prompt.

-ub 512 is load-bearing at this size. -ub is the compute-buffer dial and its price scales with context: at 262,144 each ubatch token costs ~0.695 MiB, since the attention mask is ctx × ubatch and is materialised even with -fa on (measured: 256→512 = +178 MiB, 512→1024 = +356 MiB). At 32K it is roughly a tenth as expensive. Lowering -ub costs prefill throughput and leaves decode untouched. When tuning this close to the limit, confirm with an actual generation rather than a successful load — CUDA allocates some workspace lazily on the first matmul.

Context: 256K, not 131K

These GGUFs give you 262,144 tokens by default. You do not need any flag.

The confusion is upstream's: config.json ships max_position_embeddings: 131072, but inclusionAI's own model card documents a training schedule of 8K → 32K → 256K and benchmarks at a 256K window. The released config understates the model.

This matters because llama.cpp caps a slot to the GGUF's context_length:

the slot context (262144) exceeds the training context of the model (131072) - capping

so a GGUF built naively from config.json is stuck at 131072 regardless of what you pass to -c. These files declare context_length = 262144, verified by needle-in-a-haystack out to 247,583 tokens.

To use the vendor's literal config instead:

--override-kv bailing-hybrid.context_length=int:131072

Files

Each quant is split into shards for resumable download.

file	size
Ling-3.0-flash-Q4_K_M-*-of-00002.gguf	69.70 GiB
Ling-3.0-flash-Q5_K_M-*-of-00003.gguf	84.18 GiB
Ling-3.0-flash-Q5_K_XL-*-of-00003.gguf	88.92 GiB
Ling-3.0-flash-Q5_K_XXL-*-of-00003.gguf	92.78 GiB
Ling-3.0-flash-Q8_0-*-of-00004.gguf	126.32 GiB
bailing-hybrid-llama.cpp.patch	64 KB

How these were made

Experts are 97.12% of all parameters, so the recipe spends its budget there and protects everything else cheaply: all 3.67B non-expert parameters are pinned to q8_0 for only 3.6 GiB. That covers the KDA gates (ssm_f/ssm_g), the MLA absorption tensors (attn_k_b/attn_v_b), the head-wise attention gate, embeddings and output. Norms, the router (ffn_gate_inp), exp_probs_b and the KDA scalars (A_log, dt_bias, conv1d) stay F32.

Pinning attn_k_b also avoids a silent fallback: its ncols is 128, not divisible by 256, so K-quants degrade it to q5_1/q4_1.

The expert projections are where the tiers differ:

	down_exps	gate_exps	up_exps
Q5_K_M	q5_K	q5_K	q5_K
Q5_K_XL	q6_K	q5_K	q5_K
Q5_K_XXL	q6_K	q5_K	q6_K on layers 9–41

down_proj is promoted first because it is the output side of the SwiGLU, where error propagates instead of being gated away. up_proj follows: in down(silu(gate(x)) ⊙ up(x)) an error in up passes through the product with its relative magnitude intact, while an error in gate is first attenuated by silu.

Q5_K_XXL stops where it does because of a hardware ceiling. Promoting up on all 40 MoE layers costs another 892 MiB, which forces -ub 256 and drops prefill by roughly a quarter for no measurable accuracy gain. Full Q6_K experts needs ~101.4 GiB resident against 97,887 MiB of card, and since the entire 256K KV cache is only ~2 GiB, no amount of context reduction rescues it. The seven layers left at q5_K are the seven lowest by imatrix activation energy — Σ(Act²) of 11K–24K against 193K at layer 40.

imatrix from 892K tokens of mixed code, agent traces and technical prose. With top-8 routing over 512 experts, 40 of 120 expert tensors reached only 98.24–99.80% coverage — 1–9 experts per tensor never routed to, concentrated in the deepest layers. Those tensors (blk.36/40/41) are quantized one step higher so missing importance data is paid for in bits, costing ~1.1 GiB.

Validation

From a single RTX PRO 6000 Blackwell 96 GB. Q4/Q5 run all-GPU; Q8_0 with -ncmoe 16.

• Needle-in-haystack 12/12 at depths 10/50/90% across 6.6K, 28K, 61K and 123.4K tokens, and 12/12 again at 175K and 247.6K — which is what justifies the 262,144 context_length these files declare. Measured on Q4_K_M, Q5_K_M and Q8_0; Q5_K_XXL passes at 235,379 tokens and Q5_K_XL at 179,167.
• Coherence 6/6 — factual recall, arithmetic, the bat-and-ball trap, code generation, exact-word instruction following, translation.

Divergence figures in the table above come from llama-perplexity --kl-divergence against reference logits from the BF16 master itself, not a Q8_0 proxy — quant-vs-quant comparisons overstate damage because the errors are correlated. 160 chunks × 512 tokens of held-out wikitext-2 (~82K tokens, zero verbatim overlap with the imatrix corpus). Reference PPL 4.3970 ± 0.0505.

That ordering is solid and monotonic in bit-width. A 20-instance multi-hop reasoning probe at ~111K tokens could not separate the Q5 tiers from one another, so treat KL divergence as the evidence for quality ordering, and do not read the gaps between adjacent Q5 tiers as a promise of visibly better answers.

Not measured: MTP acceptance, tool calling, and reasoning quality between 128K and 256K — retrieval is verified to 247.6K, but reasoning was only probed at ~111K.

Architecture notes

42 layers: 35 KDA (Kimi Delta Attention) + 7 gated MLA at layers 5/11/17/23/29/35/41, plus an MTP/nextn head at layer 42 that is present in the files but unused by the graph — it loads as "unused tensor — ignoring" and costs no VRAM, so MTP can be enabled later without requantizing. 512 routed experts, top-8 plus 1 shared, noaux_tc grouped routing (8 groups, 4 used), sigmoid scoring.

Three things differ from Kimi-Linear and matter to anyone porting this:

• kda_safe_gate replaces -exp(A_log)*softplus(·) with lower_bound * sigmoid(exp(A_log) * (f(x) + dt_bias)), lower_bound = -5.0. A_log is stored as +exp(A_log); the sign lives in the lower bound.
• rope_interleave: true resolves to llama.cpp's NORM rope, not the NEOX that DeepSeek-style MLA usually uses.
• attention.g_proj exists on both layer types with different shapes — [4096,2560] KDA full-rank output gate vs [32,2560] MLA head-wise attention gate. The converter renames the KDA one.

Building llama.cpp

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 6ea215d17
git apply /path/to/bailing-hybrid-llama.cpp.patch

cmake -B build -DGGML_CUDA=ON && cmake --build build -j

The patch applies cleanly to a pristine checkout of 6ea215d17 and builds without warnings. It adds src/models/bailing-hybrid.cpp and conversion/bailing_hybrid.py, plus registration in llama-arch.{h,cpp}, llama-model.{h,cpp}, models.h, llama-hparams.h and the gguf-py constants/mappings. On a later master, expect ordinary context drift in the registration files; the two new files are self-contained.

License

MIT, following the base model.