aj9o9/Ling-3.0-flash-GGUF

Ling-3.0-flash - GGUF (Q4_K_M, IQ3_XXS)

GGUF quantization of inclusionAI/Ling-3.0-flash, a 124B parameter Mixture-of-Experts model with 5.1B active parameters per token.

A note from me

This is my first quantization, and my first time patching llama.cpp. I did it to learn, and I learned a lot - including that I got part of it wrong, found it, and fixed it (see the warning below). I am sharing it because a working GGUF of this model did not exist when I started, not because I think it is authoritative.

Please report anything you find. Bad output, crashes, wrong metadata, better flags for a particular card - open a discussion on this repo or reach me at @ItsmeAjayKV. Suggestions are as welcome as bug reports.

On the llama.cpp side: I am not planning to submit this fork upstream. The architecture support was written with heavy AI assistance and the model is very new, so PR #26608 is the right place for upstream support to come from - it is independent work by someone else, it has had real review, and it handles the KDA gate parameters more cleanly than my branch does. My branch exists so these files have something to run on, and so I could understand how the pieces fit together.

This will not load in upstream llama.cpp. Ling 3.0 uses a hybrid KDA + MLA attention architecture that upstream does not yet support. These files declare the architecture bailing-hybrid, which currently exists only in the fork linked below. Loading them with a standard llama.cpp build fails with unknown model architecture.

If you cloned the fork before 2026-08-08, pull again. Earlier builds had the KDA forget gate wrong. Ling 3.0 sets kda_safe_gate: true, which selects lower_bound * sigmoid(exp(A_log) * (g + bias)) - lower_bound scales the sigmoid, it is not a clamp floor. The original build used the other branch of the reference kernel, -exp(A_log) * softplus(g + bias) plus a hard clamp, across 35 of the 42 layers.

Both versions produce values in the same range, so output stayed fluent and the problem was not visible from casual use. The curves differ substantially though: on a neutral token the old gate retained roughly 50% of the recurrent state where the reference retains roughly 8%.

Fixed in b94e0ea. The GGUF files here are unaffected - the bug was in the graph, not the weights, so there is nothing to re-download.

The fix builds and runs, but it has not been validated numerically against the reference implementation. A proper check means comparing logits against HF or vLLM on the same tokens, which I have not been able to run. If you can do that comparison, I would genuinely like to see the result.

Files

Quant	Files	Total	Bits/weight	imatrix
IQ3_XXS	Ling-3.0-flash-IQ3_XXS.gguf (single file)	47.9 GB	3.08	yes
IQ4_XS	Ling-3.0-flash-IQ4_XS-0000{1,2}-of-00002.gguf	66.4 GB	4.27	yes
Q4_K_M	Ling-3.0-flash-Q4_K_M-0000{1,2}-of-00002.gguf	75.2 GB	4.84	no

For split quants, download every shard into the same directory and point -m at the first one - llama.cpp finds the rest automatically. Do not rename them. IQ3_XXS is a single file.

IQ4_XS is smaller than Q4_K_M and imatrix-guided, so it is the better pick of the two if it fits your memory. Q4_K_M is kept because it was made first and some people prefer K-quants.

Note on IQ3_XXS: 7 of 917 tensors (blk.{5,11,17,23,29,35,41}.attn_k_b) fell back to iq4_nl because their 128 columns are not divisible by the 256-value block size IQ3_XXS requires. Those are the MLA layers, and they end up with slightly more precision than requested.

Which one? If you have 64 GB of RAM, take IQ3_XXS - it fits, and a quant that fits in RAM beats a better quant served from disk by a wide margin. Above that, Q4_K_M is the better model.

Also here:

File	Size	What it is
ling.imatrix	466 MB	The importance matrix used for IQ3_XXS. Reusable - see below.

How these were made

Both quants come from the same bf16 GGUF, converted from the original safetensors with the converter in the fork linked below.

Q4_K_M was quantized directly, with no importance matrix.

IQ3_XXS requires one (llama-quantize refuses to build it without), so:

bf16 (249 GB) --quantize--> Q8_0 (123 GiB) --llama-imatrix--> ling.imatrix
                                                                   |
bf16 (249 GB) -----------------quantize with imatrix---------------+--> IQ3_XXS

The Q8_0 exists only because the bf16 does not fit in the memory available for an inference pass. Q8_0 is near-lossless, so the activations it produces stand in for bf16's. The final quantization reads the bf16, not the Q8_0 - stacking two lossy steps would waste quality that a 3-bit quant cannot spare.

Calibration: calibration_data_v5_rc.txt (Dampf, building on kalomaze), 230 chunks at -c 512. This corpus was chosen deliberately: it mixes 20+ languages, code, config and mathematics, and v5 exists specifically to improve expert activation in sparse MoE models.

That matters here. Ling routes 8 of 512 experts per token, so a narrow corpus leaves most experts with no measurements at all, and those get quantized blind. Measured coverage:

Calibration	Expert coverage
3 chunks	62-67%
~100 chunks	~85%
230 chunks (final)	99.4-99.8%

Reusing the imatrix: ling.imatrix is the expensive artifact here - about 25 minutes on an H100. If you want a different size (IQ4_XS, IQ2_M, Q3_K_M), download it and pass --imatrix rather than recalibrating. It is architecture-specific to Ling-3.0-flash but not quant-specific.

Usage

Build the fork:

git clone -b ling-3.0-bailing-hybrid https://github.com/Ajay9o9/llama.cpp
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-completion

Run it. These files are 48-75 GB, so unless you have that much VRAM, offload the experts to CPU with --n-cpu-moe - only 8 of 512 experts are read per token, so this costs less than it sounds:

./build/bin/llama-completion \
  -m Ling-3.0-flash-IQ3_XXS.gguf \
  -p "Write a haiku about linear attention." \
  -ngl 99 --n-cpu-moe 42 -n 100

Lower the --n-cpu-moe number to keep more experts on the GPU if you have VRAM spare. On a 96 GB card with Q4_K_M fully resident, this model runs at roughly 113 tok/s.

Hardware notes

I made this quant with consumer cards in mind - specifically a 3060 (12 GB) and a 3090 (24 GB), paired with 64 GB of system RAM. Because only 8 of 512 experts run per token, pushing the expert weights to CPU costs far less throughput than the size suggests.

Setup	Expect
3090 (24 GB) + 64 GB RAM	Workable. Use --n-cpu-moe 42, lower it if VRAM allows.
3060 (12 GB) + 64 GB RAM	Tight but runs. Keep --n-cpu-moe 42 and expect a slow prompt phase.
64 GB RAM, no GPU	Possible via mmap, but slow.

This is why IQ3_XXS exists. Q4_K_M is 75.2 GB, which does not fit in 64 GB of RAM, so part of it is served from disk through mmap on every token. IQ3_XXS fits with room to spare. On a machine with 64 GB that difference matters more than the difference in quantization quality - an SSD helps, a spinning disk will crawl, and neither beats simply fitting in memory.

Rule of thumb: pick the largest quant that fits in RAM, not the largest quant you can download.

About the model

• 42 layers: 35 KDA (linear attention, recurrent state) interleaved 5:1 with 7 gated MLA layers
• 512 routed experts + 1 shared, 8 experts active per token, group-limited noaux_tc routing
• 256K context. Only the 7 MLA layers keep a KV cache, and they store a 576-value compressed latent per token rather than full per-head keys and values, so the cache stays small at long context.

See the original model card for benchmarks and training details. Those numbers were measured on the full bf16 model with the reference implementation and do not describe this quantization.

Links

• Original model: inclusionAI/Ling-3.0-flash
• Fork required to run this: Ajay9o9/llama.cpp @ ling-3.0-bailing-hybrid
• Upstream request: ggml-org/llama.cpp#26590
• Alternative implementation: ggml-org/llama.cpp#26608
• Reference implementation: inclusionAI/vllm-ling-v3

License

MIT, inherited from the original model.