bloomer010/Ling-3.0-flash-gguf

Compatibility

⚠️ Ling-3.0-flash uses the new bailingmoe3 GGUF architecture. While waiting on upstream support, use the following fork: https://github.com/aetherbird/llama.cpp/tree/bailingmoe3-support

Stock llama.cpp builds without bailingmoe3 support will not load the model.

Conversion and Quantization

Taken directly from the released inclusionAI/Ling-3.0-flash BF16 safetensors.

Conversion-specific tensor transformations include:

• A_log stored as exp(A_log)
• MLA kv_b_proj split into separate K and V tensors, with the K tensor transposed
• KDA convolution weights reshaped for llama.cpp
• Per-expert tensors stacked into GGUF expert tensors
• KDA and MLA g_proj tensors mapped separately

Norms, routing tensors, expert routing bias, KDA state scalars, dt_bias, and convolution weights remain F32.

Importance Matrix

Importance matrix generated from the Q8_0 model:

• wiki.train.raw
• 100 chunks
• 512 tokens per chunk
• 51,200 calibration tokens total
• 573 matrix entries

Quants

MXFP4_MOE:

• Quantized using llama.cpp's MXFP4_MOE quantization type

Q8_0:

• 8.51 BPW
• 126.3 GiB
• Includes MTP block

UD-Q2_K_XL:

• Model-specific Unsloth-style mixed tensor recipe
• Main expert gate/up tensors: IQ2_XS
• Main expert down tensors: IQ3_XXS
• Final target layer experts: IQ3_XXS and IQ4_XS
• Attention, shared experts, and KDA projections retained at higher precision
• MTP experts: Q3_K and Q4_K

IQ1_S:

• Expected size: approximately 24.9 GiB
• Preserves MTP functionality

Notes

The GGUF contains 43 blocks:

• 42 target-model layers
• 35 KDA layers
• 7 gated MLA layers at zero-based indices 5, 11, 17, 23, 29, 35, and 41
• One MTP/NextN block at index 42

The first two target layers use dense FFNs. The remaining target layers use 512 routed experts with top-8 selection plus one shared expert. Routing uses sigmoid scoring, expert bias, eight expert groups, and four selected groups.

The KDA safe gate is implemented as:

lower_bound * sigmoid(exp(A_log) * (f_proj(x) + dt_bias))

The lower bound is -5.0. The GGUF stores the positive exp(A_log) value, while the sign is supplied by the negative lower bound.

MTP Support

The MTP block is bundled inside every GGUF.

During ordinary inference, llama.cpp skips the MTP tensors and may report them as unused. They occupy disk space but are not loaded into the ordinary target-model buffer.

With --spec-type draft-mtp, the same GGUF is opened as an MTP draft model and block 42 is loaded and executed. No separate drafter file is required.

Validation Completed

• BF16 architecture load and tensor round-trip
• CPU and CUDA execution on a reduced-size BailingMoE3 fixture
• Target next-token parity against the Hugging Face implementation
• First three recursive MTP proposals matched the Hugging Face implementation
• Full MXFP4_MOE target and MTP graph smoke test
• Q8_0 conversion completed successfully with all 938 tensors

Build

git clone --branch bailingmoe3-support https://github.com/aetherbird/llama.cpp.git

cd llama.cpp

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j --target llama-cli llama-server

Usage

  ./build/bin/llama-server \
    -m Ling-3.0-flash-Q8_0.gguf \
    -c 131072 \
    -ngl auto \
    --flash-attn auto

Enable the bundled MTP drafter:

  ./build/bin/llama-server \
    -m Ling-3.0-flash-Q8_0.gguf \
    -c 131072 \
    -ngl auto \
    --flash-attn auto \
    --spec-type draft-mtp

MoE placement can be adjusted for available VRAM with -ncmoe N. Draft-model placement can be controlled separately with -ncmoed N and -ngld N.

Supports up to 256K context. Reasoning is enabled by default.

Upstream PR: https://github.com/ggml-org/llama.cpp/pull/26608