kingjones777/Ling-3.0-tiny-base-30T-ROCmFP4-COHERENT-GGUF

Ling-3.0-tiny-base-30T — ROCmFP4 for AMD Strix Halo (gfx1151)

4-bit ROCmFP4 quantisation of Ling-3.0-tiny-base-30T, one of the six Ling-3.0 base/training checkpoints inclusionAI released on 2026-08-20. The multi-token-prediction (MTP) head is preserved.

⚠️ This is a base checkpoint, not an instruct model

Released for continued pretraining, domain adaptation and fine-tuning. Not instruction-tuned. It ships a chat_template.jinja, but that is a tokenizer asset and does not make the weights conversational. Prompt it as a text continuation model. For chat, use inclusionAI/Ling-3.0-tiny.

The file

ftype	102 — Q4_0_ROCMFP4_COHERENT
size	4,769,674,624 bytes
architecture	bailing-hybrid — hybrid KDA linear attention + MLA
tensors	549 · block_count 25 (24 layers + 1 MTP layer)
q_lora_rank	256 — low-rank compressed queries
context	262,144

Head protection, verified in the finished file:

output.weight        Q6_K     1536 x 157184
token_embd.weight    Q6_K     1536 x 157184

tie_word_embeddings is false, so --output-tensor-type does real work here — the COHERENT tier alone leaves output.weight at 4-bit. Both heads were forced to Q6_K and audited on exact tensor names after the build.

⚙️ Requires a patched llama.cpp — details

Ling-3.0-tiny sets q_lora_rank: 256, so its MLA layers use a two-stage compressed query (q_a_proj → RMS norm → q_b_proj). Ling-3.0-flash sets q_lora_rank: null and uses a single wide q_proj. A bailing-hybrid implementation written against flash therefore cannot load tiny.

The patch, against ROCmFPX, mirrors the existing DeepSeek-V2 low-rank query path:

file	change
gguf-py/gguf/tensor_mapping.py	map attention.q_a_proj / q_b_proj / q_a_layernorm → ATTN_Q_A / ATTN_Q_B / ATTN_Q_A_NORM
gguf-py/gguf/constants.py	add those three tensors to MODEL_ARCH.BAILING_HYBRID (TensorNameMap skips anything not in the arch list)
convert_hf_to_gguf.py	emit add_q_lora_rank() when set; the null path is unchanged
src/models/bailing-hybrid.cpp	read Q_LORA_RANK as optional; when n_lora_q > 0 create wq_a / wq_b / attn_q_a_norm and run q_a → RMS → q_b at both graph sites, else keep the wide wq

Because the KV is read as optional, flash GGUFs (which lack it) keep n_lora_q = 0 and take the original path unchanged. ~51 lines across 5 files.

Verify a build carries it with strings libllama.so | grep bailing-hybrid — the architecture table lives in the shared library, not the thin CLI binary.

Measured throughput

AMD Ryzen AI Max+ 395, Radeon 8060S (gfx1151), ROCm 7.2.4, 128 GB unified memory. llama-cli, -dio -ngl 999 -st -c 2048 -n 512 --temp 0 --seed 1234, 3 repetitions.

config	flags	generation (median)	runs
no drafter	--spec-type none	107.6 t/s	107.4 / 107.6 / 107.6
MTP n-max 3	--spec-type draft-mtp --spec-draft-ngl 999 --spec-draft-n-max 3	102.9 t/s	102.9 / 103.2 / 102.4

-4.4% from MTP. Do not enable MTP on this checkpoint. It is a measured net loss of 4.4%.

The three Ling-3.0-tiny base checkpoints do not agree on this. Measured on identical hardware with identical flags: tiny-base +7.5%, tiny-base-midtrain +5.1%, tiny-base-30T −4.4%. Every comparison was range-disjoint against a 0.2–0.7% baseline spread. The MTP draft head is trained alongside the model, so its quality is a property of the training checkpoint, not of the architecture — and a checkpoint can be fully competitive on quality and speed while shipping a draft head that is a net negative. Measure before enabling it.

Sample output

Continuation from "The history of mathematics begins in ancient times. One of the earliest known":

 The history of mathematics begins in ancient times. One of the earliest known mathematical texts is the Rhind Papyrus, which dates back to around 1650 BCE. This document, discovered in Egypt, contains problems and solutions related to arithmetic, geometry, and algebra. It was used by the ancient Egyptians to teach

Not measured

Perplexity is not published for this build. No perplexity figure is quoted because none was completed. Quality evidence is the coherence check above plus the tensor-level audit.

Provenance

Converted from inclusionAI/Ling-3.0-tiny-base-30T at revision 5ffd60dad2cd5df60fb916ab431682c563290797 to BF16 GGUF (549 tensors), then quantised to ftype 102 with --output-tensor-type q6_K. Licence MIT, inherited from the base model.