6block/Qwen3.8-27B-GGUF

Qwen3.8-27B GGUF

GGUF quantizations of Qwen/Qwen3.8-27B, made with llama.cpp.

Chinese version: README_zh.md

A 27B dense vision-language model (27.8B including the vision tower). 64 decoder layers in a hybrid stack: full_attention_interval=4 means every fourth layer is grouped-query full attention (16 layers) and the rest are Gated DeltaNet linear attention (48 layers). Vocab 248320 with untied embeddings. Context length 262144 natively, extensible further with YaRN.

Text, image and video in; text out.

What is in this repo

File group	What it is
Qwen3.8-27B-<tier>.gguf	text backbone, one file per tier
mmproj-Qwen3.8-27B-*.gguf	vision projector — required for image/video input
mtp-Qwen3.8-27B-*.gguf	MTP head, usable as a speculative-decoding draft
imatrix.gguf	the calibration matrix every tier was built with

The mmproj file is separate by design: llama.cpp loads the text backbone and the vision projector as two files. Download one tier plus one mmproj. Without the mmproj the model still runs, but text-only.

The mmproj- and mtp- prefixes are what recent llama.cpp builds match on to tell the three file kinds apart, so -hf 6block/Qwen3.8-27B-GGUF picks a backbone rather than a projector, and --spec-type mtp finds the draft on its own.

The draft is larger than an MTP head alone would suggest: a standalone draft needs its own embedding and output projection, and with vocab 248320 × hidden 5120 untied those two copies dominate the file. The MTP block itself is small.

Quantizations

Tier	Size	BPW	PPL
master (BF16, not in this repo)	50.11 GiB	16.00	6.7856 ± 0.07388
Q8_0	26.12 GiB	8.34	6.7900 ± 0.07391
Q6_K	21.01 GiB	6.71	6.7951 ± 0.07403
Q5_K_M	18.70 GiB	5.97	6.8010 ± 0.07406
Q5_K_S	18.21 GiB	5.82	6.7973 ± 0.07396
Q4_1	16.89 GiB	5.40	6.8105 ± 0.07411
Q4_K_M	16.53 GiB	5.28	6.8103 ± 0.07414
IQ4_NL	15.87 GiB	5.07	6.8491 ± 0.07484
Q4_K_S	15.68 GiB	5.01	6.8291 ± 0.07440
Q4_0	15.62 GiB	4.99	6.8749 ± 0.07520
IQ4_XS	15.28 GiB	4.88	6.8529 ± 0.07487
Q3_K_M	13.23 GiB	4.23	7.1020 ± 0.07902
IQ3_M	12.56 GiB	4.01	7.0306 ± 0.07531
Q3_K_S	12.16 GiB	3.88	7.2176 ± 0.08074
IQ3_XXS	11.60 GiB	3.70	7.1994 ± 0.07848
Q2_K	11.11 GiB	3.55	7.4700 ± 0.08388
IQ2_M	10.50 GiB	3.35	7.5101 ± 0.08176
IQ2_XS	9.86 GiB	3.15	7.9668 ± 0.08803
IQ2_XXS	9.30 GiB	2.97	8.5638 ± 0.09560

Tiers a smaller file already matches

The ladder is published in full for coverage, which means a few tiers end up larger than another tier that measured no worse on this model:

Tier	Size / PPL	Smaller, PPL no higher
Q5_K_M	18.70 GiB / 6.8010	Q5_K_S — 18.21 GiB / 6.7973
Q4_1	16.89 GiB / 6.8105	Q4_K_M — 16.53 GiB / 6.8103
IQ4_NL	15.87 GiB / 6.8491	Q4_K_S — 15.68 GiB / 6.8291
Q4_0	15.62 GiB / 6.8749	IQ4_XS — 15.28 GiB / 6.8529
Q3_K_M	13.23 GiB / 7.1020	IQ3_M — 12.56 GiB / 7.0306
Q3_K_S	12.16 GiB / 7.2176	IQ3_XXS — 11.60 GiB / 7.1994

This is a size argument, not a quality one. The size gap is real; the PPL side is a tie — Q5_K_M and Q5_K_S differ by 0.004 against a ±0.074 error bar, and Q4_1 vs Q4_K_M by 0.0002. Nothing here says the left column is broken.

Q4_0 and Q4_1 are the pre-K-quant formats and IQ4_NL targets backends without full K-quant support, so they are kept for runtimes that need them. If yours loads either, take the right-hand column.

The master row is not a file in this repo. It is listed so the numbers above have a reference point.

KV cache is cheaper than the layer count suggests

Only the 16 full-attention layers keep a growing KV cache; the 48 linear-attention layers hold a fixed-size recurrent state that does not grow with context. That works out to roughly 64 KiB per token, so 32k context costs about 2 GiB — much less than a 64-layer all-attention model of this size would need.

Budget roughly tier size + 0.9 GiB (mmproj) + KV cache when picking a tier.

Calibration and protection

Every tier is imatrix-calibrated (n_ctx=512, mixed code / English / Chinese), and imatrix.gguf is published here so the calibration is auditable and reusable. The layers that suffer most under low-bit compression are protected:

Tensors	Type	Reason
ssm_alpha / ssm_beta	F32	the linear-attention state gates; 0.09 GiB in total, so full precision is essentially free
attn_q / attn_k / attn_v / attn_output	Q8_0	the 16 full-attention layers — the only ones whose KV cache grows with context
token_embd / output	Q6_K (Q4_K below 4-bit)	see below

Deliberately not pinned: ssm_out, attn_qkv and attn_gate take the tier type like any other projection. Together they are 20.6% of the backbone, so an 8-bit floor under all three flattens the ladder: a Q4_K_M built that way came out at 22.97 GiB / 7.33 bpw against the shipped 16.53 GiB / 5.28 bpw, with five tiers landing within 7 GiB of each other.

Both builds were measured on the same command. The heavy one scored 6.8161 against 6.8103 shipped — a tie inside the ±0.074 error bar. The extra 6.44 GiB bought nothing. ssm_out reads like state but is the output projection of the linear-attention block, the counterpart of attn_output; llama.cpp's own quantizer gives it no special treatment either.

token_embd and output are 9.4% of the backbone on their own — vocab 248320 × hidden 5120, and the two are untied so there are two copies. Pinning both to Q6_K at every tier would push the low tiers well above their nominal bit-width (IQ2_XS would land near 3.0 bpw effective), so at sub-4-bit tiers they drop to Q4_K instead. The BPW column above is the effective figure, protection included.

Usage

# Text only
llama-cli -m Qwen3.8-27B-Q4_K_M.gguf -c 8192

# With vision — pass the projector alongside the backbone
llama-mtmd-cli -m Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
  --image photo.jpg -p "Describe this image."

Do not pass -ngl manually. llama.cpp fits layers to free VRAM by itself; setting -ngl explicitly aborts that fitting (n_gpu_layers already set by user) and can silently fall back to CPU.

Speculative decoding with the MTP head. --spec-type draft-mtp is required, not optional: with -md alone the draft is loaded but never used, because the speculative type stays unset and the MTP path is gated on it.

llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
  -md mtp-Qwen3.8-27B-Q8_0.gguf --spec-type draft-mtp

Pulling straight from this repo, the draft is discovered from the mtp- prefix and the type is inferred, so --spec-type can be omitted:

llama-server -hf 6block/Qwen3.8-27B-GGUF --spec-draft-hf 6block/Qwen3.8-27B-GGUF

About the PPL numbers

Measured in-house with one fixed setup, master baseline run through the exact same command. These numbers are only comparable within this table. Do not compare them against PPL figures published by other repos — different corpora and chunk counts make absolute values meaningless across setups.

License

Apache 2.0, inherited from Qwen/Qwen3.8-27B. Quantized by 6block.