bowmanslayer/Qwen3.8-27B-Text-Only-GGUF

Qwen3.8-27B · Text-Only · GGUF (imatrix)

Qwen3.8-27B for llama.cpp — vision tower and MTP head removed at the weight level, importance-matrix quantized from the original BF16. Runs from 16 GB VRAM up.

Optional vision is available as a separate mmproj file (see below), so you only pay for it if you use it.

Community quantization. Not an official Qwen release; not endorsed by or affiliated with the Qwen team or Alibaba Cloud. "Qwen3.8" identifies the upstream model this artifact derives from (Apache-2.0 §6).

---

Which file do I download?

File	Size	pp512	tg128	Max context on one 3090
Q8_0	26.6 GiB	—	—	does not fit	Near-lossless. Needs 32 GB, or partial offload on 24 GB
Q6_K	20.6 GiB	1088	32.3	32,512	Best quality on one 24 GB card — but look at that context
Q5_K_M	17.9 GiB	1110	36.9	73,472
Q4_K_M	15.4 GiB	1183	41.4	111,872	Fastest, and 3.4× the context of Q6_K. The default pick
IQ4_XS	14.0 GiB	1042	31.9	133,632	Pick for a 16 GB card — see the caveat below
Q3_K_M	12.4 GiB	992	38.5	158,976	Quality drops noticeably
mmproj-*	0.9 GiB	—	—	—	Optional vision, pairs with any of the above

All three columns measured on one RTX 3090 (24 GB), Vulkan, full offload. Context figures come from llama-fit-params, i.e. what actually fits — not a calculation.

The size/context trade is steeper than the size/quality trade. Dropping from Q6_K to Q4_K_M costs a little quality and buys 3.4× the context — on the same card.

The counter-intuitive part: IQ4_XS is 9 % smaller than Q4_K_M but ~23 % slower. IQ-family quants cost more compute to dequantize. Take IQ4_XS because you need the size, not because you want speed. If Q4_K_M fits your card, it is both faster and higher quality.

16 GB card: IQ4_XS. Q4_K_M technically loads but leaves almost nothing for KV. 24 GB card: Q4_K_M for speed, Q6_K for quality. Apple Silicon: unified memory is the budget — 32 GB → Q5_K_M/Q6_K, 64 GB → Q8_0.

Vulkan numbers. CUDA builds are typically faster.

---

Why the KV cache is unusually small

Qwen3.8 is a hybrid-attention model. Of its 64 layers only 16 are full attention — the other 48 are Gated DeltaNet linear attention and hold no KV cache. With num_key_value_heads = 4, head_dim = 256:

KV dtype	Per token	32K ctx	128K ctx
f16	64 KiB	2.0 GiB	8.0 GiB
q8	32 KiB	1.0 GiB	4.0 GiB

A comparable dense-attention 27B needs roughly four times this — which is why the context figures in the table above are as large as they are.

The table is measured on a 24 GB card. For a 16 GB card with IQ4_XS (14.0 GiB) the same arithmetic gives roughly 32K at f16 KV or 64K at q8 — estimated, not measured, as the author has no 16 GB card to test on.

---

Vision

llama-mtmd-cli -m Qwen3.8-27B-Text-Only-Q4_K_M.gguf \
               --mmproj mmproj-Qwen3.8-27B-f16.gguf \
               --image photo.jpg -p "Describe this image."

The projector is the general-purpose tower from Qwen3.8-27B, unchanged. Dedicated Qwen3-VL-* models will still do better on dense OCR and small-object counting.

---

How these were made

Quantized from the original BF16 weights, not re-quantized from an existing INT4 release — so no compounding loss.

1. Vision tower (model.visual.*) and MTP block (model.mtp*) dropped at the safetensors level; model.language_model.* promoted to model.*
2. convert_hf_to_gguf.py --outtype bf16 --no-mtp → BF16 GGUF (lossless from source)
3. Importance matrix over 300 chunks of the same calibration corpus used for this author's W4A16 releases (512 passages, pile-val news text), on 2×RTX 3090
4. Every level quantized with that imatrix, K-quants included

Every file in this repo was loaded and generated with before publishing — including both mmproj files, which were checked against a synthetic image with known content. Not just checksum-verified.

---

Requirements

A llama.cpp build that knows the qwen35 architecture. Build b10502 or newer works.

Older builds fail with check_tensor_dims: tensor 'blk.64...' not found. That is a converter that counted an MTP block these files do not contain — not a corrupt download.