mb4063/Qwythos-27B-v1-ROCmFP4

Qwythos-27B-v1 — ROCmFP4 (STRIX_LEAN)

ROCmFP4 (4‑bit, ~4.38 bpw) quantization of empero-ai/Qwythos-27B-v1 for AMD GPUs, using the ROCmFPX fork of llama.cpp.

All credit for the model itself goes to Empero: Qwythos-27B-v1 is their open‑weight, full‑parameter reasoning model — a Qwen3.5‑27B fine‑tune post‑trained SFT → DPO → ESFT, shipped as a complete pre‑RL checkpoint with nothing ablated: native multi‑token‑prediction (MTP) head, full vision tower, and 1M‑token context configuration. Tool calling, <think> reasoning, and uncensored behaviour all come from upstream — this repo only changes the weight format.

What this quant changes: ~17.6 GB Q4_K_M → 14.8 GB (13.82 GiB) ROCmFP4 STRIX_LEAN — the difference between “won’t fit 128K on 16 GB” and “fits 128K comfortably”.

The first ROCmFP4 quant optimized for discrete AMD RDNA4 GPUs (RX 9000 Series / gfx1201). Existing ROCmFP4 quants on Hugging Face predominantly target Strix Halo (gfx1151). While Strix Halo is a formidable workstation-class APU, its quants are tuned for a massive unified memory architecture (up to 128GB of shared RAM).

This quant is specifically engineered for the discrete GPU environment. It is built to operate efficiently within the strict dedicated VRAM limits of modern PCIe graphics cards while capitalizing on their massively higher memory bandwidth.

All benchmarks, generation speeds, and memory metrics below are measured on a discrete RX 9070 XT 16 GB.

What’s inside

Component	Precision
Dense transformer weights (64 layers)	q4_0_rocmfp4_strix_lean (~4.38 bpw)
Attention K/V tensors	Q6_K (protected by STRIX recipe)
Token embeddings	Q5_K
Native MTP head (1 block, ~0.28 GiB)	preserved
Vision tower (mmproj)	F16 (separate file)

Quantized with llama-quantize --allow-requantize from the Q4_K_M GGUF, using the ROCmFPX build for gfx1201 (RX 9000 series / RDNA4).

⚠️ Requires the ROCmFPX fork. The custom q4_0_rocmfp4 tensor types will not load in stock llama.cpp, LM Studio, or Ollama. Build/run with charlie12345/ROCmFPX.

Quantization quality cost

Measured on WikiText‑2 (llama-perplexity, -c 4096):

	Q4_K_M source	This ROCmFP4
Perplexity	6.22	6.3884 ± 0.0419
File size	17.6 GB (16.38 GiB)	14.8 GB (13.82 GiB) (−22%)
128K ctx on 16 GB	❌ doesn’t fit	✅ fits (~15.0 GB)

Double‑quantization cost: only +0.17 PPL (2.7% increase) — negligible.

Benchmarks

Hardware: AMD RX 9070 XT 16 GB (gfx1201) | Ryzen 9 9950X | ROCm 7.2
Build: ROCmFPX build-rdna4 (patched — see ROCmFPX bug section)

The model has 65 layers total (64 transformer + 1 MTP block). -ngl 99 and -ngl 85 therefore both mean full offload — they are equivalent.

Raw GPU speed (llama-bench, pp512 / tg128)

Model	-ngl	Backend	Prefill (t/s)	Generate (t/s)
ROCmFP4 STRIX_LEAN	99	ROCm	1106	31.9
ROCmFP4 STRIX_LEAN	99	Vulkan	972	33.9
Q4_K_M source	45	ROCm	211	9.7
Q4_K_M source	45	Vulkan	246	9.2

Q4_K_M at -ngl 99 does not fit 16 GB (17.6 GB model + KV cache > 16 GB). Only 45 layers can be offloaded → 3.4× slower generation, 4.5× slower prefill.

Real‑world server (Vulkan, -ngl 99 -fa 1 -b 256 -ub 256 -ctk q4_0 -ctv q4_0)

Context	Cache config	Cold prefill (t/s)	Generation (t/s)
131K	--cache-ram 3072 -ctxcp 12 -cpent 8192	383 (24.5K tokens, 64 s)	21.1
65K	--cache-ram 4096 -ctxcp 4 -cpent 8192	815 (23.3K tokens, 28 s)	31.5

Context allocation is the dominant speed lever. The same model on the same GPU runs 1.5× faster generation and 2.1× faster prefill at 65K vs 131K — even with the same ~24K tokens actually in use. The full KV allocation size, not the used portion, determines the flash‑attention scan cost.

Prompt‑cache speed‑up:
With --cache-ram 3072 -ctxcp 12 -cpent 8192, a saved checkpoint restores a large prefix in tens of milliseconds (log examples: 535 ms for 114 tokens, 662 ms for 160 tokens, 579 ms for 88 tokens). This avoids re‑evaluating the whole history on most turns.
Note: Qwythos uses a hybrid/SWA memory pattern that occasionally forces a full re‑processing (“forcing full prompt re‑processing”), but when checkpoints are reusable the improvement is dramatic.

MTP speculative decoding (65K context)

Config	Generation (t/s)	Draft acceptance	VRAM
Without MTP	31.5	—	~95%
--spec-type draft-mtp --spec-draft-n-max 3	48–50	87–90% (mean 2.6 tokens/step)	99.4%

ROCmFPX supports MTP speculative decoding (--spec-type draft-mtp), and the MTP head is preserved in this quant. At 65K context it delivers a +53–60% generation speedup with healthy draft acceptance (position 1: ~92%, position 2: ~45%, position 3: ~25%).

However, MTP is not recommended for agent/multi‑turn workloads:

• At 131K context, MTP breaks prompt‑cache checkpoint restoration (spec-boundary-mismatch forces a full 38–41 s prefill every turn)
• In our testing, enabling MTP caused the model to end agentic turns early (planning but not executing multi‑step tool tasks); disabling MTP resolved this
• At 131K the generation gain shrinks to +14–29% (24–27 t/s vs 21 t/s) — much less than the 65K figure

Size comparison

Format	Size	bpw	128K ctx on 16 GB
Q4_K_M	17.6 GB (16.38 GiB)	~5.0	❌
ROCmFP4 STRIX_LEAN	14.8 GB (13.82 GiB)	~4.38	✅ at -ngl 99
NVFP4 (NVIDIA, CocaKova)	~20 GB	—	N/A (different format: compressed-tensors, not GGUF; NVIDIA only; listed for size reference)

Quick start

Requires the ROCmFPX fork built for gfx1201 (RX 9000 series / RDNA4) — see charlie12345/ROCmFPX.

128K context (recommended for agent use — prompt cache works):

GGML_CUDA_DISABLE_GRAPHS=1 ~/ROCmFPX/build-rdna4/bin/llama-server \
  -m ~/models/Qwythos-27B-MTP-ROCmFP4-STRIX_LEAN.gguf \
  --host 127.0.0.1 --port 8138 \
  -dev Vulkan0 -ngl 99 -fa 1 -c 131072 \
  -ctk q4_0 -ctv q4_0 \
  -np 1 -b 256 -ub 256 --jinja \
  --temp 0.7 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
  --cache-ram 3072 --cache-reuse 256 \
  -ctxcp 12 -cpent 8192

65K context (1.5× faster generation, 2.1× faster prefill, less headroom):

GGML_CUDA_DISABLE_GRAPHS=1 ~/ROCmFPX/build-rdna4/bin/llama-server \
  -m ~/models/Qwythos-27B-MTP-ROCmFP4-STRIX_LEAN.gguf \
  --host 127.0.0.1 --port 8138 \
  -dev Vulkan0 -ngl 99 -fa 1 -c 65536 \
  -ctk q4_0 -ctv q4_0 \
  -np 1 -b 256 -ub 256 --jinja \
  --temp 0.7 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
  --cache-ram 4096 --cache-reuse 256 \
  -ctxcp 4 -cpent 8192

GGML_CUDA_DISABLE_GRAPHS=1 is a no‑op on the Vulkan backend; it's included for ROCm compatibility. Use -dev ROCm0 instead of -dev Vulkan0 for the ROCm backend (slightly faster prefill, same generation, more memory‑fragile — see the bug section below).

With vision (mmproj adds ~1 GB — drop context to 16K on 16 GB):

build-rdna4/bin/llama-server \
  -m Qwythos-27B-MTP-ROCmFP4-STRIX_LEAN.gguf \
  --mmproj mmproj-Qwythos-27B-F16.gguf \
  --host 127.0.0.1 --port 8138 \
  -dev Vulkan0 -ngl 99 -fa 1 -c 16384 \
  -ctk q4_0 -ctv q4_0 -np 1 -b 256 -ub 256 --jinja --temp 0.7

To enable MTP speculative decoding (65K context only), add --spec-type draft-mtp --spec-draft-n-max 3. See the MTP caveats above before using it for agent work.

ROCmFPX bug: prompt-cache checkpoint crash (and fix)

When --cache-ram > 0 and -ctxcp N -cpent N are used together, the ROCmFPX server may crash on the second request. The bug is in ggml_backend_tensor_copy (ggml/src/ggml-backend.cpp).

Fix: In ggml/src/ggml-backend.cpp, resolve views before accessing buffer pointers in ggml_backend_tensor_copy:

// Resolve views before accessing buffer pointers
ggml_backend_buffer_t src_buf = src->view_src ? src->view_src->buffer : src->buffer;
ggml_backend_buffer_t dst_buf = dst->view_src ? dst->view_src->buffer : dst->buffer;

Then rebuild with scripts/build-rdna4.sh.

Credits

All credit for the model goes to Empero. This repo only changes the weight format for AMD GPU compatibility. Read the original model card for the full training story.

Quantization tooling: ROCmFPX fork of llama.cpp.