README.md

---
license: apache-2.0
base_model: empero-ai/Qwythos-27B-v1
base_model_relation: quantized
tags:
  - rdna4
  - gfx1201
  - gguf
  - rocmfp4
  - rocmfpx
  - rx-9070-xt
  - amd
  - quantization
  - qwen3_5
  - qwen35
  - reasoning
  - multimodal
  - vision
  - uncensored
  - long-context
  - 1m-context
  - mtp
  - multi-token-prediction
  - function-calling
  - tool-use
  - agentic
  - sft
  - dpo
  - esft
  - full-fine-tune
  - conversational

  
  
library_name: llama.cpp
pipeline_tag: image-text-to-text
quantized_by: mb4063
language:
  - en
---

# Qwythos-27B-v1 — ROCmFP4 (STRIX_LEAN)

ROCmFP4 (4‑bit, ~4.38 bpw) quantization of [empero-ai/Qwythos-27B-v1](https://huggingface.co/empero-ai/Qwythos-27B-v1) for AMD GPUs, using the [ROCmFPX](https://github.com/charlie12345/ROCmFPX) fork of llama.cpp.

All credit for the model itself goes to [Empero](https://empero.org/): Qwythos-27B-v1 is their open‑weight, full‑parameter reasoning model — a Qwen3.5‑27B fine‑tune post‑trained SFT → DPO → ESFT, shipped as a complete pre‑RL checkpoint with nothing ablated: native multi‑token‑prediction (MTP) head, full vision tower, and 1M‑token context configuration. Tool calling, `<think>` reasoning, and uncensored behaviour all come from upstream — this repo only changes the weight format.

**What this quant changes:** ~17.6 GB Q4_K_M → **14.8 GB** (13.82 GiB) ROCmFP4 STRIX_LEAN — the difference between “won’t fit 128K on 16 GB” and “fits 128K comfortably”.

**The first ROCmFP4 quant optimized for discrete AMD RDNA4 GPUs (RX 9000 Series / gfx1201).** Existing ROCmFP4 quants on Hugging Face predominantly target Strix Halo (gfx1151). While Strix Halo is a formidable workstation-class APU, its quants are tuned for a massive unified memory architecture (up to 128GB of shared RAM).

This quant is specifically engineered for the discrete GPU environment. It is built to operate efficiently within the strict dedicated VRAM limits of modern PCIe graphics cards while capitalizing on their massively higher memory bandwidth.

All benchmarks, generation speeds, and memory metrics below are measured on a discrete RX 9070 XT 16 GB.

## What’s inside

| Component | Precision |
|---|---|
| Dense transformer weights (64 layers) | `q4_0_rocmfp4_strix_lean` (~4.38 bpw) |
| Attention K/V tensors | `Q6_K` (protected by STRIX recipe) |
| Token embeddings | `Q5_K` |
| Native MTP head (1 block, ~0.28 GiB) | preserved |
| Vision tower (mmproj) | F16 (separate file) |

Quantized with `llama-quantize --allow-requantize` from the Q4_K_M GGUF, using the ROCmFPX build for `gfx1201` (RX 9000 series / RDNA4).

> ⚠️ **Requires the ROCmFPX fork.** The custom `q4_0_rocmfp4` tensor types will not load in stock llama.cpp, LM Studio, or Ollama. Build/run with [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX).

## Quantization quality cost

Measured on WikiText‑2 (`llama-perplexity`, `-c 4096`):

| | Q4_K_M source | This ROCmFP4 |
|---|---|---|
| Perplexity | 6.22 | **6.3884 ± 0.0419** |
| File size | 17.6 GB (16.38 GiB) | **14.8 GB (13.82 GiB) (−22%)** |
| 128K ctx on 16 GB | ❌ doesn’t fit | ✅ fits (~15.0 GB) |

Double‑quantization cost: only +0.17 PPL (2.7% increase) — negligible.

## Benchmarks

**Hardware:** AMD RX 9070 XT 16 GB (gfx1201) | Ryzen 9 9950X | ROCm 7.2  
**Build:** ROCmFPX `build-rdna4` (patched — see [ROCmFPX bug section](#rocmfpx-bug-prompt-cache-checkpoint-crash-and-fix))

The model has **65 layers total (64 transformer + 1 MTP block)**. `-ngl 99` and `-ngl 85` therefore both mean full offload — they are equivalent.

### Raw GPU speed (`llama-bench`, pp512 / tg128)

| Model | `-ngl` | Backend | Prefill (t/s) | Generate (t/s) |
|---|---|---|---|---|
| **ROCmFP4 STRIX_LEAN** | 99 | ROCm | 1106 | 31.9 |
| **ROCmFP4 STRIX_LEAN** | 99 | Vulkan | 972 | 33.9 |
| Q4_K_M source | 45 | ROCm | 211 | 9.7 |
| Q4_K_M source | 45 | Vulkan | 246 | 9.2 |

Q4_K_M at `-ngl 99` does **not** fit 16 GB (17.6 GB model + KV cache > 16 GB). Only 45 layers can be offloaded → 3.4× slower generation, 4.5× slower prefill.

### Real‑world server (Vulkan, `-ngl 99 -fa 1 -b 256 -ub 256 -ctk q4_0 -ctv q4_0`)

| Context | Cache config | Cold prefill (t/s) | Generation (t/s) |
|---|---|---|---|
| **131K** | `--cache-ram 3072 -ctxcp 12 -cpent 8192` | **383** (24.5K tokens, 64 s) | **21.1** |
| **65K** | `--cache-ram 4096 -ctxcp 4 -cpent 8192` | **815** (23.3K tokens, 28 s) | **31.5** |

**Context allocation is the dominant speed lever.** The same model on the same GPU runs **1.5× faster generation and 2.1× faster prefill at 65K vs 131K** — even with the same ~24K tokens actually in use. The full KV allocation size, not the used portion, determines the flash‑attention scan cost.

**Prompt‑cache speed‑up:**  
With `--cache-ram 3072 -ctxcp 12 -cpent 8192`, a saved checkpoint restores a large prefix in **tens of milliseconds** (log examples: 535 ms for 114 tokens, 662 ms for 160 tokens, 579 ms for 88 tokens). This avoids re‑evaluating the whole history on most turns.  
*Note:* Qwythos uses a hybrid/SWA memory pattern that occasionally forces a full re‑processing (“forcing full prompt re‑processing”), but when checkpoints are reusable the improvement is dramatic.

### MTP speculative decoding (65K context)

| Config | Generation (t/s) | Draft acceptance | VRAM |
|---|---|---|---|
| Without MTP | 31.5 | — | ~95% |
| `--spec-type draft-mtp --spec-draft-n-max 3` | **48–50** | 87–90% (mean 2.6 tokens/step) | **99.4%** |

ROCmFPX supports MTP speculative decoding (`--spec-type draft-mtp`), and the MTP head is preserved in this quant. At 65K context it delivers a **+53–60% generation speedup** with healthy draft acceptance (position 1: ~92%, position 2: ~45%, position 3: ~25%).

**However, MTP is not recommended for agent/multi‑turn workloads:**
- At 131K context, MTP breaks prompt‑cache checkpoint restoration (`spec-boundary-mismatch` forces a full 38–41 s prefill every turn)
- In our testing, enabling MTP caused the model to end agentic turns early (planning but not executing multi‑step tool tasks); disabling MTP resolved this
- At 131K the generation gain shrinks to +14–29% (24–27 t/s vs 21 t/s) — much less than the 65K figure

### Size comparison

| Format | Size | bpw | 128K ctx on 16 GB |
|---|---|---|---|
| Q4_K_M | 17.6 GB (16.38 GiB) | ~5.0 | ❌ |
| **ROCmFP4 STRIX_LEAN** | **14.8 GB (13.82 GiB)** | **~4.38** | ✅ at `-ngl 99` |
| NVFP4 (NVIDIA, [CocaKova](https://huggingface.co/CocaKova/Qwythos-27B-v1-NVFP4)) | ~20 GB | — | N/A (different format: compressed-tensors, not GGUF; NVIDIA only; listed for size reference) |

## Quick start

Requires the ROCmFPX fork built for `gfx1201` (RX 9000 series / RDNA4) — see [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX).

**128K context (recommended for agent use — prompt cache works):**

```bash
GGML_CUDA_DISABLE_GRAPHS=1 ~/ROCmFPX/build-rdna4/bin/llama-server \
  -m ~/models/Qwythos-27B-MTP-ROCmFP4-STRIX_LEAN.gguf \
  --host 127.0.0.1 --port 8138 \
  -dev Vulkan0 -ngl 99 -fa 1 -c 131072 \
  -ctk q4_0 -ctv q4_0 \
  -np 1 -b 256 -ub 256 --jinja \
  --temp 0.7 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
  --cache-ram 3072 --cache-reuse 256 \
  -ctxcp 12 -cpent 8192
```

**65K context (1.5× faster generation, 2.1× faster prefill, less headroom):**

```bash
GGML_CUDA_DISABLE_GRAPHS=1 ~/ROCmFPX/build-rdna4/bin/llama-server \
  -m ~/models/Qwythos-27B-MTP-ROCmFP4-STRIX_LEAN.gguf \
  --host 127.0.0.1 --port 8138 \
  -dev Vulkan0 -ngl 99 -fa 1 -c 65536 \
  -ctk q4_0 -ctv q4_0 \
  -np 1 -b 256 -ub 256 --jinja \
  --temp 0.7 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
  --cache-ram 4096 --cache-reuse 256 \
  -ctxcp 4 -cpent 8192
```

`GGML_CUDA_DISABLE_GRAPHS=1` is a no‑op on the Vulkan backend; it's included for ROCm compatibility. Use `-dev ROCm0` instead of `-dev Vulkan0` for the ROCm backend (slightly faster prefill, same generation, more memory‑fragile — see the bug section below).

**With vision** (mmproj adds ~1 GB — drop context to 16K on 16 GB):

```bash
build-rdna4/bin/llama-server \
  -m Qwythos-27B-MTP-ROCmFP4-STRIX_LEAN.gguf \
  --mmproj mmproj-Qwythos-27B-F16.gguf \
  --host 127.0.0.1 --port 8138 \
  -dev Vulkan0 -ngl 99 -fa 1 -c 16384 \
  -ctk q4_0 -ctv q4_0 -np 1 -b 256 -ub 256 --jinja --temp 0.7
```

To enable MTP speculative decoding (65K context only), add `--spec-type draft-mtp --spec-draft-n-max 3`. See the MTP caveats above before using it for agent work.

## ROCmFPX bug: prompt-cache checkpoint crash (and fix)

When `--cache-ram > 0` and `-ctxcp N -cpent N` are used together, the ROCmFPX server may crash on the second request. The bug is in `ggml_backend_tensor_copy` (`ggml/src/ggml-backend.cpp`).

**Fix:** In `ggml/src/ggml-backend.cpp`, resolve views before accessing buffer pointers in `ggml_backend_tensor_copy`:

```cpp
// Resolve views before accessing buffer pointers
ggml_backend_buffer_t src_buf = src->view_src ? src->view_src->buffer : src->buffer;
ggml_backend_buffer_t dst_buf = dst->view_src ? dst->view_src->buffer : dst->buffer;
```

Then rebuild with `scripts/build-rdna4.sh`.

## Credits

All credit for the model goes to [Empero](https://huggingface.co/empero-ai). This repo only changes the weight format for AMD GPU compatibility. Read the [original model card](https://huggingface.co/empero-ai/Qwythos-27B-v1) for the full training story.

Quantization tooling: [ROCmFPX](https://github.com/charlie12345/ROCmFPX) fork of llama.cpp.