README.md

---
license: apache-2.0
base_model:
  - EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  - Qwen/Qwen3.6-35B-A3B
tags:
  - gguf
  - rocmfpx
  - rocmfp2
  - qwen35moe
  - strix-halo
  - amd
  - speculative-decoding
  - mtp
pipeline_tag: text-generation
---

# Qwen3.6-35B-A3B-Escha-W2 → ROCmFP2 (GGUF)

A GGUF transcode of [EschaLabs/Qwen3.6-35B-A3B-Escha-W2](https://huggingface.co/EschaLabs/Qwen3.6-35B-A3B-Escha-W2)
into the **ROCmFPX** quantization formats, for AMD Strix Halo / gfx1151 and other
ROCm/Vulkan targets.

Escha's original export is a proprietary `eschamoe` packed format that only ships a
CUDA decoder. This repo contains the same expert weights decoded and repacked into
`Q2_0_ROCMFPX`, so the model runs on AMD hardware through the ROCmFPX llama.cpp fork.

> **This file will not load in upstream llama.cpp, Ollama, LM Studio, koboldcpp or
> llama-cpp-python.** It uses ggml tensor type IDs `107` (`Q2_0_ROCMFPX`) and `103`
> (`Q8_0_ROCMFPX`), which are outside upstream's range (`GGML_TYPE_COUNT = 43`).
> You need the fork below. The `qwen35moe` architecture itself *is* upstream — only
> the quantization types are fork-specific.

---

## Requirements

Build [**charlie12345/ROCmFPX**](https://github.com/charlie12345/ROCmFPX) at commit
`41db2f9eb` or later:

```bash
git clone https://github.com/charlie12345/ROCmFPX
cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151
cmake --build build -j
```

The model itself loads on any build from `db6844d9b` (build 172) onward — the
quantization types, the `qwen35moe` architecture and `draft-mtp` are all present there.
Only `--spec-mtp-strict-qwen` requires `41db2f9eb`+.

Pin the commit if you can. The ROCmFPX type IDs live in a private-use range and could
renumber between fork revisions, which would silently mismatch an older build.

## Contents

| | |
|---|---|
| File | `Qwen3.6-35B-A3B-Escha-W2-ROCmFP2.gguf` |
| Size | 13.10 GB (12.2 GiB) |
| Architecture | `qwen35moe` (35.5B total, ~3B active, 256 experts, hybrid linear+full attention) |
| Native context | 262144 |
| Routed experts | `Q2_0_ROCMFPX` (2.5 bpw) |
| Dense / attention / shared experts | `Q8_0_ROCMFPX` |
| Norms, `ssm_conv1d` | `F32` |
| MTP head experts (`blk.40`) | `Q4_0` — see note below |

### About the MTP head

Escha's export ships the `nextn` (MTP) block **without its routed experts**, so a GGUF
built from it alone fails to load with `missing tensor 'blk.40.ffn_down_exps.weight'`.
Those two tensors were grafted from the base `Qwen/Qwen3.6-35B-A3B` at `Q4_0`.

They are **not** Escha weights. This only affects the *draft* model — every speculative
token is verified against the target model before being accepted, so drafting from
base-model weights cannot introduce wrong tokens — and with `--spec-mtp-strict-qwen` the
output is bit-identical to non-speculative greedy decoding. The graft costs 453 MB and
buys a **~1.5× decode speedup**.

If you don't want it, strip block 40 and set `qwen35moe.block_count=40`,
`qwen35moe.nextn_predict_layers=0` — that yields a 12.65 GB file (+2.6% over Escha's
original, versus +6.3% with the head).

---

## Recommended flags

Measured on Strix Halo (Radeon 8060S, gfx1151), Vulkan backend.

```bash
llama-server \
  -m Qwen3.6-35B-A3B-Escha-W2-ROCmFP2.gguf \
  -np 1 \
  -dev Vulkan0 --spec-draft-device Vulkan0 \
  -ngl 999 --spec-draft-ngl all \
  -fa on --no-mmap \
  -ctk f16 -ctv f16 \
  -c 131072 -b 2048 -ub 512 \
  --jinja \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --spec-draft-n-min 0 \
  --spec-draft-p-min 0.15 \
  --spec-mtp-strict-qwen
```

> **`--spec-mtp-strict-qwen` needs `41db2f9eb` or newer** (added in
> [PR #54](https://github.com/charlie12345/ROCmFPX/pull/54)). Older binaries reject the
> flag — if yours does, either update or drop that line. Everything else works without
> it and the speed figures are unaffected, since strict costs only ~0.5%. What you lose
> is the *guarantee* that speculative output is bit-identical to non-speculative greedy
> decoding; drafts are verified against the target model either way, so output remains
> valid.

`--spec-type` is a **llama-server** flag; `llama-completion` rejects it. Note also that
with `--jinja`, the raw `/completion` endpoint returns a single token — benchmark through
`/v1/chat/completions`.

### Speculative decoding tuning

`--spec-mtp-strict-qwen` requires `-np 1`; the server refuses to start otherwise.

| Config | Decode | vs baseline |
|---|---|---|
| no speculation | 61.9 t/s | 1.00× |
| `n_max=4 p_min=0.75` (default p_min) | 77.2 t/s | 1.25× |
| `n_max=4 p_min=0.35` | 89.9 t/s | 1.45× |
| **`n_max=4 p_min=0.2`** | **96.6 t/s** | **1.56×** |
| `n_max=4 p_min=0.1` | 96.1 t/s | 1.55× |
| `n_max=4 p_min=0.1` **+ strict** | 95.5 t/s | 1.54× |
| `n_max=5 p_min=0.1` | 89.3 t/s | 1.44× |
| `n_max=6 p_min=0.1` | 80.0 t/s | 1.29× |
| `n_max=8 p_min=0.1` | 52.0 t/s | **0.84×** |

Two things worth internalizing:

1. **`p_min` matters far more than `n_max`.** The default `p_min=0.75` leaves ~20 t/s on
   the table. Drafting aggressively (lower `p_min`) wins even though per-draft acceptance
   drops.
2. **Long drafts are actively harmful on this MoE.** `n_max=8` falls *below* the
   no-speculation baseline despite a higher mean acceptance length (4.12), because each
   extra draft position widens the union of experts that must be gathered. Per-position
   acceptance is `(0.982, 0.600, 0.327, 0.145)` — the 4th token is rarely worth it.

**Strict verification is nearly free** (95.5 vs 96.1 t/s, ~0.5%) and guarantees output
identical to non-speculative greedy decoding. Leave it on.

### Reproducibility

A clean build of `main` was benchmarked against the development build on the same
model and flags, and they are indistinguishable — pp512 1252 vs 1256 t/s, tg128
65.11 vs 65.09 t/s, well inside run-to-run noise. **The public repo delivers the full
ROCmFP2 performance**; nothing here depends on unpublished work. See the variance note
below before drawing conclusions from small differences.

### Throughput

| Test | Vulkan0 | ROCm0 |
|---|---|---|
| pp512 | 1251 t/s | 1268 t/s |
| tg128 | **63.1 t/s** | 46.4 t/s |
| pp2048 | 1218 t/s | — |
| pp8192 | 1153 t/s | — |
| pp32768 | 917 t/s | — |

**Use Vulkan for decode** — it beats ROCm by 36% (63.1 vs 46.4 t/s). ROCm is marginally
ahead on prompt processing, within noise. Flash-attn on/off and ubatch 1024 changed
nothing measurable at short context.

### Long context

Decode degrades roughly linearly with depth, since ~10 of 40 layers are full attention
(`full_attention_interval: 4`) while the rest are linear-attention:

| Depth | Decode |
|---|---|
| 0 | 63.5 t/s |
| 16k | 56.1 t/s |
| 65k | 44.3 t/s |
| 131k | 34.8 t/s |
| 256k | 24.5 t/s |

KV cache at 256k with f16 is ~6 GB; total resident ~21 GB.

> **Benchmarking caveat.** Run-to-run variance on this platform is ±7 t/s through the
> chat API — identical configs produced 81, 85 and 96 t/s on consecutive runs. Warm up
> and average at least 3 runs before trusting any comparison. Differences between f16 and
> q4_0 KV, and between 32k/64k/131k context, were **within noise** in our testing.

---

## Quality

Verified working: coherent prose, chain-of-thought reasoning, Python code generation, and
**structured tool calling** (correct `tool_calls` with valid JSON arguments and
`finish_reason: tool_calls`).

Reconstruction fidelity was validated against the original `Qwen/Qwen3.6-35B-A3B` BF16
weights before quantization — decoded experts correlate **0.953** (gate_up, 2-bit source)
and **0.990** (down_proj, 3-bit source) with the base model.

No standardized benchmark scores (MMLU, BFCL, etc.) have been run on this transcode yet.
Escha reports boolq 88.38 vs 88.04 for their original quantization; expect some additional
loss here, since repacking to 2.5 bpw adds error on top of Escha's own 2-bit.

## Provenance

`EschaLabs/Qwen3.6-35B-A3B-Escha-W2` is a quantization of `Qwen/Qwen3.6-35B-A3B`. This is
verifiable rather than assumed: layer-0 expert-0's `gate_up` has 560 of 1024 all-zero rows
in the base model, and Escha's `escha_rout` has zeros at exactly those same 560 positions.

The transcode decodes Escha's packed codes, applies the documented reconstruction chain
`W_eff = diag(rin) · H · W_bare · H · diag(rout)` (where `H` is the normalized 128-wide
block Hadamard), then repacks to ROCmFPX. Escha's `s_in`/`s_out` scales are all-ones in
this export (already folded).

## License and attribution

Apache-2.0, inherited from EschaLabs' release ("model weights only, released under the
Apache License, Version 2.0"). Credit to:

- **EschaLabs** — the W2 quantization this is derived from
- **Qwen** — the `Qwen3.6-35B-A3B` base model
- **charlie12345/ROCmFPX** — the fork and quantization formats
- **exllamav3 / turboderp** — the trellis codebook design the Escha format builds on