README.md

metadata

license: apache-2.0
base_model:
  - EschaLabs/Qwen3.6-35B-A3B-Escha-W2
  - Qwen/Qwen3.6-35B-A3B
tags:
  - gguf
  - rocmfpx
  - rocmfp2
  - qwen35moe
  - strix-halo
  - amd
  - speculative-decoding
  - mtp
pipeline_tag: text-generation

Qwen3.6-35B-A3B-Escha-W2 → ROCmFP2 (GGUF)

A GGUF transcode of EschaLabs/Qwen3.6-35B-A3B-Escha-W2 into the ROCmFPX quantization formats, for AMD Strix Halo / gfx1151 and other ROCm/Vulkan targets.

Escha's original export is a proprietary eschamoe packed format that only ships a CUDA decoder. This repo contains the same expert weights decoded and repacked into Q2_0_ROCMFPX, so the model runs on AMD hardware through the ROCmFPX llama.cpp fork.

This file will not load in upstream llama.cpp, Ollama, LM Studio, koboldcpp or llama-cpp-python. It uses ggml tensor type IDs 107 (Q2_0_ROCMFPX) and 103 (Q8_0_ROCMFPX), which are outside upstream's range (GGML_TYPE_COUNT = 43). You need the fork below. The qwen35moe architecture itself is upstream — only the quantization types are fork-specific.

---

Requirements

Build charlie12345/ROCmFPX at commit 41db2f9eb or later:

git clone https://github.com/charlie12345/ROCmFPX
cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151
cmake --build build -j

The model itself loads on any build from db6844d9b (build 172) onward — the quantization types, the qwen35moe architecture and draft-mtp are all present there. Only --spec-mtp-strict-qwen requires 41db2f9eb+.

Pin the commit if you can. The ROCmFPX type IDs live in a private-use range and could renumber between fork revisions, which would silently mismatch an older build.

Contents

File	Qwen3.6-35B-A3B-Escha-W2-ROCmFP2.gguf
Size	13.10 GB (12.2 GiB)
Architecture	qwen35moe (35.5B total, ~3B active, 256 experts, hybrid linear+full attention)
Native context	262144
Routed experts	Q2_0_ROCMFPX (2.5 bpw)
Dense / attention / shared experts	Q8_0_ROCMFPX
Norms, ssm_conv1d	F32
MTP head experts (blk.40)	Q4_0 — see note below

About the MTP head

Escha's export ships the nextn (MTP) block without its routed experts, so a GGUF built from it alone fails to load with missing tensor 'blk.40.ffn_down_exps.weight'. Those two tensors were grafted from the base Qwen/Qwen3.6-35B-A3B at Q4_0.

They are not Escha weights. This only affects the draft model — every speculative token is verified against the target model before being accepted, so drafting from base-model weights cannot introduce wrong tokens — and with --spec-mtp-strict-qwen the output is bit-identical to non-speculative greedy decoding. The graft costs 453 MB and buys a ~1.5× decode speedup.

If you don't want it, strip block 40 and set qwen35moe.block_count=40, qwen35moe.nextn_predict_layers=0 — that yields a 12.65 GB file (+2.6% over Escha's original, versus +6.3% with the head).

---

Recommended flags

Measured on Strix Halo (Radeon 8060S, gfx1151), Vulkan backend.

llama-server \
  -m Qwen3.6-35B-A3B-Escha-W2-ROCmFP2.gguf \
  -np 1 \
  -dev Vulkan0 --spec-draft-device Vulkan0 \
  -ngl 999 --spec-draft-ngl all \
  -fa on --no-mmap \
  -ctk f16 -ctv f16 \
  -c 131072 -b 2048 -ub 512 \
  --jinja \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --spec-draft-n-min 0 \
  --spec-draft-p-min 0.15 \
  --spec-mtp-strict-qwen

--spec-mtp-strict-qwen needs 41db2f9eb or newer (added in PR #54). Older binaries reject the flag — if yours does, either update or drop that line. Everything else works without it and the speed figures are unaffected, since strict costs only ~0.5%. What you lose is the guarantee that speculative output is bit-identical to non-speculative greedy decoding; drafts are verified against the target model either way, so output remains valid.

--spec-type is a llama-server flag; llama-completion rejects it. Note also that with --jinja, the raw /completion endpoint returns a single token — benchmark through /v1/chat/completions.

Speculative decoding tuning

--spec-mtp-strict-qwen requires -np 1; the server refuses to start otherwise.

Config	Decode	vs baseline
no speculation	61.9 t/s	1.00×
n_max=4 p_min=0.75 (default p_min)	77.2 t/s	1.25×
n_max=4 p_min=0.35	89.9 t/s	1.45×
n_max=4 p_min=0.2	96.6 t/s	1.56×
n_max=4 p_min=0.1	96.1 t/s	1.55×
n_max=4 p_min=0.1 + strict	95.5 t/s	1.54×
n_max=5 p_min=0.1	89.3 t/s	1.44×
n_max=6 p_min=0.1	80.0 t/s	1.29×
n_max=8 p_min=0.1	52.0 t/s	0.84×

Two things worth internalizing:

1. p_min matters far more than n_max. The default p_min=0.75 leaves ~20 t/s on the table. Drafting aggressively (lower p_min) wins even though per-draft acceptance drops.
2. Long drafts are actively harmful on this MoE. n_max=8 falls below the no-speculation baseline despite a higher mean acceptance length (4.12), because each extra draft position widens the union of experts that must be gathered. Per-position acceptance is (0.982, 0.600, 0.327, 0.145) — the 4th token is rarely worth it.

Strict verification is nearly free (95.5 vs 96.1 t/s, ~0.5%) and guarantees output identical to non-speculative greedy decoding. Leave it on.

Reproducibility

A clean build of main was benchmarked against the development build on the same model and flags, and they are indistinguishable — pp512 1252 vs 1256 t/s, tg128 65.11 vs 65.09 t/s, well inside run-to-run noise. The public repo delivers the full ROCmFP2 performance; nothing here depends on unpublished work. See the variance note below before drawing conclusions from small differences.

Throughput

Test	Vulkan0	ROCm0
pp512	1251 t/s	1268 t/s
tg128	63.1 t/s	46.4 t/s
pp2048	1218 t/s	—
pp8192	1153 t/s	—
pp32768	917 t/s	—

Use Vulkan for decode — it beats ROCm by 36% (63.1 vs 46.4 t/s). ROCm is marginally ahead on prompt processing, within noise. Flash-attn on/off and ubatch 1024 changed nothing measurable at short context.

Long context

Decode degrades roughly linearly with depth, since ~10 of 40 layers are full attention (full_attention_interval: 4) while the rest are linear-attention:

Depth	Decode
0	63.5 t/s
16k	56.1 t/s
65k	44.3 t/s
131k	34.8 t/s
256k	24.5 t/s

KV cache at 256k with f16 is ~6 GB; total resident ~21 GB.

Benchmarking caveat. Run-to-run variance on this platform is ±7 t/s through the chat API — identical configs produced 81, 85 and 96 t/s on consecutive runs. Warm up and average at least 3 runs before trusting any comparison. Differences between f16 and q4_0 KV, and between 32k/64k/131k context, were within noise in our testing.

---

Quality

Verified working: coherent prose, chain-of-thought reasoning, Python code generation, and structured tool calling (correct tool_calls with valid JSON arguments and finish_reason: tool_calls).

Reconstruction fidelity was validated against the original Qwen/Qwen3.6-35B-A3B BF16 weights before quantization — decoded experts correlate 0.953 (gate_up, 2-bit source) and 0.990 (down_proj, 3-bit source) with the base model.

No standardized benchmark scores (MMLU, BFCL, etc.) have been run on this transcode yet. Escha reports boolq 88.38 vs 88.04 for their original quantization; expect some additional loss here, since repacking to 2.5 bpw adds error on top of Escha's own 2-bit.

Provenance

EschaLabs/Qwen3.6-35B-A3B-Escha-W2 is a quantization of Qwen/Qwen3.6-35B-A3B. This is verifiable rather than assumed: layer-0 expert-0's gate_up has 560 of 1024 all-zero rows in the base model, and Escha's escha_rout has zeros at exactly those same 560 positions.

The transcode decodes Escha's packed codes, applies the documented reconstruction chain W_eff = diag(rin) · H · W_bare · H · diag(rout) (where H is the normalized 128-wide block Hadamard), then repacks to ROCmFPX. Escha's s_in/s_out scales are all-ones in this export (already folded).

License and attribution

Apache-2.0, inherited from EschaLabs' release ("model weights only, released under the Apache License, Version 2.0"). Credit to:

• EschaLabs — the W2 quantization this is derived from
• Qwen — the Qwen3.6-35B-A3B base model
• charlie12345/ROCmFPX — the fork and quantization formats
• exllamav3 / turboderp — the trellis codebook design the Escha format builds on