license: apache-2.0
base_model: Qwen/Qwen3.6-35B-A3B
tags:
- qwen3
- qwen3.6
- moe
- gguf
- rocm
- rocmfpx
- amd
- r9700
- llama.cpp
- imatrix
- calibrated
language:
- en
- zh
pipeline_tag: text-generation
Qwen3.6-35B-A3B — ROCmFPX sealed (AMD-calibrated)
Custom ROCmFPX GGUFs of Qwen/Qwen3.6-35B-A3B for AMD Radeon AI PRO R9700 (gfx1201) / ROCm HIP.
Built and bench’d on donherm: dual R9700 32 GB, ROCm 7.2.4, ROCmFPX HIP pin 45bcff5, Hermes agent lab workflow.
| File | Type | Size |
|---|---|---|
Qwen3.6-35B-A3B-Q4_0_ROCMFP4_COHERENT-imatrix.gguf |
Q4_0_ROCMFP4_COHERENT + imatrix |
~19 GB |
Qwen3.6-35B-A3B-Q6_0_ROCMFPX_AGENT-imatrix.gguf |
Q6_0_ROCMFPX_AGENT + imatrix |
~31 GB |
Qwen3.6-35B-A3B-imatrix.gguf |
importance matrix | ~184 MB |
⚠️ Runtime requirement
Not stock GGUFs. Need ROCmFPX HIP llama.cpp (tested pin: 1337hero/ROCmFPX @ 45bcff5).
Will not load in stock llama.cpp / Ollama / LM Studio / Vulkan-only builds.
export LD_LIBRARY_PATH=/path/to/ROCmFPX/build/bin:/opt/rocm/lib:$LD_LIBRARY_PATH
# Prefer discrete GPUs: --device ROCm0 / ROCm1 (exclude iGPU if present)
Naming (format vs profile)
| Token | Meaning |
|---|---|
| ROCMFP4 / ROCMFPX | AMD block format (needs ROCmFPX runtime) |
| COHERENT / AGENT | Recipe: heavier emb/output or agent/tool-biased tensor routing |
| -imatrix | Quantized with importance matrix |
Examples:
Q4_0_ROCMFP4_COHERENT= 4-bit ROCmFP4 + Q6_K embeddings + imatrixQ6_0_ROCMFPX_AGENT= 6-bit ROCmFPX + agent/coherent routing + imatrix
How they were made
Shared pipeline
- Source: Unsloth BF16 GGUF of Qwen3.6-35B-A3B (~66 GB).
- BF16 full load for imatrix is impossible on 2×32 GB (+ host RAM limit).
- Built temporary Q8_0 from BF16 for activation collection only (deleted after).
llama-imatrixon dual R9700 (layer TP) over a mixed calib corpus:- code / systems prose
- agent / tool JSON
- math
- light multilingual
→*-imatrix.gguf(511 entries, 24 chunks; MoE experts may be partially covered).
- Quantized from BF16 with
--imatrix(not requant from Q4).
Quants
| Output | Command ftype |
|---|---|
| Q4 sealed | Q4_0_ROCMFP4_COHERENT + --imatrix |
| High Q6 | Q6_0_ROCMFPX_AGENT + --imatrix |
Tooling: Automated quant, A/B harness, and HF packaging via lab automation (account bakon3).
Benchmarks (donherm lab)
Hardware & software (all benches unless noted)
| Item | Value |
|---|---|
| GPUs | 2× AMD Radeon AI PRO R9700 (gfx1201), 32 GB each |
| iGPU | present — excluded via --device ROCm0 / ROCm1 |
| ROCm | 7.2.4 @ /opt/rocm |
| Binary | ROCmFPX HIP build 45bcff5 (build 119) |
| Common flags | FA on, --no-mmap, q8_0 KV when server, batch 2048 / ubatch 512 |
A) Official llama-bench — Q4 sealed vs Unsloth UD-Q4
Topology: single GPU ROCm0, -ngl 99 -fa 1 -b 2048 -ub 512 -r 3
Control: Unsloth Qwen3.6-35B-A3B-UD-Q4_K_XL (~20.8 GiB)
Date: 2026-07-16
| Test | Q4 COHERENT+imatrix | Unsloth UD-Q4 | Δ |
|---|---|---|---|
| tg128 | 93.4 | 79.5 | +17.5% |
| tg256 | 94.2 | 80.0 | +17.7% |
| tg512 | 94.2 | 80.1 | +17.5% |
| pp512 | 2708 | 2951 | −8.2% |
| pp8192 | 2310 | 2568 | −10.0% |
B) Server A/B chat — Q4 sealed vs Unsloth (no ngram)
Topology: dual simultaneous — ROCm0=sealed Q4, ROCm1=Unsloth
Server: llama-server -c 32768 parallel 1, FA on, q8_0 KV, no ngram
Harness: OpenAI-compatible /v1/chat/completions (Hermes Python suite)
Date: 2026-07-16
| Test | Sealed tg | Unsloth tg | Δ |
|---|---|---|---|
| decode_128 | 87.7 | 74.9 | +17.2% |
| decode_256 | 88.6 | 74.6 | +18.8% |
| decode_512 | 88.4 | 74.5 | +18.6% |
| decode_1024 | 88.3 | 74.7 | +18.3% |
| parallel both GPUs 256 | 88.5 | 74.9 | +18.3% |
Mean decode Δ: ~+18.2%
Quality smokes (same server suite)
| Check | Sealed | Unsloth |
|---|---|---|
17*19 → 323 |
✓ | ✓ |
123*45 → 5535 |
✓ | ✓ |
| JSON object (name/language/functions/tests_pass) | ✓ | ✓ |
iterative fib(n) |
✓ | ✓ |
| bat/ball → $0.05 | ✓ | ✓ |
C) Q6 AGENT dual-TP @ 256k (high-ctx path)
Topology: one model, layer TP -sm layer -ts 1,1 on ROCm0+ROCm1
Why TP: ~31 GB weights cannot run as two full copies on 2×32 GB
Server: -c 262144, FA on, q8_0 KV, ngram-mod 24/48/64, parallel 1, batch 2048/512
Date: 2026-07-17 · harness: same chat suite via Hermes
| Test | avg tg | notes |
|---|---|---|
| decode_128 | ~107 | ngram bimodal (cold ~67, peak ~140) |
| decode_256 | ~102 | cold ~69 / peak ~128 |
| decode_512 | ~97 | cold ~69 / peak ~140 |
| decode_1024 | ~73 | less draft help |
| repetitive code 512 | ~226 | peak ~385 with ngram |
| steady open decode | ~68–70 | without draft hits |
Prefill (chat long-prompt path): ~166 → ~133 pp as prompt grows ~0.6k→20k tokens.
Quality: all pass (323, 5535, JSON, fib, $0.05).
Launch recipes
BIN=/path/to/ROCmFPX/build/bin/llama-server
export LD_LIBRARY_PATH=$(dirname "$BIN"):/opt/rocm/lib:$LD_LIBRARY_PATH
Q4=Qwen3.6-35B-A3B-Q4_0_ROCMFP4_COHERENT-imatrix.gguf
Q6=Qwen3.6-35B-A3B-Q6_0_ROCMFPX_AGENT-imatrix.gguf
Dual GPU — Q4 @ 256k native (one full copy per GPU)
"$BIN" -m "$Q4" --host 0.0.0.0 --port 8000 --device ROCm0 \
-c 262144 -ngl 99 --parallel 3 --cont-batching --kv-unified \
--flash-attn on --no-mmap --jinja \
--cache-type-k q8_0 --cache-type-v q8_0 \
--batch-size 2048 --ubatch-size 512 --cache-ram 4096 \
--spec-type ngram-mod \
--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --verbosity 3 &
"$BIN" -m "$Q4" --host 0.0.0.0 --port 8001 --device ROCm1 \
-c 262144 -ngl 99 --parallel 3 --cont-batching --kv-unified \
--flash-attn on --no-mmap --jinja \
--cache-type-k q8_0 --cache-type-v q8_0 \
--batch-size 2048 --ubatch-size 512 --cache-ram 4096 \
--spec-type ngram-mod \
--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --verbosity 3 &
Dual GPU — Q4 @ 512k YaRN (one full copy per GPU)
Add:
-c 524288 --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 262144 \
--override-kv qwen35moe.context_length=int:524288 --context-shift
Keep batch 2048 / ubatch 512. Flash-attn may OOM on very long multi-request sessions — use --flash-attn off or lower parallel if needed.
Q6 AGENT — dual-TP 256k (single server)
"$BIN" -m "$Q6" --host 0.0.0.0 --port 8000 \
--device ROCm0,ROCm1 -sm layer -ts 1,1 \
-c 262144 -ngl 99 --parallel 1 \
--flash-attn on --no-mmap --jinja \
--cache-type-k q8_0 --cache-type-v q8_0 \
--batch-size 2048 --ubatch-size 512 --kv-unified \
--spec-type ngram-mod \
--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
--verbosity 3
Download
# recommended Q4
hf download bakon3/Qwen3.6-35B-A3B-ROCMFP \
Qwen3.6-35B-A3B-Q4_0_ROCMFP4_COHERENT-imatrix.gguf
# high Q6 agent
hf download bakon3/Qwen3.6-35B-A3B-ROCMFP \
Qwen3.6-35B-A3B-Q6_0_ROCMFPX_AGENT-imatrix.gguf
# imatrix (repro)
hf download bakon3/Qwen3.6-35B-A3B-ROCMFP Qwen3.6-35B-A3B-imatrix.gguf
SHA256
See SHA256SUMS in this repo.
License / lineage
- Base: Apache-2.0 — Qwen/Qwen3.6-35B-A3B
- BF16 GGUF: unsloth/Qwen3.6-35B-A3B-GGUF
- Runtime: charlie12345/ROCmFPX / R9700 pin work
- Methodology cousin (dense 27B Q8): 1337Hero/Qwen3.6-27B-Q8_0-ROCMFPX-GGUF
Derivative research quants — not official Qwen/Unsloth releases.
Changelog
- 2026-07-17: Add
Q6_0_ROCMFPX_AGENT-imatrix; recommended dual-GPU quant = Q4 COHERENT; full bench methodology; remove straight no-imatrix Q4. - 2026-07-16: Initial sealed Q4 COHERENT+imatrix + A/B vs Unsloth.