README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.6-35B-A3B
tags:
  - qwen3
  - qwen3.6
  - moe
  - gguf
  - rocm
  - rocmfpx
  - amd
  - r9700
  - llama.cpp
  - imatrix
  - calibrated
language:
  - en
  - zh
pipeline_tag: text-generation
---

# Qwen3.6-35B-A3B — ROCmFPX sealed (AMD-calibrated)

Custom **ROCmFPX** GGUFs of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) for **AMD Radeon AI PRO R9700 (gfx1201)** / ROCm HIP.

Built and bench’d on **donherm**: dual R9700 32 GB, ROCm **7.2.4**, ROCmFPX HIP pin **`45bcff5`**, Hermes agent lab workflow.

| File | Type | Size |
|------|------|-----:|
| **`Qwen3.6-35B-A3B-Q4_0_ROCMFP4_COHERENT-imatrix.gguf`** | `Q4_0_ROCMFP4_COHERENT` + imatrix | **~19 GB** |
| **`Qwen3.6-35B-A3B-Q6_0_ROCMFPX_AGENT-imatrix.gguf`** | `Q6_0_ROCMFPX_AGENT` + imatrix | **~31 GB** |
| `Qwen3.6-35B-A3B-imatrix.gguf` | importance matrix | ~184 MB |

---

## ⚠️ Runtime requirement

**Not** stock GGUFs. Need **[ROCmFPX](https://github.com/charlie12345/ROCmFPX)** HIP llama.cpp (tested pin: [1337hero/ROCmFPX](https://github.com/1337hero/ROCmFPX) @ `45bcff5`).

Will **not** load in stock llama.cpp / Ollama / LM Studio / Vulkan-only builds.

```bash
export LD_LIBRARY_PATH=/path/to/ROCmFPX/build/bin:/opt/rocm/lib:$LD_LIBRARY_PATH
# Prefer discrete GPUs: --device ROCm0 / ROCm1 (exclude iGPU if present)
```

---

## Naming (format vs profile)

| Token | Meaning |
|-------|---------|
| **ROCMFP4 / ROCMFPX** | AMD block **format** (needs ROCmFPX runtime) |
| **COHERENT / AGENT** | **Recipe**: heavier emb/output or agent/tool-biased tensor routing |
| **-imatrix** | Quantized with importance matrix |

Examples:
- `Q4_0_ROCMFP4_COHERENT` = 4-bit ROCmFP4 + Q6_K embeddings + imatrix  
- `Q6_0_ROCMFPX_AGENT` = 6-bit ROCmFPX + agent/coherent routing + imatrix  

---

## How they were made

### Shared pipeline
1. **Source:** Unsloth **BF16** GGUF of Qwen3.6-35B-A3B (~66 GB).
2. BF16 full load for imatrix is **impossible** on 2×32 GB (+ host RAM limit).
3. Built temporary **Q8_0** from BF16 for activation collection only (deleted after).
4. **`llama-imatrix`** on dual R9700 (layer TP) over a mixed calib corpus:
   - code / systems prose  
   - agent / tool JSON  
   - math  
   - light multilingual  
   → `*-imatrix.gguf` (**511** entries, **24** chunks; MoE experts may be partially covered).
5. Quantized **from BF16** with `--imatrix` (not requant from Q4).

### Quants
| Output | Command ftype |
|--------|----------------|
| Q4 sealed | `Q4_0_ROCMFP4_COHERENT` + `--imatrix` |
| High Q6 | `Q6_0_ROCMFPX_AGENT` + `--imatrix` |

**Tooling:** Automated quant, A/B harness, and HF packaging via lab automation (account **`bakon3`**).

---

## Benchmarks (donherm lab)

### Hardware & software (all benches unless noted)
| Item | Value |
|------|--------|
| GPUs | 2× AMD Radeon AI PRO R9700 (gfx1201), 32 GB each |
| iGPU | present — excluded via `--device ROCm0` / `ROCm1` |
| ROCm | 7.2.4 @ `/opt/rocm` |
| Binary | ROCmFPX HIP build **`45bcff5`** (build 119) |
| Common flags | FA on, `--no-mmap`, q8_0 KV when server, batch **2048 / ubatch 512** |

### A) Official `llama-bench` — Q4 sealed vs Unsloth UD-Q4

**Topology:** single GPU `ROCm0`, `-ngl 99 -fa 1 -b 2048 -ub 512 -r 3`  
**Control:** Unsloth `Qwen3.6-35B-A3B-UD-Q4_K_XL` (~20.8 GiB)  
**Date:** 2026-07-16

| Test | **Q4 COHERENT+imatrix** | Unsloth UD-Q4 | Δ |
|------|------------------------:|--------------:|----:|
| tg128 | **93.4** | 79.5 | **+17.5%** |
| tg256 | **94.2** | 80.0 | **+17.7%** |
| tg512 | **94.2** | 80.1 | **+17.5%** |
| pp512 | 2708 | 2951 | −8.2% |
| pp8192 | 2310 | 2568 | −10.0% |

### B) Server A/B chat — Q4 sealed vs Unsloth (no ngram)

**Topology:** dual simultaneous — ROCm0=sealed Q4, ROCm1=Unsloth  
**Server:** `llama-server` `-c 32768` parallel 1, FA on, q8_0 KV, **no ngram**  
**Harness:** OpenAI-compatible `/v1/chat/completions` (Hermes Python suite)  
**Date:** 2026-07-16

| Test | Sealed tg | Unsloth tg | Δ |
|------|----------:|-----------:|----:|
| decode_128 | **87.7** | 74.9 | **+17.2%** |
| decode_256 | **88.6** | 74.6 | **+18.8%** |
| decode_512 | **88.4** | 74.5 | **+18.6%** |
| decode_1024 | **88.3** | 74.7 | **+18.3%** |
| parallel both GPUs 256 | **88.5** | 74.9 | **+18.3%** |

**Mean decode Δ: ~+18.2%**

#### Quality smokes (same server suite)
| Check | Sealed | Unsloth |
|-------|--------|---------|
| `17*19` → 323 | ✓ | ✓ |
| `123*45` → 5535 | ✓ | ✓ |
| JSON object (name/language/functions/tests_pass) | ✓ | ✓ |
| iterative `fib(n)` | ✓ | ✓ |
| bat/ball → $0.05 | ✓ | ✓ |

### C) Q6 AGENT dual-TP @ 256k (high-ctx path)

**Topology:** **one** model, layer TP `-sm layer -ts 1,1` on ROCm0+ROCm1  
**Why TP:** ~31 GB weights cannot run as two full copies on 2×32 GB  
**Server:** `-c 262144`, FA on, q8_0 KV, **ngram-mod 24/48/64**, parallel 1, batch 2048/512  
**Date:** 2026-07-17 · harness: same chat suite via Hermes

| Test | avg tg | notes |
|------|-------:|-------|
| decode_128 | ~107 | ngram bimodal (cold ~67, peak ~140) |
| decode_256 | ~102 | cold ~69 / peak ~128 |
| decode_512 | ~97 | cold ~69 / peak ~140 |
| decode_1024 | ~73 | less draft help |
| repetitive code 512 | ~226 | peak ~385 with ngram |
| steady open decode | **~68–70** | without draft hits |

Prefill (chat long-prompt path): ~166 → ~133 pp as prompt grows ~0.6k→20k tokens.

Quality: **all pass** (323, 5535, JSON, fib, $0.05).

---

## Launch recipes

```bash
BIN=/path/to/ROCmFPX/build/bin/llama-server
export LD_LIBRARY_PATH=$(dirname "$BIN"):/opt/rocm/lib:$LD_LIBRARY_PATH
Q4=Qwen3.6-35B-A3B-Q4_0_ROCMFP4_COHERENT-imatrix.gguf
Q6=Qwen3.6-35B-A3B-Q6_0_ROCMFPX_AGENT-imatrix.gguf
```

### Dual GPU — Q4 @ 256k native (one full copy per GPU)

```bash
"$BIN" -m "$Q4" --host 0.0.0.0 --port 8000 --device ROCm0 \
  -c 262144 -ngl 99 --parallel 3 --cont-batching --kv-unified \
  --flash-attn on --no-mmap --jinja \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --batch-size 2048 --ubatch-size 512 --cache-ram 4096 \
  --spec-type ngram-mod \
  --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --verbosity 3 &

"$BIN" -m "$Q4" --host 0.0.0.0 --port 8001 --device ROCm1 \
  -c 262144 -ngl 99 --parallel 3 --cont-batching --kv-unified \
  --flash-attn on --no-mmap --jinja \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --batch-size 2048 --ubatch-size 512 --cache-ram 4096 \
  --spec-type ngram-mod \
  --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --verbosity 3 &
```

### Dual GPU — Q4 @ 512k YaRN (one full copy per GPU)

Add:
```bash
-c 524288 --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 262144 \
  --override-kv qwen35moe.context_length=int:524288 --context-shift
```
Keep **batch 2048 / ubatch 512**. Flash-attn may OOM on very long multi-request sessions — use `--flash-attn off` or lower parallel if needed.

### Q6 AGENT — dual-TP 256k (single server)

```bash
"$BIN" -m "$Q6" --host 0.0.0.0 --port 8000 \
  --device ROCm0,ROCm1 -sm layer -ts 1,1 \
  -c 262144 -ngl 99 --parallel 1 \
  --flash-attn on --no-mmap --jinja \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --batch-size 2048 --ubatch-size 512 --kv-unified \
  --spec-type ngram-mod \
  --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 \
  --verbosity 3
```

---

## Download

```bash
# recommended Q4
hf download bakon3/Qwen3.6-35B-A3B-ROCMFP \
  Qwen3.6-35B-A3B-Q4_0_ROCMFP4_COHERENT-imatrix.gguf

# high Q6 agent
hf download bakon3/Qwen3.6-35B-A3B-ROCMFP \
  Qwen3.6-35B-A3B-Q6_0_ROCMFPX_AGENT-imatrix.gguf

# imatrix (repro)
hf download bakon3/Qwen3.6-35B-A3B-ROCMFP Qwen3.6-35B-A3B-imatrix.gguf
```

---

## SHA256

See `SHA256SUMS` in this repo.

---

## License / lineage

- Base: **Apache-2.0** — [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)
- BF16 GGUF: [unsloth/Qwen3.6-35B-A3B-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF)
- Runtime: [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) / R9700 pin work  
- Methodology cousin (dense 27B Q8): [1337Hero/Qwen3.6-27B-Q8_0-ROCMFPX-GGUF](https://huggingface.co/1337Hero/Qwen3.6-27B-Q8_0-ROCMFPX-GGUF)

Derivative research quants — not official Qwen/Unsloth releases.

## Changelog
- **2026-07-17:** Add `Q6_0_ROCMFPX_AGENT-imatrix`; recommended dual-GPU quant = Q4 COHERENT; full bench methodology; remove straight no-imatrix Q4.
- **2026-07-16:** Initial sealed Q4 COHERENT+imatrix + A/B vs Unsloth.