Instructions to use cafonez/Escha-W2-35B-A3B-ROCmFP2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use cafonez/Escha-W2-35B-A3B-ROCmFP2 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf cafonez/Escha-W2-35B-A3B-ROCmFP2
# Run inference directly in the terminal:
llama cli -hf cafonez/Escha-W2-35B-A3B-ROCmFP2

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf cafonez/Escha-W2-35B-A3B-ROCmFP2
# Run inference directly in the terminal:
llama cli -hf cafonez/Escha-W2-35B-A3B-ROCmFP2

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf cafonez/Escha-W2-35B-A3B-ROCmFP2
# Run inference directly in the terminal:
./llama-cli -hf cafonez/Escha-W2-35B-A3B-ROCmFP2

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf cafonez/Escha-W2-35B-A3B-ROCmFP2
# Run inference directly in the terminal:
./build/bin/llama-cli -hf cafonez/Escha-W2-35B-A3B-ROCmFP2

Use Docker

docker model run hf.co/cafonez/Escha-W2-35B-A3B-ROCmFP2

LM Studio
Jan

vLLM

How to use cafonez/Escha-W2-35B-A3B-ROCmFP2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "cafonez/Escha-W2-35B-A3B-ROCmFP2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cafonez/Escha-W2-35B-A3B-ROCmFP2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/cafonez/Escha-W2-35B-A3B-ROCmFP2

Ollama
How to use cafonez/Escha-W2-35B-A3B-ROCmFP2 with Ollama:
```
ollama run hf.co/cafonez/Escha-W2-35B-A3B-ROCmFP2
```

Unsloth Studio

How to use cafonez/Escha-W2-35B-A3B-ROCmFP2 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for cafonez/Escha-W2-35B-A3B-ROCmFP2 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for cafonez/Escha-W2-35B-A3B-ROCmFP2 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for cafonez/Escha-W2-35B-A3B-ROCmFP2 to start chatting

How to use cafonez/Escha-W2-35B-A3B-ROCmFP2 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf cafonez/Escha-W2-35B-A3B-ROCmFP2

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "cafonez/Escha-W2-35B-A3B-ROCmFP2"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use cafonez/Escha-W2-35B-A3B-ROCmFP2 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf cafonez/Escha-W2-35B-A3B-ROCmFP2

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "cafonez/Escha-W2-35B-A3B-ROCmFP2" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use cafonez/Escha-W2-35B-A3B-ROCmFP2 with Docker Model Runner:
```
docker model run hf.co/cafonez/Escha-W2-35B-A3B-ROCmFP2
```

Lemonade

How to use cafonez/Escha-W2-35B-A3B-ROCmFP2 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull cafonez/Escha-W2-35B-A3B-ROCmFP2

Run and chat with the model

lemonade run user.Escha-W2-35B-A3B-ROCmFP2-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use cafonez/Escha-W2-35B-A3B-ROCmFP2 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf cafonez/Escha-W2-35B-A3B-ROCmFP2

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default cafonez/Escha-W2-35B-A3B-ROCmFP2

Run Hermes

hermes

Atomic Chat

Escha-W2-35B-A3B-ROCmFP2 / README.md

cafonez

Confirm reproducibility: clean main build matches dev build performance

eb73f4d verified about 10 hours ago

preview code

Raw

History Blame Contribute Delete

8.91 kB

	---
	license: apache-2.0
	base_model:
	- EschaLabs/Qwen3.6-35B-A3B-Escha-W2
	- Qwen/Qwen3.6-35B-A3B
	tags:
	- gguf
	- rocmfpx
	- rocmfp2
	- qwen35moe
	- strix-halo
	- amd
	- speculative-decoding
	- mtp
	pipeline_tag: text-generation
	---

	# Qwen3.6-35B-A3B-Escha-W2 → ROCmFP2 (GGUF)

	A GGUF transcode of [EschaLabs/Qwen3.6-35B-A3B-Escha-W2](https://huggingface.co/EschaLabs/Qwen3.6-35B-A3B-Escha-W2)
	into the ROCmFPX quantization formats, for AMD Strix Halo / gfx1151 and other
	ROCm/Vulkan targets.

	Escha's original export is a proprietary `eschamoe` packed format that only ships a
	CUDA decoder. This repo contains the same expert weights decoded and repacked into
	`Q2_0_ROCMFPX`, so the model runs on AMD hardware through the ROCmFPX llama.cpp fork.

	> **This file will not load in upstream llama.cpp, Ollama, LM Studio, koboldcpp or
	> llama-cpp-python.** It uses ggml tensor type IDs `107` (`Q2_0_ROCMFPX`) and `103`
	> (`Q8_0_ROCMFPX`), which are outside upstream's range (`GGML_TYPE_COUNT = 43`).
	> You need the fork below. The `qwen35moe` architecture itself is upstream — only
	> the quantization types are fork-specific.

	---

	## Requirements

	Build [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) at commit
	`41db2f9eb` or later:

	```bash
	git clone https://github.com/charlie12345/ROCmFPX
	cd ROCmFPX
	cmake -B build -DGGML_VULKAN=ON -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151
	cmake --build build -j
	```

	The model itself loads on any build from `db6844d9b` (build 172) onward — the
	quantization types, the `qwen35moe` architecture and `draft-mtp` are all present there.
	Only `--spec-mtp-strict-qwen` requires `41db2f9eb`+.

	Pin the commit if you can. The ROCmFPX type IDs live in a private-use range and could
	renumber between fork revisions, which would silently mismatch an older build.

	## Contents

	\| \| \|
	\|---\|---\|
	\| File \| `Qwen3.6-35B-A3B-Escha-W2-ROCmFP2.gguf` \|
	\| Size \| 13.10 GB (12.2 GiB) \|
	\| Architecture \| `qwen35moe` (35.5B total, ~3B active, 256 experts, hybrid linear+full attention) \|
	\| Native context \| 262144 \|
	\| Routed experts \| `Q2_0_ROCMFPX` (2.5 bpw) \|
	\| Dense / attention / shared experts \| `Q8_0_ROCMFPX` \|
	\| Norms, `ssm_conv1d` \| `F32` \|
	\| MTP head experts (`blk.40`) \| `Q4_0` — see note below \|

	### About the MTP head

	Escha's export ships the `nextn` (MTP) block without its routed experts, so a GGUF
	built from it alone fails to load with `missing tensor 'blk.40.ffn_down_exps.weight'`.
	Those two tensors were grafted from the base `Qwen/Qwen3.6-35B-A3B` at `Q4_0`.

	They are not Escha weights. This only affects the draft model — every speculative
	token is verified against the target model before being accepted, so drafting from
	base-model weights cannot introduce wrong tokens — and with `--spec-mtp-strict-qwen` the
	output is bit-identical to non-speculative greedy decoding. The graft costs 453 MB and
	buys a ~1.5× decode speedup.

	If you don't want it, strip block 40 and set `qwen35moe.block_count=40`,
	`qwen35moe.nextn_predict_layers=0` — that yields a 12.65 GB file (+2.6% over Escha's
	original, versus +6.3% with the head).

	---

	## Recommended flags

	Measured on Strix Halo (Radeon 8060S, gfx1151), Vulkan backend.

	```bash
	llama-server \
	-m Qwen3.6-35B-A3B-Escha-W2-ROCmFP2.gguf \
	-np 1 \
	-dev Vulkan0 --spec-draft-device Vulkan0 \
	-ngl 999 --spec-draft-ngl all \
	-fa on --no-mmap \
	-ctk f16 -ctv f16 \
	-c 131072 -b 2048 -ub 512 \
	--jinja \
	--spec-type draft-mtp \
	--spec-draft-n-max 4 \
	--spec-draft-n-min 0 \
	--spec-draft-p-min 0.15 \
	--spec-mtp-strict-qwen
	```

	> `--spec-mtp-strict-qwen` needs `41db2f9eb` or newer (added in
	> [PR #54](https://github.com/charlie12345/ROCmFPX/pull/54)). Older binaries reject the
	> flag — if yours does, either update or drop that line. Everything else works without
	> it and the speed figures are unaffected, since strict costs only ~0.5%. What you lose
	> is the guarantee that speculative output is bit-identical to non-speculative greedy
	> decoding; drafts are verified against the target model either way, so output remains
	> valid.

	`--spec-type` is a llama-server flag; `llama-completion` rejects it. Note also that
	with `--jinja`, the raw `/completion` endpoint returns a single token — benchmark through
	`/v1/chat/completions`.

	### Speculative decoding tuning

	`--spec-mtp-strict-qwen` requires `-np 1`; the server refuses to start otherwise.

	\| Config \| Decode \| vs baseline \|
	\|---\|---\|---\|
	\| no speculation \| 61.9 t/s \| 1.00× \|
	\| `n_max=4 p_min=0.75` (default p_min) \| 77.2 t/s \| 1.25× \|
	\| `n_max=4 p_min=0.35` \| 89.9 t/s \| 1.45× \|
	\| `n_max=4 p_min=0.2` \| 96.6 t/s \| 1.56× \|
	\| `n_max=4 p_min=0.1` \| 96.1 t/s \| 1.55× \|
	\| `n_max=4 p_min=0.1` + strict \| 95.5 t/s \| 1.54× \|
	\| `n_max=5 p_min=0.1` \| 89.3 t/s \| 1.44× \|
	\| `n_max=6 p_min=0.1` \| 80.0 t/s \| 1.29× \|
	\| `n_max=8 p_min=0.1` \| 52.0 t/s \| 0.84× \|

	Two things worth internalizing:

	1. `p_min` matters far more than `n_max`. The default `p_min=0.75` leaves ~20 t/s on
	the table. Drafting aggressively (lower `p_min`) wins even though per-draft acceptance
	drops.
	2. Long drafts are actively harmful on this MoE. `n_max=8` falls below the
	no-speculation baseline despite a higher mean acceptance length (4.12), because each
	extra draft position widens the union of experts that must be gathered. Per-position
	acceptance is `(0.982, 0.600, 0.327, 0.145)` — the 4th token is rarely worth it.

	Strict verification is nearly free (95.5 vs 96.1 t/s, ~0.5%) and guarantees output
	identical to non-speculative greedy decoding. Leave it on.

	### Reproducibility

	A clean build of `main` was benchmarked against the development build on the same
	model and flags, and they are indistinguishable — pp512 1252 vs 1256 t/s, tg128
	65.11 vs 65.09 t/s, well inside run-to-run noise. **The public repo delivers the full
	ROCmFP2 performance**; nothing here depends on unpublished work. See the variance note
	below before drawing conclusions from small differences.

	### Throughput

	\| Test \| Vulkan0 \| ROCm0 \|
	\|---\|---\|---\|
	\| pp512 \| 1251 t/s \| 1268 t/s \|
	\| tg128 \| 63.1 t/s \| 46.4 t/s \|
	\| pp2048 \| 1218 t/s \| — \|
	\| pp8192 \| 1153 t/s \| — \|
	\| pp32768 \| 917 t/s \| — \|

	Use Vulkan for decode — it beats ROCm by 36% (63.1 vs 46.4 t/s). ROCm is marginally
	ahead on prompt processing, within noise. Flash-attn on/off and ubatch 1024 changed
	nothing measurable at short context.

	### Long context

	Decode degrades roughly linearly with depth, since ~10 of 40 layers are full attention
	(`full_attention_interval: 4`) while the rest are linear-attention:

	\| Depth \| Decode \|
	\|---\|---\|
	\| 0 \| 63.5 t/s \|
	\| 16k \| 56.1 t/s \|
	\| 65k \| 44.3 t/s \|
	\| 131k \| 34.8 t/s \|
	\| 256k \| 24.5 t/s \|

	KV cache at 256k with f16 is ~6 GB; total resident ~21 GB.

	> Benchmarking caveat. Run-to-run variance on this platform is ±7 t/s through the
	> chat API — identical configs produced 81, 85 and 96 t/s on consecutive runs. Warm up
	> and average at least 3 runs before trusting any comparison. Differences between f16 and
	> q4_0 KV, and between 32k/64k/131k context, were within noise in our testing.

	---

	## Quality

	Verified working: coherent prose, chain-of-thought reasoning, Python code generation, and
	structured tool calling (correct `tool_calls` with valid JSON arguments and
	`finish_reason: tool_calls`).

	Reconstruction fidelity was validated against the original `Qwen/Qwen3.6-35B-A3B` BF16
	weights before quantization — decoded experts correlate 0.953 (gate_up, 2-bit source)
	and 0.990 (down_proj, 3-bit source) with the base model.

	No standardized benchmark scores (MMLU, BFCL, etc.) have been run on this transcode yet.
	Escha reports boolq 88.38 vs 88.04 for their original quantization; expect some additional
	loss here, since repacking to 2.5 bpw adds error on top of Escha's own 2-bit.

	## Provenance

	`EschaLabs/Qwen3.6-35B-A3B-Escha-W2` is a quantization of `Qwen/Qwen3.6-35B-A3B`. This is
	verifiable rather than assumed: layer-0 expert-0's `gate_up` has 560 of 1024 all-zero rows
	in the base model, and Escha's `escha_rout` has zeros at exactly those same 560 positions.

	The transcode decodes Escha's packed codes, applies the documented reconstruction chain
	`W_eff = diag(rin) · H · W_bare · H · diag(rout)` (where `H` is the normalized 128-wide
	block Hadamard), then repacks to ROCmFPX. Escha's `s_in`/`s_out` scales are all-ones in
	this export (already folded).

	## License and attribution

	Apache-2.0, inherited from EschaLabs' release ("model weights only, released under the
	Apache License, Version 2.0"). Credit to:

	- EschaLabs — the W2 quantization this is derived from
	- Qwen — the `Qwen3.6-35B-A3B` base model
	- charlie12345/ROCmFPX — the fork and quantization formats
	- exllamav3 / turboderp — the trellis codebook design the Escha format builds on