Instructions to use SixVolts/Qwen3.5-397B-A17B-doublestuff with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use SixVolts/Qwen3.5-397B-A17B-doublestuff with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SixVolts/Qwen3.5-397B-A17B-doublestuff
# Run inference directly in the terminal:
llama cli -hf SixVolts/Qwen3.5-397B-A17B-doublestuff

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SixVolts/Qwen3.5-397B-A17B-doublestuff
# Run inference directly in the terminal:
llama cli -hf SixVolts/Qwen3.5-397B-A17B-doublestuff

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf SixVolts/Qwen3.5-397B-A17B-doublestuff
# Run inference directly in the terminal:
./llama-cli -hf SixVolts/Qwen3.5-397B-A17B-doublestuff

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf SixVolts/Qwen3.5-397B-A17B-doublestuff
# Run inference directly in the terminal:
./build/bin/llama-cli -hf SixVolts/Qwen3.5-397B-A17B-doublestuff

Use Docker

docker model run hf.co/SixVolts/Qwen3.5-397B-A17B-doublestuff

LM Studio
Jan

vLLM

How to use SixVolts/Qwen3.5-397B-A17B-doublestuff with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SixVolts/Qwen3.5-397B-A17B-doublestuff"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SixVolts/Qwen3.5-397B-A17B-doublestuff",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/SixVolts/Qwen3.5-397B-A17B-doublestuff

Ollama
How to use SixVolts/Qwen3.5-397B-A17B-doublestuff with Ollama:
```
ollama run hf.co/SixVolts/Qwen3.5-397B-A17B-doublestuff
```

Unsloth Studio

How to use SixVolts/Qwen3.5-397B-A17B-doublestuff with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for SixVolts/Qwen3.5-397B-A17B-doublestuff to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for SixVolts/Qwen3.5-397B-A17B-doublestuff to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for SixVolts/Qwen3.5-397B-A17B-doublestuff to start chatting

How to use SixVolts/Qwen3.5-397B-A17B-doublestuff with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SixVolts/Qwen3.5-397B-A17B-doublestuff

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "SixVolts/Qwen3.5-397B-A17B-doublestuff"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use SixVolts/Qwen3.5-397B-A17B-doublestuff with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SixVolts/Qwen3.5-397B-A17B-doublestuff

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default SixVolts/Qwen3.5-397B-A17B-doublestuff

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use SixVolts/Qwen3.5-397B-A17B-doublestuff with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SixVolts/Qwen3.5-397B-A17B-doublestuff

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "SixVolts/Qwen3.5-397B-A17B-doublestuff" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use SixVolts/Qwen3.5-397B-A17B-doublestuff with Docker Model Runner:
```
docker model run hf.co/SixVolts/Qwen3.5-397B-A17B-doublestuff
```

Lemonade

How to use SixVolts/Qwen3.5-397B-A17B-doublestuff with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull SixVolts/Qwen3.5-397B-A17B-doublestuff

Run and chat with the model

lemonade run user.Qwen3.5-397B-A17B-doublestuff-{{QUANT_TAG}}

List all available models

lemonade list

Qwen3.5-397B-A17B-doublestuff

Double-stuffed: a 198 GiB custom quant sized so that two full 256K-token q8_0-KV contexts fit alongside the weights in 256 GB of VRAM — verified on 8×32 GB AMD MI100.

A custom GGUF quantization of Qwen3.5-397B-A17B (MTP), built from unsloth/Qwen3.5-397B-A17B-MTP-GGUF BF16 shards using Unsloth's calibration imatrix. All credit for the BF16 conversion, the imatrix, and the calibration set goes to the Unsloth team — the only contribution here is where the bits go.

Why this exists

At 198 GiB it fills the gap in the standard lineup between UD-Q3_K_XL (170 GiB) and UD-Q4_K_S (217 GiB) — a size class Unsloth doesn't ship — chosen specifically for the two-concurrent-256K-session serving config in 256 GB. Every routed-expert tensor's bit-width is graded by imatrix activation importance (usage-weighted mean E[x²] per token, median-normalized across gate/up/down) rather than by layer position:

the least-important layers go to q3_K,
the highest-importance tail is promoted to q5_K,
ffn_down (which feeds the residual stream) never drops below q4_K.

The imatrix says this model has no U-shape: importance spikes at blk.0–1, floors through blk.2–24, then rises monotonically ~350× toward blk.59. Grading by that measured curve — not the "middle layers matter least" folklore — is what earns the size class.

Perplexity (wikitext-2, 300 chunks @ 512, llama.cpp, one rig)

quant	size	PPL
unsloth UD-Q4_K_XL	232.4 GiB	3.5417 ± 0.0265
unsloth UD-Q4_K_S	217.0 GiB	3.5471 ± 0.0266
doublestuff (this repo)	198.4 GiB	3.5758 ± 0.0268
flat-q4 experts (ablation)	213.3 GiB	3.5790 ± 0.0269
imatrix q3-band only (ablation)	196.4 GiB	3.5809 ± 0.0269
unsloth UD-Q3_K_XL	170.3 GiB	3.6116 ± 0.0271

Honest guidance: if you have ≥217 GiB of VRAM, use Unsloth's UD-Q4_K_S — their dynamic allocation still wins at that budget. doublestuff is for max-context serving in 256 GB: it's the best PPL-per-byte in the set and beats the 15-GiB-larger flat-q4 build.

Why not just go smaller (vs UD-Q3_K_XL)?

UD-Q3_K_XL is 28 GiB smaller, but its entire routed-expert body is i-quant (IQ3_XXS on gate/up, IQ4_XS on down). doublestuff is pure K-quant, and on gfx908 (CDNA1) that wins on both axes:

	size	PPL	decode tg128	prefill pp512
doublestuff (K-quant)	198.4 GiB	3.5758	33.3 t/s	290.9 t/s
UD-Q3_K_XL (IQ)	170.3 GiB	3.6116	29.1 t/s	286.2 t/s

Despite moving 14% fewer bytes/token, UD-Q3_K_XL decodes 12% slower — IQ codebook dequant is weakly optimized on CDNA1. On a GPU with fast IQ kernels the tradeoff differs; on MI100-class hardware, K-quant wins. Its smaller footprint buys neither accuracy nor speed.

Running it (8×MI100, gfx908, 256 GB VRAM)

Common flags: -ngl 999 --split-mode layer -fa on --no-mmap, HIP_VISIBLE_DEVICES=0,2,3,4,5,7,8,9 (adjust to your cards).

Only 15 of 61 blocks (every 4th, Gated-Attention) grow KV with context; the other 45 are Gated-DeltaNet with constant state — which is why 256K slots are this cheap.

Two serve modes

# A) single-stream, latency-optimized (chat / agentic)
-ngl 999 -sm layer -fa on --no-mmap -np 1 -ub 2048 \
  --spec-type draft-mtp --spec-draft-n-max 2

# B) max-context, two concurrent 256K sessions
-ngl 999 -sm layer -fa on --no-mmap -np 2 -c 524288 -ub 1024 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --tensor-split 0.98,1.13,1.13,1.12,1.07,0.92,0.82,0.95

MTP self-speculation (--spec-type draft-mtp, the model's own blk.60 nextn head — no external draft model) speeds up single-stream decode, best at --spec-draft-n-max 2 (~+5% prose / +16% structured, lossless). n-max ≥ 3 loses to pipeline-verify cost, and plain n-gram speculation is a net loss on an all-GPU layer split. draft-mtp is -np 1 only, so it's mutually exclusive with mode B — pick latency or max-context.
The tensor-split in mode B matters. The q5 tail + output + the MTP block make the last cards heavy, so a naive contiguous split OOMs at 2×256K. The split above (measured under a real long prefill, so the compute buffer is fully accounted) flattens the peak to 30.5 GiB @ ub512 / 31.6 GiB @ ub1024 of ~32. Single 256K contexts fit on an even split.
Prefill micro-batch: the per-card compute buffer grows with -ub (~2.1 / 3.3 / 5.5 GiB at ub 512 / 1024 / 2048). In the double-slot config, -ub 2048 overflows the 256 GB aggregate — use -ub 1024 there (biggest prefill that fits both full slots) and reserve -ub 2048 for single-stream mode A, which has the headroom.

Recipe

Base ftype Q4_K_M; token_embd / output forced q6_K; routers kept F32; the MTP/nextn block (blk.60) at q4_K. Routed experts graded from the imatrix ranking:

ffn_(gate|up)_exps: q3_K on the 38 least-important layers (blk1–38), q5_K on blk53–58, else q4_K.
ffn_down_exps: q4_K floor everywhere, q5_K on blk50–59, q6_K on blk0.
ffn_(gate|up)_shexp (always-active shared expert) and attn_output: q6_K.

Full recipe file (recipe_v2.txt) and the importance tooling (compute_importance.py, gen_recipe_v2.py) are in this repo.

Reproduction

Requires llama.cpp ≥ b3ce5ce (earlier llama-quantize rejects MTP models with Bad layer 60). Quantize from Unsloth's BF16 GGUF shards — their imatrix matches their conversion's layout; a fresh convert_hf_to_gguf.py run will not.

python3 compute_importance.py imatrix_unsloth.gguf   # -> importance.json
python3 gen_recipe_v2.py 38 6 10                      # -> recipe_v2.txt
llama-quantize \
  --imatrix imatrix_unsloth.gguf \
  --token-embedding-type q6_K --output-tensor-type q6_K \
  --tensor-type-file recipe_v2.txt \
  Qwen3.5-397B-A17B-BF16-00001-of-00018.gguf \
  Qwen3.5-397B-A17B-doublestuff.gguf Q4_K_M $(nproc)

Validation

Beyond the PPL table above, this quant was checked for: 2×256K double-slot serving (per-card VRAM measured under real long-prefill load), a K-quant-vs-IQ decode head-to-head vs UD-Q3_K_XL, MTP/n-gram/prefill-batch tuning sweeps, and an imatrix-weighted per-tensor quant-error study (which confirmed the allocation is near-optimal for layer-level grading). Details in the repo's analysis/ notes.

License

Inherits the base model's license (Qwen3.5, Apache-2.0). Credit to Qwen for the model and to Unsloth for the BF16 conversion, calibration set, and imatrix.

Downloads last month: 530

GGUF

Model size

403B params

Architecture

qwen35moe

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for SixVolts/Qwen3.5-397B-A17B-doublestuff

Base model

Qwen/Qwen3.5-397B-A17B

Quantized

unsloth/Qwen3.5-397B-A17B-MTP-GGUF

Quantized

(1)

this model