CONFIGS.md

# bw24-bench — exact configs for v0.8.0 speculative-decoding board

Reproducibility configs for the speculative-decoding (ST) rows added in v0.8.0. These correspond to the README Performance section's **safetensors-path** models (Qwen3.5-9B ST and Qwen3.6-27B ST).

---

## Qwen3.5-9B NVFP4 ST (modelopt safetensors)

Model: `/data/ai-ml/hf-models/qwen35-9b-nvfp4-st-modelopt/` (modelopt NVFP4 safetensors checkpoint)

### Plain decode (no speculation)
```bash
# No flags — tuned path is the default
./target/release/run-gen /data/ai-ml/hf-models/qwen35-9b-nvfp4-st-modelopt/ --prompt "<prompt>"
```

### Speculative decode (MTP + FR-Spec trim)

**Short-code (p1):**
```bash
BW24_SPEC_K=2 \
BW24_SPEC_PMIN=0.3 \
BW24_FRSPEC_TRIM=trims/frspec-9bst-modelopt-32768.gguf \
./target/release/run-spec /data/ai-ml/hf-models/qwen35-9b-nvfp4-st-modelopt/ \
  --prompt "$(cat prompts/p1-code-short.txt)"
```

**Medium-code (p2) and Agentic-long (p3):**
```bash
BW24_SPEC_K=3 \
BW24_SPEC_PMIN=0.3 \
BW24_FRSPEC_TRIM=trims/frspec-9bst-modelopt-32768.gguf \
./target/release/run-spec /data/ai-ml/hf-models/qwen35-9b-nvfp4-st-modelopt/ \
  --prompt "$(cat prompts/p2-code-medium.txt)"  # or p3-agentic-long.txt
```

---

## Qwen3.6-27B NVFP4 ST (NVIDIA official safetensors)

Model: `/data/ai-ml/hf-models/nvidia-qwen36-27b-nvfp4/` (NVIDIA's official release)

### Plain decode (no speculation)
```bash
BW24_NV_W4=1 BW24_MMQ_F8F4=1 \
./target/release/run-gen /data/ai-ml/hf-models/nvidia-qwen36-27b-nvfp4/ --prompt "<prompt>"
```

### Speculative decode (MTP + FR-Spec trim + HPOST)

**Short-code (p1):**
```bash
BW24_NV_W4=1 BW24_MMQ_F8F4=1 \
BW24_SPEC_K=3 \
BW24_SPEC_HPOST=1 \
BW24_SPEC_PMIN=0.4 \
BW24_FRSPEC_TRIM=trims/frspec-corpus-32768.gguf \
./target/release/run-spec /data/ai-ml/hf-models/nvidia-qwen36-27b-nvfp4/ \
  --prompt "$(cat prompts/p1-code-short.txt)"
```

**Medium-code (p2):**
```bash
BW24_NV_W4=1 BW24_MMQ_F8F4=1 \
BW24_SPEC_K=3 \
BW24_SPEC_HPOST=1 \
BW24_SPEC_PMIN=0.3 \
BW24_FRSPEC_TRIM=trims/frspec-corpus-32768.gguf \
./target/release/run-spec /data/ai-ml/hf-models/nvidia-qwen36-27b-nvfp4/ \
  --prompt "$(cat prompts/p2-code-medium.txt)"
```

**Agentic-long (p3):**
```bash
BW24_NV_W4=1 BW24_MMQ_F8F4=1 \
BW24_SPEC_K=3 \
BW24_SPEC_HPOST=1 \
BW24_SPEC_PMIN=0.4 \
BW24_FRSPEC_TRIM=trims/frspec-corpus-32768.gguf \
./target/release/run-spec /data/ai-ml/hf-models/nvidia-qwen36-27b-nvfp4/ \
  --prompt "$(cat prompts/p3-agentic-long.txt)"
```

**IMPORTANT NOTE:** The p3-agentic-long column carries a degeneration caveat (see `rig5090.jsonl` tag `p3-degeneration-audit`): greedy continuations on this prompt partially degenerate into repetition on both engines symmetrically. The ratios between engines remain valid, but absolute tok/s values overstate real agentic throughput. A replacement prompt is pending.

---

## llama.cpp comparison configs

These are the **exact** commands used to benchmark llama.cpp on the same hardware (RTX 5090 Laptop). Both engines built and measured in the same session.

### Build info
- **Commit:** b9743 (c57607016)
- **CUDA:** 13.1
- **Compile flags:** `GGML_CUDA_FA=ON GGML_CUDA_GRAPHS=ON GGML_CUDA_FA_ALL_QUANTS=ON`, `FORCE_MMQ=OFF`
- **Architecture:** sm_120a SASS
- **Binary path:** `/home/avifenesh/projects/llama.cpp/build/bin/`
- **Required env:** `export GGML_CUDA_GRAPH_OPT=1` before every run

### llama-bench (raw pp/tg, no MTP)

**9B NVFP4:**
```bash
export GGML_CUDA_GRAPH_OPT=1
/home/avifenesh/projects/llama.cpp/build/bin/llama-bench \
  -m /data/ai-ml/hf-models/qwen35-9b-nvfp4-gguf/Qwen3.5-9B-NVFP4-MTP-GGUF.gguf \
  -ngl 999 -fa 1 -ctk q8_0 -ctv q5_1 -p 2048 -n 128 -d 0,4096,8192 -r 5
# MEASURED: pp512=6220 t/s, tg128=126.6 t/s (raw decode, no MTP)
```

**27B NVFP4:**
```bash
export GGML_CUDA_GRAPH_OPT=1
/home/avifenesh/projects/llama.cpp/build/bin/llama-bench \
  -m /data/ai-ml/hf-models/qwen36-27b-nvfp4-mtp/Qwen3.6-27B-NVFP4-Q4_K_M-mtp.gguf \
  -ngl 999 -fa 1 -ctk q8_0 -ctv q5_1 -p 2048 -n 128 -d 0,4096,8192 -r 5
# MEASURED: pp512=1980 t/s, tg128=42.1 t/s (raw decode, no MTP)
```

### llama-server (with MTP speculative decoding)

**27B NVFP4 with MTP (peak config):**
```bash
export GGML_CUDA_GRAPH_OPT=1
/home/avifenesh/projects/llama.cpp/build/bin/llama-server \
  -m /data/ai-ml/hf-models/qwen36-27b-nvfp4-mtp/Qwen3.6-27B-NVFP4-Q4_K_M-mtp.gguf \
  -md /data/ai-ml/hf-models/qwen36-27b-nvfp4-mtp/mtp-Qwen3.6-27B-NVFP4.gguf \
  -ngl 999 -fa on -ctk q8_0 -ctv q5_1 -c 65536 --parallel 1 \
  --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.2 \
  --jinja --temp 0.6 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8099
# MEASURED: 66.6 t/s decode with MTP (accept rate 0.762, mean accept len 3.29)
```

**9B NVFP4 serve (no separate MTP draft on disk):**
```bash
export GGML_CUDA_GRAPH_OPT=1
/home/avifenesh/projects/llama.cpp/build/bin/llama-server \
  -m /data/ai-ml/hf-models/qwen35-9b-nvfp4-gguf/Qwen3.5-9B-NVFP4-MTP-GGUF.gguf \
  -ngl 999 -fa on -ctk q8_0 -ctv q5_1 -c 65536 --parallel 1 \
  --jinja --temp 0.6 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8099
```

---

## Protocol notes

- **N=3** runs minimum, report the **median**
- **256 generated tokens**, temperature 0 (greedy)
- **Same session:** both engines measured within the same thermal regime, no clock-speed drift
- **Speculative exactness gate:** K=1..8 self-consistency check pins output token-identical to plain greedy decode
- **Numeric config:** all v0.8.0 numbers use `BW24_FA_V2=1` (tile-batched online-softmax FA decode, the engine default since 2026-07-08)

Full protocol details in the main repo: [`research/benchmarks.md`](https://github.com/avifenesh/bw24/blob/main/research/benchmarks.md) and [`docs/COMPETITOR-SETUP.md`](https://github.com/avifenesh/bw24/blob/main/docs/COMPETITOR-SETUP.md).