---
license: mit
tags:
  - benchmark
  - cuda
  - speculative-decoding
  - nvfp4
---

# bw24-bench — benchmark artifacts for bw24

Reproduction artifacts for the performance claims in [github.com/avifenesh/bw24](https://github.com/avifenesh/bw24) — a from-scratch LLM inference engine for RTX 5090 Laptop (sm_120a). If a claim depends on a specific artifact, the artifact is public here.

## Drafts (`drafts/<model>/`) — the standard regime, one file per model

Every bw24 spec board row runs **one trimmed draft file** built by the documented regime
([docs/DRAFT-REGIME.md](https://github.com/avifenesh/bw24/blob/main/docs/DRAFT-REGIME.md)):
FR-Spec ranks derived from the **model's own generations** (never transferred between
models — measured −12 acceptance pts), MTP block extracted **byte-verbatim** from the
published model GGUF, head requantized **NVFP4 after trimming** (zero measured acceptance
cost), block Q4_K_M (faster AND higher acceptance than Q8_0). Serve with
`BW24_MTP_DRAFT=<draft.gguf>` — no other flags.

| Directory | Source model (exact bytes) | Board row (e2e tok/s p1/p2/p3, vs llama.cpp same-window) |
|---|---|---|
| `drafts/qwen35-9b-nvfp4/` | Qwen3.5-9B NVFP4 MTP GGUF | 281 / 212 / 187 = 2.30x / 1.74x / 1.59x |
| `drafts/qwen36-27b-nvfp4/` | nvidia/Qwen3.6-27B-NVFP4 → Q4_K_M GGUF | 116 / 101 / 86 = 1.27x / 1.08x / 1.06x |
| `drafts/qwen36-35b-a3b-iq4xs/` | unsloth Qwen3.6-35B-A3B UD-IQ4_XS | 281 / 260 / 258 = 1.19x / 1.49x / 1.49x |
| `drafts/qwen36-27b-unsloth-nvfp4/` | unsloth/Qwen3.6-27B-NVFP4 → GGUF (recipe in bw24 jsonl) | plain parity with the nvidia daily; spec p3 win / p2 loss (jsonl 2026-07-17) |

Each directory carries the draft (`draft-owntrim-nvfp4head-q4blk.gguf`) and the rank file
it was built from (`owngen-ranks-32768.txt`, one token id per line, rank order — derived
2026-07-17/18, 218-prompt mixed corpus, ~110k own-generated tokens per model).

**Use ours for these exact models. For any other model, requant, or finetune, build your
own in two commands** (a finetune's distribution moved, so its draft must too):

```bash
./target/release/frspec-owngen model.gguf ranks.gguf 32768        # ranks from the model's OWN generations
tools/make-trimmed-draft.sh model.gguf ranks.gguf.txt draft.gguf  # extract + trim + quantize
```


## NVFP4-KV drafts (`drafts/kf4/<model>/`) — the `BW24_KV_K=nvfp4 BW24_KV_V=nvfp4` config

The nvfp4 KV-cache format (kv-fp4 lane, 2026-07-20: 18 B/32 elems both planes = −38%
KV bytes/token; self-consistency PASS, battery green in-config) is a NEW NUMERIC CONFIG —
per the regime's law 1, each model's draft is re-derived from its OWN generations *under
that config*. These directories carry the kf4-derived draft + rank files:

| Directory | Verdict vs the default-config draft (same config, K=cell, N=2) |
|---|---|
| `drafts/kf4/qwen35-9b-nvfp4/` | wash (old file marginally ahead; either serves) |
| `drafts/kf4/qwen36-27b-nvfp4/` | **kf4 draft WINS** (+2.5% e2e, acceptance 60.1 vs 57.1) — use this one under nvfp4 KV |
| `drafts/kf4/qwen36-35b-a3b-iq4xs/` | old draft stays (−1% / acceptance 67.0 vs 69.2) — kept as evidence |

Gemma models keep their existing rank files under nvfp4 KV (measured never-worse,
2026-07-20 jsonl): load-time trim + the serve-time adaptive trim absorb the format shift.

## Gemma ranks (`drafts/gemma4-*/`)

Gemma drafters ship as standalone MTP/assistant GGUFs (byte-verbatim by provenance — law 2
built in); the FR-Spec trim applies at load: `BW24_GEMMA_DRAFT_RANKS=<ranks.txt>`. Own-gen
rank files per model (law 1):

| Directory | Model | Serving note |
|---|---|---|
| `drafts/gemma4-26b-a4b-qat/` | Gemma-4 26B-A4B QAT Q4_0 | trim adopted (e2e-neutral vs corpus ranks, correct provenance) |
| `drafts/gemma4-31b-qat/` | Gemma-4 31B QAT Q4_0 | trim adopted on BOTH cells via the serve-time ADAPTIVE trim (`BW24_GEMMA_TRIM_ADAPT`, on by default with ranks): coverage escapes learned live from prompt+verify tokens, persisted to `<ranks>.learned` — flipped the chat cell from −17% to +2.5% (bw24 jsonl 2026-07-19) |
| `drafts/gemma4-e4b-qat/` | Gemma-4 E4B QAT Q4_0 | ranks published; serving stays UNTRIMMED (measured e2e wash — small head) |

## Legacy trims (top level, Qwen3.6-27B)

The pre-regime artifacts the earlier boards used — superseded by `drafts/` but kept because
published claims referenced them:

| File | Ranking | Status |
|---|---|---|
| `mtp-Qwen3.6-27B-Q4_K_M-frspec32768.gguf` | generic corpus-frequency | superseded by `drafts/qwen36-27b-nvfp4/` |
| `mtp-Qwen3.6-27B-Q4_K_M-frspec-code75-32768.gguf` | code-skewed corpus | superseded |
| `mtp-Qwen3.6-27B-Q4_K_M-frspec-balanced32768.gguf` | balanced corpus | superseded |

## Prompts + configs

`prompts/` holds the exact p1/p2/p3 board prompts; `CONFIGS.md` the per-cell engine
configs and llama.cpp pairing flags. Protocol: interleaved same-window N≥2, power state
pinned, one engine on the GPU at a time —
[research/benchmarks.md](https://github.com/avifenesh/bw24/blob/main/research/benchmarks.md).

## Related

- Engine: [github.com/avifenesh/bw24](https://github.com/avifenesh/bw24) — every published number's raw JSONL under `research/`, wins and losses both.
- [Avifenesh/Hy3-REAP-Layer103p5-bw24](https://huggingface.co/Avifenesh/Hy3-REAP-Layer103p5-bw24) — the ~100 GB Hy3 expert overlay bw24 serves on a 24 GB card via VRAM→RAM→NVMe spill.