| --- |
| license: mit |
| tags: |
| - benchmark |
| - cuda |
| - speculative-decoding |
| - nvfp4 |
| --- |
| |
| # bw24-bench — benchmark artifacts for bw24 |
|
|
| Reproduction artifacts for the performance claims in [github.com/avifenesh/bw24](https://github.com/avifenesh/bw24) — a from-scratch LLM inference engine for RTX 5090 Laptop (sm_120a). If a claim depends on a specific artifact, the artifact is public here. |
| |
| ## Drafts (`drafts/<model>/`) — the standard regime, one file per model |
| |
| Every bw24 spec board row runs **one trimmed draft file** built by the documented regime |
| ([docs/DRAFT-REGIME.md](https://github.com/avifenesh/bw24/blob/main/docs/DRAFT-REGIME.md)): |
| FR-Spec ranks derived from the **model's own generations** (never transferred between |
| models — measured −12 acceptance pts), MTP block extracted **byte-verbatim** from the |
| published model GGUF, head requantized **NVFP4 after trimming** (zero measured acceptance |
| cost), block Q4_K_M (faster AND higher acceptance than Q8_0). Serve with |
| `BW24_MTP_DRAFT=<draft.gguf>` — no other flags. |
|
|
| | Directory | Source model (exact bytes) | Board row (e2e tok/s p1/p2/p3, vs llama.cpp same-window) | |
| |---|---|---| |
| | `drafts/qwen35-9b-nvfp4/` | Qwen3.5-9B NVFP4 MTP GGUF | 281 / 212 / 187 = 2.30x / 1.74x / 1.59x | |
| | `drafts/qwen36-27b-nvfp4/` | nvidia/Qwen3.6-27B-NVFP4 → Q4_K_M GGUF | 116 / 101 / 86 = 1.27x / 1.08x / 1.06x | |
| | `drafts/qwen36-35b-a3b-iq4xs/` | unsloth Qwen3.6-35B-A3B UD-IQ4_XS | 281 / 260 / 258 = 1.19x / 1.49x / 1.49x | |
| | `drafts/qwen36-27b-unsloth-nvfp4/` | unsloth/Qwen3.6-27B-NVFP4 → GGUF (recipe in bw24 jsonl) | plain parity with the nvidia daily; spec p3 win / p2 loss (jsonl 2026-07-17) | |
| |
| Each directory carries the draft (`draft-owntrim-nvfp4head-q4blk.gguf`) and the rank file |
| it was built from (`owngen-ranks-32768.txt`, one token id per line, rank order — derived |
| 2026-07-17/18, 218-prompt mixed corpus, ~110k own-generated tokens per model). |
| |
| **Use ours for these exact models. For any other model, requant, or finetune, build your |
| own in two commands** (a finetune's distribution moved, so its draft must too): |
| |
| ```bash |
| ./target/release/frspec-owngen model.gguf ranks.gguf 32768 # ranks from the model's OWN generations |
| tools/make-trimmed-draft.sh model.gguf ranks.gguf.txt draft.gguf # extract + trim + quantize |
| ``` |
| |
| |
| ## NVFP4-KV drafts (`drafts/kf4/<model>/`) — the `BW24_KV_K=nvfp4 BW24_KV_V=nvfp4` config |
| |
| The nvfp4 KV-cache format (kv-fp4 lane, 2026-07-20: 18 B/32 elems both planes = −38% |
| KV bytes/token; self-consistency PASS, battery green in-config) is a NEW NUMERIC CONFIG — |
| per the regime's law 1, each model's draft is re-derived from its OWN generations *under |
| that config*. These directories carry the kf4-derived draft + rank files: |
| |
| | Directory | Verdict vs the default-config draft (same config, K=cell, N=2) | |
| |---|---| |
| | `drafts/kf4/qwen35-9b-nvfp4/` | wash (old file marginally ahead; either serves) | |
| | `drafts/kf4/qwen36-27b-nvfp4/` | **kf4 draft WINS** (+2.5% e2e, acceptance 60.1 vs 57.1) — use this one under nvfp4 KV | |
| | `drafts/kf4/qwen36-35b-a3b-iq4xs/` | old draft stays (−1% / acceptance 67.0 vs 69.2) — kept as evidence | |
| |
| Gemma models keep their existing rank files under nvfp4 KV (measured never-worse, |
| 2026-07-20 jsonl): load-time trim + the serve-time adaptive trim absorb the format shift. |
| |
| ## Gemma ranks (`drafts/gemma4-*/`) |
| |
| Gemma drafters ship as standalone MTP/assistant GGUFs (byte-verbatim by provenance — law 2 |
| built in); the FR-Spec trim applies at load: `BW24_GEMMA_DRAFT_RANKS=<ranks.txt>`. Own-gen |
| rank files per model (law 1): |
|
|
| | Directory | Model | Serving note | |
| |---|---|---| |
| | `drafts/gemma4-26b-a4b-qat/` | Gemma-4 26B-A4B QAT Q4_0 | trim adopted (e2e-neutral vs corpus ranks, correct provenance) | |
| | `drafts/gemma4-31b-qat/` | Gemma-4 31B QAT Q4_0 | trim adopted on BOTH cells via the serve-time ADAPTIVE trim (`BW24_GEMMA_TRIM_ADAPT`, on by default with ranks): coverage escapes learned live from prompt+verify tokens, persisted to `<ranks>.learned` — flipped the chat cell from −17% to +2.5% (bw24 jsonl 2026-07-19) | |
| | `drafts/gemma4-e4b-qat/` | Gemma-4 E4B QAT Q4_0 | ranks published; serving stays UNTRIMMED (measured e2e wash — small head) | |
| |
| ## Legacy trims (top level, Qwen3.6-27B) |
| |
| The pre-regime artifacts the earlier boards used — superseded by `drafts/` but kept because |
| published claims referenced them: |
| |
| | File | Ranking | Status | |
| |---|---|---| |
| | `mtp-Qwen3.6-27B-Q4_K_M-frspec32768.gguf` | generic corpus-frequency | superseded by `drafts/qwen36-27b-nvfp4/` | |
| | `mtp-Qwen3.6-27B-Q4_K_M-frspec-code75-32768.gguf` | code-skewed corpus | superseded | |
| | `mtp-Qwen3.6-27B-Q4_K_M-frspec-balanced32768.gguf` | balanced corpus | superseded | |
| |
| ## Prompts + configs |
| |
| `prompts/` holds the exact p1/p2/p3 board prompts; `CONFIGS.md` the per-cell engine |
| configs and llama.cpp pairing flags. Protocol: interleaved same-window N≥2, power state |
| pinned, one engine on the GPU at a time — |
| [research/benchmarks.md](https://github.com/avifenesh/bw24/blob/main/research/benchmarks.md). |
| |
| ## Related |
| |
| - Engine: [github.com/avifenesh/bw24](https://github.com/avifenesh/bw24) — every published number's raw JSONL under `research/`, wins and losses both. |
| - [Avifenesh/Hy3-REAP-Layer103p5-bw24](https://huggingface.co/Avifenesh/Hy3-REAP-Layer103p5-bw24) — the ~100 GB Hy3 expert overlay bw24 serves on a 24 GB card via VRAM→RAM→NVMe spill. |
| |
