Update nexc/README.md with O(1) constant-speed decode up to 1M
Browse files- nexc/README.md +1 -0
nexc/README.md
CHANGED
|
@@ -27,6 +27,7 @@ tags:
|
|
| 27 |
- **2.6×–3.4× Faster Long-Context Prefill & 11.8× Faster Long-Context Decode:** Achieves **`33.59 tok/s`** decode at `8K` (`25.48 tok/s` under `1M` `max_model_len`), **`13.82 tok/s`** decode (`36.7s` TTFT) at `120K`, **`13.31 tok/s`** decode (`90.3s` TTFT) at `250K`, `500K` TTFT in **`238.3s`**, and `1,000,000` TTFT in **`615.4s`** on a single NVIDIA GB10 GPU with zero required `PB27_MEMORY_*` flags.
|
| 28 |
- **$O(1)$ Constant GPU KV Cache (`~27–31 GiB` Total GPU Footprint up to `1,000,000` Tokens):** Uses only `8.1%` of a fixed `6 GiB` KV cache pool even at `1,000,000` tokens.
|
| 29 |
- **100% (`40/40 = 1.000`) Exact Single-Needle, 4-Needle (`multi4`), and 4-Hop Call-Chain Trace (`trace4`) Retrieval from `8K` to `1,000,000` Tokens:** Combines median-centered Quest block bounds, per-query Z-score normalization, post-merge `[b, b+1]` successor expansion (`k=1024, c=8192` at 1M), immutable `prompt_top` + rolling `decode_top` retention, and position-invariant 2-hop key-to-key bridge retrieval over the $75\%$ unrotated key subspace (`d // 4 :`).
|
|
|
|
| 30 |
- **W4A16 Marlin NVFP4 MoE Scale Alignment:** Includes automatic `w1`/`w3` `weight_global_scale` harmonization (`_install_nvfp4_scale_hooks`), reducing fused SwiGLU scale discrepancy from $38.1\%$ to $2.9\%$.
|
| 31 |
|
| 32 |
## Benchmark Summary
|
|
|
|
| 27 |
- **2.6×–3.4× Faster Long-Context Prefill & 11.8× Faster Long-Context Decode:** Achieves **`33.59 tok/s`** decode at `8K` (`25.48 tok/s` under `1M` `max_model_len`), **`13.82 tok/s`** decode (`36.7s` TTFT) at `120K`, **`13.31 tok/s`** decode (`90.3s` TTFT) at `250K`, `500K` TTFT in **`238.3s`**, and `1,000,000` TTFT in **`615.4s`** on a single NVIDIA GB10 GPU with zero required `PB27_MEMORY_*` flags.
|
| 28 |
- **$O(1)$ Constant GPU KV Cache (`~27–31 GiB` Total GPU Footprint up to `1,000,000` Tokens):** Uses only `8.1%` of a fixed `6 GiB` KV cache pool even at `1,000,000` tokens.
|
| 29 |
- **100% (`40/40 = 1.000`) Exact Single-Needle, 4-Needle (`multi4`), and 4-Hop Call-Chain Trace (`trace4`) Retrieval from `8K` to `1,000,000` Tokens:** Combines median-centered Quest block bounds, per-query Z-score normalization, post-merge `[b, b+1]` successor expansion (`k=1024, c=8192` at 1M), immutable `prompt_top` + rolling `decode_top` retention, and position-invariant 2-hop key-to-key bridge retrieval over the $75\%$ unrotated key subspace (`d // 4 :`).
|
| 30 |
+
- **True $O(1)$ Constant-Speed Decode up to `1,000,000` Tokens:** Decouples prefill retrieval budget (`PB27_MEMORY_SCALE=2048:384:4096:512:8192:1536`, run once on the final prefill question chunk and pinned in `Archive.prompt_top` + GPU-cached `cached_pre_k`/`cached_pre_v`) from the incremental decode retrieval budget (`PB27_MEMORY_DECODE_BLOCKS=64`, `PB27_MEMORY_DECODE_RERANK=0`, `PB27_MEMORY_DECODE_STRIDE=64`) backed by 2-level hierarchical `bf16` super-block Quest bounds (`PB27_MEMORY_SUPERBLOCK=64`, `256` super-blocks at `1M`, `<0.05 ms` bound scoring, zero NVMe pages touched for rejected candidates during decode).
|
| 31 |
- **W4A16 Marlin NVFP4 MoE Scale Alignment:** Includes automatic `w1`/`w3` `weight_global_scale` harmonization (`_install_nvfp4_scale_hooks`), reducing fused SwiGLU scale discrepancy from $38.1\%$ to $2.9\%$.
|
| 32 |
|
| 33 |
## Benchmark Summary
|