Add Qwen3-30B-A3B OSCAR KV rotations
Browse files- Qwen3-30B-A3B/README.md +6 -2
Qwen3-30B-A3B/README.md
CHANGED
|
@@ -9,14 +9,18 @@ near-orthogonal rotations (`mean |diag(R0^T R1)| ~= 0.07`).
|
|
| 9 |
| `k_rotation_perhead.pt`, `v_rotation_perhead.pt` | `format_version: 2` | `rotation: [4, 128, 128]` per layer, 48 layers -- **use these** |
|
| 10 |
| `k_rotation_qqt_r_h_pbr.pt`, `v_rotation_sst_r_h_pbr.pt` | `format_version: 1` | one shared `[128, 128]` per layer -- for comparison only |
|
| 11 |
|
| 12 |
-
|
| 13 |
|
| 14 |
| Benchmark | BF16 | INT2 shared | INT2 per-head |
|
| 15 |
|---|:---:|:---:|:---:|
|
| 16 |
| GPQA | 60.8 +- 3.2 | 43.9 +- 2.8 | **58.6 +- 0.9** |
|
| 17 |
| HumanEval | 90.9 +- 2.1 | 25.8 +- 2.5 | **89.0 +- 1.2** |
|
| 18 |
-
| AIME 2025 |
|
| 19 |
| MATH500 | 93.6 +- 0.9 | 60.8 +- 2.2 | **92.1 +- 0.4** |
|
| 20 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
Serving per-head needs a runtime that reads `format_version: 2`; the pool logs
|
| 22 |
`[per-head: 4 kv heads]` at startup when it is active.
|
|
|
|
| 9 |
| `k_rotation_perhead.pt`, `v_rotation_perhead.pt` | `format_version: 2` | `rotation: [4, 128, 128]` per layer, 48 layers -- **use these** |
|
| 10 |
| `k_rotation_qqt_r_h_pbr.pt`, `v_rotation_sst_r_h_pbr.pt` | `format_version: 1` | one shared `[128, 128]` per layer -- for comparison only |
|
| 11 |
|
| 12 |
+
TP=2, 32K generation; 3 seeds except AIME (8 seeds):
|
| 13 |
|
| 14 |
| Benchmark | BF16 | INT2 shared | INT2 per-head |
|
| 15 |
|---|:---:|:---:|:---:|
|
| 16 |
| GPQA | 60.8 +- 3.2 | 43.9 +- 2.8 | **58.6 +- 0.9** |
|
| 17 |
| HumanEval | 90.9 +- 2.1 | 25.8 +- 2.5 | **89.0 +- 1.2** |
|
| 18 |
+
| AIME 2025 (8 seeds) | 72.5 +- 4.6 | 14.2 +- 7.1 | **66.2 +- 3.8** |
|
| 19 |
| MATH500 | 93.6 +- 0.9 | 60.8 +- 2.2 | **92.1 +- 0.4** |
|
| 20 |
|
| 21 |
+
AIME is the one benchmark where per-head still loses measurably (-6.3, Welch
|
| 22 |
+
t=2.96, p~0.01 over 8 seeds); the other three sit within 1.5-2.2 of BF16. Long
|
| 23 |
+
chain-of-thought math is the hardest case for INT2 KV.
|
| 24 |
+
|
| 25 |
Serving per-head needs a runtime that reads `format_version: 2`; the pool logs
|
| 26 |
`[per-head: 4 kv heads]` at startup when it is active.
|