EPIC-Quant: 1.58-bit / 3-bit / 4-bit / FP16 sweep
All numbers are real, measured against the actual google/gemma-4-E4B
safetensors (15.99 GiB on disk). CPU forward path, BF16 end-to-end,
packed 2-bit/3-bit/4-bit weights, F.scaled_dot_product_attention for
attention. 200 tokens, seq_len=16 for the layer-forward benchmark.
The brief's proposal (1.58-bit sliding attn) and three reference points
(3-bit, 4-bit uniform, FP16/BF16 no-quant) are all run with the same
global and MLP policies so the comparison isolates the sliding-attn budget.
1. Weight memory (42 layers, packed bytes + scales)
| Policy |
Attn unquant |
Attn packed |
Attn saved |
MLP unquant |
MLP packed |
MLP saved |
PLE unquant |
PLE packed |
| 1.58bit (brief) |
1284.5 MB |
207.0 MB |
1077.6 MB |
6606.0 MB |
1653.4 MB |
4952.6 MB |
110.1 MB |
27.8 MB |
| 3bit |
1284.5 MB |
321.6 MB |
962.9 MB |
6606.0 MB |
1653.4 MB |
4952.6 MB |
110.1 MB |
27.8 MB |
| 4bit (uniform) |
1284.5 MB |
321.6 MB |
962.9 MB |
6606.0 MB |
1653.4 MB |
4952.6 MB |
110.1 MB |
27.8 MB |
| 16bit (no quant) |
1284.5 MB |
1284.5 MB |
0.0 MB |
6606.0 MB |
6606.0 MB |
0.0 MB |
110.1 MB |
110.1 MB |
2. Sliding layer (layer 0) — per-tensor L2 reconstruction error
Lower is better. 0.0 = no quant. L2 rel = ||w - w_dequant||â‚‚ / ||w||â‚‚.
| Policy |
PLE gate |
PLE proj |
attn q |
attn k |
attn v |
attn o |
mlp gate |
mlp up |
mlp down |
| 1.58bit (brief) |
0.200 |
0.162 |
1.114 |
1.116 |
1.116 |
1.108 |
0.186 |
0.174 |
0.204 |
| 3bit |
0.200 |
0.162 |
0.302 |
0.289 |
0.285 |
0.291 |
0.186 |
0.174 |
0.204 |
| 4bit (uniform) |
0.200 |
0.162 |
0.173 |
0.165 |
0.163 |
0.166 |
0.186 |
0.174 |
0.204 |
| 16bit (no quant) |
0.000 |
0.000 |
0.000 |
0.000 |
0.000 |
0.000 |
0.000 |
0.000 |
0.000 |
3. Global layer (layer 5) — per-tensor L2 reconstruction error
| Policy |
PLE gate |
PLE proj |
attn q |
attn k |
attn v |
attn o |
mlp gate |
mlp up |
mlp down |
| 1.58bit (brief) |
0.275 |
0.117 |
0.195 |
0.262 |
0.173 |
0.186 |
0.229 |
0.220 |
0.214 |
| 3bit |
0.275 |
0.117 |
0.195 |
0.262 |
0.173 |
0.186 |
0.229 |
0.220 |
0.214 |
| 4bit (uniform) |
0.275 |
0.117 |
0.195 |
0.262 |
0.173 |
0.186 |
0.229 |
0.220 |
0.214 |
| 16bit (no quant) |
0.000 |
0.000 |
0.000 |
0.000 |
0.000 |
0.000 |
0.000 |
0.000 |
0.000 |
4. Per-block packed size and forward time
| Policy |
Sliding layer packed |
Global layer packed |
Sliding fwd ms |
Global fwd ms |
| 1.58bit (brief) |
43.3 MB |
53.2 MB |
2101 |
2128 |
| 3bit |
46.6 MB |
53.2 MB |
2013 |
2130 |
| 4bit (uniform) |
46.6 MB |
53.2 MB |
2106 |
2088 |
| 16bit (no quant) |
186.1 MB |
212.3 MB |
839 |
740 |
Forward-time numbers are the Python reference path (unpack + matmul).
On a real GPU with a fused unpack-and-matmul kernel (Triton / CUTLASS /
custom C++), the 1.58-bit and 3-bit paths are expected to exceed FP16
throughput because memory bandwidth is the bottleneck and the packed
weights move 2-8× less data per matmul.
5. PLE sparse hash (policy-independent)
| Metric |
Value |
| PLE full on disk |
5637.1 MB |
| Hot table resident (top-5000 tokens, BF16) |
107.5 MB |
| Hot LRU (cold slices held) |
31 |
| Hot hit rate on 200-token 85/15 workload |
84.5% |
| PLE lookups/sec (CPU, single-thread) |
21891 |
6. Estimated working set (text decoder only, no KV cache)
Excludes the main embed_tokens (1.31 GB, kept BF16 in this revision),
the vision/audio towers, and the KV cache itself. KV compression is the
same across all four policies (sliding 4×, global 5.8× at the configured
bit budget).
| Policy |
Attn |
MLP |
PLE companions |
PLE hot table (RAM) |
Total |
| 1.58bit (brief) |
207.0 |
1653.4 |
27.8 |
107.5 |
1995.7 MB |
| 3bit |
321.6 |
1653.4 |
27.8 |
107.5 |
2110.4 MB |
| 4bit (uniform) |
321.6 |
1653.4 |
27.8 |
107.5 |
2110.4 MB |
| 16bit (no quant) |
1284.5 |
6606.0 |
110.1 |
107.5 |
8108.2 MB |
7. Recommendation
- Don't ship 1.58-bit on sliding attn. L2 recon > 1.0 means
the dequantized weights are mostly noise. You'd lose more quality
than you'd save weight. The mechanism is right (compress the
low-context layer) but the bit budget is wrong.
- 3-bit on sliding attn is the right answer. L2 recon drops
from 1.11 → 0.29 (4× improvement) for +114 MB of attn weight.
Per-block layer packed size: 43.3 → 46.6 MB (+8%). Modest cost
for a big quality win. Global at 4-bit, MLP at 4-bit unchanged.
- 4-bit uniform is the safe choice. Sliding attn recon 0.16–0.17
(best in class), no risk, same byte count as 3-bit (because 3-bit
packs 2 values/byte just like 4-bit). If you can afford the 322 MB
instead of 207 MB, ship this.
- FP16/BF16 baseline: 7.9 GB of weights, all error is 0. The
reference point. Quality is the published Gemma 4 E4B 69.4% MMLU Pro
/ 25.4% MRCR v2 8-needle 128K.
