| # EPIC-Quant: 1.58-bit / 3-bit / 4-bit / FP16 sweep | |
| All numbers are **real**, measured against the actual `google/gemma-4-E4B` | |
| safetensors (15.99 GiB on disk). CPU forward path, BF16 end-to-end, | |
| packed 2-bit/3-bit/4-bit weights, `F.scaled_dot_product_attention` for | |
| attention. 200 tokens, seq_len=16 for the layer-forward benchmark. | |
| The brief's proposal (1.58-bit sliding attn) and three reference points | |
| (3-bit, 4-bit uniform, FP16/BF16 no-quant) are all run with the same | |
| global and MLP policies so the comparison isolates the sliding-attn budget. | |
| ## 1. Weight memory (42 layers, packed bytes + scales) | |
| | Policy | Attn unquant | Attn packed | Attn saved | MLP unquant | MLP packed | MLP saved | PLE unquant | PLE packed | | |
| |---|---:|---:|---:|---:|---:|---:|---:|---:| | |
| | **1.58bit (brief)** | 1284.5 MB | 207.0 MB | 1077.6 MB | 6606.0 MB | 1653.4 MB | 4952.6 MB | 110.1 MB | 27.8 MB | | |
| | **3bit** | 1284.5 MB | 321.6 MB | 962.9 MB | 6606.0 MB | 1653.4 MB | 4952.6 MB | 110.1 MB | 27.8 MB | | |
| | **4bit (uniform)** | 1284.5 MB | 321.6 MB | 962.9 MB | 6606.0 MB | 1653.4 MB | 4952.6 MB | 110.1 MB | 27.8 MB | | |
| | **16bit (no quant)** | 1284.5 MB | 1284.5 MB | 0.0 MB | 6606.0 MB | 6606.0 MB | 0.0 MB | 110.1 MB | 110.1 MB | | |
| ## 2. Sliding layer (layer 0) — per-tensor L2 reconstruction error | |
| Lower is better. 0.0 = no quant. L2 rel = ||w - w_dequant||₂ / ||w||₂. | |
| | Policy | PLE gate | PLE proj | attn q | attn k | attn v | attn o | mlp gate | mlp up | mlp down | | |
| |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | |
| | **1.58bit (brief)** | 0.200 | 0.162 | 1.114 | 1.116 | 1.116 | 1.108 | 0.186 | 0.174 | 0.204 | | |
| | **3bit** | 0.200 | 0.162 | 0.302 | 0.289 | 0.285 | 0.291 | 0.186 | 0.174 | 0.204 | | |
| | **4bit (uniform)** | 0.200 | 0.162 | 0.173 | 0.165 | 0.163 | 0.166 | 0.186 | 0.174 | 0.204 | | |
| | **16bit (no quant)** | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | | |
| ## 3. Global layer (layer 5) — per-tensor L2 reconstruction error | |
| | Policy | PLE gate | PLE proj | attn q | attn k | attn v | attn o | mlp gate | mlp up | mlp down | | |
| |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | |
| | **1.58bit (brief)** | 0.275 | 0.117 | 0.195 | 0.262 | 0.173 | 0.186 | 0.229 | 0.220 | 0.214 | | |
| | **3bit** | 0.275 | 0.117 | 0.195 | 0.262 | 0.173 | 0.186 | 0.229 | 0.220 | 0.214 | | |
| | **4bit (uniform)** | 0.275 | 0.117 | 0.195 | 0.262 | 0.173 | 0.186 | 0.229 | 0.220 | 0.214 | | |
| | **16bit (no quant)** | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | | |
| ## 4. Per-block packed size and forward time | |
| | Policy | Sliding layer packed | Global layer packed | Sliding fwd ms | Global fwd ms | | |
| |---|---:|---:|---:|---:| | |
| | **1.58bit (brief)** | 43.3 MB | 53.2 MB | 2101 | 2128 | | |
| | **3bit** | 46.6 MB | 53.2 MB | 2013 | 2130 | | |
| | **4bit (uniform)** | 46.6 MB | 53.2 MB | 2106 | 2088 | | |
| | **16bit (no quant)** | 186.1 MB | 212.3 MB | 839 | 740 | | |
| Forward-time numbers are the **Python reference path** (unpack + matmul). | |
| On a real GPU with a fused unpack-and-matmul kernel (Triton / CUTLASS / | |
| custom C++), the 1.58-bit and 3-bit paths are expected to **exceed FP16 | |
| throughput** because memory bandwidth is the bottleneck and the packed | |
| weights move 2-8× less data per matmul. | |
| ## 5. PLE sparse hash (policy-independent) | |
| | Metric | Value | | |
| |---|---| | |
| | PLE full on disk | 5637.1 MB | | |
| | Hot table resident (top-5000 tokens, BF16) | 107.5 MB | | |
| | Hot LRU (cold slices held) | 31 | | |
| | Hot hit rate on 200-token 85/15 workload | 84.5% | | |
| | PLE lookups/sec (CPU, single-thread) | 21891 | | |
| ## 6. Estimated working set (text decoder only, no KV cache) | |
| Excludes the main `embed_tokens` (1.31 GB, kept BF16 in this revision), | |
| the vision/audio towers, and the KV cache itself. KV compression is the | |
| same across all four policies (sliding 4×, global 5.8× at the configured | |
| bit budget). | |
| | Policy | Attn | MLP | PLE companions | PLE hot table (RAM) | **Total** | | |
| |---|---:|---:|---:|---:|---:| | |
| | **1.58bit (brief)** | 207.0 | 1653.4 | 27.8 | 107.5 | **1995.7 MB** | | |
| | **3bit** | 321.6 | 1653.4 | 27.8 | 107.5 | **2110.4 MB** | | |
| | **4bit (uniform)** | 321.6 | 1653.4 | 27.8 | 107.5 | **2110.4 MB** | | |
| | **16bit (no quant)** | 1284.5 | 6606.0 | 110.1 | 107.5 | **8108.2 MB** | | |
| ## 7. Recommendation | |
| - **Don't ship 1.58-bit on sliding attn.** L2 recon > 1.0 means | |
| the dequantized weights are mostly noise. You'd lose more quality | |
| than you'd save weight. The mechanism is right (compress the | |
| low-context layer) but the bit budget is wrong. | |
| - **3-bit on sliding attn is the right answer.** L2 recon drops | |
| from 1.11 → 0.29 (4× improvement) for +114 MB of attn weight. | |
| Per-block layer packed size: 43.3 → 46.6 MB (+8%). Modest cost | |
| for a big quality win. Global at 4-bit, MLP at 4-bit unchanged. | |
| - **4-bit uniform is the safe choice.** Sliding attn recon 0.16–0.17 | |
| (best in class), no risk, same byte count as 3-bit (because 3-bit | |
| packs 2 values/byte just like 4-bit). If you can afford the 322 MB | |
| instead of 207 MB, ship this. | |
| - **FP16/BF16 baseline: 7.9 GB of weights, all error is 0.** The | |
| reference point. Quality is the published Gemma 4 E4B 69.4% MMLU Pro | |
| / 25.4% MRCR v2 8-needle 128K. | |