| --- |
| license: mit |
| --- |
| |
| # phase-sensitive-kv — reproduction checkpoints |
|
|
| Model collection for an independent, from-scratch reproduction of |
| **"Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression"** |
| (ByteDance Seed, [arXiv:2609.36322](https://arxiv.org/abs/2609.36322)). |
|
|
| > **This is an independent reproduction, not the official implementation.** |
| > Code, training details and evaluation results live in the companion repo: |
| > [ShengenWu/phase-sensitive-kv](https://github.com/ShengenWu/phase-sensitive-kv). |
|
|
| All models share the same Qwen3-0.6B backbone, FineWeb-Edu data (data_seed 777), |
| optimizer/schedule, and evaluation protocol. They differ only in attention: |
| `c0-*` uses the paper-reference chunked KV compression (W=S=8, 1 KV head, scalar |
| gates, separate K/V branches); `fullattn-*` is the data-matched uncompressed |
| control (full causal attention, unbounded cache). |
| |
| ## Checkpoints |
| |
| | Path | Variant | Training tokens | Seed | |
| |---|---|---:|---:| |
| | `c0-1p75b/seed-42.pt` | compressed (C0) | 1.75B | 42 | |
| | `c0-1p75b/seed-43.pt` | compressed (C0) | 1.75B | 43 | |
| | `fullattn-1p75b/seed-42.pt` | full attention (control) | 1.75B | 42 | |
| | `fullattn-1p75b/seed-43.pt` | full attention (control) | 1.75B | 43 | |
| | `c0-3p5b/seed-42.pt` | compressed (C0) | 3.5B | 42 | |
| | `fullattn-3p5b/seed-42.pt` | full attention (control) | 3.5B | 42 | |
| |
| Each file is a `torch.save` checkpoint dict (bf16 weights, ~2.2 GB) with keys |
| `config` (full `ResCSAConfig`) and `model` (state dict), stripped of optimizer |
| state. Load with the companion repo's package: |
| |
| ```python |
| import torch |
| from rescsa.model.modeling import ResCSAForCausalLM |
| |
| ck = torch.load("c0-3p5b/seed-42.pt", map_location="cpu", weights_only=False) |
| model = ResCSAForCausalLM(ck["config"]) # variant/layout are carried in the config |
| model.load_state_dict(ck["model"]) |
| ``` |
| |
| ## Headline result |
| |
| Phase gap (best minus worst source-phase accuracy, prefix_local protocol) |
| grows with budget in the compressed model while the uncompressed control stays |
| flat: **5.0 → 27.4 → 46.6 pp** at 500M → 1.75B → 3.5B tokens for C0 (seed 42), |
| versus **≤ 3.5 pp** for the full-attention control at matched budgets. See |
| [RESULTS.md](https://github.com/ShengenWu/phase-sensitive-kv/blob/main/RESULTS.md) |
| for full numbers and confidence intervals. |
|
|
| ## Citation |
|
|
| If you use these checkpoints, please cite the original paper: |
|
|
| ```bibtex |
| @article{zhu2026periodic, |
| title = {Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression}, |
| author = {Zhu, Xingyu and Yi, Pu (Luke) and Cheng, Ziheng and Lv, Ang and Liu, Jing and Ying, Lexing and Ma, Yiyuan and Dong, Xin}, |
| journal = {arXiv preprint arXiv:2609.36322}, |
| year = {2026} |
| } |
| ``` |
|
|
