---
license: mit
---

# phase-sensitive-kv — reproduction checkpoints

Model collection for an independent, from-scratch reproduction of
**"Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression"**
(ByteDance Seed, [arXiv:2609.36322](https://arxiv.org/abs/2609.36322)).

> **This is an independent reproduction, not the official implementation.**
> Code, training details and evaluation results live in the companion repo:
> [ShengenWu/phase-sensitive-kv](https://github.com/ShengenWu/phase-sensitive-kv).

All models share the same Qwen3-0.6B backbone, FineWeb-Edu data (data_seed 777),
optimizer/schedule, and evaluation protocol. They differ only in attention:
`c0-*` uses the paper-reference chunked KV compression (W=S=8, 1 KV head, scalar
gates, separate K/V branches); `fullattn-*` is the data-matched uncompressed
control (full causal attention, unbounded cache).

## Checkpoints

| Path | Variant | Training tokens | Seed |
|---|---|---:|---:|
| `c0-1p75b/seed-42.pt` | compressed (C0) | 1.75B | 42 |
| `c0-1p75b/seed-43.pt` | compressed (C0) | 1.75B | 43 |
| `fullattn-1p75b/seed-42.pt` | full attention (control) | 1.75B | 42 |
| `fullattn-1p75b/seed-43.pt` | full attention (control) | 1.75B | 43 |
| `c0-3p5b/seed-42.pt` | compressed (C0) | 3.5B | 42 |
| `fullattn-3p5b/seed-42.pt` | full attention (control) | 3.5B | 42 |

Each file is a `torch.save` checkpoint dict (bf16 weights, ~2.2 GB) with keys
`config` (full `ResCSAConfig`) and `model` (state dict), stripped of optimizer
state. Load with the companion repo's package:

```python
import torch
from rescsa.model.modeling import ResCSAForCausalLM

ck = torch.load("c0-3p5b/seed-42.pt", map_location="cpu", weights_only=False)
model = ResCSAForCausalLM(ck["config"])   # variant/layout are carried in the config
model.load_state_dict(ck["model"])
```

## Headline result

Phase gap (best minus worst source-phase accuracy, prefix_local protocol)
grows with budget in the compressed model while the uncompressed control stays
flat: **5.0 → 27.4 → 46.6 pp** at 500M → 1.75B → 3.5B tokens for C0 (seed 42),
versus **≤ 3.5 pp** for the full-attention control at matched budgets. See
[RESULTS.md](https://github.com/ShengenWu/phase-sensitive-kv/blob/main/RESULTS.md)
for full numbers and confidence intervals.

## Citation

If you use these checkpoints, please cite the original paper:

```bibtex
@article{zhu2026periodic,
  title   = {Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression},
  author  = {Zhu, Xingyu and Yi, Pu (Luke) and Cheng, Ziheng and Lv, Ang and Liu, Jing and Ying, Lexing and Ma, Yiyuan and Dong, Xin},
  journal = {arXiv preprint arXiv:2609.36322},
  year    = {2026}
}
```