license: mit
phase-sensitive-kv — reproduction checkpoints
Model collection for an independent, from-scratch reproduction of "Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression" (ByteDance Seed, arXiv:2609.36322).
This is an independent reproduction, not the official implementation. Code, training details and evaluation results live in the companion repo: ShengenWu/phase-sensitive-kv.
All models share the same Qwen3-0.6B backbone, FineWeb-Edu data (data_seed 777),
optimizer/schedule, and evaluation protocol. They differ only in attention:
c0-* uses the paper-reference chunked KV compression (W=S=8, 1 KV head, scalar
gates, separate K/V branches); fullattn-* is the data-matched uncompressed
control (full causal attention, unbounded cache).
Checkpoints
| Path | Variant | Training tokens | Seed |
|---|---|---|---|
c0-1p75b/seed-42.pt |
compressed (C0) | 1.75B | 42 |
c0-1p75b/seed-43.pt |
compressed (C0) | 1.75B | 43 |
fullattn-1p75b/seed-42.pt |
full attention (control) | 1.75B | 42 |
fullattn-1p75b/seed-43.pt |
full attention (control) | 1.75B | 43 |
c0-3p5b/seed-42.pt |
compressed (C0) | 3.5B | 42 |
fullattn-3p5b/seed-42.pt |
full attention (control) | 3.5B | 42 |
Each file is a torch.save checkpoint dict (bf16 weights, ~2.2 GB) with keys
config (full ResCSAConfig) and model (state dict), stripped of optimizer
state. Load with the companion repo's package:
import torch
from rescsa.model.modeling import ResCSAForCausalLM
ck = torch.load("c0-3p5b/seed-42.pt", map_location="cpu", weights_only=False)
model = ResCSAForCausalLM(ck["config"]) # variant/layout are carried in the config
model.load_state_dict(ck["model"])
Headline result
Phase gap (best minus worst source-phase accuracy, prefix_local protocol) grows with budget in the compressed model while the uncompressed control stays flat: 5.0 → 27.4 → 46.6 pp at 500M → 1.75B → 3.5B tokens for C0 (seed 42), versus ≤ 3.5 pp for the full-attention control at matched budgets. See RESULTS.md for full numbers and confidence intervals.
Citation
If you use these checkpoints, please cite the original paper:
@article{zhu2026periodic,
title = {Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression},
author = {Zhu, Xingyu and Yi, Pu (Luke) and Cheng, Ziheng and Lv, Ang and Liu, Jing and Ying, Lexing and Ma, Yiyuan and Dong, Xin},
journal = {arXiv preprint arXiv:2609.36322},
year = {2026}
}