Commit History
27 days ago
-
Fix SinkCache RoPE coordinates for absolute generation
Verified signature## Problem In Transformers 4.52.1, the standard generation loop advances Qwen2 RoPE positions absolutely. The upstream SinkCache truncates its KV window but also rerotates every surviving key into a bounded local slot. Its stored key phase therefore no longer matches the new query's absolute phase. ## Change - The custom `generate()` wrapper selects absolute-position mode. A shift retains each surviving key's original RoPE phase while dropping old K/V. - Direct callers can select `position_mode="local"` when both `position_ids` and `cache_position` intentionally use bounded slots. That path computes rerotation in FP32, normalizes the derived twiddle, then writes BF16 keys. - Appending exactly to capacity no longer evicts a token early. - README documents the position contract and the tested Transformers version. ## Verification - Qwen2.5-1.5B-Instruct, BF16, eager attention, Transformers 4.52.1, RTX 4080; two cache arms interleaved against the same frozen model and WikiTex…5382829
-
add sanity checks
Joao Gante835a8a6 -
add sanity checks
Joao Ganteffa8b5a -
allow passing a sink cache back in
Joao Gantef84586a -
add sink cache options
Joao Gantee9837c5 -
add sinkcache class
Joao Gante7f36015