Text Generation
qwen3
math
trimkv
KV
Cache
Compression
ngocbh commited on
Commit
7733268
·
verified ·
1 Parent(s): a94c6ba

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -25
README.md CHANGED
@@ -15,34 +15,15 @@ tags:
15
 
16
  # TrimKV: Token Retention for Memory-Bounded Key-Value Eviction
17
 
18
- This repository contains the weights for **TRIM-KV-Qwen3-1.7B-Math**, presented in the paper [Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction](https://huggingface.co/papers/2605.09649).
19
 
20
- The official implementation and training code can be found at [https://github.com/ngocbh/trimkv](https://github.com/ngocbh/trimkv).
21
 
22
- > TRIM-KV is an efficient and learnable key–value eviction strategy designed to improve the efficiency of large language models (LLMs) in long-horizon inference.
23
 
24
- The core idea behind TRIM-KV is to learn the intrinsic importance of each key–value pair at creation time, which we call *token retention*, and then decay this importance exponentially over time to mimic the standard inference running with eviction.
25
-
26
- The retention score is query-agnostic and captures the long-term utility of tokens. This is different from attention scores, which are query-dependent: they capture the short-term utility for predicting the next token and are recomputed at every step, making them local, myopic, and highly dependent on the transient decoding state.
27
-
28
- ### Why TRIM-KV?
29
-
30
- It's fast
31
- <div align="center">
32
- <img width="1000" alt="teaser" src="https://github.com/ngocbh/trimkv/blob/main/assets/speed.png?raw=true"/>
33
- </div>
34
-
35
- It's smart
36
- <div align="center">
37
- <img width="1000" alt="teaser" src="https://github.com/ngocbh/trimkv/blob/main/assets/performance.png?raw=true"/>
38
- </div>
39
-
40
- And it's interpretable
41
- <div align="center">
42
- <img width="1000" alt="teaser" src="https://github.com/ngocbh/trimkv/blob/main/assets/eviction.png?raw=true"/>
43
- </div>
44
-
45
- ---
46
 
47
  ## Quick Start
48
 
@@ -94,6 +75,12 @@ past_key_values = PagedTrimKVCache(
94
  ## Citation
95
 
96
  ```bibtex
 
 
 
 
 
 
97
  @article{bui2025make,
98
  title={Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction},
99
  author={Bui, Ngoc and Nguyen, Hieu Trung and Cohan, Arman and Ying, Rex},
 
15
 
16
  # TrimKV: Token Retention for Memory-Bounded Key-Value Eviction
17
 
18
+ This repository contains the weights for **TRIM-KV-Qwen3-1.7B-Math**, as presented in the paper [Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs](https://huggingface.co/papers/2512.03324).
19
 
20
+ The core idea behind TRIM-KV is to learn the intrinsic importance of each key–value pair at creation time, which we call *token retention*, and then decay this importance exponentially over time to mimic standard inference running with eviction.
21
 
22
+ The retention score is query-agnostic and captures the long-term utility of tokens. This is different from attention scores, which are query-dependent: they capture the short-term utility for predicting the next token and are recomputed at every step.
23
 
24
+ - **Paper:** [Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs](https://huggingface.co/papers/2605.09649)
25
+ - **Code:** [GitHub - ngocbh/trimkv](https://github.com/ngocbh/trimkv)
26
+ - **Arxiv:** [2512.03324](https://arxiv.org/abs/2512.03324)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
27
 
28
  ## Quick Start
29
 
 
75
  ## Citation
76
 
77
  ```bibtex
78
+ @article{bui2025cache,
79
+ title={Cache what lasts: Token retention for memory-bounded kv cache in llms},
80
+ author={Bui, Ngoc and Sharma, Shubham and Lamba, Simran and Mishra, Saumitra and Ying, Rex},
81
+ journal={arXiv preprint arXiv:2512.03324},
82
+ year={2025}
83
+ }
84
  @article{bui2025make,
85
  title={Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction},
86
  author={Bui, Ngoc and Nguyen, Hieu Trung and Cohan, Arman and Ying, Rex},