Text Generation
qwen3
math
trimkv
KV
Cache
Compression
ngocbh commited on
Commit
fbbc5b0
·
verified ·
1 Parent(s): 227597f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +13 -23
README.md CHANGED
@@ -15,34 +15,18 @@ tags:
15
 
16
  # TrimKV-Qwen3-8B-Math
17
 
18
- TRIM-KV is an efficient and learnable key–value eviction strategy designed to improve the efficiency of large language models (LLMs) in long-horizon inference.
19
 
20
- This model is a Qwen3-8B checkpoint trained with TRIM-KV on math reasoning tasks, as presented in the paper [Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction](https://huggingface.co/papers/2605.09649).
21
 
22
- - **Paper:** [https://huggingface.co/papers/2512.03324](https://huggingface.co/papers/2512.03324)
23
- - **GitHub:** [https://github.com/ngocbh/trimkv](https://github.com/ngocbh/trimkv)
24
 
25
- <a href="https://arxiv.org/pdf/2512.03324"><img src="https://img.shields.io/badge/arxiv-2512.03324-red?style=for-the-badge"></a>
26
 
27
- ### About TRIM-KV
 
 
28
 
29
- The core idea behind TRIM-KV is to learn the intrinsic importance of each key–value pair at creation time, which we call *token retention*, and then decay this importance exponentially over time to mimic the standard inference running with eviction.
30
-
31
- The retention score is query-agnostic and captures the long-term utility of tokens. This is different from attention scores, which are query-dependent: they capture the short-term utility for predicting the next token and are recomputed at every step, making them local, myopic, and highly dependent on the transient decoding state.
32
-
33
- ### Why TRIM-KV?
34
-
35
- It's fast, smart, and interpretable.
36
-
37
- <div align="center">
38
- <img width="1000" alt="speed" src="https://github.com/ngocbh/trimkv/blob/main/assets/speed.png?raw=true"/>
39
- </div>
40
-
41
- <div align="center">
42
- <img width="1000" alt="performance" src="https://github.com/ngocbh/trimkv/blob/main/assets/performance.png?raw=true"/>
43
- </div>
44
-
45
- ---
46
 
47
  ## Quick Start
48
 
@@ -94,6 +78,12 @@ tokenizer = AutoTokenizer.from_pretrained(
94
  ## Citation
95
 
96
  ```bibtex
 
 
 
 
 
 
97
  @article{bui2025make,
98
  title={Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction},
99
  author={Bui, Ngoc and Nguyen, Hieu Trung and Cohan, Arman and Ying, Rex},
 
15
 
16
  # TrimKV-Qwen3-8B-Math
17
 
18
+ > **TRIM-KV** is an efficient and learnable key–value eviction strategy designed to improve the efficiency of large language models (LLMs) in long-horizon inference.
19
 
20
+ This model is a version of `TrimKV-Qwen3-8B-Math` enhanced with TRIM-KV, as presented in the paper [Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs](https://huggingface.co/papers/2512.03324).
21
 
22
+ The core idea behind TRIM-KV is to learn the intrinsic importance of each key–value pair at creation time, which we call *token retention*, and then decay this importance exponentially over time to mimic standard inference running with eviction.
 
23
 
24
+ The retention score is query-agnostic and captures the long-term utility of tokens. This is different from attention scores, which are query-dependent: they capture the short-term utility for predicting the next token and are recomputed at every step.
25
 
26
+ - **Paper:** [Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs](https://huggingface.co/papers/2605.09649)
27
+ - **Code:** [GitHub - ngocbh/trimkv](https://github.com/ngocbh/trimkv)
28
+ - **Arxiv:** [2512.03324](https://arxiv.org/abs/2512.03324)
29
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
 
31
  ## Quick Start
32
 
 
78
  ## Citation
79
 
80
  ```bibtex
81
+ @article{bui2025cache,
82
+ title={Cache what lasts: Token retention for memory-bounded kv cache in llms},
83
+ author={Bui, Ngoc and Sharma, Shubham and Lamba, Simran and Mishra, Saumitra and Ying, Rex},
84
+ journal={arXiv preprint arXiv:2512.03324},
85
+ year={2025}
86
+ }
87
  @article{bui2025make,
88
  title={Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction},
89
  author={Bui, Ngoc and Nguyen, Hieu Trung and Cohan, Arman and Ying, Rex},