samudhan2003 commited on
Commit
89c2b46
·
verified ·
1 Parent(s): 2c7c4d8

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +21 -0
README.md CHANGED
@@ -1,3 +1,24 @@
1
  ---
2
  license: mit
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ tags:
4
+ - kv-cache
5
+ - quantization
6
+ - efficient-inference
7
+ - llm-optimization
8
+ - memory-optimization
9
+ - inference
10
  ---
11
+
12
+ # KV-Cache-Compression-Lab: Systematic Analysis of KV-Cache Quantization for Efficient LLM Inference
13
+
14
+ **Author:** Samudhan S (@samudhan2003)
15
+ **Portfolio:** Part of 5-project Efficient Inference Series
16
+
17
+ ## Abstract
18
+
19
+ The Key-Value (KV) cache is a major memory bottleneck in autoregressive LLM inference, scaling linearly with sequence length and batch size. This lab presents a systematic empirical analysis of KV-cache compression techniques (FP16 baseline vs INT8 vs INT4) and their interaction with weight quantization, measuring the trade-off between VRAM reduction, latency improvement, and quality degradation on sub-1B scale models.
20
+
21
+ ## Motivation
22
+
23
+ During decoding, KV-cache memory often exceeds model weight memory:
24
+ - For Llama-2