| --- |
| license: mit |
| tags: |
| - kv-cache |
| - quantization |
| - efficient-inference |
| - llm-optimization |
| - memory-optimization |
| - inference |
| --- |
| |
| # KV-Cache-Compression-Lab: Systematic Analysis of KV-Cache Quantization for Efficient LLM Inference |
|
|
| **Author:** Samudhan S (@samudhan2003) |
| **Portfolio:** Part of 5-project Efficient Inference Series |
|
|
| ## Abstract |
|
|
| The Key-Value (KV) cache is a major memory bottleneck in autoregressive LLM inference, scaling linearly with sequence length and batch size. This lab presents a systematic empirical analysis of KV-cache compression techniques (FP16 baseline vs INT8 vs INT4) and their interaction with weight quantization, measuring the trade-off between VRAM reduction, latency improvement, and quality degradation on sub-1B scale models. |
|
|
| ## Motivation |
|
|
| During decoding, KV-cache memory often exceeds model weight memory: |
| - For Llama-2 |
|
|
