File size: 867 Bytes
b50dab2 89c2b46 b50dab2 89c2b46 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | ---
license: mit
tags:
- kv-cache
- quantization
- efficient-inference
- llm-optimization
- memory-optimization
- inference
---
# KV-Cache-Compression-Lab: Systematic Analysis of KV-Cache Quantization for Efficient LLM Inference
**Author:** Samudhan S (@samudhan2003)
**Portfolio:** Part of 5-project Efficient Inference Series
## Abstract
The Key-Value (KV) cache is a major memory bottleneck in autoregressive LLM inference, scaling linearly with sequence length and batch size. This lab presents a systematic empirical analysis of KV-cache compression techniques (FP16 baseline vs INT8 vs INT4) and their interaction with weight quantization, measuring the trade-off between VRAM reduction, latency improvement, and quality degradation on sub-1B scale models.
## Motivation
During decoding, KV-cache memory often exceeds model weight memory:
- For Llama-2
|