metadata
license: mit
tags:
- kv-cache
- quantization
- efficient-inference
- llm-optimization
- memory-optimization
- inference
KV-Cache-Compression-Lab: Systematic Analysis of KV-Cache Quantization for Efficient LLM Inference
Author: Samudhan S (@samudhan2003)
Portfolio: Part of 5-project Efficient Inference Series
Abstract
The Key-Value (KV) cache is a major memory bottleneck in autoregressive LLM inference, scaling linearly with sequence length and batch size. This lab presents a systematic empirical analysis of KV-cache compression techniques (FP16 baseline vs INT8 vs INT4) and their interaction with weight quantization, measuring the trade-off between VRAM reduction, latency improvement, and quality degradation on sub-1B scale models.
Motivation
During decoding, KV-cache memory often exceeds model weight memory:
- For Llama-2