Language models generate one word at a time — which is slow and expensive at scale. This project implements and benchmarks the three techniques engineers use to serve them faster: smarter scheduling, weight compression, and memory management.
When you send a message to ChatGPT, the model generates each word one at a time. Every single word requires a full computation pass through billions of parameters. Doing this naively — one user at a time, sequentially — wastes most of the GPU's capacity.
Instead of waiting for one user's response to finish before helping the next user, fill the GPU's processing slots the moment any slot opens up. This keeps GPU utilization near 100% instead of ~30%. Used in vLLM and HuggingFace TGI.
Neural network weights are normally stored as 32-bit decimal numbers. Compressing them to 4-bit integers saves 75% of memory and speeds up computation — with less than 1% quality loss when done correctly (NF4 format).
As a model generates text, it needs to remember everything it wrote (called the KV cache). Naively reserving maximum memory for every conversation wastes 65% of GPU memory on average. PagedAttention uses a virtual-memory approach — only allocating what's actually needed.
All benchmarks are pre-computed — no API key or GPU needed. Use the tabs above to explore each technique: throughput charts, memory/speed tradeoffs, and the actual Python implementation.
Static batching waits for the slowest request in a batch to finish before starting the next batch — wasting GPU cycles on idle slots. Continuous batching fills those slots immediately, treating the GPU like a conveyor belt instead of a bucket.
Standard quantization divides the numeric range into equal buckets. NF4 (Normal Float 4) places buckets where most model weights actually cluster — near zero, following a bell curve. This matches how LLM weights are distributed, preserving precision where it matters most.
Memory = 2 × n_layers × n_heads × head_dim × seq_len × batch × bytes
Pre-allocating the maximum sequence length for every conversation wastes 65% of GPU memory on unused space. PagedAttention stores the KV cache in fixed 16-token pages and only allocates new pages as they're needed — like how your OS manages RAM, not how a naive array works.