Quantization

1 Spare bits

A trained weight like 0.0143729 is stored in 16-bit floating point, but the model barely cares about the trailing digits — round it to 0.014 and the network behaves almost identically. Multiply that slack across billions of weights and you have a lot of bits doing nothing. Quantization reclaims them, representing each weight with a small integer instead of a float.

The payoff is twofold. The model shrinks — int8 halves it, int4 quarters it — so it fits on smaller or fewer GPUs. And because decode is memory-bound (Chapter 22), moving fewer bytes per weight directly speeds it up: halving the weight size roughly doubles the achievable decode throughput. Smaller and faster, for the price of a little precision.

2 Rounding to a grid

Quantizing to B bits means snapping every weight to one of 2^B evenly spaced levels. You pick a scale that maps the floating-point range onto the integer range, round each weight to the nearest integer, and store that:

With 8 bits you have 256 levels — fine enough that the rounding error is tiny and the model is all but unchanged. With 4 bits you have 16 levels, and the spacing between them starts to matter. The error you introduce is exactly the gap between each weight and its nearest level, so everything comes down to placing those levels well.

3 One scale isn't enough

The naive approach uses a single scale for the entire weight matrix — per-tensor quantization. It's simple, but it forces every weight, large and small, onto the same grid. If one part of the matrix has a much wider range than another, the levels get stretched to cover the widest part, leaving the rest of the weights crammed coarsely between a few of them.

The fix is to use more scales. Per-channel quantization gives each row (or column) its own scale, so a quiet channel gets a fine grid and a loud one gets a coarse grid, each appropriate to its range. Group-wise quantization goes further, a separate scale for every block of, say, 128 weights — the modern default, because the handful of extra scale values costs almost nothing and the error drops sharply.

4 The outlier problem

Here's what actually breaks low-bit quantization: outliers. A small number of weights — and, even more troublesome, a small number of activation channels — have magnitudes many times larger than everything else. Because the scale is set by the maximum, a single outlier stretches the grid so wide that all the ordinary weights collapse onto just a couple of levels. One weight ruins the precision of thousands.

The methods you hear about — GPTQ, AWQ, and friends — are mostly clever ways to handle outliers. Some keep the salient weights in higher precision while quantizing the rest; AWQ scales weights by how much their activations matter, protecting the ones that do; GPTQ adjusts the remaining weights to compensate for the error introduced. All of them exist because, without outlier handling, four-bit quantization falls apart.

5 Quality vs size

Put the curve together and a clear story emerges. Int8 is, for most models, effectively lossless — half the size, no measurable quality drop, an easy win. Int4 with group-wise scales and proper outlier handling loses very little and is the workhorse of local model serving. Push below four bits — three, two — and quality starts dropping off a cliff that no amount of cleverness fully arrests.

So quantization isn't a free lunch all the way down; it's a steep trade-off curve, and the sweet spot for inference today sits around 4 bits. The same arithmetic applies to the KV cache (Chapter 13) and even to training in lower precision — but the principle is always the one the playground makes visible: find the bits that aren't carrying information, and only those.

6 Reading the playground

A real weight tensor — a realistic distribution with a few genuine outliers — is quantized live at the bit-width and method you choose, and the reconstruction error is measured directly. The numbers are exact arithmetic on the tensor in front of you.