Link RAB leaderboard

44307b4 verified about 2 months ago

8.08 kB

	---
	license: apache-2.0
	language:
	- en
	tags:
	- quantization
	- vector-quantization
	- llm-compression
	- research
	- methodology
	library_name: mlx
	---

	# Vector Quantization for LLM weights: a real 25% win at 3 bits — and why every cheaper number is a lie

	An honest sweep of VQ vs scalar quantization, validated the only way that counts: stacked real-data perplexity.

	> 📊 Companion agentic-LLM leaderboard, same "measure honestly" philosophy: [rab.utopiaia.com](https://rab.utopiaia.com) — RAB, the Real-world Agentic Benchmark.

	## TL;DR

	- At ~3 bits/weight, vector quantization (VQ) keeps a model essentially intact (perplexity 1.11× baseline) where ordinary scalar quantization at the same bitrate is destroyed (5.36×). Scalar needs a full extra bit (4 b/w) to match VQ's 3-bit quality — i.e. VQ is ~25% smaller than Q4 at equal quality.
	- Below 3 bits, naïve VQ collapses (2-bit VQ → 12× perplexity). The sub-1-bit and 2-bit dreams are not reachable without calibration + error feedback — at which point you have simply reimplemented AQLM / QuIP#.
	- The headline methodological result: weight-MSE and single-tensor output-fidelity both lie. A 2-bit VQ config that looked near-lossless on per-tensor cosine (0.991) was 12× worse on real perplexity. Only stacked, real-data perplexity predicts model quality. If you are evaluating a quantizer on per-tensor error, you are measuring the wrong thing.

	## The idea

	Scalar quantization rounds each weight independently to a low-bit grid (RTN), with a shared scale per block. Vector quantization instead groups each weight matrix into sub-vectors of length `dim`, clusters them into `K` shared centroids (a codebook), and stores one index per sub-vector. The cost is:

	```
	bits/weight = log2(K) / dim (+ a small codebook + scale overhead)
	```

	So `dim=4, K=4096` → 12/4 = 3 bits/weight; `dim=4, K=256` → 8/4 = 2 bits; `dim=16, K=4096` → 0.77 bits. The promise: a codebook captures correlations between adjacent weights that scalar rounding throws away — so the same bit budget should buy more fidelity.

	That promise is real, but only in a narrow band, and only if you measure it correctly.

	## Measuring it correctly

	We evaluated three ways, in increasing order of honesty:

	1. Weight MSE — reconstruction error of the matrix itself. Cheap, and misleading: it has no idea which errors the forward pass amplifies.
	2. Single-tensor output fidelity — feed random inputs `x` through `W` and `W_recon`, measure output cosine. Better in principle, but with Gaussian inputs it still lied to us (see below).
	3. Stacked real-data perplexity — quantize every layer, run the whole model on real text, measure perplexity. This is the only metric that tracks what a user would actually feel.

	The trap is error accumulation. Per-tensor metrics look at one layer in isolation. But a 30-layer model stacks 30 small distortions, each feeding the next; a per-layer cosine of 0.99 does not mean a stacked cosine of 0.99. The accumulation is the whole story, and only metric (3) sees it.

	## The sweep

	Stacked real perplexity on a 2B base model (baseline ppl 1.89):

	\| method \| bits/w \| perplexity \| ratio vs baseline \|
	\|---------------\|:------:\|:----------:\|:-----------------:\|
	\| scalar 2-bit \| 3.00 \| 10.14 \| 5.36× \|
	\| scalar 3-bit \| 4.00 \| 1.97 \| 1.04× \|
	\| scalar 4-bit \| 5.00 \| 1.91 \| 1.01× \|
	\| VQ d4 K4096 \| 3.02 \| 2.10 \| 1.11× \|
	\| VQ d4 K256 \| 2.00 \| 22.80 \| 12.04× \|
	\| VQ d2 K256 \| 4.00 \| 2.03 \| 1.07× \|

	(Scalar "N-bit" carries an extra ~1 b/w of block-scale overhead, hence the bits/w column.)

	Read the table at iso-bitrate:

	- At ~3 b/w, VQ (1.11×) crushes scalar (5.36×). Scalar has to spend a full extra bit to catch up. This is the win: VQ-3bit ≈ scalar-4bit quality at 25% less size.
	- At ~4 b/w they tie (scalar marginally better, and ~1000× faster to compute). Above 3 bits there is no reason to pay VQ's encoding cost.
	- At 2 b/w naïve VQ is broken (12×) — exactly like naïve scalar at 2 bits is broken. This is the universal "RTN-at-2-bits destroys models, which is why GPTQ exists" result, and VQ does not escape it for free.

	## The mirage we chased and killed

	Per-tensor output fidelity made `dim=4, K=256` @ 2 bits look like a free lunch: output cosine 0.991, worst-case 0.98 — visually indistinguishable from the original at half the bits. Stacked, it is 12× worse perplexity. That single result is the most important thing in this writeup: we nearly shipped a 2-bit config on the strength of a per-tensor number, and it would have produced garbage.

	We also tried to rescue sub-1-bit fidelity with a sparse high-precision residual on the worst sub-vectors (the natural "fix the outliers" move). Refuted. VQ error is diffuse — spread thinly across many sub-vectors, not concentrated in a few — so a sparse correction plus its locator bitmap wastes the budget: reaching 0.97 fidelity sub-1-bit costs more total bits than just using `dim=4, K=256` @ 2 bits. The cheap-and-good permutation and the expensive-and-good one are never the same one. (The identical trap sinks "reorder weights to compress" schemes.)

	## Where the real 2-bit lives

	Getting to a usable 2 bits is not impossible — it just isn't free. It requires calibration data, per-channel scales, and GPTQ-style error feedback that pushes each layer's quantization error into the next layer's weights. That is precisely the machinery of AQLM and QuIP#. The honest framing: the 3-bit VQ result stands on its own as a solid, shippable compression win; the 2-bit result is a research program, not a config flag.

	## Practical takeaways

	1. If you quantize, evaluate on stacked real perplexity. Per-tensor MSE and single-tensor cosine will happily approve a model that generates noise.
	2. 3-bit VQ is a genuine ~25% saving over 4-bit scalar at matched quality. If you are RAM-bound and already at Q4, VQ-3bit is the next real step down.
	3. Don't believe sub-2-bit claims that aren't backed by stacked perplexity and an error-feedback method. The bits/weight number is the easy part; keeping the model alive at that bitrate is the hard part.

	---

	Methodology and code are public. Reconstruction, packing (12-bit index bit-packing → true 3 b/w on disk) and the stacked-perplexity harness are in the repository. Model training details are out of scope here — this is a study of weight quantization on public base checkpoints.

	---

	## Repository contents (code)

	This repo is the method + code, reproducible on any public base checkpoint. No model weights are shipped here (the on-disk `.vqz` format is non-standard; a standard-format model card is a separate effort).

	\| file \| what it does \|
	\|------\|--------------\|
	\| `vq_sweep.py` \| per-tensor VQ bitrate sweep + output fidelity (the metric that turned out to lie) \|
	\| `vq_stacked_sweep.py` \| the definitive result: loads the model once, restores+quantizes per config, measures real stacked perplexity \|
	\| `vq_stacked_ppl.py` \| single-config stacked perplexity harness \|
	\| `vq_residual_lever.py` \| sub-1-bit residual lever — refuted (VQ error is diffuse, sparse correction wastes the budget) \|
	\| `vq_pack.py` \| produces a real on-disk artifact (~3 b/w) via 12-bit index bit-packing; round-trip verified \|
	\| `vq_load.py` \| reconstructs a model from the packed artifact and generates text to verify coherence \|
	\| `vq_compress.py` \| core VQ encode/decode primitives \|

	### Reproduce the headline table

	```bash
	pip install -r requirements.txt
	python vq_stacked_sweep.py # stacked real perplexity, VQ vs scalar, per config
	```

	The 3-bit VQ win (≈25% smaller than 4-bit scalar at matched perplexity) and the "per-tensor metrics lie" result both fall out of this single script.

	Study of weight quantization on public base checkpoints. Model training details are out of scope.

	---
	license: apache-2.0
	language:
	- en
	tags:
	- quantization
	- vector-quantization
	- llm-compression
	- research
	- methodology
	library_name: mlx
	---

	# Vector Quantization for LLM weights: a real 25% win at 3 bits — and why every cheaper number is a lie

	An honest sweep of VQ vs scalar quantization, validated the only way that counts: stacked real-data perplexity.

	> 📊 Companion agentic-LLM leaderboard, same "measure honestly" philosophy: [rab.utopiaia.com](https://rab.utopiaia.com) — RAB, the Real-world Agentic Benchmark.

	## TL;DR

	- At ~3 bits/weight, vector quantization (VQ) keeps a model essentially intact (perplexity 1.11× baseline) where ordinary scalar quantization at the same bitrate is destroyed (5.36×). Scalar needs a full extra bit (4 b/w) to match VQ's 3-bit quality — i.e. VQ is ~25% smaller than Q4 at equal quality.
	- Below 3 bits, naïve VQ collapses (2-bit VQ → 12× perplexity). The sub-1-bit and 2-bit dreams are not reachable without calibration + error feedback — at which point you have simply reimplemented AQLM / QuIP#.
	- The headline methodological result: weight-MSE and single-tensor output-fidelity both lie. A 2-bit VQ config that looked near-lossless on per-tensor cosine (0.991) was 12× worse on real perplexity. Only stacked, real-data perplexity predicts model quality. If you are evaluating a quantizer on per-tensor error, you are measuring the wrong thing.

	## The idea

	Scalar quantization rounds each weight independently to a low-bit grid (RTN), with a shared scale per block. Vector quantization instead groups each weight matrix into sub-vectors of length `dim`, clusters them into `K` shared centroids (a codebook), and stores one index per sub-vector. The cost is:

	```
	bits/weight = log2(K) / dim (+ a small codebook + scale overhead)
	```

	So `dim=4, K=4096` → 12/4 = 3 bits/weight; `dim=4, K=256` → 8/4 = 2 bits; `dim=16, K=4096` → 0.77 bits. The promise: a codebook captures correlations between adjacent weights that scalar rounding throws away — so the same bit budget should buy more fidelity.

	That promise is real, but only in a narrow band, and only if you measure it correctly.

	## Measuring it correctly

	We evaluated three ways, in increasing order of honesty:

	1. Weight MSE — reconstruction error of the matrix itself. Cheap, and misleading: it has no idea which errors the forward pass amplifies.
	2. Single-tensor output fidelity — feed random inputs `x` through `W` and `W_recon`, measure output cosine. Better in principle, but with Gaussian inputs it still lied to us (see below).
	3. Stacked real-data perplexity — quantize every layer, run the whole model on real text, measure perplexity. This is the only metric that tracks what a user would actually feel.

	The trap is error accumulation. Per-tensor metrics look at one layer in isolation. But a 30-layer model stacks 30 small distortions, each feeding the next; a per-layer cosine of 0.99 does not mean a stacked cosine of 0.99. The accumulation is the whole story, and only metric (3) sees it.

	## The sweep

	Stacked real perplexity on a 2B base model (baseline ppl 1.89):

	\| method \| bits/w \| perplexity \| ratio vs baseline \|
	\|---------------\|:------:\|:----------:\|:-----------------:\|
	\| scalar 2-bit \| 3.00 \| 10.14 \| 5.36× \|
	\| scalar 3-bit \| 4.00 \| 1.97 \| 1.04× \|
	\| scalar 4-bit \| 5.00 \| 1.91 \| 1.01× \|
	\| VQ d4 K4096 \| 3.02 \| 2.10 \| 1.11× \|
	\| VQ d4 K256 \| 2.00 \| 22.80 \| 12.04× \|
	\| VQ d2 K256 \| 4.00 \| 2.03 \| 1.07× \|

	(Scalar "N-bit" carries an extra ~1 b/w of block-scale overhead, hence the bits/w column.)

	Read the table at iso-bitrate:

	- At ~3 b/w, VQ (1.11×) crushes scalar (5.36×). Scalar has to spend a full extra bit to catch up. This is the win: VQ-3bit ≈ scalar-4bit quality at 25% less size.
	- At ~4 b/w they tie (scalar marginally better, and ~1000× faster to compute). Above 3 bits there is no reason to pay VQ's encoding cost.
	- At 2 b/w naïve VQ is broken (12×) — exactly like naïve scalar at 2 bits is broken. This is the universal "RTN-at-2-bits destroys models, which is why GPTQ exists" result, and VQ does not escape it for free.

	## The mirage we chased and killed

	Per-tensor output fidelity made `dim=4, K=256` @ 2 bits look like a free lunch: output cosine 0.991, worst-case 0.98 — visually indistinguishable from the original at half the bits. Stacked, it is 12× worse perplexity. That single result is the most important thing in this writeup: we nearly shipped a 2-bit config on the strength of a per-tensor number, and it would have produced garbage.

	We also tried to rescue sub-1-bit fidelity with a sparse high-precision residual on the worst sub-vectors (the natural "fix the outliers" move). Refuted. VQ error is diffuse — spread thinly across many sub-vectors, not concentrated in a few — so a sparse correction plus its locator bitmap wastes the budget: reaching 0.97 fidelity sub-1-bit costs more total bits than just using `dim=4, K=256` @ 2 bits. The cheap-and-good permutation and the expensive-and-good one are never the same one. (The identical trap sinks "reorder weights to compress" schemes.)

	## Where the real 2-bit lives

	Getting to a usable 2 bits is not impossible — it just isn't free. It requires calibration data, per-channel scales, and GPTQ-style error feedback that pushes each layer's quantization error into the next layer's weights. That is precisely the machinery of AQLM and QuIP#. The honest framing: the 3-bit VQ result stands on its own as a solid, shippable compression win; the 2-bit result is a research program, not a config flag.

	## Practical takeaways

	1. If you quantize, evaluate on stacked real perplexity. Per-tensor MSE and single-tensor cosine will happily approve a model that generates noise.
	2. 3-bit VQ is a genuine ~25% saving over 4-bit scalar at matched quality. If you are RAM-bound and already at Q4, VQ-3bit is the next real step down.
	3. Don't believe sub-2-bit claims that aren't backed by stacked perplexity and an error-feedback method. The bits/weight number is the easy part; keeping the model alive at that bitrate is the hard part.

	---

	Methodology and code are public. Reconstruction, packing (12-bit index bit-packing → true 3 b/w on disk) and the stacked-perplexity harness are in the repository. Model training details are out of scope here — this is a study of weight quantization on public base checkpoints.

	---

	## Repository contents (code)

	This repo is the method + code, reproducible on any public base checkpoint. No model weights are shipped here (the on-disk `.vqz` format is non-standard; a standard-format model card is a separate effort).

	\| file \| what it does \|
	\|------\|--------------\|
	\| `vq_sweep.py` \| per-tensor VQ bitrate sweep + output fidelity (the metric that turned out to lie) \|
	\| `vq_stacked_sweep.py` \| the definitive result: loads the model once, restores+quantizes per config, measures real stacked perplexity \|
	\| `vq_stacked_ppl.py` \| single-config stacked perplexity harness \|
	\| `vq_residual_lever.py` \| sub-1-bit residual lever — refuted (VQ error is diffuse, sparse correction wastes the budget) \|
	\| `vq_pack.py` \| produces a real on-disk artifact (~3 b/w) via 12-bit index bit-packing; round-trip verified \|
	\| `vq_load.py` \| reconstructs a model from the packed artifact and generates text to verify coherence \|
	\| `vq_compress.py` \| core VQ encode/decode primitives \|

	### Reproduce the headline table

	```bash
	pip install -r requirements.txt
	python vq_stacked_sweep.py # stacked real perplexity, VQ vs scalar, per config
	```

	The 3-bit VQ win (≈25% smaller than 4-bit scalar at matched perplexity) and the "per-tensor metrics lie" result both fall out of this single script.

	Study of weight quantization on public base checkpoints. Model training details are out of scope.