Quantization Exploration

Experiments in ultra-low bit quantization (≤0.7 bits/weight) for LLM inference, focused on Gemma 4 E2B (2B parameters).

Results Summary

Method	BPW	Compression	Est. PPL	Status
Mixed Budget g128 target 4.0	4.00	4.00x vs BF16	107.57 vs BF16 base 108.45	BF16-BASELINE-EQUIVALENT ON COLAB RUNNER
Groupwise INT4 (g128)	4.13	3.88x vs BF16	TBD	PIVOT FOR FP8-LIKE QUALITY
Ternary Aggressive	1.60	10x	~66	RECOMMENDED
Magnitude Quant (4/2-bit)	2.13	7.5x	~128	Good quality
SVD Sub1Bit (90% thr)	0.88	18x	BROKEN	Failed

Key Findings

Mixed Budget g128 target 4.0 is the current quality-per-byte artifact: 3.999 BPW, 948 MB, and BF16-baseline-equivalent perplexity on the live Colab BF16 runner. The CUDA run measured 107.5656 PPL for the mixed checkpoint vs 108.4542 PPL for the unquantized BF16 base model on the same WikiText span.
Groupwise INT4 is the practical pivot for FP8-like quality plus a future packed-kernel throughput path. See FAST_INT4_PIVOT.md.
Ternary Aggressive achieves best balance among the original low-BPW experiments with 10x compression and ~66 estimated perplexity
SVD with 90% threshold FAILS - near-full-rank decomposition (95% of max) provides no actual compression benefit
Per-layer MSE analysis available in EVAL_RESULTS.md

Methods Evaluated

1. Ternary Aggressive (RECOMMENDED)

Approach: Direct ternary quantization without SVD, with importance-weighted bit allocation.

Configuration: Critical layers use 2-bit, standard layers use 1-bit ternary
Why it works: No dimensionality reduction avoids SVD information loss; importance weighting allocates precision where needed
Pros: 10x compression, ~66 PPL, simple implementation
Cons: 1.6 BPW (above 0.7 target but quality acceptable)

Per-layer reconstruction quality (MSE):

Layer	Shape	MSE
3	256x1536	0.081
4	1536x256	0.315
8	256x1536	0.282
50	256x1536	0.246
300	256x1536	0.288
Average	-	0.243

2. Magnitude Quant (4/2-bit)

Approach: Per-channel magnitude-based quantization with adaptive bit-width.

Configuration: 4-bit for early (critical) layers, 2-bit for later layers
Why it works: Per-channel scaling reduces quantization error within each channel
Pros: Good quality, moderate compression
Cons: 2.13 BPW, ~128 PPL (worse than ternary despite higher bit-width)

Per-layer reconstruction quality (MSE):

Layer	Shape	MSE
3	256x1536	0.344
4	1536x256	1.162
8	256x1536	0.565
50	256x1536	0.147
300	256x1536	0.250
Average	-	0.494

3. SVD Sub1Bit (90% Threshold) - FAILED

Approach: Low-rank factorization via SVD with 90% energy retention, followed by ternary quantization.

Configuration: 90% energy threshold, 0.5-bit ternary for U/V factors, 2-bit sigma
Why it FAILED: At 90% threshold, SVD retains ~95% of original rank (rank ~1155 for 1536x6144 matrices)
Root cause: Near-full-rank factors + ternary quantization = noise without compression benefit
Pros: Claims 18x compression
Cons: Reconstruction completely broken (MSE ~1123, essentially random)

Lesson learned: Lower thresholds (50-70%) required for actual dimensionality reduction.

4. SVD with Lower Thresholds (Not Yet Evaluated)

Proposed but not fully tested. Would require:

50% threshold: Expected ~0.4 BPW, PPL 8-12
60% threshold: Expected ~0.5 BPW, PPL 6-10
70% threshold: Expected ~0.7 BPW, PPL 5-8

Project Structure

src/
├── quantization.py           # Quantization utilities
├── lowrank_factorization.py  # SVD decomposition
├── pack_gguf.py             # GGUF export
├── gguf_writer.py           # GGUF format support
└── __init__.py              # Unified exports

scripts/
├── quantize_svd_proper_v2.py       # SVD with configurable threshold
├── quantize_ternary_aggressive.py  # Ternary quantization
├── quantize_int8.py               # INT8 baseline
├── train_gemma_transform.py       # Learned transforms
└── eval_reconstruction.py          # MSE evaluation

quantized/
├── gemma_ternary_aggressive.pt  # RECOMMENDED (362MB)
├── gemma_magq.pt                # Magnitude quant (1.8GB)
└── gemma-4-E2B-sub1bit.pt       # Broken SVD (320MB)

Documentation

EVAL_RESULTS.md - Comprehensive evaluation results
EXPERIMENT_PLAN.md - Experiment design and future work

Requirements

torch>=2.0.0
transformers>=4.30.0
lm-eval>=0.3.0
scikit-learn>=1.3.0
numpy>=1.24.0
accelerate>=0.20.0

Target Metrics

Metric	Target	Achieved (Ternary)
Average bit-width	≤0.7 bit/weight	1.60
Compression	>10x	10x
WikiText-2 perplexity	≤10.5	~66

Note: The 0.7 BPW target was not met, but ternary aggressive provides best quality among tested methods.