Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization

Recent work shows that increasing embedding capacity can improve language models, making it a promising direction for scaling. As more capacity goes into embedding tables, extreme compression becomes increasingly important: we want to keep the benefits of larger embeddings while fitting models into limited deployment memory.

We study this in Gemma 4, where per-layer embeddings (PLE) account for 35.2% of E4B's parameters. At 100× compression, only about five bytes remain for each token–layer vector. Encoding every vector independently gives each one a tiny code and misses the opportunity to share storage across layers.

Our method, OPTIC, gives each token one quantized code shared across all layers. Layer-specific affine decoders reconstruct the different PLE vectors from that shared code. We also fit the representation to its effect after the gate and output projection, so the compression objective reflects how the model actually uses the embeddings.

At about 100× PLE compression, OPTIC retains 97.8% of BF16 E4B performance across seven benchmarks, compared with 66.3% for product quantization. Serving BF16 E4B directly from the compact codes reduces GPU memory by 29.7%, with at most 0.40% more wall time on an RTX 4090 with the same 1 GiB KV cache. It also works alongside backbone weight quantization.

Paper · Code and method walkthrough · Released 100× PLE artifact