metadata
library_name: nexusquant
tags:
- kv-cache
- quantization
- e8-lattice
- llm
- inference
- compression
license: mit
NexusQuant: E8 Lattice KV Cache Compression
Training-free KV cache compression for LLM inference. Uses E8 lattice vector quantization + Hadamard rotation. Calibration-free.
Headline
+0.276% wikitext PPL at 5.83x compression (Mistral-7B). NIAH retrieval preserved through 32K context. Validated on 9 architectures.
Head-to-head (Llama-3.1-8B-Instruct, 4K, n=30)
| Method | bpe | NIAH |
|---|---|---|
| FP16 | 16.0 | 29/30 |
| TurboQuant 2-bit | 2.125 | 0/30 |
| NexusQuant K2V2 | 2.0 | 30/30 |
Install
pip install nexusquant-kv
Usage
from nexusquant import compress_kv_cache
with compress_kv_cache(model, mode="quant_only", bits=2):
output = model.generate(input_ids, max_new_tokens=200)