| library_name: nexusquant | |
| tags: | |
| - kv-cache | |
| - quantization | |
| - e8-lattice | |
| - llm | |
| - inference | |
| - compression | |
| license: mit | |
| # NexusQuant: E8 Lattice KV Cache Compression | |
| Training-free KV cache compression for LLM inference. Uses E8 lattice vector quantization + Hadamard rotation. Calibration-free. | |
| ## Headline | |
| +0.276% wikitext PPL at 5.83x compression (Mistral-7B). NIAH retrieval preserved through 32K context. Validated on 9 architectures. | |
| ## Head-to-head (Llama-3.1-8B-Instruct, 4K, n=30) | |
| | Method | bpe | NIAH | | |
| |---|---|---| | |
| | FP16 | 16.0 | 29/30 | | |
| | TurboQuant 2-bit | 2.125 | 0/30 | | |
| | NexusQuant K2V2 | 2.0 | **30/30** | | |
| ## Install | |
| ``` | |
| pip install nexusquant-kv | |
| ``` | |
| ## Usage | |
| ```python | |
| from nexusquant import compress_kv_cache | |
| with compress_kv_cache(model, mode="quant_only", bits=2): | |
| output = model.generate(input_ids, max_new_tokens=200) | |
| ``` | |
| ## Links | |
| - [Live demo](https://huggingface.co/spaces/jmarquex/nexusquant-demo) | |
| - [GitHub](https://github.com/jagmarques/nexusquant) | |
| - Papers: [Method](https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-method.pdf), [2-bit retrieval](https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-2bit-retrieval.pdf), [FP16 dead zones](https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-fp16-deadzones.pdf) | |
| - [llama.cpp PR](https://github.com/ggml-org/llama.cpp/pull/25352) | |
| - [vLLM PR](https://github.com/vllm-project/vllm/pull/47742) | |