---
library_name: nexusquant
tags:
  - kv-cache
  - quantization
  - e8-lattice
  - llm
  - inference
  - compression
license: mit
---

# NexusQuant: E8 Lattice KV Cache Compression

Training-free KV cache compression for LLM inference. Uses E8 lattice vector quantization + Hadamard rotation. Calibration-free.

## Headline

+0.276% wikitext PPL at 5.83x compression (Mistral-7B). NIAH retrieval preserved through 32K context. Validated on 9 architectures.

## Head-to-head (Llama-3.1-8B-Instruct, 4K, n=30)

| Method | bpe | NIAH |
|---|---|---|
| FP16 | 16.0 | 29/30 |
| TurboQuant 2-bit | 2.125 | 0/30 |
| NexusQuant K2V2 | 2.0 | **30/30** |

## Install

```
pip install nexusquant-kv
```

## Usage

```python
from nexusquant import compress_kv_cache

with compress_kv_cache(model, mode="quant_only", bits=2):
    output = model.generate(input_ids, max_new_tokens=200)
```

## Links

- [Live demo](https://huggingface.co/spaces/jmarquex/nexusquant-demo)
- [GitHub](https://github.com/jagmarques/nexusquant)
- Papers: [Method](https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-method.pdf), [2-bit retrieval](https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-2bit-retrieval.pdf), [FP16 dead zones](https://github.com/jagmarques/nexusquant/blob/main/paper/nexusquant-fp16-deadzones.pdf)
- [llama.cpp PR](https://github.com/ggml-org/llama.cpp/pull/25352)
- [vLLM PR](https://github.com/vllm-project/vllm/pull/47742)