| Field | Response | |
| :------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------- | |
| Intended Task/Domain: | KV cache pruning / inference optimization for transformer-based language models (prefilling and decoding). | |
| Model Type: | Feed-forward surrogate (Linear or 2-layer MLP) applied to transformer hidden states. | |
| Intended Users: | ML researchers and engineers integrating KV cache pruning into LLM inference stacks and benchmarking long-context / long-decoding performance. | |
| Output: | Numeric pruning scores in log-space with shape \((T, H)\) (per token, per KV head), used to threshold which KV pairs to retain and to enforce a local sliding window. | |
| Describe how the model works: | KVzap maps transformer hidden states \((T, D_h)\) to per-token, per-KV-head scores \((T, H)\). Tokens below a threshold \(\tau\) are pruned from the KV cache; a recent sliding window is always retained. | |
| Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Not Applicable. | |
| Technical Limitations & Mitigation: | Over-pruning can degrade the host model; mitigate by selecting \(\tau\) conservatively and validating on target tasks. | |
| Verified to have met prescribed NVIDIA quality standards: | Yes. | |
| Performance Metrics: | Downstream accuracy under compression (e.g., RULER, LongBench, AIME25), compression ratio, and runtime overhead (KVzap compute/memory overhead is designed to be negligible relative to transformer layers). | |
| Potential Known Risks: | The scores predicted by KVzap are used to prune the KV cache. But KV Pruning may change host-model (LLM) outputs even with low thresholds as such models were not trained with KV cache Pruning. | |
| Licensing: | [Apache License 2.0.](https://www.apache.org/licenses/LICENSE-2.0) | |