File size: 2,720 Bytes
48b1885 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
Field | Response
:------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------
Intended Task/Domain: | KV cache pruning / inference optimization for transformer-based language models (prefilling and decoding).
Model Type: | Feed-forward surrogate (Linear or 2-layer MLP) applied to transformer hidden states.
Intended Users: | ML researchers and engineers integrating KV cache pruning into LLM inference stacks and benchmarking long-context / long-decoding performance.
Output: | Numeric pruning scores in log-space with shape \((T, H)\) (per token, per KV head), used to threshold which KV pairs to retain and to enforce a local sliding window.
Describe how the model works: | KVzap maps transformer hidden states \((T, D_h)\) to per-token, per-KV-head scores \((T, H)\). Tokens below a threshold \(\tau\) are pruned from the KV cache; a recent sliding window is always retained.
Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Not Applicable.
Technical Limitations & Mitigation: | Over-pruning can degrade the host model; mitigate by selecting \(\tau\) conservatively and validating on target tasks.
Verified to have met prescribed NVIDIA quality standards: | Yes.
Performance Metrics: | Downstream accuracy under compression (e.g., RULER, LongBench, AIME25), compression ratio, and runtime overhead (KVzap compute/memory overhead is designed to be negligible relative to transformer layers).
Potential Known Risks: | The scores predicted by KVzap are used to prune the KV cache. But KV Pruning may change host-model (LLM) outputs even with low thresholds as such models were not trained with KV cache Pruning.
Licensing: | [Apache License 2.0.](https://www.apache.org/licenses/LICENSE-2.0)
|