anonymousmaharaj/vllm-cross-model-kv

Cross-model KV mappers: Qwen3-0.6B to Qwen3-1.7B

Ridge mappers that turn the KV cache of Qwen3-0.6B into an approximate KV cache of Qwen3-1.7B, so the bigger model can skip most of its prefill. Independent implementation of arXiv:2608.03893 (NVIDIA, closed-form linear KV mapping inside a model family).

Serving code, fit pipeline and raw benchmarks: https://github.com/alesha-pro/vllm-cross-model-kv (runs on stock vLLM 0.26.0 as a KV connector, no fork).

Two artifacts, both fit on FineWeb-Edu (1024-token sequences, stride 4, lambda 0.01, RoPE-stripped K), each with fp32 and bf16 weights plus the full calibration manifest:

mapper	calibration	held-out R²_K	R²_V	HellaSwag-1000 retention	WikiText tail-PPL ratio
ridge-qwen3-0.6b-to-1.7b-k8-n500	500 seqs, k=8 (paper config)	0.760	0.625	0.940	1.100
ridge-qwen3-0.6b-to-1.7b-k4-n200	200 seqs, k=4	0.731	0.574	0.953	1.090

k8-n500 has the best R² and was used for the headline latency numbers (warm TTFT 2.22x over full prefill at 32K on two RTX 3090s). k4-n200 maps about twice as fast because the GEMMs are half as wide, at near-equal downstream quality. Pick k8 for quality, k4 for mapper speed.

Usage: point mapper_path of the CrossModelKVConnector consumer at a mapper directory. See the GitHub README for the full pair setup.

Trained and measured on 2x RTX 3090. Model pair: Qwen/Qwen3-0.6B -> Qwen/Qwen3-1.7B (same tokenizer, matched KV geometry: 8 KV heads, head_dim 128, 28 layers). These mappers are useless for any other pair.