File size: 1,280 Bytes
ff7679d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | # Required native Raon FP8 KV patch
Based on `krafton-ai/vllm` revision `9886842e20102e330389a505debe80d4c834940a`.
Copy `mla_attention.py` and `kv_cache_quant.py` into `vllm/models/raon1_mini/`
in that pinned source tree before building the runtime.
The relative import requires the helper beside the attention file.
Keep the remaining Raon implementation at the same revision.
The adapter is scoped to Raon attention with the original ModelOpt MXFP8 configuration
and explicit `kv_cache_quant_algo="FP8"`. Projection and expert quantization are unchanged.
It creates scalar KV parameters through BaseKVCacheMethod and applies native scale
finalization before Raon absorbs weights and computes reciprocal scales.
The cached-prefix/chunked-prefill helper multiplies gathered plain-FP8 bytes by the
K scale before latent projection or use of the positional tail.
Calibration targets E4M3FN with a maximum of 448. It does not qualify E5M2,
`fp8_ds_mla`, or NVFP4 cache execution. Existing BF16 gather paths remain unchanged.
CPU contracts verify scalar loading, finalization, reciprocal scales, and nonunit
cached-prefix restoration. Full native GPU serving and cached-prefix/chunked-prefill
GPU execution remain untested. Do not infer deployment acceptance from these checks.
|