QUANTIZATION.md

OpenMOSS-Team/MOSS-VL-Realtime-FP8

• Runtime: Transformers online_generate / create_realtime_session.
• Weights: compressed-tensors FP8 E4M3 on 252 self-attention/MLP Linear layers.
• Activations: FP8 per-token dynamic on the same 252 Linear inputs.
• BF16: cross-attention layers, vision encoder/merger, embeddings, norms and lm_head.
• KV cache: Transformers HQQ INT8, group size 64, residual length 128.
• Attention backend: flash_attention_2.