spellbound-eng/hy3-spellbound-simpo-step151-nvfp4-c8

hy3-spellbound-simpo-step151 — NVFP4 + FP8 KV cache

Quantized build of spellbound-eng/hy3-spellbound-simpo-step151-merged-bf16 (HY3 MoE, 80 layers, 192 experts + 1 shared, ~290B params) produced with Tencent AngelSlim's documented HY3 NVFP4 weight-only + FP8-KV pipeline (docs/source/features/quantization/nvfp4.md).

Format

• MoE expert weights: NVFP4 (E2M1, packed 2/byte), per-16-group FP8-E4M3 weight_scale + per-tensor FP32 weight_scale_2
• Expert activations: per-expert FP32 input_scale (gate/up/down; 45,504 scales)
• KV cache: FP8 per-tensor k_scale / v_scale on all 80 layers (absmax/448 from calibration min/max)
• Everything else BF16: attention, router, shared expert, dense layer 0, embeddings, lm_head
• config.json carries a modelopt-style quantization_config (quant_algo=NVFP4, group_size 16, FP8 kv_cache_scheme); hf_quant_config.json included

How it was made

1. tools/run.py with configs/Hy3/ptq/nvfp4_weight_only/*.yaml (data-free, scales from weight absmax, cpu_convert: true)
2. tools/run_vllm_calibrate.py (patched vLLM 0.20.0, VLLM_MOE_COLLECT_STATS=1, TP=8) over 512 conversations from an internal chat dataset — the 512 shortest validated conversations (21,149–24,909 tokens), max_length 25,000, no truncation, KV-scale search enabled (search_kv_scale: true)
3. tools/merge_hy3_nvfp4_c8.py merging the NVFP4 checkpoint with the calibration statistics and restoring BF16 shared-expert weights from the base model

Serving

Requires NVFP4-capable hardware (NVIDIA Blackwell) and a vLLM build with modelopt NVFP4 + FP8 KV cache support. Multi-token prediction (MTP) weights are not included; serve without speculative MTP.