Update README.md

README.md

@@ -89,6 +89,22 @@ VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve /path/to/Hy3-1M \
 `--kv-cache-dtype fp8` halves KV memory (recommended for long context). On Hopper/Ada or with a
 FlashInfer build that supports your GPU, you can drop the two workaround flags.
 
+## Inference tuning: MoE top-K (speed vs quality)
+
+The number of routed experts per token (`num_experts_per_tok`, native **8**) can be lowered at
+**inference time** (no re-quantization) to trade quality for less expert compute, via vLLM's
+`--hf-overrides '{"num_experts_per_tok": K}'`. Measured on this 4-bit checkpoint (greedy):
+
+| top-K | HumanEval | GSM8K | routed-expert FLOPs |
+|---|---|---|---|
+| **8** (native) | **91.5%** | **95.9%** | 100% |
+| **6** | 89.6% (−1.9) | 94.8% (−1.1) | ~75% |
+| **4** | 86.6% (−4.9) | 93.5% (−2.4) | ~50% |
+
+Degradation is **graceful** — even top-4 (half the routed-expert compute) stays coherent and
+usable. **top-6** is a sweet spot (~25% less expert compute for ≈1-2 pts). Coding is a bit more
+sensitive to fewer experts than math. (Default = 8; only lower it if you need the speed/energy.)
+
 ## Long context (YaRN)
 
 The base model is `rope_type: "default"` with `max_position_embeddings: 262144`. To go beyond,