| --- |
| base_model: Qwen/Qwen3.8-Flash-Next |
| base_model_relation: quantized |
| --- |
| |
| # Qwen3.8-Flash-Next-AWQ-W4A16 |
|
|
| Quantized version of [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next). |
|
|
| If you come here with older cards like A100, A6000 or 3090, consider using [our vLLM fork](https://github.com/wtdcode/vllm-backport) which has served billions of tokens for frontier models with older GPUs. |
|
|
| ## What is quantized |
|
|
| Weight-only INT4 (symmetric, group size 128) via AWQ, stored in the compressed-tensors `pack-quantized` format. |
|
|
| **Quantized (INT4 W4A16):** |
| - Routed MoE experts in all 48 decoder layers: `mlp.experts.{0..511}.{gate_proj, up_proj, down_proj}` (≈123B of the 180B parameters) |
|
|
| **Kept in BF16 (not quantized):** |
| - Token embedding (`embed_tokens`) and `lm_head` |
| - Gated DeltaNet linear attention (`linear_attn.*`) |
| - Qwen Sparse Attention (`self_attn.{q,k,v,o}_proj`) and its indexer (`self_attn.indexer.*`) |
| - Gated residual / hyper-connections (`*_hyper_connection.*`, `hyper_connection_mixer.*`) |
| - MoE router (`mlp.gate`), shared expert (`mlp.shared_expert.*`) and its gate (`shared_expert_gate`) |
| - Per-layer embedding (PLE) block including the 51B n-gram embedding table (`ple.*`) |
| - Vision encoder (`model.visual.*`) |
| - MTP layer (`mtp.*`, in `model_mtp.safetensors`) |
|
|
