---
license: other
library_name: transformers
tags:
  - step3p5
  - moe
  - nvfp4
  - fp4
  - modelopt
  - quantized
base_model: stepfun-ai/Step3p5
quantized_by: modelopt
pipeline_tag: text-generation
model_type: step3p5
---

# Step3p5 NVFP4

NVIDIA FP4 (NVFP4) quantized version of the Step3p5 Mixture-of-Experts model, with MoE router/gate weights dequantized to bfloat16 for vLLM compatibility.

## Quantization Details

- **Quantization method**: NVIDIA ModelOpt 0.41.0, NVFP4 (W4A4)
- **Weight format**: FP4 E2M1, packed 2 values per uint8 byte
- **Group size**: 16
- **Excluded from quantization**: `lm_head`, `*.moe.gate*` (router/gate)

The MoE router/gate weights are stored in bfloat16 (not quantized) following NVIDIA ModelOpt best practices — quantizing the router degrades routing quality with negligible memory savings.

## Serving with vLLM

```bash
VLLM_USE_FLASHINFER_MOE_FP4=0 vllm serve apandacoding/step3p5-nvfp4 \
  --quantization modelopt_fp4 \
  --trust-remote-code \
  --host 0.0.0.0 --port 8000
```

> **Note**: `VLLM_USE_FLASHINFER_MOE_FP4=0` is required to use the VLLM_CUTLASS MoE backend. The FlashInfer TRTLLM monolithic MoE kernel has a known issue with 288-expert models.

## Model Architecture

- **Type**: Mixture of Experts (MoE) with shared experts
- **Experts**: 288 routed + shared expert per layer
- **Top-K**: 8 experts per token
- **Hidden size**: 4096
- **MoE intermediate size**: 1280
- **MoE layers**: 42 (layers 3–44)
- **Attention**: GQA with 96 heads, 8 KV heads
- **Context length**: 262,144 tokens