Step-5-Preview-NVFP4-Local

A 4-bit build of StepFun's Step 5 Preview sized for four 96 GB GPUs: 347.8 GB, against 394.2 GB for stepfun-ai/Step-5-Preview-NVFP4 (revision bc246cc2848e17b37f377eda8fd825c5b3018d1f), from which it is made.

• Routed experts are NVFP4 in all 88 MoE layers: layers 3 to 87 as StepFun quantized them, and layers 88 to 90, which StepFun shipped in BF16, quantized here the same way (NVFP4 weights, BF16 activations).
• The E4M3 block scales of every routed expert are stored in b12x's compressed scale format (CSF), the format introduced with nvidia/Qwen3.8-Flash-Next-FP4-Local, about half the size of the raw scales.
• Everything else (attention, shared experts, routers, dense layers, MTP layers, embeddings, lm_head, vision tower, tokenizer, chat template) is StepFun's NVFP4 release unchanged, in BF16 as StepFun shipped it.

Step 5 Preview is StepFun AI's model: a sparse mixture of experts with 600B total and 27B active parameters, text and image input, up to 1M tokens of context. Model card: stepfun-ai/Step-5-Preview. This repository is not affiliated with or endorsed by StepFun AI. Use is subject to the StepFun Model License 1.0 (LICENSE.txt, with StepFun's Chinese text in LICENSE-CN.txt).

Status: weights complete. KL divergence against the BF16 model has not been measured yet. Serving needs a vLLM build with Step-5 support and b12x's NVFP4-CSF loader; that image is being prepared and has not been run yet (see Serving). No accuracy or serving claim is made here.

Modifications

Notice of changes under section 2(a) of the StepFun Model License. Made by Brandon M. Music on 2026-10-16 to stepfun-ai/Step-5-Preview-NVFP4 at revision bc246cc2848e17b37f377eda8fd825c5b3018d1f:

1. Layers 3 to 87: each model.layers.N.moe.{gate,up,down}_proj.weight_scale (F8_E4M3 [352, R, C]) is stored as three CSF tensors that encode the same scale bytes (see Format).
2. Layers 88 to 90: each BF16 model.layers.N.moe.{gate,up,down}_proj.weight ([352, R, K]) is replaced by NVFP4: weight (U8 [352, R, K/2], E2M1 pairs), weight_scale_2 (F32 [352]) and CSF tensors holding the E4M3 block scales. These layers have no input_scale; their activations stay BF16.
3. config.json and hf_quant_config.json: quantization config MIXED_PRECISION with one quantized_layers entry per MoE layer, model.layers.N.moe.experts: {"quant_algo": "NVFP4", "group_size": 16, "weight_scale_encoding": "csf"} for N = 3 to 87 and {"quant_algo": "W4A16_NVFP4", ...} for N = 88 to 90. The three *layers.N.moe.experts* entries for 88 to 90 are removed from StepFun's ignore / exclude_modules lists; the rest of those lists is unchanged.
4. model.safetensors.index.json and SHA256SUMS are regenerated; csf-report.json, bf16-experts-nvfp4-report.json, bf16-layer-sensitivity.json and the scripts in tools/ are added; this README replaces StepFun's card.

LICENSE.txt and LICENSE-CN.txt are StepFun's, unchanged.

Size

	stepfun-ai/Step-5-Preview-NVFP4	This repository
Tensor bytes	394,164,831,232 (394.2 GB)	347,771,251,120 (347.8 GB)
Layers 3-87 routed-expert block scales	35,295,068,160	18,154,837,272 (CSF, 51.44%)
Layers 88-90 routed experts	39,862,665,216 (BF16)	10,609,303,320 (NVFP4 9,965,666,304 + CSF scales 643,637,016)
Saved		46,393,580,112 (46.4 GB, 11.8%)

At tensor parallel 4 the weights come to about 86.9 GB (81.0 GiB) per GPU. The b12x loader keeps the compressed scale planes resident and expands one MoE layer's scales into shared scratch when that layer runs, so the CSF saving applies to GPU memory as well as disk.

Layers 88-90

StepFun kept exactly these three layers' routed experts out of quantization in both of its quantized releases: the FP8 checkpoint (stepfun-ai/Step-5-Preview, whose modules_to_not_convert starts with them) and the NVFP4 one. In the architecture they are ordinary routed-expert layers (same shapes, router, sliding-window attention and SwiGLU limit as their neighbours); they are the last three MoE layers before the final dense layer and the output head. StepFun does not say why. Here they are NVFP4 to fit four 96 GB GPUs.

Their weights are not harder to quantize than the layers StepFun did quantize: the same NVFP4 rounding gives a weight NMSE of 0.903% on layers 88-90 (all 352 experts, against BF16) and 0.900% on layers 85-87 (32 experts per layer, against StepFun's FP8 copy), with the same kurtosis and tails (bf16-layer-sensitivity.json, bf16-experts-nvfp4-report.json). That is a weight-level comparison only; their effect on the output is part of the KL-divergence measurement, which has not been run.

Format

For each routed-expert scale tensor <p>.weight_scale with shape [E, R, C] (E = 352 experts; gate and up R = 1536, C = 256; down R = 4096, C = 96; one E4M3 byte per 16 weights):

Tensor	Dtype, shape	Contents
<p>.weight_scale.nvfp4_csf_fixed	U8 [E, R/16, 16 * (1 + C/2)]	per 16-row slab: 16 row bases, then each row's 4-bit codes, two per byte, low nibble first
<p>.weight_scale.nvfp4_csf_exceptions	U32 [n]	(row * C + col) | byte << 24, grouped by expert, row-major within an expert
<p>.weight_scale.nvfp4_csf_starts	I64 [E + 1]	expert e owns exceptions[starts[e]:starts[e + 1]]

A scale byte is its row's base plus its 4-bit code, unless an exception at that position replaces it with the stored byte. Each row's base is the lowest value whose 16-value window covers the most of the row's bytes (bases are 0 to 240). The three tensors are exactly the components of b12x's CsfSource((fixed, exceptions), starts, R, C, "nvfp4", E, "csf"); a single expert is fixed[e] with exceptions[starts[e]:starts[e + 1]], the per-matrix form that nvidia/Qwen3.8-Flash-Next-FP4-Local stores. Step-5's fused [352, ...] expert layout is kept for every tensor. NVFP4 weights dequantize as E2M1 value x E4M3 block scale x weight_scale_2.

Fidelity

The fidelity measure for this checkpoint is KL divergence against the BF16 model (stepfun-ai/Step-5-Preview-BF16, revision 7734b7ffac118d70d8398bfa3eb0fab007eb45b1). It has not been measured yet. This section will carry the per-window table, the method and the teacher-logit dataset once it has been.

Conversion integrity checks (these confirm the files were written as intended; they are not a fidelity measurement): the source files match StepFun's SHA256SUMS (102 of 102); every CSF tensor passes b12x CsfSource.validate() and decodes, with an independent decoder, to the scale bytes it encodes, checked before and after writing; every packed NVFP4 matrix of layers 88-90 decodes exactly to the quantizer's output (3,168 of 3,168); all other tensors are byte-identical to StepFun's.

How it was made

• CSF encoder: csf_encode.py from Local Inference Lab's flashinfer (local-inference-lab/flashinfer main @ baa81b6d53, flashinfer/experimental/b12x/_lib/quant/csf_encode.py, SHA-256 fa97f6e13102b608ce6e41e8a172e403d87057d903df33d966bff7fa3d7abdc8), unchanged, run on NVIDIA B300 GPUs. Validator: b12x loader/_csf_source.py CsfSource.validate() (SHA-256 9178504590596cad56441c12651a80961e8a45423a5795df0abb600b41e8ca68), unchanged.
• Layers 88-90: FP32 scale per expert matrix amax / (6 x 448), E4M3 scale per 16 inputs block amax / 6 / weight_scale_2, E2M1 by round-to-nearest; the same rounding reproduces StepFun's layer 85-87 export.
• Scripts: tools/convert_step5_csf.py (CSF), tools/quantize_bf16_experts.py (layers 88-90), tools/bf16_layer_sensitivity.py (the weight-level comparison). Results: csf-report.json, bf16-experts-nvfp4-report.json, bf16-layer-sensitivity.json.

Serving

This checkpoint needs a vLLM build with both:

• Step-5 support: upstream pull request vllm-project/vllm#61727 (open as of 2026-10-16; StepFun's vllm/vllm-openai:step-5-preview image is built from it), and
• b12x's NVFP4-CSF expert loader (weight_scale_encoding: "csf", quantization modelopt_mixed, MoE backend b12x), as in Local Inference Lab's vLLM images such as ghcr.io/local-inference-lab/vllm:release-10152026, which do not include Step-5 yet.

Upstream vLLM and the step-5-preview image cannot read the CSF scale tensors; use stepfun-ai/Step-5-Preview-NVFP4 with those. An image combining both is being built on Local Inference Lab's vLLM with b12x kernels. This section will give its tag and launch command once it has been run.

Credits

• Step 5 Preview and its NVFP4 checkpoint: StepFun AI. The NVFP4 export was produced with NVIDIA Model Optimizer 0.47.0, per the checkpoint's hf_quant_config.json.
• CSF scale format, encoder, validator and loader: b12x, in Local Inference Lab's flashinfer (flashinfer/experimental/b12x), the format first shipped with nvidia/Qwen3.8-Flash-Next-FP4-Local.