Instructions to use incoai/GLM-5.3-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use incoai/GLM-5.3-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="incoai/GLM-5.3-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("incoai/GLM-5.3-NVFP4")
model = AutoModelForCausalLM.from_pretrained("incoai/GLM-5.3-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use incoai/GLM-5.3-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "incoai/GLM-5.3-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "incoai/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/incoai/GLM-5.3-NVFP4

SGLang

How to use incoai/GLM-5.3-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "incoai/GLM-5.3-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "incoai/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "incoai/GLM-5.3-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "incoai/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use incoai/GLM-5.3-NVFP4 with Docker Model Runner:
```
docker model run hf.co/incoai/GLM-5.3-NVFP4
```

GLM-5.3-NVFP4

This repository contains an NVFP4-quantized version of Z.ai's zai-org/GLM-5.3. The weights and activations of the routed-expert linear layers are in NVFP4, and the KV cache is in FP8. The checkpoint uses the NVIDIA Model Optimizer format (quant_method: modelopt) and is served directly by SGLang and vLLM on NVIDIA Blackwell GPUs (SM100+).

Base model: zai-org/GLM-5.3
Quantization: NVFP4 weights + static NVFP4 activations; FP8 KV cache
Size: 433 GiB
License: GLM-5.3 License

Quantization Method

This checkpoint was produced with our in-house post-training quantization toolchain and exported in the NVIDIA Model Optimizer format. Only the weights and activations of the linear operators within the routed MoE experts are quantized to NVFP4, with static per-tensor activation scales. The KV cache is quantized to FP8 with static unit scales (kv_cache_quant_algo: FP8).

Quick Start

The commands below enable speculative decoding with the DFlash 2 draft model (lossless, 7 draft tokens per verification step); remove the --speculative-* flags to serve without it.

Serve with SGLang (main):

python3 -m sglang.launch_server \
    --model-path incoai/GLM-5.3-NVFP4 \
    --tp 8 \
    --quantization modelopt_fp4 \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --chunked-prefill-size 8192 \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path incoai/GLM-5.3-DFlash2 \
    --speculative-draft-attention-backend trtllm_mha

Or with vLLM (v0.28.0 or later):

vllm serve incoai/GLM-5.3-NVFP4 \
    --tensor-parallel-size 8 \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --enable-auto-tool-choice \
    --speculative-config '{"method":"dflash","model":"incoai/GLM-5.3-DFlash2","num_speculative_tokens":7}'

Accuracy

We compare Z.ai's original FP8 release with this NVFP4 checkpoint. Higher is better.

Precision	GPQA Diamond	AIME 2025	MATH-500	HLE	AA-LCR
FP8	91.1	94.3	95.6	35.9	73.6
NVFP4	91.2	95.1	95.2	35.2	73.0

License

This model is a quantized version of GLM-5.3 and is distributed under Z.ai's GLM-5.3 License, which it inherits from the base model.

Downloads last month: 1,257

Safetensors

Model size

391B params

Tensor type

BF16

F32

Model tree for incoai/GLM-5.3-NVFP4

Base model

zai-org/GLM-5.3

Quantized

(24)

this model

Quantizations

2 models