Instructions to use JANGQ-AI/GLM-5.3-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use JANGQ-AI/GLM-5.3-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="JANGQ-AI/GLM-5.3-W4A16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("JANGQ-AI/GLM-5.3-W4A16")
model = AutoModelForCausalLM.from_pretrained("JANGQ-AI/GLM-5.3-W4A16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use JANGQ-AI/GLM-5.3-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "JANGQ-AI/GLM-5.3-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JANGQ-AI/GLM-5.3-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/JANGQ-AI/GLM-5.3-W4A16

SGLang

How to use JANGQ-AI/GLM-5.3-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "JANGQ-AI/GLM-5.3-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JANGQ-AI/GLM-5.3-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "JANGQ-AI/GLM-5.3-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JANGQ-AI/GLM-5.3-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use JANGQ-AI/GLM-5.3-W4A16 with Docker Model Runner:
```
docker model run hf.co/JANGQ-AI/GLM-5.3-W4A16
```

GLM-5.3-W4A16

GLM-5.3 — ~460 GB on disk (down from the ~1.5 TB BF16 source) — INT4 W4A16 quantization of the routed experts in compressed-tensors pack-quantized format. Loads and runs on Hopper (H100 / H200) via vLLM's Marlin kernels.

Source: zai-org/GLM-5.3 (GLM-MoE-DSA architecture, 78 layers, 256 routed experts top-8, 1 shared expert, MLA + DeepSeek Sparse Attention, MTP head, 1M context)
Quantization: INT4 symmetric, group-32, RTN, on the routed-expert weights only (incl. MTP) — packed with the compressed-tensors library's own pack_to_int32 for a bit-exact on-disk layout (weight_packed int32, weight_scale bf16, weight_shape int64)
Why W4A16 (not NVFP4 / MXFP4): NVFP4 and MXFP4 are Blackwell-native formats and run through slow dequant emulation on Hopper. INT4 W4A16 → Marlin is the mature, fast 4-bit path on H100 / H200. A16 = activations, attention math, and KV cache all stay 16-bit — only expert weight storage is 4-bit
Attention & router kept high-precision: MLA / DSA attention, dense MLP, the shared expert, router gates, norms, embeddings, and lm_head all stay BF16 — only the 256 routed experts per layer are INT4, and those are ~97% of the parameters
MTP included: the MTP layer (model.layers.78.*) experts are quantized like every other layer
Bundle size: ~460 GB
Runs on: H100 / H200 with vLLM tensor-parallel

What's in the bundle

Module	Source dtype	Bundle dtype
Routed experts (`experts.*.{gate,up,down}_proj`, incl. MTP)	BF16	INT4 sym g32 (`weight_packed` i32 / `weight_scale` bf16 / `weight_shape` i64)
Attention (MLA q/kv proj, o_proj, DSA indexer)	BF16	BF16
Dense MLP (first 3 layers) + shared expert	BF16	BF16
Router gates, `e_score_correction_bias`, RMSNorms	BF16 / F32	BF16 / F32
`embed_tokens`, `lm_head`, MTP glue (`eh_proj`, norms)	BF16	BF16

Dequant per group of 32: w = int4 * weight_scale (symmetric, scale = amax/7.5, range [-8, 7]).

Usage (vLLM)

vllm serve JANGQ-AI/GLM-5.3-W4A16 \
  --tensor-parallel-size 8 \
  --trust-remote-code

vLLM auto-selects the Marlin MoE kernel for compressed-tensors pack-quantized INT4 at group-32 on Hopper.

Method

Converted from the BF16 release with a streaming shard-by-shard quantizer (one tensor in memory at a time — no GPU, no large host RAM). Round-to-nearest INT4 is packed via compressed_tensors.pack_to_int32 so the layout is bit-exact with what vLLM expects; the format was ground-truthed against a known-working DeepSeek-family W4A16 model. This is a data-free RTN quant (no calibration). For maximum quality a calibrated GPTQ/AWQ pass on the routed experts can be run later on-device.

Quantized by JANGQ-AI.

Downloads last month: 304

Safetensors

Model size

753B params

Tensor type

BF16

I32

F32

Model tree for JANGQ-AI/GLM-5.3-W4A16

Base model

zai-org/GLM-5.3

Quantized

(29)

this model