Instructions to use zai-org/GLM-5.3-Flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zai-org/GLM-5.3-Flash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="zai-org/GLM-5.3-Flash")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("zai-org/GLM-5.3-Flash")
model = AutoModelForMultimodalLM.from_pretrained("zai-org/GLM-5.3-Flash", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zai-org/GLM-5.3-Flash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zai-org/GLM-5.3-Flash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.3-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/zai-org/GLM-5.3-Flash

SGLang

How to use zai-org/GLM-5.3-Flash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zai-org/GLM-5.3-Flash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.3-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zai-org/GLM-5.3-Flash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/GLM-5.3-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use zai-org/GLM-5.3-Flash with Docker Model Runner:
```
docker model run hf.co/zai-org/GLM-5.3-Flash
```

GLM-5.3-Flash FP8 running on 2×L40S (sm_89) + EPYC 9845 — setup notes, patches and benchmarks

#38

by goldvet - opened Sep 2

Discussion

goldvet

Sep 2

GLM-5.3-Flash FP8 on 2×NVIDIA L40S (Ada, sm_89) + AMD EPYC 9845 — it works, numbers inside

Since day-0 support targets Hopper/Blackwell, here are working notes for running the
native FP8 checkpoint (no quantization) on Ada GPUs with CPU expert offload,
fully offline. As far as I know these are the first published numbers for this combo.

Hardware

2× NVIDIA L40S (48 GB, sm_89, PCIe Gen4, no NVLink), TP=2
AMD EPYC 9845 (160C Zen 5c, SMT off, AVX-512, no AMX), NPS1
768 GB DDR5-6400 RDIMM running at 5200 MT/s (12×64 GB, 1DPC — platform max on
Gigabyte R263-ZG0-AAL2 / MZG3-GU0)
Debian 13 (glibc 2.41), driver-only CUDA + side-by-side toolkit 12.8

Stack

pip install "ktransformers[sglang]" in Python 3.11 → kt-kernel 0.7.0.post1,
sglang-kt 0.7.0.post1, transformers-kt 5.6.0.post4
--kt-method FP8 (CPU experts on AVX512-BF16 backend), PLE/N-gram table in host RAM
Launch: TP2, --kt-cpuinfer 152, --kt-max-deferred-experts-per-token 4,
--kt-gpu-prefill-token-threshold 2048 (layerwise GPU prefill),
--chunked-prefill-size 16384, --max-mamba-cache-size 16,
--mem-fraction-static 0.55, context 524k, KV fp8_e4m3

Patches needed on sm_89 (all one-liners, .bak kept)

config.json: index_share_for_mtp_iteration: true → false (KPool validator
rejects IndexShare-for-MTP; harmless without speculative decoding)
nsa_indexer_kpool.py ~L1109: disable the short-sequence skip path — both
fast_topk_transform_fused and fast_topk use Hopper features
(cluster launch) and crash with set_up_kernel_once failed: invalid argument
CUDA 12.8 math_functions.h: add noexcept (true) to sinpi/cospi/sinpif/cospif
(glibc 2.41 conflict, breaks all JIT compiles otherwise)
nvidia-cudnn-cu12==9.16.0.29 (torch 2.9.1 + cuDNN 9.10 Conv3d guard)
Optional: server_args.py cuda_graph_bs [1,2,4] → [1,2,4,8,16,24]
(decode without graphs collapses: 3.8 tok/s/user at n=5)

Also tuned Triton configs for L40S (6 block-FP8 GEMM shapes + fused MoE
E=288,N=1024) — 12h of tuning for +6.6% hot prefill; marginal, skippable.

Measured performance (single-run, `bench` via OpenAI API)

Metric	Value
Decode, 1 stream	18.3 tok/s (15.6 without deferred experts ×4; 9.7 initial config)
Decode, 8 streams	~47 tok/s aggregate (5.5/user)
Decode, 16–24 streams	saturates ~64 tok/s aggregate
Decode at 200k ctx	no degradation (linear attention + sparse holds)
Prefill 68k, warm	936 tok/s steady chunks, 2,226 tok/s peak, 85 s total
Prefill global budget	~900 tok/s regardless of concurrency (n=1→24: 857→926)
KV pool	2.11M tokens (fp8_e4m3), 524k per request

The prefill ceiling is structural: the generic FP8 layerwise path uses a single
GPU full-layer slot (gpu_full_layer_slots=1), so concurrent prefills serialize
FIFO. The MXFP4 path (DeepSeek V4 Flash) already has dual slots — generalizing
that to the FP8/GLM path could ~2× long-context prefill. Filed as a feature
request on the KTransformers repo.

Known limits on this fork/GPU: no MTP (KPool sm89 = non-speculative only),
radix/prefix cache force-disabled (KPool live-tail restore not implemented yet),
so multi-turn agents re-prefill history every turn (~70 s per 100k at 900 tok/s).

Quality note: everything is native FP8 weights; the only approximation is
--kt-max-deferred-experts-per-token 4 (+15% decode). Drop it if you want
bit-exact expert application.

Happy to share the full launch script / patch list.

goldvet

Sep 4

•

edited Sep 4

Update (Sept 4): prefill tuning results, two failure modes documented, and ecosystem notes

Follow-up after another optimization session on the same 2×L40S + EPYC 9845 setup.

Chunked prefill: 32k is the measured optimum on sm_89

Swept --chunked-prefill-size on the FP8 layerwise route (68k-token prompt, best of repeated runs):

Chunk	68k total	Status
16,384	85 s	baseline
24,576	69.8 s	ok (3 expert passes, same as 32k)
32,768	69.7 s	new default (−17%)
40,960	crash	OOM in the mHC fp32 fallback (~2.5 GiB transient)
40,960 + `max_split_size_mb:512`	120 s	stable but allocator thrashing — worse than 16k

Total time is quantized by ⌈prompt/chunk⌉ expert passes over PCIe, so 24k and 32k tie at 3 passes for 68k. The 40k wall is the Manifold-Constrained Hyper-Connections pre-mix: no fused MHC kernel appears to be available on sm_89, so each chunk pays a float32 (pre * residual) transient that scales with chunk size. Mitigating the OOM with max_split_size_mb trades the crash for cudaFree/retry cycles. A fused MHC kernel for Ada would remove both.

Transport comparison: native vs legacy (both at chunk=16,384)

Measured before switching the chunk default, so both transports ran the
then-default 16,384 chunks — apples to apples:

Transport	Steady chunks	68k total
native	936 tok/s	85 s
legacy (`SGLANG_KT_GLM5_NEXT_FP8_TRANSPORT=legacy`)	716 tok/s	105 s (−23%)

(The 32k-chunk numbers above are a separate, later improvement on the native path; per-chunk rates aren't comparable across chunk sizes, totals are.)
The native transport's overlap (H2D of layer N+1 over attention of layer N, per the docstring in kt_ep_wrapper.py) is real and worth keeping. Legacy also OOM'd under concurrent load where native holds.

Two operational gotchas worth knowing

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True breaks CUDA graph capture in TP=2 (invalid argument in custom_all_reduce.cuh — VMM memory can't be IPC-exported for the custom all-reduce buffers). Don't enable it on multi-GPU KT setups.
A desktop session (GNOME/Xwayland) was silently holding 0.5–0.7 GiB on GPU 0 — on a 2-GPU TP setup the most loaded card sets the limit, so headless (systemctl set-default multi-user.target) is worth the check.

Current state of the box

18.3 tok/s single-stream decode, ~47 tok/s aggregate at 8 streams, 68k prefill in 69.7 s solo / ~80 s under 8 concurrent decodes, ~1.9M-token KV pool, validated OOM-free under combined prefill+decode load.

Ecosystem notes for other sm_89 users

The community EXL3-K2 pack (2-bit routed experts, 91 GiB) runs through vLLM with native MTP and was smoke-tested on 4×L40 — proof that GLM-5.3-Flash + MTP + Ada work together in vLLM. The KT-side MTP lock is fork-specific, not a model/hardware limitation. (2-bit experts are a real quality trade-off for long-horizon agentic work, though.)
Encouraging precedent: KTransformers v0.6.4 shipped Ampere support with BF16 fallbacks, FP8 MoE Marlin for GPU experts/prefill, and RAWINT4 prefill acceleration for DeepSeek V4 Flash ~1 month after its day-0. I hope we can get the same treatment for GLM-5.3-Flash on sm_89 (MTP, radix cache tail-restore, dual GPU full-layer slots)

goldvet

1 day ago

Update: same 2×L40S + EPYC 9845 box, 2.4× faster decode

Back on the setup from the first post. Over the last few days I profiled the whole pipeline, GPU side and CPU side,
and reworked the parts that were holding it back. Same FP8 checkpoint, same deferred-experts setting, and the experts
stay at their original FP8 precision.

"Before" is my production config from this thread, re-measured with the same benchmark.

	Before	Now
Decode, 1 stream	18.1 tok/s	44.2 tok/s
Decode, 4 / 8 streams (aggregate)	34.7 / 60.3 tok/s	55.5 / 94–101 tok/s
Prefill, 2K-token prompt	~325 tok/s	~950 tok/s
Prefill, 16K-token prompt	~955 tok/s	~1,640 tok/s
Prefix cache	disabled	enabled
Agent turn adding ~600 tokens to a ~6–7K conversation	9–10 s	3.6 s
KV pool	1.68M tokens	2.40M tokens
Single-GPU mode (second GPU left free)	—	35.8 tok/s

What changed

GPU side. Several slow paths specific to this architecture now run as fused GPU kernels, and per-layer decode
overhead is lower.
CPU side. The CPU expert kernel was rewritten. Each token moves less data through RAM, with no loss of
information, and the GPU and CPU spend less time synchronizing.
Scheduling. I removed stalls that left the GPU idle between steps and rebalanced prefill between CPU and GPU.
Prefix cache. It was force-disabled for this model and now works, which is what speeds up multi-turn agent
sessions.
Memory. Freed GPU memory lets the KV pool grow to 2.4M tokens.
Single-GPU mode. It keeps the same KV pool and can switch with the 2-GPU mode without losing the cache.

Quality

Most changes give bit-identical outputs. The rest, including a small set of non-expert weights stored in an 8-bit
format, stay within the model's natural run-to-run noise, also on a 64K-token coding test with no drift over the
context. I chose not to use lower-bit experts, even though they would have been faster.

The main remaining bottleneck is the DDR5 bandwidth used by the CPU experts. MTP is still unavailable on this stack.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

GLM-5.3-Flash FP8 running on 2×L40S (sm_89) + EPYC 9845 — setup notes, patches and benchmarks

GLM-5.3-Flash FP8 on 2×NVIDIA L40S (Ada, sm_89) + AMD EPYC 9845 — it works, numbers inside

Hardware

Stack

Patches needed on sm_89 (all one-liners, .bak kept)

Measured performance (single-run, bench via OpenAI API)

Update (Sept 4): prefill tuning results, two failure modes documented, and ecosystem notes

Chunked prefill: 32k is the measured optimum on sm_89

Transport comparison: native vs legacy (both at chunk=16,384)

Two operational gotchas worth knowing

Current state of the box

Ecosystem notes for other sm_89 users

Update: same 2×L40S + EPYC 9845 box, 2.4× faster decode

What changed

Quality

Measured performance (single-run, `bench` via OpenAI API)