PhalaCloud

/

GLM-5.2-W4AFP8

like 33

Follow

Phala 13

Hi, I am trying to serve PhalaCloud/GLM-5.2-W4AFP8 with SGLang on a single-node 8×H100 setup. The model loads successfully and BF16 KV cache works correctly, but when I explicitly enable --kv-cache-dtype fp8_e4m3 as suggested in the model card, the model output becomes corrupted / nonsensical and often stops after only a few tokens.

Environment

Hardware:

8× NVIDIA H100 80GB
Single node
TP=8
PCIe-only GPU topology

Software:

torch 2.11.0+cu130
CUDA runtime 13.0
sglang 0.5.13.post1
sgl-kernel 0.4.3
flashinfer-python 0.6.12
flashinfer-cubin 0.6.12
triton 3.6.0
NCCL 2.28.9

Relevant pip freeze output:

sglang==0.5.13.post1
sglang-kernel==0.4.3
flashinfer-python==0.6.12
flashinfer-cubin==0.6.12
torch==2.11.0
triton==3.6.0
cuda-toolkit==13.0.2
cuda-python==13.3.1

Model:

PhalaCloud/GLM-5.2-W4AFP8

Working configuration: BF16 KV cache

If I do not pass --kv-cache-dtype fp8_e4m3, SGLang automatically selects BF16 KV cache on H100/SM90:

Setting KV cache dtype to bfloat16 for DeepSeek DSA on SM9 device.
Set DSA backends for bfloat16 KV Cache: prefill=flashmla_sparse, decode=fa3.

The model output is correct in this mode. With EAGLE enabled, I see reasonable acceptance rates around 0.6–0.8 and single-stream generation throughput around 50–56 tok/s.

Launch command:

python -m sglang.launch_server
--model-path /data3/models/PhalaCloud/GLM-5.2-W4AFP8
--quantization w4afp8
--disable-shared-experts-fusion
--tp 8
--reasoning-parser glm45
--tool-call-parser glm47
--context-length 204800
--max-total-tokens 204800
--mem-fraction-static 0.95
--trust-remote-code
--disable-piecewise-cuda-graph
--speculative-algorithm EAGLE
--speculative-num-steps 1
--speculative-eagle-topk 1
--speculative-num-draft-tokens 2

Example logs:

Setting KV cache dtype to bfloat16 for DeepSeek DSA on SM9 device.
Set DSA backends for bfloat16 KV Cache: prefill=flashmla_sparse, decode=fa3.
KV Cache is allocated. dtype: torch.bfloat16, #tokens: 204800, KV size: 19.11 GB
Decode batch ... accept len: 1.70, accept rate: 0.70, cuda graph: True, gen throughput: ~50-56 tok/s
Problematic configuration: FP8 KV cache

If I explicitly pass:

--kv-cache-dtype fp8_e4m3

SGLang switches to the FP8 KV path:

Set DSA backends for fp8_e4m3 KV Cache: prefill=flashmla_kv, decode=flashmla_kv.

However, the output becomes corrupted. For example, a simple chat request returns something like:

{
"message": {
"role": "assistant",
"content": "你好Comments",
"reasoning_content": null
},
"finish_reason": "stop"
}

With thinking enabled, I also saw corrupted reasoning_content, for example:

1 -1:1 –1.78:47345d1>0+##

The model often stops after only a few tokens.

The FP8 KV run also prints:

Using FP8 KV cache but no scaling factors provided.
Defaulting to scaling factors of 1.0.
This may lead to less accurate results!

I searched the local model directory and did not find any obvious kv_scales.json, quantization_param.json, k_scale, v_scale, or similar KV-cache scale file. I also did not find obvious k_scale / v_scale tensors inside the safetensors shards.

Reproduction request

A minimal request that shows the issue:

curl http://localhost:8000/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "GLM-5.2-W4AFP8",
"messages": [
{
"role": "user",
"content": "你好，请用一句话介绍你自己。树上10只鸟，开枪打死1只，还剩几只？"
}
],
"max_tokens": 128,
"temperature": 0,
"top_p": 1,
"stream": false,
"chat_template_kwargs": {"enable_thinking": false}
}'

Expected output:

A normal Chinese response explaining the model identity and answering the bird question.

Actual output with FP8 KV:

Corrupted / nonsensical output, often only a few tokens, e.g. "你好Comments", or corrupted reasoning_content.

Actual output with BF16 KV:

Correct natural-language output.
Questions
Is --kv-cache-dtype fp8_e4m3 for PhalaCloud/GLM-5.2-W4AFP8 officially supported on H100 / SM90, or was the FP8 KV path mainly tested on H200 or another environment?
Does this model require a --quantization-param-path JSON file for FP8 KV scaling factors? If yes, could you provide the expected file or format for this checkpoint?
The current public model card recommends --kv-cache-dtype fp8_e4m3, but SGLang’s GLM-5.2 DSA path automatically falls back to BF16 KV on SM90 when this flag is omitted. Should H100 users avoid forcing FP8 KV for now?
Could this be a bug in the flashmla_kv FP8 KV path for GLM-5.2 / DeepSeek DSA on H100?
Could you share the exact environment used for the reported numbers, including:
torch version
CUDA version
sglang version
sgl-kernel version
flashinfer version
driver version
H100 vs H200
whether --quantization-param-path or any private patch was used
Summary

BF16 KV cache works correctly:

H100/SM90 -> bfloat16 KV
prefill=flashmla_sparse
decode=fa3
output correct
EAGLE works with accept rate ~0.6–0.8

FP8 KV cache appears broken in my environment:

--kv-cache-dtype fp8_e4m3
prefill=flashmla_kv
decode=flashmla_kv
no scaling factors loaded, default scale=1.0
output corrupted / early stop

Any guidance on the correct FP8 KV setup for H100 would be appreciated.