| --- |
| license: other |
| base_model: zai-org/GLM-5.3-BF16 |
| tags: [glm-5.3, quantization, int4, int8, mixed-precision, compressed-tensors, dgx-spark, quanttrio] |
| pipeline_tag: text-generation |
| --- |
| |
| # GLM-5.3-Int4-Int8Mix |
|
|
| The first Int4/Int8Mix quantization of big **GLM-5.3 (743B)** — produced with the |
| QuantTrio recipe, verified structurally and numerically, and served at TP4 on four |
| NVIDIA DGX Spark (GB10 / sm121). |
|
|
| - **Base:** `zai-org/GLM-5.3-BF16` (1507 GB, genuine BF16 — chosen over the fp8 repo so the router, DSA indexer, lm_head and layer 0 stay true BF16) |
| - **Size:** 377.4 GiB, 282 shards, `compressed-tensors` / `pack-quantized` |
| - **Per-rank at TP4:** 94.6 GiB |
| |
| ## Quantization recipe |
| |
| Config groups transferred verbatim from `QuantTrio/GLM-5.2-Int4-Int8Mix` — GLM-5.2 and |
| GLM-5.3 are structurally identical (78 layers, hidden 6144, kv_lora_rank 512, 256 routed |
| experts, first_k_dense_replace 3). Data-free RTN: no calibration, static, symmetric, |
| weight-only, group size 128. |
|
|
| | group | targets | bits | strategy | modules | |
| |---|---|---|---|---| |
| | `w4a16_experts` | layers 3–77 `mlp.experts.N.{gate,up,down}_proj` | 4 | group/128 | 57,600 | |
| | `w8a16_linears` | layers 1–77 attention projs + dense & shared-expert MLP | 8 | group/128 | 616 | |
| | `w8a16_mtp_channel` | layer 78 (MTP) attention + MLP + experts | 8 | channel (gs −1) | 776 | |
|
|
| **Kept full precision:** all of layer 0, every `mlp.gate` (MoE router), the DSA |
| `indexer`/`indexers_proj`, MTP `eh_proj`/`enorm`/`hnorm`/`shared_head`, and `lm_head`. |
|
|
| Quantized by shard streaming (read one BF16 shard → quantize → pack → write one output |
| shard; ~10 GiB peak RAM, resumable), using compressed-tensors' own `calculate_qparams` / |
| `quantize` / `pack_to_int32`, with scales rounded to bf16 before quantizing so the |
| persisted scale is exactly the one quantization assumed. Runtime: ~30–57 min CPU-only on |
| one DGX Spark depending on source medium. |
|
|
| ## Verification (fail-closed gates) |
|
|
| - **Structural:** 282/282 shards open clean; exact 1:1 key mapping (59,585 source tensors |
| → 58,992 quantized modules + 593 BF16/FP32 keeps); census 57,600/616/776 matches the |
| pre-quant dry-run exactly; sacred modules plain BF16, zero packed leaks. |
| - **Numerical:** dequant vs BF16 on sampled tensors across all three groups — |
| W4-g128 nrmse ≈ 0.111 (exactly theoretical for data-free absmax RTN), W8-g128 ≈ 0.0072, |
| W8-channelwise MTP ≈ 0.009. Packing convention proven by discrimination |
| (wrong unpack scores cos ≈ −0.58). |
| - **Integrity:** full 282-entry sha256 manifest; distributed to 4 cluster nodes with |
| per-target verification, 282/282 OK each. |
|
|
| ## Serving (TP4 on 4× DGX Spark) |
|
|
| vLLM fork stack with the sm12x sparse-MLA kernel overlays (`probe-modded` image lineage), |
| native multi-node (`--nnodes/--node-rank`, workers headless), local NVMe weights per node. |
|
|
| Sweep-optimized config (autoresearch over k, max-seqs, NCCL channels, Marlin atomic-add, |
| cudagraph capture sizes): |
|
|
| ``` |
| --speculative-config '{"method":"mtp","num_speculative_tokens":3,"draft_tensor_parallel_size":1,"attention_backend":"FLASHMLA_SPARSE"}' |
| --tensor-parallel-size 4 --max-model-len 200000 --max-num-seqs 8 --max-num-batched-tokens 8192 |
| --gpu-memory-utilization 0.91 --kv-cache-memory-bytes 10950000000 --kv-cache-dtype fp8_ds_mla |
| --distributed-executor-backend mp --compilation-config '{"cudagraph_mode":"FULL"}' |
| ``` |
|
|
| Plus: cache-flushers (unconditional drop_caches during boot — page cache eats |
| CUDA-visible memory 1:1 on GB10 unified memory), `vm.swappiness=10`, NCCL pinned to the |
| RoCE fabric with `NCCL_IB_GID_INDEX` omitted (reboot-volatile GID tables). |
|
|
| ### Measured (e2e wall-clock, 256-token prose, temp 0) |
|
|
| | concurrency | tok/s | |
| |---|---| |
| | C1 | 27.2 | |
| | C4 | 59.1 | |
| | C8 | 86.9 | |
|
|
| TTFT ≈ 0.6 s at ~780-token prompt. KV pool 200,064 tokens. MTP mean acceptance length |
| 2.3–3.1/3. Beats the prior GLM-5.2 QuantTrio TP4 reference at every concurrency level. |
|
|
| ### Quality (10-sample battery, temp 0) |
|
|
| GSM8K 80% · HumanEval 100% · MBPP 100% · IFEval 90% · MMLU-STEM 90% · ARC-C 90% · |
| HellaSwag 90% → **91.4% average** — no meaningful loss vs the BF16-served baseline class. |
|
|
| ## Provenance |
|
|
| Quantized, verified, and benchmarked on a fleet of DGX Spark nodes over a RoCE fabric by |
| an agent-assisted pipeline. Companion repo with the full quantization script, verification |
| gates, launch recipe, autoresearch sweep and raw results: see the repo README / linked |
| GitHub repository. |
|
|
