File size: 4,465 Bytes
ec7df07 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | ---
license: other
base_model: zai-org/GLM-5.3-BF16
tags: [glm-5.3, quantization, int4, int8, mixed-precision, compressed-tensors, dgx-spark, quanttrio]
pipeline_tag: text-generation
---
# GLM-5.3-Int4-Int8Mix
The first Int4/Int8Mix quantization of big **GLM-5.3 (743B)** — produced with the
QuantTrio recipe, verified structurally and numerically, and served at TP4 on four
NVIDIA DGX Spark (GB10 / sm121).
- **Base:** `zai-org/GLM-5.3-BF16` (1507 GB, genuine BF16 — chosen over the fp8 repo so the router, DSA indexer, lm_head and layer 0 stay true BF16)
- **Size:** 377.4 GiB, 282 shards, `compressed-tensors` / `pack-quantized`
- **Per-rank at TP4:** 94.6 GiB
## Quantization recipe
Config groups transferred verbatim from `QuantTrio/GLM-5.2-Int4-Int8Mix` — GLM-5.2 and
GLM-5.3 are structurally identical (78 layers, hidden 6144, kv_lora_rank 512, 256 routed
experts, first_k_dense_replace 3). Data-free RTN: no calibration, static, symmetric,
weight-only, group size 128.
| group | targets | bits | strategy | modules |
|---|---|---|---|---|
| `w4a16_experts` | layers 3–77 `mlp.experts.N.{gate,up,down}_proj` | 4 | group/128 | 57,600 |
| `w8a16_linears` | layers 1–77 attention projs + dense & shared-expert MLP | 8 | group/128 | 616 |
| `w8a16_mtp_channel` | layer 78 (MTP) attention + MLP + experts | 8 | channel (gs −1) | 776 |
**Kept full precision:** all of layer 0, every `mlp.gate` (MoE router), the DSA
`indexer`/`indexers_proj`, MTP `eh_proj`/`enorm`/`hnorm`/`shared_head`, and `lm_head`.
Quantized by shard streaming (read one BF16 shard → quantize → pack → write one output
shard; ~10 GiB peak RAM, resumable), using compressed-tensors' own `calculate_qparams` /
`quantize` / `pack_to_int32`, with scales rounded to bf16 before quantizing so the
persisted scale is exactly the one quantization assumed. Runtime: ~30–57 min CPU-only on
one DGX Spark depending on source medium.
## Verification (fail-closed gates)
- **Structural:** 282/282 shards open clean; exact 1:1 key mapping (59,585 source tensors
→ 58,992 quantized modules + 593 BF16/FP32 keeps); census 57,600/616/776 matches the
pre-quant dry-run exactly; sacred modules plain BF16, zero packed leaks.
- **Numerical:** dequant vs BF16 on sampled tensors across all three groups —
W4-g128 nrmse ≈ 0.111 (exactly theoretical for data-free absmax RTN), W8-g128 ≈ 0.0072,
W8-channelwise MTP ≈ 0.009. Packing convention proven by discrimination
(wrong unpack scores cos ≈ −0.58).
- **Integrity:** full 282-entry sha256 manifest; distributed to 4 cluster nodes with
per-target verification, 282/282 OK each.
## Serving (TP4 on 4× DGX Spark)
vLLM fork stack with the sm12x sparse-MLA kernel overlays (`probe-modded` image lineage),
native multi-node (`--nnodes/--node-rank`, workers headless), local NVMe weights per node.
Sweep-optimized config (autoresearch over k, max-seqs, NCCL channels, Marlin atomic-add,
cudagraph capture sizes):
```
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"draft_tensor_parallel_size":1,"attention_backend":"FLASHMLA_SPARSE"}'
--tensor-parallel-size 4 --max-model-len 200000 --max-num-seqs 8 --max-num-batched-tokens 8192
--gpu-memory-utilization 0.91 --kv-cache-memory-bytes 10950000000 --kv-cache-dtype fp8_ds_mla
--distributed-executor-backend mp --compilation-config '{"cudagraph_mode":"FULL"}'
```
Plus: cache-flushers (unconditional drop_caches during boot — page cache eats
CUDA-visible memory 1:1 on GB10 unified memory), `vm.swappiness=10`, NCCL pinned to the
RoCE fabric with `NCCL_IB_GID_INDEX` omitted (reboot-volatile GID tables).
### Measured (e2e wall-clock, 256-token prose, temp 0)
| concurrency | tok/s |
|---|---|
| C1 | 27.2 |
| C4 | 59.1 |
| C8 | 86.9 |
TTFT ≈ 0.6 s at ~780-token prompt. KV pool 200,064 tokens. MTP mean acceptance length
2.3–3.1/3. Beats the prior GLM-5.2 QuantTrio TP4 reference at every concurrency level.
### Quality (10-sample battery, temp 0)
GSM8K 80% · HumanEval 100% · MBPP 100% · IFEval 90% · MMLU-STEM 90% · ARC-C 90% ·
HellaSwag 90% → **91.4% average** — no meaningful loss vs the BF16-served baseline class.
## Provenance
Quantized, verified, and benchmarked on a fleet of DGX Spark nodes over a RoCE fabric by
an agent-assisted pipeline. Companion repo with the full quantization script, verification
gates, launch recipe, autoresearch sweep and raw results: see the repo README / linked
GitHub repository.
|