---
license: other
base_model: zai-org/GLM-5.3-BF16
tags: [glm-5.3, quantization, int4, int8, mixed-precision, compressed-tensors, dgx-spark, quanttrio]
pipeline_tag: text-generation
---

# GLM-5.3-Int4-Int8Mix

The first Int4/Int8Mix quantization of big **GLM-5.3 (743B)** — produced with the
QuantTrio recipe, verified structurally and numerically, and served at TP4 on four
NVIDIA DGX Spark (GB10 / sm121).

- **Base:** `zai-org/GLM-5.3-BF16` (1507 GB, genuine BF16 — chosen over the fp8 repo so the router, DSA indexer, lm_head and layer 0 stay true BF16)
- **Size:** 377.4 GiB, 282 shards, `compressed-tensors` / `pack-quantized`
- **Per-rank at TP4:** 94.6 GiB

## Quantization recipe

Config groups transferred verbatim from `QuantTrio/GLM-5.2-Int4-Int8Mix` — GLM-5.2 and
GLM-5.3 are structurally identical (78 layers, hidden 6144, kv_lora_rank 512, 256 routed
experts, first_k_dense_replace 3). Data-free RTN: no calibration, static, symmetric,
weight-only, group size 128.

| group | targets | bits | strategy | modules |
|---|---|---|---|---|
| `w4a16_experts` | layers 3–77 `mlp.experts.N.{gate,up,down}_proj` | 4 | group/128 | 57,600 |
| `w8a16_linears` | layers 1–77 attention projs + dense & shared-expert MLP | 8 | group/128 | 616 |
| `w8a16_mtp_channel` | layer 78 (MTP) attention + MLP + experts | 8 | channel (gs −1) | 776 |

**Kept full precision:** all of layer 0, every `mlp.gate` (MoE router), the DSA
`indexer`/`indexers_proj`, MTP `eh_proj`/`enorm`/`hnorm`/`shared_head`, and `lm_head`.

Quantized by shard streaming (read one BF16 shard → quantize → pack → write one output
shard; ~10 GiB peak RAM, resumable), using compressed-tensors' own `calculate_qparams` /
`quantize` / `pack_to_int32`, with scales rounded to bf16 before quantizing so the
persisted scale is exactly the one quantization assumed. Runtime: ~30–57 min CPU-only on
one DGX Spark depending on source medium.

## Verification (fail-closed gates)

- **Structural:** 282/282 shards open clean; exact 1:1 key mapping (59,585 source tensors
  → 58,992 quantized modules + 593 BF16/FP32 keeps); census 57,600/616/776 matches the
  pre-quant dry-run exactly; sacred modules plain BF16, zero packed leaks.
- **Numerical:** dequant vs BF16 on sampled tensors across all three groups —
  W4-g128 nrmse ≈ 0.111 (exactly theoretical for data-free absmax RTN), W8-g128 ≈ 0.0072,
  W8-channelwise MTP ≈ 0.009. Packing convention proven by discrimination
  (wrong unpack scores cos ≈ −0.58).
- **Integrity:** full 282-entry sha256 manifest; distributed to 4 cluster nodes with
  per-target verification, 282/282 OK each.

## Serving (TP4 on 4× DGX Spark)

vLLM fork stack with the sm12x sparse-MLA kernel overlays (`probe-modded` image lineage),
native multi-node (`--nnodes/--node-rank`, workers headless), local NVMe weights per node.

Sweep-optimized config (autoresearch over k, max-seqs, NCCL channels, Marlin atomic-add,
cudagraph capture sizes):

```
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"draft_tensor_parallel_size":1,"attention_backend":"FLASHMLA_SPARSE"}'
--tensor-parallel-size 4 --max-model-len 200000 --max-num-seqs 8 --max-num-batched-tokens 8192
--gpu-memory-utilization 0.91 --kv-cache-memory-bytes 10950000000 --kv-cache-dtype fp8_ds_mla
--distributed-executor-backend mp --compilation-config '{"cudagraph_mode":"FULL"}'
```

Plus: cache-flushers (unconditional drop_caches during boot — page cache eats
CUDA-visible memory 1:1 on GB10 unified memory), `vm.swappiness=10`, NCCL pinned to the
RoCE fabric with `NCCL_IB_GID_INDEX` omitted (reboot-volatile GID tables).

### Measured (e2e wall-clock, 256-token prose, temp 0)

| concurrency | tok/s |
|---|---|
| C1 | 27.2 |
| C4 | 59.1 |
| C8 | 86.9 |

TTFT ≈ 0.6 s at ~780-token prompt. KV pool 200,064 tokens. MTP mean acceptance length
2.3–3.1/3. Beats the prior GLM-5.2 QuantTrio TP4 reference at every concurrency level.

### Quality (10-sample battery, temp 0)

GSM8K 80% · HumanEval 100% · MBPP 100% · IFEval 90% · MMLU-STEM 90% · ARC-C 90% ·
HellaSwag 90% → **91.4% average** — no meaningful loss vs the BF16-served baseline class.

## Provenance

Quantized, verified, and benchmarked on a fleet of DGX Spark nodes over a RoCE fabric by
an agent-assisted pipeline. Companion repo with the full quantization script, verification
gates, launch recipe, autoresearch sweep and raw results: see the repo README / linked
GitHub repository.