malaiwah/GLM-5.2-QSRT-Fruit

GLM-5.2 QSRT Fruit

This is a 5.04B-parameter GLM-5.2 serving proxy, not the 754B GLM-5.2 model. It is the first complete Fruit checkpoint encoded in KQuant's canonical QSRT atom format and served without reconstructing dense expert weights.

The artifact is a codec/storage/runtime integration release. Packaging, provenance, exact state decoding, and the canonical single-GPU runtime path are implemented. The checkpoint itself has no downstream chat-quality qualification; see Known limitations.

What is included

• 13 transformer layers: 3 dense and 10 MoE layers, plus the packaged MTP expert layer.
• Hidden size 1,024; MoE intermediate size 512; 256 routed experts per MoE/MTP layer.
• 2,816 QSRT experts in 11 canonical atom files.
• SQG-XOR-Cheb-T12 E4M3 codebook, three-bit trellis payload, fixed P24/P33 pair records, and physical atom rotation.
• Canonical qsrt_atoms_v1 storage with complete per-file SHA-256 manifests and a fail-closed QSRT_COMPLETE.json marker.
• W4A16 prefill/reference execution and W4A8 decode execution through B12X.

The expert allocation selected by the frozen calibration evidence is recorded in each qsrt-layer-*.json sidecar. Aggregate allocation counts are:

Allocation code	Experts
R13=0,R2=0	2,804
R13=1,R2=0	10
R13=1,R2=1	1
R13=2,R2=2	1

Size and memory

The apples-to-apples baseline is the complete BF16 tensor set. All three rows below cover the same 5,040,368,896 logical parameters and count only Safetensors files; effective bpw is stored bytes times eight divided by that parameter count, so it includes container and quantization metadata.

Tensor payload	Bytes	GiB	Effective bpw	Relative to BF16
BF16 source	10,081,800,232	9.3894	16.0017	baseline
Prior SIQ mixed	3,102,116,152	2.8891	4.9236	69.23% smaller
This QSRT model	2,909,352,104	2.7095	4.6177	71.14% smaller

The whole-model rates include 611,183,872 non-routed parameters retained in BF16. Isolating the 4,429,185,024 routed-expert weights gives:

Routed-expert format	Stored bytes	Nominal path bpw	Effective stored bpw
BF16	8,858,370,048	16.0000	16.0000
Prior SIQ mixed (1,856 K3 / 960 K4 experts)	1,879,717,272	3.3409	3.3951
QSRT P24/P33 atoms	1,686,953,224	3.0000	3.0470

QSRT is therefore 10.25% smaller than SIQ on the routed-expert component and 6.21% smaller on the compared tensor files. Package-level totals are not used for the comparison because tokenizer, card, and optional evaluation files are not model weights. The previous card's 7,593,020,594-byte BF16 row was not the complete BF16 tensor set and has been removed.

W4A8 and W4A16 use the same stored weights, so their loader weight storage is identical; W4A8 changes the decode execution path, not the checkpoint size.

Hugging Face repository-size audit

The Hugging Face model API with blobs=true reported the following immutable snapshot on 2026-08-08. Repository bytes sums every sibling's reported size; Safetensors bytes sums only *.safetensors. These are observed repository payloads, not parameter-count estimates.

Artifact	Revision	Repository bytes	Safetensors bytes
Fruit QSRT (pre-adjacent-rate-evidence publication)	c1a0c62d	2,963,027,998	2,909,352,104
Fruit BF16	ff1178d2	10,102,776,679	10,081,800,232
Fruit prior mixed SIQ	c1798e36	3,125,527,019	3,102,116,152
Full GLM-5.2 BF16	b4734de4	1,506,693,036,946	1,506,667,387,408
Full GLM-5.2 FP8	ba978f7d	755,663,676,164	755,632,050,320
Full GLM-5.2 NVFP4	aec724e8	464,874,323,992	464,823,042,096

The three full-model rows ground real download/storage scale only. They are not used for Fruit percentage claims because Fruit has 5.04B parameters while the production model has roughly 754B. The apples-to-apples Fruit tensor comparison above remains the codec-size result.

Adjacent-rate evidence

evaluation/fruit-uniform-rate-sweep.json re-encodes the same authenticated expert sample at uniform K2, K3, and K4, with fresh per-endpoint FP16 scales. It measured 16 of 19 predeclared assignments; 3 lacked the minimum routed calibration support and were skipped rather than imputed. These are pre-layer-deduplication expert-local endpoints, not package sizes.

Endpoint	Mean bpw	Mean bytes/expert	Weight NMSE	Captured-H NMSE	Validation routed NMSE
K2	2.0469	402,432	0.082011	0.024647	0.061683
K3	3.0469	599,040	0.020772	0.006091	0.014614
K4	4.0469	795,648	0.005338	0.001558	0.003707

Relative to K3, K2 is 3.948x / 4.046x / 4.221x on weight, captured-H, and validation-routed NMSE. K4 is 0.257x / 0.256x / 0.254x on the same metrics.

Evidence boundary

The completion seal covers every top-level package file and every regular file under evaluation/. The sealed adjacent-rate report measures local routed expert reconstruction on authenticated, document-disjoint calibration rows. It does not establish chat quality, broad downstream task quality, or general serving throughput.

Reproducible runtime

The model requires the matching experimental branches until the pull requests merge:

• KQuant encoder: local-inference-lab/kquant#4, encoded with KQuant revision f1ce7c8f4a9564194ea7067e1a88282a8e39135c.
• B12X kernels: local-inference-lab/b12x#129, tested revision de50e8622a8695e9829c83ad9f8c96f9b3be573a.
• vLLM loader: local-inference-lab/vllm#269, tested revision 12e7c76b15075a5b9aadf77816f13c0bd775f971.

git clone --branch feat/fruit-qsrt-runtime https://github.com/malaiwah/sparkinfer.git b12x-fruit
git -C b12x-fruit checkout de50e8622a8695e9829c83ad9f8c96f9b3be573a
git clone --branch feat/fruit-qsrt-runtime https://github.com/malaiwah/vllm-voipmonitor.git vllm-fruit
git -C vllm-fruit checkout 12e7c76b15075a5b9aadf77816f13c0bd775f971

hf download malaiwah/GLM-5.2-QSRT-Fruit --local-dir GLM-5.2-QSRT-Fruit

B12X_ROOT="$PWD/b12x-fruit" \
MODEL="$PWD/GLM-5.2-QSRT-Fruit" \
PYTHON_BIN="$PWD/vllm-fruit/.venv/bin/python" \
CUDA_VISIBLE_DEVICES=0 \
MAX_NUM_SEQS=1 \
./vllm-fruit/serve-glm52-fruit-qsrt.sh

The tested environment uses SM120, CUDA 13.2-era wheels, nvidia-cutlass-dsl >= 4.6, and the r31 vLLM/B12X image stack. The launcher defaults to one sequence because the current B12X sparse-prefill backend requires single-request prefill chunks. Only TP1 has been validated for this Fruit package.

W4A16 is used for prefill and any row count above the W4A8 decode ceiling. W4A8 is selected for decode-sized batches of at most 16 rows. Unsupported shapes, activation modes, metadata, or incomplete manifests fail closed.

Provenance and integrity

• Authenticated encoder source (safetensors_manifest) SHA-256: 8a7e30f3a948bbac203013160b2e6bb8d0ed50c36cf2ca1c3978701124cc7671.
• Calibration capture ID: cc686d28f505d62653763cdb746e830374207137d103d3c26ebbcce070059053.
• Calibration manifest SHA-256: 77fd947235b89e67549fa264a08dc1436d3913ba864d58da558d78e75d892cde.
• The encoder authenticated 256 documents / 46,223 tokens from disjoint fit, confirmation, and validation splits.
• Full encoding: 2,816 experts, 9,606.90 GPU-seconds, 2.544 GiB peak CUDA allocation.
• MANIFEST.sha256, qsrt-manifest.json, .qsrt-source-evidence.json, qsrt-calibration-evidence.json, and QSRT_COMPLETE.json bind the published package to the source and encoder fingerprints.

Known limitations

• Not chat-quality. An informal four-prompt instruction battery produced incoherent answers. No representative downstream instruction/chat evaluation is sealed by the builder, so no task-quality claim is made and this checkpoint must not be deployed as a user-facing assistant.
• TP2 atom ownership is unit-tested, but only TP1 physical serving has been qualified for this Fruit package.
• The current sparse-attention prefill backend requires max_num_seqs=1.
• This release qualifies the QSRT codec, storage, loader, and kernels. It does not establish broad downstream task quality.

License

MIT, matching the packaged Fruit source license. KQuant, B12X, and vLLM retain their respective repository licenses.