Publish GLM-5.3 One-Spark discovery card
Browse files
README.md
ADDED
|
@@ -0,0 +1,72 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
library_name: vllm
|
| 4 |
+
tags:
|
| 5 |
+
- glm-5.3-flash
|
| 6 |
+
- dgx-spark
|
| 7 |
+
- gb10
|
| 8 |
+
- exl3
|
| 9 |
+
- vllm
|
| 10 |
+
- dflash2
|
| 11 |
+
- speculative-decoding
|
| 12 |
+
- arm64
|
| 13 |
+
- deployment
|
| 14 |
+
base_model:
|
| 15 |
+
- zai-org/GLM-5.3-Flash
|
| 16 |
+
- turboderp/GLM-5.3-Flash-exl3
|
| 17 |
+
---
|
| 18 |
+
|
| 19 |
+
# GLM-5.3-Flash EXL3 2.05 — One DGX Spark
|
| 20 |
+
|
| 21 |
+
A reproducible, production-capable deployment of **GLM-5.3-Flash on one NVIDIA DGX Spark**, using:
|
| 22 |
+
|
| 23 |
+
- [Turboderp's 2.05-bpw EXL3 checkpoint](https://huggingface.co/turboderp/GLM-5.3-Flash-exl3/tree/2.05bpw)
|
| 24 |
+
- [Inco AI's DFlash2 K7 drafter](https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2)
|
| 25 |
+
- A TP1 ARM64/SM121 vLLM runtime derived from [MiaAI's two-Spark recipe](https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks)
|
| 26 |
+
|
| 27 |
+
> **64.1 tok/s structured C1 · 25.1 tok/s prose · 181.9 tok/s C4 active-stream aggregate · 262K context**
|
| 28 |
+
|
| 29 |
+
## This repository does not contain model weights
|
| 30 |
+
|
| 31 |
+
This is a **deployment/runtime discovery page**, not a new model or quant. The software, Docker recipe, benchmark harnesses, and raw evidence live in the linked GitHub repository. Target and draft weights download directly from their original publishers.
|
| 32 |
+
|
| 33 |
+
## Measured results
|
| 34 |
+
|
| 35 |
+
One DGX Spark, TP1, EXL3 2.05 bpw, DFlash2 K7, FP8 KV, thinking disabled:
|
| 36 |
+
|
| 37 |
+
| Benchmark | Result |
|
| 38 |
+
|---|---:|
|
| 39 |
+
| Structured C1, five-run median, temperature 0 | **64.053 tok/s** |
|
| 40 |
+
| Structured C1, five-run median, temperature 1.0/top-p 0.95 | **62.637 tok/s** |
|
| 41 |
+
| Open-ended prose, five-run median | **25.059 tok/s** |
|
| 42 |
+
| C4 median active stream | **40.958 tok/s/stream** |
|
| 43 |
+
| C4 summed active-stream convention | **181.944 tok/s** |
|
| 44 |
+
| C4 strict submission-to-completion wall | **91.475 tok/s** |
|
| 45 |
+
|
| 46 |
+
**C4 disclosure:** 181.944 tok/s is the sum of each stream's active decode rate, matching MiaAI's reporting convention. Because the current scheduler stages admission, strict full-batch wall throughput is 91.475 tok/s. Both numbers are published intentionally.
|
| 47 |
+
|
| 48 |
+
## Long-context results
|
| 49 |
+
|
| 50 |
+
| Prompt | Cold prefill | Cold TTFT | Warm TTFT |
|
| 51 |
+
|---:|---:|---:|---:|
|
| 52 |
+
| 8K | **786.4 tok/s** | 10.175 s | 1.775 s |
|
| 53 |
+
| 16K | **822.1 tok/s** | 19.464 s | 2.786 s |
|
| 54 |
+
| 100K | **845.7 tok/s** | 118.250 s | 8.842 s |
|
| 55 |
+
|
| 56 |
+
Every cold request had zero prefix-cache hits; every warm checksum response was correct.
|
| 57 |
+
|
| 58 |
+
## Links
|
| 59 |
+
|
| 60 |
+
- **GitHub source and instructions:** `https://github.com/gitcommit90/glm-5.3-one-spark`
|
| 61 |
+
- **Prebuilt ARM64 runtime image:** `ghcr.io/gitcommit90/glm-5.3-one-spark:general23`
|
| 62 |
+
- **Target checkpoint:** https://huggingface.co/turboderp/GLM-5.3-Flash-exl3/tree/2.05bpw
|
| 63 |
+
- **DFlash2 checkpoint:** https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2
|
| 64 |
+
- **Z.ai base model:** https://huggingface.co/zai-org/GLM-5.3-Flash
|
| 65 |
+
|
| 66 |
+
## Credits and licensing
|
| 67 |
+
|
| 68 |
+
The runtime is derived from Mia's AI Lab's MIT-licensed two-Spark recipe and substantially adapted for TP1, full-model 2.05-bpw `mul1` EXL3, ARM64/SM121, and one-Spark memory limits.
|
| 69 |
+
|
| 70 |
+
Credits: Z.ai / GLM-5 Team, Turboderp and ExLlamaV3, Inco AI, Mia's AI Lab, and the vLLM contributors.
|
| 71 |
+
|
| 72 |
+
**DFlash2 is CC BY-NC-ND 4.0 for research/evaluation.** It is not bundled here. Commercial users must obtain appropriate licensing from Inco AI.
|