Instructions to use drowzeys/keys-GLM-5.3-DFlash2-ft8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use drowzeys/keys-GLM-5.3-DFlash2-ft8 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="drowzeys/keys-GLM-5.3-DFlash2-ft8")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("drowzeys/keys-GLM-5.3-DFlash2-ft8") model = AutoModel.from_pretrained("drowzeys/keys-GLM-5.3-DFlash2-ft8", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use drowzeys/keys-GLM-5.3-DFlash2-ft8 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "drowzeys/keys-GLM-5.3-DFlash2-ft8" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "drowzeys/keys-GLM-5.3-DFlash2-ft8", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/drowzeys/keys-GLM-5.3-DFlash2-ft8
- SGLang
How to use drowzeys/keys-GLM-5.3-DFlash2-ft8 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "drowzeys/keys-GLM-5.3-DFlash2-ft8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "drowzeys/keys-GLM-5.3-DFlash2-ft8", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "drowzeys/keys-GLM-5.3-DFlash2-ft8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "drowzeys/keys-GLM-5.3-DFlash2-ft8", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use drowzeys/keys-GLM-5.3-DFlash2-ft8 with Docker Model Runner:
docker model run hf.co/drowzeys/keys-GLM-5.3-DFlash2-ft8
GLM-5.3-DFlash2-ft8
A fine-tune of incoai/GLM-5.3-DFlash2, the DFlash 2 draft model
for zai-org/GLM-5.3. It is not a standalone language model: it runs
inside a speculative-decoding server and drafts tokens for the target model to verify. Decoding stays lossless, as
with the base drafter: the target accepts or rejects every drafted token.
What changed from the base: the weights only. The architecture and config.json are the base model's
(6 layers, block size 8, selector rank 256). It was trained further "on policy": on hidden states and next-token
labels captured from a running GLM-5.3 target over about 2,900 captured sequences (prose, code and long-context
chats), so the drafter sees the same inputs in training that it sees when serving.
Which target it was trained against: a 2.75 bits-per-weight EXL3 quantisation of a modified variant of GLM-5.3, served across four NVIDIA DGX Sparks (tensor parallel 4). That is the only target it has been measured on.
Measured
Four DGX Sparks (GB10), tensor parallel 4, one stream, thinking on, the drafter alone (no MTP heads), draft depth 7, confidence 0.3. Measured 2026-10-07.
| this drafter | MTP heads only | |
|---|---|---|
| prose, short / 32K / ~120K context (tokens per second) | 39.1 / 34.2 / 31.2 | 37.1 / 32.6 / not measured |
| code, short / 32K / ~120K context (tokens per second) | 44.8 / 42.9 / 39.4 | 41.1 / 35.2 / not measured |
| prompt processing, 32K / 128K (tokens per second) | 1,196 / 1,061 | 1,188 / 1,039 |
Accepted tokens per decoding round on prose at 32K context, same protocol:
| drafter | tokens per round |
|---|---|
incoai/GLM-5.3-DFlash2 (base) |
2.029 |
| an earlier fine-tune (ft7) | 2.057 |
| this one (ft8) | 2.099 |
What is not known
- Other targets. It has not been measured against full-precision GLM-5.3, an NVFP4 target or any other quantisation. A drafter tuned to one target's hidden states may accept more or fewer tokens on another; measure before relying on it, and compare with the base drafter.
- Other servers. The file layout is the base model's (
config.json,model.safetensors), so a server that loadsincoai/GLM-5.3-DFlash2should load this in its place. It has not been tested by us on SGLang or vLLM. - Noise. The prose figures are six runs a setting, with run-to-run noise of about 3 %. The gain over the base drafter (about 3.5 % more tokens per round on prose) is small.
- Other hardware. All figures are from DGX Sparks.
Use
Point your server's draft-model path at this repository instead of the base drafter. With the base model card's
SGLang example, that is --speculative-draft-model-path drowzeys/keys-GLM-5.3-DFlash2-ft8.
model.safetensors md5: a50f76216578ff2ce3479d6873da1952.
Licence and credit
Apache License 2.0, the licence of the base drafter (see LICENSE and NOTICE). The base drafter and the DFlash 2
method are the work of the authors of incoai/GLM-5.3-DFlash2
(blog, code). The target model is
zai-org/GLM-5.3 and its licence applies to it. This repository changes
the base drafter's weights by further training and is not affiliated with or endorsed by either.
- Downloads last month
- 57