GLM-5.3-Flash INT4 for QEngine
GLM-5.3-Flash quantized for QEngine 1.5 or later, an open-source (MIT) inference engine for 3x NVIDIA RTX 3090. These files are laid out for QEngine's GLM engine; other runtimes won't read them as a normal checkpoint.
hf download bpx03/GLM-5.3-Flash-INT4-QEngine --local-dir models/GLM-5.3-Flash-INT4-QEngine
server/run.sh 8080 --model models/GLM-5.3-Flash-INT4-QEngine
What you need
- 3x 24 GB GPUs is what I use (it also starts on 2 cards, with fewer expert slots in VRAM). P2P between the cards helps but isn't required.
- About 192 GB of host RAM: the routed experts (about 157 GB) stay pinned in host RAM and the cards cache the hot ones. With less RAM the engine starts with a smaller conversation archive; below the expert size it can't load.
- An x86-64 CPU with AVX2. AVX-512 is used when the CPU has it (the CPU expert lane picks its kernels at start).
- About 186 GB of disk.
What is in it
- Routed experts: GPTQ INT4 (
q_g4_experts-*.safetensors), calibrated on my own records, with QEngine's expert layout. - KDA (linear attention) projections: GPTQ INT4 g128 with act-order, from the BF16 weights (
q_kda-*.safetensors). QEngine keeps them in INT4 in VRAM; the decode reads the packed values directly. - Everything else (attention, MLA, the shared expert, router, embeddings, head, the MTP draft layer) from the INT4
AutoRound release of GLM-5.3-Flash by Intel (W4A16; its non-quantized tensors), repacked into
model-dense-*.safetensors. SHA256SUMSfor every file.
Both quantization steps are lossy. Numbers (KL against the FP8 reference, task accuracy, speed) are on the QEngine README for 1.5.
License
MIT, as the original GLM-5.3-Flash model. Please also follow the license of the original model.
- Downloads last month
- -
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support
Model tree for bpx03/GLM-5.3-Flash-INT4-QEngine
Base model
zai-org/GLM-5.3-Flash