Pending GPU & vLLM validation
I'm working with a server equipped with 8×48GB GPUs, and I'd like to know whether it can successfully load and inference the model weights.
The BF16 model needs ~1,510 GB VRAM, so 8×48GB (384 GB) isn't enough. Options:
- Unsloth 2-bit GGUF (236 GB) — fits, runs via llama.cpp. See unsloth/GLM-5.1-GGUF
- TQ3 checkpoint (309 GB) — doesn't quite fit in 384 GB either. Available at varjosoft/GLM-5.1-Open-TQ3 but needs ~400+ GB VRAM
- You'd need 8×80GB (A100/H100) or 4×141GB (H200) for the TQ3 version
Note: the TQ3 checkpoint was created using code validated on GLM-4.7-Flash (same architecture), but we haven't run inference on GLM-5.1 itself yet — waiting on multi-GPU availability for testing. The checkpoint and model card will be updated with results once tested.
The 400+ GB estimate was conservative and assumed standard vLLM loading, which materializes the full BF16 model in VRAM during initialization (309 GB + KV cache + CUDA overhead).
We're actively working on a --quantization turboquant flag that uses meta-device initialization... the model allocates zero GPU memory at init, then loads and compresses weights one layer at a time.
With this approach, the compressed model should fit in ~75-80 GB total VRAM, which would work on 2×H200 (282 GB) or potentially 8×48 GB (384 GB) with TP=8.
This is still being validated on GPU — we'll update the model card with confirmed results and serving instructions once tested. Stay tuned.