How much vram needed for 1mil context minimum in total with everything?
Maybe some fp8 or Q8 or Q6 or even Q4... maybe these versions dont exist yet?
a lot, you'd probably need a b200 cluster, maybe h100 cluster if quantized. So unless you have $200k sitting around for gpus, then its not feasible
a lot, you'd probably need a b200 cluster, maybe h100 cluster if quantized. So unless you have $200k sitting around for gpus, then its not feasible
thats a lot of downloads for llm that needs 200k setup..wow!
a lot, you'd probably need a b200 cluster, maybe h100 cluster if quantized. So unless you have $200k sitting around for gpus, then its not feasible
you can run this on an m3 ultra mac studio
At what quant tho? If you want 1m context then even at FP8 it requires at least 800gb vram
a lot, you'd probably need a b200 cluster, maybe h100 cluster if quantized. So unless you have $200k sitting around for gpus, then its not feasible
you can run this on an m3 ultra mac studio
Directly from the vllm recipe page
“GLM-5.2 is the newest model in the GLM-5 series — a ~743B-parameter MoE (39B active) from Z-AI. The headline change over GLM-5 / 5.1 is that Multi-Token Prediction (MTP) is extended from 3 to 5 draft tokens, lifting end-to-end throughput on reasoning, coding, and agentic workloads. It ships as BF16 and native-FP8 checkpoints and keeps the GLM thinking-mode behavior.
This recipe targets the FP8 checkpoint, the practical default: it fits on a single 8xH200 / 8xH20 node and — with FP8 KV cache — reaches the full 1M-token context on 8xB200.
Prerequisites
vLLM 0.23.0 (stable). If you need tool calling and MTP at the same time, use the latest main branch.
GPU: 8xH200 or 8xH20 (141 GB each) for single-node FP8; 8xB200 (180 GB each) for the full 1M context.”
If you use the unsloth Q2 quant, you'll require at least 400GB
Don't use quant if ur planning to use VLLM .
May lab machiney.
2x h200 on Qwen 3.6 27B and 6x H200 on GLM-5.2-UD-Q4_K_XL
