Does it work with expert offloading to CPU?
#2
by MLDataScientist - opened
Vllm has expert offloading function now: https://github.com/vllm-project/vllm/pull/34535
I want to ask you if we can run this model by offloading all experts to CPU RAM while keeping attention part in GPU? I have 8 channel 256GB DDR4 3200mhz and one 5090 32GB VRAM. I will try this model in my setup later this week. But I wanted to check if you tried expert offloading and what speed you got. I hope vllm will have llama.cpp speed one day for CPU+GPU inference.
Example command for expert offloading:
VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1 python3 -m vllm.entrypoints.openai.api_server \
--model nvidia/Kimi-K2-Thinking-NVFP4 \
--trust-remote-code \
--cpu-offload-gb 350 \
--load-format dummy \
--cpu-offload-params w13_weight w2_weight