Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
Abstract
Blink removes the host CPU from steady-state LLM inference by offloading request handling to a SmartNIC and using a persistent GPU kernel for batching, scheduling, and KV-cache management, improving latency, throughput, and energy efficiency under CPU interference.
Large Language Model (LLM) inference is rapidly becoming a core datacenter service, yet current serving stacks keep the host CPU on the critical path for orchestration and token-level control. This makes LLM performance sensitive to CPU interference, undermining application colocation and forcing operators to reserve CPU headroom, leaving substantial capacity unutilized. We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path by redistributing responsibilities across a SmartNIC and a GPU. Blink offloads request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA, and replaces host-driven scheduling with a persistent GPU kernel that performs batching, scheduling, and KV-cache management without CPU involvement. Evaluated against TensorRT-LLM, vLLM, and SGLang, Blink outperforms all baselines even in isolation, reducing pre-saturation P99 TTFT by up to 8.47times and P99 TPOT by up to 3.40times, improving decode throughput by up to 2.1times, and reducing energy per token by up to 48.6%. Under CPU interference, Blink maintains stable performance, while existing systems degrade by up to two orders of magnitude.
Get this paper in your agent:
hf papers read 2604.07609 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper