Papers
arxiv:2604.07609

Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC

Published on Apr 8
Authors:
,
,
,
,

Abstract

Blink removes the host CPU from steady-state LLM inference by offloading request handling to a SmartNIC and using a persistent GPU kernel for batching, scheduling, and KV-cache management, improving latency, throughput, and energy efficiency under CPU interference.

Large Language Model (LLM) inference is rapidly becoming a core datacenter service, yet current serving stacks keep the host CPU on the critical path for orchestration and token-level control. This makes LLM performance sensitive to CPU interference, undermining application colocation and forcing operators to reserve CPU headroom, leaving substantial capacity unutilized. We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path by redistributing responsibilities across a SmartNIC and a GPU. Blink offloads request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA, and replaces host-driven scheduling with a persistent GPU kernel that performs batching, scheduling, and KV-cache management without CPU involvement. Evaluated against TensorRT-LLM, vLLM, and SGLang, Blink outperforms all baselines even in isolation, reducing pre-saturation P99 TTFT by up to 8.47times and P99 TPOT by up to 3.40times, improving decode throughput by up to 2.1times, and reducing energy per token by up to 48.6%. Under CPU interference, Blink maintains stable performance, while existing systems degrade by up to two orders of magnitude.

Community

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2604.07609
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2604.07609 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2604.07609 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2604.07609 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.