vLLM

The High-Throughput and Memory-Efficient inference and serving engine for LLMs

Docs · Recipes · Blog · Forum · Slack

• Easy — Deploy the widest range of open-source models on any hardware. Includes a drop-in OpenAI-compatible API for instant integration.
• Fast — Maximize throughput with PagedAttention. Advanced scheduling and continuous batching ensure peak GPU utilization.
• Cost Efficient — Slash inference costs by maximizing hardware efficiency. We make high-performance LLMs affordable and accessible to everyone.

uv pip install vllm --torch-backend auto
vllm serve <hf-model-id>

Runs on NVIDIA, AMD, Intel, Google TPU, AWS Neuron, Huawei Ascend, Apple Silicon and more. vLLM is a PyTorch Foundation project.