vLLM: serve open-weight models on your own GPUs

vLLM turns a GPU and an open-weight model into an OpenAI-compatible API. Version 0.31.0 shipped on 5 October 2026, and a new minor release lands roughly every two weeks with breaking changes in most of them, so pin the version you tested.

vLLM is an inference engine and an HTTP server in one package. You run vllm serve <model>, it downloads the weights from Hugging Face (or reads them from disk), loads them onto your GPUs and serves /v1/chat/completions, /v1/completions, /v1/responses, /v1/embeddings for embedding models, /v1/audio/transcriptions for speech models, and the Anthropic-style /v1/messages. It also exposes /health for probes and /metrics for Prometheus.

In the stack it is the serving layer. Your applications should not call it directly. They call a gateway such as LiteLLM, which holds the keys, budgets and fallbacks and routes some traffic to vLLM and the rest to hosted APIs. Traces and costs flow sideways to your observability backend.

It runs on NVIDIA GPUs (the default wheels and the vllm/vllm-openai image target CUDA 13.0, with a CUDA 12.9 image as well), AMD ROCm, Intel XPU and x86 or ARM CPUs. Other accelerators, including Google TPU and Intel Gaudi, run through hardware plugins maintained outside the main repository. The project is Apache-2.0 and has been hosted by the PyTorch Foundation since May 2025. In early 2026 several core maintainers founded Inferact, a company built around vLLM, and stated that the engine stays open source.

Pick vLLM when these are true:

• You serve an open-weight model with steady traffic. A GPU costs the same per hour whether it answers 10 requests or 10,000, so self-hosting only beats per-token pricing when you keep the GPU busy.
• Prompts must stay inside your network: health, finance or customer records that a policy keeps away from third-party APIs.
• You run many fine-tunes of one base model. Multi-LoRA serves dozens of adapters from one GPU instead of one GPU per fine-tune.
• You need structured outputs, tool calling or reasoning parsing on open models with an OpenAI-shaped API your apps already speak.

Skip it, or postpone it, when:

• Traffic is low or bursty. An idle GPU at 3 a.m. is pure cost; a hosted API or a serverless GPU platform is cheaper until volume is steady.
• You need a closed frontier model. vLLM serves open weights only.
• You only want a model on a laptop. Ollama or llama.cpp are simpler for single-user local use.
• Nobody on the team will own GPU drivers, capacity and on-call. A misconfigured server fails in ways that look like model quality problems.

The nearest alternatives are SGLang (Apache-2.0, similar scope) and NVIDIA TensorRT-LLM. Hugging Face archived its Text Generation Inference repository in 2026, so do not start new work on it. Cluster-level systems such as llm-d, NVIDIA Dynamo, KServe and AIBrix run vLLM underneath rather than replacing it.

Four ideas explain almost every tuning decision:

1. PagedAttention. Each request needs a key/value (KV) cache that grows with every token. vLLM stores it in fixed blocks of 16 tokens, allocated on demand like pages of virtual memory. Little memory is wasted, so many more sequences fit on one GPU, and requests with the same prefix can share blocks.
2. Continuous batching with chunked prefill. Every engine step, the scheduler fills a token budget (--max-num-batched-tokens). It schedules decode tokens for running requests first, then fills the rest with chunks of new prompts. Requests join and leave between steps instead of waiting for a whole batch. --max-num-seqs caps how many sequences run at once.
3. Automatic prefix caching. On by default. Blocks are hashed (SHA-256 by default), so a shared system prompt or the earlier turns of a chat are computed once and reused. It cuts time to first token, not decode time.
4. Preemption. When the KV cache is full, the scheduler evicts a running sequence and recomputes it later. Throughput survives, but latency spikes. A rising preemption counter is the first sign that you sized the server wrong.

The process layout matters for CPU sizing. One API server process handles HTTP and tokenization and talks over ZMQ to one engine core process, which runs the scheduler and the KV cache manager. Each GPU gets its own worker process. A four-GPU tensor-parallel server is six processes. The old V0 engine is gone, and Model Runner V2 has been the default since v0.29.0, with the V1 runner scheduled for removal in v0.32.

Memory is set by --gpu-memory-utilization, which defaults to 0.92 of the GPU for this instance. Weights, activations and CUDA graphs come out of that first, and the KV cache gets the rest. At startup vLLM logs the result in one line, for example GPU KV cache size: 643,232 tokens, Maximum concurrency for 40,960 tokens per request: 15.70x. That line is the most useful number in the whole log.