vllm

Project Overview

vLLM is a fast, easy-to-use library for LLM inference and serving, born at UC Berkeley and now maintained by over 2000 contributors. It manages GPU memory with PagedAttention and schedules requests with continuous batching, substantially raising serving throughput on the same hardware. It exposes an OpenAI-compatible API and supports 200+ Hugging Face model architectures, including LLMs, multi-modal models, and embedding models.

Turn a large model into a high-performance API service with one command, at far better hardware utilization.

Key Features

• PagedAttention: manages the KV cache like OS memory pages for high memory efficiency
• Continuous batching: requests join batches on arrival, with chunked prefill and prefix caching
• Optimized kernels: ships FlashAttention, FlashInfer, and other tuned attention kernels
• Quantization: FP8, INT8, GPTQ, AWQ, GGUF, and more
• OpenAI-compatible API: plugs into the existing OpenAI SDK ecosystem, plus Anthropic and gRPC interfaces
• Structured output: constrains generation to JSON and other formats with xgrammar
• Distributed inference: tensor, pipeline, data, and expert parallelism
• Multi-LoRA: serves many LoRA adapters from one instance
• Speculative decoding: n-gram, EAGLE, and other schemes to speed up generation

Target Audience

• Backend engineers self-hosting LLM APIs
• Data teams running large batch inference
• Enterprises deploying open models to production
• Researchers studying LLM serving performance

Quick Start

1. Install (uv recommended):

uv pip install vllm
vllm download-kernels

2. Start an OpenAI-compatible server:

vllm serve <model name>

3. Point any OpenAI SDK at the local port; see the official quickstart for details

Core Advantages

• High throughput: serves far more requests on the same hardware, lowering cost
• Ecosystem compatible: Hugging Face models work out of the box, and the API matches OpenAI
• Broad hardware: backends for NVIDIA, AMD, Intel GPUs, CPUs, and TPUs

System Requirements

Notes

• Installation: GPU builds need a PyTorch wheel matching your CUDA environment; building from source is more involved
• Kernel prefetch: run vllm download-kernels after install to speed up server startup