vLLM
Software for running models quickly for many users at once.
The fuller definition
vLLM is an open-source inference and serving system for supported language models, designed to use compute and memory efficiently across requests.
- Model
Compatible weights
- Serving engine
Manage inference requests
- Applications
Use the exposed interface
A simplified relationship, not a screenshot or a complete design.
A concrete example
A team serves a compatible model through an API for several internal applications.
The distinction that matters
Supported architectures and hardware depend on the version and configuration; installing a server does not solve access or privacy requirements.
Related terms
Model serving
Running a model so that other programs can send it requests.
APIs & softwareThroughput
How much work a system can get through in a given time.
Hardware & hostingKV cache — Key-value cache
Stored results from part of a model's earlier work. Reusing them can speed up the next step. This cache also takes memory.
Where it is taught
Choose an AI tool around the work productSource and further reading
vLLM · DocumentationThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.