vLLM — высоконагруженный инференс LLM на GPU

★ 27k

vLLM is a high-performance inference engine and server for large language models, built for speed and high throughput on GPUs. Reach for it when you need to serve LLMs in production under load: expose a model via an OpenAI-compatible API to many users at once, with efficient request batching and economical GPU-memory use (PagedAttention). It supports many open models and various GPUs. Its focus is a server engine for fast, high-load LLM inference on GPUs, not a desktop chat (Jan/LM Studio/Ollama) or an engine for modest hardware/CPU (llama.cpp): the value is maximum speed and throughput when serving models in production. It suits teams and services that need to deploy open LLMs with high performance and serve them via API to many clients. For a one-click local chat it is overkill — Ollama/LM Studio fill that.