llama.cpp — локальный инференс LLM без GPU

★ 65k

llama.cpp is a high-performance LLM inference engine in C/C++, optimized to run on ordinary hardware, including without a powerful GPU. Reach for it when you need to run open LLMs locally quickly and economically: run models on CPU or GPU, on a laptop, server, or even a modest device, using quantized GGUF-format models for compactness and speed. It is the foundation many user-facing tools are built on (Ollama, LM Studio, and others), and it includes server examples and bindings to various languages. Its focus is a low-level, fast local LLM inference engine with quantization, not a finished chat app (Jan/LM Studio/Ollama for that) or a high-throughput production server with GPU batching (vLLM for that): the value is efficient model running on varied, including modest, hardware. It suits developers and enthusiasts who want control and performance in local inference; the GGUF format is the de facto standard for quantized weights.