llama.cpp — локальный инференс LLM без GPU

★ 7.7 · orchestration

llama-cpp is a Claude Code skill that runs LLM inference on CPU, Apple Silicon (M1/M2/M3/M4), AMD, and Intel GPUs — anywhere CUDA is unavailable — using a pure C/C++ engine with minimal dependencies. It supports GGUF quantization from 1.5 to 8 bits, delivering up to 4–10× speedup over PyTorch on CPU; the recommended Q4_K_M format fits a 7B model in 4.1 GB of RAM. The skill covers CLI inference via llama-cli, an OpenAI-compatible server mode via llama-server, batch processing, grammar-constrained generation (GBNF), and hybrid CPU+GPU offloading for large models such as Llama 3 70B or 405B. Supported model families include Llama 2/3, Mistral, Mixtral, Phi-3, Gemma, Qwen, LLaVA, and any GGUF model from HuggingFace. It is the go-to choice for edge deployment on Raspberry Pi or embedded hardware, local Mac development without a discrete GPU, and memory-constrained environments where quantization quality trade-offs matter.