quant.cpp — локальный запуск LLM со сжатием KV-кэша

vibe-coding

quant.cpp is an open-source C/Python library for running LLMs on consumer hardware using 6.4× KV-cache compression with near-zero quality loss. The project ships 17,600 lines of C with zero external dependencies and installs via `pip install quantcpp`. It enables 128K-token context on a 16 GB MacBook — Llama 3.2 3B fits in 9.5 GB where FP32 runs out of memory. The CLI covers model downloads, interactive chat, and an OpenAI-compatible HTTP server (`quantcpp serve` on port 8080 with SSE streaming), while the Python API works in three lines. Full-document inference scored 7/7 on a multi-hop QA benchmark versus 0/7 for chunk-RAG, making it a practical alternative for developers who need accurate long-context retrieval without a vector database.