GPTQ — 4-битная квантизация LLM для потребительских GPU

★ 7.7 · ai-tooling

gptq is a Claude Code skill that applies post-training 4-bit quantization to large language models using group-wise quantization, achieving less than 2% perplexity degradation while reducing memory footprint by 4×. It targets deployment of 70B and 405B models on consumer GPUs such as the RTX 4090 and 3090, delivering 3–4× inference speedup compared to FP16. The skill depends on auto-gptq, transformers, optimum, and peft, and supports ExLlamaV2 and Marlin kernel backends, configurable group sizes (128 recommended), and QLoRA fine-tuning via PEFT. ML engineers can use it both to load pre-quantized checkpoints from Hugging Face and to quantize their own models with calibration data — without any retraining of the original weights.