AWQ — 4-битное сжатие языковых моделей

★ 7.7 · ai-tooling

awq is a Claude Code skill that implements Activation-aware Weight Quantization (AWQ) to compress large language models to 4-bit precision, delivering 2.5–3× inference speedup with under 5% accuracy loss — recognized as the MLSys 2024 Best Paper Award winner. Built on the autoawq library with transformers ≥ 4.45.0 and torch ≥ 2.0.0, the skill supports GEMM, GEMV, and Marlin kernel backends (requiring Ampere+ GPUs: A100, H100, RTX 40xx), plus native vLLM and HuggingFace Transformers integration. The core mechanism protects ~1% of salient weights identified via activation patterns, cutting quantization error without mixed-precision overhead. It targets production deployment of instruction-tuned and multimodal models in the 7B–70B range on memory-constrained GPUs: Mistral 7B shrinks from 14 GB to 5.5 GB, Llama 2-70B from 140 GB to 35 GB. Quantizing a 7B model takes 10–15 minutes; a 70B model takes roughly one hour.

• #quantization
• #llm-compression
• #gpu-optimization
• #4-bit
• #inference
• #model-deployment
• #awq