Flash Attention — ускорение трансформеров и экономия видеопамяти

★ 7.7 · ai-tooling

flash-attention is a Claude Code skill that optimizes transformer attention using Flash Attention, delivering 2–4x speedup and 10–20x memory reduction through IO-aware tiling and recomputation. It covers two main approaches: PyTorch native SDPA (requires PyTorch 2.2+) and the flash-attn library for advanced use cases including multi-query attention, sliding window attention, and H100 FP8 optimizations. The skill includes step-by-step workflows for replacing standard attention, forcing the Flash Attention backend, benchmarking with torch.utils.benchmark, and validating output accuracy against a baseline. Dependencies are flash-attn, torch, and transformers. It targets ML engineers and researchers training or running transformer models with long sequences (>512 tokens) who need to reduce GPU memory pressure or speed up inference.