PyTorch FSDP2 — распределённое обучение больших моделей

★ 7.7 · engineering

pytorch-fsdp2 is a Claude Code skill that integrates PyTorch FSDP2 (`fully_shard`) into training scripts with correct initialization, bottom-up sharding, mixed precision and CPU offload configuration, and distributed checkpointing via DCP. It targets scenarios where a model's parameters, gradients, and optimizer state exceed single-GPU memory, or where DTensor-based per-parameter sharding with DeviceMesh is preferred over FSDP1 for cleaner state dicts and later Tensor Parallel composition. The skill enforces a strict agent contract: launch via `torchrun`, shard submodules (e.g., Transformer blocks) before the root module, call `model(input)` so FSDP2 hooks fire, and build the optimizer after sharding so it operates on DTensor parameters. Checkpointing uses Distributed Checkpoint helpers rather than plain `torch.save`. Intended for ML engineers scaling large model training across multiple GPUs.