torchtitan — распределённое обучение LLM на PyTorch

★ 7.7 · devops

torchtitan is a Claude Code skill that enables PyTorch-native distributed LLM pretraining with composable 4D parallelism — FSDP2, tensor parallel, pipeline parallel, and context parallel — achieving 65%+ speedups over baselines on H100 GPUs. The skill covers the full training pipeline: tokenizer download via `download_hf_assets.py`, TOML-based configuration, launching with `torchrun` or `run_train.sh`, TensorBoard monitoring, and distributed checkpointing with automatic resume. It supports Llama 3.1 (8B, 70B, 405B), DeepSeek V3, and custom models, scaling from a single 8-GPU node to 512+ GPU SLURM clusters. Float8 training via torchao delivers 30–50% additional speedup on H100s, and torch.compile optimizes the compute graph. Aimed at ML engineers and researchers pretraining large language models from scratch without third-party frameworks.