Megatron-Core — распределённое обучение LLM на GPU

★ 7.7 · engineering

megatron-core is a Claude Code skill that guides training large language models from 2B to 462B parameters using NVIDIA Megatron-Core. It covers the full workflow: selecting tensor, pipeline, context, expert, and sequence parallelism strategies based on model size, launching distributed runs via torchrun or SLURM, enabling FP8 mixed precision for 1.5–2× speedup on H100, and configuring Mixture-of-Experts with expert parallelism that cuts per-GPU memory by up to 75%. The skill targets up to 47% Model FLOP Utilization on H100 and is built around production-ready configurations used for Nemotron, LLaMA, and DeepSeek. Dependencies include megatron-core, torch, apex, and transformer-engine.