nanoGPT — обучение трансформера с нуля

★ 7.7 · learning

nanogpt is a Claude Code skill that implements the GPT architecture in ~300 lines of code for learning transformers from scratch. It reproduces GPT-2 (124M parameters) on OpenWebText and supports three main workflows: character-level training on Shakespeare (~5 minutes on CPU), full GPT-2 reproduction via multi-GPU DDP (8× A100, ~4 days), and fine-tuning from OpenAI pretrained checkpoints (gpt2, gpt2-medium, gpt2-large, gpt2-xl). The codebase is split into model.py and train.py with no unnecessary abstractions — pure PyTorch, with torch, transformers, datasets, tiktoken, and wandb as dependencies. It's the go-to starting point for anyone who wants to understand GPT internals, experiment with transformer variants, or train a custom language model on their own text dataset.