Andrej Karpathy - Neural Networks: Zero to Hero (Building GPT from Scratch)

📌 What is this course?

Created by Andrej Karpathy (former Director of AI at Tesla and OpenAI founding researcher), Neural Networks: Zero to Hero is arguably the most respected hands-on deep learning curriculum on the internet.

Rather than relying on high-level abstractions in PyTorch or Keras, Karpathy starts with a tiny scalar autograd engine called micrograd, demonstrating exactly how gradients flow through computational graphs. The series progresses through character-level language modeling, multi-layer perceptrons (MLP), manual backpropagation math, batch normalization debugging, and culminates in building a generative transformer (nanoGPT, following the GPT-2 architecture) from scratch and pretraining it on Shakespeare text.

---

🎯 Who is this for?

• Developers who use PyTorch or Hugging Face but feel frustrated not understanding what happens inside the tensor graph.
• Engineers eager to master the exact mathematical and algorithmic implementation of Self-Attention and Transformers.
• Anyone who wants to observe a world-class AI researcher write, test, and debug neural network code line by line.

---

📚 Core Syllabus

1. The spelled-out intro to neural networks and backpropagation: Building micrograd from scratch
2. The spelled-out intro to language modeling: Building makemore part 1 (Bigram model)
3. Building makemore Part 2 (MLP): Following the Bengio et al. 2003 paper
4. Building makemore Part 3 (Activations & Gradients): BatchNorm and Kaiming initialization
5. Building makemore Part 4 (Manual Backpropagation): Deriving tensor calculus by hand
6. Building makemore Part 5 (WaveNet): Hierarchical convolutions for longer context
7. Let’s build GPT: from scratch, in code, spelled out: Implementing Multi-Head Attention and Transformers
8. Let’s build the GPT Tokenizer: Understanding Byte-Pair Encoding (BPE) from scratch