deep-learning-notes
CS336
中文
English
Part 4: Attention Mechanism and Transformer
Part 1: Deep Learning Fundamentals
Chapter 1: Introduction to Deep Learning
1.1 Neural Networks: A Learnable Function
1.2 Loss Function: How Does a Model Know How Wrong It Is?
1.3 Forward Propagation, Backpropagation, and Computation Graph
1.4 Gradient Descent: From Gradients to Parameter Updates
1.5 Why Neural Networks Can Be Trained: Optimization Intuition in High-Dimensional Spaces
Chapter 2: Getting Started with PyTorch
2.1 Automatic Differentiation in PyTorch: From Forward Computation to Backpropagation
2.2 Gradient Modes in PyTorch: Controlling How Computation Graphs Are Recorded
2.3 Data Loading in PyTorch: Dataset, DataLoader, and Batching
2.4 nn.Module in PyTorch: Organizing Models, Parameters, and State
2.5 Optimizers in PyTorch: From Manual Updates to Parameter Groups and State Management
2.6 Training Loop in PyTorch: Connecting Data, Models, and Optimizers
2.7 Checkpoints in PyTorch: Resuming Training After Interruption
Chapter 3: Multi-Layer Perceptron: From Single Layer to Deep Nonlinear Modeling
3.1 From Linear Classifiers to MLPs: Why We Need Hidden Layers
3.2 Activation Functions: Adding Nonlinearity to Neural Networks
3.3 Softmax and Cross Entropy: From Logits to Classification Loss
3.4 Forward and Backward Propagation of Linear Layers
3.5 Building a Complete MLP with NumPy
3.6 Train MLP on MNIST with NumPy
3.7 Backward Propagation Check: Using Numerical Gradients to Verify Handwritten Backward
3.8 Reimplementing MLP with PyTorch nn.Module
Chapter 4: Optimization Algorithms: How Neural Networks Update Parameters
4.1 From Gradient Descent to SGD
4.2 Momentum and Nesterov Momentum
4.3 Adagrad: Starting Point of Adaptive Learning Rates
4.4 RMSprop and Adadelta: Fixing Learning Rate Decay
4.5 Adam: Combining Momentum and RMSprop
4.6 AdamW: Decoupled Weight Decay
4.7 Muon: Orthogonalized Updates for Matrix Parameters
4.8 Optimizer Map: When to Use Which Optimization Algorithm
4.9 Learning Rate Schedulers: Letting the Learning Rate Change During Training
Part 2: Convolutional Neural Networks
Chapter 5: Convolutional Neural Networks: From Local Perception to Global Modeling
5.1 From MLP to CNN: Why Images Need Convolution
5.2 Convolution Computation: Kernel, Padding, Stride, and Channels
5.3 Implement Conv2d from Scratch: From Sliding Windows to a PyTorch Module
5.4 Pooling and Downsampling: Max Pooling, Average Pooling, and Adaptive Pooling
5.5 Building a Simple CNN: From Feature Extraction to Image Classification
5.6 LeNet: The Early Template of Convolution, Pooling, and Fully Connected Layers
Chapter 7: Regularization and Normalization: Making Deep Networks More Stable
7.1 Why Deep Networks Need Regularization and Normalization
7.2 Dropout: Reducing Overfitting through Random Deactivation
7.3 BatchNorm: Stabilizing Training with Batch Statistics
7.4 LayerNorm: Normalizing Features Within Each Sample
7.5 InstanceNorm: Normalizing Each Channel Within Each Sample
7.6 GroupNorm: Normalizing Features Within Channel Groups
7.7 RMSNorm: Normalizing Feature Magnitudes Without Mean Centering
7.8 A Unified View of Normalization: Which Dimensions Are Normalized?
Part 4: Attention Mechanism and Transformer
Chapter 9: Attention and Transformer: From Dynamic Retrieval to Sequence Modeling
9.1 Bahdanau Attention: From Information Compression to Dynamic Retrieval
9.2 Cross-Attention: One Sequence Querying Another Sequence
9.3 Self-Attention: Internal Information Interaction Within a Sequence
9.4 Multi-Head Attention: From Single Perspective to Multiple Perspectives
9.5 Positional Encoding: Adding Positional Information to Attention
9.6 Transformer Encoder: Stacking Self-Attention Layers
9.7 Transformer Decoder: Masked Self-Attention and Cross-Attention
9.8 Encoder-Decoder Transformer: Connecting Encoder and Decoder
9.9 KV Cache: Why We Don’t Recompute the Past During Inference
9.10 Three Different Transformer Architectures: Understanding, Generation, and Input-Output Conversion
9.11 Hugging Face Transformers API: From Structure to Calls
Chapter 10: Efficient Attention Implementations: From Memory-Efficient Attention to FlashAttention
10.1 Why Attention Is IO-Bound
10.2 FlashAttention v1: Eliminating the IO Bottleneck in Attention Mechanisms
Part 5: Modern Computer Vision
Chapter 11: Vision Transformer: From Image Classification to Visual Sequence Modeling
11.1 From CNN to Vision Transformer: Treating Images as Sequences
11.2 Patch Embedding: Cutting Images into Tokens
11.3 Class Token and Positional Embedding: Letting a Sequence Represent the Whole Image
11.4 ViT Encoder: Letting Patch Tokens Exchange Information
11.5 ViT Backbone: Pretraining and Fine-Tuning
Part 6: Generative Models
Chapter 13: VAE: From Compression and Reconstruction to Probabilistic Generation
13.1 AutoEncoder: Starting with Compression and Reconstruction
13.2 VAE: Probabilistic Modeling and the Reparameterization Trick
13.3 ELBO: Where Does the VAE Objective Function Come From?
13.4 VAE Training Phenomena and Latent Space Intuition
13.5 VAE: Advantages, Limitations, and Future Developments
Chapter 14: Diffusion Models: From Denoising to Generation
14.1 DDPM: From Denoising to Generation
14.2 The Forward Process of DDPM: From Image to Noise
14.3 DDPM’s Reverse Denoising Process and Training Objective
14.4 DDPM Network Structure and Sampling Process
14.5 DDPM from a Variational Derivation: Where Does the ELBO Come From?
Part 8: LLM Fundamentals: Using GPT-2 as an Example
Chapter 18: Implementing GPT-2 from Scratch: Core Structures for Language Model Training
18.1 What Language Models Predict: Next-Token Prediction
18.2 MiniGPT: From a Causal GPT Block to a Decoder-only Language Model
18.3 Tokenizer: Characters, BPE, and Vocabulary
18.4 Embedding, LM Head, and Weight Tying
18.5 Training MiniGPT on TinyStories
18.6 From Training to Generation: Temperature, Top-k, and Top-p
18.7 GPT-2: From MiniGPT to a Pretrained Language Model
Chapter 19: LLM Training Engineering: Memory, Computation, and Parallel Training
19.1 Where Does the Memory Go When Training an LLM? Model States, Activations, and Runtime Overhead
Part 4: Attention Mechanism and Transformer
Author
jshn9515
Published
2026-05-05
Modified
2026-08-28
Order By
Default
Date - Oldest
Date - Newest
Title
Author
Title
Author
Date
10.1 Why Attention Is IO-Bound
jshn9515
2026-03-19
10.2 FlashAttention v1: Eliminating the IO Bottleneck in Attention Mechanisms
jshn9515
2026-03-19
9.1 Bahdanau Attention: From Information Compression to Dynamic Retrieval
jshn9515
2026-04-09
9.10 Three Different Transformer Architectures: Understanding, Generation, and Input-Output Conversion
jshn9515
2026-05-08
9.11 Hugging Face Transformers API: From Structure to Calls
jshn9515
2026-05-09
9.2 Cross-Attention: One Sequence Querying Another Sequence
jshn9515
2026-04-09
9.3 Self-Attention: Internal Information Interaction Within a Sequence
jshn9515
2026-04-09
9.4 Multi-Head Attention: From Single Perspective to Multiple Perspectives
jshn9515
2026-04-09
9.5 Positional Encoding: Adding Positional Information to Attention
jshn9515
2026-04-09
9.6 Transformer Encoder: Stacking Self-Attention Layers
jshn9515
2026-05-03
9.7 Transformer Decoder: Masked Self-Attention and Cross-Attention
jshn9515
2026-05-05
9.8 Encoder-Decoder Transformer: Connecting Encoder and Decoder
jshn9515
2026-05-05
9.9 KV Cache: Why We Don’t Recompute the Past During Inference
jshn9515
2026-05-05
No matching items
Reuse
CC BY-NC 4.0