Learning Rate Warmup — How LLMs Stabilize the Start of Training

Learning Rate Warmup is a training strategy that begins with a small learning rate and gradually increases it to a target value over a fixed number of steps. At the very start of training, the model’s gradients, activations, and optimizer state have not yet settled into a stable regime. Using a large learning rate immediately can make a single weight update too aggressive, leading to a loss spike or unstable training. Warmup reduces that risk by keeping early updates small, then allowing larger learning rates as training becomes more stable.

08/29/2026

Loss Spike — Why Loss Can Suddenly Surge During Training

A Loss Spike is a sudden, sharp increase in the loss function at a particular step during deep learning training. Rather than being ordinary noise in the training log, it can signal training instability associated with a gradient norm spike, an excessively high learning rate, propagating NaN or Inf values, failed loss scaling, or an abnormal data batch. This is especially important in LLM training, where even a relatively small loss spike can force teams to discard a checkpoint, roll back training, or restart a distributed training run. For that reason, loss spikes are treated as an important operational signal when monitoring large-scale training.

08/29/2026

Output Embedding Shifted Right — Why Shift and Causal Masking Solve Different Problems

Output Embedding Shifted Right is a training structure used in Transformer Decoders where the target sequence is shifted one position to the right before being fed into the model. This prevents the Decoder from directly seeing the current correct token and forces it to predict the next token using only previous tokens, enabling true autoregressive learning.

05/24/2026

Post-LN in Transformers — How LayerNorm Placement Affects Gradient Flow and Training Stability

Post-Layer Normalization (Post-LN) is a Transformer architecture where Layer Normalization is applied after the residual connection. To understand why this matters, we first need to understand what LayerNorm does and why normalization is necessary in Transformer models. Post-LN is not just a small ordering choice. It directly changes activation scale, gradient flow, and training stability in deep Transformer stacks.

05/25/2026

Pre-Layer Normalization (Pre-LN) — How LayerNorm Before Attention Stabilizes Deep Transformers

Pre-Layer Normalization (Pre-LN) is a pre-normalization scheme in Transformers where LayerNorm is applied before Self-Attention or the Feed-Forward Network (FFN). This placement matters because deep Transformers repeatedly stack many blocks, and as depth increases, activation distributions and gradient flow can become unstable. LayerNorm keeps token representations well-scaled, while Pre-LN improves optimization stability by placing normalization before each sub-layer.

05/25/2026