☆ Save 3.2 Adaptive Optimization and Learning Rate Scheduling — How to Reduce Gradient Noise and Let Training Speed Self-Tune

02/26/2026

This post connects adaptive optimization and learning rate scheduling into one practical storyline: how gradient-based training stays stable in real deep learning. Starting from basic gradient descent, we’ll see how modern methods reduce gradient noise while automatically adjusting effective training speed so convergence is both faster and more reliable.

Table of Contents

• Gradient Descent and Gradient Noise
• Moving Averages and Momentum
• Adaptive Learning Rate Optimizers
• A Structural View of Adam
• Learning Rate Scheduling

Gradient Descent and Gradient Noise

Gradient Descent is an optimization algorithm that repeatedly updates parameters to minimize a loss function. At each step, it computes a gradient estimate and moves in the opposite direction.

• [ theta leftarrow theta – epsilon hat{g} ]

  Basic Gradient Descent update (ε: learning rate, ĝ: gradient estimate)

In deep learning, gradients are usually estimated from a mini-batch rather than the full dataset. The step-to-step variability introduced by this sampling is often called gradient noise. SGD is computationally efficient, but the noisy gradients can make the optimization path jittery and less stable.

So the goal of modern optimization is straightforward:

• Reduce noise in the gradient estimate so updates don’t wobble unnecessarily.
• Preserve fast progress so training still converges quickly in practice.

Moving Averages and Momentum

A very intuitive way to reduce gradient noise is to average gradients over multiple steps. You can view this as a linear filter over time: it smooths out short-term spikes and keeps the overall direction more consistent.

A standard choice is the Exponentially Weighted Moving Average (EWMA).

• [ v_t = alpha v_{t-1} + (1-alpha)hat{g}_t ]

  EWMA: recent gradients get larger weight

For example, if α = 0.9, the current gradient is 10, and the previous average is 6:

• vt = 0.9 × 6 + 0.1 × 10 = 6.4

The update direction becomes less sensitive to abrupt changes. This idea leads directly to Momentum, which uses the smoothed gradient to create an “inertial” update that keeps moving in consistent directions.

• Momentum

  • Uses a moving average of gradients to stabilize direction.
  • Helps traverse shallow valleys and reduces zig-zagging.

• Nesterov Accelerated Gradient (NAG)

  • A momentum variant that “looks ahead” to refine the next step.
  • Often improves responsiveness without losing stability.

Adaptive Learning Rate Optimizers

If momentum stabilizes the direction, the next practical step is to adapt the step size per parameter. This is the motivation behind adaptive learning rates.

In deep networks, some parameters get frequent, large gradients while others change rarely. A single global learning rate can be inefficient: it may be too small for slow-moving parameters, and too aggressive for sensitive ones.

• AdaGrad

  • Accumulates squared gradients and automatically shrinks the learning rate over time.
  • Works well when features are sparse, but can decay too aggressively later.

• [ theta leftarrow theta – frac{epsilon}{sqrt{sum g^2}} odot hat{g} ]

  ⊙ : element-wise product (independent per-parameter scaling)

• RMSProp

  • Fixes AdaGrad’s overly fast decay by using an EWMA of squared gradients.
  • Keeps step sizes responsive to recent gradient statistics.

• [ theta leftarrow theta – frac{epsilon}{sqrt{text{EWMA}(g^2)}} odot hat{g} ]

  Scaling based on recent second-moment behavior

A Structural View of Adam

Adam (Adaptive Moment Estimation) combines the core benefits of Momentum and RMSProp. In practice, it tracks both:

• First moment (mean of gradients) for stable direction
• Second moment (scale / variance proxy) for adaptive step sizes

• [ theta leftarrow theta – frac{epsilon}{sqrt{text{EWMA}(g^2)}} odot text{EWMA}(g) ]

  Uses both first-moment smoothing and second-moment scaling

This structure is why Adam often feels “automatic” in early training: it moves quickly when gradients are informative, then becomes more conservative as updates need to be precise. That balance is a major reason it’s a default optimizer in many modern deep learning pipelines.

Learning Rate Scheduling

The learning rate is one of the most important hyperparameters in gradient-based optimization. A common strategy is:

• Early training: take larger steps to explore and make fast progress.
• Late training: take smaller steps to refine and settle near a good minimum.

Learning rate scheduling implements this idea by gradually decaying the learning rate over time. In many loss landscapes, this improves stability and helps the model converge more cleanly near minima.

In practice, SGD + Momentum tends to be very sensitive to how you decay the learning rate, while Adam is often less sensitive—but scheduling still matters when you care about final accuracy and robustness.

※ This article is an independently organized and restructured summary based on lectures by Professor Sungroh Yoon at Seoul National University.

Recommended prerequisite reading (3/5)

+2

• Learning Rate Scheduling — A Strategy to Stabilize Optimization by Adjusting Update Size Across Training Phases
• Gradient Descent and Stochastic Optimization — Scalable learning by replacing exact gradients with efficient estimates
• Moving Average and Adaptive Learning Rate — Smoothing Noisy Gradients and Adjusting Update Speeds Across Parameters

• Supervised Loss — Why Does Learning from Ground-Truth Labels Work?
• Input Normalization — A preprocessing method that aligns input scales to stabilize learning

Recommended next reading (5/18)

+5

• Learning Rate Warmup — How LLMs Stabilize the Start of Training
• 4.4 Neural Network Optimization Challenges and Architecture Design Principles — Vanishing Gradients and Architectural Design Principles
• 4.3 Fundamentals of Learning Signals and Backpropagation — Forward Pass, Error Propagation, and Gradient-Based Learning Principles
• 3.1 Optimization in Machine Learning — Learning Parameters and Choosing What Really Works
• Exponential Decay — A Scheduling Method That Reduces the Learning Rate Exponentially as Training Progresses

• Gradient Variance — Why Update Directions Wobble in Stochastic Training
• Gradient Estimate — Approximating the Full-Data Gradient to Choose a Learning Direction
• Gradient Scaling Problem — When Gradient Magnitudes Break Down in Deep Neural Networks
• Local Gradient — Layer-wise Sensitivity Signals Passed During Backpropagation
• Dynamic Curriculum Learning — Why It Differs from a Fixed Curriculum and How Adaptation During Training Changes Performance

• Adversarial Loss — How GANs Learn to Generate Realistic Data,
• Residual Scaling — How Deep Transformers Stay Stable During Training
• Ordered Boosting — Why Does Standard Gradient Boosting Cause Prediction Shift?
• Backpropagation — The Core Idea of Learning Weights by Sending Error Backward
• Delta Rule — A Basic Learning Rule That Adjusts Weights in Proportion to the Error
• Anti-Curriculum Learning — When Starting with Hard Data Can Improve Performance
• Curriculum Design Strategy — Why Does Data Order Affect Generalization?
• Pacing Function — Why the Timing of Difficulty Expansion Affects Training Stability

Posts on the same topic (0/0)

No other posts in this section yet.

Related concepts (0/0)

No related concept posts yet.

📍 Where this concept fits in the AI learning map

See where this concept sits within the full AI Universe.

📍 Current position in AI Universe

☰

Reset Show completed · Login required Loading…

🌌 AI Universe

‹

›

⭐ Concept

Select a star.

View the full AI Universe

« 3.1 Optimization in Mach…|3.3 Regularization in Ma… »

🔖 Tags: Adam · Adam Optimizer · adaptive optimizers · Gradient Descent · Learning Rate Scheduling · sgd