☆ Save Reverse Distribution — Estimating the Previous State in Diffusion Models

09/13/2026

A reverse distribution describes the possible previous states xt-1 given the current state xt during the reverse diffusion process. A diffusion model does not turn random noise xT into a finished sample x0 in a single jump. Instead, it moves backward one step at a time. For each xt, the model constructs a probability distribution over plausible, slightly less noisy states xt-1 and then samples the state used in the next reverse step. The reverse distribution therefore captures both where the previous state may lie and how likely different regions of the state space are.

In simple terms: Recovering a clean image from a heavily corrupted one in a single prediction would be difficult. Diffusion models break that problem into smaller steps: given xt, what could the slightly less noisy xt-1 look like? During training, the neural network learns to predict the noise contained in xt. During generation, that prediction is used to parameterize the reverse distribution for xt-1. The model samples from this distribution and repeats the process, gradually turning noise into structured data.

Using noise prediction to construct a reverse distribution and sample the previous state

How It Works

• Forward Diffusion Process and Reverse Diffusion Process

  • The purpose of the reverse distribution is easiest to see by first looking at forward diffusion. Starting from the original data x0, the forward process adds noise gradually, making the sample less structured and more uncertain at each time step.
  • As noise accumulates, the original data distribution moves progressively toward a noise distribution. By the final state xT, the sample is close to random noise. Forward diffusion therefore defines a path from structured data to noise.
  • Generation follows that path in reverse. Given the current state xt, the reverse diffusion process estimates a slightly less noisy state xt-1. Repeating this one-step transition moves the sample from xT toward x0.
  • The previous state xt-1 is not usually a single predetermined value. Multiple states may be plausible for the same xt, so the reverse process represents those possibilities with a probability distribution.

• Training — Learning Noise Prediction

  • Training does not mean computing and storing a separate reverse distribution for every time step. Instead, the model learns neural network parameters θ that provide the information needed to construct the appropriate distribution for any given xt.
  • To train this predictor, known noise is added to original samples to create noisy states xt at different time steps. Because the training procedure knows exactly which noise was added, the network can compare its prediction with the actual noise and learn the noise prediction task directly.
  • After training, the network takes xt and the time step t as input and predicts the noise in the current state. In the standard DDPM noise-prediction formulation, this estimate is used to compute μθ, the mean of the reverse transition distribution for xt-1.
  • Training therefore does not produce a fixed collection of reverse distributions. It learns parameters θ that allow the model to construct the required distribution dynamically from xt and t.

• Generation — Constructing the Reverse Distribution

  • During generation, the current state xt and time step t are passed through the trained neural network. The network predicts εθ(xt, t), and the model uses that estimate to compute μθ(xt, t), the mean of the reverse transition distribution for xt-1.

  • $$ \mu_\theta(x_t,t)=\frac{1}{\sqrt{\alpha_t}}\left(x_t-\frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\epsilon_\theta(x_t,t)\right) $$

    The current state xt and predicted noise εθ determine the mean μθ of the reverse distribution. Here, βt controls the noise level at the current step, αt=1-βt, and ᾱt is the cumulative product of α values up to time step t.

  • The mean specifies the center of the transition, but a probability distribution also needs to describe variation around that center. In a standard DDPM, the one-step reverse transition is modeled as a Gaussian distribution.

  • $$ p_\theta(x_{t-1}\mid x_t)=\mathcal{N}\left(x_{t-1};\mu_\theta(x_t,t),\sigma_t^2 I\right), \qquad x_{t-1}=\mu_\theta(x_t,t)+\sigma_t z,\;\; z\sim\mathcal{N}(0,I) $$

    The Gaussian distribution is centered at μθ, while σt controls the amount of stochastic variation. To obtain an actual xt-1, the model samples standard Gaussian noise z and scales it by σt before adding it to μθ.

  • Denoising is therefore more than simply subtracting predicted noise from an image. It is a probabilistic estimation pipeline: current xt and t → noise prediction → compute μθ → construct the reverse distribution.

• Reverse Distribution and Sampling

  • Constructing the reverse distribution tells the model where xt-1 is likely to lie, but it does not yet determine the actual state used for the current step. Generation continues by sampling a specific xt-1 from that probability distribution.
  • For a simplified example, suppose candidate A has probability 0.8 and candidate B has probability 0.2. This means A is considered the more plausible previous state given xt. It does not mean that the model removes 80% of the noise.
  • Real image generation is not a choice between two discrete candidates. Images occupy a continuous, high-dimensional state space, and the reverse distribution is defined over that space. The A-versus-B example is only a simple way to illustrate what the probabilities represent.
  • The reverse distribution answers, “Where is the previous state likely to be?” Sampling answers, “Which xt-1 will be used for this reverse step?” Together, constructing the distribution and sampling from it define a single reverse transition.

• Repeating the Process from xT to x0

  • Generation starts from random noise xT. The model feeds xT and time step T into the trained network, constructs the corresponding reverse distribution, and samples xT-1 from it.

  • $$ p_\theta(x_{T-1}\mid x_T),\;p_\theta(x_{T-2}\mid x_{T-1}),\;\ldots,\;p_\theta(x_{t-1}\mid x_t),\;\ldots,\;p_\theta(x_0\mid x_1) $$

    At each time step, the model constructs the reverse distribution for the current state and samples the state used in the next reverse transition.

  • The sampled xT-1 becomes the current state for the next iteration, where the model constructs a new reverse distribution for xT-2. The same procedure continues until the final transition from x1 to x0.

  • This does not require a separate neural network for every time step. The same trained network is evaluated repeatedly with different xt and t inputs, producing the reverse distribution needed at each stage of generation.

• Generation Quality and Computational Cost

  • Because generation consists of a chain of reverse transitions, the quality of each estimated reverse distribution affects the final sample. The more accurately the model estimates a suitable distribution for xt-1 from xt, the more reliably the trajectory can move toward states consistent with the learned data distribution.
  • If one transition is estimated poorly, the sampled state may drift away from that trajectory. Since this state becomes the input to the next step, the effect can propagate through later reverse transitions.
  • The iterative process also carries a computational cost. Rather than moving directly from xT to x0, the model repeatedly predicts noise, constructs a reverse distribution, and samples a new state across many time steps.
  • In practice, generation therefore depends not only on the accuracy of each distribution estimate, but also on the speed and computational cost of repeated sampling.

Significance and Limitations

The key idea behind the reverse distribution is that a diffusion model does not have to assign a single deterministic xt-1 to each xt. Instead, it can represent multiple plausible previous states with a probability distribution. During training, the neural network learns parameters θ that allow it to predict the noise in the current state. During generation, that prediction is used to compute μθ and parameterize the reverse distribution for xt. Sampling xt-1 from that distribution at each step allows the model to gradually transform random noise into new samples that follow the learned data distribution.

The main limitation is that the true reverse distribution is not known exactly, so every reverse transition relies on an approximation produced by the trained neural network. If the model estimates the distribution of xt-1 from xt poorly, the resulting error can influence later steps and reduce the quality of the final sample. Inference is also computationally expensive because the model repeatedly constructs reverse distributions and samples from them across many time steps. Generation speed and computational cost are therefore important practical constraints for diffusion models.

Recommended prerequisite reading (3/5)

+2

• 8. Generative Modeling — Core Principles of Generative Models and Deep Generative Models
• Reverse Diffusion Process — How Diffusion Models Generate Data from Noise
• Generative AI — A Model That Learns Data Distributions to Create New Samples

• Score Matching — Learning Data Distributions Through the Score Function
• Invertible Transformations — How Normalizing Flows Preserve Information

Recommended next reading (5/15)

+5

• Noise-Conditioned Score Network — Why Diffusion Models Learn the Score Function
• Noise Scale — How Noise Levels Shape Denoising Quality in Diffusion Models
• Text-to-Image Models — Multimodal Generative Models That Synthesize Visual Images from Textual Meaning
• Score-based Generative Model — Learning the Generative Direction with the Score Function
• PixelRNN — How Autoregressive Models Generate Images One Pixel at a Time

• Stochastic Encoder — Learning Latent Representations Through Probability Distributions
• Restricted Boltzmann Machine (RBM) — An Efficient Generative Model That Learns a Probability Distribution with Two Connected Layers
• Implicit Density — Representing a Distribution Without Explicitly Computing Its Density
• Identity Preservation — Keeping the Same Identity Consistent Throughout the Generation Process
• 7.8 Advanced Positional Embeddings — APE·RPE·and RoPE in Transformer Models

• Normalizing Flow — A Generative Model That Learns Probability Density Through Invertible Transformations
• Score Function — Understanding the Gradient of Log Probability Density
• Noise Corruption — How Denoising and Self-Supervised Learning Learn from Noisy Inputs
• 7.3 Self-Attention Mechanism — From Query–Key–Value to Matrix Computation
• Music Generation — A Generative Approach That Builds a Full Piece by Predicting and Extending the Next Sound Over Time

Posts on the same topic (2/2)

• Text-to-Video Generation — Video Generation Conditioned on Text Prompts
• Latent Video Representation — Compressing Spatial and Temporal Information in Video

Related concepts (3/3)

• Diffusion Model — How Denoising Turns Random Noise into New Data
• Forward Diffusion Process — Why Does a Diffusion Model Add Noise?
• Denoising Diffusion Probabilistic Model (DDPM) — Generating Data by Predicting Noise

📍 Where this concept fits in the AI learning map

See where this concept sits within the full AI Universe.

📍 Current position in AI Universe

☰

Reset Show completed · Login required Loading…

🌌 AI Universe

‹

›

⭐ Concept

Select a star.

View the full AI Universe

« NADE (Neural Autoregress…|Entropy — A criterion th… »

🔖 Tags: diffusion-model · Generative Model · Noise Prediction · Probability Distribution · Reverse Diffusion · Reverse Distribution · sampling