☆ Save 8.8 Deep Generative Models — Major Approaches to Probability Distribution Modeling

09/13/2026

Deep Generative Models are not simply systems for creating new images·audio·or other synthetic content. Their core objective is to learn the probability distribution that gives rise to real data. Suppose we collect tens of thousands of dog images. The dogs may differ in fur color, ear shape·pose·and background, yet an arbitrary arrangement of pixels will almost never look like a valid dog. Real dog images occupy only a small, highly structured region of the enormous space of all possible images. A generative model tries to capture that structure as a probability distribution.

The difficulty is that distributions over high-dimensional data such as images can be extremely complex. Rather than using a single universal strategy, different generative model families make the problem tractable in different ways. An Autoregressive Model decomposes a complicated distribution into a sequence of conditional probabilities. A Variational Autoencoder (VAE) explains observed data through latent variables. A Flow-based Model transforms a simple distribution through a sequence of invertible mappings, while a GAN avoids evaluating probability density directly and instead trains two neural networks adversarially to produce realistic samples. This chapter develops these ideas by focusing on why such different modeling strategies are needed in the first place.

Families of Deep Generative Models
From Likelihood to NLL
Different Structures for Modeling Distributions
Normalizing Flow and Invertible Transformations
Key Differences Between Modeling Strategies

Families of Deep Generative Models

Once we define the goal of a generative model as learning a data distribution, an immediate question follows: how can such a complicated distribution be represented in a form that is actually computationally manageable? Different answers to this question lead to different families of Deep Generative Models.

At a high level, these models can be grouped into Likelihood-based Models and Likelihood-Free Models. Likelihood measures how plausible a particular observation is under the model. For a model trained on dog images, the learning objective should assign greater plausibility to images that resemble real dogs and much lower plausibility to unrelated random images.

Likelihood-based Models: Autoregressive Models, VAEs, Flow-based Models, Diffusion Models, and Score-based Models.
Likelihood-Free Models: the most representative example is the Generative Adversarial Network (GAN).

Being likelihood-based does not mean that every model computes Likelihood in the same way. Some factorize a complex distribution into conditional probabilities, some introduce latent variables, and others transform a simple base distribution into a more expressive one. What they share is the same broader goal: turning a difficult high-dimensional distribution into a structure that can be learned effectively.

Diffusion Models and Score-based Models also belong to this broader family, but for now the important point is simply where they fit within the landscape of generative modeling. Their detailed mechanisms—such as progressively adding and removing noise or estimating the Score Function—are easier to understand when treated as separate topics.

From Likelihood to NLL

To understand Likelihood-based learning, let pdata denote the distribution that generates the real data and pmodel the distribution learned by the model. If training is successful, the model distribution should approximate the data distribution as closely as possible.

Suppose most real dog images contain plausible facial features, bodies, and legs. A well-trained model should assign high probability to regions containing images with those characteristics. If it instead assigns high probability to arbitrary arrangements of random pixels, it has failed to capture the real data distribution. One standard way to quantify the difference between the data distribution and the model distribution is KL Divergence.

$$ \min_{\theta} D_{\mathrm{KL}}\left[p_{\mathrm{data}}(x)\,\|\,p_{\mathrm{model}}(x;\theta)\right] $$

The parameters θ are optimized so that the model distribution becomes as close as possible to the real data distribution.

The intuition behind this objective is straightforward: the model should place more probability mass where real data is common and less where it is not. In practice·however·training does not require optimizing the entire KL Divergence expression directly. Expanding the objective reveals which part actually depends on the model parameters and can therefore be optimized.

$$ D_{\mathrm{KL}}\left[p_{\mathrm{data}}\,\|\,p_{\mathrm{model}}\right] = \mathbb{E}_{x\sim p_{\mathrm{data}}}\left[\log p_{\mathrm{data}}(x)\right] – \mathbb{E}_{x\sim p_{\mathrm{data}}}\left[\log p_{\mathrm{model}}(x;\theta)\right] $$

The first term is determined entirely by the real data distribution, while only the second term changes with the model parameters θ.

The first term corresponds to the negative Entropy of the data distribution. Because the training data distribution is fixed, changing θ cannot affect this term. The only part the model can optimize is therefore the second term. Training pushes the model to assign higher probability to the observed data, which leads directly to the Maximum Likelihood objective.

$$ \min_{\theta}\mathbb{E}_{x\sim p_{\mathrm{data}}}\left[-\log p_{\mathrm{model}}(x;\theta)\right] $$

Training minimizes the negative log-probability that the model assigns to samples drawn from the real data distribution.

Suppose the training set contains a particular dog image A. If the model initially assigns very low probability to that image, its negative log-probability will be large. As training progresses and the model assigns higher probability to real examples of the same kind, that value decreases. In this sense, minimizing Negative Log-Likelihood (NLL) encourages the model to treat observed training examples as increasingly plausible under its learned distribution.

The overall argument can be summarized as minimizing KL Divergence → dropping the Entropy term that does not depend on θ → minimizing Cross-Entropy → minimizing NLL on the training data. However, NLL cannot always be evaluated directly in the same way for every model. When latent variables are introduced, as in a VAE, the Likelihood may become difficult to compute exactly. In that case, methods such as Variational Inference and the ELBO provide a tractable alternative objective.

This leaves the central modeling question: how can a complicated p(x) be represented in a form that we can actually compute and optimize? One strategy is to factorize the distribution into a sequence of conditional probabilities. Another is to start with a simple distribution and transform it repeatedly into a more complex one. These two ideas lead naturally to Autoregressive Models and Flow-based Models.

Different Structures for Modeling Distributions

Different generative model families solve the difficulty of distribution modeling in different ways. An Autoregressive Model breaks one large probability problem into a sequence of smaller conditional probability problems. A VAE introduces latent variables intended to capture hidden factors behind the observed data. A GAN takes another route entirely, using competition between generated and real data as the learning signal rather than directly evaluating probability density.

Consider the Autoregressive Model first. If an image is represented as a large collection of variables, modeling the full joint probability of all pixel values at once is difficult. Instead, the distribution can be decomposed sequentially: model the first value, then the probability of the second given the first, then the probability of the next value given everything that came before it, and continue in the same way.

$$ p(x)=\prod_{i=1}^{n}p(x_i\mid x_1,\ldots,x_{i-1}) $$

The full joint distribution is factorized into a product of conditional probabilities, with each variable conditioned on the variables that precede it.

The factorization is easier to see with just three variables. We first model the probability of x1. Next, we model the probability of x2 given x1, and then the probability of x3 given both x1 and x2. Multiplying these terms gives the joint probability of the complete observation.

Importantly, this decomposition is not an arbitrary approximation introduced merely to simplify computation. It is an exact factorization given by the Probability Chain Rule. An Autoregressive Model therefore turns the question “How can we model a complicated p(x) all at once?” into the more manageable problem of accurately modeling each conditional distribution. GPT, RNN, PixelRNN, PixelCNN, WaveNet, NADE, and MADE differ substantially in architecture, but they all make use of this basic conditional factorization principle.

A Variational Autoencoder (VAE) starts from a different idea. Rather than modeling every detail of the observed data directly, it assumes that the observations can be explained through simpler latent variables z. For a dataset of face images, for example, individual latent variables do not necessarily correspond to clearly interpretable concepts such as eye shape or pose. Still, the latent representation can be viewed intuitively as capturing hidden factors that help explain how different observations are generated.

$$ p(x)=\int p(z)p(x\mid z)\,dz $$

The marginal probability of the observed data x is obtained by integrating over all possible values of the latent variable z.

Here, p(z) defines the distribution over the latent variable, while p(x | z) describes how likely the observation x is given a particular latent state. Because the model must account for every possible z rather than a single latent value, the expression requires integration over the latent space. In other words, a VAE models the distribution of complex observations through a latent space.

This formulation is implemented with two connected neural networks. The Recognition Network, or Encoder, takes an observation x and produces an approximate distribution over the latent variable z. The Generative Network, or Decoder, models how an observation x can be generated from a given z. For an image, the Encoder maps the input to a distribution over a region of latent space, a latent value z is obtained from that distribution, and the Decoder uses z to generate data in the observation space.

The challenge is that the exact Posterior over z given x can be difficult to compute. Rather than evaluating that Posterior directly, a VAE uses Variational Inference to approximate it. The Encoder and Decoder are then trained together by maximizing the ELBO, which connects the approximate inference procedure to the data Likelihood. The key idea behind a VAE is therefore not simply compression followed by reconstruction, but learning a complex generative distribution through latent variables and approximate inference.

A Generative Adversarial Network (GAN) takes yet another approach. Instead of directly answering the question “What probability does the model assign to this particular image x?”, a GAN learns to generate samples that are difficult to distinguish from real data.

This setup introduces two networks: the Generator and the Discriminator. Suppose the Generator initially produces unrealistic-looking face images. The Discriminator compares generated samples with real faces and can easily identify the fakes. The Generator then learns to produce more realistic images that are harder for the Discriminator to detect. As the Generator improves, the Discriminator must also become better at recognizing increasingly subtle differences between real and generated data.

This competition between two models with opposing objectives forms a Minimax Problem. Explicitly evaluating p(x) is not the central mechanism here. Instead, the Generator uses the learning signal provided by the Discriminator to make its generated distribution increasingly resemble the real data distribution. This is the key reason GANs are classified as Likelihood-Free Models.

Normalizing Flow and Invertible Transformations

While an Autoregressive Model handles a complex distribution by factorizing it into conditional distributions, a Flow-based Model takes a different approach: it repeatedly transforms a simple distribution into a more complex one. The process begins with an easy-to-handle base distribution such as a Gaussian. By stretching·bending·and reshaping that distribution through a sequence of transformations, the model constructs a distribution capable of representing complex real data.

A Normalizing Flow (NF) uses a sequence of Invertible Transformations rather than relying on a single highly complicated mapping. It starts with a simple random variable. After the first transformation, its distribution becomes more expressive. A second transformation reshapes it further, and repeated transformations gradually produce a distribution that can approximate the target data distribution.

Simple Distribution → Invertible Transformation → Invertible Transformation → Target Distribution

The critical requirement is that every transformation must be invertible. If a simple latent value z is transformed through several mappings into a data point x, it must also be possible to start from x and trace the transformations backward to recover z. This makes it possible to identify where a point in data space originated in the base distribution and to track how its probability density changes through each transformation.

This requirement means that a Flow-based Model cannot use arbitrary neural network architectures for its transformations. Each mapping must be designed to preserve invertibility. In return, the model gains an important advantage: it can compute an explicit probability density for the final data point x. This allows the model to evaluate the Likelihood of observed data and train directly by minimizing NLL.

It is important to distinguish modeling the data distribution from reproducing the true data distribution exactly. A Flow-based Model learns a model distribution for which p(x) can be evaluated explicitly, then adjusts that distribution to assign high Likelihood to observed data. The learned distribution is therefore an approximation to the real data distribution rather than an exact copy of it.

Key Differences Between Modeling Strategies

We can now see why these model families use such different architectures. They all pursue the same fundamental goal—learning a complex, high-dimensional data distribution—but each turns that difficult distribution-modeling problem into a more tractable learning problem in a different way.

Autoregressive Model: factorizes a complex p(x) into a product of conditional distributions.
VAE: introduces a latent variable z and learns p(x) through a latent space using Variational Inference and the ELBO.
Flow-based Model: applies a sequence of invertible transformations to a simple distribution, producing an explicit probability density over the data and enabling direct NLL-based training.
GAN: does not evaluate p(x) directly. Instead, it uses adversarial training between a Generator and a Discriminator to learn a distribution that produces samples resembling real data.

The fundamental differences between Deep Generative Models therefore go beyond the particular neural network architectures they use. What matters more is how each model represents a complex probability distribution and turns that representation into a tractable learning problem. From this perspective, the conditional factorization of Autoregressive Models, the latent variables of VAEs, the invertible transformations of Normalizing Flows, and the adversarial training of GANs can all be understood as different solutions to the same underlying problem.

※ This article is an independently organized and restructured summary based on lectures by Professor Sungroh Yoon at Seoul National University.

📍 Where this concept fits in the AI learning map

See where this concept sits within the full AI Universe.

📍 Current position in AI Universe

☰

Reset Show completed · Login required Loading…

🌌 AI Universe

‹

›

⭐ Concept

Select a star.

View the full AI Universe

« 8.7 Normalizing Constant…|8.9 Diffusion Models — G… »

🔖 Tags: autoregressive models · Deep Generative Models · Diffusion Models · Generative Adversarial Network · Likelihood-based Models · normalizing flows · variational autoencoder