☆ Save 8.6 Data Distribution Modeling — How Generative Models Learn Data Distributions

09/13/2026

In Generative Modeling, the goal is not to memorize individual training examples. What matters is learning the underlying rules and structure that make those examples possible. Consider a dataset containing tens of thousands of dog photos. Each image may differ in fur color·pose·viewing angle·lighting·or background, yet most of them are immediately recognizable as dogs. That consistency suggests that the images are not arbitrary points in the data space; they share meaningful structure. A Generative Model attempts to capture that structure through a probability distribution.

The challenge is that the true Data Distribution is never directly available to us. We only observe a finite collection of samples drawn from it. Generative Modeling therefore starts with observed data and learns a Model Distribution that approximates the properties of the underlying distribution. Once that distribution is modeled well, the model can generate new samples, estimate how plausible a given observation is, and identify patterns shared across the data. This chapter develops that perspective and then examines why modeling high-dimensional data requires increasingly expressive models.

From Observed Data to a Data Distribution
Data Distribution and Model Distribution
Three Capabilities of Distribution Modeling
Why High-Dimensional Data Distributions Are Difficult
Tractability-Flexibility Tradeoff
Why We Need Deep Generative Models

From Observed Data to a Data Distribution

A useful first step is to distinguish between observed data and the Data Distribution that produced it. If a training set contains 10,000 dog images, those 10,000 images are all we can directly observe. But they represent only a tiny fraction of all plausible dog images. A dog looking sideways, running through a field, photographed at night, or covered in white fur can still form a perfectly valid image even if that exact example never appears in the training set.

The training set can therefore be viewed as a finite sample from a much larger space of possible observations. A Data Distribution provides a probabilistic description of that broader space. In standard statistical modeling, we typically assume that the training examples are i.i.d. samples drawn independently from the same unknown Data Distribution.

$$ \{x_1,x_2,\ldots,x_N\} \overset{\mathrm{i.i.d.}}{\sim} p_{\mathrm{data}}(x) $$

Each training example is assumed to be drawn independently from the same unknown true data distribution.

This does not mean that we know the exact mathematical form of the true Data Distribution. What we actually have is a finite set of observations, such as 10,000 dog images. The probability distribution that generated those observations remains hidden.

For a coin toss, the underlying distribution is simple enough to describe in terms of the probabilities of heads and tails. Natural images are fundamentally different. A single image may contain millions of interacting pixel values, and there is no simple equation that tells us exactly which combinations will produce a realistic dog. This is why Generative Modeling is fundamentally about inferring an underlying distribution from observed samples.

This also explains why large datasets are useful. The goal is not to memorize every image. Instead, the model looks for statistical regularities that appear across many examples: recurring shapes·textures·spatial relationships·poses·and other structures that characterize plausible dog images. A Model Distribution gives us a learnable representation of those regularities.

Data Distribution and Model Distribution

The Data Distribution represents the unknown process assumed to generate real observations. Because its exact form is unavailable, we introduce a Model Distribution with learnable parameters. Training adjusts those parameters using observed samples so that the model captures as much of the structure of the underlying Data Distribution as possible.

Data Distribution: The unknown distribution assumed to generate real-world data.
Model Distribution: A learnable distribution fitted to observed samples in order to approximate the Data Distribution.

Consider the dog-image example again. The true Data Distribution includes many possible images with different probabilities: dogs facing forward or sideways, standing in bright sunlight or appearing in dark environments, and countless other variations. We cannot enumerate or directly inspect this entire distribution. Instead, we train a Model Distribution from the dog images we have collected. A successful model should do more than reproduce its training examples; it should capture the broader region of the data space that corresponds to plausible dog images.

$$ p_{\mathrm{model}}(x;\theta) \approx p_{\mathrm{data}}(x) $$

The goal is for the learned Model Distribution to approximate the structure of the unknown true Data Distribution.

The parameters $\theta$ determine the shape of the Model Distribution. Before training, they do not yet encode the structure of the data. As the model processes more samples, these parameters are updated so that regions where real data frequently occurs are represented more accurately.

The learning process can therefore be summarized as observed i.i.d. samples → learning the parameters of a Model Distribution → approximating the unknown Data Distribution. We do not discover the true distribution first and then build a model around it. Instead, the model uses the samples we can observe to infer the structure of a distribution that remains hidden.

Once a Model Distribution has been learned, it supports two fundamental operations. Probability Evaluation asks how plausible a given observation is under the learned distribution, while Sampling draws new observations from that distribution. In other words, we can either ask, “How well does this example fit what the model has learned?” or, “What new example could this distribution produce?”

Three Capabilities of Distribution Modeling

A well-learned Data Distribution can support several useful capabilities within the same modeling framework. Rather than treating them as unrelated tasks, it is more useful to think of them as different ways of using the same learned distribution.

The first is Generation. Here, the model draws a new sample from its learned Model Distribution. For a model trained on dog images, the result should be a new dog image rather than a simple copy of an example from the training set.

$$ x_{\mathrm{new}} \sim p_{\mathrm{model}}(x;\theta) $$

A new data sample is drawn from the learned Model Distribution.

Suppose the training set contains black·white·and brown dogs photographed in many different poses. If the model has captured the distribution well, it should be able to combine the learned patterns into plausible configurations that were never present as exact training examples. Producing a new dog with a different pose or appearance is what it means to generate a sample from the learned distribution.

The second capability is Density Estimation. Instead of generating a new example, the model evaluates how plausible an existing observation is under the learned distribution. A useful dog-image model, for example, might assign relatively high probability to realistic dog images and very low probability to images made of randomly arranged pixels. This indicates that the model has learned something about where plausible dog images lie in the data space.

This perspective also connects naturally to Anomaly Detection. Consider a model trained on sensor measurements collected during normal operation of a manufacturing process. Familiar sensor patterns may receive high likelihood under the learned distribution, while unusual patterns that differ substantially from normal operation may receive low likelihood. That difference can provide a useful signal for identifying anomalous states.

The third capability is Unsupervised Representation Learning. To explain many observations with a shared model, the model must capture recurring structure rather than treating every value independently. Across dog images, features such as ears, facial contours, body shapes, and fur textures appear repeatedly in different configurations. Learning reusable structure of this kind can be viewed as Feature Learning: the model develops representations that describe meaningful patterns shared across the dataset.

Real-world data is also far from uniformly distributed across every possible combination of pixel values. Instead, plausible images tend to occupy highly structured regions of the enormous input space, an idea closely related to Manifold Learning. From this perspective, Generation means drawing new samples from that structure, Density Estimation asks how well an observation fits within it, and Representation Learning aims to capture the structure itself.

Why High-Dimensional Data Distributions Are Difficult

So far, the objective may sound straightforward: learn the distribution that generated the data. The difficulty increases dramatically·however·once the observations become high-dimensional, as they do with images.

Even a small grayscale image contains many pixel values, while a modern high-resolution image can be represented by hundreds of thousands or even millions of values. The challenge is not whether each individual pixel looks reasonable in isolation. What matters is the joint structure formed by all of those pixels.

Consider a dog’s face. Having realistic pixels around the eyes and realistic pixels around the ears does not guarantee that the complete image will resemble a dog. The eyes must appear in compatible positions, the ears must align with the face, the body must have a coherent shape, and object boundaries must make sense relative to the background. Plausible data therefore depends on complex relationships among a large number of variables.

A Model Distribution for high-dimensional data must be expressive enough to capture these dependencies. A simple model may make probability evaluation and training convenient, but it can miss important structure in real data. A highly flexible model can represent much richer relationships, yet that flexibility may make probability computation and learning substantially more difficult.

The challenge·then·is not simply to build a more complicated model. We need a model that is expressive enough to represent complex distributions while remaining practical to compute and train. This tension leads directly to the Tradeoff between Tractability and Flexibility.

Tractability-Flexibility Tradeoff

Tractability describes how practical it is to perform operations such as evaluating probabilities and training the model. Flexibility, by contrast, describes how broad a range of complex distribution shapes the model can represent.

A simple Gaussian Distribution provides an intuitive example of this distinction. An isotropic Gaussian, whose density has a symmetric shape around its mean, can be written as follows.

$$ p_{\mu}(x)=\frac{1}{(2\pi)^{d/2}}\exp\left(-\frac{\lVert x-\mu\rVert^2}{2}\right) $$

Here, x is a data vector, μ is the mean vector, and d is the dimensionality of the data space. A simple Gaussian makes probability evaluation tractable, but its ability to represent complex data distributions is limited.

If the data forms a single compact cloud around a mean, a Gaussian can model it reasonably well. Observations near the center receive higher probability, while probability decreases as the distance from the mean grows. Both probability evaluation and parameter estimation remain relatively straightforward, making this kind of model highly tractable.

Real image distributions·however·rarely have such a simple shape. Dog images may contain frontal or side views, standing or running poses, and many combinations of backgrounds, lighting conditions·colors·and textures. Forcing all of these variations into a single simple Gaussian can blur distinct structures into one overly coarse distribution. The model remains easy to work with, but its flexibility is too limited to capture the data faithfully.

A more expressive model can represent a much wider range of distribution shapes and therefore has a better chance of capturing the structure of real data. That additional freedom comes at a cost: probability evaluation may become difficult, and learning the model parameters may require more sophisticated methods. Generative Modeling is therefore not simply about maximizing expressivity. The practical challenge is to balance expressive power with computational tractability.

This tradeoff explains why more powerful models become necessary. If high-dimensional real-world data is too structured and diverse for simple distributions, we need models capable of representing richer dependencies and more complicated distribution shapes. This requirement naturally leads to using Deep Neural Networks for probability distribution modeling.

Why Deep Generative Models Are Necessary

The limitations of simple probabilistic models ultimately come down to Model Expressivity. Real-world datasets contain large numbers of variables connected through complex dependencies. If a model cannot represent those relationships, even a well-designed training procedure will struggle to approximate the true Data Distribution accurately.

For example, a single simple distribution is unlikely to capture frontal dog faces, side profiles, different poses, varied backgrounds, and all of their possible combinations at once. A Deep Neural Network can build much richer representations by composing transformations across multiple layers, allowing the resulting model to capture substantially more complex structure. The point of using deep networks is therefore not simply to make the model larger. It is to provide enough expressive capacity to represent the complex distributions found in high-dimensional data.

$$ p_{\theta}(x) \approx p_{\mathrm{data}}(x) $$

The central goal is to use the expressive power of a deep model to learn a Model Distribution that closely approximates the unknown true Data Distribution.

Deep Generative Models are therefore not a separate idea introduced without motivation. They emerge naturally from the problem of estimating a Data Distribution from finite observations when the underlying high-dimensional structure is too complex for simple probabilistic models. Modeling richer data requires greater flexibility, and Deep Neural Networks provide a practical way to obtain that expressive power.

※ This article is an independently organized and restructured summary based on lectures by Professor Sungroh Yoon at Seoul National University.

📍 Where this concept fits in the AI learning map

See where this concept sits within the full AI Universe.

📍 Current position in AI Universe

☰

Reset Show completed · Login required Loading…

🌌 AI Universe

‹

›

⭐ Concept

Select a star.

View the full AI Universe

« 8.5 Approximate Inferenc…|8.7 Normalizing Constant… »

🔖 Tags: Data Distribution · Generative Modeling · Generative Models · i.i.d. Samples · Model Distribution · Probability Distribution