☆ Save 8.7 Normalizing Constant — Turning Model Scores into Valid Probability Distributions

09/13/2026

Neural networks are powerful function approximators, but their raw outputs are not automatically probabilities. A model might assign a high score to an image that looks similar to the training data and a lower score to one that does not, yet those scores can be negative, arbitrarily large, and have no reason to sum or integrate to 1.

To turn such scores into a probability distribution, we need two steps. First, the values must be transformed so they are nonnegative. Then they must be rescaled so that the total probability over the entire input space equals 1. The quantity required for this second step is the normalizing constant. This chapter starts with why normalization is necessary and then shows why a quantity that is easy to compute for simple distributions can become a major challenge in complex deep generative models.

Table of Contents

• Neural Network Outputs and Valid Probability Distributions
• What the Normalizing Constant Does
• Why Normalization Becomes Difficult in Complex Models
• Generative Modeling Strategies for Handling Normalization

Neural Network Outputs and Valid Probability Distributions

Consider a simple example. Suppose a neural network assigns scores of 2, 1, and -1 to three inputs A·B·and C. These values can still express relative preference: A receives the highest score, followed by B, then C. But because one of the scores is negative, the three values cannot be interpreted directly as probabilities.

Applying an exponential function changes that. The scores 2, 1, and -1 are mapped to positive values while preserving their ordering: larger original scores still produce larger transformed values, and even the negative score becomes positive. This gives us nonnegative quantities, which is a basic requirement for a probability density.

Positivity alone is not enough, however. The transformed values still need to be normalized. If the three exponential values add up to 12, for example, each value must be divided by 12 so that the total becomes 1. For continuous data, the same idea uses an integral over the entire input space instead of a finite sum. The quantity that captures this total scale is the normalizing constant.

• $$ p_\theta(x)=\frac{e^{f_\theta(x)}}{Z_\theta} $$

  Exponentiation makes the model score nonnegative, and division by the normalizing constant turns it into a valid probability density.

Here, x is the data point whose probability density we want to evaluate, and the neural network function assigns a score to that input. A larger exponential term in the numerator can give x a higher probability density relative to other inputs. On its own·however·this is still an unnormalized value. To turn it into a valid probability density over the entire input space, we need the normalizing constant in the denominator.

What the Normalizing Constant Does

For a continuous probability distribution to be valid, its probability density must integrate to 1 over the full input space. This means we cannot simply use the exponentiated neural network scores as they are. We first need to determine their total mass across all possible inputs.

• $$ Z_\theta=\int e^{f_\theta(x)}dx $$

  The normalizing constant measures the total mass of the unnormalized values over the entire input space.

This is the continuous version of the earlier example, where each exponentiated score was divided by the sum of all three values. The difference is that real-valued data may live in a continuous space rather than a finite set such as A·B·and C, so the finite sum is replaced by an integral.

Once we divide by the normalizing constant, the resulting probability density integrates to exactly 1.

• $$ \int p_\theta(x)dx=\frac{1}{Z_\theta}\int e^{f_\theta(x)}dx=1 $$

  Dividing by the total mass of the numerator ensures that the normalized probability density integrates to 1.

This is exactly why the quantity is called a normalizing constant: it preserves the relative scale of the model’s scores while rescaling the entire function so that it satisfies the requirements of a probability distribution.

Normalization is not always difficult. For simple distributions with well-understood mathematical structure, the required constant can often be derived explicitly. A standard example is the multivariate Gaussian distribution with identity covariance.

• $$ p_\mu(x)=\frac{1}{(2\pi)^{d/2}}e^{-\frac{\lVert x-\mu\rVert^2}{2}} $$

  For a Gaussian distribution with identity covariance, the normalization factor that makes the total probability density integrate to 1 can be computed explicitly.

In this Gaussian example, the normalization factor is determined directly by the dimensionality of the data. Once the form of the distribution is known, we can evaluate its probability density at a given input without solving a new normalization problem each time.

The situation changes when the distribution is defined by a flexible neural network rather than a fixed analytical form such as a Gaussian.

Why Normalization Becomes Difficult in Complex Models

Evaluating a neural network at a single image is usually straightforward: feed the image into the network and compute its score. Computing the normalizing constant is fundamentally different. Instead of evaluating one image, we need to account for the values produced by the network across the entire space of possible images.

For a small input space, we might enumerate every possibility and add the resulting values. Real images·however·live in extremely high-dimensional spaces with an enormous number of possible inputs. The neural network itself may also represent a highly complex function, so there is no guarantee that the integral of its exponentiated output over the full input space has a simple closed-form solution.

This is the real source of the normalization problem. It is not simply that the data is high-dimensional; we must account for the values produced by a complex neural network over the entire input space. Higher dimensionality makes this problem even more severe, and exact computation of the normalizing constant can eventually become intractable.

The contrast with a Gaussian makes the tradeoff clear. A Gaussian has a fixed mathematical form, so its normalization factor can be written down explicitly. A neural network can represent much richer and more flexible distributions, but that flexibility may make the required normalization integral difficult to compute. Models that are easy to evaluate may offer limited expressive power, while more expressive models may come with harder computations. This tension is known as the Tractability-Flexibility Tradeoff.

For this reason, designing a deep generative model is not simply a matter of making the neural network more complex to represent richer distributions. We also have to consider whether the computations required to treat that model as a valid probability distribution remain practical. The normalizing constant exposes this challenge particularly clearly.

Generative Modeling Strategies for Handling Normalization

If the normalizing constant cannot be computed exactly, there is no single required solution. Some models approximate it, others constrain the model architecture so that normalization remains tractable, and still others avoid working directly with an explicit probability density and focus instead on the process of generating data.

• Approximate the normalizing constant: Energy-Based Models address the problem by approximating a normalizing constant that is difficult to compute exactly.
• Use constrained model architectures: Autoregressive Models, Flow-based Models, and Variational Autoencoders impose structural constraints that make the required probability calculations tractable.
• Model the generation process directly: GANs focus on learning how to generate data rather than explicitly computing a probability density and its normalizing constant.
• Bypass the normalizing constant: Score-based Diffusion Generative Models approach the problem without directly computing the normalizing constant.

Here, the term Restricted Neural Network Model does not simply mean a smaller or less expressive neural network. The important idea is that specific constraints are placed on the model architecture so that the required probability computations remain feasible.

This normalization problem is one reason different deep generative models rely on fundamentally different architectures. Representing complex data distributions is only part of the challenge. A model must also make the required probability calculations tractable or provide another way to handle a normalizing constant that is difficult to compute. The normalizing constant is therefore more than just a number in the denominator: it exposes the fundamental tension between expressive power and computational tractability when neural networks are used as probabilistic models.

※ This article is an independently organized and restructured summary based on lectures by Professor Sungroh Yoon at Seoul National University.

Recommended prerequisite reading (3/5)

+2

• 8.6 Data Distribution Modeling — How Generative Models Learn Data Distributions
• Unnormalized Probability Models — Why Models Use Scores Before Normalization
• Normalizing Constant — The Key to Turning Relative Scores into a Probability Distribution

• High-Probability Region — How Machine Learning Identifies Important Areas of a Data Distribution
• Continuous Uniform Distribution — Why Probability Is Proportional to Interval Length

Recommended next reading (5/16)

+5

• 8.5 Approximate Inference — Approximating Posterior Distributions with MCMC and Variational Inference
• 8.4 Posterior Inference — Posterior Distributions and Why Exact Inference Is Hard
• Probability Normalization — How Softmax Converts Model Outputs into a Probability Distribution
• Valid Probability Distributions: Why Probabilities Must Sum to 1
• 8.8 Deep Generative Models — Major Approaches to Probability Distribution Modeling

• Probability Distribution Comparison — How to Measure and Interpret Differences Between Distributions
• Sequence Probability — Modeling Ordered Data with Conditional Probabilities
• Exponential Distribution — A Continuous Probability Distribution for Modeling Waiting Time Until an Event
• Parametric Approximation — Modeling Data Distributions with Probabilistic Models
• Likelihood — A Criterion for Judging Which Model Best Explains the Observed Data

• Poisson Distribution — A Discrete Distribution for Modeling Event Counts in a Fixed Time or Space Interval
• Multinoulli Distribution — Modeling One Choice Among Many Categories
• Joint Distribution Modeling — How AI Learns Relationships Between Variables
• Probability Factorization — Computing Complex Joint Probabilities via a Chain of Conditional Probabilities
• 8.9 Diffusion Models — Generating Data by Adding and Reversing Noise
• Markov Chain — How Transition Probabilities Drive State Evolution

Posts on the same topic (8/8)

• Gibbs Sampling: Sampling from a Posterior One Parameter at a Time
• Intractable Posterior — Why Bayesian Inference Needs Approximate Inference
• Markov Chain Monte Carlo (MCMC) — Sampling the Posterior with a Markov Chain
• Metropolis-Hastings Algorithm — How MCMC Samples from a Posterior
• Stationary Distribution — How Long-Run Probabilities Emerge in a Markov Chain
• Stochastic Inference — Reasoning with Uncertainty in AI
• Variational Distribution — How Variational Inference Approximates the Posterior
• Variational Inference — Approximating Intractable Bayesian Posteriors

Related concepts (2/2)

• SMLD (Score Matching with Langevin Dynamics) — Score-Based Sampling
• Sampling-based Inference — Why Probabilistic Token Selection Matters in LLM Decoding

📍 Where this concept fits in the AI learning map

See where this concept sits within the full AI Universe.

📍 Current position in AI Universe

☰

Reset Show completed · Login required Loading…

🌌 AI Universe

‹

›

⭐ Concept

Select a star.

View the full AI Universe

« 8.6 Data Distribution Mo…|8.8 Deep Generative Mode… »

🔖 Tags: energy-based models · Generative Models · Neural Networks · Normalizing Constant · Probability Density · Probability Distribution