☆ Save 4. Multilayer Perceptron (MLP) — Fundamental Neural Network Structure and Learning Principles

10/28/2025

Multilayer Perceptron (MLP) is the most classic neural network architecture that learns complex input–output relationships by transforming data through multiple layers. By introducing hidden layers and activation functions, an MLP can model nonlinear patterns that linear models cannot capture—making it a foundational starting point for modern deep learning.

In this lesson, we connect the full story—from the MLP’s structure and information flow, to probabilistic outputs, to how learning signals propagate through the network and get optimized—so you can understand the core mechanics as one coherent pipeline.

Table of Contents

• MLP structure and information flow
• Nonlinear representation learning and the role of hidden layers
• Output layer and probabilistic interpretation
• Learning signal and the principle of backpropagation
• How optimization interacts with architecture

MLP structure and information flow

An MLP consists of an input layer, one or more hidden layers, and an output layer. Each layer takes the previous layer’s output, applies a linear transformation, and then passes it through a nonlinear activation function to gradually build more useful internal features.

The core computation at layer l is commonly written as:

• \[ \mathbf{h}^{(l)} = \sigma\!\left(\mathbf{W}^{(l)}\mathbf{h}^{(l-1)} + \mathbf{b}^{(l)}\right) \]

  The previous layer’s representation is transformed by weights and bias, then shaped by an activation function into a new representation.

For a quick numeric intuition, suppose the input is \([1.0,\,2.0]\), the weight row is \(\begin{bmatrix}0.5 & -1.0\end{bmatrix}\), and the bias is \(0.2\). The pre-activation value becomes

\[ z = 0.5\times1.0 + (-1.0)\times2.0 + 0.2 = -1.3 \]

Applying ReLU gives \( \max(0,-1.3)=0 \). Repeating this kind of transformation across layers yields increasingly abstract features.

Nonlinear representation learning and the role of hidden layers

The central job of hidden layers is to map inputs into a representation space where the task becomes easier. A single linear mapping cannot produce complex decision boundaries, but stacking nonlinearities enables an MLP to approximate highly complex functions.

In practice, this means data that is not linearly separable in the original input space can become separable in a learned feature space. Instead of hand-engineering features, an MLP supports representation learning: it discovers useful features directly from data.

Output layer and probabilistic interpretation

In classification, the output layer is often interpreted as a probability distribution over classes rather than arbitrary numbers. The most common tool for this is the softmax function.

• \[ P(y=i\mid\mathbf{x}) = \frac{e^{z_i}}{\sum_j e^{z_j}} \]

  Softmax converts logits into class probabilities that sum to 1.

For example, if the logits are \([2.0, 1.0, 0.1]\), softmax produces roughly \([0.66, 0.24, 0.10]\). This means the first class has the highest predicted probability.

Learning signal and the principle of backpropagation

An MLP learns by measuring the mismatch between predictions and ground-truth labels using a loss function, then updating parameters to reduce that loss. The mechanism that propagates this learning signal through the network is backpropagation.

Backprop uses the chain rule to compute how much each parameter contributed to the final loss.

• \[ \frac{\partial L}{\partial \mathbf{W}^{(l)}} = \frac{\partial L}{\partial \mathbf{h}^{(l)}} \cdot \frac{\partial \mathbf{h}^{(l)}}{\partial \mathbf{W}^{(l)}} \]

  The loss sensitivity at a layer decomposes into upstream sensitivity times the local derivative.

This is why neural networks can learn efficiently: the model identifies which connections are most responsible for errors and nudges parameters in the direction that reduces those errors.

How optimization interacts with architecture

Training performance is not determined by the optimizer alone. Architectural choices—such as depth, width, activation functions, and initialization—directly affect stability and convergence speed.

Deeper networks can be more expressive but may suffer from issues like vanishing gradients. Very wide networks can increase capacity but also raise the risk of overfitting. In practice, architecture and optimization should be understood as a coupled system that shapes learning dynamics.

※ This article is an independently organized and restructured summary based on lectures by Professor Sungroh Yoon at Seoul National University.

Recommended prerequisite reading (3/5)

+2

• 4.1 Multilayer Perceptron (MLP) — Intuition and Core Components
• 2.8 Fundamentals of Neural Networks
• 7. Transformer — From Self-Attention to Modern LLM Architectures

• 7.4 Multi-Head Attention, Positional Encoding, and Add & Norm — Core Components of the Transformer Block
• 5.2 Background and Design Principles Behind CNNs

Recommended next reading (5/17)

+5

• 5.3 Understanding the Core Components and Structure of Convolutional Neural Networks
• 7.1 Transformer Architecture and Core Components — Sequence-to-Sequence Models and Encoder–Decoder Structure
• 7.9 Modern Transformer Blocks in Large Language Models — Key Changes in 2024-era Transformer Architecture
• Deep Neural Network (DNN) — Learning Complex Patterns by Stacking Many Layers
• Skip Connection (Residual Connection) — A Shortcut Path That Preserves Information and Gradients in Deep Neural Networks

• Restricted Neural Network Model — How Structural Constraints Shape Neural Network Learning
• Model Expressivity — How Neural Networks Represent Complex Functions,
• Sparse Interactions — A Structural Principle That Preserves Only Necessary Connections for Computational Efficiency and Effective Representation Learning
• Activation Function — The nonlinearity that boosts a neural network’s expressive power
• Weight Sharing — Reusing the Same Filter Across Locations to Simplify Learning

• Sigmoid Function — A Classic Nonlinear Curve That Squeezes Outputs into Probabilities
• CNN Parameter Count — A Core Metric That Determines Model Complexity and Representational Capacity
• Transformer Block — Why It Became the Standard Architecture After Replacing RNNs
• Stack of Decoders — Why Multiple Decoder Layers Improve Transformer Understanding
• Pooling — Downsampling that summarizes features, reduces resolution, and improves shift robustness
• Convolution Kernels — How CNNs Capture Local Patterns
• Linear Activation Function — The Simplest Activation Where Output Scales Proportionally with Input

Posts on the same topic (8/15)

+7

• Attention Collapse — Why LLMs Overfocus on a Small Set of Tokens
• Causal Local Attention — Reducing the Cost of Long-Sequence Generation
• GShard — Scaling MoE Transformers with Automatic Sharding
• Hybrid Attention — Efficient Attention for Long-Context Transformers
• Linformer — Reducing Attention Cost with Low-Rank Attention
• MoE Transformer — Why the FFN Is Split into Experts
• Multiplicative vs Additive Mask — Why Masking Before Softmax Matters
• Quadratic Complexity — Why Attention Runs Into an n² Bottleneck

• Reformer — Scaling Attention to Longer Sequences
• Routing — How AI Models Choose a Computation Path for Each Input
• Sliding Window Attention — Why Full Attention Becomes Inefficient in Long Contexts
• Softmax-Free Attention — How Attention Can Find Important Information Without Softmax
• Strided Attention — Sparse Connectivity for Efficient Long-Context Processing
• Switch Transformer — How MoE Simplifies Scaling with Top-1 Routing
• Top-K Sparse Routing — How MoE Scales AI Models by Reducing Computation

Related concepts (2/2)

• Recognition Network — How Neural Networks Learn Features for Recognition
• Padding Mask — Why Transformers Must Ignore Padding Tokens

📍 Where this concept fits in the AI learning map

See where this concept sits within the full AI Universe.

📍 Current position in AI Universe

☰

Reset Show completed · Login required Loading…

🌌 AI Universe

‹

›

⭐ Concept

Select a star.

View the full AI Universe

« 3.3 Regularization in Ma…|4.1 Multilayer Perceptro… »

🔖 Tags: Backpropagation · Deep Learning · MLP · multilayer perceptron · Neural Networks · Representation Learning