☆ Save 7.4 Multi-Head Attention, Positional Encoding, and Add & Norm — Core Components of the Transformer Block

06/15/2026

This article explains why Multi-Head Attention uses multiple Attention Heads, how Positional Encoding injects order information into Self-Attention, and how Add & Norm improves training stability in Transformer blocks. While Self-Attention is a powerful mechanism for modeling relationships between tokens, a single attention head cannot sufficiently capture diverse relational patterns, and it also lacks awareness of token order. Multi-Head Attention, Positional Encoding, Residual Connections, and Layer Normalization address these limitations and together form a fully functional Transformer Encoder Layer.

Table of Contents

• Why Multi-Head Attention is Needed
• Single-head vs Multi-head Attention
• Attention Heads and Representation Subspaces
• Head Concatenation and Output Projection Matrix
• Why Positional Encoding is Needed
• Sinusoidal Positional Encoding
• APE·RPE·and RoPE
• Add & Norm, Residual Connections, and Layer Normalization
• Pre-LN vs Post-LN

Why Multi-Head Attention is Needed

Multi-Head Attention splits a single Self-Attention operation into multiple parallel attention computations. The key idea is not simply repeating the same operation, but allowing each head to use different weight matrices to view the input from different perspectives.

In a sentence, some tokens capture syntactic relationships, others capture semantic relationships, and others focus on long-range dependencies. A single attention head may mix these signals or overemphasize one type of relation, leading to an Attention Bottleneck.

By using multiple heads, the model can separate and learn different types of relationships. For example, when interpreting the pronoun “it”, one head may focus on its referent such as “animal”, while another head may focus on surrounding verbs or context. This allows Multi-Head Attention to produce richer representations than single-head attention.

Single-head vs Multi-head Attention

Single-head Attention computes only one attention distribution. This provides a single perspective on which tokens should be attended to. While simple and interpretable, it struggles to capture multiple types of relationships simultaneously.

Multi-head Attention processes the same input through multiple heads in parallel. Each head uses different Projection Matrices to compute its own Query·Key·and Value representations. In the original Transformer, 8 attention heads were used, each operating in a different Representation Subspace.

Thus, Multi-Head Attention is less about repeating attention multiple times and more about viewing token relationships from multiple representation spaces simultaneously. This allows the model to decompose linguistic relationships into diverse patterns.

Attention Heads and Representation Subspaces

An Attention Head is an independent unit within Multi-Head Attention that computes its own attention distribution. Since each head has separate Query·Key·and Value projections, different heads can focus on different aspects of the same input.

Each head operates in its own Representation Subspace. For example, one head may focus on local word dependencies, another on subject–verb relationships, and another on long-distance dependencies. Instead of learning identical patterns, heads specialize in different types of signals.

As a result, Multi-Head Attention allows a sentence to be interpreted from multiple perspectives simultaneously, combining syntactic·semantic·and contextual cues.

Head Concatenation and Output Projection Matrix

Each attention head produces its own output. If there are 8 heads, the outputs are \(Z_0, Z_1, \dots, Z_7\). These outputs must be combined into a single vector representation before passing to the next layer.

First, the outputs are concatenated. This combines information from all heads but does not yet map it into the final model space. Therefore, an Output Projection Matrix is applied to mix and transform the concatenated representation.

• \[ \text{MultiHead}(Q·K·V)=\text{Concat}(\text{head}_1,\dots,\text{head}_h)W^O \]

  The outputs of multiple heads are concatenated and projected back using an output projection matrix.

For example, if there are two heads each producing 3-dimensional outputs, concatenation yields a 6-dimensional vector. Multiplying by \(W^O\) maps it back to the model’s expected output dimension. Concatenation gathers information, while the projection matrix recombines it.

Why Positional Encoding is Needed

Self-Attention alone cannot capture word order because all tokens are processed simultaneously. For example, “dog bites man” and “man bites dog” contain the same words but have completely different meanings. To resolve this, the model must know token positions.

Positional Encoding is added to input embeddings to inject positional information. This allows the Transformer to represent both what a word is and where it appears in the sequence.

Positional Encoding helps the model learn not only absolute positions but also relative distances between tokens. These position vectors are added to embeddings before entering the encoder.

There are two main approaches: Learned Positional Embedding, where position vectors are trainable parameters, and Sinusoidal Positional Encoding, which uses fixed sine and cosine functions.

Sinusoidal Positional Encoding

Sinusoidal Positional Encoding uses sine and cosine functions with different frequencies depending on position and dimension. Even dimensions use sine, while odd dimensions use cosine.

• \[ PE(pos,2i)=\sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right), \quad PE(pos,2i+1)=\cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) \]

  pos is the token position, i is the dimension index, and d_model is the embedding dimension.

For example, when \(d_{\text{model}}=4\) and \(pos=1\), the positional vector consists of four values combining different sine and cosine frequencies.

• \[ PE(1)=[\sin(1),\cos(1),\sin(0.01),\cos(0.01)] \approx [0.8415,0.5403,0.0100,0.99995] \]

  Each position encodes multiple frequency components to represent positional structure.

This design does not simply assign an index to each position. Instead, multiple frequency signals allow the model to capture both absolute position and relative distance information.

APE, RPE, and RoPE

Positional encoding has multiple variants. Absolute Positional Embedding (APE) assigns a unique vector to each position, while Relative Positional Embedding (RPE) focuses on relative distances between tokens rather than absolute positions.

Rotary Positional Embedding (RoPE) encodes positional information by rotating Query and Key representations. These methods all serve the same purpose of injecting position information into the Transformer.

Add & Norm, Residual Connections, and Layer Normalization

Each sublayer in a Transformer Encoder block is wrapped with a Residual Connection followed by Layer Normalization, a structure known as Add & Norm.

The residual connection adds the input directly to the output of the sublayer, helping preserve information and improve gradient flow in deep networks.

Layer Normalization normalizes each token independently across its feature dimensions. This stabilizes training by ensuring consistent activation scales.

• \[ \text{Output}=\text{LayerNorm}(x+\text{Sublayer}(x)) \]

  The input is added to the sublayer output and then normalized.

For example, if \(x=10\) and \(\text{Sublayer}(x)=3\), the residual output becomes 13 before normalization. The key idea is preserving original information while adding transformations.

Pre-LN vs Post-LN

Layer Normalization placement leads to two variants: Post-LN and Pre-LN. The original Transformer used Post-LN, where normalization is applied after each sublayer.

Modern Transformers often use Pre-LN, where normalization is applied before the sublayer. Both designs affect training stability, especially in deep models.

Thus, Add & Norm is not just a post-processing step but a fundamental mechanism that enables stable training of deep Transformer architectures.

※ This article was independently compiled and adapted from lectures by Professor Sungroh Yoon at Seoul National University.

Recommended prerequisite reading (3/5)

+2

• 7. Transformer — From Self-Attention to Modern LLM Architectures
• 4. Multilayer Perceptron (MLP) — Fundamental Neural Network Structure and Learning Principles
• 2.8 Fundamentals of Neural Networks

• 5.2 Background and Design Principles Behind CNNs
• 4.1 Multilayer Perceptron (MLP) — Intuition and Core Components

Recommended next reading (5/17)

+5

• 5.3 Understanding the Core Components and Structure of Convolutional Neural Networks
• 7.1 Transformer Architecture and Core Components — Sequence-to-Sequence Models and Encoder–Decoder Structure
• 7.9 Modern Transformer Blocks in Large Language Models — Key Changes in 2024-era Transformer Architecture
• Transformer Block — Why It Became the Standard Architecture After Replacing RNNs
• Hybrid Attention — Efficient Attention for Long-Context Transformers

• Causal Local Attention — Reducing the Cost of Long-Sequence Generation
• Stack of Decoders — Why Multiple Decoder Layers Improve Transformer Understanding
• CNN Parameter Count — A Core Metric That Determines Model Complexity and Representational Capacity
• MoE Transformer — Why the FFN Is Split into Experts
• Reformer — Scaling Attention to Longer Sequences

• Linformer — Reducing Attention Cost with Low-Rank Attention
• Softmax-Free Attention — How Attention Can Find Important Information Without Softmax
• Attention Collapse — Why LLMs Overfocus on a Small Set of Tokens
• Sliding Window Attention — Why Full Attention Becomes Inefficient in Long Contexts
• Multiplicative vs Additive Mask — Why Masking Before Softmax Matters
• Strided Attention — Sparse Connectivity for Efficient Long-Context Processing
• Quadratic Complexity — Why Attention Runs Into an n² Bottleneck

Posts on the same topic (7/7)

• Convolution Kernels — How CNNs Capture Local Patterns
• GShard — Scaling MoE Transformers with Automatic Sharding
• Model Expressivity — How Neural Networks Represent Complex Functions,
• Restricted Neural Network Model — How Structural Constraints Shape Neural Network Learning
• Routing — How AI Models Choose a Computation Path for Each Input
• Switch Transformer — How MoE Simplifies Scaling with Top-1 Routing
• Top-K Sparse Routing — How MoE Scales AI Models by Reducing Computation

Related concepts (2/2)

• BigBird — Sparse Attention Design for Long-Context Processing
• Longformer — Why Sparse Attention Matters for Long-Context Processing

📍 Where this concept fits in the AI learning map

See where this concept sits within the full AI Universe.

📍 Current position in AI Universe

☰

Reset Show completed · Login required Loading…

🌌 AI Universe

‹

›

⭐ Concept

Select a star.

View the full AI Universe

« 7.3 Self-Attention Mecha…|7.5 Transformer Decoder,… »

🔖 Tags: Deep Learning · layer normalization · multi-head attention · positional encoding · Residual Connection · Transformer