☆ Save 7.3 Self-Attention Mechanism — From Query–Key–Value to Matrix Computation

06/15/2026

This article explains how the Self-Attention Mechanism is computed, starting from the Query–Key–Value (QKV) formulation and extending to Matrix Computation. Self-Attention is not just an intuitive idea of “looking at relevant words”; it is a mathematical process where each token computes relationships with all other tokens, forming the core of the Transformer architecture.

This article also breaks down Self-Attention into a sequence of steps: vector generation → similarity computation → normalization → weighted sum, and shows how these steps can be unified into a single matrix operation.

Table of Contents

• Self-Attention vs Cross Attention
• Query–Key–Value Vector Generation
• Attention Score Computation
• Scaling and Softmax
• Output via Weighted Sum
• Matrix Form of Self-Attention

Self-Attention vs Cross Attention

Self-Attention is a mechanism where each token in a sequence attends to all other tokens within the same sequence. In other words, all words in a sentence compute relationships with each other to update their representations.

In contrast, Cross Attention computes relationships between two different sequences. For example, in machine translation, the output sequence attends to the input sequence. Self-Attention models intra-sequence relationships, while Cross Attention models inter-sequence relationships.

Query–Key–Value Vector Generation

The first step in Self-Attention is generating Query, Key, and Value vectors from input embeddings. Each of these vectors plays a distinct role in how attention selects and transforms information.

Each token represents: “what information it wants to find (Query)”, “what characteristics it has (Key)”, and “what information it will pass on (Value)”. These roles are separated using different projections.

• \[ Q = XW_Q, \quad K = XW_K, \quad V = XW_V \]

  Query·Key·and Value are generated by multiplying the input embedding X with separate weight matrices.

For example, if the embedding dimension is 512 and the QKV dimension is 64, the input is projected into a lower-dimensional space used for attention computation. This is not simple compression but role-specific transformation.

Attention Score Computation

The second step computes similarity between Query and all Key vectors. This determines how much each token should attend to others.

• \[ \text{score} = q \cdot k \]

  Dot product between Query and Key measures token similarity.

For example, if q₁·k₁ = 112 and q₁·k₂ = 96, then the first token attends more strongly to the first key than the second. These scores are not probabilities yet but raw similarity values.

Scaling and Softmax

Dot-product values tend to grow with higher dimensionality. If Softmax is applied directly, it may produce overly peaked distributions, causing unstable gradients during training.

To address this, Scaling is applied by dividing by the square root of the key dimension.

• \[ \frac{q \cdot k}{\sqrt{d_k}} \]

  Stabilizes values to prevent Softmax saturation.

After this, Softmax converts scores into attention weights. For example, scores [14, 12] become approximately [0.88, 0.12].

Output via Weighted Sum

The final step computes a weighted sum of Value vectors using attention weights. This produces the final output of Self-Attention.

• \[ z = \sum_i \alpha_i v_i \]

  Weighted sum of Value vectors produces the final representation.

For example, if Values are [10, 20] and weights are [0.88, 0.12], the output is 11.2, meaning the first token contributes more strongly.

Matrix Form of Self-Attention

In practice, all these operations are performed simultaneously using matrix operations. Instead of processing tokens individually, the entire sequence is computed in parallel as matrices.

• \[ \text{Attention}(Q·K·V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V \]

  Self-Attention expressed as a single matrix operation.

This formulation enables the model to compute relationships between all tokens at once. Thanks to this matrix formulation, Transformers achieve efficient parallel processing compared to RNNs, significantly improving scalability and training speed.

※ This article was independently compiled and adapted from lectures by Professor Sungroh Yoon at Seoul National University.

Recommended prerequisite reading (3/5)

+2

• 5.7 Data Processing and Normalization Techniques for Optimizing CNN Training
• 5. Understanding Image Classification and Convolutional Neural Networks
• 4.2 Output Layer and Probabilistic Interpretation — Probabilistic Interpretation of Neural Network Outputs

• Attention-based Architecture — A Neural Network Architecture That Dynamically Weighs the Importance of Inputs
• Vision Transformer (ViT): How Images Are Understood Through Patch-Based Global Attention

Recommended next reading (5/20)

+5

• 7.8 Advanced Positional Embeddings — APE·RPE·and RoPE in Transformer Models
• Transformer Attention Projection — How QKV Separation Drives Attention Computation
• Head Redundancy — Why Attention Heads Learn Overlapping Roles
• Weighted Sum of Value Vectors — Why Attention Mixes Information Instead of Selecting It
• Unified Representation — Why Attention Compares Meaning and Position Together

• Sparse Attention — How Long Contexts Can Be Processed by Computing Only Selected Token Connections
• Token-wise Normalization — Why It Is Critical for Stable Transformer Training
• Global Token — How Transformers Turn an Entire Input into One Representation
• Permutation Equivariance — Why Outputs Follow Changes in Input Order
• Position Interpolation — Why RoPE Loses Its Sense of Position in Long Contexts

• Permutation-Equivariant — Why Changing Input Order Changes Output Order in the Same Way
• SiLU Activation — Why It Provides More Expressive Power Through Non-Monotonic Gating Compared to ReLU
• Swish Activation — Why It Reduces Gradient Disruption Compared to ReLU
• Weight Tying — Why Transformers Share the Space for Understanding and Generating Words
• Scaled Dot-Product Attention — Why √dₖ Scaling Stabilizes Attention
• Projection Layer — How Hidden Representations Are Transformed into Output Space
• Latent Bottleneck — Why Models Pass Information Through a Compact Latent Space
• Self-Organizing Map (SOM) — A Map-Style Neural Network That Reveals Data Structure on Its Own
• Feature Map & Activation Map — How CNNs Turn an Input into an Internal Representation
• Location-based Addressing — Accessing Memory by Position Instead of Content

Posts on the same topic (3/3)

• Distance-based Inductive Bias — Why Models Prefer Nearby Information
• Permutation Invariance — Why Changing Input Order Does Not Change the Output
• Recognition Network — How Neural Networks Learn Features for Recognition

Related concepts (2/2)

• Image Colorization — How AI Colorizes Black-and-White Images
• Hidden Variable Interaction — How Deep Learning Learns Relationships Between Hidden Variables

📍 Where this concept fits in the AI learning map

See where this concept sits within the full AI Universe.

📍 Current position in AI Universe

☰

Reset Show completed · Login required Loading…

🌌 AI Universe

‹

›

⭐ Concept

Select a star.

View the full AI Universe

« 7.2 Attention and Langua…|7.4 Multi-Head Attention… »

🔖 Tags: Attention Mechanism · Deep Learning · natural language processing · query-key-value · self-attention · Transformer