☆ Save 7. Transformer — From Self-Attention to Modern LLM Architectures

06/15/2026

This article explains how the Transformer overcomes the limitations of traditional sequential models and how it evolved from Self-Attention to modern Large Language Model (LLM) architectures. Unlike sequential processing models, Transformer processes the entire input at once and computes relationships between tokens in parallel. This property makes Transformer the core neural architecture behind modern NLP, code generation, multimodal systems, and large-scale language models.

Table of Contents

• Why Transformers Were Introduced
• Basic Transformer Architecture
• Attention and Language Modeling
• Self-Attention and QKV Matrix Computation
• Multi-Head Attention and Positional Encoding
• Transformer Decoder and Decoding Strategies
• Efficient Attention and FlashAttention
• KV Cache and Shared Key-Value Attention
• APE·RPE·RoPE and the Evolution of Positional Representations
• Modern Transformer Blocks and Latest LLM Architectures

Why Transformers Were Introduced

Before Transformers, Recurrent Neural Networks (RNNs) and traditional Sequence-to-Sequence Models dominated sequential data processing. RNNs process inputs step by step while passing hidden states through time. This naturally preserves order, but it suffers from weak long-range dependency modeling and poor parallelization.

Transformers address this bottleneck by processing the entire input sequence simultaneously and directly computing relationships between tokens. The key idea is Self-Attention, which allows each token to attend to all other tokens in the sequence.

This is not just a speed improvement. It enables Parallelization in Transformers, stronger long-context reasoning, and scalability to large datasets and large models. This is why modern LLMs are built on the Transformer architecture.

Basic Transformer Architecture

The original Transformer was designed as an Encoder–Decoder Architecture. The Transformer Encoder converts input tokens into contextual representations, while the Transformer Decoder generates output tokens based on those representations.

Input text is first split into Tokens, and each token is converted into a vector using Word Embeddings or token embeddings. These vectors form a Dense Vector Representation that captures semantic meaning.

The encoder consists of stacked Encoder Layers (Transformer Blocks), each containing Self-Attention and Feed-Forward Networks. The decoder adds Masked Self-Attention and Cross-Attention for autoregressive generation.

Attention and Language Modeling

A Language Model predicts the next token based on previous context. For example, given “I love”, it estimates probabilities for the next token such as “you”, “it”, or “this”. This forms the basis of Autoregressive Generation.

Attention Mechanism determines which parts of the input are most relevant at each step. Instead of compressing all information into a single vector, it assigns weights over all tokens and produces a Context Vector.

Attention is based on Query–Key–Value (QKV). The Query defines what we are looking for, the Key represents features of each token, and the Value contains the actual information to retrieve.

Self-Attention and QKV Matrix Computation

Self-Attention computes relationships among tokens within the same sequence. Unlike Cross-Attention, which connects different sequences, Self-Attention operates within a single sentence.

It computes Query·Key·and Value matrices from input embeddings, calculates attention scores using dot products, applies scaling and softmax to obtain attention weights, and finally aggregates values into new token representations.

• \ [ \text{Attention}(Q·K·V)=\text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V \]

  This equation represents the full Self-Attention computation.

This formulation is the core of the Transformer. It enables full parallel computation of token relationships and improves long-range dependency learning compared to RNNs.

Multi-Head Attention and Positional Encoding

Multi-Head Attention uses multiple attention heads in parallel to capture different types of relationships, such as syntactic·semantic·or long-distance dependencies.

Each head uses separate projection matrices for Query·Key·and Value, operating in different representation subspaces. Their outputs are concatenated and projected again through a final output matrix.

However, Self-Attention has no built-in sense of order. This is why Positional Encoding is required. It injects positional information into token embeddings so the model understands both meaning and position.

Residual connections and Layer Normalization are also used, known as Add & Norm, to stabilize training in deep networks. Modern LLMs extend this with Pre-LayerNorm and RMSNorm variants.

Transformer Decoder and Decoding Strategies

The Transformer Decoder generates output sequences. It uses a causal mask so that each token can only attend to previous tokens, ensuring autoregressive generation.

During training, Teacher Forcing is used by shifting the target sequence to the right and feeding it into the decoder. During inference, the model feeds its own generated tokens back as input.

The final hidden states are projected through a Language Modeling Head (LM Head) into logits over the vocabulary. These logits are converted into probabilities via softmax, and decoding strategies such as greedy search, beam search, top-k sampling, and top-p sampling are used to select the next token.

Efficient Attention and FlashAttention

Although Transformers are powerful, full attention has quadratic complexity O(n²), which becomes expensive for long contexts.

Efficient Attention methods reduce this cost. Local attention restricts computation to nearby tokens, while sparse attention selects only a subset of connections.

FlashAttention improves efficiency from a different angle by optimizing memory access patterns between GPU memory hierarchies, reducing IO overhead and speeding up attention computation.

KV Cache and Shared Key-Value Attention

KV caching reduces redundant computation during autoregressive inference by storing previously computed keys and values.

However, KV cache does not eliminate attention computation. It still requires computing attention between new queries and cached keys/values, and memory usage grows with context length.

To address this, Multi-Query Attention, Grouped-Query Attention, and Multi-Head Latent Attention share or compress key-value representations to improve efficiency in modern LLMs.

APE, RPE, RoPE and the Evolution of Positional Representations

Since Transformers do not inherently encode order, positional representations are essential. Absolute Positional Embeddings (APE) assign a unique vector to each position but struggle with unseen long sequences.

Relative Positional Embeddings (RPE) encode relative distances between tokens, improving generalization across sequence lengths.

Rotary Positional Embeddings (RoPE) rotate Query and Key vectors to encode relative positions naturally within attention computation. RoPE is widely used in modern LLMs due to its strong performance in long-context settings.

Modern Transformer Blocks and Latest LLM Architectures

Modern LLMs preserve the core Transformer structure but adapt it for large-scale training and long-context inference. A typical 2024-era architecture looks like this:

Input
→ RMSNorm or Pre-Layer Normalization
→ Self-Attention with Grouped-Query Attention and Rotary Positional Embedding
→ Residual Connection
→ RMSNorm or Pre-Layer Normalization
→ Feed-Forward Network with SwiGLU Activation or Mixture of Experts
→ Residual Connection

RMSNorm simplifies LayerNorm, while SwiGLU improves feed-forward expressiveness. Mixture of Experts (MoE) activates only a subset of expert networks using a routing or gating mechanism.

Multi-Token Prediction allows models to predict multiple future tokens simultaneously, improving training efficiency and learning signal density.

Ultimately, the Transformer is not just a model but the design language of modern LLMs. Self-Attention models relationships, positional encoding injects order, decoders generate tokens, and efficient attention plus KV caching enable scalable inference. Combined with RMSNorm·RoPE·GQA·SwiGLU·and MoE, these components form today’s large-scale Transformer systems.

※ This article was independently compiled and adapted from lectures by Professor Sungroh Yoon at Seoul National University.

Recommended prerequisite reading (3/5)

+2

• 7.4 Multi-Head Attention, Positional Encoding, and Add & Norm — Core Components of the Transformer Block
• 4. Multilayer Perceptron (MLP) — Fundamental Neural Network Structure and Learning Principles
• 2.8 Fundamentals of Neural Networks

• 5.2 Background and Design Principles Behind CNNs
• 4.1 Multilayer Perceptron (MLP) — Intuition and Core Components

Recommended next reading (5/17)

+5

• 7.9 Modern Transformer Blocks in Large Language Models — Key Changes in 2024-era Transformer Architecture
• 7.1 Transformer Architecture and Core Components — Sequence-to-Sequence Models and Encoder–Decoder Structure
• 5.3 Understanding the Core Components and Structure of Convolutional Neural Networks
• Transformer Block — Why It Became the Standard Architecture After Replacing RNNs
• Hybrid Attention — Efficient Attention for Long-Context Transformers

• Sigmoid Function — A Classic Nonlinear Curve That Squeezes Outputs into Probabilities
• MoE Transformer — Why the FFN Is Split into Experts
• Stack of Decoders — Why Multiple Decoder Layers Improve Transformer Understanding
• Switch Transformer — How MoE Simplifies Scaling with Top-1 Routing
• Linformer — Reducing Attention Cost with Low-Rank Attention

• Softmax-Free Attention — How Attention Can Find Important Information Without Softmax
• Reformer — Scaling Attention to Longer Sequences
• Causal Local Attention — Reducing the Cost of Long-Sequence Generation
• Multiplicative vs Additive Mask — Why Masking Before Softmax Matters
• GShard — Scaling MoE Transformers with Automatic Sharding
• Activation Function — The nonlinearity that boosts a neural network’s expressive power
• Sliding Window Attention — Why Full Attention Becomes Inefficient in Long Contexts

Posts on the same topic (8/8)

• Attention Collapse — Why LLMs Overfocus on a Small Set of Tokens
• Convolution Kernels — How CNNs Capture Local Patterns
• Model Expressivity — How Neural Networks Represent Complex Functions,
• Quadratic Complexity — Why Attention Runs Into an n² Bottleneck
• Restricted Neural Network Model — How Structural Constraints Shape Neural Network Learning
• Routing — How AI Models Choose a Computation Path for Each Input
• Strided Attention — Sparse Connectivity for Efficient Long-Context Processing
• Top-K Sparse Routing — How MoE Scales AI Models by Reducing Computation

Related concepts (2/2)

• BigBird — Sparse Attention Design for Long-Context Processing
• Longformer — Why Sparse Attention Matters for Long-Context Processing

📍 Where this concept fits in the AI learning map

See where this concept sits within the full AI Universe.

📍 Current position in AI Universe

☰

Reset Show completed · Login required Loading…

🌌 AI Universe

‹

›

⭐ Concept

Select a star.

View the full AI Universe

« 6.8 After Transformers a…|7.1 Transformer Architec… »

🔖 Tags: Attention Mechanism · Deep Learning · Large Language Model · LLM Architecture · self-attention · Transformer