☆ Save 7.1 Transformer Architecture and Core Components — Sequence-to-Sequence Models and Encoder–Decoder Structure

06/15/2026

This article explains how the Transformer is structured and why it has become a core neural architecture in natural language processing. Unlike sequential models that process tokens one by one, the Transformer processes the entire input simultaneously and computes relationships between tokens. Here we focus on the core components: Sequence-to-Sequence Model, Encoder–Decoder Architecture, Transformer Encoder, Transformer Decoder, Word Embedding, and Context Length.

Table of Contents

• Sequence-to-Sequence Model and Transformer
• RNN vs Seq2Seq vs Transformer
• Encoder–Decoder Architecture and Context Vector
• Transformer Encoder and Encoder Layer
• Word Embedding·Token·Vocabulary
• Transformer Decoder and Cross Attention
• Context Length and Parallel Processing

Sequence-to-Sequence Model and Transformer

A Transformer is a Sequence-to-Sequence Model that converts an input sequence into an output sequence. For example, it can take a sentence in one language and generate a sentence in another language. Both input and output consist of token sequences, and their lengths may differ.

This architecture is widely used in machine translation, text generation, and summarization. A key point is that the Transformer does not simply map words directly, but reconstructs meaning based on relationships across the entire sentence. The core mechanism enabling this is Self-Attention.

RNN vs Seq2Seq vs Transformer

RNN processes sequences step by step. It naturally captures order information by computing each state based on the previous one, but it is difficult to parallelize and may lose information in long sequences.

Traditional Seq2Seq models encode the input into a single representation and then generate the output from it. However, as input length increases, it becomes difficult to store all necessary information in a single vector.

The Transformer solves this problem by processing all tokens at once and directly computing relationships between them. For example, in “I love you”, the token “love” can simultaneously attend to both “I” and “you”. This enables parallelization in Transformers and provides stable performance even for long contexts.

Encoder–Decoder Architecture and Context Vector

The Encoder–Decoder Architecture separates understanding the input from generating the output. The Encoder converts the input sequence into context-aware representations, while the Decoder generates the output sequence based on them.

The Encoder output is not a single vector but a set of vectors, one per token, each enriched with contextual information. Collectively, this can be viewed as a Context Vector representation of the input.

The Decoder uses this Context Vector to determine which parts of the input are most relevant when generating each token. This process leads to Encoder–Decoder Attention, also known as Cross Attention.

Transformer Encoder and Encoder Layer

The Transformer Encoder is a stack of identical encoder layers. Each Encoder Layer (Transformer Block) consists of two main components:

• Self-Attention: models relationships between tokens in the sequence
• Position-wise Feed-Forward Network (FFN): transforms each token representation independently

Self-Attention is the core of the Transformer. It allows the model to consider relationships between all words when interpreting a specific token. For example, it plays a key role in resolving what pronouns refer to.

The FFN non-linearly transforms the representations produced by Self-Attention, strengthening the learned features. The same network is applied independently to each position, forming a form of Parameter Sharing.

Word Embedding, Token, Vocabulary

Transformers do not process raw text directly. Instead, text is first split into tokens. A Token can be a word or a subword unit. The total set of tokens is called the Vocabulary.

Each token is converted into a vector using Word Embedding. This produces a Dense Vector Representation that encodes semantic meaning numerically.

These representations capture semantic relationships between words, connecting to concepts such as Vector Space Semantics and Semantic Similarity. Common embedding methods include Word2Vec, GloVe, FastText, and ELMo.

In Transformers, these embeddings are used as input to the Encoder, and their dimensionality is called the Embedding Dimension.

Transformer Decoder and Cross Attention

The Transformer Decoder is responsible for generating the output sequence. It consists of three main components:

• Masked Self-Attention: prevents access to future tokens
• Encoder–Decoder Attention: uses input sequence information
• Feed-Forward Network: non-linear transformation

Masked Self-Attention uses Causal Masking to block future information, enabling Autoregressive Generation. The model generates each token based only on previously generated tokens.

Encoder–Decoder Attention determines which parts of the input sequence should be attended to during generation. This allows the model to learn alignment between input and output sequences.

Context Length and Parallel Processing

Context Length refers to the maximum number of tokens a model can process at once. A longer context length allows the model to consider more information simultaneously, which is important for Long-Context Modeling.

Transformers can process tokens in parallel, making them highly efficient for training compared to RNNs. This provides significant advantages in speed and scalability.

However, allowing every token to attend to every other token increases computational cost. This trade-off is fundamental to understanding Transformer efficiency and scalability.

※ This article was independently compiled and adapted from lectures by Professor Sungroh Yoon at Seoul National University.

Recommended prerequisite reading (2/2)

• Context Bottleneck — Why Long Seq2Seq Inputs Lose Important Information
• Encoder–Decoder Architecture — A Neural Architecture for Sequence Transduction

Recommended next reading (5/20)

+5

• 7.9 Modern Transformer Blocks in Large Language Models — Key Changes in 2024-era Transformer Architecture
• 5.3 Understanding the Core Components and Structure of Convolutional Neural Networks
• 7.4 Multi-Head Attention, Positional Encoding, and Add & Norm — Core Components of the Transformer Block
• 4. Multilayer Perceptron (MLP) — Fundamental Neural Network Structure and Learning Principles
• 7. Transformer — From Self-Attention to Modern LLM Architectures

• 5.2 Background and Design Principles Behind CNNs
• Transformer Block — Why It Became the Standard Architecture After Replacing RNNs
• Stack of Decoders — Why Multiple Decoder Layers Improve Transformer Understanding
• CNN Parameter Count — A Core Metric That Determines Model Complexity and Representational Capacity
• Switch Transformer — How MoE Simplifies Scaling with Top-1 Routing

• Top-K Sparse Routing — How MoE Scales AI Models by Reducing Computation
• GShard — Scaling MoE Transformers with Automatic Sharding
• Sigmoid Function — A Classic Nonlinear Curve That Squeezes Outputs into Probabilities
• Inception Module — A CNN architecture that learns multi-scale spatial features simultaneously by using diverse receptive fields in parallel
• Hybrid Attention — Efficient Attention for Long-Context Transformers
• Routing — How AI Models Choose a Computation Path for Each Input
• MoE Transformer — Why the FFN Is Split into Experts
• Sparse Interactions — A Structural Principle That Preserves Only Necessary Connections for Computational Efficiency and Effective Representation Learning
• Pooling — Downsampling that summarizes features, reduces resolution, and improves shift robustness
• Skip Connection (Residual Connection) — A Shortcut Path That Preserves Information and Gradients in Deep Neural Networks

Posts on the same topic (8/12)

+4

• Attention Collapse — Why LLMs Overfocus on a Small Set of Tokens
• Causal Local Attention — Reducing the Cost of Long-Sequence Generation
• Convolution Kernels — How CNNs Capture Local Patterns
• Linformer — Reducing Attention Cost with Low-Rank Attention
• Model Expressivity — How Neural Networks Represent Complex Functions,
• Multiplicative vs Additive Mask — Why Masking Before Softmax Matters
• Quadratic Complexity — Why Attention Runs Into an n² Bottleneck
• Reformer — Scaling Attention to Longer Sequences

• Restricted Neural Network Model — How Structural Constraints Shape Neural Network Learning
• Sliding Window Attention — Why Full Attention Becomes Inefficient in Long Contexts
• Softmax-Free Attention — How Attention Can Find Important Information Without Softmax
• Strided Attention — Sparse Connectivity for Efficient Long-Context Processing

Related concepts (0/0)

No related concept posts yet.

📍 Where this concept fits in the AI learning map

See where this concept sits within the full AI Universe.

📍 Current position in AI Universe

☰

Reset Show completed · Login required Loading…

🌌 AI Universe

‹

›

⭐ Concept

Select a star.

View the full AI Universe

« 7. Transformer — From Se…|7.2 Attention and Langua… »

🔖 Tags: encoder-decoder architecture · Sequence-to-Sequence Model · transformer architecture · Transformer Decoder · transformer encoder · Word Embedding