Absolute vs Relative vs Rotary Positional Encoding — How Attention Understands Position

Absolute Positional Encoding (APE), Relative Positional Encoding (RPE), and Rotary Positional Embedding (RoPE) are three ways to inject token order and distance information into attention in a Transformer. The key difference is where the position signal enters the computation: added to the input embedding, added to the attention score, or encoded by rotating the query and key vectors themselves.

05/06/2026

Rotary Positional Embedding — Why Relative Position Is Represented as Rotation

Rotary Positional Embedding (RoPE) is a way to represent positional information in a Transformer through vector rotation. Instead of simply adding a positional vector to the token embedding, RoPE rotates the Query and Key vectors according to their positions, so the attention score reflects not only semantic similarity but also the relative position difference between two tokens. The key idea is that position is not attached as extra information; it is built directly into the geometry of the vectors used in attention.

05/06/2026

Sinusoidal Positional Encoding — How Sine and Cosine Represent Relative Position

Sinusoidal Positional Encoding gives a Transformer a way to distinguish token order even though attention itself has no built-in concept of sequence position. Each position is converted into a fixed vector of sine and cosine values, and that vector is added directly to the token embedding. Because the positional vectors come from predefined functions rather than learned parameters, this approach is also called Fixed Positional Encoding. Instead of treating position as a simple integer, it turns position into a structured signal that can flow through the same attention machinery as token representations.

08/22/2026