6.4 RNN Language Modeling — Sequential Models That Learn Sentence Probability and Next-Token Prediction

This lesson explains how Language Modeling can be performed with RNNs. Rather than focusing only on the RNN architecture itself, we will connect the main ideas used in language modeling: Sentence Probability, Next-Token Prediction, Tokenization, Cross-Entropy Loss, and Autoregressive Modeling.

04/27/2026

Autoregressive Modeling — Sequential token generation via conditional probabilities on past outputs

Autoregressive modeling is a probabilistic approach that generates a sequence one step at a time: it predicts the next value conditioned on previously generated (or observed) values, rather than producing the entire sequence in a single shot.

02/15/2026

Causal vs Non-causal Self-Attention — How Context Access Separates Generation from Understanding

Causal vs Non-causal Self-Attention refers to two ways of processing context depending on whether the model is allowed to look at future tokens. Self-attention updates each token representation by referencing other tokens in the sequence. In the causal setting, each token can attend only to past and current positions, while in the non-causal setting it can attend to the full sequence on both sides.

04/08/2026

Exposure Bias — Accumulated errors caused by a mismatch between training inputs and real generation conditions

Exposure bias is an error-accumulation problem in sequence generation: during training the model conditions on the ground-truth previous token, but during real generation it must condition on its own previous outputs.

02/15/2026

Language Modeling Objective — How Next-Token Prediction Trains LLMs to Generate Fluent Text

Language Modeling Objective is the training objective that teaches a model to maximize the probability of the ground-truth next token given the current context. Through this objective, a language model learns to assign higher probability to linguistic patterns and expressions that frequently appear in real data.

05/24/2026

Masked Self-Attention — Why Transformer Decoders Must Block Future Tokens

Masked Self-Attention is a Transformer Decoder mechanism that prevents the current token from attending to future tokens. This ensures that the model learns to predict the next token using only the tokens that have already appeared, rather than looking ahead at the correct answer.

05/23/2026

Neural Network Building Blocks — Predicting Sequential Probabilities with Weighted Sums and Activations

A neural network is not a machine that memorizes inputs as-is. It’s a model that repeatedly applies a weighted sum and an activation function to transform inputs into progressively more useful representations. When you combine this with autoregressive modeling, you get a basic recipe for generative models: predict the next output (or next token) probabilistically, conditioned on what came before.

02/03/2026