6. Deep Learning and Generative AI Systems: Concepts, Architectures, and Model Landscape
This lesson connects deep learning with generative AI systems, organizing the story from basic concepts to composite architectures, generative models, large language models, and multimodal AI so that the overall landscape can be understood as one coherent picture.
10/29/2025
7. Transformer — From Self-Attention to Modern LLM Architectures
This article explains how the Transformer overcomes the limitations of traditional sequential models and how it evolved from Self-Attention to modern Large Language Model (LLM) architectures. Unlike sequential processing models, Transformer processes the entire input at once and computes relationships between tokens in parallel. This property makes Transformer the core neural architecture behind modern NLP, code generation, multimodal systems, and large-scale language models.
06/15/2026
Attention Dilution — Why Important Information Gets Lost in Long Contexts
Attention dilution is a long-context failure mode in Transformer models: as the input grows, the model has to weigh more competing information, and details that matter can end up exerting less influence on the final output. Self-Attention gives a Transformer access to relationships across the entire sequence, but seeing more tokens does not mean using every token equally well. This distinction is especially important for Large Language Models (LLMs) with long context windows, where a key instruction or supporting fact may be present in the prompt but still fail to meaningfully affect the model’s response.
08/29/2026
Context Drift — Why Key Constraints Fade in Long Contexts
Context Drift is a phenomenon where a model processing long inputs or extended conversations gradually shifts away from the original intent. Instead of consistently preserving the initial goals, constraints, and evaluation criteria, the model becomes increasingly influenced by later information, repeated patterns, or recently introduced context. The important point is not that the model completely forgets the original instructions. Rather, as the context grows longer, the relative importance of different signals is continuously recalculated, causing the initial requirements to lose influence and the model’s interpretation to slowly move in another direction.
08/02/2026
Conversational AI — A Dialogue System That Understands, Remembers, and Responds in Natural Language
Conversational AI is an AI system that interacts with people in natural language while taking into account user intent, conversational context, and task goals to produce appropriate responses. Early rule-based chatbots mostly selected from predefined phrases, but modern Conversational AI has evolved into a much richer architecture that combines understanding·memory·decision-making·and generation.
04/08/2026
Decoder Layer — Why LLMs Predict the Next Token from Previous Context
A Decoder Layer is a core component of the Transformer Architecture that continually refines contextual representations so a model can predict the next token from the tokens that came before it. GPT-style Large Language Models (LLMs) stack many Decoder Layers, allowing relationships among earlier tokens to be processed and refined across depth. The final representation is then used to produce a probability distribution over the next token. This autoregressive design is the foundation of generative AI applications such as Text Generation, Code Generation, Chatbots, and AI Agents.
08/31/2026
Decoder-only Transformer — How Next-Token Generation Works
Decoder-only Transformer is an autoregressive generation architecture that predicts the next token step by step based on previously generated tokens. Instead of analyzing the entire sequence bidirectionally, it calculates the probability of the next token using only the available past context. Each generated token is then added back to the sequence, and this process repeats to create ordered outputs such as text, code, and conversations.
08/02/2026
Error Compounding — How Small Mistakes Derail Long-Context LLM Reasoning
Error Compounding is what happens when an autoregressive model reuses its own previous outputs as context, allowing small prediction errors to influence everything that follows. This is a fundamental limitation in sequential AI systems such as Large Language Models (LLMs), Sequence-to-Sequence Models, and Reinforcement Learning policy models, where each new state is built on earlier ones. The risk becomes more pronounced in long-context generation because an early mistake can reshape the states that follow, gradually pushing both the context and the reasoning path away from the intended direction.
08/29/2026
Hallucination — When a generative model produces plausible output without grounding
Hallucination is the phenomenon where a generative model—especially a language model or a multimodal model—produces statements that are not supported by the input or any provided evidence, yet presents them as if they were factual. The model does this without explicitly “knowing” it is wrong, selecting outputs that look most plausible.
10/29/2025
Inference Bottleneck — Why LLM Inference Is Hard to Parallelize
An inference bottleneck arises from a basic constraint of large language model inference: generation is sequential. An LLM cannot produce an entire output sequence in one pass because each new token depends on the tokens generated before it. That dependency limits how much additional GPU compute can accelerate decoding, so higher raw compute performance does not translate directly into proportional improvements in latency or throughput. It also explains why KV cache efficiency, memory bandwidth, and decoding strategy matter so much in practical LLM inference.
08/29/2026