Compressed Latent KV Cache — Why Store KV in a Latent Form?

Compressed Latent KV Cache is a technique that, during Transformer inference, stores the Key-Value Cache as a smaller latent representation instead of keeping the original KV tensors in full form. Its goal is to reduce the memory footprint and memory-bandwidth bottleneck caused by KV Cache growth in long-context inference.

05/08/2026

FlashAttention IO-Awareness — Why Attention Bottlenecks on GPU Memory Movement

FlashAttention IO-Awareness is an optimization approach built on a key insight: in Transformer Attention, GPU memory movement (IO) can become a bigger bottleneck than raw computation (FLOPs). Conventional Attention repeatedly stores and reloads large Attention matrices and softmax results in GPU memory, while FlashAttention computes them directly in small blocks to reduce memory access.

05/22/2026

Multi-Head Latent Attention — Reducing the KV Cache with Shared Latents per Token

Multi-Head Latent Attention (MLA) is an Attention architecture designed to reduce the growing KV Cache cost in Large Language Model (LLM) inference. Instead of storing full Key/Value tensors for every head and every token, MLA stores a shared compressed KV latent per token. The core idea is to cache a compact representation that can reconstruct head-specific Key/Value representations when needed.

05/24/2026