Transformers & Attention: A Deep Dive
Build correct intuition for attention, transformer blocks, decoding, KV caching, and scaling laws.
advanced · 55 minutes
A rigorous but intuition-first tour of the architecture behind modern language models. We unpack self-attention, multi-head attention, and the full transformer block, then trace how decoder-only models generate text token by token. Along the way we confront the quadratic cost of attention, the role of the KV cache, and what scaling laws really promise.
What you'll learn
- Tokens, Embeddings, and Why Order Must Be Injected
- Self-Attention: Queries, Keys, and Values
- Multi-Head Attention and the Transformer Block
- Encoders, Decoders, and Causal Masking
- The Cost of Attention: Quadratic Scaling and Context Length
- Autoregressive Decoding, the KV Cache, and Scaling Laws