Attention in transformers, step-by-step | Deep Learning Chapter 6
3Blue1Brown
AI-assisted summary of Attention in transformers, step-by-step | Deep Learning Chapter 6 — original source. Check the original for full context.
Source published:
Key ideas
- Self-attention transforms static token embeddings into context-dependent representations by projecting inputs into learned query, key, and value spaces to calculate directed vector updates. (0:00)
- To prevent future tokens from leaking into past predictions during autoregressive training and inference, attention scores are masked with negative infinity before applying column-wise softmax. (11:08)
- To avoid a 150-million-parameter full square mapping per head in architectures like GPT-3, value transformations are factored into low-rank down- and up-projections through a 128-dimensional bottleneck. (15:44)
- While attention mechanisms command the conceptual spotlight, they account for under 58 billion of GPT-3's 175 billion total parameters, with the majority residing in intermediate feed-forward MLP layers. (23:19)
Detailed notes
## Background Published by Grant Sanderson on the YouTube channel 3Blue1Brown on April 7, 2024, this video is the sixth installment in his Deep Learning series. According to the page description, the lesson aims to demystify the attention mechanism—the central innovation powering transformers and modern large language models—by covering self-attention, multi-headed attention, and cross-attention step-by-step.
## TL;DR The video breaks down how transformers use attention to transform static, isolated token embeddings into rich, context-aware representations necessary for predicting the next word. Sanderson explains the linear algebra of queries, keys, dot-product relevance scores, and masking, showing how column-wise softmax produces an attention pattern grid that governs information transfer. He details how value vectors are weighted and added back into token embeddings via a low-rank factorization, which is combined with multi-headed attention across dozens of parallel heads and deep layers. Finally, he connects these theoretical matrix operations to real-world architectures like GPT-3, highlighting that modern transformer performance relies as much on GPU-friendly parallelizability as it does on conceptual expressiveness.
## Atmosphere & style The presentation carries 3Blue1Brown's signature calm, methodical, and intellectually patient rhythm, set against the gentle background music of Vincent Rubinetti. Sanderson guides the viewer through complex linear algebra with intuitive, visual metaphors—like clarifying polysemic words or tracing adjective-noun modifications—before diving into the dense formal equations. Watching the lesson feels rigorous yet deeply reassuring, transforming an intimidating technical bottleneck of deep learning into a lucid, elegant geometric journey.
## Bulletin - [00:00](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=0s) Self-attention transforms static token embeddings into context-dependent representations by projecting inputs into learned query, key, and value spaces to calculate directed vector updates. - [11:08](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=668s) To prevent future tokens from leaking into past predictions during autoregressive training and inference, attention scores are masked with negative infinity before applying column-wise softmax. - [15:44](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=944s) To avoid a 150-million-parameter full square mapping per head in architectures like GPT-3, value transformations are factored into low-rank down- and up-projections through a 128-dimensional bottleneck. - [23:19](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1399s) While attention mechanisms command the conceptual spotlight, they account for under 58 billion of GPT-3's 175 billion total parameters, with the majority residing in intermediate feed-forward MLP layers.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=0s) The core function of a transformer is to progressively adjust context-free token embeddings into rich representations where geometric directions capture contextual nuances. - [01:39](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=99s) Polysemous tokens such as 'mole' require attention blocks to calculate specific displacement vectors that push generic embeddings toward domain-specific meanings like biology, chemistry, or medicine. - [01:39](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=99s) Because generative models predict the next word exclusively from the final token position, that last vector must absorb all relevant predictive information across the entire sequence. - [04:29](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=269s) Query and key weight matrices project high-dimensional embeddings into a compact latent space (e.g., 128 dimensions) to calculate pairwise relevance via dot products. - [07:47](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=467s) Applying a softmax function down each column of the raw dot-product grid normalizes relevance scores into a valid probability distribution that sums to one. - [10:00](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=600s) Scaling dot products by the square root of the head dimension ensures numerical stability and healthier gradient behavior during training. - [11:08](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=668s) Causal masking replaces entries for future tokens with negative infinity prior to softmax, allowing a single forward pass to train on every prefix subsequence without data leakage. - [11:08](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=668s) Because full attention grids compare every token against every other token, compute and memory scale quadratically (O(N^2)) with context length. - [13:10](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=790s) Value vectors determine what semantic displacement is extracted from relevant context tokens, which is then weighted by attention scores and added back into the target embedding. - [15:44](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=944s) Factoring the value transformation into low-rank down and up projections constrains rank and balances parameter counts across query, key, and value operations. - [18:21](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1101s) Cross-attention differs from self-attention by deriving keys and values from a source sequence while queries originate from a target sequence, typically omitting causal masking. - [19:49](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1189s) Multi-headed attention runs dozens of heads in parallel (such as 96 heads per block in GPT-3), giving the model capacity to track multiple overlapping contextual relationships at once. - [22:16](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1336s) Standard transformer implementations package individual head up-projections into a single consolidated output matrix per attention block. - [23:19](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1399s) Alternating attention blocks with multi-layer perceptrons across deep layers enables models to develop abstract representations of tone, style, and domain knowledge. - [24:54](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1494s) Transformers owe much of their real-world dominance to extreme hardware parallelizability on modern GPUs rather than purely expressive superiority.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=0s) Recap on Embeddings and Semantic Space
The transformer architecture serves as the core technological engine powering large language models and modern artificial intelligence, having first been introduced to the world in the landmark 2017 paper *Attention is All You Need*. At its foundation, the fundamental objective of the model is deceptively simple: ingest a sequence of text and predict the word that comes next. To process language numerically, the raw input text is split into chunks known as tokens. While real-world tokenizers frequently break text down into sub-word fragments, it is conceptually cleaner to treat each individual word as a single token.
Once tokenized, the initial step assigns each token to a vector in a high-dimensional embedding space. The essential geometric intuition behind this space is that directions correspond directly to semantic meaning. Adding a specific displacement vector to the embedding of a masculine noun, for instance, shifts the coordinates to land on the embedding of its corresponding feminine equivalent. Countless other directions across thousands of dimensions simultaneously encode diverse grammatical, conceptual, and relational nuances. Rather than leaving words frozen in their initial static positions, the broader network is engineered to progressively transform them. As Grant Sanderson emphasizes: “The aim of a transformer is to progressively adjust these embeddings so that they don't merely encode an individual word, but instead they bake in some much, much richer contextual meaning.”
- **Foundation:** Transformers entered machine learning in the 2017 research paper *Attention is All You Need*. - **Tokenization:** Inputs are parsed into tokens (modeled conceptually here as words). - **Embedding Space:** High-dimensional vector space where specific directional steps alter semantic attributes, such as shifting gender from masculine to feminine nouns. - **Primary Objective:** Gradually refine generic token vectors into rich, context-laden representations.
[01:39](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=99s) Motivating Examples: Why Context Matters
Because the mathematical mechanics of attention can be notoriously difficult to grasp on a first pass, intuitive behavioral examples help ground the matrix multiplications before diving into the linear algebra. A classic polysemy problem appears when comparing the phrases *"American shrew mole"*, *"one mole of carbon dioxide"*, and *"take a biopsy of the mole"*. In a standard lookup table, the initial token embedding for the word *"mole"* is identical across all three sentences because the initial embedding layer possesses zero awareness of context. A well-trained attention block must calculate the exact vector delta needed to push that generic starting embedding toward one of several distinct contextual directions—whether that means an animal, a unit of chemistry, or a dermatological lesion.
```mermaid flowchart LR accTitle: Attention Contextualization Flow accDescr: Diagram showing how a generic token embedding is updated by surrounding context vectors to produce a specialized embedding. subgraph InputTokens["Input Embeddings"] direction TB E1["'American' / 'shrew'"] E2["'mole' (Generic)"] end subgraph AttentionBlock["Attention Mechanism"] AB["Calculates vector updates from surrounding context"] end subgraph OutputEmbedding["Refined Representation"] R2["'mole' (Zoological meaning)"] end InputTokens --> AB AB --> OutputEmbedding ```
Similarly, consider the word *"tower"*. In isolation, its embedding sits in a non-specific direction associated generally with tall, large structures. If it is preceded by *"Eiffel"*, the attention mechanism should pull its vector toward Paris, France, and steel construction; if the modifier *"miniature"* is added beforehand, the vector must be steered away from large physical scales entirely. On a broader scale, attention allows information to travel across vast distances within a document. Because the final token prediction depends exclusively on the very last vector of the sequence, that single vector must absorb everything that came before it. Describing this bottleneck, Sanderson observes: “If the model is going to accurately predict the next word, that final vector in the sequence, which began its life simply embedding the word was, will have to have been updated by all of the attention blocks to represent much, much more than any individual word, somehow encoding all of the information from the full context window that's relevant to predicting the next word.”
- **The Mole Dilemma:** A static embedding table assigns the same vector to *"mole"* in biology, chemistry, and medicine; attention calculates the context-driven update vector. - **The Tower Example:** Successive adjectives modify semantic associations (e.g., adding *"Eiffel"* introduces steel and Paris, while adding *"miniature"* strips away large scale). - **The Sequence Bottleneck:** In a mystery novel ending with *"therefore the murderer was"*, the vector for *"was"* must absorb relevant clues across the entire text to forecast the killer's name.
[04:29](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=269s) The Attention Pattern: Queries, Keys, and Dot Products
To inspect the underlying linear algebra, consider the sample sentence: *"a fluffy blue creature roamed the verdant forest"*. In a single attention head, one can picture a targeted behavioral pattern, such as nouns pulling descriptive content from preceding adjectives. Each word begins as an embedding vector, denoted as $e$, which encapsulates both the identity of the token and its precise sequential position within the prompt. Deep learning structures these contextual interactions around matrix-vector multiplications filled with trainable weights that adjust over time through backpropagation. Sanderson notes the pedagogical nature of this framing: “To be clear, I'm making up this example of adjectives updating nouns just to illustrate the type of behavior that you could imagine an attention head doing.”
```mermaid flowchart TD accTitle: Query-Key Interaction in Self-Attention accDescr: Embeddings are projected by query and key matrices into a lower-dimensional space, where dot products compute alignment scores. subgraph Projections["Projection into Lower-Dimensional Space"] direction LR E["Embedding (e)"] -->|"W_Q"| Q["Query (q) [dim: 128]"] E -->|"W_K"| K["Key (k) [dim: 128]"] end subgraph Matching["Relevance Scoring"] direction TB Q & K --> DP["Dot Product (q · k)"] DP --> Score["Raw Attention Score (-∞ to +∞)"] end ```
The communication process starts when a noun like *"creature"* asks an informational question: *"Are there any adjectives sitting in front of me?"* This query is represented as a lower-dimensional vector $q$ (for instance, 128 dimensions), computed by multiplying the token embedding $e$ by a learned query weight matrix $W_Q$. Concurrently, every token embedding is multiplied by a separate learned key weight matrix $W_K$ to produce a key vector $k$ in that same 128-dimensional space. The keys serve as potential answers to the queries, broadcasting attributes such as *"I am an adjective located in an earlier position."* When the query matrix projects nouns into directions seeking preceding adjectives, and the key matrix projects words like *"fluffy"* and *"blue"* into directions matching that request, the pairs align.
- **Notation:** Token embeddings are designated as $e$, query vectors as $q$, and key vectors as $k$. - **Positional Encoding:** Embedding vectors $e$ contain information detailing both the token's meaning and its position in the context window. - **Dimensionality:** Query and key vectors reside in a lower-dimensional space (such as 128 dimensions) compared to the raw embedding space. - **Weight Matrices:** $W_Q$ maps embeddings to queries, while $W_K$ maps embeddings to keys.
[07:47](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=467s) Keys, Alignment Scores, and Column-Wise Softmax
To quantify how well each key satisfies a given query, the model computes the dot product between every possible key-query pair across the prompt. Visualized as an attention grid, large positive dot products signal strong semantic alignment, whereas near-zero or negative values indicate irrelevance. When the keys generated by *"fluffy"* and *"blue"* align closely with the query generated by *"creature"*, the resulting dot products are large positive numbers. In machine learning terminology, practitioners describe this relationship by saying that the embeddings of *"fluffy"* and *"blue"* attend to the embedding of *"creature"*.
Because unconstrained dot products span any real value from negative infinity to positive infinity, they cannot yet serve as interpolation weights. The ultimate goal of this attention grid is to compute a weighted sum of contextual information down each column. To make these weights behaves like valid proportions, each column must contain strictly positive values between 0 and 1 that sum up to 1. To achieve this normalization, the attention mechanism applies the softmax function down every individual column of the grid, turning the raw alignment scores into a well-behaved probability distribution over the context tokens.
- **Relevance Measurement:** Dot products ($q \cdot k$) measure directional alignment between query vectors and key vectors. - **Attention Terminology:** Tokens yielding high dot products are said to "attend to" the querying token. - **Range of Raw Scores:** Dot products initially range across $(-\infty, \infty)$. - **Column Normalization:** Softmax is calculated along each column to ensure all weights fall strictly between 0 and 1 and sum to 1.
[10:00](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=600s) The Attention Pattern and the Transformer Formula
Once softmax is applied across all columns of the dot-product grid, the matrix fills with normalized weights that sum to one down each column. At this stage, the grid functions as an attention pattern: each column specifies how relevant every prior token on the vertical axis is to the token positioned at the top of that column.
```mermaid flowchart TD accTitle: Attention Pattern Calculation accDescr: Pipeline showing Query and Key vectors combined into an attention grid, scaled, masked, and normalized via softmax.
subgraph Inputs["1. Vector Inputs"] Q["Query Vectors (Q)"] K["Key Vectors (K)"] end
subgraph Scoring["2. Raw Dot Products"] Dot["Grid of Dot Products<br/>Q · K^T"] Scale["Scale by 1 / sqrt(d_k)"] end
subgraph PostProcessing["3. Normalization"] Mask["Mask Upper Triangle<br/>(set future to -inf)"] Softmax["Softmax (Column-by-Column)"] Pattern["Attention Pattern Grid"] end
Q --> Dot K --> Dot Dot --> Scale Scale --> Mask Mask --> Softmax Softmax --> Pattern ```
In the original transformer literature (*Attention Is All You Need*), this entire sequence of operations is condensed into a single algebraic expression:
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right)V$$
The symbols $Q$ and $K$ denote the full matrices of query and key vectors obtained by multiplying input embeddings by the respective query and key weight matrices. The numerator $Q K^T$ generates the full grid of pairwise dot products. For numerical stability during gradient calculations, each dot product is divided by $\sqrt{d_k}$, where $d_k$ represents the dimension of the smaller key-query space. Finally, the softmax operator is evaluated on a column-by-column basis so that each column forms a valid probability distribution.
---
[11:08](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=668s) Masking and the Context Size Bottleneck
During training, large language models do not merely predict the final token of a training sequence; they simultaneously predict the target next word following every initial subsequence in the passage. For an input passage like *"the fluffy blue creature"*, the model evaluates predictions after *"the"*, after *"the fluffy"*, and after *"the fluffy blue"* all in a single forward pass. Grant Sanderson emphasizes why this architecture is foundational to modern language modeling: “This is really nice, because it means what would otherwise be a single training example effectively acts as many.”
This training efficiency creates an immediate requirement: later tokens must never be permitted to broadcast information backward to earlier tokens, because reading ahead would allow the model to cheat on its next-token prediction task. To prevent future tokens from influencing past ones, the upper-right entries of the attention grid must evaluate strictly to zero. Simply overriding those slots with zeros after the fact fails because the remaining values in the column would no longer sum to one. To resolve this:
- Every entry corresponding to a future token is set to $-\infty$ prior to normalization. - Applying $\text{softmax}(z_i) = \frac{e^{z_i}}{\sum e^{z_j}}$ converts every $-\infty$ into $e^{-\infty} = 0$. - The remaining, valid historical entries renormalize cleanly to sum to $1.0$.
This technique is known as **masking**. While masking is strictly mandatory during training to enforce causal order, GPT-style models maintain this causal mask during inference as well. Because the resulting attention grid compares every token against every other token, its dimensions are $N \times N$, scaling quadratically ($O(N^2)$) with the context window length $N$. Consequently, context length constitutes one of the steepest computational bottlenecks in modern deep learning architectures.
---
[13:10](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=790s) Value Vectors and Updating Embeddings
Once the attention pattern establishes how tokens relate to one another, the architecture needs an actual mechanism to transfer semantic content between them. If the model determines that *"fluffy"* is highly relevant to *"creature"*, the embedding for *"creature"* must shift within the 12,288-dimensional embedding space to encode the specific concept of a fluffy creature.
```mermaid flowchart LR accTitle: Value Vector Weighting and Residual Addition accDescr: Sequence showing token embeddings transformed by the Value matrix, weighted by attention weights, and added back into the target embedding.
subgraph Transform["1. Value Projection"] E["Token Embeddings<br/>('Fluffy', 'Blue')"] Wv["Value Matrix (W_V)"] V["Value Vectors (V)"] end
subgraph Accumulate["2. Weighted Sum"] W["Attention Weights (Col)"] Delta["Delta e (Change Vector)"] end
subgraph Residual["3. Embedding Update"] Orig["Context-Free Embedding<br/>('Creature')"] Out["Refined Embedding<br/>('Fluffy Blue Creature')"] end
E --> Wv --> V V --> Delta W --> Delta Delta --> Out Orig --> Out ```
To achieve this, the model employs a third learned matrix: the **value matrix** ($W_V$). When multiplied by the embedding of an antecedent token (such as *"fluffy"*), the value matrix produces a **value vector**. Grant Sanderson characterizes the conceptual purpose of this operation: “When you multiply this value matrix by the embedding of a word, you might think of it as saying, if this word is relevant to adjusting the meaning of something else, what exactly should be added to the embedding of that something else in order to reflect this?”
These value vectors reside in the identical high-dimensional space as the original embeddings. The updating sequence proceeds systematically across every column: 1. The antecedent tokens produce individual value vectors via the value transformation. 2. In the target token's column (e.g., *"creature"*), each antecedent's value vector is scaled by the corresponding attention weight calculated earlier. High-attention tokens like *"fluffy"* and *"blue"* retain large proportions, while irrelevant tokens receive near-zero weights. 3. The scaled vectors are summed together into a single composite update vector, denoted as $\Delta e$. 4. This displacement $\Delta e$ is added directly to the original context-free embedding of the word.
Executing this weighted summation across all columns yields a completely updated sequence of context-rich vectors across the entire attention block.
---
[15:44](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=944s) Counting Parameters and Low-Rank Value Factorization
A single self-attention head is parameterized by its key, query, and value transformations. Tracking parameter counts using GPT-3's concrete architecture reveals an asymmetry that motivates low-rank factorization:
| Transformation Matrix | Input Dimension | Output Dimension | Total Parameters | |:--- |:--- |:--- |:--- | | **Query Matrix ($W_Q$)** | $12,288$ (Embedding dim) | $128$ (Head dim) | $\approx 1.57 \times 10^6$ | | **Key Matrix ($W_K$)** | $12,288$ (Embedding dim) | $128$ (Head dim) | $\approx 1.57 \times 10^6$ | | **Value Matrix (Naive Square)** | $12,288$ (Embedding dim) | $12,288$ (Embedding dim) | $\approx 150.99 \times 10^6$ | | **Value-Down Matrix ($W_{V,\text{down}}$)** | $12,288$ (Embedding dim) | $128$ (Head dim) | $\approx 1.57 \times 10^6$ | | **Value-Up Matrix ($W_{V,\text{up}}$)** | $128$ (Head dim) | $12,288$ (Embedding dim) | $\approx 1.57 \times 10^6$ |
If the value map were implemented directly as a full square matrix mapping the 12,288-dimensional space back onto itself, that single transformation would demand over 150 million parameters—vastly outnumbering the key and query maps. As Sanderson notes, “You could devote orders of magnitude more parameters to the value map than to the key and query. But in practice, it is much more efficient if instead you make it so that the number of parameters devoted to this value map is the same as the number devoted to the key and the query.”
In practice, the value operation is factored into the product of two rectangular low-rank matrices: - **Value-Down Matrix:** Maps the 12,288-dimensional token embedding down into a compact 128-dimensional latent channel. - **Value-Up Matrix:** Projects that 128-dimensional vector back up into the full 12,288-dimensional embedding space to form the additive displacement $\Delta e$.
In linear algebra terms, this factorization constrains the value transformation to have a mathematical rank of at most 128. By structuring the value operator as two matrices of size $128 \times 12,288$, each of the four components ($W_Q, W_K, W_{V,\text{down}}, W_{V,\text{up}}$) contains approximately 1.57 million parameters, totaling roughly 6.3 million parameters for one complete attention head.
---
[18:21](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1101s) Cross-Attention vs. Self-Attention
The attention mechanism described up to this point is specifically **self-attention**, because queries, keys, and values are all drawn from the same input sequence. This contrasts with **cross-attention**, an architecture used when models bridge two distinct streams of data, such as machine translation (pairing a source sentence in one language with a target sentence in another) or speech recognition (pairing raw audio features with output text).
In a cross-attention setup: - The keys and values typically derive from the source modality (for example, the original English text). - The queries derive from the target sequence being generated (for example, the ongoing German translation). - The resulting attention pattern aligns tokens across languages, determining which source terms inform the next translated word. - Unlike autoregressive self-attention, cross-attention generally omits causal masking, because the complete source sentence is already fully available to every query step.
Returning to self-attention in generative language models, a single attention head captures only one specific semantic relationship at a time—such as an adjective modifying a subsequent noun. Natural language exhibits countless overlapping semantic dynamics: preceding phrases like *"they crashed the"* fundamentally alter physical assumptions about a subsequent word like *"car"*. Modeling the full richness of human language requires running many attention heads simultaneously in parallel.
[19:49](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1189s) Multi-Headed Attention
The real strength of the attention mechanism emerges when the network must disentangle complex, ambiguous contextual clues. If the word "wizard" appears anywhere in the same passage as "Harry," the embedding should naturally shift toward Harry Potter. Conversely, if words such as "Queen," "Sussex," and "William" appear in that passage, the embedding for "Harry" ought to be updated to represent Prince Harry. Each distinct type of contextual update requires its own specialized set of key and query matrices to capture the relevant attention pattern, paired with a dedicated value map that decides exactly what semantic shift should be added to the embedding. While real-world trained parameters are far harder to interpret than these clean conceptual vignettes, their weights are tuned to do whatever is necessary to minimize next-token prediction loss.
To execute all these contextual adjustments at once, transformers use multi-headed attention. Rather than relying on a single attention head, a full transformer attention block runs dozens of these operations in parallel, with each head maintaining its own distinct key, query, and value maps. In GPT-3, for instance, each block contains 96 attention heads operating simultaneously.
```mermaid flowchart TD accTitle: Multi-Headed Attention Architecture accDescr: Diagram showing how multiple attention heads compute proposed updates from token embeddings in parallel and sum them into a refined embedding.
subgraph Inputs["Input Sequence"] X["Original Token Embeddings"] end
subgraph Heads["Parallel Attention Heads"] H1["Head 1: Q1, K1, V1<br/>Attention Pattern 1"] H2["Head 2: Q2, K2, V2<br/>Attention Pattern 2"] HDots["..."] H96["Head 96: Q96, K96, V96<br/>Attention Pattern 96"] end
subgraph Aggregation["Update Combination"] Sum["Sum of Proposed Updates<br/>(Head 1 +... + Head 96)"] Out["Refined Token Embeddings"] end
X --> H1 X --> H2 X --> HDots X --> H96 H1 --> Sum H2 --> Sum HDots --> Sum H96 --> Sum X --> Out Sum --> Out ```
Grant Sanderson acknowledges how bewildering this scaling can feel when trying to picture it all at once, noting that “considering that each one is already a bit confusing, it's certainly a lot to hold in your head.” He emphasizes that the underlying objective remains straightforward: “The overall idea is that by running many distinct heads in parallel, you're giving the model the capacity to learn many distinct ways that context changes meaning.”
- GPT-3 utilizes 96 parallel attention heads within each attention block. - Each head computes its own independent attention pattern using its dedicated key and query matrices. - Each head produces a sequence of value vectors and outputs a proposed update vector for every token position. - The proposed changes from all 96 heads are summed together and added directly back to the original token embedding. - With 96 heads per block (each using four matrices across keys, queries, and factored values), each multi-headed attention block accounts for approximately 600 million parameters.
---
[22:16](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1336s) The Output Matrix Formulation
When looking at literature and open-source implementations, there is a minor mechanical difference from the conceptual view of two distinct value matrices per head. In earlier conceptual breakdowns, the value transformation was explained as two separate operators: a "value down" projection into a lower-dimensional space, followed by a "value up" projection back into the full embedding space. Although one could validly implement each attention head with its own self-contained pair of up and down matrices, practical models do not structure them that way.
Instead, machine learning papers and production code combine all the "value up" matrices across every single head into one unified projection known as the output matrix, which is tied to the entire multi-headed attention block. When researchers and practitioners casually talk about the "value matrix" of an individual attention head, they are almost always referring solely to the initial projection down into the low-rank subspace. Sanderson highlights this implementation detail so that viewers encountering official Transformer codebases will not get tripped up by the differing notation.
---
[23:19](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1399s) Stacking Layers and Parameter Totals
A transformer does not rely on a single multi-headed attention layer to understand language. Instead, data passes alternately through attention blocks and multi-layer perceptrons (MLPs), repeated across dozens of sequential layers. This deep recurrence allows embeddings to become iteratively richer: once a word absorbs an initial layer of local context, subsequent attention blocks let that newly contextualized representation interact with surrounding tokens that have also been enriched.
As representations travel deeper through the network, the embeddings transcend surface-level syntax and immediate descriptors. They begin to encode abstract properties such as tone, poetic style, literary genre, and underlying scientific concepts.
```mermaid flowchart LR accTitle: Deep Transformer Layer Flow accDescr: Flowchart illustrating the sequential processing of embeddings through alternating attention blocks and multilayer perceptrons across deep layers.
subgraph Layer1["Layer 1"] Attn1["Multi-Headed Attention"] --> MLP1["Multi-Layer Perceptron"] end
subgraph LayerN["Deeper Layers"] AttnN["Multi-Headed Attention"] --> MLPN["Multi-Layer Perceptron"] end
Layer1 -->|"Contextualized Embeddings"| LayerN LayerN --> Abstract["High-Level Representations<br/>(Tone, Sentiment, Knowledge)"] ```
Summing up the total footprint across GPT-3's architecture reveals a surprising distribution of weights:
- GPT-3 contains 96 distinct layers, meaning the ~600 million parameters in a single attention block are repeated 96 times. - This results in just under 58 billion total parameters dedicated exclusively to key, query, and value attention mechanisms. - While 58 billion is massive, it represents only roughly one-third of GPT-3’s 175 billion total parameters. - Sanderson points out that “even though attention gets all of the attention, the majority of parameters come from the blocks sitting in between these steps.”
---
[24:54](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1494s) Parallelism and Recommended Resources
The true secret weapon behind the dominance of transformers is not merely their expressive modeling capabilities, but their hardware efficiency. Attention can be expressed cleanly as large matrix multiplications, making it exceptionally well suited for massive parallelization on modern GPUs. Sanderson emphasizes that “a big part of the story for the success of the attention mechanism is not so much any specific kind of behaviour that it enables, but the fact that it's extremely parallelizable, meaning that you can run a huge number of computations in a short time using GPUs.” Because the primary empirical lesson of modern deep learning has been that raw scale yields dramatic qualitative leaps in capability, architectures that maximize parallel computing throughput inherently outpace their rivals.
Sanderson concludes with pointers to foundational resources for those seeking to build or understand transformers from first principles: - The work of Andrej Karpathy (including building GPT models from scratch) and Chris Olah (interpretability and transformer circuits), describing their output as "pure gold." - Educational videos by Vivek from the channel *vcubingx*, exploring the historical motivations that lead to reinventing attention. - Britt Cruise's historical retrospective on large language models on the channel *The Art of the Problem*.
## Notable quotes > [24:54](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1494s) A big part of the story for the success of the attention mechanism is not so much any specific kind of behaviour that it enables, but the fact that it's extremely parallelizable. — Grant Sanderson
> [19:49](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1189s) The overall idea is that by running many distinct heads in parallel, you're giving the model the capacity to learn many distinct ways that context changes meaning. — Grant Sanderson
> [00:00](https://www.youtube.com/watch?v=eMlx5fFNoYc&t=0s) The aim of a transformer is to progressively adjust these embeddings so that they don't merely encode an individual word, but instead they bake in some much, much richer contextual meaning. — Grant Sanderson
## Mentioned 3Blue1Brown, Grant Sanderson, Attention Is All You Need, GPT-3, Andrej Karpathy, Chris Olah, Anthropic, Vivek, vcubingx, Britt Cruise, The Art of the Problem, Vincent Rubinetti, Manim
## Video description Demystifying attention, the key mechanism inside transformers and LLMs. Instead of sponsored ad reads, these lessons are funded directly by viewers: https://3b1b.co/support Special thanks to these supporters: https://www.3blue1brown.com/lessons/attention#thanks An equally valuable form of support is to simply share the videos.
Demystifying self-attention, multiple heads, and cross-attention. Instead of sponsored ad reads, these lessons are funded directly by viewers: https://3b1b.co/support
The first pass for the translated subtitles here is machine-generated and, therefore, notably imperfect. To contribute edits or fixes, visit https://www.criblate.com
Звуковая дорожка на русском языке: Влад Бурмистров.
------------------
Here are a few other relevant resources
Build a GPT from scratch, by Andrej Karpathy https://youtu.be/kCc8FmEb1nY
If you want a conceptual understanding of language models from the ground up, @vcubingx just started a short series of videos on the topic: https://youtu.be/1il-s4mgNdI?si=XaVxj6bsdy3VkgEX
If you're interested in the herculean task of interpreting what these large networks might actually be doing, the Transformer Circuits posts by Anthropic are great. In particular, it was only after reading one of these that I started thinking of the combination of the value and output matrices as being a combined low-rank map from the embedding space to itself, which, at least in my mind, made things much clearer than other sources. https://transformer-circuits.pub/2021/framework/index.html
Site with exercises related to ML programming and GPTs https://www.gptandchill.ai/codingproblems
History of language models by Brit Cruise, @ArtOfTheProblem https://youtu.be/OFS90-FX6pg
An early paper on how directions in embedding spaces have meaning: https://arxiv.org/pdf/1301.3781.pdf
------------------
Timestamps: 0:00 - Recap on embeddings 1:39 - Motivating examples 4:29 - The attention pattern 11:08 - Masking 12:42 - Context size 13:10 - Values 15:44 - Counting parameters 18:21 - Cross-attention 19:19 - Multiple heads 22:16 - The output matrix 23:19 - Going deeper 24:54 - Ending
------------------
These animations are largely made using a custom Python library, manim. See the FAQ comments here: https://3b1b.co/faq#manim https://github.com/3b1b/manim https://github.com/ManimCommunity/manim/
All code for specific videos is visible here: https://github.com/3b1b/videos/
The music is by Vincent Rubinetti. https://www.vincentrubinetti.com https://vincerubinetti.bandcamp.com/album/the-music-of-3blue1brown https://open.spotify.com/album/1dVyjwS8FBqXhRunaG5W5u
------------------
3blue1brown is a channel about animating math, in all senses of the word animate. If you're reading the bottom of a video description, I'm guessing you're more interested than the average viewer in lessons here. It would mean a lot to me if you chose to stay up to date on new ones, either by subscribing here on YouTube or otherwise following on whichever platform below you check most regularly.
Mailing list: https://3blue1brown.substack.com Twitter: https://twitter.com/3blue1brown Instagram: https://www.instagram.com/3blue1brown Reddit: https://www.reddit.com/r/3blue1brown Facebook: https://www.facebook.com/3blue1brown Patreon: https://patreon.com/3blue1brown Website: https://www.3blue1brown.com
[Source](https://www.youtube.com/watch?v=eMlx5fFNoYc)