Google Scraps Recurrent Networks for All-Attention Transformer Model
Yannic Kilcher
Replacing step-by-step recurrent transitions with direct attention reduces computational path lengths to single operations, eliminating sequential information loss while enabling parallel sequence training.
AI-assisted summary of Attention Is All You Need — original source. Check the original for full context.
Source published:
Key ideas
- Vaswani et al. introduce the Transformer, an architecture that completely discards recurrent neural networks and convolutions in favor of pure attention mechanisms across sequences. (10:33)
- By replacing recurrent transitions with direct attention connections, the Transformer reduces information path lengths between arbitrary tokens to single-step operations, mitigating the severe signal decay of traditional RNNs. (25:50)
- Instead of sequential recurrence across time steps, the Transformer processes entire source and target sequences simultaneously, eliminating sequential unrolling during backpropagation. (10:33)
- On translation benchmarks, the model sets state-of-the-art results (28.4 BLEU on WMT 2014 English-to-German and 41.0 BLEU on English-to-French) while requiring significantly lower training costs.
Detailed notes
## Background Published by Yannic Kilcher on YouTube on November 28, 2017, this video breaks down the landmark research paper "Attention Is All You Need" by Vaswani et al. According to the page description, the authors introduce the Transformer—a novel, simple network architecture based entirely on attention mechanisms that completely dispenses with recurrence and convolutions, achieving state-of-the-art results on machine translation tasks while dramatically reducing training time.
## TL;DR Kilcher outlines the limitations of traditional sequence-to-sequence recurrent neural networks, particularly their sequential bottlenecks, vulnerability to long-range dependency decay, and unrolling constraints during backpropagation. He explains how the Transformer architecture revolutionizes sequence transduction by processing entire sequences in parallel and replacing recurrence with sinusoidal positional encodings and three specialized attention operations. Focusing on the cross-attention module, Kilcher demonstrates how scaled dot-product attention acts as a differentiable memory-retrieval mechanism over keys, values, and queries. Ultimately, he highlights that collapsing the path lengths of information flow from sequential hops down to single steps is the primary conceptual driver behind the architecture's state-of-the-art translation performance.
## Atmosphere & style The presentation has the relaxed, whiteboard-style feel of an informal graduate seminar led by an approachable, enthusiastic researcher. Kilcher speaks conversationally and candidly, interjecting dry humor—such as clarifying that he does not work for Google—while walking viewers methodically through diagrams and mathematical intuition. The narrative arc moves fluidly from dismantling the established RNN paradigm to unpacking the elegant mechanics of multi-head attention, concluding on an upbeat, accessible note that encourages viewers to experiment directly with the authors' open-source code.
## Bulletin - [10:33](https://www.youtube.com/watch?v=iDulhoQ2pro&t=633s) Vaswani et al. introduce the Transformer, an architecture that completely discards recurrent neural networks and convolutions in favor of pure attention mechanisms across sequences. - [25:50](https://www.youtube.com/watch?v=iDulhoQ2pro&t=1550s) By replacing recurrent transitions with direct attention connections, the Transformer reduces information path lengths between arbitrary tokens to single-step operations, mitigating the severe signal decay of traditional RNNs. - [10:33](https://www.youtube.com/watch?v=iDulhoQ2pro&t=633s) Instead of sequential recurrence across time steps, the Transformer processes entire source and target sequences simultaneously, eliminating sequential unrolling during backpropagation. - On translation benchmarks, the model sets state-of-the-art results (28.4 BLEU on WMT 2014 English-to-German and 41.0 BLEU on English-to-French) while requiring significantly lower training costs.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=iDulhoQ2pro&t=0s) Traditional sequence-to-sequence NLP models rely on RNN or LSTM encoders that compress an entire input sequence sequentially into a single bottleneck vector. - [03:29](https://www.youtube.com/watch?v=iDulhoQ2pro&t=209s) Recurrent architectures struggle with long-range dependencies because early input tokens must propagate through numerous recurrent transitions before reaching decoding steps. - [07:57](https://www.youtube.com/watch?v=iDulhoQ2pro&t=477s) Attention was initially introduced into RNNs as a direct addressing shortcut, enabling the decoder to inspect intermediate encoder states rather than relying solely on the final compressed state. - [10:33](https://www.youtube.com/watch?v=iDulhoQ2pro&t=633s) The core thesis of 'Attention Is All You Need' is that recurrence is entirely expendable and sequence modeling can rely solely on attention mechanisms. - [10:33](https://www.youtube.com/watch?v=iDulhoQ2pro&t=633s) The Transformer processes input sequences in parallel rather than sequentially, removing the need for backpropagation through time across temporal unrolls. - [13:30](https://www.youtube.com/watch?v=iDulhoQ2pro&t=810s) Because the model eliminates recurrence, it injects sinusoidal positional encodings across multiple frequencies to inform the network of token positions and order. - [16:02](https://www.youtube.com/watch?v=iDulhoQ2pro&t=962s) Self-attention is applied at the encoder level to allow input tokens to directly attend to and contextualize one another simultaneously. - [16:02](https://www.youtube.com/watch?v=iDulhoQ2pro&t=962s) A masked self-attention mechanism runs over the generated target tokens to build decoder representations without looking ahead. - [16:02](https://www.youtube.com/watch?v=iDulhoQ2pro&t=962s) The centerpiece of the architecture is the cross-attention module, which integrates the encoder's keys and values with the decoder's query vectors. - [18:22](https://www.youtube.com/watch?v=iDulhoQ2pro&t=1102s) Scaled dot-product attention acts like differentiable memory lookup: query-key dot products measure alignment, and the resulting softmax distribution softly selects relevant value vectors. - [25:50](https://www.youtube.com/watch?v=iDulhoQ2pro&t=1550s) The primary theoretical advantage of attention over RNNs is reducing path lengths for signal propagation, preventing intermediate computation steps from diluting semantic context. - [26:47](https://www.youtube.com/watch?v=iDulhoQ2pro&t=1607s) Vaswani et al. released full reference code on GitHub, enabling researchers to build and customize Transformer networks directly.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=iDulhoQ2pro&t=0s) The Classical Seq2Seq Setup and Recurrent Encoders
Yannic Kilcher begins by introducing the paper *"Attention Is All You Need"* by Google, adding the humorous disclaimer that he does not work for Google, despite recently covering several of their papers. He summarizes the core argument of the authors right off the bat: researchers should move away from recurrent neural networks (RNNs). In natural language processing (NLP), tasks like machine translation—for example, taking an English sentence like *"The cat eats the mouse"* and translating it into German—traditionally follow an encoder-decoder sequence-to-sequence (seq2seq) paradigm. Under this paradigm, the entirety of the input sentence must somehow be compressed into a single vector representation, which is subsequently unpacked and transformed into the target language.
```mermaid flowchart LR accTitle: Classical RNN Encoder-Decoder accDescr: Words are processed sequentially into hidden states, then decoded token-by-token. subgraph ENC["RNN Encoder"] direction LR H0["h₀"] --> E1["Enc"] W1["'The'"] --> E1 E1 --> H1["h₁"] H1 --> E2["Enc"] W2["'cat'"] --> E2 E2 --> H2["h₂"] H2 --> E3["Enc"] W3["'eats'"] --> E3 E3 --> H3["h₃"] H3 --> E4["Enc"] W4["'the'"] --> E4 E4 --> H4["h₄"] end subgraph DEC["RNN Decoder"] direction LR H4 --> D1["Dec"] D1 --> O1["'Die'"] D1 --> H5["h₅"] H5 --> D2["Dec"] O1 -.-> D2 D2 --> O2["'Katze'"] D2 --> H6["h₆"] end ```
To explain why this formulation has bottlenecks, Kilcher traces how standard RNNs and LSTMs operate. The encoder processes the source sequence token by token. The first token (*"The"*) is mapped to an embedding vector of numbers, which is then fed alongside an initial hidden state $h_0$ (either initialized with zeros or learned) into an encoder function to generate a new hidden state $h_1$. The next token (*"cat"*) is converted to its word vector and pushed through the exact same encoder function along with $h_1$ to produce $h_2$, and this sequential recurrence repeats across every token until reaching the final state $h_4$. Kilcher explains:
> “So it's always really the same function, and from the previous hidden state and the current word vector, the encoder again predicts another hidden state.”
- In standard RNN sequence encoding, the model updates its internal representation via a recurrent transition: $h_t = f(h_{t-1}, x_t)$. - The final encoder hidden state (such as $h_4$ in a four-token sequence) acts as the bridge that is handed over to the decoder.
[03:29](https://www.youtube.com/watch?v=iDulhoQ2pro&t=209s) The Decoding Bottleneck and the Long-Range Dependency Problem
Once encoding concludes, decoding begins. The final hidden state is plugged into the decoder network, which emits the first target token (e.g., *"Die"*) along with a new hidden state $h_5$. This next hidden state, often concatenated or conditioned on the previously emitted token, feeds into the decoder to generate the subsequent word (*"Katze"*) and hidden state $h_6$. Kilcher observes that the model relies on the hidden state to capture the overall semantic meaning of the sentence, while feeding back the previously generated word helps guide local grammar and sentence flow.
However, a serious vulnerability emerges when attempting to translate words that occurred earlier in the sentence. If the decoder needs to emit the German translation for *"eats"*, the primary information it has at step $h_6$ is just the previous word and that compressed vector. Yet the information from the input token *"eats"* had to undergo a lengthy cascade: it was transformed into a word vector, passed through an encoder transition to create a hidden state, and then propagated sequentially through every subsequent input step and through the initial decoding steps. Kilcher emphasizes how brittle this makes the network:
> “So it's very hard for an RNN to learn all of this what they what we call long-range dependencies...”
Kilcher points out that one might naively think words could simply be translated word-for-word in order, which works coincidentally in toy examples, but fails completely in actual language where words get rearranged, one word can turn into multiple words, or entire phrases use completely different grammar.
[07:57](https://www.youtube.com/watch?v=iDulhoQ2pro&t=477s) Attention as a Direct Addressing Shortcut
Attention was introduced to alleviate this severe information-transfer bottleneck in RNNs. Rather than forcing the decoder to rely solely on whatever residual trace survived the chain of recurrent state transitions, attention provides an explicit mechanism allowing the decoder to look back directly at intermediate encoder states. When generating the translation for *"eats"*, the decoder can learn to place its attention directly on the encoder hidden state corresponding to the time step when *"eats"* was processed, drastically reducing the path length traversed by the information.
```mermaid flowchart TD accTitle: Attention Mechanism Addressing accDescr: The decoder sends keys and queries to directly access encoder hidden states through softmax weighting. subgraph Inputs["Encoder Steps"] H1["h₁ ('The')"] H2["h₂ ('cat')"] H3["h₃ ('eats')"] H4["h₄ ('the mouse')"] end subgraph DecStep["Decoder at Step t"] DEC["Decoder State h₆"] K["Keys / Addressing Scheme"] DEC --> K end K -->|"Softmax Attention Weights"| H3 H3 -->|"Direct Information Flow"| DEC ```
Kilcher likens this operation to an addressing scheme seen in models like Neural Turing Machines: - The decoder outputs keys or queries ($k_1, \dots, k_n$) that score or index the encoder hidden states. - These scores are normalized using a softmax architecture to produce attention weights. - The decoder directly extracts and blends the most relevant information from across the entire source sentence, bypassing the multi-step recurrent chain.
[10:33](https://www.youtube.com/watch?v=iDulhoQ2pro&t=633s) Ditching Recurrence: The Transformer Paradigm Shift
Having established how attention was traditionally used as an add-on to boost RNNs, Kilcher turns to the central thesis of the Vaswani et al. paper: ditch the RNNs entirely. The authors claim that recurrence is not actually necessary at all—researchers can simply apply attention over the entire sequence at once.
To realize this, the paper introduces the Transformer architecture. Although it retains an encoder and a decoder block, Kilcher highlights a critical conceptual difference from traditional architectures: it does not process tokens sequentially through recurrent loops across time. Instead, the entire input sentence and the partial target sentence generated so far are fed into the network simultaneously in parallel. The encoder processes the full source sequence at once, the decoder attends across both the source embeddings and its own representations, and the network immediately produces output probabilities over the vocabulary for the next token.
This fundamentally alters the mechanics of training and backpropagation. Under traditional RNNs, an entire sentence-to-sequence translation represents a single training unit that requires backpropagating through time (BPTT) across every sequential step. In the Transformer, predicting each token position operates as an independent step where the entire computation and backward pass happen all at once, without unrolling states over time. Kilcher underscores how radical this departure felt to the NLP community:
> “And this is kind of a paradigm shift in sequence processing because people were always convinced that you kind of need these recurrent things in order to to make good to learn these dependencies, but here they basically say: 'No, no, no, we can just do attention over everything and it will actually be fine.'”
[13:30](https://www.youtube.com/watch?v=iDulhoQ2pro&t=810s) Embeddings and Positional Encoding via Trigonometric Functions
The architecture begins symmetrically on both sides with standard input embeddings and output embeddings, mapping input tokens into continuous word vectors. Because the model completely removes recurrence and convolutions, it discards the sequential nature inherent to traditional recurrent algorithms. Without sequential step-by-step processing, the network needs an explicit mechanism to know the ordering of the words—identifying whether a specific word is at the beginning or the end of a sentence, or determining which of two words appears first.
```mermaid flowchart LR accTitle: Positional Encoding Overview accDescr: Diagram showing tokens embedded into vectors, combined with sinusoidal positional encodings across multiple frequencies, and fed into the Transformer blocks. subgraph Inputs["Input Processing"] A["Input Tokens"] --> B["Word Embeddings"] C["Sinusoidal Positional Encodings\n(Multiple Frequencies)"] --> D["Sum: Word + Position"] B --> D end D --> E["Transformer Attention Blocks"] ```
To encode order without recurrence, the architecture relies on trigonometric functions (sine and cosine waves) across varying frequencies. A slow sine wave can span the whole sequence, while faster sine waves oscillate rapidly. Across these differing wave scales, each sequence index receives a unique profile—acting as a continuous version of binary encoding. For instance, the first position might be evaluated as "down" across all waves, the second as "down-down-up," and the third as "up-down-up." When the network needs to compare two words, it checks their values across these sinusoidal scales: if their values match on a long, slow wave but differ on a fast wave, the network determines they are close together and resolves their precise relative order.
- Word tokens are converted to embedding vectors symmetrically on the input and output sides. - Positional encodings provide the network with word order awareness without using recurrence. - Multiple sinusoidal frequencies form a continuous analog to binary positional representation. - While the positional encodings give the model an important performance boost, Yannic notes they are secondary: “it's not that the meat of the thing.”
---
[16:02](https://www.youtube.com/watch?v=iDulhoQ2pro&t=962s) The Three Forms of Attention in the Transformer
The core innovation of the architecture is how it applies attention at three distinct locations in the network. The first application appears at the bottom-left of the diagram, where self-attention is computed directly over the input sentence. Here, the network creates a hidden representation of the entire input sequence all at once, choosing which input words to focus on or associate with one another. The second application is at the bottom-right, performing self-attention over the output sentence generated so far to build a target-side hidden state.
```mermaid flowchart TD accTitle: Transformer Attention Connectivity accDescr: Visual mapping of the three attention modules showing input self-attention, masked target self-attention, and the cross-attention mechanism bridging the encoder and decoder. subgraph Encoder["Encoder Stack"] Inp["Input Embeddings"] --> EncAtt["Input Self-Attention"] EncAtt --> Repr["Encoder Representation"] end
subgraph Decoder["Decoder Stack"] Tgt["Target Embeddings\n(Shifted Right)"] --> DecAtt["Target Self-Attention\n(Masked)"] DecAtt --> DecRepr["Target Representation"] end
subgraph CrossAttention["Encoder-Decoder Cross-Attention"] Repr -->|"Keys (K)"| XAtt["Cross Multi-Head Attention"] Repr -->|"Values (V)"| XAtt DecRepr -->|"Queries (Q)"| XAtt XAtt --> UpperLayers["Upper Feed-Forward & Output Layers"] end ```
The third, located at the top-right of the decoder stack, is the cross-attention mechanism that bridges the encoder and decoder. Yannic highlights this as the centerpiece of the system: “the top right is the most interesting part of the attention mechanism here where basically it unites the kind of encoder part with the kind of... it combines the source sentence with the target sentence that you've produced so far.” In this module, three separate connections feed into the multi-head attention block: two connections (the keys and the values) arrive directly from the encoder's source sentence representation, while the third connection (the query) comes up from the decoder's representation of the target sentence produced up to that point.
- **Bottom-left attention:** Self-attention over the input tokens to construct source representations all at once. - **Bottom-right attention:** Self-attention over target tokens generated so far. - **Top-right attention:** Cross-attention connecting the source and target representations. - Three incoming paths feed the cross-attention block: Keys ($K$) and Values ($V$) from the encoder, and Queries ($Q$) from the decoder.
---
[18:22](https://www.youtube.com/watch?v=iDulhoQ2pro&t=1102s) Dot-Product Attention as Soft Memory Addressing
The interaction inside the cross-attention module can be understood through the lens of an associative database or key-value retrieval table. The encoder discovers salient properties of the source sentence and outputs sets of keys ($K$) and values ($V$), while the decoder formulates queries ($Q$). In high-dimensional spaces, arbitrary vectors are generally close to orthogonal (roughly 90 degrees apart), meaning their dot product sits near zero. However, when a query aligns with a key, their dot product grows large.
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right)V$$
To turn these alignment scores into a selection mechanism, the network applies a softmax over the dot products. The softmax exponentiates each dot product, causing the largest value to dominate, and normalizes across the entries:
$$\text{Softmax}(x)_i = \frac{e^{x_i}}{\sum_{j} e^{x_j}}$$
This acts as a differentiable approximation of the $\max$ function. Multiplying the resulting probability distribution across the values selectively retrieves the value corresponding to the best-matching key.
```mermaid flowchart TD accTitle: Scaled Dot-Product Attention Flow accDescr: Step-by-step mathematical flow of computing queries and keys dot product, exponentiating via softmax, and taking a weighted sum of values. Q["Query (Q)\n(from Decoder)"] --> Dot["Dot Product (Q · K)"] K["Keys (K)\n(from Encoder)"] --> Dot Dot --> SMax["Softmax\n(Differentiable Max)"] SMax --> Weight["Attention Weights\n(Soft Distribution)"] V["Values (V)\n(from Encoder)"] --> Mult["Weighted Sum of Values"] Weight --> Mult Mult --> Out["Retrieved Representation"] ```
To illustrate this intuitively, Yannic presents an analogy of human attributes: the values are the actual data (such as a person's specific name, tallness, or weight), the keys serve as the indexing tags (labeled "name," "height," or "weight"), and the decoder query specifies what detail it needs (such as asking for the "name"). When the query aligns with the "name" key, the softmax selects that entry and returns the corresponding name value. While noting that the idea borrows conceptually from earlier work on differentiable neural computers and neural Turing machines, Yannic remarks: “it's a pretty ingenious it's not entirely new because it has been done of course before with all the differentiable Turing machines and whatnot but it's pretty cool that this actually works and actually works kind of better than RNNs if you simply do this.”
- Multi-head attention uses parallel sets of projection heads rather than a single vector. - Vector dot products quantify alignment; orthogonal vectors score near zero, aligned vectors yield large positive numbers. - Softmax exponentiates alignment scores, heavily weighting the maximum match while preserving end-to-end differentiability. - Multiplying the softmax distribution by the value matrix softly extracts the addressed information.
---
[25:50](https://www.youtube.com/watch?v=iDulhoQ2pro&t=1550s) Reducing Path Lengths and Improving Efficiency
The core theoretical justification for why attention outperforms recurrent architectures lies in path lengths for signal propagation. In recurrent neural networks, information must travel step-by-step through a long sequence of sequential state updates, where each intermediate transition introduces a chance to dilute or lose information. Attention collapses this distance by allowing direct, single-step connections between any two positions regardless of how far apart they sit in the sequence:
> “Basically that the point they make about this attention is that it reduces path lengths and kind of that's the the main reason why it should work better with this entire attention mechanism you reduce the amount of computation steps that information has to flow from one point in the network to another and that what brings the major improvement because all the computation steps can make you lose information and you don't want that you want short path lengths and so that's that's what this method achieves and they claim that's why it's better and it works so well.”
By minimizing the computational hops required across sequence positions, the Transformer preserves context far more effectively, leading to state-of-the-art results across machine translation and other benchmark experiments.
- Sequential recurrent models force signals through $O(n)$ transition steps, increasing the risk of information loss. - Pure attention mechanisms yield $O(1)$ path lengths between any arbitrary token pairs. - Shorter paths ensure gradients and semantic information propagate directly and efficiently across the entire context window.
[26:47](https://www.youtube.com/watch?v=iDulhoQ2pro&t=1607s) Concluding Thoughts and GitHub Codebase
Wrapping up his walkthrough of the state-of-the-art results established by the paper, Yannic Kilcher decides to bring the presentation to a close. For anyone looking to explore the implementation hands-on or experiment with the model directly, he points viewers to the authors' open-source materials, noting: “If you want to check it out yourself, they have extensive code on GitHub where you can build your own Transformer networks.” With that recommendation in place, he bids his audience farewell with a relaxed, cheerful sign-off wishing everyone a nice day.
- The authors provide extensive reference code publicly on GitHub, enabling researchers and developers to construct and train custom Transformer architectures.
## Mentioned Attention Is All You Need, Google, Transformer, Yannic Kilcher, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, Neural Turing Machines, GitHub, WMT 2014 English-to-German, WMT 2014 English-to-French
## Video description https://arxiv.org/abs/1706.03762
Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.0 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
[Source](https://www.youtube.com/watch?v=iDulhoQ2pro)