Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 2 - Large Language Models

斯坦福 CME295：Transformer 与大语言模型 | 2026年秋季 | 第2讲 - 大语言模型 本讲座深入探讨了现代大语言模型（LLM）的技术设计、训练经济学及运行机制。讲师梳理了从仅编码器（encoder-only）和编码器-解码器（encoder-decoder）系统向仅解码器（decoder-only）Transformer 的演进路径，分析了经验缩放定律、Chinchilla 计算最优解，以及带有负载均衡的稀疏混合专家（MoE）架构。他们系统地阐述了当前前沿大模型的主流方案——结合旋转位置编码（RoPE）、滑动窗口注意力（SWA）、分组查询注意力（GQA）、Pre-Norm 和 RMSNorm——以解决注意力扩展和 KV 缓存瓶颈问题。最后，讨论转向自回归生成，详细介绍了贪婪与随机采样、GPU 批处理中的浮点非确定性、上下文退化，以及思维链（Chain of Thought）和自洽性（self-consistency）等推理方法。 Chinchilla 缩放定律确立了计算最优的训练目标，即每个模型参数约对应 20 个 token；但在生产经济学考量下，通常有充分理由使模型训练量远超这一比例，以最大程度降低下游推理服务的成本。 (22:11) 稀疏混合专家（MoE）架构通过辅助负载均衡损失来避免路由崩溃，并越来越倾向于采用细粒度专家池搭配共享无条件专家的方案，以便在不增加激活 FLOPs 的情况下最大化总参数容量。 (36:03) 现代前沿大模型已基本确立了标准化的架构方案，包括旋转位置编码（RoPE）、用于缓解 KV 缓存内存瓶颈的分组查询注意力（GQA）、Pre-Norm 残差连接路径以及 RMSNorm。 (73:15) 即使温度参数严格设为 0，生产环境中的大模型推理仍具有非确定性，这是因为动态 GPU 请求批处理会改变不满足结合律的浮点运算的计算顺序。 (88:20)

Stanford CME 295 examines how modern large language models moved away from encoder-decoder designs to decoder-only transformers, pairing compute-optimal scaling and sparse mixture-of-experts architectures with micro-level layer refinements and controlled generation strategies. Convergence on Decoder-Only Text-to-Text Architectures (13:47) Early transformer variants explored different macro structures, ranging from bidirectional encoder-only setups like BERT to sequence-to-sequence encoder-decoder networks. While BERT achieved strong natural language understanding via masked language modeling and next sentence prediction, it required task-specific classification heads and small-scale fine-tuning datasets, while offering no straightforward mechanism for open-ended text generation. Modern large language models coalesced almost universally around autoregressive decoder-only architectures, exemplified by GPT. By removing the encoder and cross-attention sublayers, decoder-only systems apply causal attention across both the input prompt and output tokens. This reformulates natural language tasks into a unified text-to-text generation objective, enabling zero-shot adaptation through prompt continuation without mandatory fine-tuning. Compute Optimality and Sparse Mixture of Experts (22:11) Empirical scaling laws show that larger models are fundamentally more token-efficient, requiring fewer training tokens to achieve target loss thresholds. However, compute budgets bound model sizes. DeepMind's Chinchilla study demonstrated that prior systems were undertrained, establishing a compute-optimal ratio of roughly 20 tokens per parameter (a 1:20 ratio between parameter count and token volume) to optimize a standard cross-entropy loss. Because dense architectures engage every parameter during every forward pass, scaling models into hundreds of billions or trillions of parameters creates computational bottlenecks. Sparse Mixture of Experts (MoE) architectures replace standard feed-forward networks with routed expert pools, activating only the top-$k$ experts per token. To avoid 'routing collapse'—where early-initialized experts absorb all traffic while others idle—practitioners employ an auxiliary load balancing loss using a `stop_grad` operator. Modern implementations combine pools of dozens or hundreds of fine-grained routed experts with dedicated, universally executed shared experts to handle baseline linguistic representations. Layer Refinements: Positional Encodings, Attention, and Normalization (45:36) Contemporary models replace the foundational 2017 transformer primitives with specialized layer mechanics. Additive sinusoidal embeddings contaminated downstream feed-forward networks; frontier systems instead use Rotary Position Embeddings (RoPE), rotating query and key vectors in two-dimensional subspaces so their inner product reflects relative token displacement without learned parameters. To curb quadratic memory and compute bottlenecks, models pair Sliding Window Attention (SWA)—which expands its effective receptive field across stacked layers—with Grouped-Query Attention (GQA). By sharing key and value projections across query groups, GQA sharply reduces KV cache memory consumption during autoregressive decoding. For structural stability and throughput, architectures switch from post-norm to pre-norm skip connections, while replacing LayerNorm with RMSNorm to eliminate mean-centering overhead and bias parameters. Autoregressive Decoding, Non-Determinism, and Context Limits (81:09) Inference projects the final hidden state into vocabulary logits to select tokens. Deterministic methods like greedy decoding and beam search ensure reproducible outputs, which are essential for tasks like summarization and LLM-as-a-judge evaluation, but they often produce stiff text. Non-deterministic sampling methods—such as top-$k$, top-$p$ (nucleus) sampling, and temperature scaling—introduce conversational variety. However, even with temperature set to zero, production inference remains subtly non-deterministic because dynamic batching alters the reduction order of non-associative floating-point operations across parallel GPU kernels. For structured schema compliance, engines use guided grammar-constrained decoding to mask out invalid tokens. Context window scaling introduces practical trade-offs. Standard inputs span up to roughly one million tokens, but models experience 'context rot' in needle-in-a-haystack tasks, where performance degrades as background token noise overwhelms relevant details. To mitigate degradation, practitioners apply context compaction and clean chat sessions. For complex logic, techniques shift from few-shot prompting—which risks overfitting and inflates token overhead—toward Chain of Thought prompting, intermediate thinking tokens, and self-consistency majority voting. While context capacities and parameter budgets continue expanding, models remain constrained by inference latency, floating-point non-determinism, and contextual dilution, leaving test-time compute scaling and dedicated reasoning architectures as active areas of ongoing study.

[16:10] lived through today [22:11] undertrained [36:03] rich getting richer [69:28] Afshine Amidi: Do you really need to attend to everyone at every layer? [73:15] Afshine Amidi: no one cares about the query because the query, you use it once and then you don't need that. [81:09] Shervine Amidi: Good, and you? [81:09] Shervine Amidi: the 2nd of October at Stanford [93:41] Shervine Amidi: do this in JSON format and hope for the best [101:42] Shervine Amidi: how it gets there

[0:05] Hello, everyone, and welcome to lecture 2 of CME 295. [0:11] So today is an exciting day because we'll [0:14] be talking about large language models, [0:17] and in particular, how they relate to the transformer [0:20] architecture that we saw last lecture. [0:24] But before we start, we will recap what we saw last time. [0:29] And we will do that at every lecture [0:31] to see how this connect to what we'll see today. [0:34] So if you remember, last time we were wondering [0:39] how models could process text. [0:42] So models, they don't understand text [0:44] as it is, they understand numbers. [0:47] So if you remember, one thing that we looked at [0:49] was, first of all, how to divide our text into indivisible units. [0:55] And this process is called tokenization. [0:58] And we saw a few kinds of tokenization algorithms. [1:02] And we said that the most common tokenization algorithm [1:06] these days is the subword tokenization because it [1:10] has a nice trade-off between the vocabulary [1:14] that it was building and the sequence length [1:18] that input text would have. [1:21] But this was not it, because once you [1:23] divide a text into tokens, the next step [1:27] is to compute embeddings. [1:29] So if you remember, we saw Word2Vec as a way [1:33] to compute embeddings, but they were not aware of the context. [1:37] So then we talked about another class of models that were very [1:41] popular in the 2010s which is called RNNs, [1:45] Recurrent Neural Networks. [1:48] And then what we said was, OK, the problem [1:51] is that you tend to forget things [1:55] that happened in the past. [1:57] And so in 2017, there was this landmark paper called "Attention [2:03] is All You Need" that actually dropped the principle [2:07] of recurrence and had direct connections in order to compute [2:13] meaningful representation of tokens. [2:17] So we spent a little bit of time on some notations. [2:21] So if you remember, we had query, key, and value, [2:26] which were useful for us to know OK, [2:28] given a token, what other tokens should that token attend to? [2:35] And we saw that the architecture that the authors introduced [2:39] was called the transformer, and that the attention operation [2:44] that is here is actually something [2:47] that is happening in the multi-head attention [2:50] layers, which are present in the encoder and the decoder. [2:56] And what we said was that this architecture was first developed [3:01] in the context of machine translation, [3:04] in that in that context, that model performed really well. [3:11] So a natural question is, well, given [3:14] that this architecture performs really, really [3:16] well for machine translation, can we actually propagate [3:21] the gains to other domains? [3:25] And so this is what we're going to see in the first part. [3:29] So we're going to talk about different variants [3:32] of the original transformer architecture, such [3:36] that it was possible to apply it to other domains. [3:41] So if you recall, the transformer [3:44] is composed of two main parts. [3:47] The first part is the encoder, which is [3:50] on the left side of the figure. [3:53] And the goal of the encoder is to compute meaningful embeddings [3:59] of your input text. [4:02] So here it was in the context of machine translation. [4:04] So input text, you can think of it as the source text [4:07] that you want to translate. [4:10] And then you had a second part on the right side, which [4:12] is the decoder whose goal is to actually generate the output one [4:20] token at a time in an autoregressive manner. [4:25] And the way it does that is by attending [4:28] to all the tokens that it has predicted so far, [4:33] plus the meaningful embeddings from the input that [4:37] were computed by the encoder. [4:41] So that was the original transformer. [4:43] So a very natural question is, well, [4:46] how about we try to play with the architecture? [4:50] We try to see, well, what happens [4:53] if we only take the encoder? [4:56] Let's suppose we only take the encoder. [4:58] Can we do something with it? [5:01] Well, this is something that people have tried shortly [5:05] after the transformer came out in 2018 with a very important [5:10] paper called BERT, which stands for Bidirectional Encoder [5:16] Representation from Transformers. [5:19] So the idea is to take only the encoder parts of the transformer [5:28] and to leverage the multi-head attention layer which [5:34] performs the self-attention operation over all of its inputs [5:39] in order to have all the input tokens attend to one another. [5:47] And so here what they say is, well, [5:50] if you do that and if you introduce a special token-- [5:54] and here the special token is on the left side of the sequence [5:58] called CLS token. [6:00] So it's a classification token. [6:02] Then what you could do is rely on the meaningful encoded [6:09] embedding corresponding to the CLS token [6:13] in order to know what the input sequence is about. [6:18] Because here the encoder is doing self-attention [6:22] across all the tokens. [6:24] So the meaningful embedding that is [6:27] corresponding to the CLS token has interacted [6:31] with all other tokens. [6:35] So the idea of BERT is to say if you have some input text, then [6:42] what I can do is to compute a meaningful representation [6:47] of the input and then use that representation [6:51] for some downstream task. [6:54] So at the time we had something like sentiment extraction [6:57] as a very common task. [6:59] So is this tweet positive or not? [7:01] So typically what you could do is [7:03] take that embedding and then project that onto that class. [7:07] That's one example. [7:08] But then you could also take the embedding of a given token, [7:12] and then try to determine if that token is [7:15] corresponding to let's say, a noun or a verb and so on. [7:22] So that's the idea. [7:24] Does that make sense? [7:27] OK, great. [7:29] So now the next question is, how do you actually [7:32] train such a model? [7:34] Well, this is done in two steps. [7:37] The first step is called pre-training. [7:41] And the goal is for the model to understand how text is written. [7:48] And so the way you do that is with some proxy tasks. [7:52] So if you remember, a proxy task is a task that is not really [7:56] your end goal, but that is a way for you [7:59] to achieve something else. [8:01] And for us, this something else here [8:03] is the model understanding text. [8:08] So the first proxy task that this model is trained with [8:12] is called Masked Language Modeling, MLM. [8:18] And the idea is for the model to understand [8:24] how tokens are placed. [8:27] And so the way it does that is it takes the input text [8:31] and then it corrupts it randomly. [8:34] So the way it does that is it selects some tokens at random [8:39] and it masks them. [8:41] So actually, it's a little bit more [8:42] complicated to not introduce biases. [8:45] But the idea is to corrupt the input sequence. [8:51] And the idea is to compute meaningful embeddings [8:57] corresponding to these corrupted tokens in order [9:02] to try to predict the actual token-- the original token [9:08] that was before that corrupted token. [9:13] So here, as an example, if we take [9:14] a cute Teddy Bear is reading and let's suppose we corrupt cute [9:17] and we corrupt reading, then here [9:21] our goal could be to compute a meaningful representation [9:24] of these corrupted tokens and try [9:28] to predict the original token. [9:32] And what the model will try to do [9:35] is to rely on all the attention mechanism [9:39] that it is performing in order to deduce [9:44] what the original token was from the context. [9:50] So in other words, the model is trying [9:53] to understand how language is written [9:55] and trying to understand, what are [9:57] the words that are likely to be in places where it is missing? [10:02] So that's the first proxy task. [10:04] And then there's a second proxy task that the authors of BERT [10:08] have also tried, which is called NSP, which stands [10:12] for Next Sentence Prediction. [10:16] And the idea behind that proxy task [10:19] is for the model to understand how sentences are sequenced. [10:28] So here the idea is to have two different sentences as input, [10:33] and to ask the model if these two sentences actually [10:37] follow each other or not. [10:40] So in practice, what you would do is you would look at a text [10:43] and take two sentences that follow each other, put it there, [10:47] try to predict that it is indeed following. [10:49] And then two sentences that are not following each other [10:52] and you need to predict that it's not following. [10:55] And the idea behind that proxy task [10:59] is for the model to learn text, not at the sentence level, [11:04] but at the document level. [11:08] So that's the first step called pre-training. [11:11] And at that stage, you're assuming that your model now [11:15] understands text much better. [11:19] And that's why you can go to the second task. [11:22] So by the way, the first task is typically very expensive, [11:25] something you do just once, train on a bunch of data. [11:29] And then the second task is for you [11:33] to adapt your model to the specific use case [11:36] that you're interested in. [11:38] So if we take the example of sentiment extraction, which [11:42] is trying to predict if a sentence is, [11:46] let's say, positive or negative, what you would do [11:50] is have some projection layer that takes in the CLS token that [11:58] is carrying all the information from the sentence [12:03] and project it onto the class-- [12:05] the classes of interests which are positive or negative. [12:10] And for that, you need to have a small set of labels. [12:16] So you need the observations with actual labels. [12:19] And then what you would do is to tune the weights, [12:22] so that you can actually project that meaningful representation [12:26] onto the space of interest. [12:31] So what are the pros and cons of this approach? [12:33] So the pro is, well, it led to pretty good performance. [12:38] The other thing is it is relying on the self-attention mechanism. [12:43] So you have connections from both sides, which is actually [12:47] the reason why BERT is called that way, [12:49] because it's Bidirectional Encoder Representation [12:53] from Transformers. [12:55] Another pro is, well, you need to do fine tuning, [12:58] but you don't need that much data typically or of 100. [13:04] But the con is, well, you need a small amount of data [13:08] to do the fine-tuning, but you need to do the fine-tuning. [13:10] So in itself, that's an effort. [13:13] So that's a con. [13:15] And then the other con is when you look at this model, [13:20] you may be wondering, well, how can I generate text? [13:26] And it's something that is not obvious. [13:29] But actually, we will see in lecture 8 [13:33] that it is actually possible to generate text [13:36] with an encoder only model. [13:39] But for now, let's suppose there's no obvious way. [13:43] But more on that in lecture 8. [13:47] Cool. [13:50] So that's for encoder only model. [13:53] Well, going back to the transformer architecture, [13:56] we tried removing the decoder, only going with the encoder. [14:00] That's what we obtained. [14:01] Now how about we remove the encoder [14:04] and we try with just the decoder? [14:07] Well, let's just try that. [14:09] Well, actually, people have tried that roughly [14:13] at the same time as BERT was published. [14:16] There was another branch of transformer variants [14:19] which were actually focusing on decoder only transformers. [14:24] And this is the example of GPT, which [14:27] I'm sure you have heard plenty of times, [14:29] which stands for Generative Pre-trained Transformer. [14:34] And the idea is to only keep the decoder parts of the transformer [14:41] and formulates any task you have as a text-to-text problem. [14:50] So what does that mean? [14:52] That means you have some text as input [14:55] and your only job is to predict the next token. [14:58] So here beginning of sequence A, A, teddy, teddy bear, [15:06] is, reading, and so on. [15:10] So by the way, this kind of decoding [15:12] is called autoregressive decoding because you [15:15] predict tokens one at a time. [15:18] And this is the idea behind GPT. [15:20] So you don't need to do-- you don't necessarily [15:24] need to do any fine-tuning because your model is already [15:28] equipped to be adapted to any use case. [15:33] So I'll give you an example. [15:35] The sentiment extraction. [15:36] So let's assume you have a tweet you want to categorize [15:40] as positive or negative. [15:43] Well, you would just put into your model, [15:46] is this sentence positive or negative? [15:49] And then you put the sentence. [15:51] And then the model would just respond positive or negative. [15:56] You don't need to do fine-tuning, [15:58] unless you want to make your model even better. [16:01] So fine-tuning is not a required step, which [16:06] is already pretty appealing. [16:11] But you may be wondering, well, last time we [16:16] saw the encoder-decoder architecture, [16:19] the original transformer, which was also generating text. [16:22] Well, what is the difference between this new decoder [16:25] only model with the encoder-decoder model? [16:29] Well, so both are similar in that they can generate text [16:36] autoregressively. [16:38] But in the case of the encoder-decoder architecture, [16:42] the input has a special treatment. [16:47] It is input into the encoder, and within that encoder, [16:52] all the tokens are able to interact with one another. [16:57] So it's like pure self-attention. [17:00] Whereas in the decoder only architecture, [17:03] what you have as input, the tokens [17:06] can only interact between themselves and tokens [17:11] before them. [17:12] They cannot interact with tokens after them. [17:16] So you have causal attention for decoder only, [17:20] but you have bidirectional attention for encoder-decoder [17:24] for the input. [17:27] That's the first difference. [17:29] The second difference is that, well, you have an extra encoder [17:35] and you need to have some special care for the input. [17:40] You're treating it differently, so it may make sense [17:43] for things like machine translation [17:45] because, well, you have source texts as inputs [17:49] and you want to have target text as outputs. [17:53] But things like text generation, it's not super clear [17:57] what should be as input. [18:01] So in other words, decoder only is a little bit more general [18:05] because it treats everything the same. [18:10] So people have tried both approaches [18:14] when it came to the text-to-text paradigm, [18:18] and overwhelmingly they all pretty much [18:21] went with the decoder only. [18:24] So what I want you to remember is [18:26] that people have tried all different variants. [18:30] And in particular when it comes to generating texts, [18:34] there was both these paradigms. [18:36] But the decoder only one is the one that [18:41] actually lived through today. [18:44] And one of the reasons is, well, it's simpler [18:46] and there is no special treatment of the inputs. [18:49] And yeah, you don't need to have an encoder [18:52] to treat the input separately. [18:55] So overall, there are two paradigms. [18:58] Either you compute meaningful embeddings [19:00] and you try to fine-tuning for your use case, [19:03] or you frame it as a text to text problem. [19:06] And you can frame all problems as text-to-text problems. [19:10] Another question is, is it efficient? [19:11] It's another question. [19:12] But you can do that. [19:14] And that's what people have been doing so far. [19:22] Cool. [19:23] Well, as I mentioned, people went [19:28] with the decoder only architecture, and what they did [19:30] was just to scale that up, because they [19:33] have seen that it was performing better and better. [19:36] And in particular, in the early 2020s, [19:40] there were a couple of papers that actually quantified how [19:44] much better your model was getting as you were training it [19:49] on more data, with more parameters, with more compute. [19:55] And we'll see that now. [19:57] So this paper from 2020 is actually [20:02] saying that you can actually have almost like a law that [20:09] tells you how much better your model will [20:12] get as a function of the number of tokens [20:16] that you're training on. [20:18] So it's the middle chart, and as a function [20:21] of how big your model is, which is the rightmost chart. [20:28] And so that's the reason why in 2020, 2021, 2022, [20:32] people have been increasing model size [20:36] and training it on more tokens like increasingly. [20:40] And not only that, the authors also [20:45] saw that the bigger your model, the more token efficient [20:50] they were as well. [20:52] So what does that mean? [20:54] That means that if you have a bigger model, [20:58] you need a fewer amount of tokens [21:01] to train on in order to reach a certain performance. [21:06] So on the slide, so what are all these colors? [21:09] So on the y-axis, it's the Tesla's. [21:11] So it's the performance of your model. [21:13] The lower the better. [21:15] And then the x-axis is the number [21:17] of tokens that were processed as part of the training. [21:22] And so the beautiful colors, they're [21:24] saying that the brighter you are, [21:30] the bigger your model is, the more it tends to have a better [21:35] performance earlier. [21:38] So that's why we're saying that bigger models are also [21:41] more token efficient. [21:43] They need fewer training tokens to perform better. [21:48] But as you can imagine, well, hardware is limited. [21:53] You don't have infinite compute. [21:55] So very naturally, you're wondering, well, [21:58] if I have a certain compute budget, [22:01] how much should I scale my model in size? [22:05] And how much should I attribute that to the number of tokens [22:08] I should train it on? [22:11] So before we talk about this, I want to introduce some notations [22:14] that I think maybe you have seen, [22:16] but I want to make sure we're all on the same page. [22:19] So the first one is FLOPs, which stands for Floating Point [22:24] Operations. [22:25] And please note the little S, lowercase S. So this [22:28] is a unit that is meant to measure compute. [22:33] So the higher the FLOPs, the more compute you actually have. [22:38] So order of magnitude, so training FLOPs for LLMs [22:41] is like 10 to the power of 25. [22:44] But you may see another FLOPS out there. [22:47] So FLOPS with a capital S or even /S. [22:51] And that one is actually the compute speed. [22:55] And it's actually a measure of how performant your hardware is. [23:00] It is different compared to the previous FLOPs, [23:03] but it's easy to get mixed up. [23:05] So just wanted to call that out. [23:08] So with that said, back to what we were saying. [23:12] So we live in a world where we don't have unlimited compute, [23:16] or at least not yet. [23:18] So given the amount of compute that you have available, [23:22] a natural question that you have is, [23:25] how much should I increase my model size? [23:28] Versus how much should I increase the training size [23:33] on which I'm training my model? [23:36] And so there is this paper that came out [23:38] in 2022 which exactly asked this question. [23:44] And what it found was actually pretty interesting. [23:47] It found that there was actually a sweet spot between model size [23:53] and training data size. [23:57] And it also found that most models at the time in 2022 [24:01] were actually undertrained, meaning [24:05] that they were too big for the amount of tokens [24:10] that they were trained on. [24:11] They were undertrained. [24:14] And so it actually also showed that by training two models. [24:18] So one, which was, I believe, 70 billion parameters [24:22] and another one 200 and something billion parameters, [24:26] so higher than the previous one. [24:28] And it had the same compute budget for the two. [24:32] So the first one, given it's smaller in model size, [24:36] it trained it on more tokens. [24:39] I believe it trained it on $1.4 trillion tokens. [24:43] And the second one, which was bigger in size, [24:45] it trained it on fewer tokens because it [24:47] was the same compute budget. [24:49] And it showed that the smaller model in terms of size [24:54] was actually more performant. [24:57] So they had some relationship between number of parameters [25:01] and number of tokens that you are training it on, [25:04] which was something like one parameter for 20 tokens. [25:10] And so this was quite important. [25:12] A lot of people tried to go back and see OK, [25:15] is my model compute optimal? [25:18] So yeah, so this is also pretty important paper. [25:23] Another question you can ask yourself-- [25:25] I mean, just trying to think beyond this paper [25:28] is what your use case is. [25:30] Because here we're talking about training compute optimal. [25:35] But you also need to serve your model production. [25:41] So if you serve it with many, many requests, [25:45] you also have an incentive to not have it be too big. [25:50] But that's another question. [25:52] But I just want to put it out. [25:54] Yeah. [25:56] So the question is, is it the cross-entropy loss? [25:58] And the answer is yes, because your task [26:00] is next token prediction. [26:02] And what you're trying to do is have [26:06] your prediction be as close as the label, which [26:09] is the next token. [26:10] And yes, cross-entropy is typically the one that you use [26:13] and it's the one that is here. [26:16] Yeah. [26:19] So the question is, does the quality of the data matter? [26:22] I would say there are many papers that [26:24] came out from this, many with different conclusions. [26:29] I would say for pre-training, what matters is [26:31] the size more than the quality. [26:34] We will see in lecture 3 that next steps of the training [26:39] process matters, there's more importance towards quality [26:43] and we will see that. [26:46] OK, cool. [26:49] OK, moving on to LLM. [26:51] So why do we call LLM, LLM? [26:54] Well LLM stands for Large Language Model. [26:59] So language model is, because it's [27:01] a model that associates some probabilities to sequences [27:06] of tokens. [27:07] So in this case, it's a model that tries [27:10] to predict the next token. [27:12] So we're trying to put some probability towards what [27:16] the next prediction is. [27:19] And why is it large? [27:20] Well, it is large because the model size is large, typically [27:23] in the billions, even tens of billions or hundreds of billions [27:27] and even trillions. [27:30] Training data, also hundreds of billions of tokens. [27:33] At least nowadays, it's also on the tens of trillions of tokens. [27:37] And also compute. [27:38] You need a lot of GPUs. [27:41] And as I mentioned, nowadays people [27:46] rely on the decoder only transformer, [27:49] which is the original transformer without the encoder. [27:55] Well, you may ask me, well, if you look into the decoder, [28:01] you actually have a component, which [28:04] is the cross-attention that takes [28:07] in the meaningful embeddings from the encoder with the ones [28:13] coming from the decoder. [28:14] And you're like, OK, what is happening with that components? [28:16] Well, you also remove it. [28:18] So the decoder is not exactly the decoder. [28:20] You also remove the part that is interacting [28:24] and you obtain this very simple backbone, [28:29] which is a causal multi-head attention layer, which is only [28:34] attending to each token and the tokens that [28:37] came before it and a feed forward neural network, which [28:42] is where the compute happens. [28:47] Cool. [28:49] OK. [28:50] So what we saw here is the bigger your model size, [28:55] the better your performance. [28:57] And we saw that there was also a trade-off between the number [29:00] of tokens you were training it on [29:02] and the model size that you were having. [29:05] A natural question could be, when you have an input [29:11] x into your model, if your model is huge, [29:16] then that means that there are a lot of computations happening [29:19] in that model. [29:21] A natural question can be do you really [29:24] need all the parameters in your model for that given input x? [29:31] Do you really need that? [29:34] And I want to motivate this observation [29:37] with another observation. [29:40] So let's assume I'm here in a room full of people. [29:44] And let's say I have a math question. [29:47] Let's suppose you all are coming from different backgrounds. [29:50] So let's assume you're a math expert. [29:52] Let's assume you're a physics expert, biology expert. [29:55] And I'm coming with my math question. [29:59] Should I ask everyone? [30:00] Should I only ask you? [30:03] And this is exactly the kind of question [30:05] that people have asked themselves. [30:08] How about we actually only activate [30:14] a subset of the model for given inputs as opposed [30:18] to all parameters? [30:22] And so if you were to do that, then you [30:24] could scale your model in size, but not [30:28] incur the cost of having to go through all these operations [30:34] for every forward pass? [30:37] So how about you did that? [30:39] Well, it turns out there's a type of model out there [30:43] that was actually predating LLMs, which [30:46] is called a mixture of experts. [30:49] So who has heard of mixture of experts? [30:52] Yes. [30:52] OK, great. [30:54] So what is the idea behind mixture of experts? [30:56] So you have experts. [30:58] So think of it as your math expert, [31:00] physics experts, et cetera. [31:02] And then you have a gate, also called a router, [31:08] that tries to figure out, given your input, which expert should [31:14] you use and to what extent? [31:18] So for instance, you have some input x [31:21] and you're passing it through your gate or your router. [31:24] And the router says OK, expert 2, [31:26] so this is a math question I should ask you. [31:31] So there are several kinds of Mixture of Experts or MoEs. [31:38] One first kind is called a dense MoE. [31:41] A dense MoE, what does it do? [31:43] It still activates all the experts, [31:46] but it gives more weight towards experts [31:49] that are maybe more relevant. [31:51] So, for instance, the math question [31:53] could be high weight for the math expert, maybe lower [31:56] weight for the physics expert, but maybe a very small weight [31:59] for the biology experts. [32:03] So the bad thing with the dense MoE [32:07] is you still activate all the parameters. [32:09] It's not what you want to do. [32:10] So that's why, in the context of LLMs, [32:15] the kind that is actually very interesting [32:17] is actually not the dense MoE, it's [32:19] the sparse MoE, which is the kind of ammo that actually [32:25] selects experts that should be activated [32:30] and that ignores all the other experts. [32:35] This is the sparse MoE. [32:37] So typically you have top k experts that were selected. [32:41] So k can be equal to 1, can be equal to 2 to some number [32:45] that you fix. [32:48] And the output that you get, so y hat is the sum of the expert [32:54] output weighted by how important they are, [32:58] which is given by the gates. [33:01] And it's only a sum over some experts that were activated-- [33:06] so k experts, not all experts. [33:11] Cool. [33:11] Well, OK. [33:12] All that is great. [33:13] But now let's go back to reality. [33:15] Let's go back to our LLM. [33:16] If you remember, it's a decoder only architecture. [33:20] So we have this as a backbone. [33:23] So I have a question for you. [33:26] Where would you put this mixture of experts scheme [33:31] in the architecture? [33:33] Which layer would you put it? [33:37] Does anyone have any suggestions? [33:44] Yes. [33:47] Yep. [33:48] So the answer here is you can take a token [33:51] and project it in order to select which expert it is. [33:54] Yes. [33:54] So that is actually the role of the router. [33:57] So here the thing that I'm really interested in [33:59] is where would you put the whole mixture of experts [34:03] in all of this? [34:04] So I'll help you. [34:05] Oh, yeah. [34:06] Sorry. [34:06] Yeah. [34:08] It's a great answer. [34:09] So I'll just exactly mention what you said. [34:12] So the answer would be feedforward neural network, [34:15] because this is where all the compute typically resides. [34:19] And yes, I completely agree with you. [34:22] The multi-head attention layer is actually the place [34:26] where tokens communicate. [34:27] It's not where typically you compute meaningful embeddings [34:31] by projecting it. [34:33] So you would typically go with the feedforward neural network. [34:37] So a great answer. [34:38] And so here what you would do is back to your point, [34:41] have a router where for a given input, [34:44] decides which expert to activate. [34:48] And here the expert would be a feedforward neural network. [34:54] So yeah, great answer. [34:58] Sorry. [34:58] Yeah. [35:01] So the question is, well, or the remark I would say [35:05] is that in the attention layer, you have different heads. [35:09] Why wouldn't you treat heads as experts? [35:13] So it's a great point. [35:14] I think, at the end of the day, you can also try that. [35:19] But speaking of that, we will actually [35:21] talk about a small variation that people usually [35:26] do on multi-head attention because, not all heads are [35:29] as useful. [35:30] So I'm going to just address your question [35:32] in about 30 minutes. [35:35] Yeah, but great point. [35:37] Yep. [35:40] So the question is, how does the router know [35:44] when to activate each expert? [35:45] So by with the magic of back propagation, magic of learning. [35:50] So you have a training process and then [35:53] you want to learn how each component should behave. [35:58] And you just backpropagate and it just learns that way. [36:01] And it's actually a great point because one observation [36:07] you may tell me is, what would prevent [36:10] your model from overrelying on a specific set of experts [36:16] and not explore other experts? [36:20] And so this is actually a problem [36:23] when training MoE-based LLMs, which [36:27] is the tendency for your model to rely only [36:31] on a small set of experts instead of all the experts. [36:36] And it's called routing collapse. [36:38] So there are many reasons why it's not good. [36:40] But one of the reasons is your effective model size [36:45] is actually smaller than the total number of experts [36:51] and weights that you have. [36:52] And this typically occurs, because when [36:57] you train your model, if some experts get chosen [37:02] in the very beginning, then when it [37:05] comes to updating the weights, those experts that [37:11] were activated in the beginning, they [37:13] tend to be updated at first. [37:18] So they tend to get better and better, [37:20] whereas all the other experts are still sitting there. [37:24] So it's the problem of the rich getting richer. [37:28] And so it's for that reason that people have also come up [37:33] with a part of the last cult, so it's [37:37] a load balancing auxiliary loss, which [37:42] is aimed at forcing your model to also incentivize exploring [37:49] other experts by encouraging the fraction of the tokens that [37:56] are being rerouted to be uniform or more or less uniform [38:00] across experts. [38:02] And so maybe it's something that I didn't say, [38:04] but I will say it now. [38:06] The routing is done at the token level. [38:10] So if you have an input sentence, [38:12] each token will ask itself, which expert should I go to? [38:18] So this is done at the token level. [38:21] And so here this loss is incentivizing your model-- yeah. [38:29] So the question is, well, if you select one or two, [38:31] it's not differentiable. [38:32] It's a great point. [38:33] So you have an operator that's called stop grad that [38:37] is treating some aspects as being actually derivative of 0 [38:42] and treating these as constants. [38:45] So long story short is in the loss, [38:48] you're treating the fraction of tokens routed to expert i [38:52] as a constant. [38:54] And you're only backpropagating through the routing probability. [38:58] So you're still able to actually change those routers. [39:02] But yeah, that's a good point. [39:08] So let me see. [39:09] So I think we're talking about the fact [39:12] that these auxiliary loss is incentivizing your model [39:17] to also choose other experts and not [39:21] to have this routing collapse problem that we [39:24] mentioned at the beginning. [39:27] There are some other problems as well. [39:28] I just want to just tell you that training an MoE [39:31] based LLM is challenging, and this is part of its challenge. [39:38] But the good thing is, if you do it properly, you will have-- [39:44] so here on the slide, what you see [39:46] is an input text where the color of each section [39:52] denotes a specific experts that a given token was sent to. [39:58] And this is, for instance, at a given layer. [40:00] And this is coming from a paper from Mixtral. [40:03] So here is the model called Mixtral of experts. [40:07] And you see that the colors are pretty diverse. [40:10] So it means that you're actually leveraging your experts pretty [40:14] well. [40:17] And so I want to just say a few things about MoE-based LLMs. [40:22] So now we're in 2026. [40:23] This is something that I think the paper was [40:25] from 2024, which is quite some time ago in the AI field. [40:31] So nowadays, what people do is they use experts, [40:35] but they also have shared experts [40:40] that gets activated no matter which experts the router [40:46] or the gate selects. [40:48] And the idea behind this is for every token, [40:51] there is always some computation that is always necessary. [40:56] And you want to avoid your experts from always learning [41:00] the same thing as a foundation and then specializing [41:04] into something. [41:06] So that is something that people do nowadays. [41:08] The second thing that people do is the number of experts, [41:12] they don't choose it to be 8 or 16. [41:15] They actually choose it to be higher, 60 or in the hundreds. [41:21] And the reason why they do that is that intuitively, you [41:27] just give the chance to your experts [41:29] to be very specialized, very fine-grained. [41:32] But also on the charts, you see that the performance, [41:35] when you have more fine-grained experts is just higher. [41:40] So it's also just empirical. [41:43] And so speaking of order of magnitudes. [41:47] So when it comes to the frontier open source LLMs [41:51] that were published recently. [41:54] So here the screenshot is actually from a few days ago. [41:58] If we take the top models, what we see [42:01] is that-- so there are MoE-based LLMs. [42:03] And the number of experts that they have [42:05] is on the order of hundreds. [42:08] So here are 384, 256. [42:12] And the number of experts that are [42:14] activated at each forward pass is typically eight or so. [42:20] And you also have a shared expert, [42:22] which is not widely used. [42:26] Like in this example, the first model is not using it. [42:30] And in terms of the model size, it [42:32] is in the hundreds of billions of parameters [42:35] or even trillion parameter for the first one. [42:39] So it's pretty big models. [42:41] Yes. [42:44] So it's a great question. [42:45] The question is, does the scaling law still [42:48] hold true for a MoE-based LLMs? [42:51] So the answer is that paper was actually based on a dense LLM. [42:57] It was not like an MoE-based LLM. [42:59] So one thing that I've seen papers do [43:03] is have these scaling law experiments [43:06] for their own setup on small number of tokens, small model [43:11] size to just see what the law is for the specific setup. [43:15] So to your question, I think for MoE-based LLMs, [43:19] if authors feel that their choice, their design [43:25] is very different from what was tested before, they would just [43:29] do this very small experiment and see what [43:32] is the kind of law that comes from there, [43:35] and then choose the parameters from there. [43:39] Yep. [43:42] Yep. [43:42] So the question is, why is n the number of experts [43:46] doing in the numerator? [43:47] So here, actually, what you're doing is [43:51] you are summing across the experts. [43:53] And so if you actually write that mathematically, [43:56] you actually want to add n to-- [44:01] because fi and Pi is like over n, so it's over n squared, [44:05] times n is like over n. [44:07] So it's times n. [44:09] But n is the number of experts. [44:13] OK, great. [44:17] We're doing amazing. [44:18] Any questions, other questions? [44:24] So far, what did we see? [44:26] We saw that people have tried different variants [44:30] of the transformer. [44:31] They saw OK, how about we take the encoder? [44:34] Well, OK, you can do this. [44:35] How about you take the decoder? [44:37] You treat that as a text-to-text problem. [44:40] They scale that up. [44:41] And people have seen that the performance just [44:44] gets better and better as people increase it [44:46] in size and in number of tokens that it is trained on. [44:50] We've seen that when the model is too big, [44:53] you need to do something in order for you to not [44:57] have crazy inference costs. [45:00] And so that's why people have developed a MoE-based LLMs that [45:04] allow you to scale the number of parameters [45:07] without having to incur the cost of also activating [45:11] all the parameters in each forward pass. [45:15] So here in this part, what we will focus on [45:18] is a number of changes that people [45:22] have made in the layers that were originally [45:28] described in the original transformer architecture. [45:33] So you will see in a second what I mean. [45:36] So if you remember, the transformer architecture [45:41] relies on the self-attention mechanism, [45:44] which is the idea of letting all your tokens communicate [45:50] with everyone. [45:53] So the problem when you do that is [45:56] that you lose the information of where [45:58] tokens are positioned because you have these direct links. [46:03] So if you don't have any information about positions [46:06] about each of these tokens that are included anywhere, [46:10] well, you lose the information of where tokens are located. [46:16] So the original transformer paper tried two approaches. [46:23] The first was learning position embeddings [46:27] and adding those embeddings to the token embeddings. [46:31] So what people would do is you have an input sequence, [46:36] you have your tokens. [46:38] Each token is associated with a position index. [46:42] Let's assume 1, 2, 3, and so on. [46:45] And here the idea is to learn an embedding that is associated [46:51] with each position index. [46:53] So position 1 will have an embedding, [46:55] position 2 will have an embedding, and so on. [46:59] And these embeddings would be learned. [47:02] And what would happen is, in order for your model [47:06] to actually know about where each of these tokens [47:11] are located, so if you see on the right, what people would do [47:15] is take the token embedding and then [47:19] add that position-specific embedding to the token [47:23] just to reflect that token is at a very specific position. [47:29] But the problem with this approach [47:32] is, as we mentioned last time, that if you train your model [47:37] on a given length of input data, if you ever [47:43] come across some sequence at inference time, that is, [47:48] let's say longer, the problem is actually [47:52] have never encountered positions beyond the ones [47:55] that you encountered at training time. [47:59] So you actually do not know what embeddings for such position [48:02] are. [48:03] You don't know. [48:06] And so that's the reason why the original authors [48:09] of the transformer paper, they not only tried the learned way, [48:13] but they also tried a deterministic formula-based way [48:18] that we're going to see in a second. [48:21] And the reason why they tried this approach [48:23] is because it can generalize to any sequence length [48:31] without having you to actually train on longer sequences. [48:37] And they actually said that both approaches [48:39] were roughly leading to the same performance. [48:42] So the fixed one is better and we're going to see it now. [48:48] So there was this analogy that I mentioned last time that I will [48:53] start from in this session. [48:56] So the idea behind the deterministic formulation [49:03] is to express each position on a high dimensional vector. [49:11] And what you want is that vector to exactly inform you [49:17] in which position you're in. [49:20] And so you want that. [49:24] You also want two positions to be embedded, [49:30] such that the similarity operation between these two [49:35] embeddings are high if the positions are close [49:41] and it's low when these positions are far. [49:44] And so I'm going to just motivate [49:47] how the authors have actually designed [49:50] the formula and the design that they're [49:52] using for the position embedding that they're proposing here. [49:58] So going through the analogy that we discussed, [50:01] let's assume you're looking at your watch. [50:05] You have several hands. [50:06] You have the hour hand. [50:08] You have the minute hand. [50:09] You have the second hand. [50:12] If you put all these hands together, [50:14] you know exactly what time it is. [50:19] And each of these hands, they're moving at different frequencies. [50:26] So the hour tells you roughly when you are in the day, [50:28] the minute is a little bit more precise. [50:30] And then the second is like, even more precise. [50:33] And so what we're going to do is do [50:37] something very similar for the position embeddings, where we'll [50:41] have a function of higher and lower frequency, [50:49] such that when you put all of that together into one vector, [50:54] if you know exactly which position you're at. [50:59] So it's going to get a little bit more into the math. [51:02] So disclaimer, I hope you remember [51:05] your trigonometry formula. [51:06] But if not, don't worry. [51:08] We'll also look at it together. [51:10] So let's assume that you are at position m. [51:16] And so here you're looking at two consecutive indices-- [51:22] so 2i and 2i plus 1. [51:25] So what the authors propose is to have one be a cosine, [51:31] the other one be a sine or vice versa, [51:35] and have the frequency of these functions [51:40] be a function of the index. [51:43] So here, if i is low, then you have a higher frequency [51:49] function. [51:50] If i is high, you have a lower frequency function, [51:55] meaning your sine will vary more slowly. [52:00] So here, if you see on the graph, [52:02] I'm not sure if you exactly read the axis. [52:06] So on the right side of the slide, [52:10] you have on the y-axis the position, [52:15] and on the x-axis, the dimension. [52:18] So here what you see is if you take the first row [52:24] and you're moving down, you're going from position 1 [52:29] to later positions of your sequence. [52:34] And you see that the first dimensions, the first indices, [52:37] the one on the left, they tend to vary very quickly, [52:41] and the one on the right, the later indices, [52:44] they tend to vary much, much more slowly. [52:47] So in other words, your lower indices, [52:50] they're much higher frequency. [52:53] And the later indices, they are of much lower frequency. [52:58] So just to tell you that this seemingly scary formula is just [53:02] telling you this. [53:03] So it's varying cosine, sine with different frequencies. [53:07] It's a higher frequency for a lower index, lower frequency [53:11] for a higher index. [53:14] So let's assume you note omega i, the huge formula, [53:19] so that we don't need to worry about it. [53:21] Well, what happens if you have a position m and a position n? [53:27] How do you compute the similarity? [53:29] Well, you take the dot product. [53:32] So if you remember the way you take [53:33] the dot product of two vectors is you multiply each dimension [53:39] and then you add everything up. [53:43] So just a refresher on trigonometry. [53:47] So cosine of a minus b is cosine of a cosine [53:51] of b plus sine of a sine of b just the trigonometry formula. [53:59] So here if you take the cosine of omega i of m minus n, [54:06] it's actually the dot product between these two indices. [54:19] So the question, is i a function of the dimension? [54:22] Yes. [54:22] So i is a function of the index. [54:24] So if you have a lower i, then your omega i [54:29] will be such that it will be of higher frequency. [54:33] So yes, it's a function of the index. [54:36] That's a great question. [54:37] So here going back to our favorite trigonometric formula, [54:43] so here I just want to point out that cosine of omega i m minus n [54:49] is equal to the dot product of these two. [54:55] And if you actually want to take the dot product of the whole two [54:59] vectors, you actually obtain a sum [55:04] of the cosine of the omega i of all the indices multiplied [55:12] by m minus n. [55:16] Is everyone following so far? [55:19] Yeah. [55:20] So why am I saying this? [55:22] I'm saying this because the dot product of the two position [55:26] embeddings at position m and n is a function of m minus n. [55:32] And you may remember that cosine of 0 [55:38] is the maximum value that cosine can take. [55:41] So in other words, when m is equal to n, [55:45] the sum is maximized. [55:47] So the similarity is maximum. [55:50] So all of that to say that if you take this expression that [55:56] is graphically something that you see on the left in terms [56:00] of the value of the embeddings, what you will obtain [56:04] is that positions that are close together, [56:07] they're actually of high similarity and positions that [56:10] are farther together, they are of lower similarity. [56:14] And that's what you see on the right. [56:16] So on the right, you have the positions on the x-axis [56:21] and on the y-axis of m and n. [56:24] And you see that when the positions are equal, [56:28] the similarity is maximum. [56:32] So all of that to say that if you were to look at the formulas [56:36] without any context, I think you'd be probably scared. [56:39] But I hope that with this perspective, [56:44] it's a little bit less scary. [56:48] So now what I want to tell you is that if everything was great, [56:55] we would be using this exact method today. [56:59] But the truth is we are not. [57:01] This is actually not something that people use today. [57:04] People use something differently. [57:06] And the reason why that is the case is if you think about it, [57:13] you care about where tokens are positioned [57:19] in a very specific layer. [57:21] You only care about it in the attention layer, [57:25] because the attention layer is the layer where you actually [57:28] are communicating. [57:29] You're trying to determine which tokens are relevant when [57:33] you do your expression. [57:37] And so this is the reason why people have actually [57:43] tried to find a method that is not affecting all the layers, [57:49] but that is being surgically applied to the attention layer. [57:55] Yeah. [57:58] So the question is if you do that, then you may make [58:03] further token interact less. [58:05] And that was maybe the problem of RNN. [58:07] So why are we doing this? [58:08] So here the idea is to inject some position information [58:13] and intuitively tokens that are farther apart, [58:18] they're just less relevant. [58:21] But they're not relevant. [58:24] And so here the idea is that on a relative sense, [58:30] closer token are just more likely to be more relevant. [58:34] But you're still keeping the direct connections, [58:37] and you're still giving a chance to your model [58:39] to actually interact with those tokens. [58:42] So I do get your concern. [58:45] That being said, this is something [58:47] that proved to work well, and it just reflects the idea [58:53] that if let's say you have a 10 document page, then [58:57] probably what you have in the same sentence [58:59] is probably more relevant than something that [59:01] is very far apart, but you're still giving a chance [59:04] to these tokens to interact. [59:09] OK, great. [59:12] So going back to here. [59:13] So one major drawback of the previous approach [59:17] is that this position embedding was actually [59:20] being added to the token embedding, which was then being [59:25] processed by all the layers. [59:27] But intuitively, this is actually not what you want, [59:30] because what you want is to let your attention layer [59:34] know when two tokens are far apart. [59:39] But you don't want to let your feedforward neural network know [59:43] that your token is at position 5 or at position 12 [59:47] because the feedforward neural network doesn't care. [59:51] Can very well have a sentence where a token is [59:55] at position 24 versus 1. [59:59] That can be very making sense just because you [60:01] have multiple sentences. [60:05] So this is the reason why people have [60:08] tried to mitigate this drawback by surgically looking [60:14] into the attention layers and doing something there. [60:19] And in particular, I'll go through a few methods [60:22] that people have tried. [60:24] So one of them is a method which was meant to learn a bias [60:30] term within the Softmax. [60:34] So if you remember in the attention layer, [60:36] you have this formula which I recommend you remember, [60:39] which is Softmax of qk transpose over some scaling constant times [60:45] v. [60:46] So within the Softmax, you're actually [60:49] trying to ask yourself how relevant the query is [60:54] to all the other keys. [60:56] So this is where that matters. [60:58] And so here the idea is to learn a bias that [61:03] is dependent on how far your token m and your token n [61:11] are by learning a bias. [61:16] The problem with this is, so similar to the very first [61:19] method, you need to learn the bias. [61:22] And so first of all, maybe your training data [61:25] has some bias in itself. [61:27] So it will maybe not optimal. [61:29] But then the second thing is it's not [61:31] generalizable to all possible distances. [61:37] So that's why people have tried other methods as well. [61:40] Another one is called ALiBi-- so Attention with Linear Biases. [61:44] And so here the idea is to actually make [61:47] that bias vary as a function of m-- [61:52] sorry, n minus m. [61:53] So the distance between m and n. [61:58] But the method that is commonly used nowadays is not ALiBi, [62:04] is not the T5 bias, it's RoPE. [62:06] So who has heard of RoPE? [62:10] Yeah? [62:10] OK, great. [62:11] So we're going to see why RoPE is the way it is. [62:15] So RoPE stands for Rotary Position Embedding. [62:21] And the idea behind RoPE is in the attention layer, [62:27] you're not going to add a bias. [62:29] What you're going to do is you're [62:31] going to change the query and the key directly. [62:35] And so how are we going to change them directly? [62:38] What you're going to do is you're [62:40] going to rotate your query. [62:44] You're going to rotate your key, each of them [62:49] as a function of the position that they're in. [62:54] And why is it something that is very appealing? [62:57] Well, because the nice thing is when you rotate the query, when [63:02] you rotate the key, what you're actually doing, when you do [63:07] the self-attention formula is you're actually bubbling up [63:16] an expression that is a function of the difference between n [63:21] and m. [63:25] So we're going to see a little bit how that works. [63:27] So here, so I'm not sure if any of you [63:29] is an expert at geometry and rotation matrices. [63:35] But the way you rotate a vector is you take that vector [63:41] and you actually have a rotation matrix [63:45] that you apply to that vector. [63:48] So I'm going to maybe just write that down. [63:51] If I do have this. [63:53] So let's suppose you have some vector, let's call it x. [64:00] So you have, let's say you have two dimensions-- [64:08] let's assume you have x and y. [64:10] And let's assume that your vector is like this. [64:17] And this is let's say a radius of r. [64:22] So here the x-axis is x-- [64:26] sorry, it's R and here you have your theta angle, [64:32] cosine of theta. [64:34] And then here you have your y value, which is R sine theta. [64:42] So let's assume you have this. [64:48] So what I'm only doing is expressing [64:51] x as a function of some angle theta that it already has. [64:56] And what I want to show you is if you [64:58] were to multiply the rotation matrix with this vector x. [65:05] So let's assume you have a rotation matrix of let's say [65:09] angle phi, let's assume, which is-- so I'm going to just copy [65:14] what is on the slides. [65:15] So it's cosine of phi is minus sine of phi, sine of phi, [65:27] and then cosine of phi. [65:30] So let's assume you actually multiply your rotation matrix [65:37] with the vector x. [65:40] So for those of you who love matrix multiplications [65:45] and trigonometry formulas, this is the best case because I'm [65:51] going to exactly see. [65:55] So here you would do the matrix multiplication. [65:57] So R times cosine of theta cosine of phi [66:01] minus R cosine of theta-- [66:05] sorry, minus R sine of phi sine of theta. [66:10] What is this? [66:10] It's R cosine of theta plus phi. [66:15] And you guessed it, the other one is the same, but with sine. [66:30] So why am I writing all of this? [66:33] What I want to show you is if you multiply [66:36] your vector with the rotation matrix, [66:39] you're rotating your vector. [66:41] And this is the reason why we have this formula written [66:44] on the slide. [66:46] And the nice thing is, if you rotate your query vector [66:52] by some angle as a function of m, [66:56] and if you rotate your key vector [67:00] as a function of its position n, then [67:03] you obtain a rotation matrix that is actually [67:08] a function of n minus m. [67:11] So I will spare you the details. [67:13] But if you just do the multiplication, [67:16] you will see that it works. [67:18] And so the reason why it is pretty nice [67:22] is because this is extensible to any positions. [67:27] You're not learning anything. [67:30] And by the way, one other thing I will say is I just showed you [67:33] the 2D case. [67:34] But in order to generalize that let's say d-dimensional case, [67:41] you would actually do that by 2D blocks. [67:48] And you would do the rotation with angles that [67:54] are a function of the index. [67:56] So to your earlier question, so here you [67:58] would have also angles that would be a function, [68:03] let's say if it's a lower index, it's [68:04] a different angle compared to if it's a later index. [68:11] Cool. [68:12] And the nice thing with this method [68:14] is if you take a look at the order of magnitude of the dot [68:20] product between the query and the key [68:23] as a function of how far their positions are, then [68:27] you will see that roughly it tends to decay as m and n [68:33] are farther apart. [68:35] So it's not strictly monotonic because, well, it's [68:40] more complicated than that. [68:41] But at a very high level, it just decays [68:45] and it just reflects what we want in tokens [68:50] that are very far apart. [68:53] So that was RoPE. [68:56] So in other words, the original transformer paper [69:00] used position embeddings that were additive, [69:04] which is in contrast with nowadays where people use RoPE [69:09] as a way to encode position. [69:13] So this is how people use it. [69:16] And they don't use the additive position [69:21] that we discussed before. [69:24] So now we're going to talk about another component that [69:27] is a little bit different compared [69:29] to the original transformer, and this one is the attention layer. [69:34] So I mentioned that in the encoder part, [69:38] everyone can attend to everyone. [69:40] In the decoder part, a given token [69:44] can attend to itself and to tokens before it. [69:49] So one natural question that people have asked [69:52] is, well, do you really need to attend [69:56] to everyone at every layer? [70:01] Because it's a lot of computations. [70:04] And the answer is, well, maybe you don't need to do that. [70:09] And in particular, there is a specific variation [70:14] that is called the sliding window attention that [70:18] is actually meant for you to attend to only up [70:23] to a certain position that is farther away from you. [70:27] Not everyone, but only up to a certain level. [70:31] And so here it's something that people also [70:34] call local attention. [70:38] And what I want to tell you is you may-- so probably, you [70:43] will probably come back to me with saying, [70:44] OK, the RNN actually had this issue. [70:47] Well, why are you doing this? [70:48] So I want to point out something that I [70:53] hope will be making sense. [70:59] So your model is not just one block. [71:03] It's not just one block of the decoder. [71:05] It's actually several blocks. [71:07] So the attention operation is not just happening once. [71:12] It is happening several times. [71:15] So if you have, let's say, a sequence of tokens. [71:25] So I'm going to just replicate exactly these tokens over here [71:32] and let's say over here. [71:37] So let's assume that this token can only [71:47] interact with itself and with one token from before. [71:54] Here you only see yourself and the token before. [71:58] But if you consider that in a similar fashion [72:02] this token can attend to itself and the token before, [72:09] well, effectively, this token is able to see representations [72:15] from tokens that are beyond the limit. [72:22] Does what I'm saying making sense? [72:23] Yeah. [72:24] OK. [72:24] So in other words, even though you are limiting yourself [72:31] in the tokens that you're able to interact with, [72:35] if you have several stacked blocks, [72:39] then what you're actually able to do [72:41] is to interact with further tokens. [72:46] And for those of us who are also familiar with the vision world, [72:49] so in case you're familiar with convolutional neural networks, [72:53] it's a little bit the same idea as receptive fields. [72:58] So here when you perform the convolution, [73:02] a given element is actually a function of actually many pixels [73:08] if you go back into the chain. [73:10] So it's the same idea. [73:13] OK, I want to tell you something else about the attention layer. [73:18] So I think we briefly discussed it earlier in the lecture. [73:23] But OK, I'm going to reuse this. [73:29] So let's assume that we're back with our decoder [73:36] only model which is typically what LLMs are composed of. [73:41] And let's assume that what you want to do [73:47] is to predict the next token. [73:50] Well, this is your last predicted token. [73:53] In order to predict the next token, [73:55] you have a bunch of layers, but in particular in the attention [73:57] layer, what you're doing is you're attending to yourself. [74:01] You're attending to the token before, the token [74:04] before that, the token before that. [74:07] Let's assume you predict a new token [74:11] and that the new token, in order to predict the next token, [74:13] you're attending to yourself, you're [74:15] attending to the one before you, and so on and so forth. [74:21] So every time you want to predict the next token, what [74:27] you're doing is this causal attention, [74:32] where you're comparing the query from your token [74:37] with all the keys from the other tokens. [74:43] And every time you want to predict a new token, [74:45] you again compare it to the keys of the previous tokens. [74:52] So you could either compute all the previous keys every time [74:59] you do that operation, or you could store the keys [75:06] as you perform your decoding. [75:09] So, for instance, when I decode when I'm here, [75:11] I attend only to myself. [75:12] I computed the key. [75:13] You can save the key. [75:16] You predict this one. [75:17] So here I attend to here and to the one [75:20] before-- so I need the key. [75:21] I can reuse the key. [75:23] And I compute the key here that I save. [75:26] And I'm continuing this process so on and so forth. [75:29] So the problem with this is keys detect memory space. [75:37] And as your decoding gets longer and longer, [75:40] you need to store these keys. [75:43] And so if you remember, the attention layer [75:45] is actually a multi-head attention layer. [75:48] So it doesn't only have one key, it [75:50] has actually several keys for the same layer [75:53] for the same token. [75:55] So that is the reason why people have tried saving up [76:00] the number of keys and the number of values [76:03] that each token needs to produce. [76:05] And that's why you have this method. [76:07] So, for instance, the idea of sharing keys and values [76:13] across heads. [76:15] So the reason why people are interested in such methods [76:18] is because given that you always need the keys in order [76:23] to predict the next token, you want to save memory in there. [76:30] And so that is the reason why you have a number of methods [76:32] like group query attention, which [76:35] shares the key and the value projection [76:39] matrices across the heads, or the group query attention [76:44] which is a little bit more common, where [76:45] you have a certain number of groups where you share the key [76:51] and value matrices. [76:54] And this contrasts with the original multi-head attention [76:58] layer, which had one heads, where you had all the projection [77:03] matrices that you had to learn. [77:07] So all of that to say that given that you [77:11] are in this situation where you're always saving your keys [77:17] and your values for tokens that you have decoded [77:21] in the past, what you want is to not [77:24] have to save too much of them. [77:26] You don't want to saturate your memory. [77:29] So people are trying to find ways to optimize that. [77:33] And one such way is to just reduce [77:37] the number of different projection matrices [77:39] that you need to learn for the keys and the values. [77:43] And that is the reason why, I mean, [77:45] no one cares about the query because the query, [77:47] you use it once and then you don't need that. [77:52] So does that make sense? [77:56] So nowadays people use GQA quite a lot. [77:59] So among everything we've seen, you will see a RoPE quite a lot. [78:03] You will see local attention with sliding window quite a lot. [78:06] You will see GQA quite a lot. [78:10] And there is one last thing that I want to talk to you about, [78:13] which is the normalization. [78:17] So in the original transformer architecture. [78:21] So if you remember, so it's a little bit more [78:23] of a detail when you first read the paper. [78:25] But the way people do normalization is they first [78:31] perform the sublayer operation and then they add to the inputs [78:40] and then they normalize. [78:41] So they do normalization of x plus sublayer of x. [78:45] So it's also called post-norm. [78:48] So you're normalizing after the operation. [78:52] And the reason why people thought [78:55] it was maybe not the best way is because here [78:58] the idea behind residual connection [79:00] is that you want to let the input through just [79:05] for the gradient to flow better. [79:07] And so here the x is actually within the layer normalization. [79:11] So it's not great. [79:13] And so this is the reason why people nowadays, what they do [79:17] is they don't normalize after the operation. [79:19] They actually normalize before. [79:21] So they first normalize the inputs [79:24] and then they put it into the layer. [79:26] And then they add it to x. [79:28] It's also called pre-norm. [79:31] And it's just a way for the residual connection [79:33] to do its job of letting the gradient flow. [79:39] So you will see pre-norm as also a very common method. [79:42] And one last thing, I promise, which is the layer [79:47] normalization itself. [79:49] So in the original transformer, here we're [79:52] normalizing across all the activation by first recentering [79:57] x and then having a term, gamma, and then [80:02] some bias beta, which helps with training stability [80:06] and convergence. [80:08] But people have tried with other normalization methods. [80:11] And in particular, they have found that RMS norm-- [80:16] RMS stands for Root Mean Squared norm, [80:19] actually does the job just as well. [80:22] And here, you don't need to recenter x, [80:25] you just need to rescale it and to have just one [80:29] factor that you need to learn. [80:32] And that is the reason why people tend to use it. [80:36] And I think this was all the tricks I had. [80:38] So just as a summary. [80:41] Before on normalization front, people [80:44] used to do post-norm and layer norm, [80:47] and nowadays people do pre-norm and RMS norm. [80:52] So long story short, nowadays RoPE, local attention, gka GQA, [81:02] pre-norm, and RMS norm. [81:07] And with that, I'll give it to Shervin. [81:11] Thank you Afshin. [81:13] So now we're going to spend some time together [81:16] to talk about next token prediction. [81:19] And as Afshin has mentioned, LLMs [81:23] they use this autoregressive process [81:25] where you start with a given set of tokens that [81:28] have been processed, and then you generate the others [81:31] one at a time. [81:33] So let's say you start with just BOS. [81:37] So you generate A and then teddy, bear, is, and then [81:42] you complete the sequence. [81:44] So now the question is, how do you generate the next token? [81:48] So you know that all of these are processed [81:53] through these decoder layers. [81:55] So you have the self-attention mechanism [81:57] and the feedforward neural network [82:00] that is repeated n times because you have n layers of decoder [82:04] that are stacked. [82:06] But then at the end, it is projected on a linear layer [82:10] to predict the next token on the space of your vocabulary. [82:16] And there might be several ways to choose it. [82:20] So what could be a natural way? [82:25] So first, since you have a probability distribution, [82:28] one piece of intuition could be the probability distribution [82:32] has done a lot of work for me. [82:34] It says, what is the most probable token? [82:37] So let's just take it. [82:39] So that could be one. [82:42] But the thing is, as you compute tokens one after the other, [82:48] you will try to optimize the next token immediately. [82:53] But as you have generated the whole sequence, [82:56] you see that the global sequence might not be [82:59] with optimal joint probability. [83:02] So this is one weakness of this approach. [83:07] And you might be tempted to go the joint probability [83:11] route a bit more. [83:12] So this is why you have techniques like beam search [83:17] that doesn't do full joint probability because that [83:20] would be too computationally expensive. [83:23] But it does track k path at a time. [83:27] So let's say you want to compute the next token, [83:29] instead of only selecting the next one, [83:32] you will consider k path all the time. [83:36] So here you have beginning of sequence. [83:39] You have an there that are good like the two best ones. [83:44] And then you generate the next probability distribution, [83:47] and then you keep on keeping the k best and so on. [83:55] So who can tell me, what kind of weakness [83:58] you see out of these two techniques [84:01] that I haven't called out just yet? [84:07] Yes. [84:08] So there is a more computation. [84:11] Yes, for the second one, but not for the first one. [84:13] I'm looking for something that is common to both techniques. [84:18] So what can you say about the reproducibility [84:25] of choosing always the highest probability token? [84:29] It will always be the same. [84:32] What can you say about beam search? [84:35] Let's say you redo the computations at the same time. [84:38] Do you get the same path each time or will it change? [84:45] Yeah, great. [84:46] So yeah. [84:46] Same path and it's deterministic. [84:49] Great. [84:52] And this is where I want to bring your intuition on why that [84:56] may not be great, because as someone [84:59] who optimizes things for a living, you might say, [85:03] OK, I don't mind. [85:05] Determinism might be good. [85:07] But I just want to show you as a user [85:09] why you would care about it. [85:11] So let's suppose you are a chatbot user of all our favorite [85:16] LLMs, and you ask a given question, [85:20] like let's say a greeting message, "Good, and you?" [85:24] And let's say your friends ask the same message [85:29] and you have the same answer, and so on, you [85:32] might feel that the chatbot is robotic. [85:36] And robotic chatbots, I don't know about you, [85:38] but it's not so appealing because it feels like it's still [85:43] heavily engineered, whereas you as a human, [85:46] you might be tempted to respond a bit differently [85:49] to similar prompts. [85:51] So what I said here is mostly a thought experiment, [85:55] because in practice, the context of the LLM [85:59] might change quite a bit. [86:00] Let's say today we are the 2nd of October at Stanford. [86:03] If I ask this question one hour from now, [86:06] the context will change a bit. [86:08] It might put my new time and other pieces of information. [86:12] So this might not strictly apply. [86:15] But still you see this robotic behavior. [86:18] Whereas non-deterministic behavior [86:22] could give similarly well-sounding [86:26] answers as potential choices, which is great. [86:30] So I hope I convinced you that determinism is not the best. [86:36] So it can be good for some use cases. [86:38] I'll talk about it in a bit. [86:42] And one natural next step to combat determinism [86:47] is to go back to this probability distribution [86:51] and simply sample from it, because you have a probability [86:54] distribution, you have weights. [86:56] All our computational methods enable [86:59] us to sample the next word. [87:01] So let's just do that. [87:03] And you can do so with two methods, [87:08] because you might not want to deal with the whole distribution [87:11] all the time. [87:12] So you might want to select amongst top-k most [87:17] probable next tokens. [87:20] And the second technique that is used quite often [87:24] is top-p where you want to select [87:27] tokens that have a given probability [87:30] distribution in cumulate. [87:36] So I promised you that I would tell you [87:40] when determinism is needed. [87:42] So, for example, if you perform natural language tasks that [87:46] are tricky and need reproducibility, [87:50] for example, let's say you want to summarize a piece of text. [87:52] You want the same output all the time, that can be good. [87:55] In one of the later lectures, we're [87:57] going to talk about evaluations and particularly LLM [88:01] as a judge, where you want consistency between the ratings [88:04] that you give. [88:05] So these two are good examples. [88:12] In the rest of the time, this non-deterministic path [88:16] might be the most attractive. [88:20] So now I'm going to go into the specifics [88:24] that I mentioned on a high-level, [88:25] but making sure that it makes sense. [88:27] So in the last layers, you have this linear projection [88:31] between d model, the embedding space of your token [88:35] at a given time. [88:37] And then you project it on the space of vocabulary. [88:41] And then based on this, you apply this Softmax operator [88:47] to convert these numbers into a probability distribution. [88:52] And why am I telling you that? [88:54] It's because I want to introduce a parameter that you might have [88:59] heard as users quite bit-- the temperature [89:02] T, which helps modulate the probability [89:07] distribution that you get. [89:09] So here, as you see, so the regular Softmax operator [89:16] has T equals 1. [89:17] So the formula that you see on the right is T equals 1. [89:20] You introduce T like so. [89:23] I want to ask your intuition of how you feel the probability [89:29] distribution to be then. [89:31] So let's say you take T bigger than 1. [89:35] If you have a given probability distribution, [89:38] how do you feel the probability distribution will change? [89:46] So yeah, the one comment is flatter. [89:48] Yeah, great. [89:49] And let me ask you the opposite. [89:51] Let's say you have very small t. [89:53] What is your intuition? [89:55] Yeah and the comment is sharper. [89:57] Yeah, great. [89:58] So is it clear to everyone? [90:01] Because if you have small T, then the Xi that is high [90:06] will be more emphasized. [90:10] And the for the other i's, the some at the bottom [90:14] will take it over. [90:15] So you will have just one term-- [90:18] so you will have just one index that [90:21] will have a probability close to one [90:23] and the other ones will be close to 0. [90:25] And when T is large, then all of these x exponential [90:29] of something over T will tend to 0 within the exponential. [90:33] So everything will be more or less uniform. [90:37] And this is exactly what these graphs show. [90:40] And one thing, so this is a determinism [90:44] is continuous more so than truly like binary choice. [90:49] So you're not committed to either turn it on or off. [90:54] You can adjust the temperature as a setting. [90:58] And I want to say-- so I talked about determinism. [91:02] Does it really happen? [91:04] So you could think that it does, because when [91:06] you look at your decoder architecture, [91:09] you have embeddings that are learned. [91:11] OK, these are deterministically set at inference time. [91:14] It goes through the masked self-attention mechanism. [91:17] You have fixed weights, fixed operations. [91:20] You could think that's fixed, feedforward neural network. [91:22] You have given weights that are fixed. [91:25] You multiply. [91:25] This should be fixed and so on. [91:29] So with a temperature of 0, you would [91:31] think that this is completely deterministic. [91:34] But it turns out that not because in practice LLM [91:39] inference engines, they will batch your request among others. [91:44] And the reason why nondeterminism comes into play [91:48] is a purely computational one. [91:50] In theory, it should still be deterministic. [91:53] But there is this concept of batch invariance [91:57] that is not respected in some cases. [92:00] And I strongly recommend this blog [92:03] from [INAUDIBLE] as a great piece of reading. [92:07] So he goes through the exact reason. [92:10] And if you want me to break it down to you, [92:13] it all goes to the floating points non-associativity [92:19] with respect to the addition operation, [92:22] where the order of addition does matter on the result [92:29] that you get. [92:32] So let's say you add a big number to a small number [92:37] and then you retract that big number, you might get 0. [92:42] Whereas if you do that operation separately, [92:44] you will get that small number back. [92:46] So it all has to do with the representation of floating point [92:49] numbers that are approximated. [92:51] So this relationship does apply in the world of floating points. [93:01] Yeah. [93:05] So the comment here the tensor should [93:08] be computed independently. [93:10] So it turns out that depending on the implementation, [93:14] the order of operation within these tensors will change. [93:19] Yes, that is the reason. [93:23] So yeah, the blog has more information on exactly why [93:28] and then how to build kernels that [93:30] do apply this as great reading. [93:34] Any other questions? [93:39] OK, great. [93:41] And so now we talked about decoding in the usual case. [93:46] Sometimes, and more particularly in an industry setting, [93:50] you might want to use these LLMs to perform tasks you would [93:54] have used complex systems for. [93:57] So you might want to generate let's say structured output, [94:02] such as JSON. [94:04] And one question that you might ask yourself, [94:07] let's say you want to generate a piece of data [94:10] in this specific format and then this matters to you. [94:13] Because if it doesn't do it right, [94:15] then parsing fails downstream and you're in trouble. [94:18] How do you do so? [94:20] So one first approach that you could think about [94:23] is just ask the LLM for it, do this in JSON format [94:27] and hope for the best. [94:29] With today's models, even the lightest, [94:32] this usually works well. [94:35] But it's not 100% and what you want is 100%. [94:38] So how can you do that? [94:40] So there is a concept called guided decoding, [94:44] where the span of available tokens [94:48] to be predicted at the next step is restricted, [94:52] depending on the syntax of the language you [94:56] want to decode this in. [94:58] So let's say you are in JSON. [94:59] You know That your first token must [95:03] be a specific symbol and so on. [95:10] And this is how you could deal with it. [95:14] OK, so I hope all made sense. [95:17] And now we're going to talk about one exciting part, which [95:20] is prompting. [95:23] And I want to introduce some terms, [95:27] so that we all know some basics of vocabulary there. [95:32] So let's say you want to issue a request to an LLM. [95:36] Usually, you do not put just the BOS token [95:39] and let it generate everything afterwards. [95:42] You have a given prompt. [95:44] So here my prompt could be a teddy bear. [95:47] And I want to generate the next tokens out of it. [95:50] So what happens is that I will put all of them as input. [95:54] And then autoregressively, it will compute each token one [95:59] at a time. [96:01] And you might have heard of this concept called context window. [96:05] Where does it come at play here? [96:09] So at the stage of next token prediction, anywhere [96:14] you are, what is as input is your context window. [96:19] And when you look at your whole prediction, [96:23] once it's completely done, whatever tokens were generated [96:26] are called the output tokens. [96:30] So you might be tempted sometimes [96:31] to say that the context window is exactly what you [96:34] put as input as a user, but it actually is more than that, [96:38] because every token that you generate [96:41] needs to be put back at the input level, which [96:44] is why when you think about these large numbers, [96:47] you think I can input anything. [96:50] These are large numbers as context window, [96:52] as we'll see in the slide. [96:54] You need to take that into account. [96:56] So it's linked with output length. [97:00] Yep. [97:02] So the question is, why if you need [97:05] to have a fixed context length, like a maximum context length? [97:08] So the reason is a computational one. [97:10] And we'll see very soon why it might not even [97:14] be desirable to have higher context length. [97:17] So there is also an intuition based one reason. [97:21] We'll it in seen in a second. [97:25] That makes sense? [97:27] So I want to give some orders of magnitude here. [97:30] So in terms of context length, you [97:33] might all know about that O of 1 million tokens [97:36] as input, which is the industry standard. [97:39] And then as output tokens, it's usually in the order [97:43] of magnitude of 100k. [97:46] That was true until two days ago where [97:49] I had to rework this slide. [97:51] Actually, maybe soon, we're going [97:52] to move into the million territory there as well. [97:55] And this new LLM that came out didn't publish an input maximum [98:00] length. [98:01] So maybe we might have gone one order of magnitude higher there. [98:05] But these are some good orders of magnitude. [98:08] And then to give you some intuition on what token means. [98:13] So usually, a token is 3/4 of a word and a page is about 500 [98:21] tokens. [98:22] So when you talk about orders of magnitude, [98:24] if you take 100k tokens, it's about a book, and a million, [98:29] it's a large book. [98:33] And now coming back to your question, [98:35] is it the case that always having more tokens as input [98:39] is better? [98:40] Well, you need to beware of a phenomenon that's called [98:43] "context rot," where there is this test called needle [98:48] in a haystack, where you put a piece of information that you [98:51] care about somewhere in that context window and the task is [98:55] a retrieval one. [98:56] And it turns out that as the context length goes higher, [99:01] the LLM has more and more difficulty [99:04] to retrieve that piece of information. [99:07] So sometimes it's a problem that comes from the LLM itself. [99:11] So you will always see in the model releases [99:14] the hallucination rate that goes down and so on. [99:18] But this is not something that you could perfectly [99:21] solve, because conceptually, when [99:24] you mix a piece of information, among others, [99:27] it might be the case that there is so much noise [99:29] around that the task of retrieving [99:31] exactly that piece of information is not realistic. [99:36] And in general, putting more and more as inputs [99:41] pushes that risk up, which is why in general you [99:45] hear about context compaction. [99:47] You should start a new chat every time. [99:49] These things still apply conceptually. [99:52] And this is why we are not here with unlimited context window. [99:57] It's also like a good practice. [100:00] OK, awesome. [100:02] And I'm going to spend the last minutes on techniques [100:05] to improve performance without you changing [100:08] any weights of your model. [100:11] So there is a large concept called in-context learning, [100:16] where instead of teaching in terms of weights tuning [100:21] your LLM, you give examples and solutions of what you want. [100:27] So let's say you are in the task of asking your LLM about the age [100:33] of your teddy bear. [100:35] So if you just ask it in zero shot versus give some question [100:40] answers of what answer you expect, [100:43] the LLM will learn the format and maybe even the reasoning [100:49] that it needs to go through implicitly. [100:53] And these days, you will see few shot learning [100:57] being less and less used because the reasoning models, [101:00] they get better and better. [101:02] And the bad effects of few shot learning [101:05] is that it will make your model overfit on the specific examples [101:09] that you give it, whereas what you want [101:11] is be more reasoning-based. [101:13] So you will see some prompt tuning techniques that [101:17] turn these examples into instructions, [101:20] and people typically handcraft such instructions [101:23] and put them instead of examples. [101:26] And then again, some other obvious downsides are also [101:31] you have more input tokens, which [101:34] increases cost and latency. [101:36] So on that reasoning part, you might [101:38] say, how about we put some reasoning in there? [101:42] So there is this landmark paper called chain of thought, [101:48] where it takes in-context learning to the next stage [101:52] by teaching the model in these few short prompts how [101:57] it gets there. [101:59] And the authors, they saw that the model learned the reasoning [102:04] path, and overall it improved performance. [102:08] And I don't want you to consider chain of thought [102:11] to be purely an in-context learning technique. [102:17] It's way more than that. [102:18] And we will talk about reasoning models [102:20] where thinking tokens are used and the same concept applies. [102:24] So we'll talk about it then. [102:26] And one technique that one might be tempted to use offline [102:31] is called self-consistency. [102:33] So how about, instead of generating one reasoning path, I [102:38] generate several ones? [102:40] Then one nice thing is that it makes the output that you might [102:44] be interested in, let's say in the arithmetic world, [102:47] more stable. [102:49] And by majority voting, you can get to an answer [102:53] where more generations agree to. [102:56] Yeah, we'll see in reasoning models lectures, more [103:00] about these. [103:02] And that's all I had for you today. [103:03] And I hope you have a great weekend. [103:07] Thank you. [103:08] [APPLAUSE]