Optimizing AI Agents: Prompts vs. Weights (GEPA, GRPO) | Stanford CS329Z Lecture 9 with Michael Ryan

Optimizing AI Agents: Prompts vs. Weights (GEPA, GRPO) | Stanford CS329Z Lecture 9 with Michael Ryan — Stanford Online

Stanford Online

AI-assisted summary of Optimizing AI Agents: Prompts vs. Weights (GEPA, GRPO) | Stanford CS329Z Lecture 9 with Michael Ryan — original source. Check the original for full context.

Source published:

Key ideas

  • Context optimization is primarily a mechanism for last-mile functional specification and discovering unstated workflow constraints rather than mere semantic phrasing tweaks. (5:18)
  • Group Relative Policy Optimization (GRPO) replaces the memory-intensive critic network of PPO with group-level z-score advantage estimation, substantially reducing GPU overhead in reasoning models like DeepSeek-R1. (62:53)
  • On-policy distillation outperforms standard SFT and pure RL on the AIME benchmark (achieving 74% accuracy) by using reverse KL divergence to provide fine-grained, token-level teacher feedback during student rollouts. (72:56)
  • Prompt optimization demonstrates substantially higher sample efficiency than reinforcement learning, making an alternating schedule—optimizing prompts, fine-tuning weights, and re-optimizing prompts—the most effective path for compounded performance gains. (78:51)

Detailed notes

## Background This lecture was published on YouTube by Stanford Online on November 17, 2026, as part of the Stanford CS329Z: Engineering AI Agents course recorded on October 21, 2026. Presented by DSPy contributor and Stanford CS PhD student Michael Ryan, the talk covers whether to optimize an AI agent's prompts or fine-tune its weights, diving into context optimization, weight optimization techniques like SFT, PPO, and GRPO, on-policy distillation, and autonomous self-optimizing agents.

## TL;DR Michael Ryan breaks down the principles and trade-offs of engineering AI agents through context optimization versus weight optimization. He examines in-context methods such as bootstrapped few-shot selection, OPRO, MIPROv2, TextGrad, and GEPA, illustrating how reflection and Bayesian search solve last-mile specification challenges. He then contrasts these with parameter-tuning strategies, exploring SFT on teacher rollouts, the memory efficiencies of GRPO over PPO, and on-policy distillation as implemented in models like DeepSeek-R1 and Qwen. Finally, Ryan explores agents acting as self-improving optimizers through frameworks like the Darwin Gödel Machine before offering a practical decision framework for choosing between prompts and weights.

## Atmosphere & style The lecture carries an engaging, grounded academic tone blending technical depth with self-deprecating humor and pragmatic engineering insights. Ryan speaks with the fluid authority of a researcher who has built and deployed these optimization algorithms at scale, freely sharing candid anecdotes about model quirks and unglamorous design realities. It feels like an advanced, highly practical masterclass where theoretical machine learning abstractions are continually anchored in real-world GPU bottlenecks, benchmark results, and system trade-offs.

## Bulletin - [05:18](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=318s) Context optimization is primarily a mechanism for last-mile functional specification and discovering unstated workflow constraints rather than mere semantic phrasing tweaks. - [01:02:53](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3773s) Group Relative Policy Optimization (GRPO) replaces the memory-intensive critic network of PPO with group-level z-score advantage estimation, substantially reducing GPU overhead in reasoning models like DeepSeek-R1. - [01:12:56](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=4376s) On-policy distillation outperforms standard SFT and pure RL on the AIME benchmark (achieving 74% accuracy) by using reverse KL divergence to provide fine-grained, token-level teacher feedback during student rollouts. - [01:18:51](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=4731s) Prompt optimization demonstrates substantially higher sample efficiency than reinforcement learning, making an alternating schedule—optimizing prompts, fine-tuning weights, and re-optimizing prompts—the most effective path for compounded performance gains.

## Key takeaways - [03:34](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=214s) AI system optimization depends on three non-negotiable pillars: representative rollout data, the pipeline or model architecture, and a verifiable metric or evaluation framework. - [01:26](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=86s) Engineering optimal prompts cannot rely on human intuition alone, as unintuitive or non-standard instructions frequently outperform traditional human-crafted prompts. - [10:26](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=626s) Few-shot demonstration ordering significantly impacts model accuracy, and optimal demonstration orderings can be identified without ground-truth labels using output entropy metrics. - [17:08](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=1028s) DSPy and the STaR framework bootstrap reasoning rationales in-context using rejection sampling on correct task outcomes without updating model weights. - [27:40](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=1660s) MIPROv2 optimizes multi-stage agent pipelines through a three-stage framework combining demonstration bootstrapping, grounded instruction generation, and Bayesian search over prompt combinations. - [38:02](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=2282s) TextGrad implements backpropagation for LLM pipelines using natural language critiques passed upstream through computational graphs instead of numerical derivatives. - [40:20](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=2420s) GEPA employs genetic evolution along a Pareto frontier across diverse task benchmarks, utilizing reflective mutation and module merging to prevent optimization collapse. - [51:26](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3086s) Supervised fine-tuning (SFT) for agents post-trains student models on complete teacher execution trajectories, including thoughts, tool calls, observations, and verifications. - [53:32](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3212s) Selecting an overly capable teacher model can degrade distillation performance if the capability gap prevents the student model from effectively mimicking latent reasoning steps. - [59:24](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3564s) Proximal Policy Optimization (PPO) introduces severe GPU memory bottlenecks in LLM training by requiring concurrent hosting of policy, value, reference, and reward models. - [01:02:53](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3773s) GRPO eliminates the dedicated value model by normalizing scalar trajectory rewards across a group of sampled rollouts for each prompt. - [01:06:56](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=4016s) Reinforcement learning is fundamentally required to push frontier model capabilities beyond existing human expert limits because no teacher exists to supply purely supervised demonstrations. - [01:09:37](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=4177s) Standard off-policy SFT fails to teach models how to recover from their own early errors, whereas pure RL provides sparse rewards that struggle to assign credit to specific failure-inducing tokens. - [01:15:43](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=4543s) Self-modifying agent systems like the Darwin Gödel Machine and Andrej Karpathy's auto research demonstrate that agents can autonomously optimize their own harness code and experimental configurations. - [01:18:51](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=4731s) Context optimization is best suited for black-box API models and dynamic workflows, whereas weight fine-tuning is preferable for open-weight models requiring latency reduction and domain-specific stability.

## Detailed walkthrough ### [00:05](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=5s) Introduction, Announcements, and Reverse-Engineering Keynote

Michael Ryan opens the lecture by expressing his excitement for the topic of optimization, noting that of all the work he contributed to the DSPy framework, optimization was the part he worked on most intimately. As an introductory announcement, he reminds the class that the upcoming Monday session will feature a guest lecture on TerminalBench delivered by two speakers.

He then shares an anecdote from his preparation process where he got to “feel the AGI a little bit.” While putting together the lecture slides, he wanted to incorporate material from a talk he previously gave on one of his research papers. However, the original slide deck was formatted completely differently. When he asked co-instructor Diyi Yang whether shifting visual formats mid-presentation would be acceptable, Diyi suggested simply handing the task over to OpenAI’s Codex. Michael, who designs his presentations in Apple Keynote, was amazed by the result: Codex independently reverse-engineered the proprietary file storage structure of Keynote and generated code that transformed the presentation directly into the standard slide format used for CS329Z. As Michael Ryan puts it: “So Codex went and reverse engineered how Keynote stores the the data for your presentations and wrote code that was able to convert it to the format of slides that we use for this class.”

- Next Monday features a guest lecture covering TerminalBench with two visiting speakers. - Codex successfully inspected Keynote’s underlying data format and converted the researcher’s external presentation slides into the course template.

---

[01:26](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=86s) Prompt Optimization Poll: "Take a Deep Breath"

To motivate the unpredictable nature of prompt engineering, Michael conducts an interactive class poll. He asks students to identify which of three suffix prompts yields the highest performance on math reasoning problems for Google's instruction-tuned PaLM 2-L model. The three candidate prompts are: - A: `"Let's think step by step"` - B: `"Take a deep breath and work on this problem step by step."` - C: `"Let's combine our numerical command and clear thinking to quickly and accurately decipher the answer."`

A show of hands splits the classroom almost perfectly into thirds across options A, B, and C. Michael reveals that option B is actually the best-performing prompt for that model. He points out how unintuitive this outcome is, noting that an engineer has no clear principled reason or a priori knowledge to expect telling a model to breathe to outperform standard chain-of-thought phrases. Because intuition fails, developers cannot guess their way to optimal systems; they must build evaluations, run the options empirically, and measure what works. Explaining the conceptual shift, Michael Ryan observes: “So optimization enables us to specify intent through data and eval which allow us to improve our system.” This replaces the brittle paradigm of manually hard-coding prompts and workflows with an automated, metric-driven process.

---

[03:34](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=214s) Lecture Roadmap and the Three Pillars of Optimization

Michael outlines the day's agenda and learning objectives: identifying when to apply optimization to AI architectures, detailing common context and weight optimization algorithms, and discerning when to apply each strategy in practice. The lecture is structured into five core modules: the foundational conditions required to optimize an AI system, context optimization techniques (prompt and demonstration selection), weight optimization (reinforcement learning and distillation methods), agent-driven optimization frameworks, and a concluding decision guide for practitioners.

```mermaid flowchart TD accTitle: The Three Pillars of AI System Optimization accDescr: Diagram showing the three interconnected requirements needed to optimize an AI system: Data, System, and Metric.

subgraph Pillars ["Foundations of AI System Optimization"] D["Data<br/>Rollouts, demonstrations, human feedback"] S["System<br/>Language model, agent, workflow"] M["Metric / Eval<br/>Pass rate, exact match, LM judge, cost, latency"] end

D -->|"drives tuning"| S S -->|"produces output"| M M -->|"provides feedback"| D ```

Before diving into algorithmic details, Michael revisits the core paradigm introduced in earlier framework lectures: the three pillars of an AI system. Optimization is only possible when all three pillars exist simultaneously: 1. **Data:** Training rollouts, demonstration trajectories, and human preference feedback. 2. **System:** The underlying language model, programmed workflow, or agentic control loop. 3. **Metric / Evaluation Benchmark:** Quantitative objective signals, ranging from pass rates, exact match, and LLM-as-a-judge scores to operational constraints like cost and latency.

Emphasizing that optimization cannot occur in a vacuum without this infrastructure, Michael Ryan states: “You need to first build your system, collect data that you're going to use to optimize a system and have some way of measuring the impact of your optimization.”

---

[05:18](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=318s) Context Optimization and Last-Mile Adaptation

Addressing a common industry skepticism, Michael tackles the question of whether prompt and context optimization remain necessary in an era where foundation models continually scale in capability. Skeptics often argue that engineers will no longer need to fiddle with prompt wording as base models improve. Michael acknowledges part of this premise: simple prompt fiddling is indeed becoming obsolete as models get smarter. However, he distinguishes superficial prompt tweaking from true context optimization.

Michael argues that context optimization is fundamentally about solving “last-mile adaptation.” While foundation model providers build models that excel across general domain benchmarks, real-world deployment requires an agent to satisfy the idiosyncratic, bespoke requirements of a specific business logic or user workflow. Developers can attempt to bridge this gap by manually hand-authoring every nuance of their intent into a massive system prompt, but that approach is fragile. A far more robust method is establishing a rigorous evaluation suite that defines operational success, and then optimizing the context to maximize that benchmark. Explaining why he prefers precise terminology, Michael Ryan states: “I'm calling it context optimization cuz I do think prompt optimization calls to mind things like fiddling with the exact wording of things and that is less necessary with language models today.”

- Foundation models scale well generally, but they cannot anticipate the bespoke rules of proprietary enterprise systems without last-mile adaptation. - Context optimization uses systematic evals rather than manual prompt fiddling to guide model behavior toward 100% compliance on domain benchmarks.

---

[07:26](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=446s) When to Optimize: Capturing Hidden Intent and Multi-Component Systems

Michael outlines the concrete conditions under which automated optimization pays off. Practitioners should turn to optimization when they possess domain-specific data and evals, especially in situations where edge cases, formatting constraints, and user expectations are difficult to articulate exhaustively in a natural language prompt.

To illustrate, he references an email parsing agent designed to extract dates and times from email threads to create calendar events. While a developer can start with a trivial prompt like `"extract the date and time"`, true system intent contains complex hidden constraints: - The output format must match an exact ISO-compliant schema that downstream calendar APIs can parse without crashing. - The system must have an explicit policy for handling emails containing multiple competing events.

Instead of anticipating and writing hundreds of edge-case rules into prompt instructions, developers can let context optimization discover these hidden expectations automatically from evaluation feedback. Furthermore, when an agent consists of multiple interconnected modules, tuning individual prompts by hand becomes intractable. As Michael Ryan summarizes: “This is an instance of where context optimization be can be useful because we can learn the underlying hidden parts of your intent and put that into the system.” Ultimately, automated optimization delivers significantly higher system reliability across complex pipelines.

[09:07](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=547s) Conditions for Optimization: Latency, Cost, and Adaptability

Optimization serves several distinct goals beyond basic task accuracy, starting with system reliability to ensure an agent behaves consistently across every execution. Another core goal is domain adaptation to specialize general-purpose models for targeted tasks. Optimizing context or weights also delivers dramatic latency and cost reductions: developers can leverage a larger, more capable model to teach or format tasks so that a much smaller model achieves equivalent performance at a fraction of the operating cost. This connects directly to weight distillation, where a high-capacity teacher model is compressed into an efficient student model.

```mermaid flowchart LR accTitle: System Re-Optimization Cycle across Model Generations accDescr: A system built on an older LLM version is re-optimized using a principled evaluation suite when newer LLM versions arrive. subgraph Generation1["Initial Deployment"] A["Agent Pipeline (e.g., Calendar Tracking)"] --> B["LLM Generation N (e.g., GPT-5.4)"] end subgraph Generation2["Continuous Adaptation"] C["New LLM Released (e.g., GPT-6.1 / GPT-7)"] --> D["Automated Eval Suite"] D --> E["Re-Optimize Agent System"] end B -.->|"Model deprecation / upgrades"| C ```

Furthermore, optimization enables systems to learn from experience over time and adapt dynamically when underlying dependencies shift. Michael Ryan illustrates this with a forward-looking scenario: if an engineering team builds an entire calendar tracking application around GPT-5.4, and OpenAI subsequently releases GPT-6.1 or eventually GPT-7, the application should not break or degrade. Michael Ryan emphasizes that “if we've set up good evals, we can reoptimize when a new model comes out and ideally we're just getting the best performance we can at every generation.” With these core optimization conditions established, the focus transitions to context optimization algorithms.

[10:26](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=626s) Few-Shot Prompting and Prompt Ordering Sensitivity

Context optimization begins with few-shot optimization, a concept rooted in foundational research establishing that large language models are capable few-shot learners. Rather than running gradient updates to modify network weights, practitioners supply a task description (such as translating English to French) along with several input-output demonstration pairs within the prompt. As an aside, Ryan jokes about his attempt to pronounce the French translation of "sea otter" (*loutre de mer*): “well I don't actually speak French but I'm assuming this is like loutre or something like that... Apologies if uh if you do speak French.” With these demonstrations provided, the model infers the target behavior and improves its performance on unseen inputs.

```mermaid flowchart TD accTitle: Permutation and Probing Search in Fantastically Ordered Prompts accDescr: Demonstrations are permuted into diverse prompt candidate orders, evaluated across a probing set, and ranked via entropy metrics without ground-truth labels. subgraph Candidates["Demonstration Permutations"] T["Training Examples"] --> P1["Order Permutation A"] T --> P2["Order Permutation B"] T --> P3["Order Permutation N"] end subgraph Probing["Probing / Dev Set Evaluation"] P1 --> LM["Frozen LLM"] P2 --> LM P3 --> LM LM --> M["Label Distribution & Logprobs"] end subgraph Scoring["Metric Evaluation"] M --> GE["Global Entropy (Spread across set)"] M --> LE["Local Entropy (Token-level logprobs)"] GE --> Best["Select Best Prompt Ordering"] end ```

To motivate how prompts are engineered, Ryan highlights the influential 2022 paper *"Fantastically Ordered Prompts and Where to Find Them."* The authors identified a striking sensitivity in LLMs: the exact sequence in which few-shot examples appear inside a prompt dramatically changes model accuracy. Using a binary sentiment classification task over movie reviews (e.g., mapping text like *"has a way of seeping into your consciousness"* to positive sentiment), the authors sought an algorithmic method to identify the optimal permutation of training exemplars.

The experimental setup permutes the ordering of few-shot demonstrations in the prompt, passes them into a frozen pre-trained language model, and evaluates inference over a held-out "probing set" (equivalent to a dev set) one review at a time. Interestingly, this method does not rely on ground-truth probing labels to select the prompt; instead, it optimizes purely over the statistical distribution of the model's generated outputs.

[14:00](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=840s) Entropy-Based Prompt Selection: Global vs. Local Metrics

To choose the optimal demonstration ordering without consulting gold labels, the authors evaluate two primary metrics: global entropy and local entropy. Global entropy measures the compressibility of the output label distribution across the entire probing set:

- **High global entropy**: The model outputs a balanced, diverse mix of predictions across the dataset (e.g., labeling *"the ending is universally panned"* as 0, *"features multiple endings"* as 1, *"hard to sit through"* as 0, and *"nice movie"* as 1). High entropy indicates a rich, non-degenerate distribution that is difficult to compress into a naive heuristic. - **Low global entropy**: The model collapses into majority-class bias, outputting the exact same label regardless of the input text (e.g., answering 0 to every single probe).

Local entropy, by contrast, examines the model's token-level log probabilities ($\text{log probs}$) on individual instances. If the model assigns nearly equal probability to token `0` and token `1` for *"the ending is universally panned"*, the local entropy on that specific example is high. If it places extreme confidence on a single token, local entropy is low. While low local entropy makes sense for an unambiguous, obvious test instance, the authors hypothesized that maintaining relatively high local entropy averaged over the whole probing set prevents overconfident, brittle predictions.

Evaluating these metrics across model sizes like GPT-3 2.7B and 175B (referencing the older naming era of Ada, Babbage, and DaVinci), the results revealed that global entropy served as the most effective unsupervised heuristic for selecting prompt orders. Except for the theoretical "Oracle" setting (which directly evaluates against ground-truth validation labels to identify the upper bound), global entropy consistently surpassed random baseline orderings, producing accuracy improvements as large as 8%.

[17:08](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=1028s) Bootstrapping Reasoning: Rejection Sampling and the STaR Framework

Moving beyond fixed classification tasks, few-shot prompts often require intermediate rationales. While Chain-of-Thought (CoT) prompting significantly improves reasoning fidelity, real-world data typically contains labeled input-output pairs without step-by-step reasoning rationales. In sentiment analysis, a dataset might map *"the ending is universally panned"* directly to `0`, omitting the implicit rationale that *"panned"* is an intensely negative term signaling audience dislike.

```mermaid flowchart TD accTitle: STaR Bootstrap Reasoning Loop accDescr: An LLM generates a rationale and answer; correct solutions are filtered via rejection sampling and fed back to fine-tune or prompt future iterations. Q["Unreasoned Question / Input"] --> LLM["LLM"] LLM --> R["Generate Rationale + Answer"] R --> Eval{"Check Ground Truth Answer"} Eval -->|"Correct"| Keep["Keep Reasoning Trace (Rejection Sampling)"] Eval -->|"Incorrect"| Drop["Discard or Re-prompt with Hint"] Keep --> Prompts["Augment Few-Shot Exemplars / Fine-Tune"] ```

To bridge this gap, self-taught reasoning algorithms extract and validate reasoning paths automatically. Ryan introduces the STaR (Self-Taught Reasoner) framework, emphasizing that “we can rejection sample our way to mining reasoning traces.” Under this architecture: 1. An input question (for instance, *"What can be used to carry a small dog?"* paired with multiple choices) is fed to the language model. 2. The language model generates both a rationale and a final answer choice. 3. The generated answer is verified against the ground-truth target. 4. When the model outputs the correct answer, the accompanying reasoning rationale is saved as a verified demonstration trace.

Through this rejection-sampling loop, the system autonomously bootstraps chain-of-thought demonstrations from unlabeled or unreasoned problem sets, filtering out flawed reasoning paths and accumulating high-quality rationale traces.

[18:25](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=1105s) Bootstrapping Chain-of-Thought Reasoning Without Fine-Tuning

Michael Ryan explains the STaR (Self-Taught Reasoner) paradigm for bootstrapping rationales: given an input, the model outputs a question rationale and a multiple-choice answer. If it answers correctly, the triplet (question, rationale, answer) becomes a high-quality training example to fine-tune the language model. If the model gets the wrong answer, it is provided a hint. In the original paper, the authors literally append `"hint: this is correct"` beside the gold answer, prompting the model to generate backward reasoning toward that provided ground truth. Ryan quips with a laugh that after giving away the right answer, the model should certainly arrive at the correct result, “otherwise you're working with a really stupid model.”

While that earlier line of work leans entirely on parameter weight updates via supervised fine-tuning (SFT), Ryan emphasizes that context optimization allows developers to reap the exact same benefits without touching model weights at all. Citing the 2023 DSPy paper, he outlines how bootstrapped few-shot optimization operates purely in-context:

```mermaid flowchart TD accTitle: DSPy Bootstrapped Few-Shot Search accDescr: An input passes through the LM to generate rationale and output, which is evaluated by a metric. Successful traces are kept, while failures are discarded. Combinations of successful traces are then sampled and evaluated over a dev set.

subgraph Generation ["1. Rejection Sampling Rollouts"] A["Input Data"] --> B["LM Pipeline"] B --> C["Generate Rationale & Output"] C --> D{"Evaluate with Metric"} D -- "Pass" --> E["Saved Trace Example"] D -- "Fail" --> F["Discard Trace"] end

subgraph Search ["2. In-Context Dev Set Search"] E --> G["Candidate Pool of Traces"] G --> H["Sample K Few-Shot Examples"] H --> I["Evaluate Program on Dev Set"] I --> J{"Best Score?"} J -- "Yes" --> K["Optimal Few-Shot Prompt"] J -- "No" --> H end ```

Rather than fine-tuning the base weights, the system runs raw inputs through the language model pipeline, generating intermediate reasoning traces and final outputs. A deterministic or programmatic metric tests the rollout. High-scoring instances are retained as demonstrations, while failed rollouts are discarded. These vetted traces are then dynamically slotted into the prompt context to find combinations that maximize performance on a validation split.

- **Rejection Sampling**: Only model rollouts that achieve a passing metric score are retained as candidate few-shot exemplars; bad outputs are pruned immediately. - **Bootstrap Random Search**: Candidate exemplars are repeatedly sampled at random, stuffed into context, and evaluated over the development set to identify the highest-performing prompt combination. - **Superiority over Human Baselines**: Empirical results from the DSPy paper demonstrate that automated reasoning bootstrapping outperforms standard zero-shot, few-shot without reasoning, and even human-written chain-of-thought demonstrations. - **Handling Demonstrations vs. Labels**: Ryan clarifies that while the system can bootstrap vanilla input-output labels, its primary value in chain-of-thought setups is bootstrapping the synthetic reasoning steps.

Responding to an audience question asking what is actually being interchanged across combinations, Ryan explains the mechanics of the search step:

> “All you're doing is changing what few shots examples are there. And the way that we got the few shot examples was this rejection sampling.”

If rejection sampling produces a candidate pool of 30 successful traces and a hyperparameter dictates $k = 6$ demonstrations in context, the algorithm randomly samples subsets of six, measures full program accuracy over the dev set, samples another six, and continues the search until it converges on the top subset.

---

[21:50](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=1310s) Reflective Optimization and Google DeepMind's OPRO

Ryan transitions from few-shot demonstration search to a broader family of techniques he groups under the umbrella of "reflective optimization." He defines this paradigm as running an agent or pipeline, systematically reflecting on failure modes and successes, and iteratively refining instructions based on that analysis.

The foundational study in this space is Google DeepMind's 2024 paper *Large Language Models as Optimizers* (OPRO). The OPRO framework turns prompt engineering into a meta-prompt optimization problem:

```mermaid flowchart LR accTitle: OPRO Optimization Loop accDescr: Candidate instructions are scored on a dataset, and historical instruction-score pairs are fed to a meta-prompt LM to generate improved candidate instructions.

A["Candidate Instruction"] --> B["Run Evaluation on Task Dataset"] B --> C["Compute Accuracy Score"] C --> D["Meta-Prompt History<br>(Instructions + Scores in Ascending Order)"] D --> E["Optimizer LM<br>(Generates Novel Prompt)"] E --> A ```

In OPRO, an optimizer LLM receives a history of previously attempted system instructions alongside their resulting benchmark accuracy scores, arranged in ascending order. The meta-prompt instructs the model to inspect past attempts, discern what stylistic or instructional patterns correlated with higher metrics, and write a brand-new instruction wrapped in brackets that is designed to score even higher. Ryan notes that this core mechanic—simply feeding execution history and scores back into an LLM—presaged modern agent design:

> “And so it's a very simple idea. Just throw everything at a language model and see what it does. And you'll see actually uh maybe this paper was pretty prescient because this comes back with agents. Just throw everything at them and and see what it does and it starts to work.”

OPRO demonstrated that automated prompt generation uncovered counterintuitive instructions that consistently outperformed human prompt engineering: - The standard baseline phrase from the original Chain of Thought paper, `"Let's think step by step"`, yielded an accuracy of 71.8% on target reasoning benchmarks. - When PaLM 2 was employed as the optimizer to refine instructions for itself, it produced the famous phrase `"Take a deep breath and work on this problem step-by-step"`, which achieved the top evaluation accuracy on math benchmarks. - When GPT-4 was tasked as the optimizer, it generated a dense, numerical command-oriented instruction that also surpassed human-crafted chain-of-thought baselines, though it landed slightly below PaLM 2's `"take a deep breath"` prompt.

---

[24:54](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=1494s) Evaluating Non-Determinism and Prompt Optimization Intuitions

A student in the lecture hall asks how OPRO controls for LLM non-determinism during prompt evaluation, especially when running over external APIs. Ryan makes a distinction between internal frontier research and real-world deployment:

- **Direct Model Access**: Because OPRO was published by Google DeepMind, the researchers ran evaluations using direct, local access to PaLM 2 checkpoints rather than public black-box APIs, allowing tighter control over decoding parameters like temperature. - **Handling API Variance**: For practitioners restricted to commercial APIs where non-determinism, backend routing, or silent model updates occur, the operational standard is running repeated trials across the evaluation set to explicitly measure and bound score variance.

Another audience member asks whether an underlying theoretical intuition explains why prompt optimization yields such consistent empirical gains. Ryan splits his answer into historical perspectives versus modern agent workflows:

1. **The Early Era (Distribution Matching)**: When prompt engineering centered on finding eccentric phrasing like `"take a deep breath"`, the prevailing hypothesis was that models write prompts closer to their native training distribution. If a developer supplies phrasing that is out-of-distribution relative to the pretraining corpus, the model underperforms. Prompt optimizers essentially conducted an automated search for the semantic subspace that most effectively activated the base model's latent capabilities. 2. **The Modern Era (Last-Mile Specification)**: Contemporary prompt optimizers—such as MIPROv2, TextGrad, and GEPA—focus on functional specification rather than esoteric keyword matching.

Ryan highlights that real production agents fail not from a lack of base intelligence, but from omitted edge cases and formatting requirements:

> “Now the prompt optimization algorithms are more about getting that last mile specification. And that's really why we do it. It's like you may have forgotten to say you need to do ISO 8601 format or whatever in the prompt, but when we run it, it can reflect and say I failed because I output the date in the wrong format, so I need to incorporate that in the prompt.”

[27:40](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=1660s) Introduction to MIPROv2 and Bootstrapped Few-Shot Generation

Wrapping up a brief aside on papers treating few-shot learning as a Bayesian process, Michael Ryan transitions to the next major method: MIPROv2. Acknowledging the playful name, Ryan highlights that this was one of the research papers he personally led. The algorithm operates through three sequential phases: bootstrapping demonstrations via rejection sampling (running the program multiple times to collect successful traces), proposing instruction candidates by placing comprehensive context about the pipeline into a prompt-generating language model, and searching over combinations of instructions and demonstrations using a Bayesian search rather than a naive random search.

```mermaid flowchart LR accTitle: MIPROv2 Three-Stage Pipeline accDescr: Pipeline showing data moving through bootstrap demonstration collection, grounded instruction proposal, and Bayesian search over prompt combinations. subgraph S1["1. Bootstrap Demos"] D1["Run LM Program"] --> D2{"Eval on Metric"} D2 -->|"Score Pass"| D3["Save Trace"] D2 -->|"Score Fail"| D4["Discard"] end subgraph S2["2. Instruction Proposal"] P1["Program Code"] & P2["Training Data"] & P3["Saved Traces"] & P4["Prompt Tips"] --> P5["Proposal LM"] P5 --> P6["Instruction Candidates"] end subgraph S3["3. Joint Optimization"] D3 & P6 --> O1["Bayesian Surrogate (TPE)"] O1 --> O2["Optimized LM Program"] end ```

To make the mechanics concrete, Ryan walks through an example built on the classic Iris flower classification dataset, often found in standard packages like scikit-learn. The program consists of a single DSPy language model call (though the framework readily accommodates multi-call pipelines). Given four input properties—petal length, petal width, sepal length, and sepal width—the model predicts the iris species (such as *Setosa*, *Versicolor*, or *Virginica*). Ryan self-deprecatingly jokes about the domain:

> “To be honest, I don't even know what's I'm not a very flower oriented guy. I don't even know what a sepal is.”

The bootstrapping step works by running labeled training examples through the program: - If a trace yields an incorrect species prediction on the exact-match metric, it is discarded immediately. - If the output scores correctly, the full rationale—including step-by-step reasoning (“Let's think step by step”)—is preserved as a candidate few-shot demonstration for downstream prompts.

[30:37](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=1837s) Instruction Proposal via Grounded Context

Once candidate few-shot demonstrations are gathered, MIPROv2 generates task instructions using an instruction-proposing language model grounded in four rich contextual signals:

1. **Program-aware grounding:** The raw DSPy program code is fed directly to the model, which outputs a plain-English functional summary explaining that the program classifies iris species using petal and sepal dimensions via chain-of-thought reasoning. 2. **Data-aware grounding:** The model inspects raw samples from the training dataset (carefully excluding any development set data to prevent data leakage or cheating) and extracts structural patterns, such as the three distinct target classes and their dimensional distributions. 3. **Successful bootstrapped traces:** The high-scoring input-rationale-output traces generated during the first stage are passed along as concrete demonstrations of what good execution looks like. 4. **Prompting tips / heuristics:** A set of meta-prompting guidelines is provided to nudge the generator toward diverse strategies, such as assigning relevant personas, keeping instructions concise, or introducing artificial urgency.

Ryan notes with amusement that the high-stakes framing heuristic has shown its age:

> “This I think hasn't aged as well as some of the other things. Uh, but this is what we did at the time. So we do don't be afraid to be creative. Um, the instruction should include a high-stakes scenario that the LM must solve to solve the task.”

As a consequence of these prompts, the model outputs candidate instructions that range from clean analytical directions (“Given the petal length... provide a step-by-step reasoning for your prediction”) to melodramatic framings, such as directing the language model to classify the flower accurately in order to “save a critically endangered species from extinction.”

[33:13](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=1993s) Joint Bayesian Optimization and Audience Q&A

With pools of instruction candidates and few-shot demonstration sets in hand, MIPROv2 searches for the highest-performing combination using a Bayesian surrogate model based on a Tree-structured Parzen Estimator (TPE), implemented through the Optuna library. Initially, prior beliefs across all instructions and demonstration sets are modeled uniformly. The search process then proceeds iteratively:

```mermaid flowchart TD accTitle: MIPROv2 Bayesian Search Loop accDescr: Iterative evaluation loop sampling prompt parameters, evaluating on mini-batches, and updating surrogate beliefs before running full validation. A["Initialize Uniform Beliefs over Prompts & Demos"] --> B["Sample Instruction & Demo Pair (TPE)"] B --> C["Inject Pair into LM Program"] C --> D["Evaluate on Training Mini-Batch"] D --> E["Update TPE Beliefs (Reward / Penalize)"] E --> F{"Full Eval Cadence Reached?"} F -->|"No"| B F -->|"Yes"| G["Run Full Dev Set Evaluation"] G --> H{"Target Iterations Complete?"} H -->|"No"| B H -->|"Yes"| I["Ship Best Scoring Candidate Pair"] ```

The algorithm samples an instruction and a demonstration set according to current beliefs, injects them into the DSPy program, and tests that configuration against a mini-batch of training instances: - If the mini-batch score improves, the surrogate raises its belief values for both that instruction and that demonstration set. - If performance drops, the surrogate lowers its belief values accordingly. - At set intervals governed by hyperparameters, the system runs a full evaluation across the dev set to guard against mini-batch sampling noise. The configuration that achieves the highest full-evaluation score is selected for deployment.

In this Iris example, the optimizer converged on the high-stakes prompt (“given the critical situation in which the rare species is on the brink of extinction...”), which delivered significantly better evaluation scores than the basic baseline prompt.

During the subsequent Q&A, an audience member asks whether the system verifies the intermediate reasoning steps of the program traces, pointing out that an output could produce the correct answer despite flawed logic. Ryan clarifies that standard MIPROv2 runs did not monitor reasoning traces directly:

> “The question is do you monitor the things like the reasoning traces and make sure that this is correct? So, uh there are tools for observability that enable you to monitor things but in practice we didn't do any reasoning trace monitoring and stuff. You could try to make your metric based on that and then enforce that like is like you use an LM as a judge was the reasoning faithful to get to the answer and include that as a part of your metric.”

He adds that while systems can technically reward-hack or adopt bizarre internal reasoning under pure outcome-based metrics, modern frontier models adhere much more faithfully to instructions than earlier generations did, reducing the practical risk of unfaithful reasoning traces. Another audience member then begins asking whether this style of prompt optimization continues to work similarly on newer model families.

[36:52](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=2212s) Q&A: Bandit Formulations and Returning Single Prompts

An audience member asks whether MIPROv2 and related prompt optimization techniques still work on newer frontier models. Michael Ryan confirms that they do, pointing out that people are still actively reporting strong empirical results with MIPRO. However, he notes that as models have grown significantly more capable, researchers and practitioners can now unlock even more performance by leveraging advanced reflective methods.

Another student asks whether the optimizer outputs a probability distribution over instructions and demonstration sets, or if it settles on a single final selection. Ryan clarifies that while the optimization process explores across a distribution of candidates, the algorithm ultimately returns just one top-performing combination:

> “We're exploring the distribution and then when it's time to like to give a final answer to the user, we just give them one.”

When asked if this setup is essentially a multi-armed bandit problem, Ryan agrees completely, noting that the MIPRO paper explicitly grounds its theoretical formulation in multi-armed bandit theory.

- **MIPRO on modern LLMs**: Continues to be effective, though newer reflection-based techniques leverage enhanced model capabilities for better gains. - **Output selection**: A single candidate that scored best on evaluations is returned, rather than a sampleable distribution. - **Theoretical framing**: Framed and solved as a classic multi-armed bandit exploration-exploitation problem.

---

[38:02](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=2282s) Feedback Gradients and TextGrad

Ryan transitions to discussing feedback gradients as an introduction to deeper reflection-driven optimization, highlighting the open-source library TextGrad. In TextGrad, an agentic program is defined as a computational graph comprising a chain of LLM calls, tool interactions, and intermediate computations. At the end of the pipeline, the output is evaluated against a loss metric—such as an evaluation score or correctness check. An LLM then inspects the error, generates textual feedback explaining what went wrong and how the output could be improved, and passes that natural language critique upstream to the preceding stage.

```mermaid flowchart LR accTitle: TextGrad Feedback Backpropagation Flow accDescr: An LLM chain passes text forward and backpropagates natural language feedback gradients to update prompts. subgraph ForwardPass["Forward Execution"] P["Prompt & Input"] --> M1["Stage 1 LLM"] M1 --> M2["Stage 2 LLM / Tool"] M2 --> Out["Final Output"] end subgraph FeedbackPass["Textual Backprop"] Out --> Loss["Task Loss / Eval"] Loss -->|"Text Critique"| FB2["Feedback to Stage 2"] FB2 -->|"Text Critique"| FB1["Feedback to Stage 1"] FB1 -.->|"Update"| P end ```

By recursively backpropagating natural language critiques across previous module inputs, outputs, and prompts, the system iteratively updates prompt text (e.g., converting a simple "think step by step" instruction into a specialized, high-performing reasoning prompt). In response to a student question about whether real vector gradients or latent embeddings are involved, Ryan emphasizes that the term is an analogy rather than traditional deep learning calculus:

> “So this is not an embedding or anything like that. This is literally having the model look at the response and what output it got and reflect on it and say like what should change based on that. So this is, I think, like the connections to PyTorch are nice, but there's no actual derivative happening here.”

- **PyTorch analogy**: Reuses familiar automatic differentiation nomenclature (forward passes, loss functions, backpropagation), but executes entirely via textual reflection. - **Multistage credit assignment**: Critiques travel backward through chains of calls, enabling upstream prompts to adapt based on downstream failures. - **No numerical derivatives**: Updates rely strictly on discrete prompt rewrites guided by qualitative text feedback rather than numerical weight gradients.

---

[40:20](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=2420s) GEPA: Genetic Pareto Prompt Evolution

Ryan introduces GEPA (Genetic Pareto Prompt Evolution), a prompt optimization algorithm integrated directly into DSPy that powers automated prompt tuning across industry workflows. The process begins by initializing a candidate pool with a baseline default prompt ($P_0$). The optimizer evaluates $P_0$ against a development set of distinct tasks. When early prompts fail across tasks, the algorithm applies mutations and merges to discover new candidate prompts that gradually solve individual slices of the benchmark.

```mermaid flowchart TD accTitle: GEPA Optimization Loop accDescr: Iterative cycle of Pareto filtering, mutation or merging, mini-batch evaluation, and candidate pool updates. Init["Candidate Pool (P0)"] --> EvalDev["Evaluate on Dev Set"] EvalDev --> Pareto["Extract Pareto Frontier"] Pareto --> Choice{"Candidate Generation"} Choice -->|"Strategy 1"| Mutate["Reflective Prompt Mutation (Mini-batch rollouts + reflection)"] Choice -->|"Strategy 2"| Merge["System-Aware Merge (Cross-module prompt recombination)"] Mutate --> MiniEval["Mini-batch Evaluation"] Merge --> MiniEval MiniEval --> Imprv{"Improved?"} Imprv -->|"Yes"| AddPool["Evaluate all tasks & add to Candidate Pool"] Imprv -->|"No"| Discard["Discard Program"] AddPool --> Pareto Discard --> Choice ```

To maintain diversity and avoid over-optimizing for one metric at the expense of another, GEPA extracts the Pareto frontier of candidates across all benchmark tasks. Ryan illustrates this with an example: if prompt $P_1$ only solves Task 2, while prompt $P_2$ solves both Task 2 and Task 3, $P_2$ Pareto-dominates $P_1$. Consequently, $P_1$ is eliminated, and only the Pareto-optimal candidates are retained for the next evolutionary iteration.

To generate new prompts from this frontier, GEPA alternates between two core generation strategies:

1. **Reflective Prompt Mutation**: A single Pareto-optimal prompt (such as $P_2$) is selected, run across rollouts on a mini-batch of training examples, scored using a defined metric $\mu$, and scrutinized by an LLM that reflects on output errors to propose targeted textual edits. 2. **System-Aware Merge**: In multi-module pipelines (e.g., an agent with separate modules to extract an email and pick an emoji), GEPA splices together successful prompts from different candidates—taking the emoji-picking prompt from $P_3$ and the email-extraction prompt from $P_2$ to form a combined candidate program.

A newly mutated or merged program is verified via a mini-batch evaluation. If its score improves, it is fully evaluated across all dev set tasks and inserted back into the candidate pool; otherwise, it is immediately discarded. Addressing student questions regarding whether non-Pareto candidates persist, Ryan confirms:

> “I believe you discard the non-Pareto frontier because you only you'll only sample from the filtered [pool].”

- **Pareto dominance**: Filters the candidate pool down to programs that are strictly optimal across differing evaluation subsets, preventing the premature loss of specialized behaviors. - **Dual search operators**: Combines targeted qualitative mutation (LLM reflection over failures) with crossover merging (recombining modular prompts across disparate pipelines). - **Compute budget**: The cycle of mutation, recombination, mini-batch screening, and Pareto updating repeats continuously until the optimization compute budget is exhausted.

[46:04](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=2764s) Q&A: GEPA Candidate Pools and Local Optima

Fielding questions about GEPA's evolutionary mechanics, Michael Ryan clarifies how candidate tracking works during prompt optimization. Candidates that fail to achieve Pareto dominance are discarded from the active search pool: “and you're only ever actually sampling from the filtered pool. So if it's not Pareto dominant, you get rid of it.”

```mermaid flowchart TD accTitle: GEPA Candidate Filtering and Proposal Cycle accDescr: Diagram illustrating how candidate prompts are proposed, evaluated against a Pareto frontier, and either retained or discarded.

subgraph CandidatePool["Candidate Population"] P0["Seed Instructions"] P1["Pareto-Dominant Prompts"] end

subgraph Evaluation["Evaluation & Filtering"] EVAL["Evaluate on Task Metric"] CHECK{"Pareto Dominant?"} DROP["Discard Non-Dominant Prompts"] end

subgraph Mutation["Instruction Proposal"] MUT["Reflective Mutation / Branching"] end

P0 --> EVAL P1 --> EVAL EVAL --> CHECK CHECK --"No"--> DROP CHECK --"Yes"--> P1 P1 --> MUT MUT --> EVAL ```

A student asks whether starting from a single seed instruction risks trapping the model in a local optimum, asking why the optimizer cannot invent completely fresh instructions from scratch. Ryan agrees this is a legitimate risk, noting that techniques from MIPRO could bridge this gap: “I think you could probably do something where you mix this together with something like what we did for MIPRO where you generate a bunch of seed instructions.” He notes that he would have to check with the paper's first author, Lakshya, to see if generating multiple initial seed instructions was explored during the research.

- Non-Pareto dominant instruction candidates are pruned, ensuring the system samples only from the filtered Pareto frontier. - Mixing seed instruction generation (as done in MIPRO) into GEPA provides a promising direction to prevent the optimizer from getting stuck in local optima.

---

[47:06](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=2826s) Multi-Stage Prompt Evolution in Privacy-Preserving Inference

Ryan walks through an application from the Papillon paper on privacy-preserving inference. The architecture handles queries containing sensitive user data (such as Social Security numbers) by running a local model that rewrites the private query into an sanitized request, sending that request to a powerful remote API model for reasoning, and finally having a local model combine the remote model's answer with the original query to produce a personalized output without leaking private data externally.

```mermaid flowchart LR accTitle: Papillon Privacy-Preserving Inference Architecture accDescr: Pipeline showing local sanitization of user data, external API processing, and local aggregation.

subgraph LocalSystem1["Local Device (Stage 1)"] UQ["User Query (PII)"] PC["Local Prompt Creator (Optimized Prompt)"] end

subgraph RemoteAPI["Remote Cloud (Stage 2)"] EXT["External LLM (API Reasoning)"] end

subgraph LocalSystem2["Local Device (Stage 3)"] AGG["Local Aggregator (Optimized Prompt)"] OUT["Personalized Response"] end

UQ --> PC PC -->|"Sanitized Request"| EXT EXT -->|"General Reasoning Response"| AGG UQ --> AGG AGG --> OUT ```

In GEPA's experiments, prompts for both the local sanitization model and the final aggregator were optimized: - **Part 1 (Local Prompt Creator):** Initialized with: *"Given a private user query, create a privacy-preserving request for a powerful external LLM. The LLM may assist without learning private information about the user."* - **Part 2 (External API Model):** Not optimized; receives the rewritten query directly. - **Part 3 (Local Aggregator):** Initialized with: *"Respond to a user query. For inspiration, we found a potentially related request to a powerful external LLM and its response."*

Tracking how the prompt creator evolved, GEPA first produced expanded privacy strategies—adding explicit guidance on identifying personally identifiable information (PII) and analyzing query intent—yielding an immediate 8% boost on the privacy-preservation benchmark metric. From there, subsequent mutations enforced output structures by banning specific phrasing, required transparent transformation rationales, and implemented exhaustive verification protocols. Benchmark tables demonstrate that GEPA, particularly when paired with system-aware merging, consistently achieves top-tier accuracy across diverse evaluations.

---

[49:56](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=2996s) Context Optimization Summary and Population Limits

Wrapping up the prompt and context optimization segment, Ryan emphasizes that two primary paradigms remain foundational: few-shot optimization and reflective optimization. Developers bootstrap reasoning traces to locate demonstrative few-shot exemplars, execute agentic tasks, reflect on performance failures, and evolve prompts along the Pareto frontier. He highlights that GEPA remains widely used in production.

When an audience member asks how GEPA compares to classic genetic algorithms that maintain pools of 100 or more diverse candidates, Ryan explains that GEPA's pool size is constrained by the Pareto frontier: > “So it's somewhat up to your run how many things you have in the candidate pool. And then I think that there's certainly interesting ways you could probably build on top of and modify this algorithm to have a more diverse candidate pool. But currently this the the limit is like that Pareto frontier requirement.”

Because candidates must maintain Pareto dominance across evaluation instances, the diversity and size of the active pool are bounded by the dataset's task diversity and Pareto trade-offs.

---

[51:26](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3086s) Transition to Weight Optimization: SFT on Agent Rollouts

Ryan shifts the lecture from prompt/context manipulation to weight optimization, starting with supervised fine-tuning (SFT). Revisiting the token-level cross-entropy loss from Lecture 2, he explains how target tokens are assigned probability weights, backpropagating negative log probabilities to increase the likelihood of desired token trajectories. In traditional instruction fine-tuning, inputs like factual questions ("What is the boiling point of nitrogen?") are tied directly to target textual answers.

Applying SFT to AI agents involves rolling out a capable teacher model on complex tasks and training a student model on the generated trajectories: > “And so SFT for agents, what we will typically do is roll out a teacher on agentic tasks. Then we look at successful trajectories or in some cases we trust the teacher enough that we don't even check that they were successful. And we then post-train the student model on these teacher completed tasks.”

The instruction input serves as the agent's task environment prompt, while the target output encompasses the complete trace: chain-of-thought thoughts, tool invocations, bash terminal commands, tool output observations, and self-verification checks.

---

[53:32](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3212s) Pipeline Design and Teacher Selection in OpenThoughts-Agent

Illustrating agent SFT in practice, Ryan highlights the OpenThoughts-Agent paper. In one terminal task example, an agent is instructed to find all case-insensitive files ending with `.ample.com` in the current directory while excluding a `beta` subdirectory, capturing both `stdout` and `stderr`: 1. The teacher emits a bash search command. 2. The terminal executes the command and returns the output. 3. The teacher checks the byte size (e.g., 48 bytes) and verifies standard streams. 4. The teacher concludes the task has succeeded.

Remarkably, the OpenThoughts-Agent framework does not always explicitly verify whether the teacher solved the task successfully; the authors rely on the raw competence of high-end teacher models across varied tasks sampled from diverse Hugging Face repositories.

```mermaid flowchart LR accTitle: Agent SFT Distillation Workflow accDescr: Pipeline showing task sourcing, teacher rollout collection, and student supervised fine-tuning.

subgraph DataPrep["Data Preparation"] T["Source Tasks & Benchmarks"] F["Filter & Combine Scenarios"] end

subgraph Rollout["Teacher Generation"] SEL["Select Teacher Model"] RO["Generate Agent Trajectories (Thoughts + Actions + Verification)"] end

subgraph Training["Post-Training"] SFT["Supervised Fine-Tuning on Student Model"] end

T --> F F --> SEL SEL --> RO RO --> SFT ```

When building this pipeline, picking the single most powerful model available is not always optimal for distillation: > “Interesting. They they find that you shouldn't always use the best model as a teacher. We'll talk about the intuition for this a little bit more with on-policy distillation, but sometimes like if you're going to use GPT-5.6 to train like a 3B model, that's not always the best because GPT-5.6 is so much more capable than that 3B model. Then you're going to have problems where the model is like not able to learn from it. So you want to select a teacher that's actually closer a bit to your student in capabilities.”

- High capability mismatches between teacher and student degrade imitation learning because the student lacks the capacity to represent the teacher's latent reasoning jumps. - Effective pipelines source heterogeneous tasks across multiple repositories, filter them down to cohesive sets, collect full environment interactions, and train student weights via next-token prediction.

[55:21](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3321s) Supervised Fine-Tuning (SFT) and the FireAct Paradigm

Michael Ryan explains how supervised fine-tuning (SFT) is commonly applied to AI agents using rollouts gathered from a high-capability teacher model. After gathering trajectories across various benchmark tasks, practitioners filter the rollouts based on specific criteria—such as length constraints or strict verification to discard incorrect teacher solutions. Ryan notes that fellow instructor John Yang will explore these data-filtering recipes deeper in upcoming lectures. Plotting training dataset size on the x-axis against task accuracy on the y-axis reveals a clear trend across multiple benchmarks: “As you train on more data from all of these different benchmarks, the model gets better.” While SFT proves straightforward, it serves as a remarkably reliable engine for translating raw data scale into systematic performance gains.

```mermaid flowchart TD accTitle: Supervised Fine-Tuning Pipeline for Agents accDescr: Pipeline showing data questions running through various reasoning patterns into a teacher model, followed by verification filtering, reformatting, and student fine-tuning.

subgraph DataGen["Trajectory Generation"] Q["Benchmark Questions"] --> RP["Reasoning Strategies<br/>(CoT, ReAct, Reflection)"] RP --> T["Teacher Model<br/>(e.g., GPT-4)"] end

subgraph FilterTune["Filtering & Training"] T --> F{"Verifier Filter<br/>(Remove Incorrect)"} F -->|Pass| RF["Reformat to ReAct Format"] RF --> SFT["Fine-Tune Student Model<br/>(e.g., Llama 2 13B)"] end ```

To ground this approach, Ryan highlights the FireAct framework, developed shortly after ReAct. In FireAct, benchmark questions are executed under various reasoning design patterns previously discussed in the course—Chain of Thought, ReAct, and reflection—using GPT-4 as the teacher. Incorrect trajectories are filtered out, and the surviving traces are reformatted into standard ReAct trajectories to fine-tune an open-source model such as Llama 2. At test time, this training allows the student model to implicitly select and combine appropriate reasoning patterns to solve novel tasks.

- Fine-tuning Llama 2 13B on GPT-4 rollouts via FireAct improves agent accuracy from 31% (standard ReAct prompting on GPT-3.5) to 34%. - When fine-tuning GPT-3.5 directly on these rollouts using historical OpenAI fine-tuning APIs, performance scales even further. - Ryan classifies standard SFT on teacher traces as "hard distillation," clarifying that while it relies strictly on black-box input/output traces, on-policy distillation methods can exploit white-box model weights if available.

---

[58:15](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3495s) Reinforcement Learning for Agents: Verifiability and Setup

Moving from imitation to reinforcement learning (RL), Ryan outlines the core mechanics of training agentic student models via trial and error. Instead of merely cloning static trajectories, the student model actively rolls out policies directly within varied environments and across tasks of escalating complexity. The system then issues reward signals when the student successfully finishes a task. Verifiability is crucial: in domains like software development, unit tests provide an absolute programmatic truth metric, whereas environments lacking deterministic ground truths must rely on an LLM-as-a-judge to assign rewards.

```mermaid flowchart LR accTitle: Reinforcement Learning Loop for Agents accDescr: An agent model interacting with an environment, receiving evaluations, and adjusting weights.

A["Student Agent Policy"] -->|"Rollout Trajectory"| B["Environment / Task"] B -->|"Output State"| C{"Verifier / Reward Engine<br/>(Unit Tests or LLM Judge)"} C -->|"Scalar Reward"| D["RL Update Algorithm<br/>(PPO / GRPO)"] D -->|"Weight Update"| A ```

Using the OpenThoughts Agent framework as an illustration, Ryan points out that automated verification can be as simple as evaluating whether generated output files achieve an exact match against expected test outputs. When verified, the student receives positive reinforcement. Ryan highlights that while numerous RL algorithms exist, two specific methods dominate discussions in modern agent architectures: Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO).

---

[59:24](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3564s) Proximal Policy Optimization (PPO) Mechanics and Bottlenecks

Ryan examines Proximal Policy Optimization (PPO), detailing the mechanics required to safely optimize an agent's policy. The policy model generates an output trajectory, which is subsequently evaluated across three distinct downstream components: a reward model (either trained or derived from verifiable environmental execution), an unoptimized reference model, and an auxiliary value model. The reference model serves a critical stabilizing function: “in RL you're straying the model from its initial distribution and so you can end up with weird behavior if you don't regularize it back to the base model somehow.” This regularization is enforced via a token-level Kullback–Leibler (KL) divergence penalty.

```mermaid flowchart TD accTitle: PPO Architecture accDescr: Standard PPO flow combining policy, reference, reward, and value models to update policy weights via GAE.

P["Policy Model<br/>(Optimized)"] -->|"Output Rollout"| Ref["Reference Model<br/>(Fixed Base)"] P -->|"Output Rollout"| RM["Reward Model / Verifier"] P -->|"Output Rollout"| VM["Value Model<br/>(Per-token scalar critic)"]

Ref -->|"Per-token KL Divergence"| GAE["Generalized Advantage<br/>Estimation (GAE)"] RM -->|"Output-level Reward"| GAE VM -->|"Per-token Value Baseline"| GAE

GAE -->|"Advantage Update"| P ```

The value model—often an identical architecture to the policy model modified to emit a scalar score for each token—estimates the likelihood that an individual token contributes toward the final solution. Using Generalized Advantage Estimation (GAE), based on foundational research by John Schulman, the algorithm assigns blame or credit to each individual token relative to counterfactual alternatives. Tokens scoring higher than average receive positive policy gradient boosts, while unhelpful tokens are suppressed.

Despite its effectiveness, Ryan emphasizes that PPO introduces severe computational bottlenecks for long agentic trajectories: - **Memory Footprint:** Up to four models (policy, reference, reward, and value) must reside in GPU memory simultaneously. - **Dual Model Updates:** Both the policy model and the value model must undergo continuous backpropagation and gradient updates at every single training step, doubling optimizer state overhead. - **Per-Token Cost:** Calculating per-token values and per-token Generalized Advantage Estimation across large rollout horizons requires exhaustive forward passes.

---

[01:02:53](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3773s) Group Relative Policy Optimization (GRPO)

Ryan introduces Group Relative Policy Optimization (GRPO), originating from the DeepSeekMath research paper. Ryan comments on its unusual origins: “GRPO is a very very impactful method, but it's like buried on page like 10 of a paper that's talking about like a dozen other things.” GRPO eliminates the heaviest architectural bottlenecks of PPO by discarding the dedicated critic/value model entirely. Instead of predicting fine-grained per-token baseline values with a secondary neural network, the policy model samples a group of candidate trajectories for the exact same input task prompt.

```mermaid flowchart TD accTitle: GRPO Architecture accDescr: Policy model generating a group of outputs evaluated against one another to compute relative advantage without a value model.

P["Policy Model<br/>(Single Model Trained)"] -->|"Sample Group of Rollouts"| Outputs["Group of Outputs<br/>(O_1, O_2,..., O_N)"]

Outputs -->|"Scalar Evaluation"| Rewards["Output Rewards<br/>(e.g., Unit Test Passing Rate)"] Outputs --> Ref["Reference Model<br/>(Per-token KL regularization)"]

Rewards --> ZScore["Relative Advantage<br/>(Group Z-score Normalization)"] ZScore --> Update["Policy Gradient Update"] Ref --> Update Update --> P ```

To determine token credit without an explicit value model, GRPO evaluates the entire group of rollouts against the reward metric (such as unit test success rates on a coding problem, where different runs pass 60% or 70% of tests). The relative advantage of each rollout is then derived statistically by computing a z-score across the rewards obtained by that sampled group.

- **Reduced Memory:** Eliminating the value network cuts memory requirements to just the policy model and reference model (along with any environmental reward mechanism). - **Single Model Training:** Only the policy model receives gradient updates and optimizer tracking, reducing training time. - **Granular Token KL Divergence:** Token-level computation is restricted entirely to the KL divergence penalty against the reference model, while rollout advantages are computed at the holistic output level.

[01:04:23](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3863s) GRPO Objective and Advantage Computation

Michael Ryan explains how Group Relative Policy Optimization (GRPO) computes advantages across sampled outputs without relying on a separate critic or value network. Instead of learning an explicit value function, the algorithm runs multiple rollouts for a given query and evaluates performance based on empirical outcomes, such as the fraction of unit tests passed. For example, if an agent averages a 40% pass rate across attempts, some individual rollouts might pass seven out of ten test cases while the mean rollout passes only four. By subtracting the mean reward across the group and dividing by the standard deviation, the algorithm converts the score into a standard $z$-score indicating how many standard deviations above or below the group mean a specific output performed. This relative score constitutes the advantage signal propagated back through the policy model.

Ryan then deconstructs what he humorously calls a “scary equation”—the formal GRPO objective function—showing that it breaks down into intuitive components:

```mermaid flowchart LR accTitle: GRPO Objective Pipeline accDescr: Pipeline showing data sampling, importance ratio weighting, advantage scaling with clipping, and KL regularization in GRPO.

subgraph Sampling ["1. Group Sampling"] A["Sample prompt x"] --> B["Sample group of outputs {y_1... y_G}"] end

subgraph Scoring ["2. Token Importance & Advantage"] B --> C["Compute ratio: π_θ / π_old"] C --> D["Scale by Advantage (z-score)"] end

subgraph Regularization ["3. Stability Checks"] D --> E["PPO-style Clipping"] E --> F["D_KL Penalty vs. Reference Model"] end ```

The objective takes the expectation over a prompt sampled from the training dataset alongside a group of generated responses. For each candidate response, it averages across the group and across the sequence tokens. The core update term examines token importance by calculating an importance sampling ratio: the probability of producing a token under the current policy being optimized ($\pi_\theta$) divided by its probability under the old policy ($\pi_{\text{old}}$). A ratio of $1.0$ indicates equal likelihood before and after the update; values above $1.0$ mean optimization has made the token more probable, whereas values below $1.0$ denote decreased likelihood. This ratio is multiplied by the advantage derived from the group comparison.

To maintain training stability and prevent over-optimizing on a single trajectory, GRPO employs policy clipping: “We clip the reward essentially to stay within a certain bounds so that we're not over optimizing on a particular trace that's really good.” Finally, it incorporates a Kullback–Leibler ($\mathbb{D}_{\text{KL}}$) divergence penalty against a frozen reference model to ensure the policy distribution does not drift excessively far from its starting distribution.

---

[01:06:56](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=4016s) Case Study: DeepSeek-R1 and SFT vs. RL

Turning to real-world empirical validation, Ryan highlights the training recipe from the DeepSeek-R1 paper. The DeepSeek team demonstrated that reinforcement learning could be applied directly to a raw base model without prior instruction tuning, producing DeepSeek-R1-Zero and showing that pure RL can induce reasoning behaviors from scratch. However, to produce the final production model (DeepSeek-R1), they used an alternating pipeline: sampling reasoning trajectories from the RL-trained model to create a curated dataset for a cold-start Supervised Fine-Tuning (SFT) stage on the base model, followed by another round of RL, another refinement via SFT, and a final RL stage.

```mermaid flowchart TD accTitle: DeepSeek-R1 Iterative Training Pipeline accDescr: Iterative training stages moving from base model to DeepSeek-R1-Zero, cold-start SFT, and final DeepSeek-R1.

Base["Base Model"] -->|"Pure RL (GRPO)"| Zero["DeepSeek-R1-Zero"] Zero -->|"Sample Reasoning Traces"| Data["Cold-Start Dataset"] Data -->|"Bootstrap SFT"| SFT1["SFT Model"] SFT1 -->|"RL Phase"| RL1["Reasoning Checkpoint"] RL1 -->|"Distill / Re-SFT"| SFT2["Refined SFT Model"] SFT2 -->|"Final RL"| R1["DeepSeek-R1"] ```

Comparing algorithms on mathematical problem-solving benchmarks, DeepSeek reported that GRPO outperformed standard PPO while being significantly simpler and more compute-efficient to run because it removes the auxiliary critic network.

A student asks about the distinct roles of SFT versus RL in this workflow: - **Role of SFT:** Primarily bootstraps basic reasoning and task-following capabilities so the model can begin producing coherent trajectories. - **Role of RL:** Exposes the model to diverse, complex problem environments where it learns by trial and error. Ryan points out: “Essentially to train a frontier model you actually need to use RL because there is no teacher that could just SFT from.” - **On-Policy Learning Advantage:** Under RL, the model generates its own action sequences rather than passively cloning an expert teacher's demonstrations, leading to more resilient learning trajectories.

Another student asks whether one should still use RL if an empirical trial shows it performing worse than SFT. Ryan laughs and replies pragmatically that decisions depend entirely on the operational setting: “If the SFT is better, just do that.” However, he reiterates that when pushing the frontier beyond human expert demonstrations, relying purely on supervised teacher data is not an option.

---

[01:09:37](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=4177s) On-Policy Distillation and the Chess Analogy

To introduce on-policy distillation, Ryan shares a pedagogical analogy drawn from a Thinking Machines blog post about learning chess. One could attempt to master chess by watching grandmasters like Magnus Carlsen and Hikaru Nakamura stream and analyze their games online. However, watching players rated 2700 to 3000 ELO is of limited help to an amateur because they navigate intricate board positions and counter-strategies that lower-rated players will never encounter: “In one instance, you're just watching the best of the best do what they do and trying to learn as much as you can from it. In the other instance, you're trying things and learning from the trials.” Real improvement happens when an amateur plays on-policy moves from their own skill level and has an expert coach sitting beside them evaluating whether each specific move was sound or flawed.

Standard SFT corresponds to passive observation: the teacher solves the problem, and cross-entropy loss trains the student on teacher tokens. If a student model lacks basic arithmetic order-of-operations, it might generate an incorrect sequence like evaluating `5 + 2` as `7` before multiplying. Under pure reinforcement learning with a binary final outcome, the student simply receives a sparse reward of zero. The model knows the final outcome failed, but receives no granular signal indicating which token derailed the reasoning chain.

On-policy distillation resolves this credit assignment failure by combining student rollouts with teacher token probabilities. The student generates rollouts under its own policy (on-policy), and a white-box teacher model evaluates the student's output token by token using conditional log probabilities:

| Property | Supervised Fine-Tuning (SFT) | Reinforcement Learning (RL) | On-Policy Distillation | |:--- |:--- |:--- |:--- | | **Sampling Policy** | Off-policy (Teacher generates output) | On-policy (Student generates output) | On-Policy (Student generates output) | | **Reward / Loss Signal** | Dense (Token-by-token cross-entropy) | Sparse (Trajectory-level final reward) | Dense (Token-by-token teacher log probabilities) | | **Teacher Requirements** | Requires target labels / demonstrations | Requires no teacher (only reward function) | Requires white-box access to teacher log probabilities |

When the student erroneously outputs a `2` after `5 +` instead of opening parentheses, the teacher's conditional log probability for that token drops dramatically, providing immediate, local negative feedback at the exact step where the mistake occurred. Mathematically, the loss minimizes the reverse KL divergence over student rollouts, measuring the expectation of the difference in conditional log probabilities between the student and the teacher policy distributions.

[01:12:56](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=4376s) Real-World Reasoning and On-Policy Distillation in the Qwen Series

Michael Ryan illustrates token-level reward assignment using a trick problem designed to evaluate physical common sense against misleading numerical framing. In the scenario, ice cubes are placed into a hot frying pan, accompanied by an arithmetic distraction: five ice cubes are added, the quantity is doubled, and the model is asked how many total ice cubes remain in the pan after an hour. The physically correct answer is zero because the pan is on and the ice melts. Instead, the student model executes the entire arithmetic calculation and concludes that twenty ice cubes remain.

```mermaid flowchart TD accTitle: Qwen Model Training and Distillation Pipeline accDescr: Base models undergo SFT and RL to create large teacher models, which then train smaller models via on-policy distillation.

subgraph Teachers ["Teacher Pipeline (Large Scale)"] Base["Base Models"] --> SFT["SFT on Long CoT"] SFT --> RL1["Reasoning RL"] RL1 --> RL2["Thinking Additions & Further RL"] RL2 --> Large["Capable Teacher Models<br/>(Qwen 2.5 32B & 235B / 22B Active)"] end

subgraph Students ["Student Pipeline (Sub-32B Scale)"] SmallBase["Small Base Models"] Large -->|"On-Policy Distillation<br/>(Teacher Evaluates Student Rollouts)"| Distill["Distillation Process"] SmallBase --> Distill Distill --> SmallFinal["Sub-32B Production Models"] end ```

Under a per-token reward scheme, the teacher penalizes generation tokens containing rationalizations such as claims that the ice cubes are not melting or are removed during frying. As Ryan points out:

> “The answer you're supposed to output is zero because the frying pan is on and the ice cubes are going to melt.”

Turning to industry-scale deployment, Ryan discusses the recipe used for the Qwen series. Alibaba trains base models using supervised fine-tuning (SFT) on long chain-of-thought traces, followed by reasoning reinforcement learning (RL), expanded thinking traces, and additional RL. This produces highly capable teacher models, such as the 32B dense model and a 235B mixture-of-experts model with 22B active parameters. For all models below the 32B threshold, instead of running the identical, compute-heavy RL pipeline, they perform on-policy distillation from these large teachers into smaller base checkpoints.

- **Distillation benchmark comparisons on AIME:** - Standard off-policy distillation (SFT on teacher rollouts) reaches approximately 55% accuracy. - Pure reinforcement learning reaches roughly 67% accuracy. - On-policy distillation achieves the highest performance at 74% accuracy while demanding significantly fewer GPU hours. - **Reference sheet dimensions:** Ryan provides a recap comparing SFT, RL, and on-policy distillation across where trajectories are sourced, where the learning signal originates, and the respective objective functions. - **Audience question on convergence:** When an audience member asks whether on-policy distillation and SFT would converge to the same model given enough GPU hours, Ryan responds: “I don't know the theoretical computations behind it, but intuitively I think yes in the limit if you're spending a lot of compute.”

---

[01:15:43](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=4543s) Agents as Optimizers: Darwin Gödel Machine, Meta-Harness, and AutoResearch

Moving into self-optimizing architectures, Ryan describes approaches where the agent is empowered to rewrite its own operational framework. In the Darwin Gödel Machine, an agent modifies its own execution harness rather than relying on an external developer. The system maintains an archive of candidate configurations tracked as full Git commits. To foster both exploitation and exploration, parents are selected for modification in direct proportion to their benchmark performance scores and in inverse proportion to the number of children they have already spawned.

```mermaid flowchart LR accTitle: AutoResearch Execution Loop accDescr: An autonomous agent iterative loop reading repository files, executing code, and logging or reverting changes.

Repo["Read Repo Context"] --> Propose["Propose & Commit Edit"] Propose --> Run["Execute Experiment"] Run --> Check{"Metric Improved?"} Check -- Yes --> Keep["Accept Commit"] Check -- No --> Revert["Revert Change"] Keep --> Log["Append to Experiment Log"] Revert --> Log Log --> Repo ```

Through open-ended exploration and iterative self-modification, the Darwin Gödel Machine independently discovers robust scaffolding heuristics. It learns to compact and summarize context upon hitting token limits, generate history-aware patches, and enable granular file viewing. Ryan emphasizes that both self-improvement (the parent modifying itself rather than being altered by an external meta-agent) and open-ended exploration (sampling across the historical commit lineage rather than strictly taking greedy steps off the latest best commit) are critical for superior results. A similar concept appears in Meta-Harness, where an agent iteratively edits harness code based on past execution history, outperforming standard evolutionary baselines on Terminal-Bench.

Turning to developer workflows, Ryan cites the "auto research" paradigm popularized by Andrej Karpathy. Here, an agent lives inside a codebase (such as Karpathy's nanoGPT speedrun repository), reads surrounding files, proposes modifications, runs benchmark experiments, inspects the resulting metrics, keeps improvements, reverts regressions, and logs findings to an ongoing run journal.

> “And so agents themselves are relatively capable optimizers. You can set them up in an environment where they can log results, ideate, execute; and then the metric and eval become the most critical part to control.”

- **Discovered harness modifications:** The Darwin Gödel Machine uncovered context compaction at token limits, granular file viewing, and history-aware patch construction purely through code self-modification. - **Micro-adjustments in nanoGPT experiments:** Karpathy’s autonomous agent systematically tested structural variations, including halving batch sizes, altering learning rate warmdown schedules, and adjusting random training seeds.

---

[01:18:51](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=4731s) Choosing Optimization Strategies: Context vs. Weights

In the final minutes, Ryan synthesizes when to choose context optimization (prompt engineering, bootstrapping, GEPA) versus weight optimization (SFT, RL, distillation), noting that they can be combined effectively. The *Better Together* paper demonstrates that an alternating optimization schedule—first optimizing prompts, then fine-tuning weights, and subsequently re-optimizing prompts—yields compound performance gains.

```mermaid flowchart TD accTitle: Prompt vs Weight Optimization Selection Framework accDescr: Decision branches determining whether to choose context optimization, weight optimization, or a combined strategy.

Decision{"Key Constraint or Objective"}

Decision -->|"API-only model access / Low budget"| C1["Context Optimization"] Decision -->|"Rapid daily iteration / Expensive rollouts"| C1 Decision -->|"Small models failing prompt sensitivity"| W1["Weight Optimization"] Decision -->|"Teacher available / Need cheap inference"| W2["On-Policy Distillation"] Decision -->|"Context window exhaustion / Excessive bloat"| W1 Decision -->|"Maximizing absolute task capability"| B1["Staged Combination<br/>(Prompt -> Weight -> Prompt)"] ```

Sample efficiency is another critical trade-off. Findings from the GEPA paper demonstrate that prompt optimization requires vastly fewer environment rollouts to climb the performance curve than reinforcement learning algorithms like GRPO. Conversely, the OpenThoughts agent research indicates that weight optimization is data-hungry, requiring expansive trajectory datasets to realize its ceiling.

> “The GEPA paper shows that prompt optimization can be more sample efficient than reinforcement learning.”

| Consideration | Choose Context Optimization | Choose Weight Optimization | |:--- |:--- |:--- | | **Model Access** | API-only access where weights cannot be tuned directly | Full access to open weights (or fine-tuning endpoints) | | **System Volatility** | Workflows changing daily requiring fast, cheap optimization runs | Stable target tasks justifying heavy upfront training compute | | **Rollout / Environment Cost** | High cost per interaction (demands high sample efficiency) | Low-cost or simulator environments capable of scaling to thousands of rollouts | | **Model Capacity** | Large models sensitive to subtle prompt modifications | Smaller models insensitive to in-context instructions | | **Inference Cost & Footprint** | Tolerant of longer prompts and accumulated in-context demonstrations | Requires minimal latency/cost by baking behavior into model parameters | | **Context Length Overhead** | Tolerant of prompt bloat across multi-turn tool traces | Prompt bloat approaching token limits; requires parameterizing behaviors |

Ryan concludes by reminding the class that smaller models often struggle with complex prompt structures, making distillation from a capable teacher the optimal path toward cheap, fast execution without context exhaustion.

## Notable quotes > [01:06:56](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=4016s) Essentially to train a frontier model you actually need to use RL because there is no teacher that could just SFT from. — Michael Ryan

> [59:24](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=3564s) in RL you're straying the model from its initial distribution and so you can end up with weird behavior if you don't regularize it back to the base model somehow. — Michael Ryan

> [09:07](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=547s) if we've set up good evals, we can reoptimize when a new model comes out and ideally we're just getting the best performance we can at every generation. — Michael Ryan

> [03:34](https://www.youtube.com/watch?v=e_m99aR7_3Q&t=214s) You need to first build your system, collect data that you're going to use to optimize a system and have some way of measuring the impact of your optimization. — Michael Ryan

## Mentioned Michael Ryan, DSPy, TerminalBench, Diyi Yang, John Yang, Codex, OpenAI, Apple Keynote, CS329Z, Google, Google DeepMind, PaLM 2-L, GPT-3, GPT-3.5, GPT-4, GPT-5.4, GPT-6.1, GPT-7, Ada, Babbage, DaVinci, Fantastically Ordered Prompts and Where to Find Them, STaR, Large Language Models as Optimizers, OPRO, MIPRO, MIPROv2, TextGrad, GEPA, Iris flower dataset, scikit-learn, Optuna, Papillon, OpenThoughts-Agent, FireAct, ReAct, Llama 2, Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), DeepSeekMath, DeepSeek-R1, DeepSeek-R1-Zero, John Schulman, Thinking Machines, Magnus Carlsen, Hikaru Nakamura, Qwen, Alibaba, AIME, Darwin Gödel Machine, Meta-Harness, Andrej Karpathy, nanoGPT, Better Together, OpenThoughts

## Video description Browse Stanford's online courses in Agentic AI: https://stanford.io/4AIRb2z

Lecture Overview: Should you optimize your AI agent's prompts or fine-tune its weights? In this Stanford CS329Z: Engineering AI Agents lecture, Michael Ryan, whose prompt-optimization algorithms are used across industry, covers when optimization pays off, context optimization (bootstrapped few-shot, OPRO, MIPROv2, TextGrad, GEPA), weight optimization (SFT on agent rollouts, PPO vs. GRPO, on-policy distillation), agents as optimizers (Darwin Gödel Machine, autoresearch), and a practical guide to choosing between prompts and weights.

Speaker: Michael Ryan

Michael Ryan is a CS PhD student at Stanford University developing human-centered, open-source AI systems that augment human intelligence and reflect the plurality of human values. His contributions to the DSPy open-source project enable developers to optimize multi-stage AI systems, and his algorithms are used for automatic AI prompt optimization at Meta, Google, Dropbox, Amazon, and other organizations.

Course website: https://cs329z.stanford.edu/

Watch the full CS329Z playlist: https://youtube.com/playlist?list=PLPRcF5HR02_U&si=3Z4OYphngcmQ0GIy

Recorded 10/21/2026 in CS329Z: Engineering AI Agents, a graduate course at Stanford University taught by Diyi Yang, Michael Ryan, and John Yang.

#AIAgents #AgenticAI #Stanford

[Source](https://www.youtube.com/watch?v=e_m99aR7_3Q)

Reading summary

Michael Ryan argues that modern AI agent optimization requires moving beyond manual prompt fiddling toward systematic context and weight adaptation driven by rigorous evaluations. He frames prompt and parameter tuning as complementary methodologies, where prompt optimization provides sample-efficient last-mile adaptation and weight optimization permanently embeds operational behaviors into compact models. Context Optimization for Last-Mile Adaptation (5:18) Optimization requires three pillars to exist simultaneously: data (trajectories or human feedback), an underlying system (model or workflow), and a quantitative evaluation metric. Ryan emphasizes that as base language models scale in general capabilities, manual prompt tweaking becomes obsolete, but context optimization remains essential for solving last-mile adaptation. General-purpose models cannot inherently predict bespoke business logic, strict ISO output schemas, or specific edge-case policies without explicit grounding. Reflective and Genetic Prompt Search Frameworks (40:20) Context optimization has evolved from unsupervised few-shot exemplar ordering toward reflective and algorithmic search. Methods like Google DeepMind's OPRO treat LLMs as optimizers by feeding historical prompt scores into a meta-prompt, while TextGrad backpropagates natural language critiques upstream across computational graphs. In multi-stage systems, algorithms like MIPROv2 combine grounded instruction proposals with Bayesian optimization over Tree-structured Parzen Estimators. Ryan highlights GEPA (Genetic Pareto Prompt Evolution), which addresses multi-objective benchmarks by retaining only Pareto-dominant candidate prompts. Rather than collapsing into local heuristics, GEPA mutates and merges instructions across task subsets, pruning underperforming candidates to maintain a high-performing instruction frontier. Weight Optimization: Agent SFT and GRPO (62:53) When prompt engineering reaches context capacity limits or latency constraints, practitioners turn to weight optimization. Supervised fine-tuning (SFT) trains student models on execution traces—including tool invocations and verification checks—collected from high-capacity teacher models. However, imitation learning alone cannot explore complex problem spaces, necessitating reinforcement learning (RL) in verifiable environments. To overcome the computational overhead of Proximal Policy Optimization (PPO), which requires maintaining separate critic and value networks, Ryan points to Group Relative Policy Optimization (GRPO) from DeepSeekMath. GRPO samples a group of candidate trajectories for a single prompt, measures task outcomes like unit-test pass rates, and normalizes scores into relative group z-scores to compute policy gradients without an auxiliary critic. Resolving Credit Assignment via On-Policy Distillation (69:37) Binary reward reinforcement learning often fails on complex reasoning tasks because sparse terminal feedback cannot identify which specific token derailed an entire chain of thought. Ryan contrasts this with learning chess: watching grandmasters corresponds to passive SFT, whereas real progress requires an amateur to play moves from their own skill distribution under the critique of an expert coach. On-policy distillation resolves this credit assignment issue by having the student generate trajectories under its own policy while a white-box teacher evaluates those rollouts token by token using conditional log probabilities. This per-token supervision penalizes logical rationalizations—such as calculating melting ice cubes incorrectly in the Qwen training series—without requiring costly manual annotation. Trade-Offs and Synergies in Agent Optimization (78:51) Practitioners must weigh distinct constraints when selecting an optimization pathway. Context optimization is significantly more sample-efficient than reinforcement learning and is ideal for API-only setups, low computational budgets, and rapid iteration. Conversely, weight optimization is data-hungry but cuts unit inference costs, prevents prompt-length context window exhaustion, and stabilizes smaller models. Rather than treating these paradigms as mutually exclusive, Ryan cites research showing that combining both yields superior performance. Staging an alternating curriculum—first optimizing prompts, then fine-tuning model weights, and subsequently re-optimizing prompts—compounds the architectural benefits of both approaches. While automated context and weight optimization deliver substantial reliability gains, practical adoption remains bounded by real-world trade-offs: API non-determinism complicates evaluation metrics, genetic prompt pools risk local optima when seeded from single prompts, and full reinforcement learning pipelines demand high rollout scale and rigorous verifiers.

Source quotes

[66:56] Michael Ryan: Essentially to train a frontier model you actually need to use RL because there is no teacher that could just SFT from. [59:24] Michael Ryan: in RL you're straying the model from its initial distribution and so you can end up with weird behavior if you don't regularize it back to the base model somehow. [9:07] Michael Ryan: if we've set up good evals, we can reoptimize when a new model comes out and ideally we're just getting the best performance we can at every generation. [3:34] Michael Ryan: You need to first build your system, collect data that you're going to use to optimize a system and have some way of measuring the impact of your optimization.

Original transcript

[0:05] Today's lecture is on optimization and [0:08] I'm very excited to give this lecture [0:10] because uh I if you recall a few weeks [0:13] ago when I was presenting on DSPI I [0:14] mentioned that optimization is the part [0:16] of DSPI that I worked on most closely. [0:19] So I'd say I have good background [0:21] knowledge on some of this optimization [0:23] stuff and I'm looking forward to [0:25] teaching it today. Uh let's start with [0:27] some announcements really quick. [0:28] >> [snorts] [0:28] >> So, first of all, guest lecture next [0:31] Monday. Um, and so that will be on [0:34] terminal bench. So, please be sure to [0:36] come. I think our we'll have two [0:37] speakers and it should be a really, [0:39] really engaging talk. Uh, so I actually [0:41] have a fun story while I was preparing [0:42] this lecture where I got to feel the AGI [0:45] a little bit. I have some slides you'll [0:46] notice that come from a talk that I gave [0:49] for one of the papers that I'm [0:50] presenting. Uh, but the format was [0:52] completely different. And so I was like, [0:54] oh, I was talking to D and I was like, [0:56] do you think it's okay that the format [0:57] is going to change mid-presentation? And [0:59] she's like, why don't you just give this [1:00] to Codeex? Uh, so Codeex went and [1:02] reverse engineered I I do my [1:04] presentations in Keynote. So Codex went [1:06] and reverse engineered how Keynote [1:08] stores the the data for your [1:11] presentations and wrote code that was [1:13] able to convert it to the format of [1:15] slides that we use for this class. So [1:17] you can kind of keep an eye out and see [1:18] if you notice, but I guess uh it was [1:20] very impressive to me that Codex was [1:22] able to reverse engineer Keynotes like [1:24] file structure. Um anyway, let's get [1:26] started. So uh we're going to start with [1:28] a quick poll. Um this is a interactive [1:31] activity and there's no there there is a [1:33] correct answer, but no shame if you get [1:35] the wrong answer because this is the [1:36] point of the activity. So the poll is [1:38] going to be uh which prompt is the best [1:41] for palm 2L uh instruction tuned. So [1:44] this is an old Google model. I specified [1:46] the model just in case that updates your [1:48] belief on which prompt is the best [1:50] because it is model specific. But we [1:51] have three options here. We have let's [1:54] think step by step uh as like a a suffix [1:58] to a math problem and then you give it [2:00] and try to have it solve the math [2:01] problem. We have take a deep breath and [2:03] work on this problem step by step. Uh [2:05] and we have let's combine our numerical [2:07] command and clear thinking to quickly [2:09] and accurately decipher the answer. So [2:11] uh let's do a show of hands. And again, [2:13] there is a correct answer, but there's [2:15] going to be no shame if you get it [2:16] wrong. Uh, so raise your hand if you [2:18] think it's A. All right, we got maybe [2:22] roughly a third of the class. Uh, how [2:23] about raise your hand if you think it's [2:24] B? Um, okay, this is pretty even. I I'd [2:28] say maybe another third. And then how [2:30] about C? Who thinks it's C? And [2:32] obviously that should be a third, too. [2:33] So good. I'm glad everyone participated. [2:35] We we summed to one. Uh, the correct [2:38] answer uh is actually the second one. [2:40] take a deep breath and work on this [2:42] problem step by step, which I think was [2:44] probably a bit unintuitive. I know some [2:46] of you did get it correct, but uh there [2:48] isn't really a clear reason why you'd [2:50] expect this prompt to be better than [2:51] another. Um now, we'll talk a little bit [2:54] about how prompt engineering is changing [2:56] because models are getting better. But, [2:58] uh I think the interesting motivation [3:00] here is you wouldn't really know a [3:02] priority that one prompt is going to be [3:03] better than another. So what we have to [3:05] do is set up evals and run these prompts [3:08] and see which one actually works the [3:10] best. And this is the whole idea of [3:11] optimization. So uh optimization enables [3:14] us to specify intent through data and [3:16] eval uh which allow us to improve our [3:19] system. Uh and this is opposed to how [3:21] you would normally specify intent [3:23] through actually how you build the [3:24] system or how you write up prompts for [3:27] different component components in your [3:28] system. Um and today we're going to [3:30] break down common optimization [3:31] algorithms and when to use each of them. [3:34] So the learning objectives for today are [3:36] uh to identify when to use uh [3:38] optimization for AI systems uh describe [3:42] various common context and weight [3:43] optimization algorithms uh and [3:46] differentiate times to use each [3:47] optimization strategy. So the way we're [3:50] going to go about this uh here's our [3:52] lecture overview for today. We're going [3:53] to start with a brief section about what [3:55] conditions do you need to optimize an AI [3:57] system. Um then we'll talk about context [4:00] optimization for a bulk of the class and [4:02] weight optimization. So different uh [4:04] prompt optimization algorithms and then [4:07] different RL and distillation [4:08] algorithms. Uh then we will talk about [4:11] optimization with agents. So using [4:13] agents as your engine for optimization. [4:16] Uh and then finally we'll have a brief [4:17] discussion of like which strategy should [4:19] you use in practice uh depending on your [4:21] situation. So any questions before we [4:24] dive into the conditions for [4:25] optimization? [4:29] Cool. Uh oh, we got no questions. Okay. [4:32] So then first conditions for [4:34] optimization. So I'm going to pull up a [4:35] slide that we have talked about uh [4:37] before which is these three pillars of [4:39] AI systems. Uh and so we have the data [4:42] uh you'll remember this from the [4:44] frameworks lecture potentially uh data [4:46] rollouts demonstrations human feedback. [4:48] This is all the data that can power our [4:50] system. We have the system which is [4:51] actually the language model or workflow [4:53] or the agent. And then we have a metric [4:55] or evaluation benchmark uh where we [4:58] measure things like pass rate, exact [4:59] match, LM judge or things like cost and [5:02] latency. And so if we have all of these [5:04] things uh these are these are the [5:06] necessary components to optimize an AI [5:09] system. You need all three. You need to [5:11] first build your system, collect data [5:13] that you're going to use to optimize a [5:14] system and have some way of measuring [5:15] the impact of your optimization. Um, I [5:18] think a question that comes up a lot, [5:21] uh, especially when I talk about some of [5:22] the prompt optimization work I've done [5:24] in the past is like, is context [5:26] optimization necessary today? I think [5:29] it's a good question. Uh, so, uh, the [5:31] questions as it's usually asked is like, [5:34] do we really need to fiddle with prompts [5:35] anymore or aren't models just going to [5:37] get better with scale? Uh, I think to [5:40] answer to these questions is actually [5:42] less about fiddling with prompts. So no, [5:44] maybe we don't need to fiddle with [5:45] prompts. And yes, models are going to [5:47] get better with scale, but context [5:49] optimization is actually about more than [5:50] prompt optimization. So I know that my [5:53] motivating example a second ago was [5:55] telling the language model to breathe. [5:56] Um, but we're going to talk about some [5:58] more serious cases in a bit, but really [6:00] the reason you want to use uh context [6:02] optimization is that it's useful for [6:05] last mile adaptation. So what do I mean [6:07] by that? Well, uh, the companies are [6:09] going to get the model to be as good as [6:11] they can at every general task. Um, but [6:14] at the end of the day, when you're [6:16] building a system that has to fit a [6:18] specific purpose and solve a specific [6:20] problem, uh, there's this last little [6:21] bit of adaptation that needs to happen [6:23] to make the model really good at exactly [6:25] what you're trying to do. And, uh, the [6:27] way you can actually do this last mile [6:29] adaptation, there's several ways. I [6:31] mean, we'll talk about context and [6:32] weights. I mean you could really really [6:34] um design your system very very [6:36] carefully specify every single bit of [6:38] your intent in the prompt uh or other [6:42] components of the system to make this [6:43] happen. Um and that's like one way to do [6:46] last mile adaptation. But another way to [6:48] do last mile adaptation is that you can [6:50] set up an eval and say if I can score [6:52] 100% on this I will feel that my system [6:55] really understands what I'm trying to [6:56] accomplish and then you can optimize [6:58] your system to score well on the eval. [7:00] Um, so I think this is a reason why [7:02] you'd want to use context optimization. [7:04] Uh, prompt optimization. I I'm calling [7:06] it context optimization cuz I do think [7:08] prompt optimization calls to mind things [7:10] like fiddling with uh the exact wording [7:12] of things and that is less necessary [7:14] with language models today. They're [7:15] getting very very good. Uh, but context [7:17] optimization I believe is something [7:19] that's here to stay. Um, so any [7:21] questions about is this necessary? [7:26] [snorts] Nice. Then uh let's talk about [7:28] when to optimize. So uh you should [7:31] optimize when you have data and eval for [7:33] your particular domain um and the [7:35] examples and feedback can communicate [7:37] the requirements that would otherwise be [7:39] difficult to specify in the prompt or [7:41] the way you've architected the system. [7:43] So um I think an example that you might [7:47] recall is we had the uh email uh email [7:50] parsing instance where we were trying to [7:52] get the date and time of calendar events [7:54] from emails so we could add them to our [7:55] calendar. And I think something that was [7:58] nice about that is we were able to write [7:59] a very simple prompt like extract the [8:01] date and time. Uh but there was more in [8:03] our intent than was specified in what we [8:06] actually wrote there. We cared about um [8:08] for system purposes. We cared that it [8:10] was using like whatever ISO compliant [8:12] date format that we could parse uh to [8:14] actually run this and run it on the and [8:17] then send it off to the calendar app. Uh [8:18] and we cared that if there were multiple [8:20] events that that was handled somehow. Um [8:23] and like there was like ways that we [8:25] were not writing this into our exact [8:26] specification. Um so this is an instance [8:29] of where context optimization be can be [8:31] useful because we can learn the [8:33] underlying hidden parts of your intent [8:35] and put that into the system. Um so [8:37] sometimes proper evals convey your [8:39] intent better than uh uh better than you [8:42] could actually have written. Uh and then [8:44] you can tune the system to fit these [8:45] evals. Um and then uh if the system has [8:49] many complex components that are [8:50] difficult to tune manually uh it's also [8:52] a reason to use automatic optimization. [8:55] And so what are some of the benefits of [8:57] optimization? Uh the system uh [8:59] appreciates greater reliability. Um so [9:01] if you can optimize your system on some [9:04] eval, you know, one of the metrics you [9:07] can optimize for is reliability and like [9:09] making sure that your system behaves the [9:10] same way every time. uh domain [9:12] adaptation. We've kind of already talked [9:14] about this, but you can specialize your [9:16] system. Um lower cost and latency. Uh [9:19] this is in particular something that you [9:20] could see with uh I think actually if [9:23] back to that email example, we showed [9:24] that you could use a larger model and [9:26] then have a smaller model perform just [9:28] as well. So you're saving money. Um or [9:31] we'll talk about weight distillation in [9:33] a bit, but this is another strategy [9:34] where you have a larger model that's [9:35] your teacher and you try to distill it [9:37] into a smaller model for cost savings or [9:40] efficiency. Um the other benefit you get [9:42] is learning from experience. So over [9:43] time the system can evolve uh and [9:45] adaptability. So as uh underlying things [9:48] change like let's say um we build our [9:51] whole calendar tracking application with [9:53] GPT 5.4 uh and then GPT 5 uh 6.1 comes [9:58] out next or you know someday GPT7 is [10:00] going to come out. Um and we care that [10:02] our system still works with the new [10:04] models then we can reoptimize the [10:06] system. uh if we've set up good evals, [10:08] we can reoptimize when a new model comes [10:10] out and ideally we're just getting the [10:12] best performance we can at every [10:14] generation. So uh any questions about [10:17] the conditions for when to optimize [10:19] otherwise we'll we'll jump into some [10:21] context optimization algorithms. [10:26] Cool. Uh all right, let's talk first [10:28] about so we're going to break context [10:30] optimization into a few uh key [10:34] approaches. Uh the first one we're going [10:35] to talk about is fshot optimization. Um [10:38] so uh to recall uh what fshot prompting [10:42] is. Uh so this this is somewhat classic [10:45] uh work at this point when uh some of [10:47] the original GPT models were coming out. [10:49] Uh there's this paper that large [10:50] language models are fshot learners. And [10:52] what does that mean? Well, you can give [10:54] several examples of how to solve your [10:56] task in a prompt. Uh and this will help [10:59] the model get better at solving your [11:01] prompt or following your prompt. So in [11:03] addition to a task description uh such [11:06] as translate English to French uh you [11:08] can show a few examples of your task and [11:10] you don't have to update the weights uh [11:12] but the model will learn from these [11:13] examples. So we have sea otter to well I [11:17] don't actually speak French but I'm [11:18] assuming this is like lu or something [11:21] like that. [laughter] [11:22] I don't know if that was correct [11:23] pronunciation. Apologies if uh if you do [11:25] speak French, but so we we do have all [11:27] of these um examples and then ideally [11:30] when we give a new example, the model [11:32] will be better at understanding what [11:34] we're trying to do here. Um and so the [11:36] first paper I want to talk about for fot [11:38] optimization uh just kind of to motivate [11:40] some of the methods, but I don't know [11:43] this strictly necessary anymore is [11:45] fantastically ordered prompts and where [11:47] to find them. So this was a paper in [11:49] 2022. It won some awards. It was a very [11:52] very well-received paper. Um the idea [11:54] was the authors had found this problem [11:56] which is when you have few shot examples [11:58] that you want to put in the prompt the [12:00] ordering the model is actually very [12:01] sensitive to the ordering that you put [12:03] it in. Um and so they wanted to find [12:05] what is the best fantastically ordered [12:09] prompt uh and how do you find that? Uh [12:11] so uh the way that they uh that set up [12:14] this task here is this is like a uh [12:16] sentiment classification task. Um, and [12:19] so they have these movie reviews. Uh, [12:22] and so, uh, it has, uh, an example here [12:25] has a way of seeping into your [12:27] consciousness, which come comes from a [12:28] positive movie review. Um, and so they [12:31] have all of these train examples, um, [12:33] and they want to try to permute the [12:35] order, uh, put it through the [12:37] pre-trained language model and then, uh, [12:40] have the language model classify what [12:42] sentiment came from the text. Um and so [12:45] the main idea behind the motivation and [12:47] this is remains true of other few shot [12:50] optimization strategies although less of [12:52] them are about optimizing the order [12:53] these days. Um but uh the idea is [12:56] actually try a bunch of prompt orderings [12:58] in this case orderings but in other [12:59] cases just try a bunch of prompts on the [13:02] train set and measure how this uh [13:04] changes labels on the probing set. So in [13:06] this paper they call it a probing set. [13:08] Uh we'll also call it like a dev set. [13:10] Um, and then what they do, uh, in this [13:13] paper, it's not something that all [13:14] papers do, is they ignore the ground [13:16] truth labels on the probing set. Uh, and [13:18] they're actually just trying to optimize [13:20] statistics about the way that the [13:21] probing set, uh, is parsed. So, let me [13:24] just make sure that this setting makes [13:25] sense. We have examples like this. We're [13:27] going to put them all in the prompt in [13:29] various different orders. We're going to [13:30] run it on a probing set where we just [13:32] give one example at a time and have the [13:34] model produce sentiment. And so, we have [13:36] things like the ending is universally [13:38] panned zero. So the model classified [13:40] that as negative sentiment uh features [13:42] multiple endings, one that's positive [13:44] sentiment and so on. And so we have our [13:46] whole probing set. Uh and then I'm going [13:48] to break down what these uh equations [13:50] actually are showing. But any questions [13:52] about what the sort of method is here of [13:55] like permuting the order and then [13:56] running it on a probing set. [14:00] Cool. Uh so then here is uh what the [14:04] global uh entropy is measuring. So um [14:08] the actual math behind it uh I'm not [14:11] going to assume you understand or not [14:13] but I'm going to show you an example. Uh [14:16] so uh this is an example of a of a [14:18] sequence from the probing set that would [14:20] get high global entropy. Uh because the [14:22] entropy is measuring something about the [14:23] compressibility of the distribution. And [14:26] so we say uh the ending is universally [14:28] panned get a zero. Features multiple [14:30] endings we get a one. Hard to sit [14:32] through get a zero. Nice movie gets a [14:34] one. You can see there's a distribution [14:36] of uh scores here uh of of labels. Uh [14:40] whereas a low global entropy the model [14:42] has maybe collapsed to always saying the [14:44] same output. Um and so uh this is a lot [14:48] easier to specify uh that the the idea [14:50] of entropy is like how compressible is [14:52] the distribution. You could very easily [14:54] say uh the model always outputs zero. [14:57] It's a lot harder to specify what is the [14:58] rule that the model is operating under [15:00] under this probability distribution. So [15:02] this would be an instance of getting [15:04] high global entropy. U the other metric [15:06] they have is local entropy. And so this [15:09] is actually for a particular output. [15:10] They they do average over the outputs [15:12] but for a particular output we can look [15:14] at the log props of the model. Um and so [15:16] if you remember back to my large [15:17] language model lecture, we talked about [15:19] how the model outputs a probability of [15:20] each token. And so we look at the log [15:22] props of outputting 0 and one for the [15:24] ending is universally panned. And so [15:27] here uh the uh log props are pretty [15:30] uniform between the two. uh so this is [15:32] considered high local entropy and then [15:34] if the model is very confident about one [15:36] answer than another then it's considered [15:38] low local entropy and so on a single [15:40] example it might be good to have low [15:41] local entropy um because the model is [15:44] just confident if it's an obvious [15:45] example but their idea is like over the [15:47] whole probing set you want to have [15:49] somewhat high local entropy uh because [15:51] it means the model is like kind of [15:52] considering the options and it's less [15:54] obvious so at least that's the intuition [15:56] uh although when we look at the results [15:58] I do think uh global entropy tends to be [16:00] the better way deciding the prompt [16:02] ordering. But uh here are the results [16:04] for GPT3, 2.7B uh and 175B. I think uh I [16:09] don't remember the names of the models, [16:10] but this is like back when we were [16:12] calling them uh Da Vinci, ADA, Babage, [16:14] all of this. Uh but these are roughly [16:16] approximations of the model size. And so [16:18] you could see on all of these data sets [16:20] when they did this search, um Oracle is [16:23] like if you actually looked at how it [16:24] scored on the dev set. So this is the [16:26] best ordering you could possibly find. [16:28] And then global entropy was looking at [16:30] when we shuffle them and look at the the [16:32] entropy of the probing set. Uh we picked [16:35] based on that. Uh then uh this was the [16:38] best method you know besides oracle [16:39] which they they didn't bold because [16:41] that's the truly best score you could [16:42] possibly achieve. And you can see that [16:44] they were able to find better prompt [16:45] orderings than like random chance and [16:47] and sometimes it was a significant lift. [16:49] You can see like uh several like 8% [16:51] lifts and stuff like that. Um so the [16:54] idea was just use the probing set and [16:56] measure the ordering accuracy. Uh any [16:59] questions about that? I think the main [17:00] thing to understand is this motivation [17:01] of like trying a bunch of things in the [17:03] prompt and seeing how it affects the [17:05] outcome. [17:08] Cool. Then let's talk about bootstrap [17:10] reasoning. Um so we know uh this is [17:13] another thing to recall. Chain of [17:15] thought prompting is important but often [17:17] these labeled fshot examples don't [17:18] contain reasoning. Uh so I just showed [17:20] you those examples of the uh movie [17:23] probing set. Uh so we have like labels [17:26] potentially for uh in the probing set [17:28] the ending is universally panned gets a [17:31] zero. Uh and so we know that but we [17:32] don't have like a chain of thought [17:33] reasoning of like uh well panned is a [17:36] negative term so clearly people don't [17:38] like this movie so then I'm going to say [17:40] it's a negative sentiment. Um and so uh [17:43] we want to uh be able to get this chain [17:46] of thought reasoning somehow uh even [17:47] when we have unlabeled or unreasoning [17:51] uh labeled examples we don't have [17:52] reasoning for them. So one paper that [17:54] does this is a paper called star um and [17:57] so the idea is like we can rejection [17:59] sample our way to mining reasoning [18:00] traces. So they start with a question [18:03] that they give to a large language model [18:05] um and then they generate a rationale [18:07] and an answer. So this would be if we [18:09] were doing this movie review example uh [18:11] like I gave it the ending is universally [18:13] panned. I had it say okay I think that [18:16] that's a negative thing because panned [18:18] is bad and so I'm going to say negative. [18:19] So so we produce that. Um here's another [18:22] example. Uh what can be used to carry a [18:24] small dog? We give it some choices and [18:25] it gives us a multiple choice answer. Um [18:28] so then if we get it correct uh we get [18:31] question rationale answer as like a [18:33] training example and we can use that to [18:35] fine-tune the language model. Um if we [18:38] get a wrong answer uh we give it a hint [18:40] uh which in this paper what they do is [18:42] they actually just say next to the next [18:43] to the correct answer they say hint uh [18:46] this is correct and then they have it [18:47] generate the reasoning for that and uh [18:50] then it should definitely get the [18:51] correct answer otherwise you're working [18:52] with a really stupid model [18:54] >> [laughter] [18:54] >> um and then uh they use that as training [18:57] examples and then they fine-tune the [18:58] model on that um and so um the actual [19:02] key thing I wanted to talk about here is [19:04] the the idea so there's fine-tuning in [19:06] this meth method, right? We we produce [19:07] these examples and we fine-tune the [19:09] model and that works great. But actually [19:11] uh we can bootstrap reasoning and we [19:13] don't even need to fine-tune. We can do [19:14] context optimization. Um so uh this [19:17] comes from the uh DSPI paper uh itself [19:21] in 2023. But the idea was we just uh run [19:24] the input through the model or like LM [19:27] system. We generate reasoning and output [19:29] and then we test it on a metric. Uh if [19:31] we score well, we keep this as an [19:33] example. Um and then we we can run [19:36] another input through uh we score uh [19:38] poorly then we discard it and then what [19:41] we do is just try out different [19:42] combinations on the dev set. So we have [19:44] all of these examples that we saved. We [19:46] throw them into the context of the model [19:48] and see how does it score on our dev [19:49] set. Uh if this combination scores well [19:51] then this is maybe a good optimized [19:53] fshot prompt uh set. Otherwise we just [19:56] keep randomly sampling and trying again. [19:58] Um and so uh this is uh called bootstrap [20:02] random search. Um so basically uh this [20:06] is a way of representing what I just [20:07] described. We run the data through the [20:09] system several times. We find which ones [20:12] score well uh and we keep those and then [20:14] we just take these as examples and we [20:16] slot them into our system, test it over [20:18] the dev set, search combinations of [20:20] these and then see which one scores the [20:22] best. And if I look at the results from [20:25] the DSPI paper, uh you can see that this [20:28] bootstrap method uh tends to be best [20:31] performing over uh like for example [20:34] fshot here is just take the labeled no [20:37] reasoning type things and just put those [20:39] uh in uh and then human train of thought [20:42] is like actually have the human write [20:43] the reasoning. Um so one thing to be [20:45] careful of is I've been talking about [20:47] this as like bootstrapping the [20:48] reasoning. Technically, this also works [20:50] for bootstrapping labels uh which is [20:52] what vanilla is here. But uh in the [20:54] chain of thought instance, we are [20:55] bootstrapping the reasoning and you can [20:57] see that this leads to the best [20:58] performance to do this sort of like [20:59] rejection sampling approach where we [21:01] throw out instances that don't score [21:02] well. Uh yes, question. [21:05] >> So in the com different combinations, [21:07] what are you like interchanging? [21:09] >> Yeah, so the question is for the [21:10] different combinations, what are you [21:11] interchanging? Uh you're actually just [21:14] we have a set that we said um the model [21:16] scored well on these instances. So let's [21:18] say we have 30 examples now, 30 fshot [21:20] examples. Uh and we we specify with a [21:22] hyperparameter we want to put six in [21:24] context. So we randomly sample six of [21:26] them. Put them in context. Run the whole [21:28] program on the dev set and see what it [21:30] scores. Then we randomly sample another [21:32] six, run it, see how it scores. It's a [21:34] very very simple method. All you're [21:35] doing is changing what few shots [21:36] examples are there. And the way that we [21:38] got the few shot examples was this [21:40] rejection sampling. Does that make [21:41] sense? Awesome. Good question. I'm glad [21:43] we had the clarifying question. Any [21:45] other questions? Awesome. [21:50] Now I'm going to So this is fshot [21:52] optimization. Now I'm going to talk [21:54] about something uh that I'm kind of [21:56] bunching a lot of papers under this [21:57] umbrella, but I'm going to call it [21:59] reflective optimization. Uh and the way [22:01] we're motivating this is reflective [22:03] optimization is about uh running the [22:06] program in some way and reflecting on [22:08] what went well, what could be better and [22:10] improving the system based on that. Um [22:12] so the first one that I will talk about [22:14] is a 2024 paper called large language [22:17] models as optimizers or they call their [22:20] method OPRO. Um and so uh this was a [22:23] Google DeepMind paper and they uh it's a [22:27] simple idea. Um so they start with um [22:31] running a generated solution uh through [22:34] a meta prompt here where they have all [22:37] of these different solutions that [22:38] they've tried. Uh so so I guess the [22:41] generated solutions. So let's say this [22:43] is like um uh a math setting here. Um so [22:48] we're going to this is actually the [22:50] paper that had the the take a deep [22:51] breath thing. So let's let's use that as [22:53] our motivating example. So we we say [22:55] take a deep breath and let's uh think [22:57] step by step and then we run it on a [22:59] whole bunch of math problems that give [23:01] us scores for how accurately we scored [23:02] on the math problems. And we now have [23:05] the combination of what we generated as [23:08] a potential prompt and the score that it [23:11] got. So we have take a deep breath that [23:13] got 80. We have um think step by step [23:16] that got 76. We have all of these and we [23:18] just put this into a prompt and we give [23:20] that to a language model and say given [23:22] all of these past things that we've [23:24] tried and how well they scored, what [23:26] else should we try? Um, and so you allow [23:28] the LM to see past instructions, how [23:30] good they were, and ask them to write a [23:32] new instruction. So an example of this, [23:35] uh, this is what goes into the meta [23:36] prompt. So you say something like, I [23:37] have some text along with their [23:38] corresponding scores. Uh, they're [23:40] arranged in ascending order based on [23:42] their scores. Um, and so here's one. [23:45] Let's figure it out. That scored 61. [23:47] Let's solve the problem that scored 63. [23:49] We give a bunch more examples. And then [23:50] this part is like about telling it how [23:52] this will actually be used. Uh, and [23:54] says, write your new text that is [23:55] different to the old ones. and has a [23:57] score as high as possible. Write the [23:59] text in square brackets. And so it's a [24:01] very simple idea. Just throw everything [24:03] at a language model and see what it [24:04] does. And you'll see actually uh maybe [24:06] this paper was pretty preient because [24:08] this comes back with agents. Just throw [24:10] everything at them and and see what it [24:12] does and it starts to work. Uh but uh [24:14] they find here's the table where I [24:17] pulled that um poll at the beginning of [24:19] class from. Uh they say let's think step [24:21] by step that got an accuracy of 71.8. Um [24:24] this comes from you know the chain of [24:27] thought paper itself and then uh when [24:29] they used palm to optimize itself uh [24:32] then they said take a deep breath and [24:34] that scored highest uh when they used [24:36] GPT4 uh that's what came up with that [24:38] other prompt that I showed you about [24:39] numerical command and that was still [24:42] better than the original chain of [24:43] thought uh prompt uh but not quite as [24:45] good as this take a deep breath idea. [24:47] Um, so any questions about about this [24:50] first fora into reflective optimization? [24:54] >> Yes. [24:58] >> Assuming that they're doing API. [25:01] >> Yes. Yes. They're using APIs and here [25:03] they're just using it as that metaprompt [25:05] model that gets all of the context and [25:07] writes a new instruction. But yeah, [25:09] >> I'm asking about the accuracy. So when [25:11] they when they actually run evaluation, [25:13] how do they control for the [25:14] nondeterminis? [25:16] >> Oh, uh this is a good question. You're [25:17] saying like how do they control for [25:18] nondeterminism when they run their [25:20] evaluation? [25:21] >> Uhhuh. I'll say two things. So first of [25:22] all, um the when they did all of their [25:25] evaluations, uh they actually uh used [25:28] Palm 2 for it in this paper. So all of [25:30] these accuracy scores is how well did [25:31] Palm 2 do on it. And this is a Google [25:33] paper, so they weren't using the API in [25:35] this instance um because they just had [25:37] direct access to the model. But in [25:39] practice, we do often have to run [25:41] evaluations on the API and the kind of [25:44] gold standard thing is just to like run [25:46] multiple trials. Uh you just have to [25:47] account for variance based on that. [25:49] Yeah, good question. Any other [25:51] questions? Uh yes. [25:53] >> Um [25:54] is there like I know this is like more [25:56] of like [26:01] why like these strategies help? [26:05] >> Uh yeah, there's a this is a good [26:07] question. And the question is like is [26:08] there an intuition for why these [26:09] strategies will improve performance [26:12] besides just the empirical side? Um I [26:14] think I can say two things that come to [26:16] mind. Uh first of all is like back in [26:18] the era where we were doing these like [26:20] fiddles uh why why uh did it help and [26:23] then I can say today why it helps. So [26:24] back in this era u doing a little [26:26] fiddling to the prompt um you you can [26:29] notice things like the model when it [26:30] writes its own prompt actually gets [26:32] better performance. And so uh part of [26:35] the intuition for this uh and I don't [26:37] know if there's like any really solid [26:39] explorations of this but I I think the [26:41] colloquial intuition uh and common [26:44] intuition was that if models write their [26:46] own prompts uh they're seeing things [26:47] that come more from their training [26:49] distribution and so that can lead to [26:51] better performance because the model is [26:52] le like if you write a prompt that's [26:54] very different from what the model has [26:56] been trained on then it's going to [26:59] potentially perform worse. And so that [27:01] was one reason why we wanted models to [27:02] write their own prompts and and tweak [27:04] the prompts like this. Um, and like part [27:07] of the search process is also just [27:09] trying to find things that were more [27:10] similar to the training distribution of [27:11] the model. But that that was back in [27:14] this era. Now, and I'm going to be [27:16] talking about prompt optimization [27:17] algorithms from today. Now the prompt [27:18] optimization algorithms are more about [27:21] getting that last mile specification. [27:23] And that's really why we do it. It's [27:24] like you may have forgotten to say you [27:26] need to do ISO 8,600 format or whatever [27:30] in the prompt, but when we run it, it [27:31] can reflect and say I failed because I [27:33] output the date in the wrong format, so [27:34] I need to incorporate that in the [27:36] prompt. Stuff like that. But it's a good [27:37] question. Um, there are some theoretical [27:40] papers on fshot learning as like a [27:42] Beijian process and stuff like that too. [27:44] Uh, but I think this is the the for the [27:46] the quick like one minute answer, that's [27:48] what I'll say. Um, any other questions? [27:52] Cool. Um, next paper I'm going to talk [27:55] about is Meepro V2. Uh, funny name. This [27:58] is actually uh one of the papers that I [28:00] was leading. So, we're going to get into [28:01] some slides that I had from that. But, [28:04] uh, the idea is, um, uh, that we are [28:07] going to start by bootstrapping [28:09] demonstrations. The same thing that I [28:10] talked about before of like this [28:12] rejection sampling, running your program [28:14] several times and getting these few shot [28:16] examples. Then, we're going to propose [28:17] instruction candidates. And the way we [28:19] do that is like learning as much about [28:21] your system as we can. um just getting [28:23] that all in context for the model and [28:25] writing instructions and then we're [28:27] going to search combinations just like [28:28] we did in the bootstrap uh search but [28:30] instead of a random search it'll be like [28:32] a beige uh search. Um and so let me [28:35] explain what each of these things is by [28:37] walking through a quick example. Uh so [28:40] so these are the slides that uh codeex [28:42] helped me change the format on. Um okay, [28:44] first thing is bootstrap fshot [28:46] generation. And so we're going to run [28:47] your training example several times to [28:49] collect se successful traces of your [28:51] program and use these as fshot [28:53] demonstrations. So let's say we have [28:54] this LM program. It's a very simple [28:56] program because it has one language [28:57] model call. Uh but you could have [28:59] several calls chained together. Uh this [29:01] is again like a DSPI thing uh like we [29:03] talked about in the past. And so our our [29:05] method is going to be classify iris. Uh [29:08] this is actually a classic machine [29:10] learning data set. If you've played [29:11] around in like uh scikit or some of [29:14] those libraries, this should be built [29:16] in. Um and so uh the task is that you [29:19] get pedal length, pedal width, seepil [29:21] length, sele width. To be honest, I [29:23] don't even know what's I'm not a very [29:25] flower oriented guy. I don't even know [29:26] what a sele is. But um uh but you have [29:29] these four properties of the of the [29:31] flower. uh and then you give it to a [29:33] model and it's supposed to classify [29:35] which species of iris this this uh this [29:40] particular plant is. Um and so we have [29:43] training inputs. Uh these will be [29:45] several labeled instances of petal [29:47] length, pedal width, select sele width. [29:49] Uh and then we have a metric and that's [29:51] going to just be like exact match. Did [29:53] you predict the correct species of uh [29:55] the flower here? And we're going to run [29:57] this through. Uh so uh we first run it. [30:00] It goes runs our program. It produces an [30:02] output. We run it on the metric. We [30:05] score it and we got the answer wrong. [30:06] So, we discard this. Uh if the output [30:09] scores poorly on the metric, we discard [30:11] the trace. Uh then we run it again. [30:13] Species prediction and it's correct. So, [30:16] we're going to save this. Uh and then if [30:18] the output scores highly, we we keep it. [30:20] And so, here's just a quick example. [30:21] It's like here's the properties of the [30:23] plant. We give the rationale. Let's [30:25] think step by step. And it goes and it [30:27] describes what it thinks. And it says [30:29] it's likely. And that was correct in [30:31] this instance. So we save this as an [30:32] example that we would put in the prompt. [30:34] Um and so then the next step is [30:37] instruction proposal. And so uh we [30:39] collect several details about your task [30:41] to inform the prompt proposal language [30:43] model. And so first of all we have [30:45] program aware grounding. Um so this is [30:48] like we actually run we just give your [30:50] DSPI code to the model and say you know [30:53] read this and it says the task of the [30:56] program is designed to solve that that [30:58] it's designed to solve is classifying [31:00] iris flowers into one of three species [31:02] based on the given dimensions of their [31:03] pedals and seepoles. Uh the program uses [31:05] a language model to predict the species [31:07] by reasoning through the provided [31:08] dimensions. So it writes like a [31:10] description of what is trying to be [31:11] solved by your program. Uh then data [31:14] aware grounding. So we actually look at [31:16] examples from your data set. Uh so [31:18] here's like examples of doing this [31:21] classification task and their labels. Um [31:25] and then we have we feed a sample of [31:27] this into a language model. [31:29] [clears throat] And we say uh the data [31:31] set appears to have three classes. [31:33] Satossa, versol, and virginica. Each of [31:35] the classes has a distinct distribution [31:37] on the pedal length and pedal width blah [31:39] blah blah. You know, all of these [31:40] observations that it makes about about [31:41] your training data. Note that we don't [31:43] feed the dev set data. That would be [31:46] cheating. And so then, uh, we do [31:48] successful bootstrap program traces. Uh, [31:51] and so this is again what we ran here. [31:53] So we run this example, we get [31:55] VersaColor. That's not correct. So we [31:57] discard it. We run this one. We get [32:00] Virginica. That is correct. So we save [32:02] this one. And we just run this a bunch [32:04] of times. And we get uh here's just [32:06] another example. So we save these. Uh, [32:09] and then prompting tips. Uh this I think [32:11] hasn't aged as well as some of the other [32:13] things. Uh but this is what we did at [32:15] the time. So we do don't be afraid to be [32:18] creative. Um the instruction should [32:19] include a highstakes scenario that the [32:21] LM must solve to solve the task. Uh [32:23] provide the language model with a [32:25] persona that's relevant to the task or [32:27] like keep the instruction clear and [32:28] concise. Uh and so we take all of this [32:31] program summary, data summary, uh [32:33] successful traces and tips and we feed [32:36] this into the language model and we get [32:38] instructions uh that we can use. And so [32:40] what do these instructions look like? [32:42] They look like uh using the provided [32:45] pedal length, pedal width, sele width [32:47] measurements in centimeters, predict the [32:48] iris species accurately to save a [32:50] critically endangered species from [32:52] extinction. uh or uh given the pedal [32:55] length, you know, all of these uh [32:56] provide a step-by-step reasoning for [32:58] your prediction based on the given [32:59] measurements. And so we have a few [33:00] examples. Uh here's another one about [33:02] brink of extinction. This is what I'm [33:05] talking about about some of the tips [33:06] being about putting in a highstake [33:07] scenario, which worked really well at [33:09] the time, but maybe isn't so necessary [33:10] today. Um and so the last thing we do is [33:13] optimize the combination of instructions [33:15] and demonstrations. And so given our set [33:17] of instructions and demonstrations from [33:18] the first two steps, we iterative [33:20] iteratively sample combinations to [33:22] update our belief that the given [33:24] instruction or demonstration is good [33:25] using a Beijian surrogate model. What [33:27] does this actually mean? I'll show you [33:28] visually. So we have all of these uh [33:31] instructions and demos that we have [33:33] collected and we have beliefs over all [33:36] of them on how useful they are and we [33:38] initialize this uh as a uniform [33:40] distribution. [33:42] Uh then uh we're going to sample [33:44] parameters for our program based on [33:45] these beliefs. So this is a Beijian [33:47] search pro problem. If you're familiar [33:49] this we use something called a tree [33:51] structured parson estimator but I won't [33:53] talk about the details there but you can [33:55] look into this library called optuna [33:56] that has a nice implementation of it. Um [33:58] and so we sample instruction one uh [34:00] because this is a uniform thing that [34:02] we're sampling from and demo set two we [34:04] sampled. Uh so we're going to take these [34:06] we're going to slot them into our [34:08] program uh by just changing the prompt [34:10] to the language model uh and run the [34:13] whole program on our dev set. And so we [34:15] take s we sample a mini batch from the [34:17] training data uh every few iterations. [34:19] We also do a full evaluation. So details [34:21] of that uh you can find in the paper [34:23] like how we decide like when to do a [34:24] full evaluation and stuff but us it's [34:26] mainly hyperparameter tuning. Um and [34:28] then we uh so we have a subset of our [34:31] our training [34:33] examples and then we run that through uh [34:36] on the updated program. Uh so here we go [34:38] run it and we get a good score. And so [34:40] we update the beliefs based on the mini [34:42] batch results. And so here we're going [34:45] to increase the belief that this was a [34:47] good instruction and increase the belief [34:49] that that was a good uh set of few shot [34:51] examples. Uh and then what we do is uh [34:54] we just repeat numbum batches as a [34:56] hyperparameter times doing a full [34:58] evaluation every few iterations. Um and [35:01] then uh we just say this was bad, lower [35:03] those beliefs. This was good, raise [35:05] those beliefs, raise those ones. Um and [35:07] then at the end um we when we run the [35:11] full example, this is like to make sure [35:13] that it wasn't just noise from the mini [35:14] batch. uh those are the when we run the [35:16] full example is the when we decide this [35:18] is the instruction and demo set pair [35:19] that we actually want to to ship and so [35:22] we take this uh here's an example [35:24] optimize module so here we started with [35:26] given the pedal and sele dimensions in [35:28] centimeters predict the iris species uh [35:30] we go we run this and we say uh given [35:33] the critical situation in which the rare [35:35] species on the brink of extinction you [35:36] know here and this was like uh a lot [35:38] better so that's uh okay that's all for [35:42] talking about meepro v2 Any questions [35:45] about this? Uh, which I'd be happy to [35:48] answer. Yes. [35:52] >> For example, it might have like results [35:55] correct but the incorrect like reasoning [35:56] process. [35:58] >> Yeah. The question is do you monitor the [36:00] things like the reasoning traces and [36:01] make sure that this is correct? So, uh [36:03] there are tools for observability that [36:05] enable you to monitor things but in [36:08] practice we didn't do any reasoning [36:10] trace monitoring and stuff. You could [36:12] try to make your metric based on that [36:14] and then enforce that like is like you [36:17] use an LM as a judge was the reasoning [36:19] faithful to get to the answer and [36:20] include that as a part of your metric. [36:22] uh in in this example and in most of the [36:24] work people just have an eval and they [36:27] if it scores higher on the eval it's not [36:30] it usually it could reward hack and it [36:32] in fact does do some weird things [36:34] sometimes but as models have gotten [36:36] better they tend to actually adhere to [36:38] the instruction and so um yeah it's a [36:41] good question but no you're not usually [36:43] doing reasoning monitoring but you you [36:44] could yeah another question [36:46] >> does this kind of optimization work [36:49] similarly on newer models models or does [36:52] it look like a different kind of [36:54] >> Yeah, the question is does this still [36:56] work on newer models? Um, and the answer [36:58] is yes. Uh, so some I've seen people [37:00] still reporting results on on Meepro. Uh [37:03] there are better methods that I'll talk [37:05] about in a bit uh that you can do even [37:06] more because models have be become more [37:08] capable. Uh but this does still work. [37:10] Yeah, good question. Another question [37:13] over uh is that the distribution of the [37:16] instruction and demo sets or are you [37:18] going to pick one of them? Uh yeah. So [37:21] you do uh so the question is like are [37:23] you optimizing like a distribution over [37:25] them or do you ultimately pick one? Uh [37:27] the answer is that we pick one. So we're [37:29] we're exploring the distribution and [37:31] then when it's time to like to give a [37:33] final answer to the user, we just give [37:35] them one. Although I do think that's [37:36] kind of interesting. I don't know. I'd [37:38] have to think more about if you want to [37:40] return the distribution and then like [37:41] let them try to sample from it or [37:43] something like that. That could be kind [37:44] of cool. But no, we just we say this is [37:46] like the one that scored the best and we [37:48] give them that. Is this essentially a [37:50] multarm bandit problem? [37:51] >> Yes. Yeah. The question is, is it a [37:52] multi-arm bandit problem? Yeah. And in [37:54] the paper we talk about that as like the [37:55] theoretical for formulation, but it is a [37:57] multi-arm bandit problem. Yes. Good [37:58] question. Any other questions? [38:02] Uh, cool. Then we're going to talk about [38:04] feedback gradients very briefly. Uh, [38:06] just this one slide. Uh, so there's this [38:09] library called text grad. Uh, and the [38:11] idea um, and I think this is like a nice [38:13] transition to the next method we're [38:14] going to talk about which has more LLM [38:15] reflection. uh but basically uh you [38:18] define your program to be this chain of [38:20] calls and tools and things like that. Uh [38:22] and then at the end uh you either get [38:24] the correct answer or not and you have [38:26] the language model reflect on you know [38:29] this response could be improved by what [38:31] uh and then uh basically you back [38:33] propagate it. So you give the feedback [38:35] that it got at one stage and you go to [38:37] the pri previous stage and you say okay [38:39] this was the output this was the uh [38:42] feedback that the next stage got uh this [38:44] was the input that you got here's all [38:46] this detail like how could we improve [38:48] that and you just back propagate the [38:49] feedback through your program um and so [38:52] they have this example of you will [38:54] answer a reasoning question think step [38:56] by step uh and then through back [38:58] propagating based on diffs of like uh [39:01] looking at how it scored on your your [39:03] loss function. Uh this is all a little [39:06] bit like they really wanted to match [39:08] this to pietorch. So they use a lot of [39:10] terms that come from pietorch and deep [39:13] learning and stuff but it is all stuff [39:15] that's happening in text but then you [39:17] have the loss which is just like you get [39:18] your score on some eval and then have [39:21] the language model reflect on it. Uh but [39:23] basically by doing this optimization [39:25] they get a different prompt uh that is [39:27] more accurate. Um, so I think I didn't [39:30] talk about this in too much detail. So [39:31] if you do have questions, happy to [39:33] answer, but I think this is a motivation [39:34] for the next method that we'll talk [39:36] about, which is Jeepa. Uh, any [39:38] questions? Yes. [39:41] >> Are you thinking about like or [39:43] >> Yeah, there's no no. So this is maybe [39:45] something that's uh it could be [39:47] considered a strength or a weakness of [39:48] this paper, but they really reuse terms [39:51] from deep learning to not be deep [39:53] learning things. So So this is not an [39:56] embedding or anything like that. This is [39:58] um literally having the model look at uh [40:02] the response and what output it got and [40:05] reflect on it and say like what should [40:07] change based on that. So so this is I [40:09] think like the connections to pietorrch [40:11] are nice but there's there's no uh [40:13] actual derivative happening here. Yeah, [40:15] good question. Uh any other questions? [40:20] Cool. Then let's talk about genetic paro [40:22] prompt evolution. uh uh which uh the [40:25] method is called Jeeppa. Um and so this [40:28] one uh we start by initializing our [40:32] search space um by writing or whatever [40:35] our default prompt is. So uh this is [40:38] also built into DSPI. So you can kind of [40:39] be thinking like this is the same format [40:41] we're in. Maybe you think of the iris [40:43] problem I just talked about or this was [40:45] actually the exact method I showed you [40:46] when we were optimizing the prompt for [40:48] the email thing. Um and so we start with [40:50] our default prompts. uh and this goes [40:52] into our candidate pool. So we'll start [40:54] and we just have P 0. So this this will [40:56] get filled out later, but we start with [40:57] P 0. Um and we run uh for instance P 0 [41:02] on our dev set. Uh and so we have [41:04] multiple tasks in our dev set. Sorry if [41:06] the text is a little bit small here, but [41:07] we have task one, task two, task three. [41:10] All these are different tasks within our [41:11] dev set. And we start with P 0 and it it [41:14] fails all of them. Um, and so, uh, [41:17] eventually we'll get more of these and [41:18] you can see they start to succeed on [41:20] some of these examples and and we'll [41:22] have modified the prompt to find these. [41:23] So, you can think of like I'll explain [41:25] how we do the prompt modification, but [41:26] you can think we're doing some sort of [41:28] modifications to the prompt and it [41:29] starts to succeed on some of these tasks [41:31] in our dev set. And so, we take the best [41:34] candidate per task. And so, how does [41:36] this work? We look at task one. Um, the [41:39] only thing that solved it was was prompt [41:41] three. And so, we save that one. uh task [41:44] two uh two different ones solved this [41:46] one. P1 solved it. Uh prompt one and [41:49] prompt two also solved task two. So we [41:51] we pull that over here. Um but we say uh [41:54] prompt two paro dominates prompt one [41:57] because uh prompt two also solved task [42:00] three whereas prompt one only solved [42:02] task two. So we'll store pro uh p2. Uh [42:05] and then task three will look and see it [42:07] was only solved by t2. So that's another [42:09] vote to save p2. And so then you you [42:11] filter uh out just the paro frontier. Um [42:15] and so we save just the ones that were [42:17] like paro optimal on our task and we [42:19] won't be exploring over any that [42:21] weren't. Um and then we go and we want [42:23] to create new candidates. So how do we [42:25] do that? Uh we're going to sample [42:27] examples from our train set. We have two [42:28] strategies to create new candidates. I'm [42:30] going to talk about one at a time. Uh we [42:32] sample one candidate for mutation. And [42:34] so here we sampled P2. uh and we take a [42:36] mini batch of data and we run rollouts [42:39] where P2 is the prompt that we slotted [42:41] into the program and we obtain feedback [42:44] on this using our metric u mu and then [42:47] we reflect to propose a new prompt. So [42:50] this is really just run this prompt on [42:52] your whole mini batch of training [42:54] examples score it and say what uh [42:57] looking at the actual outputs what could [42:59] be improved about this prompt and so [43:01] this is called reflective prompt [43:02] mutation. Uh so this was method one. [43:05] [laughter] Method two is merging [43:07] prompts. Uh so this is maybe the [43:09] evolutionary component. And so we sample [43:11] you know are we going to use which [43:12] strategy are we going to use at each [43:14] step. And so we here uh select P2 and [43:17] P3. Remember those were the paro optimal [43:19] ones uh that we saw. And so the idea [43:23] here is we might have a multimodule [43:25] program. Uh and so we might have a call [43:27] that's like extract the email pick the [43:29] emoji if we're calling back to that [43:31] example a few lectures ago. And so uh [43:34] maybe P2 and P3 they have different [43:36] prompts for each of those. But what we [43:37] want to do is try to select one prompt [43:40] uh from we want to select the pick emoji [43:42] prompt from program 3 and the pick uh [43:45] the extract email from program one uh [43:48] program two. And so we merge these [43:50] together into a complete program where [43:52] we selected pieces of each of them. And [43:54] so this is called the system aware [43:55] merge. Um and then uh we are going to [43:59] again perform a mini batch eval on the [44:01] new example. Uh and then did the [44:04] performance improve? If yes, uh you [44:07] evaluate on all tasks and add it to our [44:09] candidate pool so that we know if it's [44:11] now at the paro frontier. Uh otherwise [44:13] we discard the new program and we try [44:16] again and we just loop on this until we [44:18] run out of budget to do this [44:20] optimization. So this is the way that uh [44:22] Jeep works. Uh and I'll show some [44:24] examples. Any questions about the [44:26] method? I know the diagram can be a [44:27] little bit confusing. Yeah. [44:30] >> The candidate pool already contains [44:32] candidates that you previously [44:34] consider. [44:35] >> Uh, sorry. The question is, would you [44:36] add it to the candidate pool if the [44:38] candidate pool contains things that you [44:40] didn't consider or sorry, what was the [44:41] question? [44:41] >> Why do we add it back to candidate pool [44:43] if we already sample from the candidate [44:45] pool [44:47] sample and had not consider? Oh, you're [44:50] saying why do we add it back to the [44:51] candidate pool to like build off of it [44:53] again when there's like other things we [44:55] might not have tried in the candidate [44:56] pool. Uh yeah, so we're samp we're [44:58] always sampling from the candidate pool [44:59] and this is the idea of like this sort [45:01] of like evolution of it is like we found [45:04] something that might be better, we put [45:05] it in and so now it can be sampled from [45:07] to build on top of it, but we can still [45:09] sample some things that we haven't tried [45:10] yet. Um and and it's it's constantly [45:13] growing. So actually most things or I [45:14] think everything in the candidate pool [45:16] actually has already been tried. Um, so [45:17] we're just pulling new things from the [45:19] candidate pool each time to like try to [45:21] merge together or build on top of if [45:23] that makes sense. Yeah, good question. [45:25] Uh, another question. Yeah, [45:26] >> the candidate pool just means that like [45:28] P1 and P2 were created based on P 0. [45:31] >> Yeah. Yeah. So this is exactly when [45:33] we're showing the arrows here. This is [45:35] just saying that P 0 was the initial [45:36] thing and then we did this reflected [45:39] prompt mutation to create P1 2 or like [45:42] Yeah. Or it came from a system where [45:44] merge from it. Yeah. It's like from a [45:46] merge. [45:48] >> Yeah, I think the arrows come together [45:49] on a merge. So, I don't think this is [45:50] showing any merges. Um, [45:52] >> after like the iteration, do you throw [45:54] away the nonparto frontier? Like, [45:58] >> yeah, you I I believe you discard the [46:01] nonparto frontier because you only [46:03] you'll only sample from the filtered [46:04] pool. Uh, so like I don't know, however [46:06] you want to consider your data structure [46:08] for the I don't know why my cursor is [46:10] not appearing. Yeah, I like maybe they [46:12] all stay visually here, but uh and like [46:14] in but yeah, and you're only ever [46:15] actually sampling from the filtered [46:16] pool. So if it's not parto dominant, you [46:18] you to get rid of it. Yeah, good [46:20] question. Any other questions? Yeah. [46:24] >> So you start from like one instruction, [46:26] right? Yeah. And then you like branch [46:27] out and like but like when you propose [46:30] and you kind of do it like why not allow [46:32] models like come up with like an [46:33] entirely new instruction because [46:35] otherwise you're like maybe you're like [46:36] in a local [46:37] >> Yeah, this is a good question. And like [46:39] are you stuck in a local optimum because [46:40] you started with a certain instruction? [46:42] Um I think you could probably do [46:44] something where you mix this together [46:46] with something like what we did for mero [46:47] where you generate a bunch of seed [46:49] instructions. I'm not sure if uh the if [46:53] locka he's the first author on on this [46:55] work. I'm not sure if he tried that [46:56] along the way and if he found it worked [46:58] or didn't work. So I have to ask him. Um [47:00] but it's a good thought. Yeah. Yeah. Uh [47:02] any other questions? [47:06] Cool. Um so here's an example. This oops [47:10] uh this comes from this uh papalon paper [47:13] um which is doing this privacy [47:15] preserving inference. And so what they [47:17] want to do here is issue a user query uh [47:20] to a prompt creator. And so this user [47:23] query might have like very user [47:24] sensitive info in it. uh and you say [47:26] like uh here whatever here's like all [47:28] these details my social security number [47:30] whatever they they issue this like [47:32] private prompt and you take a local [47:34] model and you rewrite it to be privacy [47:36] preserving you send that off to an API [47:38] based LM to actually do the hardcore [47:40] reasoning and answering over this [47:42] question you get a response and then you [47:44] take the original user query and the LLM [47:46] response and you aggregate it together [47:48] to give the personalized final output [47:50] but the the API based model never needed [47:52] to see your private details. Um, so the [47:55] baseline prompts that are used in the [47:56] JEPA paper, which uh, interestingly I [47:58] noticed are a little bit different than [47:59] the ones in the Papalon paper. So they [48:01] they must have tweaked it a little bit [48:03] uh, or when when they were running the [48:04] experiments, but basically uh, given a [48:07] private user query uh, create a privacy [48:09] preserving request for a powerful [48:10] external LLM. The LLM may assist without [48:13] learning private information about the [48:14] user. So this is the initial prompt for [48:16] this part one. We don't optimize the [48:18] prompt for part two. It's the API based [48:20] remote model and the prompt is literally [48:22] the the query. Um and then we optimized [48:25] the third one which is respond to a user [48:26] query for inspiration. We found a [48:28] potentially related request to a [48:30] powerful external LM and its response. [48:32] Um and so uh this is the example of how [48:35] it evolved. Um and so you can see it [48:38] started with that base instruction given [48:39] a private user query create a privacy [48:41] preserving request. Then it did some [48:44] expanded privacy strategies. Uh so this [48:46] is a summary of like what it was trying [48:48] in the prompt because these prompts get [48:49] kind of long. uh but it said it added [48:51] detailed guidance on how you identify [48:53] PII um analyze the query intent things [48:57] like that um then branching off of that [49:00] and so you can see that got like an 8% [49:01] boost on whatever the metric was about [49:04] privacy preservation then uh it said [49:07] okay actually we need to split uh and [49:09] structure the output better um so we're [49:11] going to ban certain types of uh outputs [49:14] and like this is how we'll get a [49:16] structured output um then do detailed [49:19] transparent transformation rationale so [49:21] explain your rationale better rigorous [49:23] exhaustive protocol so these again are [49:25] summaries but you can see it like learns [49:27] and branches and builds on top of the [49:28] prompts um and some results on Jeepa uh [49:31] so these are a bunch of different [49:32] benchmarks across the top and then some [49:35] you'll notice I me I kind of taught you [49:37] guys a lot of these methods so you can [49:39] actually almost understand this full [49:40] table except for opto prime we didn't [49:42] talk about but essentially um you could [49:45] see that Jeepa and especially Jeepa with [49:48] the system aware merging uh tends to get [49:52] the highest accuracy across all of these [49:53] benchmarks that they they tested it on. [49:56] Um so to summarize on context [49:58] optimization and we'll move on to weight [50:00] optimization um there are two core [50:02] strategies that are still useful today [50:05] which is fshot optimization and [50:07] reflective optimization. And so you want [50:09] to bootstrap uh reasoning and output to [50:11] find useful fshot examples that you [50:13] could put into your prompt. Um and then [50:15] running the task and reflecting on what [50:16] to change is a strong pattern and [50:19] evolving the prompt based on the paro [50:20] frontier is like one of the current ways [50:22] that's very very good and and just to be [50:25] clear about what's in the past what's [50:26] present. Uh Jeppa is still very [50:28] frequently used today for context [50:30] optimization. Uh any any questions about [50:32] this? Yes. [50:34] >> The applications of algorithms people [50:36] often maintain large population like 100 [50:39] things. So so here for complex [50:41] optimization how many candidates are you [50:45] Yeah. So the question is like for [50:47] genetic algorithms usually you have a [50:48] really wide population pool to sample [50:50] from and then you sustain population [50:52] diversity. Um here the size of your pool [50:55] is kind of limited by the amount of [50:57] tasks in your evaluation set actually [50:59] because of that paro optimal uh point. [51:03] So it's somewhat up to your run how many [51:06] things you have in the candidate pool. [51:07] Um and then um I think that there's [51:10] certainly interesting ways you could [51:11] probably build on top of and modify this [51:13] algorithm to have a more diverse [51:14] candidate pool. Uh but currently this [51:16] the the limit is like that paro [51:19] frontier requirement. Yeah, good [51:22] question. Any other questions? [51:26] Um cool. Then we are going to talk about [51:28] weight optimization which will be [51:30] methods that perhaps uh if you've taken [51:32] a large language model course before [51:34] you'll be more familiar about. Um so the [51:36] first one we'll talk about is supervised [51:38] fine-tuning. Uh so these are these are [51:40] very very core algorithms very commonly [51:42] used. Um so I want you to recall from [51:45] lecture 2 the cross entropy loss. Uh [51:47] which we talked about uh it's the sum [51:50] over every token in the vocabulary of [51:52] the model. Uh we look at uh when we have [51:55] an exact output that we are trying to [51:57] fine-tune on. uh we say we score one uh [52:01] or we we set this y variable to one when [52:04] this is the correct token uh and zero [52:06] when the token is not correct and then [52:08] we just look at the log probability of [52:10] that token and and then we make it [52:11] negative because you want lower loss to [52:14] be a good thing so basically did we give [52:16] a high probability to the correct token [52:18] is our loss function and then we can do [52:20] back propagation on this and so I also [52:22] want you to recall instruction [52:23] fine-tuning uh which is when we give all [52:26] of these instructions such as please [52:27] answer the following question, what is [52:29] the boiling point of nitrogen? And we [52:30] give an answer. Uh, and we fine-tune. [52:33] Basically, we just take all of these as [52:35] instructions. And then we do the uh [52:37] cross entropy loss. Um, we run the model [52:39] and we do the cross entropy loss on the [52:41] actual outputs we want the model to [52:44] output and we back propagate. So this is [52:46] SFT and SFT for agents. Um, what we will [52:49] typically do is roll out a teacher on [52:52] agentic tasks. Um then we look at [52:55] successful trajectories or in some cases [52:57] we trust the teacher enough that we [52:58] don't even check that they were [53:00] successful. Um and we uh then post-train [53:03] the student model on these teacher [53:05] completed tasks. So if we go back to the [53:08] the figure here, sorry, it's like our uh [53:11] instruction is going to be whatever [53:14] prompt we give some agentic task and [53:16] then our output is like everything the [53:19] teacher model did to solve that task. [53:21] Um, and then oops. Uh, then we're going [53:24] to be focusing on a diversity of tasks [53:26] and environments and hoping that the [53:28] student learns the task execution from [53:30] the teacher. And so an example of this [53:32] comes from this open thoughts agent [53:34] paper. Uh, and they give this task [53:36] locate all files case insensitive whose [53:39] names end with.ample.com under the [53:41] current directory, but ignore anything [53:43] inside the beta subdirectory. Uh, [53:45] capture both standard out and standard [53:47] error of the command uh, for inclusion [53:49] in the generated task. And then it says [53:52] okay uh this is the teacher agent. Um it [53:54] says I need to capture both standard out [53:57] and standard error. Uh here it writes a [53:59] command. Uh the terminal responds with [54:01] the the response of that command and [54:04] then it says I should verify. So it [54:05] outputs the what was captured and it [54:08] sees that that was correct. So it says [54:10] both standard out and standard error [54:11] were captured. Uh this is 48 bytes of [54:14] content. Uh I'm going to say that I've [54:15] completed the task. And so uh in in the [54:18] open thoughts agent paper, they actually [54:20] don't even always verify uh the the [54:23] accuracy of the teacher. They just say [54:24] we're going to use very capable teachers [54:26] and hopefully they can solve all the [54:27] tasks that we've sampled from all of [54:29] these data sets. Um and then they run [54:32] this and then they SFT a model on these [54:34] outputs. And so this was the the sort of [54:36] pipeline that they use. They source [54:38] tasks from a bunch of like uh papers and [54:41] hugging face repositories and they mix [54:43] them together uh to find like which ones [54:46] are useful in conjunction with one [54:48] another. They filter to a few that they [54:50] find to be very good. They select a [54:52] teacher model. Interesting. They they [54:54] find that you shouldn't always use the [54:56] best model as a teacher. We'll talk [54:57] about the intuition for this a little [54:58] bit more with on policy distillation, [55:01] but sometimes like if you're going to [55:02] use GPD 5.6 six to train like a 3B [55:05] model. Um, that's not always the best [55:07] because GPD 5.6 is so much more capable [55:09] than that 3B model. Um, then you're [55:12] going to have problems where the model [55:14] is like not able to learn from it. Uh, [55:16] so you want to select a teacher that's [55:17] actually closer a bit to your student in [55:19] capabilities. Uh, you generate rollouts [55:21] from that teacher. Then you filter those [55:23] rollouts based on some criteria. And so [55:25] they did like different length uh, [55:27] filtering and stuff, but sometimes your [55:29] filter might just be was the teacher [55:30] correct if you have a verifier. And then [55:32] that's how you get your final recipe for [55:34] fine-tuning the model. And I put this [55:36] note here, stay tuned for the agent data [55:38] lectures because I believe John uh when [55:40] he teaches that is going to go a lot [55:42] more in depth about some of these [55:43] things. Um but just to show you some [55:46] results here. Um they show that um data [55:49] set size on the x-axis and accuracy on [55:52] the y- axis. As you train on more data [55:55] from all of these different benchmarks, [55:56] the model gets better. Uh so SFT is like [55:59] a nice method uh when you have a strong [56:02] teacher, you have good data, you can [56:04] just fine-tune and it looks like, you [56:05] know, as your data size increases, you [56:07] can get models that are better and [56:09] better. And so uh SFT does a nice job of [56:11] using a lot of data to over time improve [56:14] your model. Um and just another example [56:17] paper uh that I wanted to throw in is [56:19] Fire Act, which came out a little bit [56:21] after React. Um and it was about [56:23] fine-tuning agents. And so the idea here [56:26] was just take a bunch of these uh [56:28] questions from all of these data sets uh [56:31] benchmarks and you run different [56:34] reasoning design patterns. Uh I believe [56:36] we talked about all of these in a prior [56:38] lecture. So we have chain of thought [56:39] react reflection. We run a capable [56:42] teacher model in this case GPT4 and we [56:44] generate trajectories. um they do check [56:47] if they're correct and remove the ones [56:48] that aren't and then they reformat them [56:51] into React format and fine-tune llama 2 [56:54] on it. Uh and then when you get a new [56:56] question um the language model is able [56:59] to uh implicitly select which reasoning [57:02] strategy it uses because it was trained [57:03] on all of them uh and then solve the [57:05] task. And so they fine-tune llama 2 on [57:08] GBD4 rollouts. And then they find that [57:11] uh interestingly uh that when you prompt [57:14] like GBD 3.5 with react for instance, [57:17] you can actually get llama 2 13b to be [57:19] better as an agent by doing this fire [57:21] act. You get 34% accuracy versus 31%. Um [57:25] and then when you fine-tune GBT3.5 [57:27] because OpenAI used to have all these [57:28] APIs that allowed you to fine models, [57:30] you can even make it better. Uh, but [57:32] this was just showing that there's some [57:33] lift from doing this like teacher based [57:35] SFT. Any questions about SFT? That [57:38] should hopefully be a pretty [57:39] straightforward method, but very very [57:41] commonly used. [57:44] >> Yes, [57:44] >> is the easiest. [57:47] >> Uh, yeah. So, there's two type there's [57:48] actually many many distillation [57:50] algorithms. Yes, people call this [57:51] distillation. Uh, there's hard [57:52] distillation, which is what I just [57:54] talked about. SFT is like hard [57:55] distillation. Then we're going to talk [57:56] about on policy distillation in a bit [57:58] which is doing more if you have an open [58:00] weight or like white box model you can [58:03] do even more from it. Uh here you can do [58:05] this because you just can take the [58:07] traces from the teacher model but [58:08] there's actually cool things you can do [58:10] if you have access to the weights of the [58:11] model too. Good question. Any other [58:13] questions? [58:15] Uh cool. Let's talk about reinforcement [58:17] learning. So [clears throat] [58:18] reinforcement learning for agents the [58:20] way that we typically do this is we are [58:22] going to roll out the student on agentic [58:24] tasks and reward the student for [58:26] successfully completing the task. Uh and [58:28] we will again focus on varying the [58:30] environments and also task complexity is [58:32] another thing that you you vary in RL. [58:34] Um but verifiability of the task is [58:36] super key here. Uh what does [58:38] verifiability mean? Well a lot of times [58:40] it's like can you get a truly verifiable [58:42] reward which is nice like for coding you [58:44] can pass unit tests and stuff like that. [58:45] Uh you can also use an LM as a judge if [58:47] verifiability isn't like strictly [58:49] possible in your environment. Uh but you [58:51] do need some way of assigning a reward. [58:53] Uh and so just to go back to the open [58:54] thoughts agent paper uh I took the exact [58:56] same example but here uh we might have [58:59] like a ground truth of like these this [59:01] was the expected output. And so if we [59:03] get an exact match when we compare the [59:05] output file to the expected output file [59:08] um then we give it a reward and then the [59:11] student model gets rewarded for doing [59:12] that. And I'll show you algorithmically [59:14] how that actually happens. Um but yeah, [59:16] so we just run this u several times uh [59:20] and and run several tasks to do this. So [59:24] uh two methods that are used for [59:25] reinforcement learning. There are many [59:26] many more than this but two common ones [59:28] are PO and GRPO. Um and so this comes [59:31] from this like deepseek math uh paper I [59:34] believe and they uh you know the paper [59:37] that actually came up with GRPO. It's [59:39] actually kind of interesting cuz uh GRPO [59:41] is a very very impactful method, but [59:43] it's like buried on page like 10 of a [59:45] paper that's talking about like a dozen [59:47] other things. So, it's kind of an [59:49] interesting read. But, uh GRPO is [59:51] actually very very impactful. So, we're [59:52] going to start by talking about PO. Um [59:54] the way that PO works is we have our [59:56] policy model. This is the model that [59:58] we're optimizing. We generate an output [60:00] and then we have we score it in on three [60:03] different models. So our reward model um [60:05] will be something that was trained or or [60:08] is like a verifiable reward from the [60:10] environment but we train it to score the [60:12] model. Um we have the reference model [60:15] which is a unoptimized version of the [60:18] model and we do this because um in RL [60:21] you're straying the model from its [60:23] initial distribution and so you can end [60:25] up with weird behavior if you don't [60:26] regularize it back to the base model [60:28] somehow. Uh and then we have a value [60:30] model. And the value model is typically, [60:32] it doesn't have to be, but it's [60:33] typically the same as the policy model, [60:35] but you have it outputting a scalar [60:37] score for every token. Um so the policy [60:39] model maybe tries to solve the task. [60:42] Then for every token, we give it a score [60:43] of how useful we think this this part of [60:46] the trajectory was towards getting to [60:48] the final answer. And then um we will [60:51] get a reward from combining the KL [60:54] divergence from the reference model and [60:55] what the reward model scored. we get a [60:57] value per token from the value model. We [60:59] use generalized advantage estimate uh to [61:02] assign blame basically to each token and [61:05] then we this this gives us an advantage [61:07] and we can use that to update our policy [61:09] model. So uh like this is kind of what I [61:12] just talked through but we generate an [61:13] output uh we get the KL divergence per [61:15] token we get one reward for the entire [61:18] output and we get the value per token. [61:21] Uh this gives us a per token reward and [61:23] a per token value. We use this [61:25] generalized advantage estimate which is [61:27] how useful is this token compared to [61:29] other possible tokens that we could have [61:30] output at this stage. Um and this comes [61:33] from like a John Schulman paper. Um and [61:35] then uh ultimately uh we we use that to [61:39] update the model based on basically was [61:42] this token better than the average token [61:43] we could have output. If yes then we we [61:46] give the model some boost to make that [61:48] token more likely. Otherwise uh if it [61:50] was less useful we degrade that token. [61:53] And so this is how you kind of RL the [61:55] model based on the reward. Um so there's [61:57] some challenges with PO. First of all, [62:00] there's four models to keep in memory. [62:02] Um so yeah, all all four of these have [62:05] to be saved in memory when you're [62:07] running this. Uh which the reward model, [62:09] like I said, could be an external reward [62:12] from the environment. But certainly the [62:14] reference model, value model, and policy [62:16] model all have to be in memory and that [62:17] gets kind of expensive. Also, there's [62:20] two models that you have to train at [62:21] every step. The value model is a trained [62:23] model that updates as the policy model [62:25] updates. So you now have two models to [62:27] take a gradient step on which means you [62:29] have to have the optimizer state and all [62:30] of that saved in memory as well. And [62:33] it's uh it's also time expensive. Um [62:37] then there's also several expensive per [62:39] token steps uh since we're computing the [62:42] generalized advantage estimate which [62:43] requires computing the value over all [62:45] tokens. Um and so you have the forward [62:47] pass over all of these value uh tokens. [62:49] uh and so these are these are some [62:51] problems I guess with with PO uh that [62:53] gpo resolves. So if we look at gRPO, we [62:56] start with uh the policy model and we [62:58] output several outputs. Uh so we try to [63:01] solve the task many many times and this [63:03] is what this creates what we call a [63:05] group. Um then we have token level KL [63:08] still um but our reward is per output [63:11] and now we have several rewards, right? [63:14] And so every one of these outputs we [63:15] gave it a reward. Um and then we get a [63:18] group advantage. And I'm going to show [63:20] you this in some uh equations. But [63:22] basically um what we look at is uh first [63:25] of all we have three models to keep in [63:27] memory. So uh and the reward model like [63:29] I said doesn't even have to be a model. [63:30] So we really just care about keeping our [63:32] policy model and our reference model in [63:33] memory. We no longer need a value [63:35] estimate. Um and then uh only one model [63:38] to train at every step. Uh and only the [63:41] KL is per token. everything else is at [63:43] the whole output level. Um so this [63:45] becomes a lot more efficient. Um and so [63:48] let's talk about how this advantage [63:49] calculation is done. So how do we do [63:50] advantage calculation without a value [63:52] model? Um and so the idea is we actually [63:56] just compute the zcore for every output [63:58] reward. So like I said, we have our [64:01] policy model that outputs all of these [64:03] different potential rollouts and then [64:05] our reward model uh gives them let's say [64:08] it's a coding task actually. We we try [64:10] to solve the coding task a dozen times [64:12] and maybe three out of those times we we [64:15] maybe have like 10 unit tests and so uh [64:18] 70% of the unit tests pass on these [64:20] three and 60% that pass on these three [64:21] and so our reward is like how many unit [64:23] tests did we pass and so then we can [64:26] compute our advantage based on the mean [64:28] reward. Uh so maybe on average we pass [64:31] 40% of test cases and so some of those [64:33] rollouts will have passed seven out of [64:35] 10 some and the average one passed four [64:37] out of 10. So then a particular output [64:40] was better than the other outputs. We [64:42] can say uh because it it had a higher [64:44] reward and then we divide by the [64:46] standard deviation. This is how we make [64:47] it a zcore. Um uh and so you can say how [64:50] many standard deviations above the mean [64:51] was I. And then uh now we have this [64:54] advantage that we can go and propagate [64:56] back to the policy model. And so I'm [64:58] going to show you a scary equation uh [64:59] but it's kind of simple when you break [65:01] it down. This is the gpo uh update rule. [65:05] And so uh let's let's talk about this [65:08] piece by piece. So this is the [65:10] expectation over a prompt pulled from [65:12] our data set and a group of sampled [65:14] outputs for that prompt. So this is this [65:17] whole idea of we're sampling multiple [65:19] outputs here for a given query that [65:20] we've sampled from our pool of tasks. Uh [65:23] we're going to average over the group [65:25] and average over the number of tokens in [65:27] the response uh to to get our uh update [65:30] rule our score. And we are going to look [65:33] at how important was this token based on [65:35] our new policy compared to our old. So [65:37] all this is looking at is how likely was [65:40] this token to be output by our current [65:42] policy that we're optimizing divided by [65:45] how likely was this token to be output [65:46] by our old policy. So this gives you [65:48] basically a score. If you get 1.0, it [65:51] means it was equally likely pre and post [65:53] our optimization. Uh if you get above [65:56] one, it means our optimization has [65:58] pushed this to be a more likely token. [66:00] If it's below one, it means this was a [66:01] less likely token. Uh we multiply this [66:04] by the advantage, which remember we [66:06] computed based on how good was this [66:08] output compared to other outputs. Um and [66:11] then this is like a sort of uh stability [66:13] thing where it's like basically you can [66:15] in RL you could end up pushing the model [66:17] really really far down a path that's [66:19] like the model already knows that this [66:22] was a good thing to do and uh it was [66:25] ended up being really good and you could [66:26] keep pushing it. So we clip the reward [66:29] uh to essentially stay within a certain [66:31] bounds so that we're not over optimizing [66:33] on a particular trace. That's really [66:35] good. Um and then and then separately we [66:38] also make sure we don't deviate too far [66:39] from the reference model. So we have the [66:41] KL divergence here and this uh and I [66:44] think like probably if this is the first [66:46] time you're seeing the equation you [66:47] probably didn't understand it fully from [66:48] what I just presented but hopefully this [66:50] gave you an intuition of what GRPO is [66:52] doing. So any questions about that? um [66:56] otherwise I'll talk about a real world [66:57] example. Um so this comes from the [67:00] DeepSseek R1 paper. Uh and essentially [67:04] uh they take the base model and they run [67:06] RL on it to produce DeepSseek R10. Uh [67:09] then they sample from that to get [67:10] reasoning prompts that they then use to [67:12] cold start SFT for the base model again. [67:15] Uh and they run RL on that and they do [67:17] this sort of iterative process and then [67:19] now they have SFT again and RL again and [67:21] this is how they produce DeepC R1. So I [67:23] wanted to show this just to make the [67:25] case that like people use uh you can [67:27] actually RL from from nothing to get a [67:29] model that can actually do reasoning. Uh [67:31] but in practice people typically SFT and [67:34] then RL. Um and then we have GRPO versus [67:37] PO for Deepseek R1. And you can see that [67:40] they find that GRPO was better uh on [67:44] doing these math tasks which is nice [67:46] because it's also simpler to run uh more [67:48] efficient. Uh okay question. Yeah. Can [67:51] you talk about like what would you um [67:54] kind of focus on in the SFT part and [67:57] what would you focus on part? [67:59] >> Yeah. Yeah. So the question is what [68:00] would you focus on in SFT and what would [68:02] you focus on in RL? Uh typically the SFT [68:05] is almost meant to bootstrap like [68:07] reasoning in the model. So you start [68:08] from a base model that can't do [68:09] reasoning. you SFT to get it to the [68:11] point where it can start to solve the [68:12] tasks. And then once you can start to [68:14] solve the tasks, then RL is all about [68:16] like creating diverse and complicated [68:19] environments that like uh the model [68:22] should be able to sol and so essentially [68:24] to to train a frontier model you [68:26] actually need to use RL because there is [68:28] no teacher that could just SFT from. Uh [68:30] so that's one reason to do it. Uh, [68:32] another one is like typically the model [68:34] can learn better when it's doing RL [68:36] because uh in the RL setting um the [68:40] model is on policy which means it's [68:42] taking the actions that it would choose [68:44] to take there and instead of just [68:46] showing it like this is what the best [68:47] thing you could have done was or like [68:49] what a teacher does. So I think it's [68:51] like useful for the learning trajectory. [68:52] You could honestly do either of these [68:54] things with SFT or RL, but typically um [68:57] it's easier for a smaller model to learn [68:59] with RL I guess is is one answer. Yeah, [69:01] it's a good question though. It maybe [69:03] not fully satisfying. There's a deeper [69:04] discussion that could happen here, but [69:05] yeah. Uh, another question. Yeah, [69:08] >> if the is not even better than the [69:13] >> What's that question? Sorry. [69:15] >> So, if you like dol and you realize that [69:17] it's even worse, would you just [69:19] [clears throat] go with [69:20] >> Oh, uh, the question is if you do RL and [69:22] realize that it was worse than the SF, [69:23] would you still do it? I mean, I think [69:25] this is all dependent on your setting, [69:26] but probably not. I think, [laughter] [69:27] yeah, if the SFT is better, just do [69:29] that. Yeah, good question. Uh, but [69:31] sometimes SFT isn't an option, like if [69:33] you're training a frontier model and you [69:34] don't have a teacher. But yeah, uh, [69:37] cool. Uh, the last method we'll talk [69:39] about here, and I have to go a little [69:41] quick because we're running out of time, [69:42] is on policy distillation. Uh, but I do [69:44] think this method has like a really [69:46] elegant motivation. And so the example I [69:49] want to show you comes from this [69:50] thinking machines blog post, which is [69:52] uh, this chess game. Uh, and so um, you [69:55] can learn chess a few ways. One way you [69:57] could learn chess is, you know, um you [70:00] could go on YouTube and watch, you know, [70:02] Magnus Carlson and Hikaru, like some of [70:04] the best chess players in the world, [70:05] watch them play their games and talk [70:07] about what they did. Um but I would [70:09] argue if this is the only way that you [70:11] try to learn chess, you probably won't [70:12] get very good at chess because these [70:15] guys are, you know, ELO 2700, 300, 3000, [70:18] stuff like that. And so like you're [70:20] watching the best of the best play games [70:22] that are getting into moves that you'll [70:23] never see. Um, and unless you are that [70:26] good, which some of you may be. I'm not [70:27] actually very good at chess at all. Um, [70:29] and so, uh, the other way you can learn [70:31] chess is by playing a game and having a [70:33] coach tell you was that move good or [70:35] bad. Um, and that is a lot better of a [70:38] way to to at least learn chess because [70:40] in one instance, you're just watching [70:41] the best of the best do what they do uh, [70:43] and trying to learn as much as you can [70:44] from it. In the other instance, you're [70:46] trying things and learning from the the [70:48] trials. Uh, so this is the idea of on [70:50] policy distillation. And so here's an [70:52] example of SFT versus RL versus on [70:55] policy distillation. We start with a [70:56] prompt. Uh what is the answer to this [70:59] question? Uh and then we have SFT where [71:02] the teacher solves the problem and we we [71:04] uh use the cross entropy loss and we [71:06] train the student how likely was it to [71:08] predict each of these tokens and we [71:09] update based on that. Um but maybe our [71:12] student was actually not even in the [71:14] ballpark, right? like it didn't know [71:16] order of operations and so it said 5 [71:18] plus 2 is seven and then does this [71:20] multiplication which is completely [71:22] incorrect. Uh and so we get reward zero. [71:24] Uh this is also not that great because [71:27] the student has learned that it was [71:28] wrong but it's very confusing like uh it [71:31] doesn't know where it messed up. So on [71:33] policy distillation if we have access to [71:36] the log probabilities of the teacher so [71:38] it does have to be a white box model. Uh [71:40] but then we can take what the student [71:42] output and we can score each of these [71:44] tokens based on how likely the teacher [71:45] was to have output that token. And so we [71:48] can say we start with five plus and then [71:51] all of a sudden the student had output [71:53] two and the teacher says that was a [71:55] really really bad next token to output [71:57] because you're dropping the parenthesis. [71:59] Uh and so this is not a good order to to [72:01] to run this task. Um and so this is the [72:05] idea of on policy distillation is you [72:06] apply the reward based on the teacher's [72:08] conditional log probability. um instead [72:10] of the students uh and you do it on [72:13] policy. So here's the comparison. Uh we [72:16] say our sampling for supervised [72:18] finetuning versus reinforcement learning [72:19] versus on policy distillation, our [72:21] sampling is off policy for SFT because [72:23] it's the teacher's policy that we're [72:24] sampling from. Whereas reinforcement [72:26] learning and on policy distillation, [72:28] we're sampling from the student both [72:29] times. And the reward signal is dense [72:32] for both SFT and on policy distillation [72:35] because per token we have a score. Um [72:37] but reinforcement learning is sparse [72:39] because we only get a reward on the [72:40] entire output. And so uh this is the [72:43] loss function that we use. It's the [72:44] reverse KL divergence. And we're [72:46] essentially saying the expectation over [72:48] the response that's sampled from the [72:49] policy. And we just look at the [72:51] difference in the log props between the [72:52] student and the teacher. That's what [72:53] this is saying. Um and so an example uh [72:56] this is like a problem with a which is [72:58] like meant to be testing real world [73:00] reasoning. We place ice cubes in a [73:02] frying pan and then we give it a math [73:03] problem like we add five ice cubes and [73:05] then we doubled it and then and this is [73:07] just to be distracting and then the [73:09] student outputs how how many total ice [73:11] cubes are in the pan. The answer you're [73:13] supposed to output is zero because the [73:15] frying pan is on and the ice cubes are [73:16] going to melt. Uh but the student does a [73:18] whole math problem to determine that [73:19] there's 20 ice cubes in the frying pan [73:22] after like an hour. Um and so uh the [73:25] teacher uh you can see we reward per [73:28] token and you could see um that the [73:31] student is penalized for saying things [73:33] like oh the ice cubes are not melting or [73:36] removed uh during the frying process and [73:38] stuff like that because the teacher [73:40] understands that the point of the [73:41] problem is that you understand the [73:43] physical intuition that the ice cubes [73:44] will melt. Um and so uh I'm moving a bit [73:47] quickly here just in the interest of [73:49] time but just a real world example is [73:51] training the quen 3 model series. So [73:53] they start by taking their base models. [73:55] They do SFT on long chain of thought. [73:58] They do reasoning RL. Um they add more [74:01] uh uh thinking uh to the model. They do [74:04] more RL. And they get these really [74:07] capable large models uh 235B, 22 active [74:10] par billion active parameters and 32B. [74:13] And then for all of their smaller [74:15] models, they want to just take the base [74:17] models and do on policy distillation [74:19] from these teachers. And so this is how [74:22] they produce all the Quen models that [74:24] are below the 32B scale. Um and and they [74:27] actually find that you could have done [74:29] the same RL process with the models. Um [74:32] and so this is um looking at off policy [74:35] distillation versus uh which is SFT. Um [74:39] and they find that you got like 55% [74:41] accuracy on AM. Um with reinforcement [74:44] learning we can get up to 67% accuracy [74:46] but with on policy distillation we get [74:48] highest of 74% accuracy and it actually [74:51] was way less expensive to do this we did [74:53] it in fewer GPU hours um so this is a [74:56] this is a strong motivation for on [74:58] policy distillation and then this slide [75:00] is just kind of comparing the three of [75:01] where do we get the trajectories where [75:03] does learning signal come from and what [75:05] is the objective function so uh I just [75:07] put this as a little reference sheet for [75:08] you but we covered all of this in the [75:10] prior slides so Sorry, I'm going a [75:13] little quick here, but uh any questions [75:15] about on policy distillation? Yeah, [75:17] >> enough GPU hours would on policy [75:20] distillation and SFTP [75:22] converge to the same model. [75:24] >> Yeah, the question is with enough GPU [75:25] hours would on policy distillation SFT [75:27] converge to the same model? Uh I I don't [75:30] know the theoretical uh computations [75:32] behind it, but intuitively I think yes [75:34] in the limit if you're spending a lot a [75:36] lot of compute. Yeah, good question. Um [75:40] any other questions? [75:43] Cool. Then uh I'm going to talk about [75:45] optimization with agents which can be a [75:46] quick section. Uh first of all we'll [75:48] talk about self-improvement. So there's [75:50] this paper called the Darwin Godal [75:52] machine and the idea is that the agent [75:54] modifies its own harness. And so uh we [75:57] start with a uh archive of all of these [75:59] things that we tried and these are full [76:01] uh commits in a repo history and then we [76:04] allow the model uh it's running its own [76:06] code and it can selfmodify. Uh so we [76:10] select the parent to selfm modify [76:12] proportional to the performance scores [76:13] on some benchmark and inversely [76:15] proportional to the number of children. [76:16] So we're trying to explore more um and [76:19] then we evaluate and we add this as a [76:20] child. Uh and so you can see we have [76:22] this sort of branching history of all of [76:24] these harnesses that we tried and we [76:26] over time learn uh these are things that [76:28] you should include in your harness. So [76:31] uh on context limit you should compact. [76:33] It was something that it discovered that [76:35] you should summarize your context. um [76:37] and uh things about history aware patch [76:39] generations, more granular file viewing. [76:42] All of these things are edits that it [76:43] made to its own code. Um and you can see [76:46] over time the model over iterations it [76:49] self-improves um and modifies itself. Um [76:52] and that the uh open-ended exploration [76:55] and self-improvement were both [76:56] necessary. So the self-improvement idea [76:58] is the model the parent is modifying [77:01] itself rather than you have some meta [77:02] agent that just modifies all of the [77:05] potential uh candidate children. And the [77:08] open-ended exploration is that idea of [77:10] like sampling past things in the trace [77:12] in the history rather than always [77:14] building on your latest or greatest [77:16] model. Um and so they find that you get [77:18] better performance by doing that. And [77:20] then this meta harness paper is very [77:21] similar ideas like an agent is writing [77:24] harness code and looking at all of the [77:26] things that it's tried in the past in [77:27] just iterating on harness code. Um they [77:30] find that the meta harness is better [77:32] than other evolution strategies and [77:34] scores highest on this terminal bench. [77:36] And so the idea is the agent can [77:38] optimize itself. Uh and that's all I [77:39] wanted to convey with that. Um another [77:42] thing to talk about is auto research. [77:44] This is like an idea that I think many [77:46] people have had but Andre Carpathy [77:48] definitely popularized it. Uh, and so [77:50] the idea is that you have an agent [77:52] working in a repo and it can just um [77:54] read all of the context in that repo, [77:56] propose changes, commit them to the repo [77:59] and uh execute experiments, look at if [78:02] the metrics approved, if so accept the [78:05] change, if not revert the change, log it [78:07] to a log file and just keep repeating. [78:09] And uh so he has a specific prompt that [78:12] he uses, but the setting that he really [78:14] initialized using it for was auto [78:16] research of training models on his nano [78:18] chat uh speedrun, nano GPT speedrun. And [78:21] you can see that it's making edits to [78:22] the codebase like having the batch size [78:25] uh changing the warmdown. Um eventually [78:28] it gets down all the way here to like [78:29] changing the random seed of training the [78:31] model. But you can see that the agent is [78:33] just trying to propose experiments and [78:35] try things within the repo. And so uh [78:38] agents themselves are relatively capable [78:39] optimizers. You can set them up in an [78:41] environment uh where they can log [78:43] results, ideulate, execute. Uh and then [78:45] the metric and eval become the most [78:47] critical part to control. So uh I think [78:51] I'll pause on questions because we're at [78:52] time and I just want to get through the [78:54] last bit. Um which strategy should you [78:56] use? Um better together uh is a paper [78:59] that shows weight and prompt [79:00] optimization can work together to solve [79:02] tasks. And so they find that sometimes [79:04] you should actually first optimize the [79:06] prompts then the weights and then the [79:07] prompts again. Um uh so this is kind of [79:10] an interesting argument of like [79:12] sometimes using them together is good. [79:13] Uh the Jeepa paper shows that prompt [79:16] optimization can be more sample [79:17] efficient than reinforcement learning. [79:19] Um and so this is showing number of [79:21] rollouts and you can see that Jeepa in [79:23] blue here improves a lot faster than gpo [79:26] and the open thoughts agent paper shows [79:28] you that over time uh weight [79:31] optimization is data hungry but it can [79:33] make use of a lot of examples. So [79:35] reasons to use context optimization uh [79:37] optimization on API access models so [79:40] some strong models don't provide [79:42] fine-tuning access. So if you want to [79:43] use the best model, sometimes you have [79:44] to do context optimization. If you need [79:46] to rerun optimization frequently, like [79:48] daily, something's changing about your [79:50] system, context optimization is a lot [79:52] faster. Uh and environment interactions [79:54] are expensive. If every roll out is [79:56] super expensive, you want something [79:57] that's sample efficient. Reasons to use [79:59] weight optimization. Um context [80:01] optimization might fail, especially for [80:03] small models. They aren't very prompts [80:05] sensitive. Sometimes uh you have a [80:07] capable teacher, but you want a cheaper [80:08] model, then you can use distillation. [80:10] And your context is getting too long. So [80:12] sometimes these context optimizations [80:13] keep adding and adding and adding to [80:15] your context. Um and so you might want [80:18] to do weight optimization to prevent [80:19] that from happening. Uh so that's my [80:21] last slide. Uh this was what we talked [80:23] about today. Uh sorry I went a little [80:25] bit over but I appreciate everyone's [80:26] time and hopefully this was very useful [80:27] to you guys.