AI Engineer
Standard benchmarks fail to predict production application performance because prompt engineering cannot bridge multimodal perception gaps or agent execution breakdowns, requiring spec-driven evaluation frameworks and automated prompt compilers to safely deploy cheaper models.
AI-assisted summary of The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft — original source. Check the original for full context.
Source published:
Key ideas
- Standard AI benchmarks fail to predict application behavior; model capabilities diverge sharply on basic tasks like letter counting, where GPT-5.5 and Kimi K2.6 succeeded while DeepSeek V4 Flash and Mistral Large 3 failed on direct prompts. (5:18)
- Multimodal vision perception cannot be fixed through prompt engineering: either a vision model possesses the resolution and OCR accuracy to resolve an image or it fails entirely. (13:03)
- In multi-turn agentic workflows, model efficiency diverges dramatically; GPT-5.5 executed flight and hotel searches in parallel, while Kimi and DeepSeek ran sequential tool calls and Mistral dropped execution halfway through. (26:17)
- Microsoft Research's ASSERT tool automates test generation by constructing stratified user scenarios and judging behavioral rule violations across full intermediate agent traces. (40:07)
- DSPy and the Genetic Pareto Algorithm (GEPA) allow developers to compile model-specific prompts from declarative Python constraints, enabling cheaper or open-weights models like Mistral to meet strict formatting specs. (45:52)
Detailed notes
## Background Published by the AI Engineer channel on YouTube on October 19, 2026, this video documents a hands-on technical workshop delivered by Pamela Fox, Principal Cloud Advocate at Microsoft, at the AI Engineer World's Fair in San Francisco. According to the page description, the session explores how models behave when swapped inside real applications—comparing GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3 across tasks like letter counting, RAG groundedness, vision, tool calling, automated evaluations, Microsoft's ASSERT framework, and DSPy prompt optimization.
## TL;DR Pamela Fox walks developers through live model-swapping experiments on Microsoft Foundry to demonstrate why published benchmarks fail to reflect real-world application performance. Beginning with basic letter counting and RAG groundedness tests, she illustrates how models diverge significantly in hallucination tendencies, vision perception, and tool-calling loops within a LangChain travel agent. She demonstrates that while prompt engineering can fix some text reasoning gaps, multimodal perception limits are hard barriers, and agentic workflows require rigorous evaluation. The workshop culminates in automated evaluation methodologies, comparing deterministic string checks and LLM-as-a-judge architectures before introducing Microsoft's ASSERT for spec-driven synthetic test generation and DSPy for automated prompt compilation.
## Atmosphere & style The workshop carries an engaging, pragmatic, engineer-to-engineer tone that feels unvarnished and thoroughly grounded in real production pains. Pamela speaks with candid humor and casual confidence, freely joking about her hatred of YAML, managing live API rate limits, and poking fun at models trying to do mental math. The atmosphere feels like sitting in an active conference room where code is tested on the fly, providing an authentic look at the iterative, trial-and-error reality of building and evaluating AI systems.
## Bulletin - [05:18](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=318s) Standard AI benchmarks fail to predict application behavior; model capabilities diverge sharply on basic tasks like letter counting, where GPT-5.5 and Kimi K2.6 succeeded while DeepSeek V4 Flash and Mistral Large 3 failed on direct prompts. - [13:03](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=783s) Multimodal vision perception cannot be fixed through prompt engineering: either a vision model possesses the resolution and OCR accuracy to resolve an image or it fails entirely. - [26:17](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1577s) In multi-turn agentic workflows, model efficiency diverges dramatically; GPT-5.5 executed flight and hotel searches in parallel, while Kimi and DeepSeek ran sequential tool calls and Mistral dropped execution halfway through. - [40:07](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2407s) Microsoft Research's ASSERT tool automates test generation by constructing stratified user scenarios and judging behavioral rule violations across full intermediate agent traces. - [45:52](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2752s) DSPy and the Genetic Pareto Algorithm (GEPA) allow developers to compile model-specific prompts from declarative Python constraints, enabling cheaper or open-weights models like Mistral to meet strict formatting specs.
## Key takeaways - [02:23](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=143s) Model swapping is an essential discipline for AI engineers because public benchmarks do not reflect real-world application performance, and access to any specific model provider can change unexpectedly. - [07:48](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=468s) Failure modes on elementary arithmetic and symbol tracking can often be salvaged with structured chain-of-thought prompt engineering, such as forcing word-by-word decomposition. - [10:00](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=600s) In Retrieval-Augmented Generation (RAG), models vary substantially in groundedness: GPT-5.5 and DeepSeek admit missing knowledge, whereas Mistral Large 3 answers from parametric weights even under negative constraints. - [13:03](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=783s) Unlike text reasoning, poor visual perception cannot be resolved with prompt modifications; if a vision model misreads fine details or low-resolution tables, structural prompting provides minimal benefit. - [16:18](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=978s) Evaluating multimodal vision in languages other than English introduces compounding errors due to simultaneous visual processing and cross-lingual lexical interference. - [17:13](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1033s) Writing JSON schemas for tool calling is a fundamental form of prompt engineering, where parameter names and descriptions dictate whether models extract arguments accurately. - [21:03](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1263s) When running tool calls inside loops, models display vast differences in execution efficiency: GPT-5.5 consolidated a multi-step word problem into a single call, while DeepSeek required four sequential round-trips. - [26:17](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1577s) Evaluating multi-turn agents is notoriously difficult because models differ in their ability to parallelize independent tool calls, maintain context across turns, and avoid dropping early instructions. - [30:00](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1800s) Deterministic string-check evaluations on tool call arguments offer a lightweight, fast baseline for regression testing, but test suites should incorporate edge cases like date math and multilingual slang. - [30:00](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1800s) Because LLMs struggle with mental calendar arithmetic across boundaries and leap years, developers should provide a dedicated tool for date math rather than forcing the model to calculate in-context. - [35:12](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2112s) Evaluating fuzzy outputs or RAG groundedness requires the LLM-as-a-judge paradigm, configured with a binary pass/fail verdict and an explicit reasoning trace to calibrate against human domain experts. - [35:12](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2112s) Decoupling application generation from judge evaluations reduces operational expense and allows engineers to iterate on judging prompts without repeatedly re-running primary app inference. - [43:32](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2612s) Generic evaluators like fluency and coherence provide zero diagnostic value for modern LLMs; production test suites require custom evaluators aligned with functional business constraints. - [45:52](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2752s) DSPy transforms prompt engineering into a compilation process where declarative constraints in Python are automatically optimized via genetic hill climbing into model-specific prompts. - [49:22](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2962s) Anthropic's Claude models feature native structured citation extraction within their Messages API, providing programmatic text offsets for verified citations without custom regex parsing.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=0s) Workshop Setup and the Case for Model Swapping
Pamela Fox kicks off the guided workshop by sharing access details for a hands-on GitHub repository preconfigured with Microsoft Foundry model deployments. Because her workshop proposal was accepted on very short notice, she warns attendees that rate limits (HTTP 429 errors) are a distinct possibility if everyone hits the token capacity simultaneously. To mitigate this, the downloadable `.env` file includes multiple commented-out endpoints and keys that participants can toggle between if they encounter limits, including a dedicated endpoint reserved for Anthropic models due to differing project terms. Pamela's colleague Cedric is on hand in the room to assist participants with setup issues and share links.
```mermaid flowchart TD accTitle: Workshop Environment Setup Flow accDescr: Participants set up their local environment or launch a browser-based Codespace, configure credentials, sync dependencies, and call Microsoft Foundry endpoints.
subgraph Prep ["Environment Setup"] A["Clone Repo or Launch GitHub Codespace"] --> B["Download & Populate.env Credentials"] B --> C["Run 'uv sync' for Python Dependencies"] end
subgraph Runtime ["Execution Flow"] C --> D["Python Script using OpenAI SDK"] D --> E{"Microsoft Foundry Project"} E --> F["GPT-5.5 / Kimi K2.6"] E --> G["DeepSeek V4 Flash / Mistral Large 3"] E -.->|"Fallback on 429 Errors"| H["Alternate Deployment Endpoints"] end ```
To run the sample code, participants using a local machine need Python and `uv`; running `uv sync` installs all required project dependencies. Alternatively, users can launch a GitHub Codespace directly from the repository page to get a preconfigured VS Code environment in the browser without local setup friction. After polling the room on who uses `uv` versus standard Python, Pamela underscores why model swapping has become a vital discipline for AI developers:
> “And I think this is a really important thing for all us to be doing as AI engineers because, you know, as you've seen, like there's so many new models that come out all the time.”
Engineers must retain architectural flexibility and maintain options, especially because access to any given model provider could be restricted or disrupted at any point.
- **Primary Models Deployed on Microsoft Foundry:** - **GPT-5.5** (baseline high-capability model) - **Kimi K2.6** (recent high-performing model) - **DeepSeek V4 Flash** (fast/lightweight model) - **Mistral Large 3** (frontier open-weights model) - **Tooling:** Python SDK for OpenAI (`openai`), `uv` package manager, and GitHub Codespaces.
---
[05:00](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=300s) Benchmarks vs. Reality, Test 1: Counting Letters
Pamela points out that while standard quality, safety, throughput, and pricing benchmarks give an approximate sense of model tiering, they do not reflect actual application behavior. As Pamela emphasizes:
> “...you don't really know how good a model is until you use it for your particular scenario.”
To demonstrate baseline differences, the workshop starts with a classic limitation test: counting letter frequencies. Every example file defaults to a `model` variable set to `GPT-5.5` and calls the Foundry endpoint using the standard OpenAI Python SDK. The test prompt submitted to each model is:
> *"How many times does the letter E appear in the following sentence? The elderly gentleman eagerly entered the elevator. COUNT CAREFULLY."*
When executed via `uv run`, **GPT-5.5** correctly counts 13 instances of the letter "E", reflecting strong arithmetic and symbol-level tracking capabilities that Pamela notes colleagues have observed across internal finance benchmarks. Switching the active model string to **Kimi K2.6** yields the exact same correct count of 13. However, running the prompt through **DeepSeek V4 Flash** fails significantly, returning an answer of 6 (missing more than half the occurrences). **Mistral Large 3** fares slightly better but still falls short, returning 8.
| Model Tested | Output on Direct Prompt | Correct (13 E's)? | |:--- |:--- |:--- | | **GPT-5.5** | 13 | Yes | | **Kimi K2.6** | 13 | Yes | | **DeepSeek V4 Flash** | 6 | No | | **Mistral Large 3** | 8 | No |
---
[07:48](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=468s) Fixing Failure Modes with Prompt Engineering
Faced with a model that fails on a direct instruction, developers do not necessarily have to abandon it; prompt engineering can often bridge the performance gap. Pamela alters the counting prompt on the fly to enforce step-by-step chain-of-thought reasoning:
> *"Break the sentence into words. Count per word. And sum. Show your reasoning."*
Rerunning the updated prompt against **Mistral Large 3** completely resolves the arithmetic shortfall. By explicitly forcing the model to decompose the sentence token-by-token, count each word individually, and display its intermediate tally, Mistral arrives at the correct answer of 13. This simple demonstration establishes a core thesis of the workshop: model limitations can often be mitigated with structured prompt adjustments, but developers must test scenarios directly to identify which prompts require extra scaffolding.
---
[08:38](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=518s) Test 2: RAG, Groundedness, and Hallucination Risks
Moving from toy counting prompts to real-world architectures, Pamela examines Retrieval-Augmented Generation (RAG). She argues that almost all production LLM workflows—including autonomous agents—are fundamentally variations of RAG, where an LLM must answer questions strictly based on supplied external context.
The central challenge in RAG across varying model tiers is **groundedness**: ensuring the model restricts its answers entirely to the provided retrieved sources without confabulating details drawn from its pre-training weights. Pamela notes that smaller or less capable models are far more prone to pulling unverified facts from internal parameters and presenting them as if they were part of the retrieved text.
To test this behavior, Pamela sets up an asymmetric retrieval scenario: - **Test Context:** A factual document containing information strictly about **carpenter bees**. - **User Question:** *"What are the key differences between how honey bees and carpenter bees build their nests?"* - **Target Grounded Behavior:** The model should extract and explain the nest-building habits of carpenter bees from the context, and explicitly refuse to answer regarding honey bees because that information is absent from the provided source material. - **Instructional Prompt:** Instructs the model to answer based solely on the sources and cite them, establishing clear guardrails against ungrounded external knowledge.
[10:00](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=600s) RAG Groundedness and Model Hallucinations
Testing Retrieval-Augmented Generation (RAG) performance across models begins with a baseline scenario where a basic search query is executed against a local dataset. When evaluating a prompt about honeybee nest construction, the source documents deliberately omit this specific information. Pamela Fox notes that GPT-5.5 correctly admits the omission, responding that the dataset does not provide details on honeybee nest construction: “GPT-5.5 is very good at knowing what it doesn't know.” Skipping full runs on every model for time, she tests DeepSeek V4 Flash, which also cleanly reports that the provided sources lack clear information.
```mermaid flowchart TD accTitle: RAG Groundedness Behavior Comparison accDescr: Visual flowchart comparing how GPT-5.5, DeepSeek, and Mistral handle queries when retrieval sources lack the requested information.
subgraph Input["Retrieval Query"] Q["User Query: Honeybee nest construction"] --> S["Vector / Text Search"] S --> D["Retrieved Docs: Carpenter bees only (No honeybee data)"] end
subgraph Evaluation["Model Groundedness"] D --> M1["GPT-5.5 & DeepSeek V4 Flash"] D --> M2["Mistral Large 3 (Baseline)"] D --> M3["Mistral Large 3 (Constrained Prompt)"] end
subgraph Output["Observed Behavior"] M1 --> R1["Refuses: Data not in source"] M2 --> R2["Hallucinates from parametric weights"] M3 --> R3["Marks 'no source provided' but still generates answer"] end ```
The contrast appears immediately when switching to Mistral Large 3 (prompting an aside to audience member Cedric on the proper French pronunciation of "Mistral"). Unlike GPT-5.5 and DeepSeek, Mistral Large 3 falls back into its parametric weights and provides an answer despite the facts not being present in the context:
> “So this is an example where a model went into its weights even though we didn't want it to do.”
Attempting to curb this behavior through prompt engineering, Pamela adds strict formatting requirements: forcing the model to state "no source provided" for any claim absent from the context, strictly forbidding fabricated answers, and demanding that every assertion be tied back to source evidence. While Mistral Large 3 reflects the new instruction by outputting `"no source provided"` alongside the details, it still insists on supplying the information from its weights rather than refusing outright. Pamela underscores that this gap is critical because models must be thoroughly vetted on whether they admit ignorance or fabricate missing context before being placed into production RAG pipelines.
- **GPT-5.5**: Refuses cleanly when source context lacks the requested data. - **DeepSeek V4 Flash**: Successfully refuses without hallucinating. - **Mistral Large 3**: Defaults to answering from internal training weights; strict prompt modifications cause it to label assertions as `"no source provided"`, but it still outputs the parametric data instead of remaining silent.
---
[13:03](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=783s) Multimodal Image Input and Prompting Limits
Moving beyond text, the evaluation tests multimodal vision capabilities across three local images: 1. A photograph of an aurochs (wild cattle). 2. A photograph of Costa Rican crocodiles taken by Pamela. 3. A low-resolution screenshot of a plant price table with small typography.
Because DeepSeek V4 Flash only accepts text, it is excluded from image evaluation, leaving GPT-5.5, Kimi K2.6, and Mistral Large 3.
| Image Challenge | GPT-5.5 Result | Kimi K2.6 Result | Mistral Large 3 Result | |:--- |:--- |:--- |:--- | | **Aurochs Identification** (Check if unicorn) | Rejects unicorn; accurately recognizes an aurochs | Rejects unicorn; accurately recognizes an aurochs | Rejects unicorn; fails to recognize it as an aurochs | | **Crocodile vs. Alligator Snout** | Correctly identifies crocodiles via snout shape | Correctly identifies crocodiles | Fails; guesses alligators | | **Cheapest Plant Table Search** | Extracts the true cheapest plant ($0.58) | Extracts the true cheapest plant ($0.58) | Fails; extracts a plant priced at $2.30 |
GPT-5.5 and Kimi K2.6 successfully identify the aurochs, spot the crocodile snout features, and parse the low-resolution plant catalog down to the 58-cent item. Mistral Large 3, however, struggles across the board: it knows the animal is not a unicorn but fails to name the aurochs, misclassifies the crocodiles as alligators, and pulls a $2.30 plant from the table.
Pamela highlights a fundamental rule for multimodal architectures: you cannot easily prompt-engineer your way around poor visual perception:
> “Now, here's the thing. When it comes to multimodal understanding, you can't really do much prompt engineering because either the model understands that image or it doesn't.”
While specialized domains (like scientific graphs) can benefit from textual structural hints, vision models either have the OCR and visual resolution to resolve the data, or they fail.
---
[16:18](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=978s) Cross-Lingual Vision Failures
Evaluating vision models in languages other than English introduces compounding errors. When the crocodile picture is queried in Spanish—asking whether the animals are *"caimanes o cocodrilos"*—models that succeeded in English stumble.
Pamela explains that the model faces a double hurdle: it must simultaneously parse fine-grained biological snout geometry and navigate cross-lingual lexical interference. In Spanish, *"caimán"* often translates to or overlaps with alligator, distinct from the specific caiman species found in Florida habitats. The cognitive load of mapping visual features to language-specific taxonomies causes the model to guess incorrectly. Given that applications built in regions like California often require bilingual support for English and Spanish speakers, models must be benchmarked directly across target languages rather than assuming English vision performance carries over.
---
[17:13](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1033s) Raw Tool Calling and Schema Prompt Engineering
Tool calling (function calling) serves as the core foundation for all agentic workflows, relying on LLMs explicitly fine-tuned to accept structural definitions and return argument payloads. Pamela sets up a low-level, bare-metal demonstration without higher-level agent frameworks to inspect the exact JSON structures exchanged with the models.
```mermaid flowchart LR accTitle: Low-Level Tool Calling Execution Flow accDescr: Sequence diagram showing how user input and JSON schema prompt the LLM to emit a structured tool call payload.
subgraph Preparation["Schema & Context"] S["JSON Schema: create_calendar_event"] P["System Prompt: Today is Monday, June 29"] U["User: Team sync tomorrow at 1:30 PM"] end
subgraph LLM_Inference["Model Evaluation"] M["LLM (GPT-5.5 / DeepSeek)"] end
subgraph Emitted_Payload["Tool Invocation"] A["Tool Call: function name + arguments JSON"] C["Client Application: Executes actual code"] end
S & P & U --> M M --> A A --> C ```
The model is supplied with a JSON schema defining `create_calendar_event`, specifying property fields, types, and descriptions for arguments such as title, attendees, date, time, and meeting medium. Pamela emphasizes that creating this JSON schema is itself a primary form of prompt engineering: parameter names, type constraints, and descriptions directly dictate how reliably the model generates arguments.
The test inputs a conversational scheduling request: - **User input:** *"Can you throw something on my calendar? It's a platform Team Sync. Me, Sarah from Eng, Marcus, the new PM Priya. Tomorrow at 1:30 PM, it's virtual on Microsoft Teams."* - **System context:** *"You are a calendar assistant. Today is Monday, June 29. Use the available tools to process the user's request."*
Examining the emitted JSON, Pamela highlights that the model itself never runs the function; it only emits the requested name and arguments payload for the host application to run. Comparing the actual generated payload to the expected baseline, GPT-5.5 maps the attendees, relative date (June 30), and platform correctly, though it redundantly lists `"me"` as an attendee. The stage is then set to evaluate how DeepSeek handles identical function schemas and argument extraction.
[20:00](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1200s) Tool Calling Ambiguity and Argument Constraints
Evaluating individual tool calls reveals whether a model strictly adheres to provided schemas and constraints. When testing DeepSeek, Pamela Fox points out that it failed to respect the exact instructions: it listed "you" as an attendee, and for the meeting location, it output `"virtual (Microsoft Teams)"` instead of following the requested format. When models deviate like this, it exposes ambiguities in the prompt that require immediate tightening:
> “So once I see this, I can start to realize, like, okay, there's some ambiguity, right? And, you know, if it called it with the wrong arguments, that means there's ambiguity, I need to go back into my descriptions and say, you know, don't include me or you just to, named attendees.”
Iterating on descriptions—such as explicitly requiring full names or specifying exact format rules—allows developers to clarify expectations before re-evaluating across diverse test cases.
[21:03](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1263s) Tool Calling in a Loop: Efficiency in Word Problems
Moving toward agentic architectures requires tool calling inside an execution loop where the model repeatedly invokes functions until reaching a stated objective. Fox sets up an experiment with a simple `calculate` function that evaluates a mathematical string expression to solve a store sale word problem (e.g., purchasing three shirts, applying discounts, and factoring in sales tax). To prevent models from cheating by doing mental math, she enforces a strict system prompt rule:
> “The only system prompt is do not do math in your head. OK. So let's see. That one's really for GPT-5.5, 'cause it's so proud of its math abilities that it's always trying to do it in its head, like, come on.”
This setup directly measures execution efficiency: - **GPT-5.5** proves to be the most efficient: it bundles the full multi-step word problem into a single, comprehensive arithmetic expression, completing the entire operation in exactly one tool call. - **DeepSeek V4 Flash** reaches the correct answer but requires four separate sequential tool calls, creating significant latency overhead because each call translates to an extra network request. - **Kimi K2.6** matches GPT-5.5's quality most of the time, but Fox notes its tendency to enter redundant loops:
> “Yeah, the other day when I did Kimi, it got the right answer on the first try and then decided to check its work with the four other calls. I was like, no, you're good, you're good.”
```mermaid flowchart TD accTitle: Single-Turn vs Multi-Turn Tool Calling accDescr: Diagram contrasting how GPT-5.5 solves math in one compound call versus DeepSeek solving it across multiple steps.
subgraph Problem["Word Problem Input"] P["Store Sale & Tax Query"] end
subgraph Efficient["GPT-5.5 Pattern (1 Round-Trip)"] G1["Batch full expression"] --> G2["Single calculate() call"] end
subgraph Iterative["DeepSeek Pattern (4 Round-Trips)"] D1["Sub-calc 1"] --> D2["Sub-calc 2"] D2 --> D3["Sub-calc 3"] D3 --> D4["Sub-calc 4"] end
P --> Efficient P --> Iterative ```
[23:58](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1438s) Building Agents: LangChain v1 Travel Agent
To demonstrate full agent loops, Fox uses a travel planner scenario, selecting LangChain v1 after taking an informal audience vote over PydanticAI and other agent frameworks. The agent connects to Microsoft Foundry models and exposes four specific tools: 1. `search_flights` 2. `search_hotels` 3. `check_budget` (determines if the combined flight and hotel stay fit the allowance) 4. `search_activities` (discovers local events using remaining funds)
Custom middleware inspects and logs the tool interactions in real time. The agent receives a scenario-specific prompt: a user planning a 3-night trip from San Francisco (SF) to New York City (NYC) next month with a total budget of $600, requiring direct flights and an activity suggestion within the leftover budget.
[26:17](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1577s) Parallel Tool Calling and Model Discrepancies
The travel planner run highlights major differences in how models handle parallelization, context length, and instruction following over multi-turn interactions. Evaluating these differences is notoriously difficult:
> “And you can start to see why it's hard, because everybody asks me, like, how do you evaluate agents? Agents are the hardest to evaluate because there's so much going on. There's so many different things that are happening along the way. It's like, is it calling the right tools? Is it calling them efficiently? Is it calling things in parallel?”
| Model | Parallel Execution | Execution Breakdown | Result / Failure Mode | |:--- |:--- |:--- |:--- | | **GPT-5.5** | **Yes** | Turn 1: parallel calls to `search_flights` and `search_hotels`. Turn 2: `check_budget`. Turn 3: `search_activities`. | Achieved maximum efficiency by parallelizing independent search operations. | | **Kimi K2.6** | **No** | Turns 1–2: calls `search_flights` and `search_hotels` sequentially. Turns 3–4: calls `check_budget` twice. Turn 5: calls `search_activities`. | Required two extra round-trips; did not invoke independent tools in parallel. | | **DeepSeek V4 Flash**| **No** | Sequential execution across four single tool calls. | Reasonably efficient with no extra redundant calls, but lacks parallelism. | | **Mistral Large 3** | **No** | Called `search_flights` and `search_hotels` sequentially, then stopped. | Forgot subsequent instructions; failed to call `check_budget` or `search_activities`. |
Fox points out that as context windows fill up across agent turns, models like Mistral can drop early instructions and require external steering to finish their work:
> “So Mistral decided to only search flights and search hotels, and then not do anything else. So it did, it kind of gave up there. So that's an example where, okay, we'd have to do some additional steering there to be like, hey, are you sure that you have everything, right? You could use something like the Ralph loop. Jeff Huntley is here today. He made the Ralph loop.”
[30:00](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1800s) Bulk Automated Evaluations and String-Check Evals
Pamela Fox concludes her agent discussion by observing that across different agent frameworks, whether in LangChain or elsewhere, the underlying mechanics remain essentially the same: “I tend to think most agent frameworks are, you know, more similar than they are different at this point.” She shifts the workshop focus toward evaluations, moving away from manually executing prompts and eyeballing outputs toward building bulk, automated test suites that can reliably compare model performance in production.
```mermaid flowchart TD accTitle: Single Tool Call String-Check Evaluation Pipeline accDescr: Test inputs flow into the candidate model, producing tool call arguments that are validated using string comparisons against ground truth expectations.
subgraph Inputs ["Test Suite"] TC["Test Cases<br>(Date math, slang, multilingual)"] end
subgraph Execution ["Model Under Test"] M["Model executes<br>create_calendar_event"] end
subgraph Verification ["Deterministic Assertions"] SC{"String Checks"} Pass["Pass"] Fail["Fail"] end
TC --> M M --> SC SC -->|"Matches expected values"| Pass SC -->|"Missing or incorrect args"| Fail ```
Fox begins with the simplest form of evaluation: deterministic string checks on structured tool calls. Using the `create_calendar_event` tool, she demonstrates how to inspect whether the model invoked the expected tool and populated arguments correctly—such as verifying that a meeting location was returned as `"virtual"`. She stresses that real-world test suites need dozens or hundreds of test cases and must be easy to update once production users inevitably uncover edge cases. Once an application is live, user traffic exposes unforeseen failure modes, requiring continuous updates to the evaluation dataset to prevent evaluation drift.
- **Date Math Vulnerabilities**: Fox deliberately incorporates date math edge cases because LLMs consistently struggle with mental arithmetic across calendar boundaries, especially during leap years. Her concrete engineering recommendation: “So another tip, if you're doing dates, just give them a tool to do date math, okay? Because it's too hard to do date math in your head.” - **Diverse Edge Cases**: The test suite includes ambiguous durations, multilingual prompts (such as queries in Spanish), and informal slang inputs like: *"Yo, can you book the big comp room?"* - **Execution Strategy**: While production workflows typically run evaluations on a single model and compare against historical benchmarks, Fox runs the entire test suite across all four models concurrently to display comparative pass/fail rates in real time. - **CI/CD Practicalities**: Fox warns against treating LLM evaluations like standard unit tests in pytest: “Now, here's the thing about evals, is that you don't want to just have these like, just be part of like your pytest like your test suite because these are slow these are expensive.”
---
[33:42](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2022s) Q&A: Evaluation Frameworks and Observability Tools
An audience member asks whether existing frameworks handle evaluations more elegantly than basic string comparisons, particularly for parallelization and complex assertions. Fox jokes in response, “You don't like my string checks,” before detailing the modern ecosystem of evaluation and observability tooling.
She notes that nearly all major observability platforms come equipped with built-in evaluation capabilities. Teams using LangFuse, LangSmith, Arize Phoenix, or LangChain can leverage native evaluation primitives. Within the Microsoft Foundry ecosystem, developers have access to both the Azure AI Evaluation SDK and `openai.evals` from the OpenAI package. Other notable libraries include DeepEval and Promptfoo (the go-to evaluation framework for JavaScript and TypeScript developers). An audience participant mentions VALL-E / Vally, which Fox acknowledges as one of many emerging tools. However, she points out that for straightforward string assertions on single tool arguments, rolling lightweight custom checks avoids unnecessary framework complexity.
---
[35:12](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2112s) LLM-as-a-Judge for Fuzzy Outputs and RAG Groundedness
As evaluation requirements advance beyond exact string matches to unstructured text and fuzzy outputs, Fox introduces the LLM-as-a-judge paradigm. When evaluating conversational responses or RAG groundedness, simple string checks break down, requiring an evaluator LLM to reason over the outputs.
```mermaid flowchart TD accTitle: LLM-as-a-Judge Architecture for Groundedness accDescr: An application query and context yield a model response, which is fed alongside the context and query into a judge model to produce a structured pass or fail verdict with a reasoning trace.
subgraph AppRun ["Inference Run (App)"] Q["User Query"] CTX["Retrieved Context"] M["Model Under Test"] Q & CTX --> M M --> R["Model Response"] end
subgraph JudgeRun ["Evaluation Run (Judge)"] JM["Judge Model<br>(GPT-5.5)"] Q & CTX & R --> JM JM --> JOUT["JSON Output<br>Verdict: Pass / Fail<br>Reason: Detailed Explanation"] end ```
To configure an LLM judge effectively, Fox puts the evaluator model into judge mode with clear system instructions: it must return a binary pass/fail verdict along with a mandatory, explicit reasoning trace explaining its judgment. For groundedness evaluations, the judge must receive the user query, the candidate model's response, and the retrieved context documents. Evaluating more complex interactions like agentic tool use requires providing full execution traces, tool definitions, and intermediate state transitions.
- **Dual-Call Latency and Expense**: In LLM-as-a-judge pipelines, every single evaluation requires at least two separate inference calls—one to generate the application response and another for the judge model to evaluate it. Running multiple specialized evaluators (such as groundedness, relevance, and safety) multiplies the number of LLM invocations per test case. - **Production Dataset Sizes**: Fox highlights that a robust production evaluation dataset for RAG requires a minimum of 200 test cases, often scaling to 800 or more, making full runs across multiple models time-consuming and costly to execute on every commit. - **Decoupling Generation from Evaluation**: Fox recommends pre-running and caching application questions, contexts, and answers. Decoupling data generation from evaluation allows engineers to iterate on and refine the judge prompt independently without re-invoking the primary application pipeline. - **Judge Calibration**: When evals fail, engineers face a core dilemma: “So that's what you have to figure out when you're doing LLM as judge, you have to figure out like, wait, is the problem my app? Or is the problem my judge, right?” Fox stresses that automated judges must be calibrated against human domain experts who label ground truth and validate whether the judge's pass/fail verdicts align with human standards. - **Model Groundedness Results**: In Fox's live run using GPT-5.5 as the evaluator judge, most evaluated models scored 2 out of 3 on the test cases, while Mistral achieved only 1 out of 3, demonstrating remaining gaps in grounded response adherence. - **Community Resources**: Fox highlights Hamel Husain (`hamil.dev`) as an essential industry resource for learning LLM evaluations, course material, practical debugging methodologies, and evaluation memes.
[40:07](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2407s) ASSERT: Generating Evals and Test Sets from a Spec
Pamela recommends following Hamel Husain and taking his course to learn industry best practices for crafting effective evaluations, pointing out how difficult it is to engineer realistic test sets before going into production. To address this exact problem, Microsoft Research recently released an evaluation tool called ASSERT, which automatically constructs LLM-as-a-judge rubrics and generates synthetic user test sets directly from a high-level application specification. Pamela demonstrates how the tool was applied to the LangChain travel planning agent shown earlier in the workshop, working from a YAML configuration file stored under the `assert_eval` directory.
The configuration file defines the context of the agent, specifies the judge model, and establishes the behavioral dimensions under test. A core capability highlighted by Pamela is stratification: developers can explicitly define user segments—such as categorizing budget pressure into ample budget, tight budget, or infeasible budget—or allow ASSERT to derive the stratification automatically. Because hand-authoring evaluation specifications can be tedious, an agent skill is provided to help generate the YAML file. When ASSERT executes, it hooks into the LangChain travel planner and traces every intermediate tool call, since full trace visibility is essential for judging whether an agent followed instructions or violated constraints.
```mermaid flowchart TD accTitle: ASSERT Spec-to-Eval Pipeline accDescr: Pipeline showing how ASSERT turns a YAML spec into stratified scenarios, runs agent traces, and evaluates behavioral rule violations. subgraph Specification ["1. Spec Definition"] A["YAML Spec & Context"] --> B["Stratification & Dimensions<br/>(e.g., Budget Pressure)"] end subgraph Generation ["2. Test Generation"] B --> C["Synthetic User Scenarios<br/>(Allowed vs. Disallowed Behaviors)"] end subgraph Evaluation ["3. Trace & Audit"] C --> D["LangChain Travel Agent Run<br/>(Full Tool Call Tracing)"] D --> E["Violation Scoring & Rubric Dashboard"] end ```
Running ASSERT takes significant compute time, so Pamela walks through pre-computed evaluation runs displayed in ASSERT's dedicated frontend viewer. The tool derived distinct behavioral categories, such as budget and feasibility rules, direct flight preference violations, and required tool usage compliance. To simulate real-world traffic, ASSERT generated an extensive, colorful test set of synthetic user scenarios with distinctly LLM-generated titles, including "scenario 65 red eye before the installer walked through the coach with six parent text one weekend to see ama" and "coach with the clipboard."
“What I realized about YAML is that I might hate YAML, but it's also better than everything else for what I need it for, so there we go,” Pamela jokes while inspecting the configuration. Explaining the fundamental challenge the tool solves, she emphasizes: “You go into production but you want to have evals before you go into production right so how do you figure out you know what your test set is going to be and so this assert framework helps a lot with making that test set and bootstrapping your evals.”
- **Tool Origins**: ASSERT was open-sourced by Microsoft Research approximately one week prior to the workshop. - **Spec Stratification**: Supports pre-stratification or automatic stratification across application-specific variables (e.g., budget levels from ample to mathematically impossible). - **Trace Auditing**: Evaluators inspect the full execution trace and tool calls rather than grading only the final text output.
---
[43:32](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2612s) ASSERT Results Across Models
Pamela displays the comparative ASSERT benchmark results for the travel planner across DeepSeek, Kimi, and GPT-5.5. The metric reported in this evaluation suite tracks the total number of rule and constraint violations across scenarios, meaning lower scores indicate superior performance. In this trial, GPT-5.5 outperformed the other models by a wide margin, registering the lowest number of violations. Pamela admits she was surprised by the gap, as her earlier interactive tests throughout the day suggested that Kimi was performing roughly on par with GPT-5.5.
Reflecting on broader evaluation practices in the developer community, Pamela observes a pervasive antipattern: teams either avoid running evaluations entirely, or they rely exclusively on generic out-of-the-box metrics like coherence and fluency. She warns the audience against relying on these superficial checks, explaining that modern frontier models are universally fluent and coherent by default. While standard groundedness and relevance evaluators have utility in retrieval-augmented generation (RAG) pipelines, genuine agent validation requires bespoke evaluators tailored to the application's actual business rules and safety boundaries.
“Never run those,” Pamela warns regarding default coherence and fluency metrics. “Like, LLMs are great at being coherent. You never have to check if an LLM is coherent. If they're not, like something really bad has happened.” She underscores the necessity of targeted evaluation: “You need custom evaluators to really see if your agent is doing a good job, and but you know you need some help bootstrapping those custom evaluators.”
- **Scoring Direction**: Metrics measure behavioral rule violations; lower numerical scores represent better agent compliance. - **Model Standouts**: GPT-5.5 achieved the fewest rule violations, beating DeepSeek and Kimi on the travel planner test suite. - **Eval Traps**: Off-the-shelf fluency/coherence evaluators provide zero signal on agent correctness; only custom evaluators aligned with functional constraints reveal failures.
---
[45:52](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2752s) Prompt Optimization with DSPy
With only a few minutes remaining on the workshop timer, Pamela addresses the friction of prompt engineering across different model families. Manually rewording instructions every time an application swaps its underlying model is labor-intensive, particularly because developers rarely know the idiosyncrasies of every alternative model. To demonstrate programmatic prompt discovery, she introduces DSPy, a framework that flips traditional prompt development: instead of hard-coding prompts, developers write declarative program constraints and evaluation checks, allowing an optimization algorithm to compile and optimize the prompt automatically.
```mermaid flowchart TD accTitle: DSPy Prompt Compilation Pipeline accDescr: Diagram illustrating how DSPy compiles programmatic constraints into model-specific prompts via genetic hill-climbing algorithms. subgraph Codebase ["1. Declarative Code"] A["Task Constraints & Spec"] --> B["Deterministic Checks & Assertions"] end subgraph Optimizer ["2. Hill-Climbing Loop"] B --> C["GEPA / Genetic Pareto Optimizer"] C <-->|"Student Model + Reflection Model"| D["Iterative Prompt Mutation"] end subgraph Artifact ["3. Output System Prompt"] D --> E["Optimized Model-Specific Prompt<br/>(e.g., Mistral Large)"] end ```
Pamela demonstrates a complex formatting challenge: prompting a model to produce a numbered list where every individual line matches an exact word count and the first letter of each line forms an acrostic spelling a target word. In DSPy, these requirements are expressed as deterministic Python validation functions. DSPy then runs an optimization loop using GEPA (Genetic Pareto Algorithm), which pairs a student model with a reflection model to perform hill-climbing search across candidate instructions until it finds a prompt that satisfies the constraints.
On this test task, the DSPy compilation process ran for approximately six minutes and successfully produced an optimized system prompt tailored specifically for Mistral Large—the model that had struggled most during the workshop's manual tests. Pamela notes that major tech companies like Dropbox and Shopify have shared talks documenting how they use DSPy alongside GEPA to migrate production workloads from expensive flagship models to substantially cheaper, smaller models deployed on private infrastructure without sacrificing output quality.
“You kind of have to rewrite your whole code because now instead of checking your, like, your prompt isn't in your code,” Pamela explains. “Your constraints are in your code. Your spec is your code. The prompt is an output from this DSPy process.” Highlighting the cost benefits of automated prompt compilation, she adds: “So especially when using smaller models, this means we can use the cheapest model and still get the results we need, right?”
- **Paradigm Shift**: The system prompt is treated as a compiled artifact rather than source code; the program source contains only the input/output signatures and validation tests. - **Optimization Engine**: Employs GEPA (Genetic Pareto Algorithm) to mutate and hill-climb candidate instructions evaluated against defined test assertions. - **Cost Reduction**: Enables developers to systematically unlock baseline performance on cheap, open, or self-hosted models (such as Mistral) that fail on unoptimized zero-shot prompts.
---
[49:22](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2962s) Anthropic Native Citations and Ollama Local Models Wrap-Up
In her final remarks, Pamela highlights a notable feature of Anthropic's Claude models that distinguishes them from standard OpenAI-compatible endpoints: native, structured citation extraction. Although Anthropic requires using a separate Messages API rather than the unified OpenAI chat completions format used throughout the workshop, their models are natively trained to handle structured source inputs. When a developer provides text documents in Anthropic's structured source format and enables citations, the model returns response blocks paired with precise, programmatic citation offsets, making it straightforward to render clickable source links in user interfaces without post-hoc parsing hacks.
Pamela concludes by letting attendees know that the workshop GitHub repository will remain available for them to continue experimenting with model swapping. However, she warns that the API keys currently embedded in the repository's `.env` file will be rotated immediately, joking that she trusts the audience, but not enough to leave live credentials active after being scolded by her colleague Cedric. To encourage free, private experimentation without hosted cloud keys, she notes that she added Ollama support to the codebase the previous day; developers can point the workshop examples at a local Ollama server simply by adjusting the base URL and model identifiers.
“With Anthropic, they actually have this notion of citations built into the models,” Pamela explains, detailing their unique API architecture. “So you say, hey, this is my source. It's a document. It's text. Here's the text of it and then you enable citations and then Anthropic models are actually trained to return back blocks of text with citations.” Addressing the workshop credentials, she quips: “I do need to rotate the keys that I put in that.env because I mean I trust you but I don't know how much I trust you and Cedric already said I was bad for doing that.”
- **Anthropic Native Citations**: Accepts structured document payloads via the Messages API and returns machine-readable text blocks tied to citation references. - **Local Testing via Ollama**: The entire workshop repo supports local model execution by pointing client instances to an Ollama server running compatible endpoints. - **Key Rotation**: Exposed workshop `.env` keys are actively revoked; users must supply their own hosted endpoint keys or run locally via Ollama.
## Notable quotes > [13:03](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=783s) When it comes to multimodal understanding, you can't really do much prompt engineering because either the model understands that image or it doesn't. — Pamela Fox
> [05:00](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=300s) you don't really know how good a model is until you use it for your particular scenario. — Pamela Fox
> [43:32](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=2612s) LLMs are great at being coherent. You never have to check if an LLM is coherent. — Pamela Fox
> [30:00](https://www.youtube.com/watch?v=UGjk3Uk0MxM&t=1800s) I tend to think most agent frameworks are, you know, more similar than they are different at this point. — Pamela Fox
## Mentioned Pamela Fox, Cedric, Jeff Huntley, Hamel Husain, Microsoft, Microsoft Foundry, Microsoft Research, Anthropic, GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, Mistral Large 3, Claude, LangChain, PydanticAI, DSPy, ASSERT, GEPA, uv, Python, GitHub Codespaces, OpenAI SDK, Ollama, Dropbox, Shopify, LangFuse, LangSmith, Arize Phoenix, DeepEval, Promptfoo, Azure AI Evaluation SDK, pytest
## Video description Benchmarks won't tell you how a model behaves in your app. Swapping it in and testing will.
Pamela Fox, Principal Cloud Advocate at Microsoft, runs the same scenarios against GPT-5.5, Kimi K2.6, DeepSeek V4 Flash and Mistral Large 3 on Microsoft Foundry so you can see where models differ and what you can do about it. She works up from a letter-counting prompt to RAG groundedness, image input, multilingual pitfalls, tool calling, tool calling in a loop and a LangChain v1 travel agent, then moves to evaluation: string-check evals, LLM-as-a-judge, Microsoft's new ASSERT tool for generating test sets from a spec, and DSPy prompt optimization that finds a prompt the cheapest model can satisfy.
In this video: - Where top models differ: counting, groundedness, vision, tool calls and parallel tool calling - How prompt engineering closes some gaps but not image understanding - Bulk evals with string checks and LLM-as-a-judge - Generating test sets with ASSERT and optimizing prompts with DSPy
SPEAKER Pamela Fox, Principal Cloud Advocate, Microsoft X: https://x.com/pamelafox LinkedIn: https://www.linkedin.com/in/pamela-s-fox GitHub: https://github.com/pamelafox
LINKS ASSERT (GitHub): https://github.com/responsibleai/ASSERT
CHAPTERS 0:00 Workshop setup 2:23 Why you need to be able to swap models 3:08 The four models 5:18 Benchmarks vs your scenario 5:38 Test 1: counting letters 7:48 Fixing it with prompt engineering 8:38 Test 2: RAG and groundedness 13:03 Test 3: image input 16:18 Switching languages 17:13 Test 4: tool calling 21:03 Test 5: tool calling in a loop 23:58 Agents with LangChain v1 26:17 Parallel tool calls 30:27 Evals with string checks 33:42 Q&A: eval frameworks 35:12 LLM as a judge 40:07 ASSERT: evals from a spec 43:32 ASSERT results across models 45:52 Prompt optimization with DSPy 47:52 A DSPy prompt for Mistral 49:22 Anthropic and citations 50:32 Wrap-up
Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI.
AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/
#LLM #Evals #AIEngineer
[Source](https://www.youtube.com/watch?v=UGjk3Uk0MxM)