Snorkel 4B Model Beats 235B Qwen on Tool Tasks for Under $500
AI Engineer
Enterprise language models fail at analytical tasks due to poor tool discipline rather than weak reasoning, allowing small models trained on schema discovery to outperform massive networks.
A 4B parameter model fine-tuned using GRPO outperformed the 235B parameter Qwen 3 on financial tool-use tasks by mastering disciplined environment interaction rather than raw reasoning scale. (0:16)
The entire 4B model RL training run completed in roughly 21 hours and cost under $500, demonstrating that high-performing domain-specific behavior is accessible without massive infrastructure. (10:48)
Training exclusively on single-table queries transferred cleanly to complex multi-table tasks, nearly doubling performance on the FinQA reasoning benchmark from 13.9% to 26.6%. (16:23)
Decomposing model evaluations into multi-question rubrics reveals exact behavioral failure points to guide data curation, before condensing metrics into a scalar reward for reinforcement learning. (18:37)
Detailed notes
## Background
Published on June 10, 2026, by the AI Engineer channel on YouTube, this talk features Kobie Crawford, a developer advocate at Snorkel. The presentation demonstrates how a 4B parameter model fine-tuned with reinforcement learning for under $500 outperformed a 235B parameter model on financial tool-use tasks by emphasizing tool discipline over model scale.
## TL;DR
Crawford challenges the standard industry reflex of swapping in larger, expensive models when an AI agent fails at complex tasks. Through a financial analysis benchmark developed with UC Berkeley researchers, he contrasts a 235B Qwen model—which hallucinated after failing to verify database schemas—with a disciplined 4B model trained via GRPO that discovered tables, inspected schemas, and self-corrected queries. Crucially, training exclusively on single-table tasks cleanly transferred to multi-table reasoning, proving that tool discipline and structured diagnostic rubrics matter far more than raw reasoning scale.
## Atmosphere & style
The presentation carries the brisk, pragmatic, and high-signal energy characteristic of an AI Engineer stage talk. Crawford speaks with engaging conversational humor and measured confidence, contrasting the absurdity of using massive reasoning models as a 'sledgehammer to crack a walnut' with accessible, cost-effective engineering realities. It feels like an encouraging, practical wake-up call for production engineers, blending rigorous experimental benchmarks with approachable, relatable storytelling.
## Bulletin
- [00:16](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=16s) A 4B parameter model fine-tuned using GRPO outperformed the 235B parameter Qwen 3 on financial tool-use tasks by mastering disciplined environment interaction rather than raw reasoning scale.
- [10:48](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=648s) The entire 4B model RL training run completed in roughly 21 hours and cost under $500, demonstrating that high-performing domain-specific behavior is accessible without massive infrastructure.
- [16:23](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=983s) Training exclusively on single-table queries transferred cleanly to complex multi-table tasks, nearly doubling performance on the FinQA reasoning benchmark from 13.9% to 26.6%.
- [18:37](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=1117s) Decomposing model evaluations into multi-question rubrics reveals exact behavioral failure points to guide data curation, before condensing metrics into a scalar reward for reinforcement learning.
## Key takeaways
- [03:32](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=212s) Teams frequently default to deploying larger models when production tasks fail, misdiagnosing behavioral execution failures as a lack of general reasoning capacity.
- [03:32](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=212s) Deploying 4B parameter models provides crucial operational advantages in latency, cost efficiency, and local on-premise data control for regulated industries like finance and healthcare.
- [03:32](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=212s) Reinforcement learning is uniquely suited for tool use because it alters operational behavior and environmental interaction rather than attempting to force factual memorization.
- [07:10](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=430s) The 235B parameter Qwen 3 failed on a financial query because it guessed non-existent table names and hallucinated after errors instead of inspecting schema tools.
- [08:53](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=533s) Curating high-quality verifiable datasets with domain experts in the loop is the foundational prerequisite for effective reinforcement learning.
- [10:48](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=648s) Using the RLLM framework and GRPO, a self-contained FinQA training run was completed in 21 hours for under $500.
- [12:09](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=729s) The open FinQA environment was engineered to be completely self-contained to avoid external execution failures and ensure benchmark portability.
- [13:55](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=835s) The fine-tuned 4B model doubled its pass-at-one accuracy by systematically calling discovery tools, checking table schemas, and correcting its own SQL query syntax upon error.
- [16:23](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=983s) Single-table training curricula unexpectedly outperformed mixed and curriculum setups, proving that core tool discipline generalizes directly to complex multi-table reasoning benchmarks.
- [18:37](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=1117s) Multi-question diagnostic rubrics allow engineers to identify exact failure points in model execution before authoring targeted training datasets.
- [18:37](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=1117s) While detailed qualitative rubrics are necessary for diagnostic evaluation and dataset curation, GRPO training loops still require distillation into a single scalar value.
## Detailed walkthrough
### [00:16](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=16s) Introduction and Research Motivation
Kobie Crawford begins by reflecting on the high-signal nature of the AI Engineer conference, noting his role at Snorkel as a developer advocate. He introduces Snorkel, which he describes as the "Frontier AI Data Lab," focusing on high-quality data curation with an "expert-in-the-loop" methodology. By bringing in domain experts to help generate training data, Snorkel aims to provide the high-quality assets necessary for larger labs to achieve model performance gains.
Crawford clarifies the title "Stop Making Models Bigger, Make Them Behave," noting that it is a provocative, "punchy" title rather than a literal argument against large models. Instead, the team advocates for applying the right data to the right problem statement to achieve efficiency. He introduces a collaboration with the RLLM team—a research group at UC Berkeley—focused on an agentic project aimed at financial analysis.
- Goal: Make a 4 billion parameter model outperform a 235 billion parameter model on a tool-use task for financial analysis.
- Philosophy: "Stop making models bigger" is about identifying opportunities where the right data and behavioral tuning yield superior performance compared to simply scaling up the model.
### [03:32](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=212s) The Case for Behavioral Control Over Scaling
Crawford identifies a common pattern in enterprise AI: teams struggle to get a model to perform reliably in production and assume that the solution is to "just drop in a larger model." He argues that while larger models have greater reasoning capabilities, they are often an inefficient use of resources when faced with specific, constrained production tasks. By opting for a 4 billion parameter model, companies can gain significant advantages in speed, cost, and data control—the latter being especially critical in domains like finance and healthcare where keeping data on-premises is a major priority.
Crawford explains that when the goal is to improve how a model handles tools or interacts with an environment, Reinforcement Learning (RL) is an ideal tool because it focuses on changing the model's *behavior* rather than trying to force it to memorize new facts. He uses a "sledgehammer to crack a walnut" metaphor, comparing the brute-force application of massive reasoning models to the "Terence Tao effect": a mathematician of immense, general brilliance is not necessarily what you need to perform routine SQL queries and basic arithmetic for a financial dashboard.
- "We see very often that people choose like, 'Well, okay, we didn't get the performance that we wanted with this right now. We'll just drop in a larger model.'" — Kobie Crawford.
- "The idea that you must always get to a much smarter model to do something or deeper reasoning to get something done well is the thing we're challenging here." — Kobie Crawford.
### [07:10](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=430s) Failing to Use Tools: The 235B Model
To demonstrate the limitations of raw model size versus proper tool discipline, Crawford shows a test case using the 235 billion parameter Qwen 3 model. When asked to determine the year-over-year ad revenue growth for YouTube between 2023 and 2024, the model failed to inspect the available tools or schema.
Instead of performing a discovery phase—checking what tables actually existed in the environment—the model attempted to query a table that did not exist. After receiving no data from the first attempt, it guessed again, failed a second time, and ultimately hallucinated an answer. Crawford points out that the model’s general reasoning prowess was entirely unhelpful because it lacked the "discipline" to correctly navigate the tool-use environment.
- The model’s failure sequence:
1. Failed to inspect the environment/schema.
2. Queried a non-existent table (first attempt).
3. Guessed a different query (second attempt).
4. Hallucinated a final answer when both queries returned null.
```mermaid
flowchart TD
subgraph Attempt
A["User Prompt"] --> B["235B Model"]
B --> C["Query Non-Existent Table"]
C --> D["No Data"]
D --> E["Guess Again"]
E --> F["No Data"]
F --> G["Hallucinated Answer"]
end
accTitle: Failure loop of the 235B model
accDescr: The 235B model fails to use tools by guessing tables rather than inspecting them, leading to a hallucination.
```
### [08:53](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=533s) Generating High-Quality Training Data
Kobie Crawford emphasizes that the foundation of a successful model—especially when using Reinforcement Learning (RL) on smaller parameter models—is the generation of a high-quality dataset. At Snorkel, this process relies heavily on an "experts in the loop" methodology. They utilize an internal platform to solicit work from specialists across various domains, ranging from PhD-level researchers to deep industry veterans with extensive financial analysis experience.
The team places a premium on quality, ensuring that every task defined within the dataset is verifiable and fit for purpose. They conduct rigorous verification steps to confirm that the tasks can be queried effectively and will return a concrete, expected answer. As Crawford notes: “Data quality again is a big emphasis for us. So we really make sure that's a the key.”
### [10:48](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=648s) The RL Workflow and Cost Efficiency
For the training phase, the team utilized a 4 billion parameter model and applied the GRPO (Group Relative Policy Optimization) technique. They employed an RLLM framework developed in partnership with UC Berkeley and built a custom FinQA environment. A standout detail of this experiment is the accessibility and affordability of the process: the entire training run took roughly 21 hours and cost under $500.
Crawford uses this result as a "call to action" for engineers who might feel discouraged by the perception that model improvement requires massive, expensive infrastructure. He suggests that if you are already hosting models on-premise or working with smaller architectures, RL is a highly tractable path to achieving production-grade performance. He jokingly adds, “Even if Karpathy doesn't like it.”
### [12:09](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=729s) The FinQA Environment
The FinQA environment is designed to be fully self-contained, eliminating external dependencies that could cause failures in remote data centers. It is designed for portability and has been published on GitHub, with support for PrimeIntellect’s infrastructure and integration into Hugging Face spaces. The environment contains two distinct benchmarks:
* **Standard FinQA:** 290 samples.
* **FinQA Reasoning:** 79 advanced samples requiring multi-table queries.
### [13:55](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=835s) Comparing Performance and Behavioral Shifts
When comparing the fine-tuned 4B model to the 235B model, the performance improvement was striking; the smaller, fine-tuned model essentially doubled its "pass at one" accuracy. Beyond just raw numbers, Crawford highlights a significant behavioral difference. While the larger model hallucinated after failing to find a table, the 4B model demonstrated disciplined "tool use":
1. **Discovery:** It queried `get_table_names` to identify available data.
2. **Schema Inspection:** It used `get_table_info` to understand the structure before querying.
3. **Self-Correction:** When it initially requested a "revenue" column that did not exist, it observed the error and successfully corrected its own SQL query.
Crawford remarks: “The reality is what it needed to do was to learn how to use tools.”
### [16:23](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=983s) Surprising Generalization Results
The team experimented with different training curricula: single-table only, mixed datasets, and curriculum learning (starting with single and graduating to multi-table). Surprisingly, training exclusively on single-table questions yielded the greatest performance uplift. Even more counterintuitively, this single-table training regime generalized effectively to the harder, multi-table FinQA reasoning benchmarks, where performance jumped from 13.9% to 26.6%.
This outcome leads to a core takeaway for the project: the performance bottleneck was not the model's inherent "reasoning" depth, but its tool discipline. By fixing the core failure mode—the inability to correctly interact with the environment—the model became significantly more capable across both simple and complex task sets. As Crawford concludes: “Interestingly enough again, the tool discipline, just knowing how to use the tools that are in the environment turned out to be a bigger deal than anything else in terms of how to make these models actually get better at what they need to do in this space.”
[18:37](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=1117s) Using Rubrics to Diagnose Model Behavior
Kobie Crawford emphasizes that the primary obstacle in model performance often lies in specific, granular behaviors rather than general model scale. To address this, the research team at Snorkel advocates for the implementation of detailed rubrics within their evaluation framework. By decomposing a model’s output into a comprehensive list of checkable questions, developers can move beyond simple binary "yes or no" grading. This structured feedback allows engineers to pinpoint exactly where a model is failing across multiple potential points of failure, providing the diagnostic clarity needed to determine what training data is actually required.
While these rubrics provide deep, qualitative insight for human analysis, Crawford notes that the Reinforcement Learning (RL) process—specifically methods like GRPO—typically requires a single numerical scalar to function effectively. Consequently, the team uses the detailed rubric for high-level diagnostic work and dataset generation, while still distilling the final outcome into the single value required to drive the actual RL training cycle.
* **Rubric-Based Evaluation Strategy:** Break model responses down into a set of specific questions to identify precise points of failure.
* **The Workflow:**
* **Diagnostic:** Use rubrics to analyze the "why" behind model errors and guide data selection.
* **RL Training:** Use a single value scalar to maintain compatibility with standard GRPO training loops.
* **Quote:** “And looking at each of those individual questions, you can then start to use the rubric as a way to find and intuit like and find where the actual problem is among all the multiple possible arenas.”
[19:45](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=1185s) Closing Remarks and Resources
Concluding the presentation, Crawford highlights the broader implications of solving specific, well-defined problems rather than simply relying on scaling models. He points the audience toward a blog post detailing the study, which includes further documentation and a partner post from the Agentyca team at UC Berkeley. As the session reaches its conclusion, Crawford notes that he will be available outside the venue to answer any follow-up questions from the audience, bringing the talk to a close amidst applause.
* **Further Reading:** The speaker references a blog post on the Snorkel website and a collaborative post with the Agentyca team from UC Berkeley.
* **Final Sentiment:** The focus remains on the "opportunity of what you can do with solving the right questions or the right problems."
## Notable quotes
> [03:32](https://www.youtube.com/watch?v=TNwJ1LMiENk&t=212s) The idea that you must always get to a much smarter model to do something or deeper reasoning to get something done well is the thing we're challenging here. — Kobie Crawford
## Mentioned
Kobie Crawford, Snorkel, AI Engineer, Qwen 3, RLLM team, UC Berkeley, Terence Tao, GRPO, FinQA, Andrej Karpathy, PrimeIntellect, Hugging Face, YouTube, Agentyca team
## Video description
Qwen 3 235B was asked for YouTube's year over year ad revenue growth from 2023 to 2024. It queried a table that didn't exist, tried again, got nothing back both times, and hallucinated an answer. The 4B model Snorkel finetuned with RL called `get_table_name` first, inspected the schema, ran a query, hit a column error, self-corrected, and got the right answer. The training run cost under $500.
Kobe Crawford covers why tool discipline matters more than reasoning depth for this class of tasks, how single table training transferred cleanly to harder multi table problems (13.9% to 26.6% on the FinQA reasoning benchmark), and why breaking evals into rubrics helps identify which specific behavior to fix before writing any training data.
Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page:
https://ai.engineer/talks/TNwJ1LMiENk-stop-making-models-bigger-make-them-behave-kobie-crawford-snorkel
Join our upcoming AI Engineer conferences (see https://ai.engineer landing page):
- AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers
- AIE CODE in SF, November 10–12 — the top AI Coding conference returns
- AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney
Snorkel developer advocate Kobie Crawford argues that enterprise AI failures often stem from poor tool discipline rather than a lack of reasoning power, demonstrating that a 4-billion-parameter model can outperform a 235-billion-parameter model when behaviorally tuned with reinforcement learning.
The Limits of Raw Model Scale (3:32)
Crawford observes that enterprise teams struggling with production reliability frequently default to dropping in larger models. While massive models possess broader reasoning capacities, Crawford characterizes applying them to routine tasks as using a sledgehammer to crack a walnut—analogous to hiring a world-class mathematician like Terence Tao to run basic SQL queries and arithmetic for a financial dashboard. In privacy-conscious sectors like finance and healthcare, running outsized architectures also exacerbates compute costs and complicates on-premises deployment.
To illustrate this limitation, Crawford highlights a benchmark test with the 235-billion-parameter Qwen 3 model asked to compute year-over-year YouTube ad revenue growth. Rather than inspecting the environment schema, the model repeatedly guessed non-existent table names. After consecutive queries returned no data, it hallucinated an answer instead of investigating why its queries failed, showing that general reasoning cannot compensate for undisciplined tool use.
Low-Cost Behavioral Tuning via GRPO (10:48)
To demonstrate an alternative to model scaling, Snorkel collaborated with UC Berkeley's RLLM team to train a 4-billion-parameter model for financial analysis. The team applied Group Relative Policy Optimization (GRPO) using a custom, self-contained FinQA environment designed to eliminate external runtime dependencies.
The entire training process completed in approximately 21 hours and cost under $500. Crawford presents this efficiency as evidence that teams already hosting compact models on-premises can achieve production-grade performance through behavioral reinforcement learning without prohibitive capital or compute requirements.
Tool Discipline and Unexpected Generalization (13:55)
Post-training evaluations revealed that the fine-tuned 4-billion-parameter model roughly doubled its pass-at-one accuracy compared to the 235-billion-parameter baseline. Crucially, the improvement reflected structural behavioral changes: the smaller model executed discovery calls (`get_table_names`), inspected database schema (`get_table_info`), and self-corrected when it encountered SQL syntax errors, such as requesting a missing column name.
Furthermore, the training curriculum revealed unexpected generalization. When trained exclusively on simple, single-table tasks, the model not only excelled at basic queries but also improved performance on complex, multi-table FinQA reasoning benchmarks from 13.9% to 26.6%. Crawford notes that tool discipline—learning how to reliably interact with the environment—proved to be the decisive bottleneck rather than deeper intrinsic reasoning.
Expert Data and Rubric-Based Diagnostics (8:53)
The behavioral tuning pipeline relied heavily on high-quality, verifiable training data produced through an expert-in-the-loop workflow, drawing on PhD researchers and finance veterans. Each task was curated to ensure deterministic, verifiable queries and concrete answers.
To diagnose failure modes, Snorkel implemented rubric-based evaluations that decomposed model outputs into granular, checkable criteria instead of binary pass/fail grades. While standard GRPO training loops require a single numerical scalar to optimize policy updates, human evaluators used the multi-criteria rubrics to locate specific mechanical breakdowns and curate targeted training data.
Crawford concludes that enterprise practitioners should isolate behavioral execution errors before assuming a task requires a larger model. For tool-assisted environments, teaching a compact architecture to inspect, query, and self-correct offers a faster, cheaper, and more controllable route to production reliability.
Source quotes
[3:32] Kobie Crawford: The idea that you must always get to a much smarter model to do something or deeper reasoning to get something done well is the thing we're challenging here.
Original transcript
[0:07] [music]
[0:16] >> This is the the last presentation I have
[0:17] to give this uh conference. So, I'm
[0:19] feeling already a little bit of the
[0:20] euphoria of like, "Ah, it's all done."
[0:23] Um
[0:24] I know we're we're at a uh
[0:26] at a uh also closer to the end of the
[0:28] whole sequence uh is hasn't been I mean
[0:32] I I keep finding these conferences to be
[0:34] like some of the highest signal that I
[0:35] get like wherever I'm wherever I go. So,
[0:37] how how generally is it people feel like
[0:39] like they're getting what they came for
[0:40] here?
[0:41] I'm just curious because like I you know
[0:42] we're we're Snorkel um when we we put in
[0:45] a sponsorship and we want to know that
[0:46] like people are getting what they want
[0:47] out of they know they're going to come
[0:48] back because we want to sponsor next
[0:49] time. We want to know people are happy
[0:50] about it. So, did did did you did you
[0:52] guys see what you wanted to see? Yeah?
[0:53] Yeah. Yeah, really good. Brilliant.
[0:55] Brilliant.
[0:57] Um so, now that it is 3:45, I'm going to
[1:00] go ahead and start the the official
[1:02] thing. Uh so, uh my name is Kobe
[1:04] Crawford. Uh I'm a developer advocate at
[1:07] Snorkel. Uh we call ourselves the
[1:08] Frontier Data Frontier AI Data Lab. And
[1:11] the what we're doing right now our main
[1:14] thing is
[1:15] starting from the research-backed work
[1:17] that Snorkel's been doing since its
[1:19] inception.
[1:20] Uh we've been working on a variety of
[1:22] things about data quality and then then
[1:24] at this point now where where we're
[1:25] focused is actually providing data sets
[1:27] where we assure a certain level of
[1:29] quality. We're very attentive to being
[1:32] very very very motivated about making
[1:33] sure the data is high quality. And part
[1:35] of how we get to high quality is we
[1:37] always make sure to have sort of an
[1:38] expert in the loop as part of the
[1:40] process. So, we have uh expert
[1:42] contributors that we work with and we
[1:43] bring people in uh to provide their
[1:45] expertise to make sure that the data
[1:46] that we generate uh is of top quality.
[1:48] And then for the for the top labs that
[1:50] want to use our data to uh improve their
[1:52] models and get them get the hill
[1:53] climbing done uh in the right way, uh
[1:55] that's what we that's what we do at
[1:57] Snorkel. Um
[2:00] because of that, um you know, uh a lot
[2:03] of what goes on is uh still more
[2:04] research. And uh this is a a talk that's
[2:07] talking about some of the work that we
[2:08] did
[2:09] that our research team did. And um one
[2:11] of the keys in this research is uh that
[2:14] we're looking at uh
[2:16] how the best quality data can be best
[2:19] applied and like where it is that we
[2:20] need to be looking for where there are
[2:22] opportunities uh to get that done. So,
[2:25] in this particular case, talking about
[2:27] stop making models bigger. I mean, it's
[2:30] a nice punchy title. Of course, we don't
[2:31] really mean like models shouldn't be
[2:32] large intrinsically, but uh the point
[2:34] broadly speaking is that sometimes we
[2:36] find great wins to be had with the right
[2:38] data applied to the right uh to the
[2:40] right problem statement. So, this is
[2:42] something we're going to talk about a
[2:43] specific use case that we that our
[2:44] research team discovered and uh and in
[2:47] partnership with the RLLM team,
[2:50] uh which is a research group
[2:52] uh with a part of UC Berkeley.
[2:54] Uh and so, the UC Berkeley team over
[2:56] there, RLLM, the Agentic Project, their
[2:58] their lab partnered with us on this
[3:00] particular work.
[3:02] So, the goal this is the the as I said
[3:04] is making a 4 billion parameter model
[3:05] outperform outperform a 235 billion
[3:08] parameter model on a tool use task for
[3:10] financial analysis.
[3:14] So, we'll start with the research
[3:15] objective.
[3:17] And then we'll iterate in through
[3:18] talking about the approach that we that
[3:19] was used for this particular process and
[3:21] then talk about the results and
[3:24] um happy to report that we got what we
[3:27] were looking for. So,
[3:28] uh so so, good things to be had.
[3:30] So,
[3:32] couple of quick level setting
[3:33] backgrounds of the what we're talking
[3:34] about here. First is that
[3:36] as we see enterprise use cases uh take
[3:40] on some greater complexity, we have
[3:42] obviously we've got the massive
[3:43] explosion with people are doing in terms
[3:44] of personal assistance. And as people
[3:46] are working in the context of
[3:48] enterprise, a lot of times you still
[3:49] need a several more constrained uh
[3:52] choice about how to implement something
[3:54] and make sure that it's reliable. And
[3:56] like when you're looking for things that
[3:57] are going to be done uh for uh
[4:00] you know, enterprise production use
[4:01] cases, you kind of almost have to make
[4:03] sure there's a lot of safety and
[4:04] security things done. So, these other
[4:06] kind of priorities and that that fold
[4:08] into what people typically want to do,
[4:10] we're looking at these things and
[4:11] saying, "Okay, well, these are the
[4:13] enterprise use cases that people have."
[4:14] And as people try to solve the problems
[4:17] of making the models perform at the
[4:19] level that makes it like acceptable for
[4:20] actually being deployed as a production
[4:22] service,
[4:23] um
[4:24] we see very often that people choose
[4:26] like, "Well, okay, we didn't get the
[4:27] performance that we wanted with this
[4:29] right now. We'll just drop in a larger
[4:31] model. It'll be smarter. It has greater
[4:33] reasoning skills. And we'll just sort of
[4:34] expect that the important performance
[4:35] will improve uh
[4:37] commensurate with the additional load of
[4:39] the size of the model and the the the
[4:41] greater inference cost that goes along
[4:42] with that." And in some cases that might
[4:44] not always be the right thing. So, we
[4:46] see people saying, "You know, we'll just
[4:47] get a bigger model. That'll solve the
[4:48] problem." And uh sometimes maybe that
[4:51] isn't quite the answer.
[4:55] In this case, what we're trying to do is
[4:56] to say, "Can't we take a smaller model
[4:58] and then use RL
[5:00] with the right data
[5:02] to yield the kind of performance gains
[5:04] that we're looking for and to deliver
[5:05] the kind of application functionality
[5:06] that we want?" And so, that's the target
[5:08] here.
[5:09] Uh and again, for these various reasons,
[5:11] cost, speed, security, and then the the
[5:15] idea that in general, you know, you
[5:16] start with a really big model and
[5:18] make your POC and make it work and
[5:19] everybody's happy that it works. It's
[5:21] like, "Okay, now what do we do to
[5:22] productionize it?" You want to roll to
[5:23] production, you want to think about how
[5:24] you're able to deploy that. Do you need
[5:26] to keep everything on premise? You do do
[5:28] you have the ability to deploy and run
[5:30] your service yourself so that you don't
[5:32] have to have external dependencies and
[5:34] worry about the data export aspects and
[5:36] data control. Especially in the context
[5:38] of financial data and health care and
[5:40] other domains like that, people have to
[5:42] be concerned about those aspects as
[5:43] well. So,
[5:45] for getting a smaller model to be able
[5:48] to perform as well as larger models,
[5:50] uh we feel like in the particular case
[5:52] of talking about tool use for financial
[5:54] analysis that RL is the right time to to
[5:56] be uh making the kind of training.
[5:58] You're talking about like changing the
[5:59] behavior. And so, that's kind of more of
[6:01] a behavior thing. The RL's kind of
[6:02] better for behavior than say you're
[6:04] talking about like changing the core
[6:06] data and knowledge that's inside of
[6:07] that. So, that's that's an intuition
[6:08] about like how we've approached it and
[6:10] that's part of what's going on here.
[6:13] So, a larger model,
[6:15] sometimes it's more like uh taking a
[6:16] sledgehammer to crack a walnut. It's
[6:18] like just adding all of this capability
[6:20] is like this. And the RLLM team uh that
[6:23] we worked with, they talked about this
[6:24] and their description of it was
[6:26] the Terence Tao effect. Uh uh Terence
[6:29] Tao, the famous mathema- tician who's
[6:31] uh uh
[6:32] I forget what awards he's won and
[6:33] whatnot, but well known for being, you
[6:35] know, generally brilliant about
[6:36] mathematics across the board. And
[6:38] therefore, like could approach and
[6:40] manage any kind of mathematical problem.
[6:42] But that much brilliance might not
[6:44] necessarily be what a financial analyst
[6:46] actually has to have. They don't have to
[6:48] know all the kinds of math. They don't
[6:49] have to do latent Dirichlet algorithm
[6:52] stuff uh to talk about, you know, doing
[6:54] a SQL query and getting some math
[6:56] getting some some data's back and then,
[6:57] you know, doing some addition and
[6:58] subtraction, right? So, the the idea
[7:01] that you must always get to a much
[7:02] smarter model to do something or deeper
[7:04] reasoning to get something done well uh
[7:06] is the thing we're challenging here.
[7:08] Um so, here is that 235 billion quant
[7:11] three model responding to the question
[7:15] in this environment that we built. I'm
[7:17] going to talk about the environment a
[7:18] little bit more in detail later. But I
[7:19] point this out to sort of show
[7:22] here's a reasoning model, a smarter
[7:24] model, and its response in the context
[7:26] of needing to actually use tools.
[7:29] So,
[7:30] the response that it got that it
[7:33] generated to the question, "What is the
[7:35] year-over-year growth rate of YouTube
[7:37] ads revenue from '23 to '24?"
[7:40] began with uh making a query to find an
[7:44] existing uh some values, but they the
[7:46] query it chose was to a non-existent
[7:48] table. The table didn't exist. It didn't
[7:49] actually inspect the environment and
[7:52] inspect the tools to find out what
[7:54] tables it could query. It just threw a
[7:56] query out uh without doing that. Um so,
[7:59] the table it wasn't there and didn't get
[8:00] anything back. It guesses again.
[8:03] Still doesn't get anything back. And
[8:04] then having not gotten anything back in
[8:06] either of those two uh attempts, it
[8:08] falls back to just hallucinating an
[8:10] answer.
[8:11] And so, out comes this hallucinated
[8:12] answer. It's
[8:14] completely, you know, don't know what
[8:15] the weights told it to say, but that
[8:18] what came out. And, you know, it's it's
[8:20] not very useful.
[8:21] So, even though the model is incredible
[8:23] in terms of like much better at
[8:25] reasoning than a much smaller model
[8:26] would be,
[8:27] uh that greater reasoning did not help
[8:29] it when it needed to use the tools.
[8:32] We're going to come back to the same
[8:34] question again against the model that we
[8:36] fine-tuned that's only 4 billion
[8:37] parameters. And you're going to see the
[8:38] difference. We'll talk a little bit more
[8:39] about those differences later. So, put a
[8:41] pin in that. Come back. We'll see that
[8:42] that year-over-year question from
[8:44] come back.
[8:45] So, here,
[8:47] this is what we're talking about.
[8:48] Summarizing it again, no discipline in
[8:50] tool use even though it has all the
[8:53] the abilities to reason that it has.
[9:01] Moving forward to then what we did for
[9:03] this uh
[9:05] attempt to use RL to make the smaller
[9:07] model work well.
[9:10] Uh
[9:11] the first thing is to generate a
[9:13] high-quality data set.
[9:15] Um at Snorkel, our general approach is
[9:17] again to have experts in the loop. I
[9:18] don't know if I said I say again now,
[9:19] but I don't think I said it.
[9:20] We have experts in the loop for the data
[9:22] that we do. Um the way we generate data
[9:24] and the way we work on it is we have a
[9:26] platform that we've used internally for
[9:28] interacting with things we we solicit
[9:30] the work and support of experts on
[9:32] various tasks and various topics. So, if
[9:34] we need somebody who is
[9:36] in the financial analysis space already,
[9:38] then we get them and pull them in. We'll
[9:40] work with people at the PhD level for
[9:41] the their domains of expertise. Uh and
[9:43] also, of course, people who are deep in
[9:45] the industry and have been working for
[9:47] some time and then know the space well.
[9:49] Um
[9:50] The process of doing that that's like
[9:53] one of the things that we put an
[9:54] emphasis on is how we work at Snorkel
[9:55] for our data generation.
[9:57] And then broadly speaking,
[10:00] naturally that can be augmented with
[10:02] other kinds of things. But that's what
[10:04] we're really key about like what we want
[10:05] to do in terms of emphasizing quality as
[10:07] a core element. So we have the data set.
[10:12] And then we
[10:13] go through and make sure there's a
[10:15] verification step done to make sure that
[10:16] the tasks that are defined from that
[10:18] data set are actually appropriately
[10:22] fit fit it to the task and are actually
[10:24] like good tasks that they do in terms of
[10:25] like the it can be queried. You know,
[10:27] you you know that you're going to get
[10:28] the results that you need from it. And
[10:30] that that we should be able to have a
[10:32] verifiable answer that we're looking
[10:33] for. So we do all the verification steps
[10:35] to make sure that everything is correct
[10:36] on that front. And that's another part
[10:38] of what means to you know, put together
[10:40] the data set and have it ready for use.
[10:42] Data quality again is a big emphasis for
[10:44] us. So we really make sure that's a the
[10:45] key.
[10:48] And then it was time to do our RL with
[10:49] it. And the way we did our the way that
[10:51] this was done we're talking about very
[10:54] few surprises in terms of like you've
[10:55] seen the state of the art in this in
[10:56] this space. GRPO Again, we started with
[10:59] a 4 billion parameter model. And then
[11:03] the
[11:04] environment that we used the RLLM
[11:06] framework again through the UC Berkeley
[11:08] partnership through the the developers
[11:09] of that framework. And we have our a
[11:12] FinQA environment that we built. And
[11:15] we're going to talk a little bit more
[11:15] about the details about that environment
[11:16] in just a moment. But then
[11:19] this is something that was able to be
[11:20] done like in a
[11:22] 24 21-hour job and the total cost of
[11:26] running that job was under $500 per run.
[11:29] So
[11:30] um
[11:31] RL is not have to be a very expensive
[11:33] thing to be able to get non-trivial
[11:35] performance gains. And if you're already
[11:37] working about working on working with
[11:39] models that you want to host yourself,
[11:41] if you're already thinking about like
[11:43] what you'd like to do to be able to do
[11:44] things we have on prem kind of solutions
[11:46] or things where you're doing it with a
[11:47] smaller models. And you aren't already
[11:49] thinking that you can improve the models
[11:51] the way you that you want, then this is
[11:53] like a call to action that you actually
[11:54] can. That it's actually a very tractable
[11:56] thing to get a model that you want to
[11:58] work with actually up to the performance
[11:59] levels that you need using RL.
[12:03] Even if Karpathy doesn't like it.
[12:07] Um
[12:09] So
[12:10] our FinQA environment is something that
[12:12] we built. It's set up for being able to
[12:16] host the kinds of questions that are
[12:18] being done here. It provides a specific
[12:20] set of tools.
[12:21] It's set up where like everything is
[12:23] built into the environment. So there's
[12:24] no external dependencies that are that
[12:26] might be you know, in some remote
[12:30] data center that you don't have access
[12:31] to. So when you deploy the environment
[12:32] it's fully self-contained. Kind of roll
[12:34] out that if you worked with something
[12:35] like Harbor before or if you worked with
[12:37] like OpenEnv, you're familiar with the
[12:38] same thing about using an environment
[12:40] like this. And this is an environment
[12:41] that we've actually built and published.
[12:43] It's available
[12:44] on PrimeIntellect's infrastructure as
[12:46] well. You can so somebody can load up
[12:48] right there at PrimeIntellect. Also on
[12:50] OpenEnv and and and actually saved into
[12:52] the OpenEnv repo on GitHub. And then the
[12:55] OpenEnv the PyTorch folks and and
[12:57] Hugging Face folks team up and host
[12:59] these
[12:59] in Hugging Face spaces. So
[13:02] these kinds of things are accessible and
[13:04] easy to find if you want to take a look
[13:05] at them and see how you might take take
[13:07] them apply them to your needs. And then
[13:09] again like getting getting started with
[13:10] RL is actually
[13:12] easier and easier these days.
[13:15] We have the
[13:16] FinQA set up where we have 290 samples
[13:19] that way and we have our more advanced
[13:21] 79 samples called FinQA reasoning that
[13:23] requires multi-table
[13:26] queries. And so there's enough enough of
[13:29] the reasoning that has to be done across
[13:30] that to make that we've we've identified
[13:32] that these are harder tasks. And so we
[13:33] have like essentially two benchmarks
[13:35] that are built inside of this
[13:37] environment.
[13:41] So that's the set up of like how we get
[13:43] this done. We're going to go about
[13:44] talking about the evals and the results
[13:46] that we got working with this now given
[13:48] the RL that we just did on this 4
[13:50] billion parameter model.
[13:55] So we did it. It's
[13:56] performs better than the 235 billion
[13:59] parameter model
[14:01] with that RL training loop.
[14:03] And the performance in terms of pass at
[14:05] one was essentially double
[14:08] of what it had been percentage-wise in
[14:09] terms of like solving problems. So it's
[14:11] a very significant uplift that was done
[14:13] with this $500 loop. And again, the
[14:15] right data set
[14:17] and
[14:18] and and is really a key. You want to get
[14:19] the questions and answers to be actually
[14:21] things that are really going to help the
[14:22] model learn.
[14:23] But what is also interesting is what was
[14:27] important about what the model needed to
[14:28] learn.
[14:30] So
[14:32] just to give you a little flavor of like
[14:33] what that 4 billion parameter model
[14:35] looks like in terms of like how it
[14:37] behaves. And if you recall what we
[14:38] talked about earlier, the 235 billion
[14:40] parameter model
[14:41] tried some queries without knowing what
[14:43] the tables were.
[14:45] Didn't find anything and then
[14:46] hallucinated an answer. This 4 billion
[14:49] parameter model having been
[14:50] fine-tuned on this
[14:52] data set
[14:54] tries a table and actually first
[14:56] discovers the tables by using the tool
[14:58] get table names. The tool existed for
[15:00] the other model as well and it just
[15:01] didn't choose to try it. So the first
[15:03] thing it did was actually query to find
[15:05] out what tables it had available to it.
[15:06] So that's already like win. All right.
[15:08] The the second thing is then from there
[15:10] it went on to actually
[15:12] inspect the schema. Let me find out
[15:13] what's in that table so I know how to
[15:15] make the right SQL query.
[15:17] And so it's like does get table info to
[15:19] get the information back to know what to
[15:20] query it.
[15:22] Following that
[15:24] it runs a query.
[15:26] Actually ran into an error. It actually
[15:28] asked for
[15:29] the revenue column.
[15:32] But that column was not actually a part
[15:34] of the the data in the table.
[15:37] Given that error
[15:40] it actually corrected. It
[15:42] self-corrected. It observed the error,
[15:44] responded to that error by actually
[15:46] correcting to find the
[15:48] actual column that it needed.
[15:51] And so you're seeing both the error
[15:52] correction
[15:54] that it had learned how to do as well as
[15:57] the use of the tools to discover the
[15:59] right information in the first place. So
[16:00] between those two, those behaviors are
[16:03] the real keys to succeeding at these
[16:05] questions. And this is actually
[16:06] something like maybe not quite intuitive
[16:07] about like where it is that the model
[16:09] was failing. The reality is that what it
[16:12] needed to do and here it is getting the
[16:13] correct answer.
[16:15] The reality is what it needed to do was
[16:17] to learn how to use
[16:19] tools.
[16:23] Couple interesting things that go along
[16:24] with it that are
[16:26] more
[16:27] more fun and also really useful and good
[16:29] for our situation here.
[16:32] The training data that we talked about
[16:34] at the beginning, there were
[16:35] single-table questions, multi-table
[16:37] questions included in the overall data
[16:39] set.
[16:40] And for as part of the evaluation study,
[16:43] one of the things that they said was
[16:44] like let's take a look and see if we
[16:45] train with single table only, train with
[16:47] multi-table mixed in so the full data
[16:48] set across both types, or try to do some
[16:51] curriculum learning and actually start
[16:52] with single table, let the model kind of
[16:54] climb a bit and then progressively add
[16:56] multi-table. And it turned out the
[16:58] single-table only
[17:00] training was actually the one that
[17:01] yielded the greatest uplift for these
[17:04] kinds of questions.
[17:06] So that was a nice pleasant surprise.
[17:09] And
[17:11] the
[17:12] the other surprising thing
[17:14] was that
[17:15] even though the single-table only
[17:18] training regime was the best training
[17:21] regime the uplift that we see in terms
[17:23] of the model's performance on that
[17:25] harder benchmark that is a that has
[17:26] multi-table questions
[17:29] was a similar doubling in in in
[17:31] percentage improvement.
[17:33] So the harder multi-question multi-table
[17:36] Q&A
[17:38] in the FinQA reasoning
[17:40] question set
[17:42] also saw
[17:44] 13.9 to 26.6
[17:46] percentage
[17:47] jump after this training.
[17:50] So
[17:52] interestingly enough again, the tool
[17:54] discipline, just knowing how to use the
[17:56] tools that are in the environment turned
[17:58] out to be a bigger deal than anything
[18:00] else in terms of how to make these
[18:02] models actually get better at what they
[18:04] need to do in this space.
[18:07] So
[18:08] turned out it wasn't the reasoning that
[18:09] was the issue. It was the tool use.
[18:12] We focused only on single step for the
[18:14] best performance.
[18:16] And we're able to fix that core failure
[18:17] mode. And given that core failure mode
[18:20] being fixed, it turned out that that
[18:22] then made the model better in terms of
[18:24] like the improvement generalizing to
[18:26] other question sets.
[18:30] And so that means like that's the key to
[18:33] talk to take away from this is that
[18:35] sometimes the the idea is to find the
[18:37] specific behavior that's really the
[18:39] problem. And one of the things that I
[18:41] would go back to what we do at Snorkel,
[18:42] one of the one of the things that our
[18:43] research team has been talking about a
[18:44] lot lately is building rubrics as part
[18:47] of our evals. And then those rubrics
[18:50] by breaking down
[18:52] the rightness or wrongness of a model's
[18:55] response into a full
[18:57] list of different questions that can be
[18:59] answered. And looking at each of those
[19:01] individual questions, you can then start
[19:02] to use the rubric as as a way to find
[19:05] and intuit like and find where the
[19:06] actual problem is among all the multiple
[19:09] possible arenas. So instead of simply
[19:11] knowing yes or no at the final, which is
[19:14] good for the RL part
[19:16] you can use the rubric to help you do an
[19:18] analysis of what are the behaviors that
[19:20] you want to actually generate data sets
[19:21] to help you with. So you make decisions
[19:23] about what which data sets you need or
[19:25] which data you want to work with based
[19:26] on what you see coming out of the richer
[19:28] feedback that the rubric gives you.
[19:30] And then the RL still gets a single
[19:33] value or so GRPO just usually works with
[19:35] a single value. That's it. That's part
[19:36] of how it works. So you use that for the
[19:39] actual
[19:40] RL cycle.
[19:45] So that's the that's the summary of what
[19:46] we we did with that. We think it's a
[19:48] really interesting
[19:49] result to know. And you know, again the
[19:52] opportunity of what you can do with the
[19:54] solving the right questions or the right
[19:56] problems really helps. This link here is
[19:58] to a blog post that we have about this.
[20:00] So if you have questions about the
[20:02] details of this particular study and you
[20:04] want to see more about it, you can drill
[20:06] down within that. It also links to a
[20:08] partner post from the Agentyca team over
[20:11] UC Berkeley. So their post also has
[20:13] additional information you can see from
[20:14] them and and this is
[20:16] you know, the significant thing we
[20:18] wanted to talk about. So thank you for
[20:20] your time and I I don't know how much
[20:21] time we have left for questions or not.
[20:22] How are we doing?
[20:25] Are we already at time? Looks like it.
[20:26] Yeah, sorry. Okay, so I'm sorry we don't
[20:28] have questions. I'll hang out right
[20:29] outside if you want anybody has any
[20:30] follow-up questions they want to ask
[20:32] and thank you very much. Appreciate it.
[20:34] >> [applause]
[20:45] [music]