Going In Deep On Data | YC Paper Club

深度剖析数据 | YC Paper Club 主持人 Francois Chaubard 在开场时指出，在生产级 AI 系统中，数据整理而非模型架构才是价值千亿美元的真正瓶颈。来自 Snorkel AI 的 Vincent Sunn Chen 探讨了将专家监督拓展至“Data 2.0”的路径，介绍了 Senior SWE-bench 以及通过真实工程品味和执行力评估编程智能体的混合验证智能体。Inception 的 Volo Kuleshov 阐述了扩散语言模型如何实现超高速生成并结合 Tau-Forge 构建合成强化学习环境，而 Shayne Longpre 则概述了 ATLAS 框架，该框架形式化了跨语言迁移动态，并将 Chinchilla 缩放定律扩展至英语以外的数据混合场景。 数据基础设施和以数据为中心的企业已创造超过 1000 亿美元的企业价值，打破了早期风投认为数据是零利润大宗商品的设想，促使工程重心从模型架构转向数据调试。 (0:00) Snorkel AI、普林斯顿大学和威斯康星大学麦迪逊分校联合推出了 Senior SWE-bench，利用动态“验证智能体”从高级规范生成确定性测试，并通过“高品味通过”（tasteful pass）指标评估智能体的代码质量。 (18:59) Inception 介绍了非自回归扩散语言模型 Mercury 2，该模型在商用 GPU 上的生成速度超过每秒 1,000 个 token，并配合 Tau-Forge 合成复杂的企业客户强化学习环境。 (28:41) ATLAS 项目通过 700 多次实验运行将 Chinchilla 缩放定律扩展到多语言预训练，发现跨语言迁移具有不对称性，在很大程度上由共享文字系统驱动，并受模型参数容量增加的支持。 (40:21)

Presenters at the YC Paper Club argue that curated data pipelines, synthetic simulation environments, and empirical data mixing—rather than model architecture or sheer parameter scaling—form the primary operational bottleneck and performance differentiator in modern artificial intelligence. Data Debugging Over Architectural Tweaks (1:08) Francois Chaubard argues that treating data as an undifferentiated commodity was a historic industry miscalculation, noting that data infrastructure and data-centric companies have created over $100 billion in enterprise value since 2016. In production computer vision and deep learning, model performance bottlenecks routinely stem from physical edge cases—such as foggy grocery refrigerator glass or customers blocking shelf views—rather than suboptimal activation functions or network depth. Citing Andrej Karpathy's transition from spending 5% of his time on data during his PhD to 75% in industrial production, Chaubard contends that the Transformer era has shifted this ratio further to roughly 97% data and 3% architecture. Because models reliably break under production distribution shifts, deploying frontier AI across the economy requires continuous curation of specialized preference-style expert datasets and verifiable reinforcement learning environments rather than static file collections. Programmatic Supervision and Dynamic Agent Validation (18:59) Vincent Sunn Chen outlines how manual labeling fails in complex domains due to linear cognitive costs ($O(N)$), specification brittleness, and an absence of audit trails. Snorkel initially countered these constraints through weak supervision, using programmatic labeling functions and an unsupervised label model to estimate voter error rates and denoise conflicting heuristics. However, current agent workloads—characterized by long sequence lengths and high environment complexity—demand a shift to "Data 2.0." To evaluate autonomous coding agents without relying on brittle unit tests (which generate false negatives) or ungrounded LLM judges (which generate false positives), Snorkel and academic collaborators introduced Senior SWE-bench. The benchmark uses "validation agents" that ingest concise specifications from senior engineers and dynamically write deterministic execution scripts against candidate patches. Candidate code is graded via a "tasteful pass" metric, which evaluates adherence to codebase conventions, architectural design, and patch bloat alongside raw functional execution. Synthesizing Realistic Environments for Agent Evaluation (33:24) Volo Kuleshov asserts that standard agent evaluations like Tau-bench suffer from "benchmaxing," where models scoring above 90% on public leaderboards drop drastically in performance when exposed to live enterprise customer support logs. Because production environments feature roughly an order of magnitude more edge cases and operational specifications than public benchmarks, saturated test suites conceal wide disparities across model sizes. To bridge this gap, Inception developed Tau-Forge to synthesize complex reinforcement learning environments from corporate interaction logs and a web-scale Business Knowledge Graph. The platform builds domain policies, synthetic databases, and API tools, then introduces "hardening traps"—such as conflicting instructions or withheld information—to ensure tasks test agent reasoning. Training Inception's non-autoregressive diffusion model, Mercury 2.5, on these synthesized tasks improved accuracy on held-out business environments by over 23%, matching top frontier models. Empirical Data Mixing Beyond English-Centric Scaling (40:21) Shayne Longpre emphasizes that standard scaling laws are calibrated almost entirely on English tokens, leaving builders of lower-resource models caught between frontier dilution and monolingual overfitting. In MADLAD-400, for example, Thai represents only 0.6% of the token volume of English; pre-training on Thai alone quickly causes severe overfitting across repeated epochs, while general frontier mixes dilute the language to negligible fractions. Through the ATLAS project, Longpre shows that cross-lingual transfer efficiency is asymmetric and depends on model capacity: larger architectures convert representational competition into positive synergy, whereas small models suffer cross-lingual interference. Furthermore, shared script provides a stronger transfer benefit than language family membership due to subword tokenization properties. By decomposing Chinchilla's single dataset term ($D$) into target tokens, transfer tokens, and residual data weighted by a learned transfer factor ($\tau$), the framework accurately predicts loss across more than 700 pre-training runs and determines precise compute thresholds for deciding whether to fine-tune or pre-train from scratch. Across software engineering benchmarks, real-world business environments, and multilingual pre-training, the presentations demonstrate that model capability is fundamentally constrained by supervision quality, environment fidelity, and data distribution design. However, as autonomous agents take on increasingly subjective and prolonged tasks, creating verifiers and synthetic tasks that reliably reflect human expertise without reward hacking remains an ongoing technical challenge.

[0:00] Francois Chaubard: but he does know my name now, so that's good. [0:00] Francois Chaubard: I already have downloaded ImageNet. How much more data do I possibly need? [1:08] Francois Chaubard: You've trained your model, you're at 85% F1 score on hot dog / not hot dog. What do you do next? [1:08] Francois Chaubard: Nope. Next. Literally no. No. No. No. No. [6:33] Francois Chaubard: How many apps do you have on your phone? That's probably the answer. [6:33] Vincent Sunn Chen: for giving data its flowers. [17:21] Vincent Sunn Chen: YOLO access to your desktop. [27:46] Vincent Sunn Chen: putting our money where our mouth is [48:20] Shayne Longpre: team script [51:12] Shayne Longpre: You get bold numbers in a table state-of-the-art. Yay. It works.

[0:00] [music] [0:08] Welcome to YC data club this time. I [0:11] hope you guys like the little picture [0:13] that we have this time. Paul Graham [0:15] didn't like that I was slacking off in [0:16] my little crown over there, but he does [0:18] know know my name now, so that's good. [0:20] All right, so why data? Why does it why [0:23] does data warrant uh its own night to [0:25] talk about? Um, in 2016 I left my PhD [0:29] program and I started focal systems. And [0:32] back then the notion was that data is [0:34] commodity. I already have downloaded [0:36] imageet. How much more data do I [0:38] possibly need? And there was lots of [0:40] chatter about scale being worth a [0:42] billion dollars. There's no chance. And [0:45] there was a lot of VCs except who was [0:47] the guy that did scale series A? I think [0:49] it was Levy at Excel. Um, besides him, [0:52] everyone said the terminal value of data [0:54] businesses was zero dollars. And since [0:57] then, I think it's like a hundred [0:58] billion dollars in market cap creation. [1:00] So, rest assured, the entire VC uh [1:03] community has changed their tune on [1:05] that. And why? Well, the interview [1:08] question I would typically ask at Focal, [1:10] I still ran the deep learning team uh [1:13] even until I left for nine years later [1:15] was um you know, let's say I had to [1:17] interview these PhDs uh from Stanford or [1:20] MIT, super smart people, and I would say [1:22] okay, you have you trained your model, [1:24] you're at 85% F1 score on hot dog, not [1:28] hot dog. What do you do next? [1:31] And if your answer was, oh, I would go [1:33] read some papers. I would like try Reu [1:35] versus Gellu. I would like add more [1:37] layers. I do like nope. Next. Literally [1:40] no. No. No. No. No. And the reason why [1:42] is because you have to look at the data. [1:44] The correct answer is you look at the [1:46] data. You classifi look at all the false [1:48] positives. You look at all the false [1:49] negatives. You put them in purto [1:50] buckets. You say, "What's the biggest [1:52] issue?" And for us, for in stock, out of [1:54] stock, it's like, "Okay, it's when the [1:56] fridge has some fog on it. We can't see [1:58] through the fog or when there's a person [2:00] in the way or when something else is [2:02] happening." like the the number of [2:04] layers didn't was not going to change if [2:06] there's a person in the way and you [2:08] can't possibly tell if there's an in [2:09] stock or out of stock. And so it's just [2:11] looking at data and Andre I would say [2:13] was the most on this um and he famously [2:17] did the first measurement of what human [2:19] level is on ImageNet and where he [2:21] actually looked through and tried to see [2:22] how good he could be. And we're not that [2:25] good. I can't tell the difference [2:26] between Eskimo sheep dog and uh Siberian [2:29] husky. I'm not that I'm I'm not good at [2:31] that. And like 30 40% of imageet is dogs [2:34] are types of dogs. And like people kind [2:36] of don't know that stuff. And so the [2:37] data that you're training on ends up [2:39] being in much more important than the [2:41] architecture itself especially when the [2:43] architecture is very expressive. And so [2:46] in your PhD he says he spent maybe uh 5% [2:50] of his time thinking about data um and [2:52] 95% of his times focus on the [2:55] architecture. And then when you go into [2:57] production that flips and I would say he [3:00] was saying 25 74% I think nowadays it's [3:02] like 3% 97%. And it's really about the [3:05] data now that we have the transformer uh [3:07] and it works really well. And so and why [3:10] is this why does the data matter so [3:11] much? It's because in your nice little [3:13] imageet train set everything is really [3:15] nice. The train distribution is [3:17] beautiful. It's amazing. And then when [3:19] you launch it into production which [3:21] raise your hand if you've ever had this [3:23] experience. [3:26] that happens and then you start getting [3:28] a bunch of points in in in the in the [3:31] test distribution that you didn't have [3:32] in the train and the model you know [3:35] pukes itself and can't it doesn't work. [3:37] So to automate the economy if this is [3:39] the GDP val uh we need both expert data [3:44] and expert RL environments RL [3:47] environments for the verifiable uh [3:49] rewards and then for the pref style data [3:53] like very subjective do I do you like [3:55] this design do you not like this design [3:56] is this more usable or less usable you [3:59] you may the code may work but the coding [4:01] style is awful and it's unmaintainable [4:03] code for things like medical like two [4:05] doctors don't agree on the [4:07] answer for a given patient and and same [4:09] thing with judges. Um and so these are [4:11] preference style answers and that runs a [4:14] lot of the economy and the right way to [4:15] think about data sets and our [4:18] environments for sure is these are [4:20] products. If you talk to Greg at ArcGI, [4:24] the amount of craftsmanship that goes [4:27] into each game that Prime Intellect just [4:29] crushed um to be fair is incredible. [4:33] It's very very difficult to make these [4:35] data sets really really good fully [4:36] comprehensive and even for focal such a [4:39] simple thing of like in stock and out of [4:41] stock the amount of time that was spent [4:43] and how do we handle product pushers [4:45] like it looks like a product um but it's [4:47] just pushing product to to the front and [4:50] so how do should should we have that be [4:51] a separate uh uh class in the ontology [4:55] or should we classify that as an out of [4:57] stock because it technically is an out [4:58] of stock but it's going to confuse the [5:00] model because an out of stock and a [5:01] product look very [5:02] Um, so how do you handle all this stuff? [5:04] It's very very difficult. Um, and a lot [5:07] of people think that they're just zip [5:08] files and it's just really not. And the [5:10] other thing too that is I I deal with a [5:12] lot in the in the batch is let's say you [5:16] are automating Salesforce, right? And [5:18] you have a bunch of traces and you have [5:20] screen captures of people using [5:21] Salesforce. So you have 100 billion [5:24] hours. You have all the data that you [5:25] could possibly capture. Then Salesforce [5:28] changes the UI. [5:31] Then what? You better get some more [5:33] data, right? Because it's not going to [5:35] work. And so you can have all the data, [5:37] you still need more data. And so this [5:39] idea that it was going to the terminal [5:41] value is going to be zero is just so so [5:43] wrong. Brendan Emer just just presented [5:45] this slide at Sequoia and I really agree [5:47] with it. It is the bottleneck right now. [5:48] It is not the architecture. It is not [5:50] GPUs. It is not watts. Um it's largely [5:53] why does chat GBT still uh you know I [5:57] just did this analysis and posted on my [5:58] Twitter. Um, there's no model gets beats [6:02] random on predicting next seven-day [6:05] returns on the S&P. Literally, if you [6:07] just like roll a coin is better than [6:08] every single AI model that you can use [6:10] in production right now. Um, and so if [6:13] we had a bunch of traces from like [6:15] Goldman Sachs traders that are [6:17] definitely better than random, I would [6:19] think that they would do better, but [6:20] they don't have that. And so, how do you [6:22] get there? Same thing with accountants, [6:23] doctors, lawyers. you need experts on [6:26] each one of these uh domains for us to [6:29] uh match or exceed human uh capabilities [6:32] on it. And like I said, in the last 10 [6:35] years, despite most venture capitalists [6:37] turning a blind eye to this, um it's [6:39] produced over hundred billion dollars of [6:40] market cap in the last 10 years. It's [6:42] probably been one of YC's uh best [6:44] categories that we've invested in. And [6:47] so the mental model, what is the right [6:48] mental model to think about this? Um I [6:50] don't know, but I propose something [6:53] here. If I were Apple and I manufactured [6:56] these things and I wanted them to be [6:58] able to deliver groceries to your door [7:01] or to um bring a a burrito to your house [7:05] or bring a car and pick you up or turn [7:08] it on to a flashlight or give you [7:10] directions, you need these things called [7:12] apps. [7:13] And if you want the LLM to be a good [7:16] doctor, accountant, lawyer, carpenter, [7:19] um trader, uh or therapist, right? a lot [7:23] of people using for that as well. um [7:25] you're going to need data in RL [7:27] environments and if you don't have [7:29] upto-date very good curated data then [7:33] your LM can't do those things and so may [7:35] and so a lot of people say like how many [7:37] possible data companies can we have in [7:39] this space and the answer is how many [7:41] apps do you have in your phone that's [7:43] probably the answer I really do think [7:45] about it like that and then even better [7:47] on this analogy is does it make sense [7:50] for there to be an Instacart only for [7:53] Apple and an Instacart only for Android [7:55] or should probably just be one company [7:59] or should Apple get into the Instacart [8:01] business and start delivering groceries? [8:03] Doesn't make any sense. And so some of [8:04] these it makes sense like Apple should [8:06] and Android should have a flashlight [8:08] functionality because that's kind of [8:09] easy to do. But like for a lot of things [8:12] it's really really difficult to stand up [8:14] the entire Instacart network and then [8:17] start competing with Instacart if you're [8:19] Apple. Like why? Just that's not what [8:21] you are great at. And in the same way [8:24] going really deep into being a great [8:26] doctor like dealing with HIPPA and all [8:29] that stuff. Should Anthropic uh uh um [8:32] OpenAI, uh Google, Apple, all these [8:36] companies all get really great on [8:38] curating an AI doctor or should you just [8:41] like use one company that does a great [8:43] job of creating a environment, has a [8:45] network of doctors, curates all this [8:46] prep data, uh you know, parallel uh data [8:50] for given a specific patient. here's one [8:52] trajectory, here's another trajectory, [8:54] and and a doctor is preferring one or [8:56] the other. Probably just do that. Have [8:58] one company that just focuses just on [8:59] that. And that's what we're seeing. All [9:01] right. I'm very excited by uh the [9:04] speakers that we have uh tonight. Um May [9:07] couldn't make it. She had got sick [9:09] unfortunately. We're going to try to [9:11] squeeze her into the next one. Um but we [9:13] have uh Vincent Chen, who is a a Hazy [9:16] alumni, one one of the founding team [9:19] members of Snorkel. uh grew it to a [9:22] billion dollar market cap now and and [9:24] well beyond I think. Vololo, founder of [9:27] Inception Labs, Cornell professor and [9:29] expert surfer. [9:32] Um and then Shane uh MIT PhD uh student [9:37] just defended focused on pre-training uh [9:40] founded the data provenence initiative [9:42] and then recently joined anthropic. [9:44] Please help me give a round of applause [9:46] to this our speakers. [applause] [9:49] and Franuis. [9:53] >> Wonderful. Thank you, Francois, for the [9:55] awesome introduction and thank you for [9:57] giving data its flowers. I feel like [9:59] we've been trying to shout from the [10:00] roofs about this and uh it's awesome to [10:03] see it come front and center these days. [10:05] My name is Vincent. I'm one of the [10:07] founding team members at Snorkel. I uh [10:09] started our uh Frontier Lab business at [10:11] the company and now I lead research on [10:13] our benchmarks and evaluation at uh the [10:16] company. I'm going to talk about uh [10:18] scaling expert supervision. Um, and I'll [10:21] talk about it. I'm cheating a little [10:22] bit, but I'll talk about a few research [10:24] themes and how they've evolved over the [10:25] years. We've been working on data for [10:28] over a decade now at Snorkel, dating [10:30] back to our roots at the Stanford AI [10:32] lab. And um our key thesis is that [10:35] scaling expertise, actually giving [10:36] leverage to experts in the field, [10:39] doctors, clinicians, um journalists, [10:41] people who actually have the spec in [10:43] their head for what good looks like is [10:44] the real bottleneck for building really [10:46] effective data sets. And effectively, [10:48] that's what all of our research is about [10:50] at the company. So I'll talk to you a [10:51] little bit about that. As I mentioned, [10:52] we're a frontier lab focused on data in [10:55] particular. Um we're uh super excited to [10:58] be you know partnered with uh [11:00] effectively every global frontier lab [11:02] and uh Frontier um you know Fortune 10 [11:04] enterprises uh on the data problem in [11:06] particular and each one of these [11:08] engagements again is anchored on this [11:10] key bottleneck of scaling expertise. How [11:12] do we actually get the judgment and [11:13] knowledge out of these people's heads [11:15] and into the data sets that they're [11:17] working on? [11:18] And so I first want to start by framing [11:20] the problem and how it's evolved over [11:22] time. Expert supervision, right? The [11:25] process of taking what is in experts [11:27] heads, what is in messy documents or [11:29] knowledge corpuses um and putting it [11:31] into really effective data is a real [11:34] bottleneck. There's a real process that [11:36] goes from okay this is raw data to [11:38] something that can that can be used for [11:39] evaluation and training and this has [11:41] evolved over time right data 1.0 know as [11:44] I'll call it was uh basic labeling in Q [11:46] uh you know for question answering for [11:48] example right these might have been [11:49] thumbs up thumbs down against uh kind of [11:51] preference labels um basic you know [11:53] prompt and response pairs and you know [11:56] let's say it's 30 seconds of human [11:57] judgment right to produce one of these [11:58] labels as Frantois alluded to the shape [12:02] of data today is evolving significantly [12:05] right now we have entire worlds that [12:06] we're building against um that represent [12:08] tasks and rubrics and verifiers and [12:10] really nuance grading mechanisms all [12:13] packaged in Docker environments um that [12:15] actually need to represent the types of [12:17] spaces that these agents are going to [12:19] operate in. And these could take you [12:20] know uh single digit to you know triple [12:24] digits of hours for humans to develop on [12:26] their own. And so the key bottleneck [12:27] here is really how do we scale expertise [12:30] so that we can effectively build these [12:32] types of data sets when experts you know [12:35] are are one there are only so many of [12:37] these experts in the world with [12:38] increasing complexity and they only have [12:40] so much time in a day. And so let's [12:42] start with data 1.0. I know right uh [12:43] this is a problem that I believe still [12:45] exists today but when we started snorkel [12:47] um I want to you know share a little bit [12:49] about how we thought about this problem [12:50] right the the problem of manual labeling [12:52] right with the imageet example that [12:54] Francois gave um there are a bunch of [12:56] problems right uh one it's not a [12:58] scalable approach um think of the cost [13:00] of manual labeling as an O of problem [13:02] for every single data point you need a [13:04] manual label you need um new cognitive [13:06] effort to actually develop that two [13:09] these types of approaches aren't robust [13:11] to noise right you can Imagine that the [13:13] only way to actually uh reduce noise in [13:16] these settings is to um you know uh [13:18] introduce redundancy. So you need a k [13:20] byn um relabeling effort to actually uh [13:24] you know reduce noise in these settings [13:26] with changes in spec you know schema [13:28] task definition. You need to start from [13:30] zero and critically there's no [13:32] provenence. There's no notion of what [13:34] what is my rationale? How did I actually [13:35] think about this problem? And so this [13:38] was the key challenge that we faced you [13:40] know up to a decade ago and still face [13:42] to today in many ways where the problem [13:45] of manual labeling only um compounds in [13:48] expert domains. Think of a you know MD [13:50] PhD or a cardiologist in a very niche [13:53] subdomain. How do you actually get them [13:55] to produce these large large corpuses of [13:58] MRI data sets or you know EHR uh data? [14:01] Uh it's actually fundamentally [14:02] intractable in a lot of these settings. [14:04] And so that is the key problem to solve [14:06] in some of these um kind of manual uh uh [14:08] labeling settings. And so the the line [14:10] of work that we worked on here uh we [14:12] called kind of data programming. Our CEO [14:14] Alex was was a kind of grad student at [14:15] the time and we um all kind of [14:17] contributed to this work. The key [14:18] intuition here was we want to encode [14:21] expert supervision in software and [14:23] there's a few benefits when you do that [14:25] right you get the scale of software. You [14:27] now get to label data programmatically. [14:29] You get the adaptability of software. [14:30] you can refactor it and adjust it over [14:32] time as your spec changes and um [14:35] critically you can actually audit it and [14:36] have conversations about it and [14:38] collaborate on it. Um and so the you [14:40] know in a nutshell right a bunch of our [14:42] kind of key um grad grad school work in [14:45] a in a nutshell was was this uh one on [14:47] the left hand side um the main idea here [14:49] was um to first model expertise as uh [14:52] labeling functions right so this could [14:54] be a number of different forms this is a [14:56] pretty general uh abstraction but the [14:58] main idea is um if you express the [15:00] ration or reasons why people are [15:02] labeling spam versus not spam into [15:05] specific heruristics this um helps get [15:07] that knowledge out of people's heads and [15:09] into a form factor that's actually way [15:11] more reproducible and scalable. Number [15:13] two, um, in the middle, the challenge [15:15] with this is obviously that, uh, these [15:17] signals, uh, these sources of [15:18] supervision overlap. They're inaccurate. [15:21] Um, and so we introduce a bunch of [15:23] theory and work to actually, um, work [15:25] through this notion of weak supervision, [15:26] right? How do you actually understand [15:28] quality in these settings where uh, [15:29] signals are overlapping and your sources [15:31] are are weak or aren't, you know, fully [15:33] ground truth? And at the end of this um [15:35] taking these uh D noiseis signals and [15:37] using them in kind of end models that [15:39] are noise aware help generalize this [15:41] beyond um you know the coverage of the [15:43] initial labeling functions. So one click [15:45] down you know the intuition behind some [15:47] of this and the label model without [15:49] getting into too much of the of the gory [15:51] details. Um number one the goal here is [15:54] to model um each voters's accuracy in a [15:58] fully unsupervised way. So the analogy I [16:00] like to share is hey you have a number [16:02] of students in your class with different [16:04] levels of skill or quality. Um you don't [16:06] have a grading key, you don't have an [16:08] answer key. How do you estimate the true [16:11] ground truth answer when you only have [16:14] the votes from each of these students? [16:15] And so that's the fundamental kind of [16:17] modeling problem you're trying to [16:18] introduce here. Um you want to learn [16:22] this noise model and understand each [16:24] source's um quality effectively. you [16:27] know, given that you don't have any [16:28] ground truth in the first place. So, uh, [16:30] step one again is to kind of learn the [16:32] voters's accuracy. Step two, this is [16:34] this is Baze, right? You're trying to [16:36] kind of compute the the probability um [16:38] of of the ground truth solution given [16:40] each one of those voters um uh uh kind [16:43] of actual votes. Um, and then three, if [16:45] you train a kind of noiseaware model, [16:48] um, you're able to actually extend the [16:50] coverage of this label model and [16:52] minimize some of the noise in in uh, [16:54] broader distributions and and scale this [16:55] a lot more effectively. So, I'll zoom [16:58] out for a sec. Right? The main intuition [16:59] here was a lot of our work early days [17:01] was about scaling expertise via [17:03] introducing [17:04] labels and expert judgment as software. [17:07] And one of the key bottlenecks there was [17:08] actually denoising the sources of signal [17:11] that these came from in in in a a regime [17:13] where you were missing ground truth and [17:16] um you know some of these techniques uh [17:17] we still use today in a lot of our kind [17:19] of production settings. [17:21] So I want to talk briefly about data [17:23] 2.0. Now you know as the surface area [17:25] for data and frontier progress has [17:28] expanded um it's grown in uh complexity [17:31] and importance as well. And so let's [17:33] let's use coding as an example here. you [17:35] know, I have plotted on the x-axis uh [17:37] complexity, right? Input complexity, how [17:39] complex are your prompts and the specs [17:40] that you're giving models, output [17:42] complexity, environment complexity, are [17:44] you working in just Q&A settings now, or [17:46] are they fundamentally new, you know, uh [17:48] environments where you give it YOLO [17:50] access to your desktop? And on the [17:52] Y-axis, we have sequence length. How [17:53] long are these agents actually working [17:55] in practice? Now, you might think for a [17:57] domain like software or coding that uh [17:59] the eval are actually saturated, the [18:01] data is actually saturated. And I would [18:03] push to uh contend that as the frontier [18:05] advances, the data challenges actually [18:07] continue to grow, right? We're we've [18:09] seen basic eval like human eval saturate [18:12] at this point, which is which is true, [18:13] but we're still seeing, you know, um a [18:16] big spectrum of uh you know, terminal [18:18] based agents continue to grow. Uh [18:20] program bench, you know, I think the [18:21] latest model just cracked 1%. Um we [18:24] introduced the benchmark called senior [18:25] SWE, which I'll talk about in a second. [18:27] But as the complexity and responsibility [18:30] of agents, you know, grows, the [18:32] importance of data that you're uh [18:34] producing to both train and evaluate [18:35] these models also grows. With this in [18:37] mind, I want to anchor on some recent [18:39] work that we've done uh from a little [18:41] bit uh later than the last decade um [18:44] which was a senior SWEBench. And so the [18:46] the key here, we actually uh were very [18:48] humbled and privileged to work with the [18:49] original SWEBench team out of out of [18:51] Princeton on this. Um the key idea here [18:53] was we're already using coding agents [18:56] like software uh like senior engineers. [18:58] We're vibe coding. We're trusting them [18:59] to make architectural decisions to [19:01] refactor entire swaths of of code, but [19:04] we don't have good ways to evaluate [19:05] them. We're still evaluating them as as [19:07] junior engineers. And so the key [19:09] challenge we wanted to solve here was [19:11] that we wanted to build a benchmark. We [19:12] wanted to build data sets that were [19:13] really representative of actual work [19:15] that senior engineers would produce. We [19:17] wanted to scale the expertise needed to [19:19] produce and build these types of data [19:21] sets and uh build a really highquality [19:24] benchmark that uh represented all of [19:25] these factors. And so I'll talk you [19:27] through some of the methodology we used [19:29] to get here. Obviously we won't get [19:30] through everything but um you know [19:32] hopefully build some intuition for how [19:33] some of the methods we've used have have [19:35] evolved over time. And so a little bit [19:37] about the task themselves, right? So uh [19:39] in senior Sweetbench um the task design [19:42] specifically required many points of [19:44] expert supervision or expert touch [19:46] points. So on the left hand side uh this [19:49] is a harbor task. We're big fans of [19:50] harbor as an eval um uh framework. Um [19:54] these are instructions that are natural [19:56] language instructions. So the the key [19:58] diff here was instead of representing [20:00] you know if you're familiar with a [20:01] Swedebench or or kind of traditional [20:02] coding benchmarks if you actually look [20:04] at the data if you actually look at the [20:05] tasks a lot of the samples are very [20:08] oversp specified very kind of EDDD level [20:11] um you know guidance for what to do what [20:13] not to do we wanted to represent more [20:15] realistic higher level abstractions in [20:17] terms of how we actually interact with [20:19] these agents in practice so more of a [20:20] Slack message here's a here's a dump of [20:23] logs you know figure out what to do with [20:24] it or um here's a few users stories [20:27] about an idea that I have. Um can you [20:29] help me implement this? So actually [20:30] injecting expertise and realism into the [20:32] instructions was a super non-trivial [20:34] part of building this. Um on the right [20:36] hand side um you see a whole reward [20:38] phase which we uh spent a lot of time [20:40] designing and specifically um [20:42] calibrating as well. I'll go into a [20:44] little bit more detail but the intuition [20:45] here is we wanted to capture a number of [20:48] different components of how you actually [20:50] measure these agents in practice. So not [20:52] just um is this correct or mergeable but [20:54] does this exhibit taste? Does this [20:56] exhibit what you'd trust a staff or [20:58] senior or principal level engineer to [20:59] actually do in your codebase uh [21:01] completely unsupervised? Let's go a [21:03] click deeper into the specific uh reward [21:05] design. Um and this was a bit nuance and [21:08] again an area where we wanted to get [21:10] creative in scaling expertise. So we had [21:12] you know snorkel engineers and [21:14] researchers. We have a really strong [21:15] network of very senior principal staff [21:18] level engineers um but not unlimited [21:20] time. And so we wanted to find ways to [21:21] scale their intuition and efforts rather [21:23] than having them handgrade every single [21:25] one of these samples and agent [21:27] trajectories that came out of this. So [21:29] the axis we tried to articulate here um [21:32] there there's a trade-off between [21:34] reliability and flexib uh flexibility [21:36] when it comes to traditional verifiers [21:38] and rewards especially in the software [21:40] domain. On the top you have um high [21:43] reliability and low flexibility. These [21:44] are pre-written verifiers and think of [21:46] these as unit tests, right? These are [21:48] tests that reliably execute against your [21:50] codebase. They're um actually um running [21:52] against the code so you know that [21:54] they're doing the thing. Um but the [21:56] challenge is they don't actually adapt [21:57] to any solution, right? If you have um a [22:00] you know post/profile implementation [22:03] versus a um you know another way to [22:06] implement a way to kind of update a [22:08] user's information, you might actually [22:09] have a false negative in the verifier, [22:11] right? might actually penalize behavior [22:13] that is fully okay and fully valid um if [22:16] you're too constrained in your [22:18] verification approach. And so on the [22:20] other side of the spectrum, right, low [22:22] reliability, high flexibility, you have [22:24] LLM judges, right? This is a very common [22:26] and effective way in many cases to [22:28] actually adapt to different solution [22:30] shapes, right? The um code can be a [22:32] little bit more flexible. uh you're [22:33] actually just asking an agent or LLM to [22:35] look at the code and specifically um you [22:37] know grade it uh using its own kind of [22:40] uh calibrated or unccalibrated [22:41] intuition. Um and obviously the [22:43] challenge here is that this can be [22:44] really unreliable because you're um not [22:47] really exercising any of the solutions. [22:49] You're not exercising the code path [22:50] directly. You might be just, you know, [22:52] asking the agent to kind of take a [22:54] glance at the code and it's it's hard to [22:56] actually uh control for false positives [22:58] here, right? Where you may o overreward [23:00] plausible but incorrect solutions. And [23:02] so we found a middle ground for this [23:04] which I'll go into a little bit more [23:05] detail and introduced this notion of a [23:07] validation agent uh which specifically [23:10] tried to capture and scale the expertise [23:13] of a bunch of our senior researchers and [23:15] engineers and experts in our community [23:17] and and network um to one express the [23:21] user stories or kind of specification of [23:23] what they thought good looked like and [23:25] two use that specification to then write [23:27] deterministic tests against the code [23:29] surface that was actually relevant to [23:32] you know the solution space. So um a [23:34] simple idea but did take a lot of [23:36] calibration and effort to actually um [23:39] come up with this notion of hey how do [23:40] we actually capture the expertise of of [23:42] someone who's you know defining user [23:45] stories for a particular task while [23:47] actually exercising the real code path. [23:49] So again in a little bit more detail [23:50] right what does this actually look like [23:52] in practice the input to this validation [23:54] agent think hey I have codeex I have uh [23:56] clawed code it generates a patch against [23:59] my instruction set the expert produces [24:01] what we call the validation spec so this [24:03] was just a few bullet points think a [24:05] slack message of okay here are my user [24:07] stories here are a few you know uh [24:09] functional or non-functional [24:10] requirements not so much that it was [24:13] oversp specified and overly prescriptive [24:15] about a specific solution but uh high [24:17] level enough so that it to capture the [24:19] space of you know what we wanted the [24:21] model to do or not do. We then had a [24:23] validation agent take the patch and the [24:25] validation spec and specifically in uh [24:28] implement test scripts. So these were uh [24:30] deterministic um scripts that would uh [24:34] execute and and run over the code [24:35] itself. And at the end of this, we would [24:38] uh score against the both both the [24:39] execution and use a judge um to ensure [24:42] that the runs were were kind of sane or [24:44] you know uh resulted in kind of [24:46] collusion or reward hacking or some sort [24:48] of behavior that we thought was um not [24:50] valid for the trial. And so this this [24:52] idea of encoding expertise in a way that [24:56] um was still scalable was really the [24:59] crux of how we tried to design this [25:00] validation agent to map to dynamic [25:03] implementations right in the specific [25:05] codebase. Um and so we did a lot of work [25:07] actually to calibrate this. We aligned [25:09] these against you know specific test [25:11] engineers internally and in our network. [25:13] Um we introduced LLM judges that [25:15] measured things like uh fidelity, [25:17] completeness, collusion and ultimately [25:19] each test was parameterized so that um [25:22] you know in the validation spec you you [25:24] can think of it as like a ramble of kind [25:26] of um hey what is the shape of the type [25:28] of use cases edge cases I want to [25:29] capture without having to write you know [25:31] the full spec at the end or the the full [25:33] tests you know at the end of the day [25:34] which could take days to weeks to [25:35] months. [25:37] So this is um live now and one thing I [25:41] want to uh point out before you know [25:43] talking about the results which are kind [25:44] of fun. Um we introduced this notion of [25:45] a tasteful pass. Um this is a new metric [25:48] that we introduced in this benchmark to [25:49] specifically measure not just [25:51] correctness or mergeability but a more [25:53] nuanced definition of is this actually [25:56] code that aligns to your codebased [25:57] practices. Does this actually map to um [26:00] you know how a senior engineer would act [26:02] in practice? Does it minimize bloat in [26:04] terms of patch size? Um we merged that [26:06] with uh notions of correctness to [26:08] actually um rank a bunch of these models [26:11] and remarkably um and we double and [26:13] triple check this uh Fable, Opus and [26:15] Soul are all tied for first place as of [26:18] uh last week. This is pretty exciting [26:20] because it means that um you know the [26:22] Paro Frontier is definitely getting [26:24] pushed in these cases, but this is a [26:25] kind of live dashboard and a living [26:27] benchmark that we've been keeping up to [26:29] date. So definitely reach out and check [26:30] it out if you have um if you're curious [26:32] about it. This is some of our recent [26:33] work um on this topic. [26:36] And so I want to definitely give some [26:38] flowers to the team here. Henry, our [26:40] co-founder, uh was the was the lead [26:41] here. We were very humbled to work with [26:43] Cararthik's group at Princeton who led [26:44] the original SweetBench and uh our chief [26:46] scientist Fred uh who has the lab over [26:49] at uh uh Wisconsin Madison. Um and some [26:52] of his students were also involved. And [26:53] so I'll zoom out and and wrap up here. [26:55] Um as we enter data point 2.0, this is [26:58] maybe the the takeaway here. As we enter [26:59] data 2.0 know and see a lot more [27:01] complexity. We need a lot more data [27:03] research, right? It's not just a problem [27:04] of throwing humans at these tasks. These [27:07] days, we need a lot of thoughtful design [27:09] to give these experts, software [27:11] engineers, doctors, uh lawyers, people [27:13] who actually understand the spec [27:15] leverage to exercise their own [27:16] supervision and judgment in these data [27:18] sets. And um you know, this this slide [27:21] at least shows a view of all the axes [27:23] that complexity is going to continue to [27:25] grow in, right? The environments are [27:26] going to get more complex and dynamic. [27:27] the outputs are going to get more and [27:29] more unverifiable and nuanced and [27:31] subjective and the autonomy of these [27:33] agents are going to also continue to [27:35] extend and you could imagine every one [27:37] of these axes introduces compounding [27:39] complexity and so it's really a research [27:41] question. It's really a research problem [27:43] to think through how do we scale and um [27:46] provide experts more judgment and that's [27:47] really what we're focused on and [27:49] encourage the community to to work on as [27:50] well. And so here's a very concrete ask. [27:53] Uh we need more benchmarks and we think [27:54] this is actually a really high lever way [27:56] for folks to drive new data research. [27:58] We're super humbled to to work with a [28:00] number of the folks on the screen from [28:02] agents/exam um over at Berkeley [28:04] continual learning bench oorld uh the [28:06] terminal bench folks. Um we've learned a [28:09] lot working with these folks and um in [28:11] general we're we're uh very excited to [28:13] see a lot more benchmarks and so we like [28:15] to put our money where our mouth is. We [28:16] have this notion of a open benchmarks [28:18] grants. Um hopefully this is us actually [28:20] a research a resource to folks in the [28:23] community. We're very excited to you [28:24] know accelerate and and focus on more [28:26] research that is out there and help you [28:28] accelerate your data devel data [28:30] development and partner as a as a [28:32] research team. But um yeah, we're really [28:34] excited about the future of benchmarks [28:36] and data for data 2.0 and thanks so [28:38] much. [applause] [28:41] >> All right, next up we have Volo. [28:43] >> Uh okay, great. Thank you for having me. [28:45] Super excited to be here today to tell [28:47] you about some of our uh work some of [28:49] the work that we've been doing at [28:51] Inception on training uh real world [28:54] large scale diffusion language models uh [28:56] with uh my co-founders Stephano and [28:59] Aditia and a really really talented team [29:01] of engineers and researchers. So in [29:04] short uh what we're working on is a new [29:06] generation of language models that is [29:08] powered by diffusion. And what that [29:11] means is that instead of generating [29:13] tokens one at a time sequentially left [29:16] to right, a diffusion model starts by [29:19] generating tokens all at once uh [29:21] starting from an initial guess of the [29:23] sequence. So starting from some some [29:24] kind of noisy initial version of the [29:27] output and then producing all the tokens [29:29] in parallel over multiple steps of [29:31] refinement. We're very excited about [29:33] this technology. we think is going to be [29:35] the future of language models. And in [29:38] particular, a very important advantage [29:40] these models have today is speed. [29:43] Because these models can produce [29:44] multiple tokens per step, they can [29:46] produce many more tokens per second and [29:49] they can reach speeds at over a thousand [29:51] tokens per second uh which is way beyond [29:54] what is achievable with uh traditional [29:56] ultra reagressive modeling. There's all [29:59] kinds of applications for ultraast [30:01] inference in real time AI. One area that [30:05] we've been working in uh a lot is [30:07] real-time voice. Um so voice agents for [30:11] example uh in customer support uh but [30:13] also in other domains like education. Uh [30:15] so these are agents that uh synthesize [30:18] voice and talk to you in real time. [30:20] Usually uh the voice uh state-of-the-art [30:23] voice pipelines are still built on a [30:25] cascaded architecture where you have a [30:28] uh speech to text system, a texttospech [30:31] system at the output and at the middle [30:33] you have an LLM. The overall latency of [30:36] the system is highly bottlenecked by the [30:38] LLM. This is a critical critical piece [30:41] of this workflow. And if you can make [30:44] LMS go much faster, if they can go at [30:46] 1,000 tokens per second like our Mercury [30:48] 2 models are able to achieve, then you [30:51] can make real-time voice interactions [30:53] feel more seamless or you can deploy a [30:55] bigger model or you can get the model to [30:57] reason for longer which will then [30:59] significantly improve your quality. So [31:01] this is a slide that shows the [31:03] performance of Mercury 2 on voice [31:05] benchmarks compared to other models. And [31:07] what you see is Mercury 2 draws a a new [31:10] parto frontier of quality and uh and [31:14] latency. So latency on the x-axis, [31:17] quality on the y-axis. Uh so we have a a [31:22] new parameter frontier that's achievable [31:23] with a fusion. I also want to highlight [31:25] some other models here that are [31:27] baselines. So here for example we have a [31:29] uh 120 billion parameter GPOSS model [31:32] running on cerebras and a diffusion [31:35] mercury model can achieve both higher [31:37] quality at least as measured here by [31:38] tobench and I'm going to argue on real [31:40] data as well uh and it can also run [31:44] faster. So this is really exciting [31:46] because uh so again three are [31:48] specialized chips uh designed and built [31:51] to make other aggressive models run [31:52] really fast. If you have diffusion, you [31:55] can achieve similar speeds on GPUs by [31:57] having more intelligent software. So you [31:59] can build software to make models really [32:01] fast instead of hardware. That's really [32:03] exciting. You can deploy these models on [32:05] uh on on a lot more GPUs that are more [32:07] easily available. [32:09] Um but to build a really good voice [32:12] models, algorithms and diffusion is one [32:14] component, but the other part that's [32:16] really really really important is the [32:17] data. uh you need data both for training [32:20] the model and for eval [32:22] right now perhaps the most widely used [32:24] and representative benchmark for running [32:26] evals is tbench [32:29] uh tabbench has a few problems first of [32:31] all tbench doesn't really capture all [32:33] the range of complexity that you see in [32:36] real world data uh so if you look at a [32:38] tabbench environment it is it probably [32:40] has an order of magnitude fewer [32:42] specifications than you would have in a [32:44] in a real world uh setting where like a [32:47] business would be doing real world [32:48] customer support. There's fewer tools in [32:50] Towerbench. Uh it doesn't have certain [32:52] kinds of specifications that you would [32:54] want to provide. Uh and then another [32:56] problem with Towbench is that it's [32:58] extremely benchmaxed. If you go to your [33:00] favorite benchmarking website, you're [33:01] probably going to see most of the models [33:03] achieving uh scores that are in the [33:05] '90s. But then if you run these same [33:07] models on real world production logs, uh [33:10] you're going to get much lower scores, [33:11] and there's also going to be much more [33:13] variance across models of different [33:15] sizes. So data is a key key ingredient [33:17] of building the kinds of performant [33:19] models uh that that we want to build uh [33:21] in addition to the algorithms. [33:24] Now our approach for getting good data [33:27] for our models involves a system that we [33:29] call Dowo Forge which uh synthesizes a [33:34] wide range of realistic RL environments [33:37] similar to Dowbench based on real data [33:40] based on real user interactions or uh [33:43] either based on data that we're getting [33:45] from our data providers or we also have [33:47] partnerships with companies that are [33:48] using uh this model in production and so [33:51] we get a lot of real world data that [33:53] then allows us to synthesize uh talbench [33:56] like environments across different [33:57] domains which can then be used for evals [33:59] as well as for training the model. So [34:01] this slide here summarizes or gives an [34:04] overview of tao forge. Uh taforge is an [34:08] agentic system. So it's a harness that [34:10] orchestrates uh a large number of [34:12] different agents and it produces this [34:15] output synthesized so synthetic RL [34:18] environments that are similar to [34:20] Towbench. The input to Dow Forge is a [34:23] description of the kind of environment [34:25] that you would like to generate. So here [34:27] for example, you could ask it to [34:28] generate a banking environment as well [34:30] as artifacts that represent real world [34:33] usage of users in that real world usage [34:37] of the model in uh that target domain. [34:40] The first step is to synthesize uh an [34:44] initial environment that is based on [34:46] that data. that involves for example a [34:50] policy that specifies the kinds of uh uh [34:54] task or the kind of domain where you [34:56] want the model to operate. So for [34:57] example, if this is meant to simulate [34:59] customer support in banking, then you [35:00] could generate a policy that describes [35:02] how the bank would operate, what are the [35:04] hours, who are the users, different [35:05] accounts, maybe synthesize a small [35:07] database. Uh and also uh it creates [35:10] tools that the model can use to then [35:12] interact with this environment. Uh again [35:14] crucially this can be uh conditioned on [35:18] real usage of the model which ensures [35:21] that this environment is representative [35:22] of how the model is going to be used [35:23] later in in production. If this data is [35:26] not available, we've also created what [35:28] we call the business knowledge graph [35:29] which is data that has been crawled all [35:31] across the internet and that describes [35:33] various representative businesses. uh so [35:36] we can also synthesize uh uh domains for [35:39] which we don't have real world data but [35:40] we have crawled the internet to gain [35:42] real world knowledge about these [35:44] businesses. [35:45] Now given an environment then the most [35:48] important next thing that we want to [35:50] generate are tasks which means these are [35:52] tasks that uh uh so so we're going to [35:55] have uh simulated users that will try to [35:57] achieve certain goals in this [35:58] environment and the agent will assist [36:00] them in these goals and we need to [36:02] synthesize what those goals are. In [36:05] order to do this we start with what is [36:07] called a database of seeds. Uh so these [36:09] are abstract scenarios. For example I [36:11] want to open an account, close an [36:12] account, I want to change my phone [36:13] number. uh I would like to I don't know [36:16] get some information uh as well as [36:18] personas which are um certain types of [36:21] humans that might be interacting with [36:22] the model given that we have an [36:24] environment we can take our library of [36:27] scenarios and other seeds and specialize [36:30] it to this domain again by invoking an [36:32] agent uh and then once we have a good [36:35] database [clears throat] we can start to [36:37] generate tasks. So again these are real [36:39] world interactions that a user would [36:40] have with these models and then we will [36:42] simulate those users using other agents. [36:44] Now getting a good set of tasks is also [36:47] non-trivial. Uh we need to strike a [36:49] careful balance of tasks that are uh not [36:53] too easy to solve. So if a if a model [36:55] consistently solves this task, it's not [36:57] useful for learning. And conversely, if [36:59] the task is too hard, there's also not [37:00] enough learning signal. So there is an [37:02] iterative procedure where we filter and [37:06] reconstruct tasks based on the ability [37:08] of the model to solve them. Uh and to do [37:11] this we use these go these what we call [37:13] uh hardening traps which means that uh [37:15] we can uh modify the task to make it [37:17] more difficult. So for example you can [37:19] take away certain information or you can [37:20] make the ask more complex. Uh so you can [37:23] do all kinds of uh tricks to harden the [37:25] task or you can also filter out [37:27] difficult tasks uh until we get [37:29] something that we feel strikes a good [37:31] balance between being realistic and [37:33] providing learning signal to the model. [37:35] So again the output here are RL tasks [37:38] and then uh of course this is [37:40] continuously inspected by looking at [37:42] real world data. So uh models are well [37:45] first of all these environments can [37:46] still be verified the harness can be [37:48] updated uh and also models that are [37:50] coming out of this environment we get [37:53] real world feedback from users and we [37:55] can use that to further refine the [37:56] harness and further refine this uh [37:59] synthetic generation procedure. [38:02] So as an example of what this model can [38:04] achieve uh here this is an experiment [38:06] where we used our latest preview model [38:09] mercury 2.5 um uh so before it was uh [38:13] trained on any sort of domain specific [38:15] data you know it achieved about 50% [38:17] accuracy and training it on an initial [38:20] set of tasks synthesized from tforge [38:22] improved performance on another set of [38:25] environments by over 23% matching the [38:28] performance of uh all kinds of [38:30] state-of-the-art openour or is in closed [38:32] sourced uh model and here just to [38:34] clarify the setup is that uh there are [38:37] there's a diverse set of environments [38:39] and businesses so for example I don't [38:41] know banking or you know airplane [38:44] um booking airplane tickets and uh you [38:47] know getting doctor's appointment so you [38:49] can have a whole range of businesses and [38:50] you can split that into a test set and a [38:53] training set you hill climb on one set [38:55] of businesses and you test on a [38:57] different set of businesses uh which [38:59] ensures that uh which which helps reduce [39:01] overfitting. Um and and again these [39:04] these these these tasks are still [39:07] especially the test set is closely [39:08] derived from real world logs which gives [39:10] us confidence that this will correlate [39:12] with real world performance once we [39:13] deploy into into production. Um, so this [39:16] is just a a bit of uh a bit of [39:19] information on the kinds of techniques [39:21] that we can use to make our model really [39:23] high quality, not just fast, but also [39:25] improve their uh intelligence in some [39:27] very specific priority domains like uh [39:30] voice. Uh and if you'd like to try these [39:32] models, they're available. Uh so they [39:34] run at 1,000 tokens per second. This is [39:36] the pricing. And even though I mainly [39:38] talked about voice in this talk, uh we [39:41] also have uh customers across other [39:44] domains including search as well as [39:46] code. Uh and so these are all domains [39:49] where uh mercury 2 models are really [39:52] good for latency sensitive tasks. Uh and [39:55] there's if you're interested in [39:56] feedback, we have companies and other [39:58] companies that are using these. Uh so [40:00] happy to tell you more after the after [40:02] the talk. And maybe I'll just briefly [40:03] add by saying that we have a program for [40:05] startups. If you are in YC, we have a we [40:09] have $500,000 in credits that we're very [40:11] happy to give away. Uh if you're [40:13] interested, please uh grab me and uh I [40:16] would love to tell you more. Thank you. [40:18] [applause] [40:21] All right, next up we have Shane. [40:25] Awesome. Thank you. Thank you so much [40:27] for your attention. I think I'm the last [40:28] talk and uh I'm Shane. I just graduated [40:32] from my PhD at MIT and I think this is [40:35] my favorite project uh during my 5-year [40:38] PhD. It was work that I did while [40:40] interning at Google. Thank you for the [40:42] TPUs to make this happen. Um we [40:46] basically looked at multilingual [40:48] pre-training and within that transfer [40:51] between the languages. So the synergies, [40:52] interference and interactions between [40:54] the different training sets uh data set [40:57] sources that are different languages. [41:00] While I walk you through this project [41:01] for the next 10 minutes, you can imagine [41:03] all of these experiments and all of [41:05] these results and methods could be [41:07] generalized to many different sources [41:10] within your data. They don't have to be [41:11] languages. They could be um different [41:14] domains or different quality sources of [41:17] data, different things um however you [41:19] want to splice it up. And I think that's [41:21] important because increasingly when we [41:24] think about uh our model's capabilities, [41:26] its risks, its limitations, we think [41:29] about how all of those different pieces [41:31] interacted and came together. So diving [41:33] into this, I'm going to talk a little [41:35] bit about why this is important, [41:37] motivating it a little bit beyond [41:38] English. Then I'm going to go through [41:40] really quickly, rapidfire, some of the [41:42] research questions we asked and the [41:44] answers we arrived at before bringing it [41:46] together. So this was presented at iClar [41:49] earlier this year with these phenomenal [41:51] folks at Google, Udub, Stanford and some [41:54] other places. If you look at scaling law [41:57] and even mixing law papers in literature [41:59] right now, uh they are 99% about English [42:02] training data and English evaluation. [42:04] There's very little about sort of the [42:07] rest of the world and the community. Uh [42:09] which is actually kind of shocking. And [42:10] so this is thinking what if the [42:12] objective was a single language that [42:14] wasn't English or multiple languages a [42:17] cluster of them and exploring that [42:19] domain. And why is that different than [42:22] in English? The reason is because [42:24] there's severe constraints when you go [42:26] past the first few languages that are [42:28] very abundant online. And we also uh [42:31] provide toolings and methods both for [42:33] scaling laws, mixing laws, fine-tuning [42:35] versus pre-training and other decisions [42:37] that developers have to make. And this [42:39] has implications for uh developing [42:41] models for your own culture, society, [42:43] language and also trying to figure out [42:46] if you are Turkish um what are the [42:49] constraints in pre-training for uh [42:52] Turkish language models, how much will [42:54] they lag behind uh language models for [42:56] English based off of just the [42:57] availability of data that's there and [42:59] what synergies and interference we can [43:01] expect in pre-training. So let me start [43:05] with the problem setup. Very simply, [43:06] let's pick a language. We're going to [43:08] pick Thai because there's something like [43:09] 80 million speakers in the world. It's a [43:12] very unique language, but we could be [43:13] talking about Swahili, Polish, it [43:15] doesn't matter. The point is that in [43:17] Madlad 400, a popular common crawl based [43:21] pre-training corpora, there are 0.6% [43:25] the number of tokens in Thai as compared [43:28] to English. So less than a percent of [43:31] the amount of English data that you [43:32] have. You know, some frontier model, [43:34] CHBT, whatever it is, we don't know. but [43:37] it has some training mixture among the [43:38] natural languages and it's probably [43:41] going to roughly follow like every other [43:42] language mall the distribution you know [43:44] they'll do different sampling techniques [43:46] and things like that but you're going to [43:47] roughly have more English than anything [43:49] else followed by the next languages you [43:51] care about that are more abundant more [43:53] uh heterogen heterogeneous [43:55] um all the way down that list and by the [43:58] time you get to Thai it's a tiny sliver [44:01] of the overall mixture much less than 6% [44:05] and so the performance performance on [44:07] Thai is going to be pretty poor. Um, it [44:10] might be pretty pretty good overall [44:11] because it's a massive model, but it's [44:13] not going to be great for Thai speakers. [44:15] And this is shown uh in the literature [44:18] and in evaluations. [44:20] So, you might say, okay, let's train a [44:22] model just for Thai. We're going to [44:24] monolingually only train on all of our [44:26] TAI data. But the problem is there's so [44:28] little of it that we end up repeating it [44:30] again and again and again. And those are [44:32] those dash lines, epochs. And you've [44:35] overfit to your tie data. You need a [44:37] much smaller model. And so actually, you [44:40] don't do very well there either. [44:42] Ideally, what you want to do if we skip [44:44] ahead to the solution is something like [44:46] this. It's actually a mix where you use [44:49] a lot of your tie data, but by the time [44:51] you're hitting severe diminishing [44:52] returns, you're also incorporating, this [44:54] isn't a curriculum. This is all, you [44:56] know, mixed together. You're using [44:57] highquality synergistic data that [45:00] happens to be English, Indonesian, [45:02] Malay, Lao, uh, Croatian, it turns out. [45:06] But you're not going to know that [45:07] because this isn't just about what [45:10] language families are similar. It's [45:12] about what data empirically is not noisy [45:15] and that other source is actually [45:16] helpful, covers similar topics. Um, and [45:19] that is very distributed and uncertain [45:22] on the web. It doesn't just mean that [45:24] they are from the same language family. [45:26] So it's something you have to [45:28] empirically measure. You can't just [45:30] bring a linguist and they can theorize [45:32] about linguistics for this particular [45:34] case. So [45:36] how do we get to the fantastic Thai [45:38] model or sahili model or Turkish model [45:40] and figure out the right mix and the [45:42] right model size? [45:45] First we want to measure language [45:46] synergies and this is where a lot of the [45:48] contributions of the paper comes in. [45:50] There's many ways you can think about [45:51] doing this. And the way that we landed [45:53] on was imagine this. You have a training [45:56] curve. So over your training steps, the [46:00] red line is where you're training only [46:02] on Thai data. So it's monolingual. So it [46:04] does pretty well, but it flattens very [46:06] early. Now imagine that compared to uh [46:09] models that are trained 5050 on Thai and [46:12] English, Thai and um Telugu, Thai and [46:16] other languages. And you can see that [46:18] those learning curves are much higher, [46:21] meaning that they're not as good. And so [46:24] actually the distance between these [46:26] curves tells you something about how [46:29] good that language is in being helpful [46:31] for Thai. And so we can say, okay, well, [46:34] Indonesian is better than English, which [46:36] is better than Telugu because that line [46:37] is longer, the distance. Um, and this [46:40] efficiency gap can be formalized into [46:43] something that looks like this. And let [46:46] me abstract all this away. Uh you don't [46:48] have to go into too much detail. If [46:50] these two lines are really close, the [46:52] number is big and positive. If the lines [46:54] are very far away, it's going to be a [46:56] negative transfer, meaning there's [46:58] interference. These languages are [46:59] competing for tokens or neurons in order [47:02] to represent one another and perform [47:04] well on them. [47:06] And if you do this over and over again [47:08] with many tricks that I'm not going to [47:10] get into, you get this big cross-lingual [47:12] transfer matrix where red means high [47:14] synergy and blue means high [47:17] interference. And so if you look at, you [47:19] can't see it, but imagine you could zoom [47:21] in and look at Spanish, the row. The [47:24] bright red uh spots empirically happen [47:27] to be Portuguese, Italian, um, and [47:31] French. And the most negative harmful [47:35] language um is Japanese. Meaning if you [47:37] train with Japanese, it's going to hurt [47:39] your Spanish. And so this is all [47:42] empirically sort of born and there's, [47:44] you know, some surprises in here, some [47:45] interesting things. Some languages are [47:47] more helpful broadly than others. Um and [47:50] it's complicated. [47:52] The other really cool thing about this [47:54] matrix is it's at one fixed size of [47:56] language model. If we increase the size [47:58] of the language model and make it much [48:00] bigger, then actually a lot of these [48:02] blues start to turn red or deeper red, [48:05] there's more synergy because the model's [48:07] bigger and can accommodate uh [48:09] understanding multiple languages at once [48:11] or together and seeing their common [48:13] patterns. But if the model gets really [48:15] small, all of these turn to blue and [48:17] they all interfere competing for space [48:19] within the model. All right, quick [48:21] question for the audience. who thinks [48:23] that if um Portuguese is helpful for [48:27] Spanish, that means that Spanish is [48:30] helpful for Portuguese and that symmetry [48:33] exists uh from what we measured. Raise [48:36] your hand if you think that is true. [48:39] We have a few people. If you think it's [48:42] not true, raise your hand. Okay, it's [48:45] kind of 50-50. Well, actually, it's kind [48:47] of hidden on the screen. I know it's [48:48] late. Uh it's not symmetric as you can [48:51] see. Um and actually there's this weird [48:55] sort of like uh branching factor that [48:58] happens and this is important because if [49:00] you're a practitioner and you do [49:01] experiments you say okay Telugu is [49:04] really helpful for Swahili you can't [49:06] assume the opposite you have to go [49:08] measure it and so that's where this [49:10] breaks down and we actually show that [49:12] okay second question you can redeem [49:14] yourselves um what is more helpful do [49:17] you think if two languages share the [49:19] same script or if they share the same [49:22] language, family. Who wants to vote for [49:24] family? [49:26] We have some. Who wants to vote for [49:28] script? All right, congratulations team [49:30] script. They both matter, but script [49:32] matters a little bit more probably [49:34] because of tokenization artifacts and [49:37] how those are represented. If you're [49:39] taking pictures, don't worry. It's all [49:40] online. I will point you to it. Um, I'm [49:43] going to go through in the last three [49:45] minutes really quickly a little bit of [49:46] math. We have this great matrix. So we [49:48] have all these numbers and relationships [49:50] between languages. But the last step is [49:53] how do we actually create a scaling law [49:54] from this? How do we actually understand [49:56] how to predict estimate loss or figure [49:59] out the best mixture of languages for a [50:01] given target which is our original goal. [50:03] You have to fit scaling laws. Those of [50:05] you that are familiar with Chinchilla [50:07] probably remember this. You predict the [50:09] loss based off of the model size and the [50:12] amount of data you've trained on and you [50:14] fit it using these blue parameters here. [50:18] you fit those. Um, however, if you have [50:21] a multilingual data source and you're [50:23] trying to figure out how good it is on [50:24] French, the thing that matters most is [50:26] how much French data you had and if you [50:29] have tons of other data, you know, how [50:30] is that factored in? 1D is not helpful. [50:32] You need to model all of it. And so, um, [50:35] that's what we do. There's many ways to [50:37] model this. We explored many options but [50:40] simple solution that actually works [50:43] really well is to break down [50:45] chinchilla's d into a composite of [50:47] monolingual transfer languages [50:49] monolingual source in this case Thai the [50:52] close transfer languages in the matrix [50:54] plus a bucket for everything else that [50:57] looks and yeah we account for [50:58] diminishing returns when you run do [51:00] multiple epochs of certain data there's [51:02] a lot of details long story short it [51:04] looks something like this and uh the tow [51:07] how is sort of a waiting factor that you [51:09] can learn from the matrix. I'm going to [51:12] rush along in the interest of time. But [51:13] you get bold numbers in a table [51:15] state-of-the-art. Yay. It works. It [51:17] works really well actually. Maybe [51:19] because a lot of people can't run 700 [51:21] pre-training experiments. Maybe because [51:23] this is unders underserved and people [51:24] aren't looking at this enough. But [51:26] either way, I think this is really [51:28] useful machinery to use for mo for uh [51:31] mixing multiple sources and [51:34] understanding how they interact to [51:35] predict the final loss for your model [51:37] and how to scale your model given what [51:39] you have. So it gets us back to this. [51:42] I'm going to wrap up by saying we did [51:44] some experiments to say you have a big [51:46] pre-training models available that are [51:47] multi language. You just fine-tune or [51:49] maybe you should pre-train from scratch [51:51] for the language that you care about. We [51:53] tell you which to do based on how much [51:55] compute you have. We also look at the [51:57] curse of multilinguality. So if you um [52:01] want to increase the size of a language [52:03] model from four languages and you want [52:05] to retrain it for eight languages, how [52:07] much more data do you need and how much [52:09] bigger does the model need to be in [52:11] order to maintain the same performance [52:15] as before? We do a bunch of math. It's [52:17] very cool and we show you exactly what [52:20] to do as a practitioner and it actually [52:22] holds really well over different target [52:24] languages. So I'd recommend consulting [52:27] this if you're interested. Thank you so [52:29] much. Appreciate your attention. [52:31] [applause]