Brilliant Develops AI Tutor to Force Students Through Difficult Math

Brilliant开发AI导师促使学生攻克高难度数学 由于AI文本生成助长学生作弊并削弱认知耐力，教育平台必须拒绝自动化答案，转而采用智能体交互界面，让学习者在核心问题解决中经历必要的磨砺。 Brilliant联合创始人兼首席执行官Sue Khim与Peter Yang探讨了美国教育系统为何在严重的成绩通胀掩盖下陷入“学习低谷”，以及如何利用AI培养认知毅力而非助长作业作弊。Sue展示了Brilliant的语音AI导师Koji，该导师特意不直接提供答案和现成解释，促使学生必须经历富有成效的探索与挣扎，从而自行领悟核心见解。在技术实现方面，她分享了人类设计者如何构建教学叙事，而AI智能体则利用由交互式基元构成的确定性函数库来组装课程，并通过自动化UI评估和数千次对抗性合成学生模拟进行验证。最后，他们探讨了前沿实验室为何难以仅用标准强化学习复刻人类辅导，并分享了关于培养专注力以及热爱应对严峻挑战的个人育儿理念。 Brilliant的AI导师Koji特意不提供直接答案和现成解释，而是通过启发式语音提问，让学生保持在获得真正理解所需的富有成效的探索与挣扎中。 (10:49) 由于提示前沿大语言模型端到端生成课程会导致教学内容缺乏连贯性，Brilliant依靠人类教育工作者设计教学叙事，同时将AI智能体限制在跨越数千个交互基元的确定性UI API范围内。 (18:05) 每门新课程在逐步推出前，都要经过自动化UI与物理机制评估、人工编辑审核，以及大约1000次对抗性合成学生的压力测试会话。 (29:30) Brilliant通过在自动化强化学习失效的不可验证领域中运营，建立起针对前沿AI实验室的防御壁垒，利用数百万次实时辅导会话将专有的学生学习信号直接训练到模型权重中。 (34:39)

Brilliant CEO Sue Khim argues that instead of functioning as an answer engine that erodes competence, artificial intelligence should be designed to preserve productive cognitive struggle and teach students how to think independently. Speaking with host Peter Yang, she outlines how declining U.S. academic standards, flawed homework incentives, and specialized AI architectures shape the future of interactive learning. The U.S. Learning Trough and Broken Incentives (1:34) While the United States allows exceptional freedom for educational experimentation, Khim describes a broader system caught in an unprecedented "learning trough." Standardized test scores in basic literacy and numeracy are falling even as student grades climb. Khim points out that historically rigorous university STEM programs now must offer remedial algebra covering middle-school concepts, and the AP English exam has drifted over recent years from assessing college-level reading comprehension down to middle-school levels. Rather than addressing root causes, schools appease parents through grade inflation, creating pressure for neighboring schools to award easy A's so their students' GPAs do not suffer. This decline is compounded by how U.S. schools incentivize homework. Unlike countries that grade strictly on exams—making daily assignments practice for the test—American schools count homework toward final grades. Khim argues that when smart kids are told the goal of an assignment is a grade, using AI to cheat is a rational shortcut. Fixing the problem requires changing what schools measure rather than merely panicking over AI tools. Why Explanations Are Close to Cheating (10:49) Khim compares using AI to skip the work of learning to "bringing a robotic arm to the gym to lift weights for you." To counter this, Brilliant's AI tutor, Koji, avoids not only direct answers but also ready-made explanations. Khim contends that explanations feel satisfying in the moment but let the learner off the hook without doing the mental work. In a live session demo where a student incorrectly shaded five unequal parts to represent 5/12, Koji used voice prompts and peripheral questions rather than explaining that the shapes had different sizes, forcing the student to discover the core insight independently. Keeping students in this struggle without causing them to quit requires balancing non-AI mechanics—such as interactive object manipulation and Super Mario-like difficulty ramps—with AI judgment. Koji constantly evaluates whether a student made a minor slip-up or lacks a foundational concept, and whether to offer a hint or let them think longer. Khim admits that withholding easy answers hurts short-term engagement and problem-completion metrics, which she views as a "local maximum," whereas Brilliant optimizes for formative and summative mastery over time—testing whether a student can execute a skill today, a week later, and a month later. Human Pedagogy and Deterministic Agent Interfaces (18:05) Under the hood, Brilliant avoids relying on large language models for end-to-end lesson generation. When the team previously prompted models like Codex and Claude to generate coordinate-plane lessons from scratch, the output looked superficially right but produced an incoherent sequence that frustrated learners. Referencing Balaji Srinivasan's view of AI as an "80% middle solution," Khim enforces a strict division: human educators design the pedagogical narrative and sequence, AI implements the code, and human reviewers verify the result. To prevent hallucinations and latency spikes, Brilliant builds dual interfaces—a visual canvas for the student and a deterministic API layer of composable "primitives" for the agent. Instead of inventing visual apps on the fly, the LLM calls structured tools to instantiate graphs, highlight grid areas, or ask sub-questions; covering Algebra 1 alone required thousands of modules. Before release, lessons undergo automated primitive evaluations (checking math logic, physics rules, visual overlaps, and mobile tap targets), human editorial review, and roughly 1,000 simulated sessions with adversarial synthetic students—which Khim estimates are about 20% as good as real human data—followed by staged rollouts. Defensibility Against Frontier AI Labs (34:39) To survive competition with frontier model providers, Khim follows a clear rule: if big AI labs can apply reinforcement learning (RL) to a domain, they will dominate it. However, she argues that human tutoring cannot be easily verified by an automated pass/fail reward function like a code compiler, because true success depends on whether a human sustained attention and deeply learned a concept. Although millions of students prompt ChatGPT for homework help, Khim notes that these transactional exchanges end as soon as the "answer machine" outputs a solution, yielding no signal on whether the student understood the material. By contrast, Brilliant captures proprietary telemetry on real-world misconceptions across millions of interactive tutoring sessions and feeds that verified learning data back into its model weights to build a vertically integrated data flywheel. Ultimately, Khim and Yang conclude that cultivating independent thinkers requires setting high standards rather than delaying rigorous subjects like eighth-grade algebra. At home and in software, the goal is not chasing frictionless entertainment or effortless straight A's, but helping children build the concentration and resilience to tackle hard problems on their own.

[0:00] Sue Khim: We believe technology can and should be making our kids stronger, not dumber and weaker. [9:03] Sue Khim: there's no fear of embarrassment or failure. [9:03] Sue Khim: using AI as a way to skip the part where you learn is like bringing a robotic arm to the gym to lift weights for you. Like we're really losing the plot of why you went to the gym in the first place. [10:49] Sue Khim: we try not to offer explanations either. Like explanations feel good, but you're not doing the work yourself. [10:49] Sue Khim: that explanation only makes sense if you kind of already understand it. [27:27] Peter Yang: building the machine that builds the thing. [34:39] Sue Khim: Our rule of thumb is if the big model companies can RL your domain, they will eat you alive, but there’s lots of alpha finding domains that no one else can RL on. [36:16] Sue Khim: You got to build the product, you got to get the data, and then that data makes your product better, [39:10] Sue Khim: School is your job, not my job... if you can't do your homework, I guess you can't do your homework. [39:10] Sue Khim: you've like lost the plot, right? And it's really, really hard to undo. [44:35] Sue Khim: Set high standards. Set high standards. And like you'll be very surprised by what kids can do.

[0:00] AI today is mostly [music] used by [0:02] students for cheating. We believe [0:04] technology can and should be making [0:06] [music] our kids stronger, not dumber [0:08] and weaker. [0:09] >> Feel like AI is making me dumber, too, [0:10] cuz it's just easier to just ask it to [0:12] do stuff than to actually learn. [0:13] >> This is an actual playback of a real [0:16] tutoring session on Brilliant. What's [0:18] much more surprising is that we try not [0:19] to offer explanations either. [music] [0:21] Explanations feel good, but you're not [0:24] doing the work yourself. If we truly [0:26] have super intelligence, we should get [0:27] learners to stick with something hard [0:29] and actually [music] use it themselves [0:30] and get the students to do better [0:32] thinking. [0:35] >> Hey everyone. Today I'm really excited [0:37] to welcome Sue, the co-founder and CEO [0:39] of Brilliant. Sue has been thinking [0:40] about AI and education for over a [0:43] decade. And just as a parent, I'm really [0:45] excited to talk to her about our [0:46] education system and the best way to use [0:48] AI to help kids learn without just [0:49] giving them all the answers. So, welcome [0:51] Sue. [0:51] >> Thanks, Peter. It's great to be here. [0:53] >> Yeah. Um, I I really love how Brilliant [0:56] is affordable and very approachable. Why [0:58] don't we start with this? Like I'm [0:59] curious what your thoughts are on kind [1:01] of just like the overall US education [1:03] system. Like where do you think is doing [1:05] well and where do you think it needs [1:06] improvement? [1:07] >> Yeah, that's uh the US education system [1:10] is really interesting. High highs uh low [1:13] lows. I think one thing that's uniquely [1:16] good about the US system relative to a [1:19] lot of other places is that there's more [1:21] freedom for experimentation. So you know [1:24] you see new home schools and [1:25] experimental schools everywhere with [1:27] very talented people working on them. [1:30] And so we do have the conditions for new [1:33] solutions. The uh terrible thing that's [1:36] happening in the US is this phenomenon [1:39] of test scores are down but grades are [1:42] up somehow. Kids can't read or do math [1:44] anymore. And we hear all the time from [1:47] colleges that used to have strong STEM [1:49] degree programs and they're now looking [1:51] at remedial algebra solutions for [1:53] college students. I mean, we're talking [1:54] about middle school level math. And you [1:58] know, some of the reporting on this is [1:59] that part of how we've been dealing with [2:02] worse performance is by lowering [2:04] standards, meaning we've introduced [2:06] grade inflation. And so we are changing [2:08] the standard so that more students can [2:10] get an A instead of fixing the [2:13] underlying problem. And obviously if one [2:15] school does it then of course other [2:16] schools are also going to feel pressured [2:18] to do it too cuz you don't want your [2:19] school to be the only one where a lot of [2:21] students are getting seized. And so this [2:23] is sort of happening behind the scenes. [2:24] I think it's pretty opaque to parents [2:27] and doing this grade inflation works for [2:29] a while to keep the parents happy but [2:31] obviously it is the worst way to deal [2:33] with actual falling performance. So we [2:36] are in this real you know learning [2:38] trough and I think it's going to take [2:40] time to get ourselves out of it. But [2:42] conversely, I think that the super [2:44] intelligence that we're building at [2:45] these AI labs also is a real [2:47] opportunity. Like we think that if it's [2:49] used in the right way, AI can make our [2:51] kids into geniuses. It's like a sea [2:53] change like how the invention of writing [2:55] suddenly allowed everyone to have [2:57] infinite memory. [2:58] >> Mhm. Do you think um I feel like that [3:01] for schools it comes down to incentives [3:03] and and maybe their funding is based on [3:04] like is is it based on how many students [3:06] can get high grades or something or [3:08] >> uh I certainly think that you know [3:09] teachers are measured uh to a degree on [3:15] how uh I mean it's obviously like very [3:18] different from different schools like [3:19] charter schools are different public [3:20] schools are different yeah [3:21] >> you know private schools are a different [3:23] beast home schools have their own [3:24] standards you know I I think that what [3:26] they are first order responding to is [3:29] parents being upset and you know because [3:32] the standardized tests are also getting [3:33] easier. One very easy way to appease the [3:37] parents is by also you know lowering the [3:40] standard at the school to match because [3:42] they can do that. And so we have this [3:44] phenomenon where on the standardized [3:46] tests you have lower performance just on [3:49] an absolute basis. And that's masked by [3:51] the fact that you know at the school [3:53] we've also lowered the standards. And [3:55] really the only place that this is [3:56] coming up is internationally graded [3:58] exams where you can see the delta [4:00] relative to years past. [4:02] >> I see. And so like personally I went to [4:04] grade one in China. [laughter] I grade [4:07] one in China. And then I remember like [4:08] that grade one was intense like I was [4:09] learning tons of math trying to copy [4:11] like the all the mandering and I [4:13] remember getting like straight C's or [4:14] something in in grade one. And then I [4:17] went to school in Canada. I feel like [4:19] even in Canada, it was like pretty much [4:21] easy coasting all the way up to like [4:23] grade grade seven, [4:24] >> especially after taking grade one math [4:26] in China. [4:26] >> Yeah. [4:27] >> But but but I feel like I said just like [4:28] trade-offs. I I feel like some of the [4:30] Asian countries, they kind of teach kids [4:31] to just memorize stuff and like just [4:33] grind it out. And I think one good thing [4:34] about like the schools here is like, you [4:36] know, you have more free time for better [4:38] for so so you can explore things your [4:40] explore your own interests. Um I'm I'm [4:42] I'm just curious like have you had kind [4:44] of like a similar experience? Did you [4:45] grow up here or [4:46] >> I did grow up here. I mean, I've also [4:48] gone to school in Korea [4:52] and so I've had a few and I moved around [4:54] a lot as a kid. So, I've had a lot of [4:56] different educational experiences. [4:58] School is very long and I think the [5:01] hours are really more than you need for [5:04] uh the amount that you are actually [5:06] learning in that period. And schools [5:08] serve other functions uh in society [5:11] besides teaching kids things. And you [5:14] know to think that you know I I also [5:16] shared your experience where you know it [5:18] was like pretty boring in school a lot [5:20] of the time and to think that like now [5:21] we have made all of the tests a lot [5:23] easier even since then is is pretty [5:26] scary to think about. You know, I was [5:27] reading this data that came out recently [5:28] about the AP English exam and how, you [5:31] know, it's an AP. So, it used to test [5:33] undergraduate and graduate level [5:34] material and then it slowly drifted down [5:37] to, okay, we're testing high school [5:38] level reading passages and now we're [5:40] apparently in the last two years we're [5:41] testing middle school level passages and [5:43] it's like we've just given up. This [5:45] episode is brought to you by linear. [5:47] When engineers use tools like cursor, [5:49] clock code, and codecs, a lot of work [5:51] happens invisibly. Someone can go from a [5:54] bug report in Slack to a shipped fix [5:56] without creating any record of what [5:58] happened outside of the code editor. And [6:00] that's fine for speed, but it makes [6:02] coordination harder as you scale. Linear [6:04] integrates with the very best agent [6:06] coding tools directly like cursor and [6:08] codecs. That way, anyone can see what an [6:11] agent is working on and who assigned [6:13] them to the task. You get the speed of [6:15] agents without losing visibility across [6:17] the team. product teams at OpenAI, RAMP, [6:20] and Block are all using Linear to [6:22] collaborate with AI agents. And I use [6:24] Linear myself to run my creator [6:26] business. So, check it out at [6:28] linear.app/ [6:30] aents. That's linear.app/ [6:33] aents. Now, back to our episode. Yeah, I [6:35] guess just as an Asian parent, like like [6:37] a tiger parent. I try not to be a tiger [6:39] parent, but [laughter] [6:41] it's it's it's hard. Um, like I I feel [6:43] like uh you got to supplement in some [6:45] ways with like you know just like [6:47] teaching your kids your yourself, right? [6:48] And we're going to get to the AI tutors [6:50] soon, but like I'm curious if you have [6:52] any thoughts about what kind of learning [6:54] works best in the classroom environment [6:55] versus like one-on-one tutoring versus [6:57] like, you know, hands-on stuff. Have you [6:59] thought about this? Cuz I feel like 25 [7:01] kids in a classroom, unless I'm doing [7:02] like a school project with my [7:04] classmates, it's kind of boring just [7:05] listening to your teacher draw on about [7:07] stuff. [laughter] [7:08] >> Yeah. [7:09] >> Yeah. [7:09] >> Yeah. So classrooms are this unique [7:12] environment. You have a teacher, you [7:13] have a lot of peers. And I think a [7:16] really great use of classroom time is [7:18] peer learning and interaction with a [7:21] teacher who can motivate people and [7:23] inspire everyone in the class to learn. [7:25] You know, enthusi we are humans. [7:27] Enthusiasm is infectious. The best [7:29] teachers are creating excellent [7:32] collaborative learning environments and [7:34] they are devising ways for peers to [7:37] learn a lot from each other and that's [7:39] really powerful. But in most schools [7:42] that is not how classroom time is used [7:44] uh the majority of the time and so you [7:46] know you're spending most of the time [7:48] delivering lectures and there's a kind [7:50] of instructional delivery and practice [7:53] that happens through lectures and the [7:56] kid just actually practicing that I [7:59] think is not the best use of classroom [8:01] time and is not the best use of a [8:03] human's special abilities. like people [8:05] don't typically get into the teaching [8:07] profession so that they can stand at a [8:08] chalkboard and write stuff on there and [8:10] then watch students just stare at their [8:12] paper. Uh so so there's this like kind [8:14] of individualized instruction reporting [8:18] personalization that you know you can [8:20] get some of from a human tutor who you [8:24] know personalizes and tells the parent [8:26] what's going on and a great deal more [8:28] from software. You know, this is a lot [8:30] more than what a teacher can provide to [8:33] every student in a large classroom. And [8:35] I think software on many axes is going [8:37] to exceed what you can get with private [8:40] human one-on-one tutoring. You know, [8:41] obviously on cost and convenience, [8:44] software is on a whole different order [8:46] of magnitude. Human tutors are very, [8:47] very expensive and so most people are [8:49] priced out. And then you know for [8:52] features like if you are working [8:54] one-on-one with the student in a [8:56] software environment you get things like [8:58] real-time graphical interaction you can [9:00] adapt the behavior on the fly to the [9:03] student as they're problem solving. You [9:05] are able to do that based on a huge [9:07] amount of data and reps on how kids best [9:09] learn a particular concept. You can you [9:11] know generate personalized review. And a [9:13] thing we hear a lot is also like there's [9:15] no fear of embarrassment or failure. So [9:18] you're comfortable, you know, trying and [9:19] trying again. Like most kids would not [9:21] ever raise their hand in a classroom. [9:23] Like just the thought of that is [9:25] horrifying. But, you know, with an AI [9:27] tutor, like kids are very comfortable [9:28] doing that. [9:29] >> Interesting. Yeah. Yeah. I I think if [9:31] you have 25 kids in a classroom, it's [9:32] just kind of hard to have all the [9:34] personal contacts on each kid and like [9:35] give each kid your personal attention, [9:37] right? [9:37] >> Yeah. Yeah. It's impossible. I mean, [9:38] we're already asking teachers to do so [9:40] much. [9:41] >> Yeah. So, let's talk about that then. [9:42] So, so the mission of Brilliant is to [9:44] bring a world-class tutor to every home, [9:47] right? [9:47] >> Yeah. [9:48] >> Do you want to like talk about what that [9:49] looks like and maybe you can show us how [9:50] the product works too? [9:52] >> Yeah. Yeah. You know, when we say [9:53] worldclass tutor in every home, there's [9:55] some core philosophies underlying that. [9:58] So, for example, we believe technology [10:01] can and should be making our kids [10:03] stronger uh not dumber and weaker. And [10:07] you know, AI today is mostly used by [10:09] students for cheating. And one of the [10:12] analogies that we say is using AI as a [10:14] way to skip the part where you learn is [10:17] like bringing a robotic arm to the gym [10:19] to lift weights for you. Like we're [10:21] really losing the plot of why you went [10:23] to the gym in the first place. And so [10:25] what we built our AI tutor Cooji to do [10:28] is to use its own intelligence to [10:31] protect the time and effort that the [10:33] student spends thinking for themselves [10:34] just like a great teacher would. And [10:36] maybe what would be helpful is for me to [10:38] actually just show you. Yeah, I feel [10:40] like it's not even just the kids. I feel [10:41] like AI is making me dumber, too, [10:43] because it's just easier to just just [10:45] ask it to do stuff. [10:47] >> It's so It's so seductive. It's so [10:49] seductive. Okay, so let me go ahead and [10:52] play. So, you know, we uh this is an [10:54] actual playback of a real tutoring [10:57] session on Brilliant. Um and I think [11:00] it's just instructive to see, you know, [11:02] how someone gets stuck and what we're [11:05] doing there. Um, and so you'll see the [11:08] user checking the answer. Um, it's not [11:11] correct and then they're going back and [11:13] forth with the tutor Cooji. And you [11:16] know, one of the things that surprises [11:19] people about Brilliant is I think [11:21] everyone knows that just giving the [11:22] answer doesn't work for learning. But [11:24] what's much more surprising is that we [11:26] try not to offer explanations either. [11:29] Like explanations feel good, but you're [11:32] not doing the work yourself. And so what [11:35] we do is try to put the learner through [11:37] the harder work of doing the work [11:40] themselves. And so um you know in this [11:42] interaction the student and Koji are [11:44] going back via voice. You know for [11:46] privacy reasons we not share that. Um [11:49] but this is a student trying to learn [11:51] fractions by shading parts of a shape. H [11:55] they need to make 51 12ths by shading in [11:58] parts of the shape. and they shaded in [12:00] five different parts which makes [12:02] intuitive sense but the thing is that [12:04] the shapes are worth different amounts [12:07] and that is the insight that we want [12:09] them to discover here. So here the [12:12] learner understands the five but not the [12:13] 12th. And so not only do we not want to [12:15] give the answer, we don't want to just [12:17] tell them h and just give that [12:20] explanation that I gave you. You know, [12:22] it's the most natural thing for you to [12:24] want to do as a teacher or as a parent [12:26] that's like hovering over your child. [12:28] And it's actually not the most effective [12:30] tutoring intervention. Like that [12:31] explanation only makes sense if you kind [12:34] of already understand it. And you know, [12:36] one of the things that we have built [12:38] into the product is that is this idea [12:40] that cheating and explanations are [12:42] closer than you'd think. Like they are [12:45] both a way of letting the learner off [12:47] the hook. And we want the learner to [12:50] actually sit there in the struggle of [12:53] why is this wrong? Figure out the [12:55] insight themselves and realize how far [12:57] they can go in thinking through the [12:59] problem themselves. And you know, if we [13:02] can get them there, that is such a more [13:04] powerful learning session than if we had [13:06] just tell told them, oh, hey, this is [13:08] why it's wrong. [13:09] >> Yeah, that makes a lot of sense. Yeah. [13:10] But I I feel like I feel like building [13:12] this product, you kind of have to [13:13] balance the teaching and the motivation. [13:16] >> Totally. Totally. Like cuz if if it's [13:18] just I I guess a couple things. I was [13:20] watching my kid you use the software, [13:22] right? And and like in the beginning it [13:23] was pretty easy. So she didn't have to [13:25] talk to the AI at all. She can just like [13:26] answer all the questions. And then at [13:28] some point it got harder. And then I I [13:30] told her Yeah. I told her that there's [13:31] this avatar on the left that you can [13:32] talk to and uh yeah, it kind of walked [13:34] her through the steps and kind of like [13:35] it didn't actually explain the answers [13:37] to her, but it kind of asked her like a [13:39] bunch of prayerful questions to kind of [13:40] try to help her get to the answer. [13:42] >> Yeah. [13:43] >> Yeah. Yeah. Yeah. [13:43] >> But I think it's kind of like um just as [13:45] a building this product, how do you [13:47] think about keeping the kids going [13:48] without like making it too easy for them [13:50] or making it too hard for them or like [13:51] you know, how do you think about that? [13:52] >> Yeah. So, you know, there are both AI [13:55] and nonI interventions for that. So in [13:58] the breadand butter building of the [14:00] product, there are things that we do [14:01] around pacing and feedback and hints. [14:04] You know, as you noticed in the [14:05] onboarding, she didn't really need to [14:07] fall back to the tutor at all. She's [14:09] just like kind of building up reps, [14:10] figuring out, okay, how does this all [14:11] work? You know, what is a lesson? Oh, I, [14:14] you know, solve these problems and get [14:16] feedback. And it's getting a little bit [14:17] harder every time. And, you know, it [14:19] also helps to keep the learner just [14:20] actively doing something. you know [14:22] whether that's manipulating things or [14:24] making predictions or testing ideas and [14:26] getting feedback from the UI and you [14:29] know it's just all these little [14:30] interactions that then build up to the [14:32] student saying oh wait I I see it like I [14:35] get it now and then once you are [14:37] struggling then on the AI side there is [14:40] Cooji's personality and pedagogy like [14:43] how he responds when you're struggling [14:45] and that is one of our most active areas [14:48] of work right now so you know a Good [14:51] tutor is constantly making decisions [14:53] about the student's next move. So, it's [14:56] not just can I explain this concept. [14:58] It's what is the student confused about? [14:59] Are they ready for a hint? Did they make [15:01] a small mistake or is there, you know, [15:04] something bigger that they're missing? [15:06] You know, should I simplify the problem? [15:08] Should I ask them to draw it? You know, [15:10] do I just let them struggle for another [15:12] minute? And you know, as you rightly [15:14] point out, it really is a balance [15:16] between the student getting so [15:17] frustrated that they drop off and then [15:19] they're act they're not going to learn [15:20] it at all. Um, and actually doing the [15:23] thing that is the most effective thing [15:25] for that student to learn. And that [15:28] judgment and pedagogy is the trade-off [15:31] that we are building into the product. [15:33] You know, it's very early. We've got a [15:35] long way to go, but we ran the beta for [15:37] over six months and you know we've been [15:40] live for a few weeks now and we are [15:42] seeing improvement every month in [15:44] learners sticking with it more and it's [15:47] our topmost priority. Like the thing we [15:49] are asking learners to do on Brilliant [15:51] is stick with something hard. [15:52] >> Yeah, [15:53] >> it is absolutely like worse in the short [15:55] term for engagement. Like if we just [15:57] gave people the answer or like a you [15:59] know great hint that like totally [16:01] unlocks the problem for them when [16:02] they're stuck, people would do more and [16:04] they would probably get way less [16:05] frustrated. But uh I think that's like a [16:07] local maximum that you reach with the [16:09] product. And if we truly have super [16:10] intelligence, we should get learners to [16:12] stick with something hard and actually [16:14] use it themselves and you know get the [16:16] students to do better thinking. [16:17] >> Yeah. It's like the same incentives as [16:19] kind of like the US education system we [16:20] talked about earlier, right? Like it's [16:22] probably easier to just like help your [16:23] students all get A's and like [16:25] >> easy. Yeah. And as a teacher, it's like, [16:27] "Oh, man." Just uh, you know, and I have [16:30] kids, too, and I just want to tell them [16:31] like that it would be so much easier. [16:33] Like, they're about to give up. I just [16:34] want to tell you. Um, and so I really [16:37] think like so much of the magic of a [16:39] great human tutor is exactly how you [16:42] exercise that judgment and keep the kid [16:45] going and and basically extend the [16:47] amount of time that they can focus and [16:49] concentrate. [16:50] So, so let's talk about like uh how the [16:53] product works behind the scenes as as [16:55] builders like um so so I so I guess like [16:57] the default experience is like you have [16:58] a problem and then you can ask Cooji [17:01] about the problem, right? Or like you [17:02] can have a conversation with it. Um [17:05] >> so is it just as simple as that or like [17:07] what what what are some product [17:08] principles that that you have to kind of [17:10] make this thing a good experience? [17:12] >> Yeah, so the userfacing UI is very [17:16] simple. the what we do under the hood. [17:20] There's a lot and I came with a couple [17:22] demos to to show you. Uh but I think we [17:25] can boil down the you know what I'm [17:26] going to show to three things. Uh one is [17:30] that AI models are very spiky. And so [17:34] it's been really important for us to use [17:36] AI for what it's good at and not what [17:38] it's bad at. And I can talk a little bit [17:41] more about that. Uh second thing is the [17:44] agent interface is just as important as [17:46] the user interface when you're building [17:47] products today. Uh and the third is what [17:50] this means is everyone's job is now to [17:53] define an agent interface for their [17:56] area. So I can actually I mean that that [17:58] all sounds very abstract when I say it [18:00] like that. So let me uh let me pull up [18:01] my screen again and show you. Okay. So [18:05] here I'm going to show you a specific [18:06] example. Uh oh, let me okay of how [18:12] brilliant was built with agentic [18:14] workflows. So this is an example of a [18:18] video showing a learning designer who [18:20] has sketched out a lesson for teaching [18:22] the coordinate plane. You know when [18:24] we've tried to use codeex or claude to [18:26] simply generate this content h even [18:29] after training it on our entire body of [18:31] pedagogical interventions and content [18:34] you know it looks almost right but it is [18:36] actually nowhere close to the level of [18:38] quality control on the output that we [18:40] get when a master teacher designs it. [18:43] And people can tell uh you know the kid [18:45] struggles, they get super frustrated by [18:47] the sequence, like the teaching is a [18:49] mishmash. And so the core story or [18:53] pedagogy for every single concept on [18:55] Brilliant is all designed by humans. And [18:58] then what you're seeing here is that [18:59] it's implemented by AI. So the human is [19:02] designing the lesson and then handing [19:04] off to the AI to implement the design. [19:08] And what we're doing here is uh [19:10] equipping the LLM with a lot like [19:13] there's an entire product around it in [19:15] order to make uh a good teacher. [19:18] >> So the human design is actually each [19:20] question and and also the order and that [19:22] the LM executes. [19:24] >> Correct. Correct. Um, and that's, you [19:26] know, taking into account again like [19:28] what are models good at, what are they [19:30] bad at? And, you know, one belief that [19:33] we have is that companies that are [19:34] selling an experience that is under the [19:36] hood driven end to end by an LLM are [19:38] going to disappoint their customers. [19:40] Like it's really appealing to just let [19:42] the LLM swallow your entire product. [19:44] Like your users talk to the model, the [19:47] model goes off and calls other models [19:48] which then call other models and then [19:50] they magically solve the user's problem. [19:53] And I think general solutions like this [19:55] uh feel really seductive. But um you [19:59] know the issue is that any job that [20:00] you're trying to do for the user in our [20:02] case tutoring is made up of a dozen [20:05] different tasks. The models are very [20:07] unevenly shaped. And so they're amazing [20:09] at some of those things like this [20:10] implementation task that I just showed [20:12] you. And they're terrible at others like [20:14] designing an excellent pedagogical uh [20:16] learning experience. And so those tasks [20:19] that it's bad at are going to ruin the [20:21] whole experience. It's like and you see [20:22] this in coding too. The models are [20:24] excellent at writing working code. We [20:26] use them a lot for that. They are not so [20:28] good at architecture and that becomes [20:30] the bottleneck on how effective they're [20:33] going to be. And you know what we've [20:34] seen is that the models for our use case [20:37] are really bad at creating engaging [20:40] interactive learning experiences. Like [20:42] we can't let hallucinate that part. And [20:44] I really liked what Bology had to say [20:46] about this which is that AI is a [20:47] middle-to- solution not the endto-end [20:50] solution. And we found that this is like [20:51] a super useful way to think about, you [20:53] know, integrating AI into building [20:56] products. Like if you can limit the role [20:58] of AI to what it really shines at, [20:59] that's when it feels the most magical. [21:01] >> It's like it's Yeah, that's my [21:02] experience too. It does really well in [21:04] the middle 80% execution. [21:05] >> Yeah. [21:06] >> But then the the planning you have to be [21:07] involved and like the verification got [21:09] to be involved at the either. [21:10] >> Totally. [21:11] >> Yeah. Yeah. you have to really constrain [21:14] the use of what AI does in the product [21:16] or you know your product experience is [21:18] going to be governed by the thing that [21:20] the AI is bad at because that's what [21:21] people are going to notice and [21:23] >> maybe that that's where the secret sauce [21:25] is cuz like you know I I could probably [21:27] use cloud and generate like you know 200 [21:29] lessons too right [21:31] >> yeah exactly [21:31] >> about coronets and stuff but like you [21:33] know I think I I think just got to carry [21:35] enough to actually review his work and [21:36] make sure it's actually good instead of [21:39] >> pump slop yeah [21:40] >> yeah it's so easy to pump out slop. And [21:42] I think that like the, [21:45] >> you know, it's like going to get worse [21:46] before it gets better. There's a lot of [21:48] edgy slop out there right now. [21:50] >> Yeah. [21:51] >> Um, and I think that, you know, for [21:54] anyone building education products, what [21:55] I would say is that people can tell. You [21:57] might not think that they can tell, but [21:59] they can definitely tell. And you see it [22:00] in the engagement metrics. [22:02] >> Interesting. [22:03] >> Yeah. People just get frustrated and [22:04] drop off when they're having to read [22:06] slop. [22:08] >> Yeah. Even Even the kid kids, right? [22:10] Yeah, can tell. [22:12] >> Yeah, they can totally tell. [22:13] >> Yeah. And I feel like if you have enough [22:15] slop in your product, at some point it [22:16] becomes like the whole product becomes [22:18] slop and you don't even know what what's [22:19] going on anymore. It's like [laughter] [22:21] >> which parts are actually human craft and [22:22] which parts are slop, you know? [22:24] >> Totally. [22:25] >> So let's let's talk about the second one [22:26] though. What do you mean by designing [22:27] for agents? Because the user is still [22:29] the kid, right? Or like what do you mean [22:31] by designing product for a agents? [22:32] >> Yeah. So the user is the kid and you [22:35] want both an agent UI and a user UI. So [22:40] um I can I can go ahead and show you uh [22:42] you know what I mean by agent UI. But [22:45] before I do that like in brief when you [22:47] are building an AI native consumer [22:49] product um obviously you want an amazing [22:52] user interface. You want delight and [22:55] ease of use and you know visual product [22:57] interventions. You want it to feel you [22:59] know delightful and intuitive. And you [23:02] know we did this across all of our [23:04] learning apps but equally important now [23:07] is the agent interface. So these are the [23:10] tools and actions skills the harness [23:13] that the model uh needs to have to match [23:16] its intelligence to what's important in [23:17] the product. So let me show you [23:18] something uh that is our primitives [23:21] library. uh when we are building an [23:24] interactive app for teaching quadratics [23:26] uh we built the UI to be awesome but [23:30] under the hood uh for each of these so [23:33] you can see uh tons and tons of [23:35] primitives on the uh left rail here uh [23:39] each of these there is also an API that [23:42] a model can write to uh to create a [23:46] problem to annotate a problem to [23:48] highlight or ask a sub question Um, and [23:53] so these primitives can talk to each [23:56] other and talk to an LLM. And so these [24:00] were all intentionally designed so that [24:02] the model isn't hallucinating what to [24:04] do. And this is how you get really good [24:08] combination of latency and accuracy and [24:12] reliability because there is this sort [24:14] of deterministic grounding. And so, you [24:17] know, designing this agent interface for [24:18] our universe of learning applications [24:20] makes it much easier for uh a learning [24:23] designer to, you know, sketch a sequence [24:26] with an agent. And basically, this [24:28] library of primitives with the agent [24:30] interface helps power the agentic [24:32] workflow that I just showed off uh [24:34] previously. like the agent can grab the [24:36] learning designer sketches, use the [24:38] graphing module that an engineer built, [24:40] assemble the lesson uh following the [24:42] guidelines from our learning design team [24:44] and upload it. Uh and you know, you like [24:48] this kind of speed up is like part of [24:49] the magic of AI because just to cover [24:51] algebra 1, we had to build thousands of [24:54] modules and we had to build an interface [24:55] for the model to read and write to based [24:57] on what the user is doing in the UI in [24:59] real time. And so, you know, this is [25:01] what the AI is good at, like taking in [25:02] that multimodal input and reasoning what [25:04] the right next thing to do is and [25:06] avoiding the thing that it's bad at. [25:07] >> So, the primitives are kind of like uh [25:09] components that that um [25:11] >> Yeah, it's like modular composable [25:13] components. Yeah. [25:14] >> Got it. And so, so then they kind of put [25:15] some guard rails for the AI to make sure [25:17] to use the components to design lessons [25:19] with the human, right? Such generate [25:21] stuff. [25:21] >> Yeah. And so, you you know, you you're [25:23] not hallucinating the whole interactive [25:25] from scratch. You're not hallucinating [25:27] all of the misconceptions or uh tutoring [25:31] interventions from scratch. Instead, [25:33] you're providing the model with a a [25:35] schema that it can navigate very quickly [25:38] uh to figure out based on like what I've [25:41] ingested as multimodal input from the [25:42] user, what is the correct multimodal [25:44] output, but it doesn't have to [25:46] hallucinate all of the multimodal [25:47] output. Like if you did that, it would [25:49] take a long time and it wouldn't be as [25:50] accurate. [25:50] >> I see. Okay. Yeah. Cuz these elements [25:52] like structure day, right? They like [25:54] like they like little boxes and stuff to [25:57] stick to. Yeah. [25:58] >> Yeah. Yeah. If you can if you can [26:00] provide more structure so that they can [26:02] do less thinking, it uh you know has a [26:05] lot of benefits for your product. [26:06] >> Okay. Cool. Um what about the what about [26:08] the third one that you mentioned like I [26:10] think some something about the user [26:11] interface hasn't been invented yet or [26:14] >> Oh. Oh. So this is about how everyone's [26:16] job is defining an interface for their [26:19] area. So this is a way in which like [26:20] work at Brilliant has really changed. um [26:23] in that you know everyone who's [26:25] designing for users is also in some [26:27] sense also designing for agents like you [26:29] have to consider both parties and the [26:31] way that we think about AI is like the [26:34] glue or the connective tissue you know [26:36] we are building an ecosystem of [26:39] interactive learning tools that AI works [26:41] really really well with and you know [26:43] your job is to make it so that any [26:45] person uh at Brilliant with an agent can [26:48] get good information out and make good [26:50] changes so that you know people with [26:52] agents can navigate, you know, this [26:55] whole network of things with a lot more [26:57] autonomy. So, you know, one goal that we [26:59] have is to make it so that someone on [27:01] day one at the job can take advantage of [27:03] everyone else's expertise because it's [27:05] already baked into the system. You know, [27:06] how do we teach? What tools do we [27:08] already have? What do we use AI for? You [27:10] know, what do humans do and why? And [27:12] then you can bring the special skills [27:14] that you were hired in for and already [27:16] have a you know sort of comprehensive [27:18] set of interfaces and documentations for [27:22] uh agents to help you navigate the whole [27:24] network of things that we have. [27:25] >> Okay. So this one is more about like [27:27] employee productivity I guess. [27:29] >> Yeah. It's about like how we work you [27:31] know what are you thinking about [27:32] day-to-day you know are you building [27:34] something or are you building the [27:36] machine that builds the thing? And I [27:37] think more and more you know the way [27:39] that people's brains around their work [27:41] have shifted is uh you know we are [27:44] building the machine to build the thing. [27:45] >> Yeah that that's really interesting [27:46] because normally before all this AI [27:48] stuff like when you onboard an employee [27:49] they have to read a bunch of [27:51] documentation right they got to read a [27:52] bunch of wiks they got to have a bunch [27:53] of one-on- ones to onboard. Yeah. I [27:55] mean, and and then you have to write the [27:56] wikis and the one-on- ones. You know, I [27:58] I think one of the best inventions of [28:00] innovations of AI is the wiki that [28:02] writes itself and kind of plugs into [28:05] your workflow. And, you know, that used [28:07] to be a huge task to keep all that stuff [28:10] up to date. And now, you know, we don't [28:12] have to do that anymore, which is [28:13] amazing. So, so how do you so, so like [28:15] when you hire somebody like you know uh [28:17] how do you give them like do you just be [28:19] like hey here's kind of all of our [28:20] internal skills and workflows and like [28:22] you know like how how do you kind of [28:24] onboard them to this [28:25] >> whole and I think it's different for [28:28] different domains but I I would say one [28:31] core thing that's true across all the [28:33] domains is you learn by doing like we [28:36] really still to this day uh try to have [28:39] technical people ship something right [28:41] away because that requires you having [28:44] everything installed and working and [28:45] like really seeing from end to end like [28:47] how does something go from idea to [28:50] executed for the end user. You know [28:51] similarly if you are joining as a [28:54] learning designer you know our entire [28:56] learning design team at this point is [28:57] quite technical and we are you know [28:59] asking you to be able to use the tools [29:02] well enough to actually ship something [29:04] to the end user. So you know exactly [29:07] what the onboarding steps are you know [29:09] what you have to read what you have to [29:11] learn to do that change dayto day but I [29:13] think ultimately like we've done a good [29:15] job if it is uh efficient for you to [29:18] make an impact that makes the product [29:19] for users on day one [29:20] >> okay so actually ship something on day [29:22] one [29:24] >> user facing yeah [29:25] >> yeah I mean we're we're imperfect uh but [29:27] but I think philosophically you know [29:28] that is what we want everyone to do [29:30] >> let's talk about like how do you make [29:32] sure let's say you hired me as a [29:33] learning designer right Yeah. [29:35] >> And and I make like a math lesson. How [29:37] do you make sure that like the stuff is [29:39] up to quality? Like do you have um some [29:41] evals that you run or like you royal to [29:44] some percent of users first? [29:46] >> Uh we do have comprehensive eval. So all [29:48] of those like primitives that I showed [29:50] you. There are evals that run on them. [29:52] It's not just eval correctness. It's [29:54] eval obeying the laws of physics. You [29:57] know, not having areas of visual [30:00] overlap, having generous enough tap [30:02] targets. like we eval across a bunch of [30:04] different uh axes but one of the things [30:07] that we I think will be doing [30:09] indefinitely is that there are multiple [30:11] human reviewers for you know everything [30:14] that we put out and you know learning [30:16] designers are hired for their you know [30:19] particularly their craft and judgment [30:21] and taste on is this good so that's the [30:24] first order like before we put something [30:25] out there are humans judging whether or [30:27] not it's good there's evals that run we [30:30] also have synthetic students that we put [30:32] into, you know, whatever a thousand [30:35] tutoring sessions in that lesson to [30:37] check the tutoring interventions and [30:39] see, you know, is this going off the [30:41] rails, like, you know, are the tutoring [30:43] interventions like sensical for all [30:45] these different paths that users can go [30:47] down. Uh, it's nowhere near as good as [30:49] like real human data. Um, but it is, you [30:54] know, it's still a useful exercise to do [30:56] to catch any, you know, serious errors [30:59] uh before we ship it to users. And then [31:01] roll out to users is incremental. So [31:04] there's a lot of checkpoints along the [31:05] way. [31:05] >> So you have synthetic users that maybe [31:06] that have different personalities or [31:08] something. [31:09] >> Yeah. Yeah. So we have synthetic users [31:10] that go down like various different [31:12] adversarial or unhappy paths and that [31:16] just you know bark up the wrong tree and [31:17] then we have the tutor try to save that [31:20] session and then we see okay how is the [31:21] tutor working and you know it's like I [31:25] don't know it's 20% as good as real [31:26] human data but it's better than nothing. [31:28] >> Interesting. Okay. So maybe you have [31:29] like a legacy user versus like a [31:31] [laughter] like a someone who's too [31:33] smart or something like skips all the [31:34] stuff. Yeah. Okay. [31:36] >> Yeah. Yeah. Yeah. So there's different [31:38] different personalities. Um but you know [31:41] ultimately I would say the like most of [31:44] the arbitration of quality comes from [31:46] the team and there's some sanity checks [31:48] before it goes out the door and then we [31:50] learn a lot from actually just you know [31:53] watching the tutoring sessions and [31:54] watching how it's going and looking at [31:56] real student data. And there's no [31:59] there's no replacement for that you know [32:00] like once you have tens of thousands of [32:03] students going through a single concept [32:04] like you are going to get such more [32:06] comprehensive data on the right way to [32:08] teach that concept so that people people [32:10] understand it. How do you uh like okay [32:12] you can obviously get qualitative [32:14] feedback from the real students but how [32:16] do you measure the thing we talked about [32:17] which is like this thing is just [32:18] difficult enough to keep them going like [32:20] because cuz like for example if you only [32:21] measure like completion rate or [32:22] something then you just make it too easy [32:25] you know how do you you know is there [32:27] some quantifiable way of [32:29] >> yeah so we actually uh we actually don't [32:32] measure completion rate the only thing [32:34] that we care about is your performance [32:36] on the assessment and [32:38] >> Oh that's right there assessment that's [32:40] right yeah Yeah. Yeah. So, this is [32:41] something that we're uh you know rolling [32:44] out new features to across different [32:46] courses all the time and just testing [32:48] like how do we structure the assessment [32:51] to sort of get the best most incisive [32:54] data on whether or not the student has [32:55] actually learned. But you know when [32:57] we're building a course we are working [32:58] backwards from okay you know at the end [33:01] of this chapter what should the user be [33:04] able to do like and you know all of the [33:08] learning sequence is measured against [33:09] okay did the student actually learn how [33:10] to do that thing and got it [33:13] >> and so you know if you can do it in less [33:16] time with fewer nodes completed that's [33:18] great like we should be speeding you up [33:19] like maybe it's better for you to not [33:22] complete every you know thinly sliced [33:24] thing and you know if you're really [33:27] struggling then your goal might not be [33:29] to complete that course. Your goal might [33:30] be to go back to some like way earlier [33:33] thing and do that instead. So I think [33:35] it's really important to not measure [33:37] stuff like time spent learning or you [33:40] know quantity of stuff like ultimately [33:42] what matters is like can you do the [33:44] thing and so you know can you do the [33:46] thing can you do it today can you do it [33:48] in a week from now can you do it a month [33:49] from now like can you do it and have you [33:51] retained it? Yeah, because that that's a [33:53] really good point because I I remember [33:54] like when I was going through with my [33:55] kid like uh every lesson had a lesson [33:57] part and an assessment part, right? And [33:59] then like um [34:00] >> Yeah. Then there's a summitative [34:01] assessment part at the end too. [34:02] >> Yeah. So So I guess in some ways it's [34:04] it's kind of like a bunch of mini tests [34:06] which [34:06] >> Yeah. [34:07] >> I guess in the US education system it's [34:08] not very kosher to have a lot of tests [34:10] but [laughter] [34:10] Yeah. [34:11] >> Yeah. Yeah. I mean we're Yeah. I mean I [34:13] guess this is one benefit of being a [34:15] consumer company is you know it doesn't [34:17] matter sort of like what's kosher or [34:19] not. like we can just focus on what [34:21] works. And you know, we're always [34:22] testing things like, you know, do you [34:24] want more assessments closer together or [34:26] do you want them further apart? You [34:28] know, how big of a range of stuff do you [34:30] want to test uh per assessment? You [34:32] know, the thing that that's nice about [34:34] having a ton of usage is that you can [34:37] come up with ideas and then test them [34:38] pretty quickly. [34:39] >> That makes sense. That makes sense. Do [34:40] you have any thoughts on like how to [34:42] build a I mean Bernie has been around [34:44] for many years but do you have any [34:45] thoughts on how to build kind of like a [34:46] durable AI product with everything [34:48] changing all the time? [34:49] >> Yeah, I mean AI products feel pretty [34:51] easy to copy generally, right? [34:54] >> Yeah. [34:54] >> And so the way that we think about it is [34:55] that the most defensible products that [34:58] are going to come out of this era are [35:00] the ones that are generating valuable [35:02] data in domains that aren't [35:04] automatically verifiable. So if it's [35:05] automatically verifiable like coding, [35:08] you're going to get run over by a [35:10] frontier lab like anything where the AI [35:13] model has a verifiable reward is you [35:17] know I think it's just going to be tough [35:19] uh for startups to you know do anything [35:21] differentiated there that isn't just [35:23] like that isn't just beaten by the pure [35:25] scale unless you're in a super niche [35:26] base that they don't care about but then [35:28] [laughter] maybe maybe that business is [35:29] not the most promising startup. But [35:31] yeah, our rule of thumb is if the big [35:33] model companies can RL your domain, uh, [35:35] they will eat you alive, but there's [35:37] lots of alpha finding domains that no [35:40] one else can RL on. And tutoring is one [35:43] of those because the outcome that you [35:44] want there is a real human student [35:47] having actually learned something well [35:49] and having actually sustained attention [35:51] on the task. And so you need actual [35:54] tutoring sessions at scale to generate [35:56] that data. Like you need real student [35:58] mistakes and misconceptions. You need to [36:01] find out what tutoring interventions [36:03] work best. You need to develop judgment [36:04] on how to sustain their attention on the [36:06] task uh across you know billions of [36:09] problems solved. And in our case we're [36:11] doing that in a kind of graphical [36:13] tutoring environment that is totally [36:16] novel. And so we are using this data [36:19] from our tutoring sessions to build [36:21] better and better tutoring instincts now [36:23] into the model weights and uh in doing [36:27] so create you know a vertically [36:29] integrated optimized [36:32] experience for this very specific point [36:34] of view that we have on learning and how [36:37] we can make it more efficient and [36:39] effective. And so, you know, what I [36:41] would say is just like try to find areas [36:44] where you have unique data that makes [36:46] your product stronger over time [36:48] >> or or maybe like using your product will [36:50] produce the unique data. [36:51] >> Correct. Yeah. Yeah. You got to build [36:53] the product, you got to get the data uh [36:54] and then that data makes your product [36:55] better. Like that's a great flywheel. [36:57] >> Uh just play devil's advocate. Like I'm [36:59] I'm I'm sure a lot of kids are talking [37:00] to Chad GBT, right? Like [laughter] [37:02] >> Yeah. [37:02] >> But but but I guess they're probably not [37:04] asking this. They're probably not [37:05] following the same learning process that [37:07] that you're trying to do. They're [37:08] probably just asking Chad to do their [37:10] homework. [37:10] >> Yeah. [37:11] >> So, so in some ways, OpenAI has a bunch [37:12] of data, but it's probably not as [37:14] >> correct. They don't know if you've [37:15] learned it as we have. [37:16] >> Yeah. They don't know if you've learned [37:17] it. And actually, you know, I think [37:19] every Frontier Lab has run into this [37:20] with the homework help aspect of their [37:22] products, which is that students will, [37:25] you know, slurp in their homework, the [37:26] answer machine gives them the answers, [37:28] and then the interaction's over. And so, [37:30] you know, you don't know if you are [37:32] chatbt whether or not the student [37:33] actually understands it. That that's not [37:35] what the exchange was about. the [37:36] exchange was, you know, the user gave [37:38] you their homework and you solved their [37:39] homework for them. And so, you know, we [37:42] would say in in our case, like that's [37:44] not useful data for us. [37:46] >> Yeah. [37:46] >> Like the only data we got is what was on [37:48] your homework. Like what are you [37:50] learning right now? Um but whether or [37:52] not you know it's not data that we would [37:54] feed into making the tutor better. [37:56] >> Yeah. Yeah. I I I think kids are already [37:58] smart, right? They're going to find the [37:59] shortest path to, you know, finish the [38:02] homework and then they can go off and [38:03] play. But but yeah, but I think this is [38:05] the incentives problem, right? Like the [38:07] US is unique in that we make homework [38:10] count towards the grade. You know, in [38:11] other countries, the homework isn't like [38:14] you don't get graded on your homework. [38:16] You get graded on the test. [38:17] >> And so, you know, I I think students [38:20] very quickly understand, oh, cheating on [38:21] the homework doesn't help me with [38:22] anything. Like the homework is there to [38:24] help me [38:24] >> interesting, [38:25] >> you know, get reps on the things I need [38:26] to do to pass the test. And so, you [38:28] know, I know people are very concerned [38:30] about homework cheating as a use case of [38:32] AI, but it's like, well, you know, maybe [38:34] homework shouldn't have counted towards [38:35] the grade in the first place and like [38:36] now it's finally time for us to fix that [38:38] incentives problem. [38:39] >> Yeah, that's a really good point. And [38:40] maybe homework can be like more project [38:42] based. So, like is it actually like what [38:44] kind of stuff you make as opposed to did [38:46] you fill this sheet correct correctly? [38:48] >> Yeah. [38:48] >> Yeah. Yeah. I mean, I I think like what [38:50] outcome are you measuring? Like what are [38:52] you asking the student to actually do? [38:53] Like kids are smart. If you tell them, [38:56] hey, do this thing and do this thing so [38:59] that you can get a grade and that's the [39:01] purpose like and there's an easy way to [39:03] do that and get a good grade. Like, of [39:04] course, they're going to do that. But, [39:05] you know, that wasn't the point. That [39:06] wasn't the point of the homework, but we [39:07] have made it the point by creating this [39:09] incentive. [39:10] >> How how do you I mean, okay, you're [39:11] you're like a CEO of an education [39:13] startup. How do you with your own kids, [39:15] do you want them to get straight A's or [39:16] like what are your thoughts on this? [39:18] What are your thoughts on what they [39:19] should optimize for? [39:21] >> You know, my parents were very free [39:24] range. Like I don't think they ever [39:25] looked at a report card. Um [39:28] >> Wow. [39:29] >> I would Yeah, [39:30] >> they just uh they were very clear. [39:33] School is your job, not my job. You [39:35] know, I I think like other kids like [39:36] their parents like were paying a lot of [39:38] attention and they got a lot of help. Uh [39:40] and my parents are like if you can't do [39:41] your homework, I guess you can't do your [39:42] homework, you know? Like that's not my [39:44] job. You know, one thing I learned is [39:45] that they had very high standards. They [39:47] talked a lot about how hard it is to [39:50] achieve great things. uh they were [39:52] themselves unafraid to do hard things. [39:55] Um they you know said I need to use my [39:59] talents to solve really hard problems [40:01] and you know they did like also buy a [40:04] lot of biographies uh of which you know [40:07] the primary sort of like story in the [40:10] biography was people spending a really [40:12] hard time uh sorry spending a really [40:14] long time toiling away at super hard [40:17] problems that no human had ever solved [40:19] before. [laughter] [40:20] you know, just like literally like Marie [40:22] Cury uh giving herself, you know, po, [40:25] you know, radiation poisoning, like [40:27] these kinds of these kinds of stories. [40:30] And uh and I think that, you know, for [40:33] my kids, the thing that I really pay [40:35] attention to, and they're young still, [40:36] you know, they're five and three, and [40:38] the thing that I really pay attention to [40:39] is just helping them cultivate [40:41] concentration and attention. And I think [40:45] that, you know, they're interested in [40:47] different things. And so like the [40:49] easiest way to do that is just getting [40:51] them, you know, creating space for them [40:52] to be immersed in their own interests. [40:55] And what I really want is for them to [40:57] get reps of the idea that it should feel [41:00] hard to learn something new. Uh but also [41:04] it can be super satisfying. And so [41:08] however I can help them, you know, learn [41:10] that it's fun to do hard things, it's [41:13] satisfying to do hard things, it's also [41:15] hard to do hard things. And I don't [41:17] think it has to be math, you know. It [41:19] could be [41:20] >> building something. It can be monkey [41:21] bars, you know. It could be uh joining a [41:25] chess club or a lemonade stand or like [41:27] learning magic tricks. But I think as [41:30] soon as they start to believe that [41:32] learning sucks and like you do it to [41:34] please authority figures and maybe a [41:36] good way to do that is to have catchup [41:38] do your homework so that you can get a [41:39] good like you know at this point you've [41:40] like lost the plot, right? And it's [41:41] really really hard to undo. So I spend a [41:44] lot of time talking to them about you [41:45] know learning isn't about the answer. It [41:48] is about strengthening that part of you [41:50] that can find one that can concentrate [41:52] and struggle that's unafraid of trying [41:54] new problems. And so you know maybe this [41:57] will change as they get a little bit [41:59] older but right now I'm much more uh [42:02] interested in how deep of a rabbit hole [42:06] can they go down by themselves because [42:08] they're really interested in something. [42:09] >> Yeah, that's really important. And and [42:10] it's funny. I guess your parents kind of [42:12] like passively aggressively made you [42:14] [laughter] all those biographies. So [42:18] >> yeah. Yeah. That's really good. So I [42:19] think a lot about this too. My my kids a [42:21] little bit older. They're like five and [42:23] eight. [42:23] >> Yeah. [42:24] >> And and like I I I feel like definitely [42:25] one principle is like let them try many [42:27] things and see what they like. Right. [42:28] Yeah. [42:29] >> But I think like the whole basis of [42:30] brilliant is like even if you like the [42:32] thing, it doesn't mean that it's not [42:34] going to become hard, you know? Like to [42:35] actually become good at it, you have to [42:37] you have to struggle. So, so that I [42:39] think that's the part that I think is [42:40] really hard to even as a parent really [42:42] hard to teach because like you know like [42:45] for for example let's say she likes like [42:46] she she likes vibe coding games for [42:48] example she likes voding games [42:50] >> but then but then like you know [42:51] sometimes she runs with bugs and stuff [42:52] and and then she gets demotivated. [42:54] >> Yeah. [42:54] >> She gets demotivated and and then she's [42:56] like oh maybe I'm not very good at vi [42:58] games. [42:59] >> So so like you kind of have to be there [43:01] as a parent to kind of like re reward [43:02] her in some way shape or form. [43:04] >> Yeah. Yeah. And I think this is what [43:06] like learning products will get, you [43:07] know, better and better at exponentially [43:10] is creating that correct ramp like just [43:14] at the edge of your difficulty, like [43:16] that pacing that makes it so that you [43:18] stick with it. Um because every time, [43:21] you know, like when you play Mario, [43:23] right, and you're like jumping, there's [43:26] never a point where the kid is like, "Oh [43:29] man, I'm just not good at Mario. Like I [43:31] was not meant for this." You know like [43:32] it's structured in such a way that like [43:34] that next jump feels really hard but you [43:37] feel like if you try 10 more times you [43:38] can do it. [43:39] >> Yeah. [43:39] >> And I think that if more of learning [43:41] were like that where you know students [43:44] cultivated a belief that you know it is [43:46] going to get harder and harder and it's [43:47] going to require more and more effort [43:48] and my brain's going to have to work [43:49] harder and harder but you know I can do [43:52] that. That's like an amazing thing to [43:55] walk away with from you know the [43:56] learning that you do in your childhood. [43:58] >> Yeah. It's like a sense of [43:59] accomplishment, right? That you can do [44:00] something that other people might have [44:01] given up on. [44:03] >> Yeah. And a sense of, you know, [44:04] willingness to do something hard. [44:06] >> I think another thing I'll say is like I [44:08] love how you made Brilliant really [44:09] approachable and like cuz in the Bay [44:11] Area a lot of, you know, we talked about [44:12] this like a lot of private schools are [44:14] like 50k a year or something. [44:16] >> Yeah. [laughter] [44:16] >> And and I I I love how accessible [44:19] Britain is. [44:20] >> Yeah. And your kids are young, you know, [44:21] they're like way before the age that [44:24] it's built for. Like this is a product [44:25] for middle school and high school [44:27] students. But we we're getting there. [44:28] We're we're working on it. [44:29] >> Oh, okay. Yeah. Interesting. Okay. She [44:31] got pretty far on the code coding [44:32] course, actually. [44:33] >> Amazing. Amazing. Yeah. Kids are [44:35] >> hard. Kids are like way capable, more [44:38] capable than they think. I I I feel like [44:39] grade one kids can understand like [44:41] algebra. Like I feel like explain to [44:43] them. [laughter] [44:44] >> Yeah. And you know, we're having fights [44:46] about, you know, should we allow the [44:47] teaching of algebra in 8th grade in [44:50] California? [44:51] >> And it's just, you know, set high [44:54] standards. Set high standards. And like [44:56] you'll be very surprised by what kids [44:57] can do. [44:58] >> Yeah. All right. So, so where can people [45:00] find more about Brilliant and find you? [45:02] >> Yeah. So, we got our website at [45:04] brilliant.org. Uh you can find Brilliant [45:07] in the App Store and Play Store. Uh you [45:10] can also follow us on Twitter where we [45:12] post uh updates and uh preview things [45:14] that are shipping soon. Um I'm Sue Kim. [45:18] Uh the account for Brilliant is [45:19] Brilliant. [45:20] >> Awesome. Well, well, I have a ton of [45:22] respect for what you're doing. I'm I'm [45:23] I'm sure it's way harder to build a [45:25] education startup than like just build a [45:26] mobile game or something else, you know, [45:28] [laughter] [45:28] as Peter entertainment, but uh I I I [45:30] think I think it could have massive [45:32] potential on all of our kids. Thank you [45:34] so much. [45:35] >> Yeah, it's a labor of love. Thanks so [45:36] much, Peter.