OpenAI Embeds Medical AI in Flagship Models to Cut Care Bottlenecks

OpenAI将医疗AI融入旗舰模型以化解诊疗瓶颈 OpenAI正直接在通用前沿基座模型中训练医疗智能，而非开发专用的衍生分支，寄望于算力成本的断崖式下降和电子病历系统的整合，将AI打造为弥合破碎临床系统的环境化安全网。 Karan Singhal解释称，OpenAI直接将医学知识与安全评估整合至通用前沿模型中，而非分化出专用模型，从而借助基座规模带来的复合推理飞跃。OpenAI的双支柱策略旨在通过普及可靠且免费的基础医疗智能来“抬高底线”，同时借助主动式个人健康助理、盘活沉睡的纵向数据以及加速临床发现来“拔高上限”。AI的定位并非取代实体医生或侵入性手术治疗，而是充当环境化临床安全网和运营协调中枢，以减少行政浪费、协调医疗机构与支付方的利益机制，并将传统指南采纳长达5至17年的滞后期压缩至近乎实时。 每周有超过3亿人使用ChatGPT咨询健康与医疗问题，这促使OpenAI直接在通用前沿模型中训练医疗能力，而非将其孤立在专门模型中。 (1:00) OpenAI依托由数百名医生组成的国际网络对其模型进行评估，特意将临床安全性与审慎权衡置于用户顺应度或互动参与度指标之上。 (16:46) ChatGPT与Epic电子病历软件的集成就双向临床工作流程赋能，协助加利福尼亚大学旧金山分校等医疗系统简化病历审查，同时企业智能体已将AdventHealth的行政工作流程成本削减了约80%。 (24:53) OpenAI的GPT-4.1 Luna模型在面向临床医生的Healthbench Professional评估中表现优于GPT-4.5 South，同时将API成本降至三代同堂水平的三十分之一（下降约34倍）。 (43:39)

[43:39] Karan Singhal: And so one bet we've had at OpenAI is that the cost of health intelligence will will plummet at a given level of health intelligence. [8:12] Karan Singhal: We are very excited about this vision of kind of a proactive personal health assistant. I don't think that's any kind of replacement for actually having a having a doctor or having access to a doctor. [24:53] Karan Singhal: The way we think about this is you can have a model that's really good but if the activation energy to use the model to improve health is too high then people won't do it right. [47:11] Karan Singhal: The ideal world in my view is a world in which providers and payers kind of have a handshake like, hey, here's our ground truth of what happened in this visit. [41:18] Karan Singhal: I think one of the most interesting things about agents is they kind of solve a lot of the bottlenecks in analyzing data that already exists. [47:11] Karan Singhal: And so, you know, what I'd say to to our partners is, you know, build on top of models that are really capable. And the most important thing is bet that those models will become more capable.

[0:00] 5.3 plus trillion we spend on [0:02] healthcare. If it were its own country, [0:04] it'd be the third largest GDP. And yet [0:06] we have a ton of scarcity. Open's [0:08] mission is to ensure AGI benefits all of [0:09] humanity and give people access to [0:11] trustworthy, abundant, reliable health [0:13] intelligence. One of the stories that I [0:14] was most touched by was the story of [0:16] Nathan Levent. He basically had this [0:18] kind of long saga with his son where at [0:20] some point he was suffering a lot of [0:21] fatigue and irritableness and AI helped [0:24] them navigate the process and it turns [0:25] out he was diagnosed with an aggressive [0:27] form of leukemia. some of the work that [0:28] you've done around mental health. [0:29] There's a lot of complexity. How do you [0:31] help guide a user on that given that [0:33] could mean so many things? [0:34] >> We've just been working with a bunch of [0:35] licensed mental health experts across [0:37] the world to think about the right ways [0:38] to define model behavior and a bunch of [0:40] tricky scenarios and ensure that we can [0:41] get to a level of consensus where we can [0:44] actually measure performance. [0:45] >> Let's talk about trust. A person can [0:47] [music] be speaking to an LLM really [0:49] hoping to hear that everything is okay, [0:50] but from a medical perspective, they [0:52] need to hear perhaps something very [0:53] different. How do you teach an AI to [0:56] recognize the difference? Um, [1:00] >> hey everyone. Welcome back to the [1:01] podcast. I'm Jay Rugani, a partner at [1:03] A16Z, leading investments in healthcare [1:06] and biotechnology companies. I'm joined [1:08] by Karin Singal of OpenAI. He leads [1:11] health AI research at OpenAI and we've [1:13] got a lot to talk about. Thanks for [1:15] being here. [1:15] >> Thanks for having me. [1:16] >> I want to start with the big question. [1:18] Uh, there was a report earlier this [1:20] year. You all have talked about it [1:21] publicly. Hundreds of millions of people [1:24] each week use chat GPT for health [1:26] rellated questions. Uh what are they [1:29] asking? What are the kinds of questions [1:31] they're asking that they're not getting [1:33] from our main healthare system? [1:35] >> Yeah. Um let me zoom out and talk a [1:37] little bit about why we're investing in [1:38] health in the first place. Um so you [1:42] know open's mission is to ensure AGI [1:43] benefits all of humanity. And so we see [1:45] that at a high level happening in three [1:47] ways. Um the first is building and [1:49] deploying AGI. So building capable [1:50] models, building products around them, [1:52] deploying them, partnerships. The second [1:54] is preventing downside um safety of [1:57] various kinds. The third is making [1:58] upside happen. Um and so we view health [2:00] as a really really concrete way to make [2:02] upside happen and also work on some of [2:04] the safety problems that come up in [2:06] healthcare along the way. And so really [2:08] we're really invested in in making sure [2:10] basically two things. One is that we can [2:11] raise the the floor of human health and [2:13] give people access to trustworthy [2:15] abundant reliable health intelligence [2:17] for for free or nearly free and really [2:19] raise the ceiling as well and make [2:21] things possible that weren't possible [2:22] before. And so the best way to see the [2:24] kind of raising the floor is really to [2:25] think about how we've been working in [2:27] the consumer space. Um and so as we [2:29] reported there's more than 300 million [2:31] people weekly using CHP for for health [2:34] and wellness types of questions and we [2:36] take that as like an incredibly [2:37] important responsibility to to get that [2:39] right. Um, and that's been rapidly [2:41] growing um, over time. I think one of [2:43] the things that you see is just if you [2:45] if you think about the experience of of [2:46] getting healthcare even in the US, even [2:48] in the Bay Area, you know, if you're [2:50] looking to get a specialist appointment, [2:51] even here in the Bay Area, sometimes it [2:53] can take weeks or months, um, you might [2:56] um, spend a lot of time figuring out how [2:57] to get an appointment and scheduling and [2:59] all these kinds of things. And after all [3:01] that time, maybe get 15 minutes with the [3:02] doctor where you don't feel adequately [3:04] prepared and you maybe walk out of it [3:06] thinking, "Oh, I had like six more [3:08] questions that I wanted to ask." And our [3:10] hope is that AI can be part of this care [3:12] journey with you and like health is not [3:14] something that is just kind of [3:15] restricted to these kind of 15-minute [3:17] intervals. And AI can actually help you [3:20] navigate your care journey. And so that [3:21] actually reflects how people are using [3:23] AI on the consumer side today. Like [3:25] people are coming in with questions and [3:27] concerns, information that they'd like [3:29] to have. Um they want to figure out how [3:31] to kind of navigate their care visits or [3:33] ask questions or figure out how to [3:35] follow up things that they that the [3:37] doctor told them or things like this. [3:38] And we're finding that in between these [3:40] visits is is when AI can help people [3:42] most on the consumer side. And you know, [3:44] we're making similar investments on the [3:45] clinician side and on the enterprise [3:47] side as well. Yeah, it's amazing. We [3:49] have an extraordinary health care [3:51] system. Uh if I got really sick, there's [3:54] no country I'd rather be in than the US. [3:56] And yet we have a ton of scarcity. We [3:59] have, you know, you've seen some of [4:00] these statistics, over 100 million [4:02] people don't access to a regular primary [4:04] care doc. Even the ones who do often [4:06] have to wait 40 plus days to see that [4:08] doctor. Sometimes you meet the doctor [4:11] exactly as you said and it's a quick [4:12] 15inute visit and you have more [4:14] questions. I saw a dermatologist last [4:16] week about something. I wrote down all [4:18] my questions beforehand in an Apple [4:20] notes and I still forgot to ask [4:22] something and now I'm scheduling a [4:23] follow-up and it's another four months [4:25] out. It happens to all of us. We've all [4:26] felt that. Um, is there a specific [4:30] patient story or recent case that you [4:32] all have seen that kind of help us [4:34] imagine where the world is going and [4:36] where your work is taking us? [4:39] >> One one way in which the the work of our [4:40] team is gratifying is just like seeing [4:42] the numbers and seeing how many people [4:43] are using the tech and how useful it [4:45] seems to be for them. Um, one other way [4:47] is just hearing from people, right? So [4:49] we get people coming in sending us [4:51] emails or whatever it is reaching out in [4:53] various ways multiple times a week [4:55] saying like here's how this has helped [4:57] me. [4:58] >> I think one of the things that I one of [4:59] the stories that I you know was most [5:01] touched by was the story of Nathan [5:02] Levent and his um his son um so he hosts [5:06] the Cognitive Revolution podcast um and [5:08] kind of told his story in that podcast [5:10] but he basically had this kind of long [5:12] saga with his son where at some point he [5:14] was kind of suffering a lot of fatigue [5:16] and irritableness and and things like [5:18] this. um his his very young son. Um and [5:21] um AI helped them kind of navigate the [5:24] process of realizing that this was like [5:25] a a pretty serious thing while they were [5:27] on vacation and and deciding to take [5:29] them for testing and repeatedly telling [5:31] them like, "Hey, take this seriously [5:32] even when when doctors were a little bit [5:34] less sure and actually kind of [5:35] downplaying it." And it turns out he was [5:37] diagnosed with an aggressive form of [5:39] leukemia. Um, and throughout his care [5:42] journey, I mean, I mean, Nathan reports [5:44] kind of using GBD5 Pro at that time, you [5:46] know, quite a bit for helping navigate [5:48] the process, um, helping understand lab [5:50] reports and things like this even [5:51] before, you know, a specialist had time [5:53] to kind of help interpret. And I think, [5:55] you know, it brought them a lot of peace [5:56] of mind. Um, and so Nathan actually [5:59] tells that story. I don't I don't know [6:00] if I'm allowed to advertise other [6:01] podcasts here, but Nathan actually tells [6:03] that story in his podcast and would [6:04] recommend that for folks [6:06] >> in many ways. So, OpenAI is building [6:09] health intelligence and health [6:11] intelligence into your model at every [6:12] step, but it's the same model that folks [6:15] use to write code or plan a vacation. [6:17] Can you help folks understand why you're [6:20] building it that way from an [6:21] architectural standpoint? Why is a [6:23] general purpose AI the right foundation [6:25] for something as complicated as [6:27] medicine? [6:27] >> Yeah, I mean, people often ask us like, [6:29] you know, this kind of general versus [6:31] specialized model debate. And I used to [6:33] work on specialized models actually in a [6:35] previous role. And so so I definitely [6:36] have kind of experience on on kind of [6:38] both sides of this um this question. I [6:41] think the the kind of the kind of the [6:42] most important thing here is like what [6:44] we're talking about here is actually [6:46] specialized training in a general model [6:47] which is I think actually the best of [6:49] both worlds. So what we do um for for [6:51] Chash PT and all the models that we ship [6:53] in Chash PT every across every stage of [6:55] model training for JHPT we actually have [6:57] specialized healthcare data added to [6:59] each phase of training um since all of [7:01] our models in the last year. as a result [7:04] is now you have a model that benefits [7:05] from all the scale and general [7:07] improvements and intelligence and [7:08] reasoning of the big big frontier models [7:11] and also the benefits from the [7:13] specialized training that we get from [7:14] for example integrating data that we get [7:16] from working with a bunch of physicians [7:17] across the globe and so I think you get [7:19] the best of both worlds with this [7:20] approach if you if you kind of forked [7:22] and said like hey we're going to train a [7:24] kind of smaller specialized model that [7:26] is just for healthcare [7:27] >> um like I used to and like many people [7:29] have um what you'd find is that model [7:31] would just kind of fall behind the [7:32] frontier relatively really quickly [7:34] >> because the pace of the frontier and the [7:35] general model improvements is so fast. [7:37] Then if you bet on scale, what we found [7:39] as OpenAI as an industry is that things [7:41] will get better. Um, and so what we're [7:44] doing is kind of hitching a ride on that [7:45] kind of scale frontier. Um, and [7:47] leveraging the fact that we have the [7:49] ability to kind of integrate evals and [7:51] training data into the models to make [7:52] the models both specialize and also [7:54] benefit from that general capability. [7:56] [snorts] [7:57] >> I want to almost start imagining what's [7:59] possible with the technology that you're [8:01] building. Obviously where it is today is [8:03] amazing, but you're not stopping and [8:05] there's a lot lot of work to go. I think [8:07] many people are allured by the idea of a [8:10] personal doctor in your pocket. [8:12] Something that is perfectly [8:14] knowledgeable, has knowledge across all [8:17] medical specialties, so you don't [8:18] necessarily need to go see a [8:19] cardiologist and a dermatologist for [8:21] different questions. um something that [8:23] is perfectly personalized to you, has [8:25] all your medical history in context, um [8:28] remembers everything that you've said, [8:30] remembers your medical history, but also [8:31] your preferences. Um and is also able to [8:34] motivate you and nudge you so that uh [8:38] the information that it gives you is [8:40] given to you at the right time in a way [8:42] that can nudge you towards better health [8:44] behaviors or adhering to the protocol [8:46] that your doctor uh gave you. Is that [8:49] sort of the experience you're building [8:51] at OpenAI in the fullness of time or and [8:52] how close are we to that? [8:54] >> Yeah, that's something that we're we're [8:55] really excited about. We are very [8:57] excited about this vision of kind of a [8:58] proactive personal health assistant. I [9:00] don't think that's any kind of [9:01] replacement for actually having a having [9:02] a doctor or having access to a doctor. [9:04] But really thinking about the ways in [9:06] which we can take that experience of [9:08] getting health information and making it [9:10] more personal to kind of your context is [9:12] a really important part of kind of our [9:13] investment in health and chd which is [9:16] kind of consumer offering that we have [9:17] that enables users to connect their [9:19] health information for that information [9:21] to be used kind of securely not to be [9:22] used for for training data or for [9:24] targeting ads or anything like this. [9:25] Mhm. [9:25] >> And that's a big part of how we think [9:27] about how we're going to move towards [9:28] this world of really like increasingly [9:30] personal access and proactive access. [9:32] It's also been how we've been thinking [9:33] about deployment. So we've actually from [9:35] the beginning been deploying this as [9:37] widely um as possible to our free users [9:40] as well as our pro users. And so we have [9:42] been of the kind of 300 million weekly [9:45] active health users, the vast majority [9:47] of them are free users. And we take that [9:49] and most of them are outside the US as [9:51] well. M and so we we take that kind of [9:53] um question of like distribution here [9:55] really important really um really [9:57] centrally and so as we're thinking about [9:59] this kind of next frontier of like a [10:00] proactive personal health assistant [10:02] we're kind of bringing that to bear as [10:03] well [10:04] >> I think use cases help us imagine help [10:07] people imagine what's actually possible [10:09] what all this work is contributing to [10:11] the idea of a personal doctor in your [10:12] pocket I think is exciting when Greg [10:15] Brockman came on the podcast he spoke a [10:17] little bit about this situation he had [10:19] with a friend of her friend of is was in [10:21] the hospital and uh her physician was [10:25] about to administer a certain [10:26] medication. She double check double [10:29] checked that medication in chat GBT [10:31] realized based on her medical history [10:33] that she would have some drug uh [10:35] interaction or some um adverse event [10:37] that may come from that. She showed that [10:39] to her doctor her doctor agreed and said [10:42] I didn't have enough time to look [10:44] through your medical history uh to [10:45] notice that. So thank you so much for [10:47] for raising that. Shouldn't we do that [10:49] for everyone all the time? If it's cheap [10:52] enough and convenient enough, can't we [10:54] do a second, third, fourth opinion for [10:56] for every clinical scenario? Is that [10:58] where we're going? [10:59] >> Yeah. I I love this idea of a kind of [11:01] safety net for both patients and [11:02] clinicians that is just kind of there in [11:04] the care journey. And you know, not not [11:06] always interfering or or or being the [11:08] middleman, but like really thinking [11:09] about the ways in which it can augment [11:10] the existing experience and build on it. [11:12] Um you know we did this work actually [11:14] kind of more directly studying this on [11:16] the clinician side um in a kind of a [11:18] series of experiments that we ran uh [11:19] with Penda Health where we studied this [11:22] idea of like a clinician co-pilot that [11:24] was deployed and it was exactly deployed [11:25] in this way where you know it was a [11:27] safety net for the clinician as they [11:28] were doing their work. It kind of came [11:29] in and proactively suggested like hey [11:32] this might be off like you know this [11:34] there might be this medication [11:35] interaction that you might not be might [11:36] not be tracking [11:38] >> can we you know potentially adjust that [11:40] and things like this. I think we're [11:41] going to see the same thing happen on [11:43] the patient side and I'm very very [11:44] excited about this idea of like kind of [11:45] like a proactive p proactive agent in [11:48] your care journey really helping you and [11:49] making sure that like you know if your [11:51] A1C rises and you're diabetic like that [11:54] you get like review of your medications [11:56] by your doctor and things like this and [11:57] making sure that that's kind of flagged [11:59] for kind of appropriate review. [12:01] >> You mentioned sort of personalized [12:02] proactive agents. personal agents in [12:05] general have kind of emerged on the [12:07] scene from a popularity perspective over [12:09] the last few months. Instinct, Muse, [12:12] Grockbot, OpenAI's dots. Uh you can give [12:16] these AI apps an objective for folks who [12:18] are not familiar and it can execute a [12:21] series of tasks. So it's not just hey do [12:23] this task and then it executes the task [12:25] uh with a single loop. It can use your [12:28] preferences. It can access all of your [12:30] applications that you give it permission [12:32] to. uh it can access your credit card [12:34] information if you give that to them and [12:36] it can execute an objective on your [12:38] behalf. One can't help but imagine the [12:41] possibilities there in navigating our [12:44] wonderful but often disjointed health [12:46] care system. Scheduling a appointment [12:49] and making sure uh your insurance is [12:51] covered uh for that appointment or [12:53] covering that appointment. Uh you know [12:56] helping you and guiding you to follow a [12:59] care protocol that your doctor gave you [13:00] post discharge. Even negotiating with [13:04] your insurance company on a medical [13:05] billing error. all kinds of tasks that [13:08] just create friction in our healthcare [13:09] system. [13:10] >> Yeah. [13:10] >> Is that where you're kind of going with [13:12] with uh with your work? Should we look [13:14] forward to that from OpenAI? [13:16] >> I'm very excited about that vision. I [13:18] think if you look at from both a patient [13:20] and clinician perspective, one of the [13:22] most difficult things about the [13:23] healthare system and how it serves [13:24] people is this kind of question of like [13:26] fragmentation and kind of lack of [13:28] connection of the ecosystem. And [13:30] healthcare is also one of those [13:32] industries where you shouldn't expect [13:33] everything to have an MCP connector [13:35] tomorrow, right? It is a it is a place [13:37] where there's legacy software. There's [13:39] not like necessarily the best [13:41] connectivity with AI or software in the [13:43] first place. And many people are, you [13:44] know, running their primary EHR software [13:46] in the healthcare ecosystem on like a VM [13:48] for example. So like [13:49] >> virtual machine that they're running in [13:51] the cloud. Yeah. [13:51] >> Exactly. And so there's actually like an [13:53] interesting component where like a [13:55] virtual machine actually makes things [13:56] easier in some ways for for AI agents, [13:58] but this is absolutely ripe for agents [14:01] and computer use. Um, so, so I [14:02] definitely think that this is like a [14:04] place where a lot of the barriers that [14:06] have traditionally existed between [14:08] patients and getting the care they need [14:09] and clinicians and delivering the care [14:10] that they need and healthcare data in [14:12] general, which is often kind of a [14:14] bottleneck to the whole system working [14:15] well for patients. I think a lot of [14:17] those will dissolve in the coming months [14:19] and so we're really excited about that. [14:21] Okay, so we we're talking about the use [14:22] cases, you're helping us dream the [14:24] dream. Uh, before we keep going down [14:26] that that uh that road, let's talk about [14:28] trust. Obviously trust is the most [14:30] important here. The same underlying [14:33] model um can help me understand my lab [14:37] results or something that I heard from a [14:39] doctor um but it can also help a highly [14:42] trained specialized physician make sense [14:44] of a very complicated case. [14:46] What do you change in the answer so that [14:49] it's legible to each of those two [14:51] personas or any other persona that might [14:53] be engaging with the same underlying [14:54] model? Yeah, I mean as you said, health [14:57] is is a place where, you know, it moves [14:59] at the speed of trust. And so, you need [15:01] to be able to not only make the models [15:03] better and measure them, but also kind [15:04] of communicate and bring the world [15:06] along. And so, that's been a very [15:08] important part of our thinking and our [15:09] strategy for both how we work with [15:11] patients and how we work with end users [15:13] and how we work with clinicians. And so [15:15] um the the brief answer to your question [15:17] is basically we we evaluate and train [15:19] the models by working with experts [15:21] across the world and and getting [15:22] feedback in various ways to kind of [15:25] tailor their behavior um to whether a [15:27] person is a lay user or kind of an [15:29] expert lay user that has been like [15:30] navigating their care for a while a [15:32] complex care journey or clinician or [15:34] other kinds of user personas. And so [15:36] back even for the original healthbench [15:39] evaluation that we released back in May [15:40] 2025, there was actually a theme or part [15:43] of the the evaluation that was really [15:44] focused on this distinction of how the [15:46] model behaved and tailored its level of [15:49] kind of use of technical jargon and [15:51] things like this to whether a person was [15:52] a lay user or clinician. And since then [15:54] we've both gotten better at measuring [15:56] whether the models are able to cater to [15:58] different kinds of users and also [15:59] training the models towards that. Um, [16:01] and so a lot of our approach has really [16:04] been like not only do how do we make the [16:06] models better for these different users, [16:08] but how do we bring them along? And part [16:09] of that is how we do evaluations and [16:11] communicate about them as well. So [16:13] there's a few components to trust. So [16:14] we've talked about um sort of tailoring [16:18] the answer to the user. Another big [16:20] component is disagreement. [16:22] um a person can be speaking to an LLM or [16:26] speaking to chat and be sharing all of [16:30] their symptoms but really hoping to hear [16:31] that everything is okay. So from a [16:33] consumer user NPS perspective, net [16:36] promoter score perspective, they want to [16:38] hear reassurance, but from a medical [16:39] perspective, they need to hear perhaps [16:42] something very different. Yeah. [16:43] >> How do you teach an AI to recognize the [16:46] difference and manage different product [16:49] objectives there either from consumer [16:51] engagement and retention versus sort of [16:54] correctness and and sort of clinical [16:55] quality? [16:56] >> Yeah, this is super central to how we do [16:58] both evaluation and training. And so [17:00] zooming out a little bit and the [17:02] question of like agreement and [17:02] disagreement around how we um how we [17:06] respond and how we evaluate like one [17:08] thing you could do and one thing that we [17:09] you could you could kind of base your [17:11] approach on here is like we could bring [17:13] in a few clinicians into let's say the [17:15] opening high health team and say like [17:17] hey you guys figure out how to navigate [17:18] these kind of uncertain cases and you [17:21] know we'll we'll ask you about these [17:22] uncertain cases and you'll help us build [17:24] evaluations and training data and stuff [17:25] like this. um and leave it to those few [17:29] physicians to figure it out. Um [17:31] >> this is actually not the best approach. [17:32] So it turns out if you ask five [17:33] physicians or something this um any [17:35] question you'll get like eight different [17:36] answers. And so the the one of the first [17:39] things that we did when we first started [17:40] focusing on health at OpenAI is really [17:42] build this kind of global physician [17:43] network of hundreds of experts that we [17:45] work with kind of across the globe [17:47] across you know dozens of specialties [17:48] and languages and countries of practice [17:51] >> to think about the right ways to kind of [17:52] aggregate the wisdom of experts so that [17:54] we weren't relying especially in areas [17:56] where there could be disagreement we [17:57] weren't relying on one or two or three [17:58] physicians to kind of adjudicate. So [18:00] that's that's one part of it um is kind [18:02] of like thinking about um evaluation and [18:04] training data coming in from different [18:05] experts rather than relying on one or [18:07] two people. But then the second part of [18:09] your question is like how do you balance [18:10] kind of competing objectives, right? [18:12] Let's say the physicians and the [18:13] aggregate wisdom of the physicians is [18:14] pointing in one direction [18:15] >> and then maybe the user actually wants [18:17] you to say something comforting even [18:18] when the situation is not comforting [18:20] there. It's basically a prioritization [18:22] of safety. Um and so [18:23] >> it's very important to us that like we [18:25] we take our obligation very seriously [18:27] when there's hundreds of health hundreds [18:29] of millions of health conversations [18:30] coming in. And so across you know the [18:32] evaluation stack and training stack um [18:34] we have evaluations that track these [18:36] kind of difficult situations where the [18:38] user should where the model should [18:39] actually go and say like hey this is [18:41] unsafe like you should seek care. Um [18:43] even if the user doesn't want to hear it [18:44] or you know wants to get some kind of [18:46] alternative treatment or something like [18:47] this. Um, and that's, you know, a very [18:49] important part of how we do evaluation. [18:51] And over time, we've not only improved [18:53] how we've done the evaluation with [18:54] working with physicians, but also seen [18:56] pretty dramatic model improvements [18:58] across the board. Uh, in those areas, [19:00] there's another example or another [19:02] instance of disagreement, which is me as [19:04] the consumer. I ask a medical question [19:07] of chat GPT. I then show that response [19:10] to my doctor during a visit. My doctor [19:12] loves me, by the way. Um, and and then [19:15] they disagree. they disagree with the [19:17] response. How should I handle that [19:19] situation? [19:20] >> Yeah, I mean I think you should handle [19:21] it similarly um as if you know [19:23] disagreement was happening in some other [19:25] way as well. Like people people do [19:26] disagree about um different things and I [19:28] think like you have a a right as a [19:31] patient to get you know ask for another [19:33] doctor to take a look and and get a [19:35] sense and go to another doctor to get a [19:36] second opinion [19:37] >> and yeah that's what I would advocate [19:39] for whether or not you're also using CHP [19:41] to get health information as well. [19:43] >> Yeah, [19:44] makes a ton of sense. There's another [19:47] element of trust which is just absolute [19:49] correctness and a lot of the [19:53] a lot of the work that you all have done [19:56] it seems to be at least I'm hearing from [19:58] you seems to be about grounding answers [20:00] in the world's medical literature and [20:02] evidence. Some parts of medicine there's [20:05] not a lot of agreement or disagreement [20:07] about, but there is much of medicine [20:08] that's still uh being discovered or [20:11] being studied. And there is disagreement [20:14] even in the medical field. And for that [20:17] um you know for that for those kinds of [20:20] questions it's difficult to ground it in [20:21] truth when we don't we're still [20:22] discovering you know truth science as a [20:24] process etc. How do you think about [20:27] correctness in those kinds of instances? [20:30] >> Yeah it's a really great question. I [20:31] mean it's important with or without AI. [20:34] So if you kind of zoom out and look at [20:35] like how is medical consensus [20:37] distributed today, right? Like what is [20:39] the process of that today? You basically [20:42] have um a bunch of people um writing [20:45] papers, biomedical papers of various [20:46] kinds um suggesting kind of improvements [20:48] or treatments or things like this that [20:50] could that could improve the quality of [20:51] care. And then you have this process, [20:53] yearly process across different [20:54] specialties of building clinical [20:56] guidelines which kind of describe like [20:59] hey in this situation if you're a [21:00] cardiologist and you see a patient with [21:02] X Y and Z here's what you should [21:04] probably do. Um and so these these [21:05] clinical guidelines are roughly you can [21:07] consider them kind of flowcharts that [21:09] are put out each year by the different [21:12] kinds of bodies across the different [21:14] across the world that describe like [21:16] roughly what you should do in different [21:17] scenarios based on the latest research. [21:19] >> Mhm. And then the final step is so you [21:21] have you biomedical literature and then [21:22] you have the um clinical guidelines and [21:24] then you have like what do people [21:25] actually do in practice and so the final [21:29] step it turns out is a is a ma massive [21:30] bottleneck. So there are these studies [21:32] on like how long does it take for um [21:34] clinical guidelines to penetrate the [21:36] practice of everyday doctors across the [21:38] world. [21:39] >> And um in the past people have found [21:40] that this takes depending on the study [21:42] like 5 to 17 years for that for that [21:44] penetration to actually happen [21:45] >> which is massive. Like you don't want to [21:47] be practicing medicine 5 to 17 years [21:49] behind. And so one of the things that we [21:51] think about is how do you compress that [21:54] kind of period of of kind of practicing [21:57] in in in the past um down to weeks or [22:00] days or even hours, right? And so this [22:03] is one very important principle behind [22:04] our work in chachpd for healthcare and [22:06] chachpd for clinicians which are [22:08] clinician and healthcare professional [22:09] facing offerings where we're bringing in [22:11] the latest biomedical literature in the [22:13] moment of care and allowing people to [22:15] very quickly in the course of like let's [22:16] say a 15-minute visit at the point of [22:18] care get answers to their questions. So [22:20] that's one way of answering the question [22:21] which is like you know in practice you [22:23] know while there is medical a lack of [22:25] medical consensus in many areas in some [22:27] cases there is medical consensus but [22:29] it's not being practiced and so one [22:30] thing you can do is like really [22:31] dramatically reduce the amount that [22:33] that's happening and so that's one one [22:34] way of answering your question a second [22:36] way of answering your question is like [22:37] how to handle the places where there's [22:38] still like genuine medical uncertainty [22:40] and there I think I mean there's a [22:42] couple things so one is you want to [22:43] train the models to be able to recognize [22:45] their own uncertainty like after after [22:47] browsing and getting information and [22:48] things like this you want the models to [22:50] be able to know, hey, like there just [22:51] isn't enough information and to be [22:52] upfront about that. And so, you know, [22:55] that's been an important part of how [22:56] we've done evaluation that's actually [22:57] like a public aspect of um healthbench, [23:00] which includes kind of evaluations on [23:01] hedging, for example, when there is this [23:03] kind of uncertainty. So, that's one one [23:05] important thing. The second thing, I [23:07] mean, this kind of goes to this question [23:08] of like how do you raise the ceiling of [23:09] human health is like really thinking [23:11] about the ways that models can actually [23:13] accelerate the process of doing science [23:14] and actually pushing the medical [23:15] frontier. And so, maybe we'll we'll chat [23:17] about that later. [23:19] Some of these areas are gray areas and [23:22] controversial. And I'm curious if you if [23:25] you think the models or you're designing [23:26] the models in a way to take a stance on [23:28] these things, unapproved peptides, [23:31] vaccines, [23:33] all things that just get politicized [23:34] oftent times in the media. How should a [23:37] product like like chat engage on some of [23:39] those topics where there's still debate? [23:43] Yeah, I mean where there's debate or a [23:44] lack of consensus across the board [23:46] whether it's politicized or not, we want [23:47] to make sure that we are relying on the [23:48] experts to help define kind of standards [23:50] for how the model should respond. And so [23:52] that's kind of the approach that we take [23:53] across the board. Hearing you talk, it [23:56] feels like you're always thinking about [23:59] one of three constituents. It's either [24:01] the patient, it might be the individual [24:03] clinician that signed up, or it might be [24:05] the enterprise, whether it's a [24:06] healthcare provider enterprise or some [24:08] other enterprise. How do you think about [24:10] designing the product at the [24:12] architectural level and at the let's say [24:14] the harness level to meet the needs of [24:16] all these different users? [24:18] >> Yeah, I mean I think maybe there's three [24:20] parts to that. One is making the models [24:21] really really good for these users. The [24:23] second is building the right harness or [24:25] product around it and the third is [24:27] really like the deployment or [24:28] partnership motion to kind of re reach [24:29] out to the right reach out to the users. [24:31] And like by default I think a lot of the [24:33] time the the models are often pretty [24:35] good these days. And so we talked about [24:37] our approach of evaluation working with [24:39] clinicians and things like this. We are [24:41] seeing pretty dramatic improvements in [24:42] kind of performance and reliability. You [24:44] know we talked a few months ago about [24:46] the improvements that we've been making [24:47] on kind of the free tier and bringing [24:50] free tier performance to the frontier. [24:51] So you know of the 1.2 billion people [24:53] who use Chaship PT weekly um giving them [24:56] a model for health that is just like [24:58] incredibly useful at the frontier for [24:59] free. Um so that's been a big part of [25:01] our strategy obviously for for making [25:03] healthcare health intelligence abundant. [25:05] The second thing is our kind of product [25:07] offerings. And so the way we think about [25:09] this is you can have a model that's [25:11] really good but if the activation energy [25:12] to use the model to improve health is [25:14] too high then people won't do it right. [25:15] So a lot of our product efforts are [25:18] really like thinking about these [25:19] constituents that you're talking about [25:20] and thinking about the ways in which we [25:22] can reduce the activation energy for [25:23] people to use compute and intelligence [25:25] to improve human health. Um and so that [25:28] goes you know along the lines of you [25:30] know our work in chash PT uh health and [25:32] chash PT um chd for healthcare um on the [25:35] enterprise side and chash for [25:36] clinicians. A lot of the way we design [25:38] these offerings is basically to think [25:40] about like what are the barriers and [25:41] bottlenecks and activation energies for [25:43] people to be able to use the models. Um [25:45] and so for example um you know what are [25:47] the um security and compliance [25:49] bottlenecks in providing an enterprise [25:52] offering of chachbt to healthcare [25:54] systems. So that was like one question [25:55] that we were thinking about a lot 2 [25:56] years ago. And so we thought, okay, we [25:59] need to we need to kind of invest in [26:00] security and compliance. We need to make [26:02] sure we're able to offer HIPO [26:03] compliance. We need to make sure that [26:05] we're able to offer the kinds of of um [26:07] privacy protections that healthcare [26:08] organizations need so that we're able to [26:10] handle the healthcare data that they [26:11] need and be able to offer services that [26:13] they trust. [26:14] >> And so that was like a very important [26:15] part of the work we were doing for like [26:16] roughly two years ago to like roughly a [26:18] year ago. and um and we kind of just go [26:21] after there from from there like based [26:23] on the bottlenecks that we see to kind [26:25] of adoption. The third thing is kind of [26:27] partnerships and deployment. And again, [26:28] it's really like, you know, a few years [26:31] ago, I think OpenAI was very early to [26:32] speaking the language of healthcare. Um, [26:34] and when we um interacted with [26:37] healthcare partners, I think we were, [26:38] you know, early in those conversations. [26:40] So, I think one of the things that we've [26:41] brought to bear since then is really, [26:43] you know, thinking about each of those [26:44] constituents that you're talking about [26:46] and really thinking about how to speak [26:47] their language and how to kind of bring [26:48] them along this this journey because [26:50] even if the models get really smart, if [26:52] nobody's like really trusting them or [26:53] using them, that's not really achieving [26:54] our mission. And is that uh you're [26:56] alluding to a little bit some of the [26:58] work you've done with Epic to kind of [26:59] integrate the product directly into [27:01] Epic. Maybe share a little bit more [27:02] about that because a lot of the folks [27:04] who spent a lot of time in healthcare uh [27:06] were very interested to see that level [27:07] of collaboration with with Epic. [27:09] >> Yeah, I mean this is a very important [27:10] part of our approach. You like like we [27:12] talked about fragmentation um in [27:14] healthcare has been one of the issues um [27:16] and the fact that there is kind of data [27:18] and workflows that are disconnected from [27:20] each other I think is is a big obstacle [27:21] to the to the use of AI productively in [27:24] healthcare. Um and so you know our work [27:26] with Epic or work with other um partners [27:28] we had a recent announcement where we [27:29] announced that there's now a way to use [27:31] Epic in CHGPT um if you're if you're a [27:33] provider or healthcare professional and [27:35] also for chat GPT to be useful in Epic [27:37] itself. And so on in the Epic workflow [27:39] itself you can have like a little um [27:41] sidebar where you can actually use Chat [27:43] GPT as well. [27:44] >> Um and so that's one thing we we've [27:47] we're also working on like different [27:48] other kinds of connectors that we've [27:49] also announced recently and we have more [27:50] coming. And so there is quite a bit of [27:53] work that we're doing to make sure that [27:54] we are kind of breaking down these [27:55] barriers to kind of the healthcare [27:56] ecosystem being connected. And I think [27:58] you know Epic is just one of many [27:59] examples where we're just identifying [28:01] those bottlenecks and trying to fix [28:02] them. The last thing I want to hit on [28:04] with respect to trust is uh some of the [28:07] work that you've done around mental [28:08] health. And uh this is another area [28:11] where there's a lot of complexity and [28:13] there may be other objectives that you [28:15] need to solve for beyond um the clinical [28:18] complexity that that we talked about. If [28:21] I ask my openai dot, hey, I want to feel [28:24] better over the next few weeks. How do [28:26] you even interpret that question? How do [28:28] you help act on that? How do you help [28:30] guide a user on that given that that [28:32] could mean so many things? [28:34] >> Yeah, I mean and regardless of the [28:35] surface, we are super committed to like [28:37] helping users when they're having a [28:38] difficult time or when they're in [28:39] crisis. And so, for example, for DOTS, [28:41] we've been investing the same kinds of [28:43] safeguards that we've been putting in [28:44] all of our product surfaces in that [28:46] product surface as well. So, that's an [28:48] important part of our approach is kind [28:49] of like a a baseline safety floor. And [28:52] then you know with mental health bench [28:53] we've really taken this approach of of [28:55] not just measuring the safety floor but [28:57] thinking about the ways in which the [28:58] models can actually like proactively [28:59] help kind of raise the ceiling um and [29:01] really think about the ways in which [29:02] models can you know help provide [29:05] positive mental health experiences and [29:06] move from move from just um worrying [29:09] about the safety floor to measuring [29:10] those those kinds of upsides. And so [29:12] again, the approach is is actually very [29:14] much mirroring our work on Healthbench [29:16] where we've just been working with um a [29:18] bunch of licensed meth mental health [29:20] experts across the world to think about [29:21] the right ways to define model behavior [29:23] and a bunch of tricky scenarios and [29:25] ensure that we can get to a level of [29:27] consensus. Um people definitely disagree [29:28] even more here than physical health. Get [29:31] to a level of consensus in some of these [29:32] tricky situations where we can actually [29:34] measure performance and create this kind [29:36] of like shared benchmark from the [29:37] industry. And um so we released that [29:39] that benchmark very recently and similar [29:41] to healthbench and healthbench [29:42] professional the goal has really been [29:44] can we create this benchmark that kind [29:45] of raises the ceiling for everybody and [29:46] kind of creates across all model [29:48] developers and the healthcare industry [29:50] and and mental health experts like this [29:52] really shared ruler or measuring stick [29:55] that we can use to kind of make sure [29:56] that we're moving in the right [29:57] direction. [29:59] >> Okay. So we've spoken about intelligence [30:01] and the use cases that are coming around [30:03] uh intelligence approving. We've spoken [30:04] about trust which is critical. Let's [30:06] talk about medical care. And I want to [30:08] draw the distinction between medical [30:11] information and medical care. Uh I think [30:13] for medical information, I think if you [30:16] have a health concern, if I have a [30:18] health concern about my myself or [30:20] something that I'm dealing with, chat [30:22] and other LLMs are increasingly the [30:25] better, faster, cheaper way to get an [30:26] answer and a resolution to that [30:28] question. But if uh in the scenarios [30:31] where medical information alone doesn't [30:33] get me to resolution, I then uh at least [30:37] today am thrusted into the traditional [30:39] medical care system. And so if I need to [30:42] go get a prescription, if I know need to [30:44] go see an in-person specialist, if I [30:47] need to get some follow-up blood work or [30:49] some MRI or some other form of scan [30:52] done, if I need a clinician to examine [30:54] me in some way, I still need to navigate [30:56] the traditional health care system. And [30:59] a lot of that work um uh is sort of uh [31:03] governed and regulated by the US medical [31:06] license and sort of other infrastructure [31:07] that we have in our healthcare system. [31:09] Um it's the multi-turn conversation [31:12] where it's not just a single question [31:13] resolution. It's I got to go talk to a [31:15] few people. I need to go get a few [31:16] things done. We need to collect some [31:18] information. In many ways, medical care [31:21] is a it's a bundled product. you know, [31:24] we're getting the diagnosis and triage [31:26] and the care plan. We're getting the [31:28] empathy from the clinician and the [31:31] assurance and the support. We're getting [31:34] the legal liability associated with that [31:36] and what they're taking on. Will AI [31:38] provide all of that? Uh where will it [31:40] stop short? Uh where do you think where [31:42] do you think this is all going? [31:43] >> Yeah. No, I don't think AI can can or [31:45] will provide all of that. And I think [31:47] you're right that there's like this [31:48] distinction between health information [31:50] and and medical care. And so I think I [31:53] think the way we see our role here is [31:55] really to enable the whole ecosystem [31:56] because there's no future where you know [31:59] any one company is delivering all the [32:00] medical care right and so our role here [32:03] is really to think about what is the [32:04] last mile of medical care really look [32:07] like for for people. Can we really [32:09] understand that and can we make that [32:11] easier and um easier for everybody so [32:14] that last mile of actually having impact [32:15] on patients just looks like the last 100 [32:17] meters for all of our partners. And so [32:19] that's kind of how we think about it is [32:21] like really like making it very easy for [32:22] for our partners to use AI to support [32:24] everybody in the care journey. I think [32:26] there is stuff that OpenAI can do [32:27] relatively uniquely here because of the [32:29] front door to health that it has. And so [32:31] for example, it can help you navigate a [32:33] lot of these kinds of questions or help [32:34] you navigate this journey or you know as [32:36] we get increasingly helpful personal [32:38] agents help you with some of the [32:40] scheduling and logistics and things like [32:41] this and just act as a quarterback for [32:43] your health. Um that said, I don't think [32:45] it'll resolve all these challenges that [32:47] we'll have, at least not in the short [32:49] term. And I think to a large extent, [32:50] we're looking to enable a lot of our [32:52] partners that are delivering care that [32:53] are, you know, overwhelmingly using our [32:55] models in various ways [32:56] >> to like make these operations happen [32:58] smoother and easier so that patients are [33:00] not left in this kind of fragmented [33:02] experience. [33:03] >> So should the as a consumer should the [33:05] experience that I look forward to uh [33:07] from OpenAI be um sort of the [33:11] quarterback care that you described? [33:13] Will I be talking to a clinician in the [33:15] interface at any point or would you [33:17] perhaps route me to a doctor and help me [33:19] find that given that's a whole challenge [33:21] just scheduling and making those visits [33:23] happen. What should I look forward to? [33:25] >> I I think I think you'll see like we we [33:27] will be hopefully very helpful in [33:30] helping you navigate the process of [33:31] finding care. I don't think we'll be in [33:33] the business of doing all the things [33:34] that you mentioned. [33:35] >> Um but and that that's where we're [33:37] really relying on all of our startup and [33:39] care provider organizations and partners [33:40] and things like this. So, so coming back [33:43] to a question I mentioned earlier, which [33:44] is you are building that future, but [33:47] you're building it with partners, you're [33:49] building it for consumers and patients, [33:50] you're building it for clinicians, [33:51] you're building it for enterprises. For [33:53] clinicians and enterprises, like give us [33:55] some texture on uh on some of the tools [33:58] that you're launching and and some of [34:00] the collaborations that have really [34:01] worked. [34:02] >> Yeah. Um we've had a few really exciting [34:05] partnerships and we've had some really [34:06] amazing launch partners for Cheshpd for [34:08] healthcare. We've had lots of users [34:10] using catch for clinicians and really [34:12] enjoying it. It's been um the retention [34:14] on the on the offering has been like a a [34:16] very pleasant surprise. People people [34:17] seem to be really enjoying it. Um yeah, [34:20] we've been working with a variety of [34:21] health systems. I think one um a couple [34:23] that stand out you know one is um we've [34:24] been working with UCSF and we've been [34:26] chatting about that and they've been you [34:28] know one of our earliest adopters for [34:29] this connection between Epic and the EHR [34:31] and CHPT and they've been really on the [34:34] front lines of thinking about for [34:35] example clinicians using it for chart [34:38] review or things like this um and [34:39] thinking about the right ways in which [34:41] like the experience of caring for a [34:43] patient doesn't need to be this super [34:44] fragmented you need to switch tabs or [34:46] switch experiences and things like this [34:48] and so they've been some of our power [34:49] users on that front. Um I think another [34:52] one um is Advent Health and they've [34:54] actually been some of our power users in [34:56] think and thinking about deploying [34:57] agents across the workspace actually [34:59] across all of OpenAI. They've been um [35:00] they've been a leader um and they've [35:02] been really um they've had amazing [35:04] results. They've reported roughly 80% [35:07] reduction a lot of their administrative [35:09] um workflow costs using AI in these [35:12] kinds of ways. And I think a a lot of [35:14] health systems I think should look to [35:15] them. I think they're they're really [35:16] impressive. If I go into my doctor's [35:18] office and I see chat GBT open up on [35:21] their screen, should I be upset? Should [35:23] I be happy? Yeah. [35:24] >> How do you think people will react to [35:26] that? [35:26] >> Yeah. I I I hear I hear differing things [35:28] and I think it's actually changing over [35:29] time. And so [35:31] >> I I think it's this interesting thing [35:32] whereas as the Overton window is [35:34] shifting in real time. I think you know [35:35] there might have been some level of [35:37] concern before [35:38] >> but I think increasingly a lot of you [35:40] know on both sides you know there's this [35:41] question of like you know whether [35:42] clinicians would push back against [35:44] patient use of AI and whether patients [35:46] would push back on clinician use of AI [35:48] in the way that you're describing but I [35:50] think both parties are actually finding [35:51] incredibly useful for their own personal [35:53] purposes. M [35:54] >> and so I think what we're finding in [35:55] practice especially over time is that [35:57] both parties might actually feel safer [35:59] with AI in their care journey whether [36:00] it's you know their clinician using it [36:02] or the patient using it as kind of like [36:04] a a sanity check and I think that that [36:06] attitude is definitely changing over [36:07] time but I wouldn't say that we're fully [36:08] there but that's my prediction of how [36:09] things go. [36:10] >> Okay. So then taking that one step [36:12] further, if a patient has done all their [36:15] research before a visit and they have [36:17] all their health data integrated and [36:18] they've asked a bunch of questions in [36:19] chatbt and then the clinician before the [36:22] visit has all automated through loops uh [36:25] has run the entire patient case through [36:28] uh through chatbt and then the visit [36:31] happens. What should they talk about? [36:34] >> Um my hope is that they talk about the [36:35] human aspect of healthcare. I mean there [36:37] there's still like a lot of uncertainty. [36:38] There's a lot of empathy. There's a lot [36:39] of bedside manner that goes into the [36:40] human aspects of healthcare. There's [36:42] also a lot a lot of physical interaction [36:44] that goes into it. I don't think you [36:46] know I don't think all the information [36:48] exists for the AI just information the [36:50] theoretically all the information exists [36:51] for the AI to just figure out what all [36:53] should be done and part of the reason [36:54] for a visit is to get more information [36:56] and do the process of history taking [36:58] investigation and things like this. And [37:00] so I think that will not go away [37:01] especially the the physical elements of [37:03] that. And I think even from a cognitive [37:05] perspective, the elements of of kind of [37:07] empathy and getting comfort and getting [37:08] that second check um I think will still [37:11] be something that people demand. [37:13] Can what do you think uh AI in general [37:16] or let's say dots would do in between [37:19] visits and will you think you'll cross [37:22] the line into providing providing care [37:24] in between visits and then share that [37:26] back with the doctor? [37:27] >> Yeah. No, I I I don't think we'll um I [37:29] think we'll we'll we'll definitely be [37:30] focused on health information, but I [37:32] think the whether it's dots or any other [37:34] product surface, I think we're really [37:36] excited about this kind of future of [37:37] heading more in this proactive direction [37:38] that you were outlining. Like imagine if [37:41] somebody is kind of dealing with a [37:42] chronic care journey and you know [37:43] they're again like this example of like [37:45] someone's A1 someone gets a lab test [37:47] back where their A1C is higher and [37:49] they're pre-diabetic or diabetic you [37:51] know can you flag for them like hey [37:53] maybe you should get a medication check [37:54] and go to your you know cardiologist or [37:57] whatever [37:58] >> and think about the right ways in which [37:59] you can um you know flag to users [38:03] medication adherence issues or other [38:05] kinds of issues. So I think those kinds [38:06] of issues are places where you know AI [38:09] can have potentially a really high [38:11] health impact especially like these kind [38:13] of chronic care journeys but I don't [38:15] think um I don't think it'll be we'll be [38:17] in the business of like managing all [38:18] these things by ourselves. [38:19] >> Sure. [38:20] >> Yeah. Another thing I'm curious about is [38:22] in the fields of AI applied to other [38:25] domains and situations like let's say [38:28] games like chess and go. Um AI as AI got [38:32] really good at chess and go they also [38:34] revealed other strategies and ways of [38:36] playing the game that even the best [38:38] experts in those domains were not [38:39] exploiting or not pursuing and that [38:42] helped us learn more about these games [38:44] and um also helped us learn more about [38:46] AI. Are there equivalents like that that [38:49] you've seen in AI applied to health from [38:51] the work that you've done uh in [38:53] research? Are there already findings [38:56] that are surprising you and perhaps [38:58] introducing new ways of delivering care [39:01] or understanding a clinical scenario uh [39:04] that can give us some optimism? Yeah, I [39:06] mean I think we've we've seen this at [39:08] smaller scales, but let let me let me [39:09] zoom out and say like, you know, one [39:11] thing that's been motivating our team in [39:13] general recently around raising the [39:14] ceiling of human health is this notion [39:16] of bringing to bear like kind of a move [39:19] 37 for for health. And this idea of a [39:22] move 37 is it comes from from Alph Go. [39:24] Um and uh a game that the Alpha Go [39:28] system which plays this kind of [39:30] complicated board game uh played with [39:32] Lisa Dole [39:33] >> where there was a specific move that [39:35] just kind of in the moment was seemed [39:37] like a a really dumb move and a [39:39] surprising move for all the human [39:40] experts watching this game [39:42] >> uh between this AI system and this world [39:44] expert and go but in hindsight is this [39:46] kind of like brilliant and insightful [39:48] move that humans are now just u [39:50] beginning to understand and learn from [39:52] the patterns of and things like this. [39:54] Yeah. [39:54] >> And this this is very interesting [39:55] example. Um I think so so this idea of [39:59] like how do you get to a move 37 in [40:00] health of a move or prediction where the [40:03] models predict something that a human [40:05] might not have been able to predict in [40:07] hindsight seems seems correct and [40:09] brilliant and insightful and impactful. [40:11] >> So I think I think we're very early and [40:12] I think you're going to see much more of [40:14] this in the future than in the past. But [40:16] um I think you know I mean one kind of [40:18] example is again thinking about these [40:19] the ways in which these systems have [40:20] been deployed as safety nets to [40:21] clinicians where clinicians are actually [40:23] delivering care and in the moment the AI [40:27] actually flags kind of like hey [40:28] something might be off here like like [40:30] our study with Panda Health and thinking [40:32] about the patterns of of results that we [40:33] saw there [40:34] >> as an example you know um in that [40:36] context uh the clinicians were a little [40:38] bit um prone to overuse of antibiotics [40:41] >> and a very common pattern that the the [40:43] models flagged is just kind of like [40:45] avoiding this overuse of of antibiotics [40:46] and thinking less about the local case [40:48] and more about the kind of global um [40:50] issue of resistance to antibiotics. So [40:52] that's like one, you know, specific [40:54] example, but but again, I think you're [40:55] going to see more of these examples of [40:57] models taking in like a longitunal [40:59] history of a user over um 10 years [41:01] across all their data, across their HR [41:03] data, across their chat data, all the [41:05] all these things and making predictions [41:06] that seemed impossible before cuz a [41:08] human can't even integrate all that data [41:10] in their head. [41:11] >> And making predictions about their [41:12] health and their risks and things like [41:14] this that seemed like they were [41:15] impossible. And I I think that would [41:16] rise to the occasion of of being called [41:18] the Move 37. [41:19] >> Yeah. Yeah, I mean there are there's so [41:21] many things that consumers are curious [41:23] about uh that um at least at one point [41:26] in time we weren't sure if it was [41:28] healthy for you and then the to your [41:30] point the Overton window changed and uh [41:33] and there was sort of more evidence that [41:35] was sort of conclusive in one direction. [41:37] You know, at one point people weren't [41:39] sure if smoking was uh bad for you. Is [41:41] coffee good or bad for you? Is sugar [41:43] good or bad for you? Uh sometimes the [41:46] views on some of these things have [41:47] changed and today consumers are they [41:50] want to know is creatine good for you? [41:52] Is melatonin good for you? Uh you know [41:54] all these sort of new emerging [41:56] phenomenons that perhaps are [41:58] underststudied or at least um uh you [42:01] know not widely uh sort of agreed upon [42:04] this safety and and effectiveness of [42:06] certain of certain supplements or [42:08] certain medications and things like [42:09] that. Can do you do you anticipate as [42:12] people are sort of entering this [42:13] information into chat GPT or other LLMs [42:17] that we'll have more evidence that we [42:19] can generate to study some of these [42:22] things that folks are curious about but [42:23] we don't really know uh or we don't have [42:25] conclusive evidence one way or the other [42:27] on on where they stand. [42:29] >> Yeah. I mean I think one of the most [42:30] interesting things about agents is they [42:33] kind of solve a lot of the bottlenecks [42:34] in analyzing data that already exists. [42:36] Right. There's various places across the [42:39] healthcare ecosystem where data is being [42:40] generated. EHR data, longitudinal [42:42] histories, claims, whatever [42:44] >> where data is being generated and it is [42:46] kind of just stored there and the [42:48] insights from it are not fully mined or [42:50] used to generate new knowledge. [42:52] >> Yeah. And so I think one of the [42:53] frontiers for how how health might [42:55] improve in the future and how the [42:56] ceiling of of health might might raise [42:58] will be thinking about the ways in which [43:00] we can leverage the data that is just [43:01] kind of out there in boxes in various [43:03] places and have models generate new [43:05] insights based on this data. So so I'm [43:07] also optimistic about that future. Um [43:09] I'm also optimistic about a future where [43:11] you know you mentioned these two [43:12] constituencies the the user the patient [43:15] and the clinician. I think I think there [43:17] is a third constituency which is the [43:18] scientists right which was helping kind [43:20] of advance the the frontier of knowledge [43:22] in the space and I'm also very [43:24] optimistic about a future in which you [43:26] know um AI helps accelerate a lot of the [43:28] bottlenecks in validation [43:30] >> which I think will end up being the [43:31] bottlenecks as opposed to kind of [43:32] proposing new ideas or treatments or [43:35] things like this um and so um very [43:37] optimistic about that as well. Switching [43:39] gears slightly, you've also shared [43:41] thoughts around um just the cost of [43:44] intelligence as an input to medical [43:46] intelligence coming way down and you [43:48] know at some point it'll be too cheap to [43:50] meter. Um I spent a lot of time and we [43:52] as a team spent a lot of time trying to [43:55] understand how these technologies will [43:56] make an impact on our healthcare system [43:58] that uh is inflating well ahead of uh of [44:03] inflation. uh 5.3 plus trillion we spend [44:07] on healthcare. If it were its own [44:09] country, it'd be the third largest GDP. [44:11] Um it's the biggest part of our economy. [44:13] It's also the biggest employer uh in our [44:15] economy. 23 million Americans work in [44:17] healthcare. There's so much um there's [44:20] just so much we spend on healthcare. [44:23] What do you what do you sort of make of [44:24] the work that you're doing and the [44:26] impact that it'll have on the cost of [44:28] care? I think this is like one of the [44:29] most important ways that we can pursue [44:30] our mission is to not only make our [44:32] models better at the frontier, but make [44:33] that widely accessible to everybody. So [44:35] that's, you know, obviously a big part [44:37] of how we've been able to deploy health [44:38] intelligence to a lot of people across [44:40] the world for free. I think it's also a [44:43] big part of how we think about, you [44:44] know, um API and how how good our models [44:46] are there for health and pricing there [44:48] and stuff like this. You know, one one [44:50] recent update was we recently released [44:53] uh GPD6 Luna [44:55] >> and GP6 Luna's um is our current [44:58] cheapest model on the API um or [45:00] cheapest, you know, frontier model on [45:01] the API. And what we found with GP6 Luna [45:04] was that it was actually doing better on [45:06] our benchmark Healthbench Professional, [45:08] our our current latest benchmark of of [45:10] kind of clinician workflows that are [45:12] like really difficult selected for [45:13] difficulty for our current models across [45:15] all the different ways in which [45:16] clinicians are using our models and and [45:18] the models um still could be could use [45:20] some improvement. [45:21] >> Mh. [45:21] >> Um it performed better than 5.6 Soul, [45:24] which was literally our best model like [45:26] a month ago. And so and it's it performs [45:28] better despite being 34 times cheaper. [45:30] Um, and we've had a couple of those [45:31] moments even even in the past year. And [45:34] you know, there's been many trends [45:35] showing that AI compared to other [45:37] technologies that have been introduced [45:38] in the past has been reducing a cost at [45:40] a given level of kind of capability or [45:42] usefulness faster than many of those [45:44] other technologies. And so one bet we've [45:46] had at OpenAI is that the cost of health [45:48] intelligence will will plummet at a [45:50] given level of health intelligence. [45:52] >> And the the [clears throat] level of [45:53] health intelligence where you know [45:54] clinicians and patients feel like they [45:56] can trust in the care journey is [45:58] starting to get there, right? So that's [45:59] like a fixed level of intelligence in [46:01] some sense. Hopefully the bar for that [46:02] raises over time as well. Um my [46:05] expectation is once we get close to that [46:07] level of intelligence the cost for that [46:09] level of intelligence also dramatically [46:10] reduces like it has been as the hope [46:12] there is that we get like a kind of this [46:14] commoditization of health intelligence [46:16] right we don't want to hoard that health [46:17] intelligence we wanted to everybody to [46:18] use it [46:19] >> so my hope that that contributes but you [46:21] know to answer your question about the [46:23] whole of healthcare spending I don't [46:25] think all of that is about health [46:26] intelligence I don't think like health [46:28] intelligence has had yet a macro factor [46:30] on the a macro effect on the problem [46:31] that you're describing [46:33] >> if you look at like the whole spend of [46:35] health. There is quite a bit of areas [46:36] though where if you look at for example [46:38] chronic care management or documentation [46:40] or administration or a lot of kind of [46:42] administrative workflows of various [46:43] kinds, back office workflows, there's a [46:46] lot of money spent in ways that probably [46:48] don't need to be spent that way. Um and [46:50] so we are very optimistic about kind of [46:52] an overall compression of spend over the [46:54] over the coming years. I don't think [46:55] that's happened yet and I don't think [46:56] that just happens from making health [46:58] intelligence abundant. I think that [47:00] happens from integration and workflows [47:01] and deployment and scaling that up and [47:04] really making sure these models are [47:05] valuable in the economically valuable [47:07] tasks that that people are using them [47:08] for. But I think it's a a prerequisite [47:10] to making that future happen. [47:11] >> More care, less paperwork is sort of [47:14] where we're heading. At least there's a [47:15] lot of the cost savings. [47:17] I'm sure you saw some of the headlines [47:18] over the last few weeks of uh there's a [47:20] New York Times article uh that basically [47:23] provider AIS and payer AIs are fighting [47:26] and there was some Blue Cross Blue [47:28] Shield Association analysis that showed [47:30] that uh the respective billing bots uh [47:33] and paperwork bots effectively uh [47:35] increased spend by a billion dollars or [47:37] something like that. when you are doing [47:40] cutting edge research on health and [47:43] trying to bring down the cost of medical [47:44] intelligence to near zero and then you [47:47] see headlines like that, what do you [47:48] make of that? [47:49] >> I I think this might be one of those [47:50] areas where there might be some kind of [47:52] short-term pain around this kind of [47:53] provider payer arms race, but in the [47:56] long run there's no reason this has to [47:57] exist, right? And so [47:59] >> re really I think you know you're [48:01] entering a world where a lot of visits [48:03] actually have some kind of ambient [48:04] intelligence in the room listening, [48:06] transcribing things like this. um there [48:09] is notes, there's clinical notes, [48:10] there's AI adjudicate AI details of of [48:13] clinical notes and things like this. And [48:15] so you have a lot of the details you [48:16] need to adjudicate visits, right? And so [48:18] the ideal world in my view is a world in [48:20] which providers and payers kind of have [48:22] a handshake like, hey, here's our ground [48:25] truth of what happened in this visit. [48:26] Like here's all the information we have. [48:27] There's just like no other source of [48:28] information that there is, [48:29] >> right? [48:30] >> And then an AI helps adjudicate that [48:32] situation. And maybe there's like some [48:34] limited appeals process or things like [48:35] this, but I don't think you need to have [48:37] all this process where where you have [48:38] like an AI arms race between payers and [48:40] providers. I think they can just be [48:42] completely avoided. So I I view that as [48:43] a a bit of a short-term distraction. No, [48:46] there's a lot of entrepreneurs that are [48:48] building technology and healthcare that [48:50] are that are likely watching this and uh [48:53] they're probably trying to figure out uh [48:56] you know as you continue to advance your [48:58] work and you bring down the cost of [48:59] medical intelligence to near zero and [49:02] this becomes this sort of ubiquitous [49:03] utility that flows through all of our [49:05] healthcare interactions for consumers, [49:07] for clinicians, for the administrative [49:09] part of of the healthcare technology or [49:11] the healthcare stack. Um, what should [49:14] folks be building on top? What should [49:16] specialist applications be doing [49:19] assuming that this gets 100x plus [49:22] better? [49:23] >> Yeah. Um, it's a great question. Like I [49:25] mentioned, like we think about a lot of [49:27] our health impact through OpenAI as not [49:29] being directly through the products that [49:30] we offer, but actually through our [49:31] partners and and that last mile of [49:33] delivering care. [49:34] >> And so, you know, what I'd say to to our [49:36] partners is, you know, build on top of [49:37] models that are really capable. And the [49:39] most important thing is bet that those [49:40] models will become more capable. So bet [49:42] on the future. I think many people in [49:44] healthcare are on some kind of spectrum [49:46] between betting against the models, [49:47] betting neutrally for the models and [49:49] betting uh with the models or betting [49:51] for the models. And I think um I would [49:54] say you know bet on the models getting [49:55] better. That's probably the most [49:56] important thing. And think about the [49:58] ways in which a lot of the technology [50:00] that's being developed whether that's [50:01] you know agents or computer use or other [50:03] kind of frontier tech will accelerate [50:06] healthcare and and bet on that and build [50:08] towards that future. I also think a lot [50:10] of the medical care um will actually [50:12] like um become more accessible to [50:14] patients with AI. You know, we talked [50:15] about this idea of kind of a quarterback [50:17] and dissolving a lot of these barriers. [50:19] And so just being focused on delivering [50:21] really good quality care um and having [50:23] AI be part of, you know, monitoring that [50:25] care journey, I think is a very [50:26] important part of that as well. [50:28] >> Zooming out to the overall healthare [50:29] system and now just looking at it at [50:31] large, you know, we talked about this [50:32] 5.3 trillion. A lot of that spend is [50:36] paperwork between providers and payers [50:38] and other stakeholders within the five [50:40] Ps of healthcare. A lot of that is [50:42] cognitive care that you're arguing can [50:44] be augmented and um enhanced and sort of [50:48] we can have bring on a lot more [50:50] cognitive care capacity uh online which [50:53] is exciting and exciting from an access [50:55] perspective. And then there's a lot of [50:56] spend in healthcare that is procedural [50:59] either requiring some imaging, some [51:01] blood work, some follow-on examination [51:03] or it's some surgical intervention. Um, [51:07] where do you see sort of the [51:09] intelligence work that you're doing in [51:11] the medical space impacting procedural [51:13] care? And is that going to be the [51:15] ultimate is that going to be the new [51:16] sort of or an enhanced uh or perhaps an [51:19] exacerbated bottleneck or do you expect [51:22] to do a lot of work there as well? [51:23] Robotics etc. Yeah, I mean I I think in [51:26] the short term with procedural care, I [51:28] think you want a better targeting when [51:29] that's needed, right? And I think AI can [51:31] help, you know, provide information that [51:33] is helpful for knowing when to seek that [51:35] kind of care as well. [51:36] >> Probably the the biggest macro factor in [51:38] the next, you know, year or two will [51:40] just be better kind of um proactive [51:43] preventative stuff, right? So you don't [51:44] have to actually go in later and get [51:46] that procedure right because if you make [51:47] the right moves sooner and AI helps you [51:50] navigate that kind of journey then you [51:52] can hopefully avoid a lot of the [51:53] downstream cost of the most expensive [51:54] procedures. So that that might be like [51:55] one macro factor to kind of look out for [51:57] is my guess. [51:58] >> I think the question of kind of robotics [52:00] and care I think is a a bit of a longer [52:02] term one both from a technology [52:03] perspective and kind of like a [52:05] reimbursement and kind of industry um [52:07] integration perspective. My expectation [52:10] is that it's not like a next few years [52:12] kind of thing. Very cool. Okay, we've [52:15] covered a lot of ground. We've talked [52:16] about intelligence improving. We've [52:18] talked about trust. We've talked about [52:19] moving from medical information to [52:21] medical care. Help us understand what we [52:24] can look forward to as things uh as sort [52:27] of things continue to unfold. Um as [52:29] agents and personal agents and the [52:32] underlying models become increasingly [52:34] performant, what can we look forward to [52:36] from chatb health? [52:38] >> Yeah, I mean I think there's like really [52:39] two highle things that we're really [52:40] focused on making happen. One is again [52:43] thinking about like the ways in which we [52:45] can raise the floor of human health and [52:46] make trustworthy reliable health [52:48] intelligence abundant for all clinicians [52:50] and all users and patients. And so [52:52] getting to the point where you know like [52:54] we talked about like all patients and [52:56] clinicians just feel safer by having AI [52:58] in the care journey like when you see [52:59] the the chatpt you actually when the [53:01] doctor's using chach you actually feel [53:03] safer and vice versa. getting to that [53:05] point and again that's that's a question [53:06] of not just you know the models being [53:08] good and and eval being good but also [53:10] how we deployed and the activation [53:11] energy around that and um the products [53:14] and things like this um I think a second [53:16] thing is really thinking about um [53:18] raising the ceiling so what are the ways [53:19] in which AI can actually do things that [53:21] were not possible before so one of them [53:23] actually I think is this kind of like [53:25] you know proactive health proactive [53:26] personal health assistant that is part [53:28] of your care journey is helping you [53:29] navigate these moments being your [53:31] quarterback you know helping you [53:32] schedule and all these things but also [53:34] flagging the places where you know you [53:36] can intervene in your health today and [53:37] prevent a lot of know expensive [53:39] deterioration later in the future. Um [53:42] and I think that's going to be an [53:43] important part of how we think about you [53:44] know improving the future of of health [53:46] for for people as many people as [53:48] possible. And I think the second thing [53:50] is just what you mentioned before which [53:51] is like really thinking about the ways [53:52] in which we can unlock the value of [53:54] existing data that is sometimes stored [53:56] in boxes of various kinds and thinking [53:59] about the ways in which that can be [54:00] leveraged to produce new insights that [54:01] benefit patients. It is striking and [54:04] incredible and humbling to realize that [54:08] chat GPT I think came out three years [54:10] ago roughly. Uh and so it's been a lot [54:12] of progress uh in a very very short [54:15] period of time. So to even talk on a [54:17] five or a 10-year basis feels academic [54:20] but can you give us some predictions? I [54:21] mean is there a feature that we might [54:24] have that feels science fiction today uh [54:26] but we could have one day 3 5 10 years [54:28] from now? Yeah, I mean is it is super [54:31] super hard to predict the future these [54:32] days. So um um this is a difficult one, [54:35] but what I would say is like I think the [54:38] thing that might have the most impact on [54:39] human health in in that kind of time [54:41] frame in the next 5 to 10 years is [54:43] probably going to be newer treatments [54:44] that come out of maybe two things. One [54:47] is um AIG guided um generation of new [54:51] targets or molecules that that that [54:53] could be interesting for you know for [54:55] new treatments and the second is AI [54:57] guided prioritization and validation. So [54:59] thinking about the right ways, we're [55:00] already at a point where the number of [55:03] molecules and drugs that people want to [55:05] suggest to be used in in clinical trials [55:07] and things like this is way more than [55:08] the number of slots available in [55:10] clinical trials. So thinking about the [55:11] ways in which models can help prioritize [55:13] and run these clinical trials so that [55:15] they just run smoother and faster. My [55:17] guess is in in that kind of time frame, [55:19] you're you're talking about kind of like [55:21] improved medicines, more personalized [55:23] medicines for people, things like this. [55:25] And that will only be possible because [55:26] you invested in those two pillars. That [55:29] is awesome. Uh well, we've covered a lot [55:31] of ground. I'm leaving this conversation [55:33] even more optimistic than I was before. [55:35] Uh so Karen, thank you so much for [55:37] coming on the A16Z podcast. [55:39] >> Thank you for having me.