AI in Healthcare Series: When AI Stops Being a Project

医疗健康AI系列：当AI不再只是一个项目 企业内部AI应用正从对话式问答转向长时间运行的自主智能体任务，部分工作流已能承担耗时数小时的人工作业。例如UnitedHealth Group已大规模部署模型与跨领域智能体，并通过严格的月度高管审查机制全面评估token成本、投资回报率与临床成效，以防止AI效益脱离实际财务业绩；与此同时，具备成本优势的前沿开放权重模型虽引发关注，但其依赖消耗数倍token的测试时计算来实现同等性能，对基础设施提出了巨大要求。 企业内部应用正从对话式问答转向自主、长时间运行的智能体任务；OpenAI的数据显示，25%的企业智能体工作流所涉及的工作，通常需要人类连续投入8小时或更长时间才能完成。 (2:26) UnitedHealth Group的内部平台United AI Studio管理着117个模型和92个跨领域智能体，在严格的企业支出限额下，每天处理100亿至250亿个token。 (11:21) 像GLM-5.2这样的前沿开放权重模型凭借显著的成本优势在企业界引发巨大反响，但它们是通过测试时计算消耗5到10倍的token来实现同等性能的，需要庞大的基础设施支持。 (13:28) 为避免出现“除了财务底线以外到处都是AI”的局面，UnitedHealth Group每月强制召集所有业务部门的CEO、CFO、CIO、CTO和CMO进行长达6小时的AI质询，以审查token成本、ROI及临床结果。 (29:16)

UnitedHealth Group's Sandeep Dadlani and hosts Matt Lungren and Justin Norden argue that health systems must move artificial intelligence past isolated pilot projects into core enterprise infrastructure, projecting a two-year window before AI shifts from an exploratory experiment to an organization's primary operating model. The Shift to Long-Horizon Autonomous Agents (2:26) Matt Lungren points to OpenAI research showing that roughly 25% of evaluated agent tasks represent knowledge work previously estimated to take a human worker eight continuous hours or more to finish. This development moves generative AI beyond transactional, single-turn search queries toward sustained, multi-step execution across complex domains such as corporate finance, recruiting, and legal compliance. However, Lungren observes that health systems remain fundamentally unprepared for this transition because hospital operations and administrative workflows are historically structured around compartmentalized, iterative human handoffs. Sandeep Dadlani warns that enterprise adoption is hampered by widespread 'agent washing,' where engineering teams rebrand basic APIs as agents despite lacking autonomous capabilities to reason, decide, and act. He contends that confining automation to narrow, defensive departmental silos prevents systemic progress. To unlock meaningful administrative and clinical value, health systems must establish enterprise-wide governance and security frameworks that permit autonomous agents to traverse entire operational journeys from intake through reimbursement. Managing Token Economics Through Modular Architecture (11:21) Autonomous agent workflows introduce steep financial and technical risks, as iterative planning and tool use consume large volumes of tokens without predictable expenditure ceilings. In addition, rapid frontier model releases and open-source models like GLM-5.2 create acute vendor-lock-in risks for health systems that hardcode workflows to single proprietary APIs. To control these variables, UnitedHealth Group built the United AI Studio, a centralized harness hosting 117 language models—including approximately 20 internal, purpose-built open-weight small language models—and orchestrating 92 cross-domain agents across payer, provider, and pharmacy operations. The studio incorporates an enterprise token gateway to monitor and enforce consumption limits across an organization-wide throughput of 10 billion to 25 billion tokens per day. Dadlani explains that Optum avoids training foundation models from scratch on proprietary clinical data, relying instead on modular architecture where models can be swapped out as frontier capabilities evolve. This strategy reflects Rich Sutton's 'bitter lesson,' acknowledging that generalized foundation models frequently eclipse specialized domain models, making it more advantageous to focus engineering on workflow context, tool integration, and user experience. Model Commoditization Versus Operational Legacy Realities (22:06) Lungren and Justin Norden observe that base foundation models are rapidly commoditizing, shifting competitive advantage toward proprietary context, reinforcement learning, and custom workflow harnesses built above the model layer. Furthermore, an 18-month academic peer-review cycle often means published healthcare AI studies evaluate models that are already obsolete, widening the gap between theoretical benchmarks and practical clinical implementation. Dadlani contrasts industry enthusiasm for frontier architectures with the entrenched realities of healthcare administration, emphasizing that the United States healthcare sector still processes an estimated 9 billion faxes annually. Within UnitedHealth Group, operational teams continue using legacy optical character recognition (OCR) systems alongside modern multimodal systems because OCR remains fast, inexpensive, and deeply integrated into document processing pipelines. Bridging this gradient requires deploying advanced automation without prematurely abandoning reliable, cost-effective infrastructure. Frontline Immersion and Mandatory Executive Accountability (29:16) To counter the reality where healthcare chief financial officers report seeing 'AI everywhere but the bottom line,' Dadlani champions a 'zero distance' operating model. By removing organizational consensus layers and embedding cross-functional engineering teams directly into frontline operational settings—such as customer support call centers—teams can identify root issues and rapidly build functional prototypes in minutes using modern code-generation harnesses. To ensure AI initiatives deliver measurable bottom-line value, UnitedHealth Group conducts a mandatory monthly six-hour review bringing together 50 top executives, including business unit CEOs, CIOs, CTOs, CFOs, and Chief Medical Officers. This committee scrutinizes hard operational metrics, including token expenditure, Net Promoter Scores, financial return on investment, and clinical quality indicators. Dadlani and the hosts stress that without this cadenced, executive-level financial and clinical scrutiny, AI remains an unmeasured IT novelty rather than a driver of core enterprise performance. The discussion concludes that sustainable healthcare transformation requires uniting top-down executive accountability with frontline development, using an 'inside out' model that scales internally proven tools (such as Optum Real, Crimson, and Digital Prime) to the broader market. A critical unresolved challenge remains whether smaller health systems lacking UnitedHealth Group's massive engineering footprint and capital can build similar architectural harnesses to manage token volatility and avoid proprietary platform lock-in.

[5:10] Sandeep Dadlani: Everything was an agent. Even an API was called an agent because it's so cool to call anything an agent. And oh, if you create a super API, it became a super agent. And it was all wrong. An agent needs to have, as the name indicates, agency to reason, to decide, to act. [7:07] Justin Norden: We see AI everywhere but the bottom line. [9:10] Sandeep Dadlani: I feel I'm like one weekend Twitter read away from again feeling obsolescent, completely irrelevant. [11:21] Sandeep Dadlani: Dude, you're done. You can't mess around because you're not really moving things to production. You're just messing around. [13:28] Matt Lungren: unless you've got a garage full of Mac minis. [25:14] Sandeep Dadlani: We were down. Like, we were like, OK, if these guys came up with this, what the heck. [29:16] Sandeep Dadlani: I want some of my life back... these are draining meetings, [29:16] Sandeep Dadlani: injects paranoia through the organization. [29:16] Sandeep Dadlani: In two years, either AI will be a project, or AI will be the business. [34:47] Sandeep Dadlani: could have danced all night,

[0:11] Welcome back to the Stanford Healthcare AI Podcast. [0:15] We're so thrilled to have Sandeep Dadlani, the CEO [0:18] of Optum Insight, who really needs no introduction, who's [0:23] been leading, talking a lot about what [0:24] they're doing in AI kind of across the country for health. [0:27] So Sandeep, welcome. [0:29] Thank you for having me. [0:30] Justin, Matt, it's so cool to hang with the cool kids, [0:33] finally. [0:34] So thank you. [0:35] That is very generous and untrue given us nerds. [0:41] But you were just starting to say something fun you were [0:44] doing with AI on Father's Day. [0:46] So-- [0:46] So on Father's Day-- [0:47] --elaborate more. [0:49] On Father's Day, my daughter's 23 and 19 [0:52] have a tradition that I buy them gifts. [0:56] Figure that. [0:57] Anyway, so we decide we haven't bought anything yet. [1:01] So we go to the mall. [1:03] Like who goes to the mall nowadays. [1:04] But anyway, we hang-- we go to the mall [1:06] and there's really nothing there. [1:08] There's very few SKUs. [1:10] There's very few depth of variety. [1:13] And as you can imagine, I've not been in the mall [1:15] for a long time. [1:17] But they find some things which are interesting. [1:20] But of course, it's not in the right size. [1:22] It's not in the right color, it's not in the right fit. [1:24] And what amazes me right now is you take a picture, [1:29] I just upload it to Gemini at that point, [1:31] and then Gemini just figured out exactly the color, size, fit, [1:37] and in-stock availability. [1:39] It did the work for us, and where it was available [1:43] online with which brand. [1:46] And we had a setup that it ordered it. [1:50] I've been in retail consumer most of my life, [1:52] and I'm just I was just amazed at how much work it did for me [1:58] and help save Father's Day. [1:59] So that is a recent instance which I really enjoyed. [2:02] So does that make sense? [2:04] Absolutely, and it's a perfect example also, [2:08] which one of the things Matt and I [2:11] have done continuing on this podcast is just [2:13] try to keep up with what's happening, [2:14] how is usage changing, and what you just mentioned, [2:19] just how much work it did for me. [2:21] I think this is actually one of the most fascinating things [2:23] that's come up in the last week. [2:26] And so actually, there is a new paper. [2:29] OpenAI released some of their internal data [2:32] on how the work has changed. [2:35] Because most of the world-- [2:37] OpenAI talks about a billion users [2:39] using ChatGPT thinks about this as a Google replacement. [2:43] Hey, give me back my information. [2:46] And what has changed that they showed, and this [2:49] was their own internal data, is how not just [2:51] engineering and coding replacement has worked, [2:54] but across finance, recruiting, and legal, [2:57] agents are now doing the work and just pulling that [3:00] so much farther, like you mentioned in that shopping [3:03] example. [3:04] And I was just with a health system board last week, [3:09] this is something that we're not yet ready for in health. [3:13] We've done question answering, a little bit of turns. [3:17] But really doing the work is somewhere that it's hard. [3:23] The whole world is starting to comprehend what this looks like. [3:26] And it'll start in consumer and shopping examples. [3:29] But this is where it's starting to change. [3:31] I mean, my point though, I would say your example is great [3:35] because it does start to-- and this paper teases that out a bit [3:38] because you're really now talking about again, [3:40] not just a Q&A, which is quick, short, transactional, [3:45] which I would still argue is significantly helpful. [3:48] But now we're talking about longer time horizon [3:51] chains of tasks us together. [3:53] And in this paper, they even had I think it was like 25% of them, [3:57] like the tasks that the agents were doing for OpenAI use cases. [4:01] Again, not in engineering. [4:03] We're over eight hours, or at least [4:04] that was the estimates of the human that was saying, [4:07] how long would this really take you to do About eight hours. [4:09] I mean, I don't know how long it would [4:11] take you to do all the sourcing to find the right-- [4:13] I mean, it's an insane amount of work, [4:15] even if you're super facile with search tools or whatever. [4:19] But again, this is kind of like that overhang that we always [4:22] are bringing up, which is like, again, we [4:24] used to say with GPT-4, if you paused it there, [4:27] there's still plenty of things that it can actually [4:30] operationalize and make my life better. [4:32] But we're so far, like I'm running out of-- [4:34] I don't even know if I have an eight hour [4:36] task, an eight hour continuous task to give an agent right now. [4:40] I would struggle to come up with one. [4:42] And that's the challenge I think we're posing to health systems [4:46] and enterprises is like, we're going [4:48] to have to almost break the bone to reform it [4:51] because our systems just aren't built [4:53] on sending a bunch of folks out to do eight hour tasks [4:56] and then returning. [4:57] It's much more iterative than that. [4:59] How do we break up the work or silo up the work [5:02] so we can actually take advantage [5:04] of the full capabilities of where the frontier is? [5:06] I actually don't have a great answer right now. [5:10] No, and look, I mean, we've grappled with this [5:13] inside UnitedHealth Group. [5:15] Now, this is I'd say, for large enterprises, a reasonably [5:18] sophisticated shop. [5:19] You've got about 22,000 engineers who can play around [5:23] with LLMs and agents and of course, do the job. [5:28] But when it comes to long form agentic work, [5:34] I don't think we are there in terms [5:36] of the governance, the guardrails, the security, [5:40] and frankly, most importantly, the imagination. [5:43] So I was just reviewing one of our businesses [5:46] last week where of course, they're [5:48] showing off the latest AI stories [5:51] and everything was an agent. [5:52] Even an API was called an agent because it's so cool [5:55] to call anything an agent. [5:56] And oh, if you create a super API, it became a super agent. [6:01] And it was all wrong. [6:02] I mean, this is the eagerness of people [6:05] to impress and show agent. [6:08] An agent needs to have, as the name indicates, [6:11] agency to reason, to decide, to act. [6:18] And then right now, I see agents in the enterprise, [6:22] in silos where we are comfortable, [6:25] where there's domain context, sort of integrity, secularity. [6:32] Except that we won't reimagine healthcare, [6:34] if we just have agents in silos in an end to end process. [6:38] So the opportunity is there, and it's apparent to us. [6:43] And we are building the guardrails, the governance, [6:46] the gateways to do that. [6:50] Ironically, I think the rest of health, [6:51] I feel this represents the opportunity of a lifetime [6:57] because there's so much that's possible now. [6:59] And Justin, you're probably deep into health systems as well. [7:01] You see this all the time, right. [7:03] The opportunity is right there in front of us. [7:07] Yes, and I think you mentioned a few things that are fascinating, [7:10] which is the siloed versus not siloed aspect [7:17] to go solve a problem. [7:18] I think the knee jerk reaction is to go after the silo. [7:23] Hey, an agent's a task, an automation [7:27] can fix this one thing in this one task. [7:30] But we haven't done the work to reimagine, like, [7:32] why do we do this task. [7:34] This is actually one task in a much, much larger end [7:38] to end flow to whether that's getting through a system [7:41] to get paid, getting a patient to the right spot. [7:43] And there's laser focus right in front of the work at hand. [7:49] And the re-imagination to say, what does this workflow, [7:52] how do we do it, and how do we reimagine [7:54] it, given the new state of technologies, [7:56] is what is most impactful. [7:59] And I think most exciting, but also hardest to do. [8:01] And I think an interesting conversation [8:03] I have with health systems-- there's [8:05] a recent conference where people are talking about, [8:09] hey, we're just going to make the investment. [8:12] We don't exactly know where it's going to go, [8:14] but we're going to hope for the best. [8:18] We're going after a few point solutions, and it just-- [8:23] it makes sense because it's hard to rethink [8:27] through the many people involved in a certain task. [8:33] And then there's this incrementalism, [8:37] which we talk a lot about where just one thing after another, [8:40] but they don't really add up. [8:41] Some CFOs will talk about we see AI everywhere [8:44] but the bottom line. [8:45] That's a refrain I've heard from a few times in health systems. [8:48] And it's because we're not asking the hardest questions [8:50] for what is the metric, do we know it to begin with, [8:54] and then how do we work to change it at scale. [8:57] And that's the thing I face over and over again. [9:00] And this is a people challenge candidly, more [9:04] than it is a technical challenge today. [9:06] And this is what's coming for healthcare. [9:08] But I-- [9:08] The tools are getting better and better. [9:10] I mean, but on that point, though, when I hear agent [9:13] and I put my CFO or enterprise hat on, I'm thinking, [9:16] oh, this is going to be expensive because to me, [9:18] I know you talked about you have to have agency, but agency to me [9:21] means blowing through tokens, going off and doing something. [9:25] And we're not really even sure the efficiency [9:28] of that token use for a given. [9:29] So there's that part of it. [9:31] And on the other side, when you talk about silos [9:33] and you talk about enterprise level engagements, [9:35] now we're having real conversations about, [9:37] hold on a second, I need to have more agency as an enterprise [9:41] over both my spend, which we can get into, [9:43] and that's top of mind, I think, but then the other part [9:46] is of the model. [9:47] Like, I'm starting to feel like there's a backlash right [9:50] now against some of the frontier labs in the use [9:54] locking in with one model and being beholden, [9:57] if I've restructured my entire enterprise or workflows [10:00] or systems around one provider or oneAPI. [10:04] Now and then I see a GLM-5.2, or I see so and so, or let's say [10:08] that there's an issue with the models, [10:11] security risk and that gets pulled from me like that is [10:13] really concerning because now I'm really worried about cost. [10:17] And now I'm also worried about the [10:19] not being able to keep my finger on the pulse of the race [10:22] and the geopolitical and all the things. [10:24] It's starting to get very complicated. [10:25] But underneath all that, 100% agree, [10:29] I think across the board everyone [10:30] would say this is transformative for my business, [10:33] for my patients, for my health system without question. [10:36] How do I actually put myself in a position [10:40] to take advantage of it without losing or falling behind, [10:42] or getting stuck or locked in? [10:44] That is not, to me, solved at all right now. [10:48] I feel I'm like one weekend Twitter read away [10:52] from again feeling obsolescent, completely irrelevant. [10:58] What we've done so far, which I think you'll find fascinating, [11:01] is I think my first three years as CTO at UHG, [11:06] we just went absolutely berserk with embedding these AI use [11:12] cases and workflows. [11:14] Prompt engineering was a thing. [11:15] Generative AI and LLMs were a thing. [11:18] The only thing I feel we got lucky on [11:21] is we created this thing called a harness, a United AI Studio. [11:27] And now this year, I feel like there [11:30] are people questioning, saying, why [11:31] have you created this studio. [11:32] This year, I feel a little more vindicated. [11:35] I'm sure next year I'll feel embarrassed about something, [11:37] but this year, I feel vindicated. [11:39] It has 117 LLM models and about 20 of them [11:44] are internal, open-weight, SLM models [11:48] that we built for purpose. [11:49] Others are commercial. [11:52] It has a token gateway, which we hurriedly put together as soon [11:56] as we started seeing headlines of other companies being [11:58] embarrassed by token costs. [12:00] Now, please. [12:01] I read the headlines tomorrow if one of our guys [12:05] just blew up the tokens. [12:06] But right now we consume between depending on the day $10 to $25 [12:09] billion tokens a day, a billion with a B. [12:13] And my personal token limit is $1,000. [12:17] At 1,000, the CTO of the enterprise called me and says, [12:20] dude, you're done. [12:22] You can't mess around because you're not really [12:24] moving things to production. [12:25] You're just messing around. [12:27] Sure. [12:29] And there are about 92 agents that we have registered [12:35] that work cross domain. [12:37] So within a domain, I can have many agents, coding agents, [12:39] this agent but when, you know, UnitedHealth Group, [12:42] payer provider, pharmacy, et cetera. [12:46] Domains that can work across agent [12:49] with responsible secure data sources, 92. [12:52] And we have to register them so that we can monitor them [12:55] in a post sort of scenario. [12:58] Having said that, I've now read about GLM-5.2 and I'm like, OK, [13:03] this looks cool. [13:05] And so maybe this whole thing is going to change all over again. [13:08] So every weekend, we really walk into Monday's thinking, [13:15] should we change the world again. [13:16] We designed it to perfection. [13:18] We were proud on Friday. [13:20] And then we are back to paranoid and behind and irrelevant [13:24] on Monday. [13:25] Does that make sense? [13:26] Do you feel that way? [13:27] Absolutely. [13:28] All the time. [13:28] Because we've mentioned it so many times, I just pulled up, [13:32] so GLM-5.2, to your point, kind of blew up Twitter or X recently [13:37] on really from different benchmarks coming in achieving [13:41] the last frontier model performance, [13:44] not the new fable mythos coming out and then un-coming out as is [13:50] happening, but that prior version and specifically [13:52] in coding where the token costs have blown up. [13:56] As you mentioned, there's lots of headlines around costs [13:58] and public companies pulling back on usage [14:01] and restricting usage. [14:03] But then GLM-5.2, an open source Chinese based model that is far [14:08] cheaper. [14:09] And people can even run on their own infrastructure. [14:13] And so to your point, this has uprooted a lot of the frontier [14:19] conversations on spend. [14:20] And a separate study was talking about how actually open router, [14:24] a company that looks at different ways for people [14:28] to access what models, the majority of models now for US [14:31] based companies through open router [14:33] is now Chinese open source models. [14:35] And so it is changing. [14:37] Do you see large-- what's the largest-- [14:39] and you don't have to name the enterprise. [14:41] What's the largest enterprise that's using GLM-5.2 today, [14:46] if you know of any? [14:48] Matt, I don't know if you have one. [14:49] There are many, many people, if you look on who are testing, [14:54] I don't know if there's a large institution to your point that-- [14:56] I don't think there's anyone-- [14:57] --that said, hey, we've switched-- [14:58] Name it. [14:59] --everything we're running, But everyone should be testing it. [15:03] Yes, well I-- [15:03] Right. [15:04] Right, you should know your use cases, [15:06] which it sounds like, Sandeep, you guys are like really-- [15:08] if you're paranoid, you're doing something right [15:10] would be my response to that comment, by the way. [15:12] But on the GLM stuff, like I have not seen to be fair, just [15:18] some sobering aspects of this. [15:20] So it is an amazing model. [15:22] It is a 4.8. [15:22] I think most people would agree in terms of quality. [15:25] Those who've tested it mainly for engineering [15:27] and other more basic tasks. [15:29] But I wouldn't say that-- we haven't [15:31] seen that health, benchmarks. [15:34] It's not blowing those away to our knowledge yet. [15:36] But the other thing is interesting. [15:38] This is going back to something that Noam Brown started a thread [15:41] on, who's the CTO of OpenAI, where [15:43] one of the lead scientists at OpenAI, where he said, [15:47] listen, we don't have benchmarks that can really [15:50] show the upper bounds, because if you just give it [15:53] more tokens, like it keeps-- there doesn't [15:57] seem to be an upper limit. [15:58] So with that in mind, if you look at the GLM release, [16:02] again, it gets to those frontier levels. [16:04] But it's using 5 to 10x the number of tokens to achieve [16:08] the same outcome as a 4.8, for example. [16:12] So yes, they're cheaper, they're more cost efficient. [16:15] I would almost argue that this would not be [16:18] one you can run at your home. [16:20] It's a 750-plus billion parameter model. [16:22] Unless you've got a garage full of Mac minis or something. [16:27] And I'm not sure how you're running this on your own prem. [16:30] But in general, though, I think it's the higher level discourse [16:34] which is around how much do we empower the community to take [16:40] the model, shape it, use it for what we want [16:42] and almost have ownership over it versus writing [16:45] the frontier with the expectation [16:47] that the partner that I've chosen [16:50] or the model that I've chosen, or in your case, [16:53] do we put a harness around it to make those decisions, [16:56] maybe more on the fly more intelligently for a given use [16:59] case. [16:59] I don't have a great answer, but what [17:01] I'm starting to come to the conclusion [17:03] is our benchmarks aren't great publicly [17:05] to really inform a lot of our decision making [17:07] and the work that sounds like you're doing [17:09] and I think I know Justin's doing, which [17:12] if I know my own use cases super well, [17:14] I can almost tell within a couple of days [17:17] with a new release. [17:18] Hey, this is going to be better. [17:20] I'm going to use this or not. [17:21] Like, do you know what I mean? [17:22] Like, there's almost that-- [17:23] your AI literacy is high enough that you [17:26] can make some of those decisions around your own business use [17:28] cases. [17:29] Yeah, and Sandeep, I want to come back to you in a second [17:34] and talk about the models. [17:35] Also very publicly, you've mentioned spending [17:38] a lot of money this year. [17:41] Candidly, I've used that I think $1.5 billion line. [17:44] But please correct it, to scare some of our health system [17:47] partners of hey, you should think about investing more here. [17:51] But before we do that because we're talking about models, [17:55] I feel like we'd be remiss if we didn't bring up this paper. [17:58] I feel like that has blown up at least the medical AI [18:02] sphere on performance. [18:05] And so this was a study done by a number of researchers [18:08] that looked at general purpose, large language models [18:11] compared to some of these specialized clinical AI tools, [18:15] as Matt said. [18:16] And there's been a lot of discussion on the benchmarks [18:18] and what is performing better or not. [18:20] The short version being these generalized models, again, [18:24] a little bit out of date, if you look, this did clot, Opus 4, 6, [18:28] GPT-5.2, Gemini 3.1, but showed that they are actually [18:31] performing better than OpenEvidence and updates expert [18:34] AI models. [18:36] People are talking about the bitter lesson [18:38] on trying to encode all this specific medical expertise. [18:41] Matt worked so hard to learn as a sub subspecialist, [18:45] but if you give the models enough data and time [18:47] and compute that they start to do better. [18:49] And so I guess we talked about model performance. [18:53] We talked about model costs, Sandeep, [18:55] how does-- you talked about how many models [18:57] you're using, like how are you keeping [18:59] your head around these different aspects and tools [19:02] as you start to develop what you're doing internally. [19:06] I mean, I think we-- [19:07] so this is fascinating and I think [19:10] I saw early versions of this, and then Eric Topol [19:13] published it himself. [19:15] And I normally take that a little more seriously. [19:17] So then I saw OpenEvidence kind of retort on it as well. [19:20] So I love the theater around this. [19:23] But zooming out of this specific example, when we were earlier [19:29] thinking of a product roadmap of how should we think about claims [19:35] and reimbursement, how should we think about the patient intake [19:37] experience, how should we think about the patient clinical and [19:41] care pathways. [19:42] How should we think about-- and remember, Optum Health touches [19:45] 20 million patients a year. [19:47] We employ 10,000 physicians. [19:49] So when we're thinking about the tech landscape that [19:53] improves the patient experience and the provider experience, [19:57] the option we have with this engineering firepower [20:00] and with this kind of AI resources [20:03] is to build versus buy. [20:05] And remember, for somebody who's grown up in AI, [20:08] it's really a matter of pride to build. [20:10] And nowadays build is like training a open weight [20:14] model on a whole bunch of data and getting a model right is not [20:17] rocket science anymore. [20:18] You feel like you can do it. [20:21] And I think we refrain from doing that. [20:24] More often than not. [20:25] In this realm, I feel that I need a harness that [20:29] can change out the model. [20:33] It could be a specialized LLM supplier today, [20:36] and tomorrow a guard model may completely [20:38] replace that out of the box because it makes sense. [20:41] It's just getting trained on all digital assets in the world. [20:46] So at some point, the reasoning has [20:48] got to get better and better. [20:49] So I won't comment on the specific case [20:52] because there are many of our vendors and partners in that [20:55] graph you showed. [20:56] And, they're all good-- they're all good partners. [20:59] But I can see the trend. [21:01] I can see the guard models getting better, [21:04] and I can see that responsible organizations, because we [21:07] got to keep focus on affordability [21:09] and then speed to market with end to end functionality, [21:12] I'd rather focus on the experience and the workflow [21:16] and embedding different models into the workflow. [21:20] Clinical reasoning is one of the most critical pieces [21:23] of that puzzle. [21:23] Make sense? [21:26] Absolutely, and I think it's been interesting to your point. [21:32] And early on the conversations, hey, [21:34] we're going to take a bunch of medical data [21:36] and just train a frontier model from scratch. [21:39] I've told our team and our partners and customers like, [21:42] we are not doing that. [21:44] Again, we could be wrong. [21:45] But yes, it takes time and compute [21:49] to do that and specialized data. [21:50] But these models to your point are changing so fast [21:52] by the week. [21:54] It's are there-- it's not that you [21:55] couldn't get to slightly better performance, [21:58] and especially at your scale. [21:59] You have the data where you really could. [22:01] But what is the urgency and what is the priority to do so? [22:04] Getting the context right, getting [22:06] the details of the use cases as well as [22:09] the team alignment on we're going to rethink the work. [22:12] There are higher priorities, I think as we do this. [22:16] Someday, fast forward down the line, [22:19] once we've done all of the low hanging fruit, [22:22] and now we're trying to eke out bits of like, [22:24] sure, let's run at that. [22:26] But candidly, my own opinion is we're years away. [22:30] And like many things-- [22:31] Agreed. [22:31] --where we're predicting the future could be wrong, [22:34] but it just seems like there's bigger fish [22:36] to fry at the moment. [22:36] I mean, it's interesting to hear both of you [22:38] talk about this because what you're kind of saying-- [22:41] and I think this paper, the controversy that swirled around [22:44] it also hints at it too. [22:46] But we really are talking about a commoditization. [22:48] I mean, let's just say what it is. [22:50] And what we're talking about is the value [22:53] that sits on top of that. [22:54] Now, whether that value is your own business context, [22:57] your specific use cases, again, the harness, or maybe [22:59] you do RL and use some new fancy tools [23:02] to improve for a big given. [23:04] But what we're talking about is above the model layer. [23:07] And the other thing I think that's [23:09] interesting about this discussion is, papers like this [23:12] keep happening. [23:14] In other words, peer review papers come out. [23:17] They test models that were 18 months old because that's [23:20] how long the peer review process is. [23:21] But those are the-- but we're trained in the clinical world [23:24] to really pay attention to peer review work. [23:28] And again, we're having these debates that [23:29] are almost 18 months old, to me, in the peer review literature [23:33] around, OK, this model versus this model and that benchmark, [23:36] and-- but what we're talking about in the real world, [23:38] the day to day is the value that we're getting for the use case, [23:42] irrespective of the model. [23:44] And I think that disconnect is really important to highlight. [23:46] And I think it's causing a lot of controversy in the back [23:49] and forth that you're seeing because the folks that [23:51] have a vested interest, as you might in a given model's horse [23:56] race, you're going to have a visceral reaction to that. [23:58] But it's very telling that the folks that are actually [24:02] translating the model into value are [24:04] more focused on the actual value that than the model. [24:07] I think it's a fascinating insight that [24:09] maybe is kind of a lot of us are missing that in the debates [24:12] that we're having. [24:14] Yeah, and I think-- [24:16] We work at an industry that still processes [24:19] 9 billion faxes a year. [24:22] Let's get perspective here before we [24:23] get to whether it fell short of Opus 4.8 [24:27] or not is one end of the question. [24:30] We do process 9 billion faxes. [24:32] I go to some of my team members who are still [24:34] using some parts of UnitedHealth Group, [24:37] still use OCR, optical character recognition for some use cases [24:40] because it's been embedded in the workflow. [24:42] It was a new invention for years back, [24:44] and now it's embedded in the workflow [24:45] and extricating it out to use multi-modal LLMs. [24:50] And when I challenge that team, why are you still using OCR. [24:58] And they're like, well, we're trying to replace the 9 billion [25:00] faxes. [25:02] And this is the cheapest, fast-- anyway, [25:05] so this is a gradient in an industry [25:08] that you're just seeing the tip of the iceberg here [25:11] versus what's really below. [25:13] Yeah. [25:14] So one of the things that's always fascinating [25:17] that we discuss clearly we're all excited about these models. [25:20] We're excited about the potential. [25:22] One thing I struggle with certainly is timing. [25:27] When? [25:28] How fast? [25:30] How do we prioritize this? [25:32] I mean, Sandeep, how do you talk about timing with your team [25:36] and take it in any direction you can. [25:39] Obviously, you're a public company. [25:41] You have many different kind of people [25:44] to communicate with on this, but it's just [25:47] something I find is so hard given [25:50] the pace that these things are moving. [25:53] How frustrated should we be that something's not achieved yet? [25:57] How much time should we give our team? [25:59] Should we OCR the 9 billion facts or something else? [26:02] But like, how do you think about timing with all of this. [26:05] Well, I mean, so actually Matt and I [26:07] work together when I was at Mars in the pet radiology space. [26:11] And at that time, at least the mantra [26:14] I had tried to evangelize among colleagues [26:18] was this idea that a large-- [26:20] Mars is about $50, $70 billion company, like big, big company. [26:24] The idea that a large legacy company could move 100x faster, [26:31] and the idea was 100x, that was the big-- [26:33] every poster had 100x on it and so on. [26:35] My office in Minnesota also has a big 100x logo. [26:40] And the idea was that you worked 100x harder. [26:43] The idea is in a large company by design or in large fragmented [26:48] industries by design, most people [26:50] are not 0 distance to the user, to the end user, to the patient, [26:55] to the receptionist at the front at the provider clinic, [26:59] to the physician, to the moment of truth. [27:02] And so everyone's abstracting and because everyone's [27:05] abstracting, 90% of the people need consensus layers, [27:10] more data, more reports to abstract. [27:13] Unless you take the problem solvers, in this case, [27:16] I'd say engineers, product managers, product managers now [27:20] with cloud and Codex can be engineers, so get them [27:24] to the patient interaction, get them to the call center [27:28] and have them do side by side calls. [27:30] That then injects a rare form of speed. [27:33] And frankly, my experiments, and many fail and many are OK, [27:40] have been around what happens in a large legacy, fragmented [27:46] industry or institution where you get the problem solvers [27:50] zero distance to the problem. [27:55] And then start solving 100 times faster. [28:01] And look, it's mixed results. [28:03] We are particularly at this time when the problem solvers [28:07] have so many superpowers. [28:10] So even next week, I'm trying to set 6 Tiger teams [28:14] on six processes end to end, where I'm not [28:17] seeing fast enough results. [28:18] I'm saying, OK, get the Tiger teams, [28:19] but the Tiger teams need to be top on plot 4.8. [28:23] They need to be top on Codex, they [28:24] need to be top on, and so on. [28:26] In my leadership meeting, strategy meeting [28:29] last week, somebody came up with a competitor [28:32] who has a beautiful experience, just [28:35] the app experience was like breathtaking, [28:38] and it depressed us. [28:39] Like, we were down. [28:40] Like, we were like, OK, if these guys came up [28:42] with this what the heck. [28:44] And then somebody just threw a login password in to-- [28:49] and you know the rest. [28:51] You threw it to Cloud Code saying, [28:52] build a more beautiful experience. [28:54] Get inspired by that. [28:55] But take our context. [28:57] And in about 60 seconds, an HTML file [29:00] appeared, which was even more breathtaking. [29:04] So my point is, we can reconcile ourselves [29:08] to say, oh, this industry will never change. [29:12] There are lots of people like Justin and Matt who came over, [29:14] but this industry will never change. [29:16] Or this is a large organization, it won't move fast enough. [29:19] I think at UnitedHealth Group, particularly this regime [29:22] of leadership, and I won't say this just for positioning [29:27] ourselves every month, every CEO, not CIO, [29:33] not CTO, every CEO of every business, including myself, [29:36] we are in a room with our CTOs, with our CFOs. [29:40] CFOs, very importantly to your point on metrics, [29:42] Justin, what is the metric with the chief medical officers [29:46] in terms of outcomes. [29:48] So the CIO, CTO, CFO, CMO, CEO are [29:51] in one room for all the businesses that 50 [29:53] people, literally 50 people together, six hours. [29:57] I want some of my life back. [29:59] Justin, these are draining meetings things, [30:01] but six hours a month we just obsess [30:05] over AI use cases, AI tokens the value and the outcome [30:10] from AI, the NPS results before and after for AI and so on. [30:16] And so I thought I was doing great with AI for the last three [30:19] years. [30:19] But now this regime, with this leadership, now with this kind [30:22] of cadence. [30:24] And why are we doing that? [30:25] I think it injects just rare-- [30:29] it injects paranoia through the organization. [30:31] Now to prepare for the next one. [30:33] So I better show results in 30 days, [30:36] otherwise my peers-- like so you see [30:39] the tension in the organization and the financial tension [30:42] as well. [30:43] So there's investment, there's a return [30:44] we have to solve for affordability. [30:48] When I'm advising my clients, I actually [30:51] am just trying to tell them, hey, just replicate this, guys. [30:54] And I wish you could do it every week. [30:56] I mean, this is so effective. [30:59] Does that makes sense? [31:01] We don't do it with all of our health system partners, [31:03] but we do this with a handful of them [31:05] where we're running, in some cases every other week [31:10] with the CFO on progress, on tracking [31:13] with that CEOs, and the leadership team. [31:15] And I love that at that scale it's [31:19] the whole team, because I think at a lot of places [31:22] AI is a delegated issue. [31:25] It is delegated to-- [31:27] it's an IT problem. [31:28] It's an innovation problem. [31:30] Hire someone like Matt. [31:33] It's the AI guy's problem. [31:34] Like he'll go figure this out. [31:36] And it really is a CEO level issue and leadership discussion, [31:41] if you want the pace. [31:43] And granted, it's interesting when-- [31:45] obviously, we're here having this discussion. [31:48] The fact that you want to join us on an AI and Health Podcast [31:51] is telling in and of itself. [31:55] The early adoption historically of technology [31:57] at least in the health system side, [32:01] they haven't been the winners. [32:03] Historically, it hasn't been the right strategy [32:06] to super adopt early and pave the way. [32:08] And so that's the historical context for our field. [32:11] And I think we all believe AI is different. [32:15] We all believe these tools are already changing. [32:17] Forget model improvements. [32:19] But it is useful, I think, to recognize the historical context [32:24] for where we come in health for why what you're mentioning [32:28] is so out there, is not the norm for most leadership teams [32:32] and what they're doing. [32:33] But it is starting to happen. [32:34] And so my own opinion is you'll get different groups, [32:37] groups like yours who have leaned in and are making this [32:40] transformation, and many groups who won't. [32:42] And if you believe the outcomes are going to be an order [32:45] of magnitude with these technologies as they're coming, [32:48] they're going to be the organizations that persist [32:50] and the ones that don't. [32:52] I want to say in two years, the health systems you advise, [32:56] Justin, in two years, either AI will be a project [33:01] or AI will be the business. [33:05] It's a sense I get, at least from my clients. [33:07] Yeah, I think, and that's what makes this different to me. [33:11] And I think you guys are dancing around [33:12] this a little bit, but the platform shift conversation, [33:15] do you want-- did you want to be the first one to go to Cloud? [33:18] Like, all up. [33:19] I don't know. [33:19] I don't know, if those folks were happy with the decisions. [33:23] There were so many things they had to learn. [33:24] But because this is a shifting landscape, [33:27] and I think you're in a position now where you're actually [33:32] going from it sounds like a top down, but also [33:34] a bottoms up, which is where this technology kind of touches [33:38] every aspect of your organization. [33:39] This isn't as much of a static platform shift to me [33:42] as it is a way of working change. [33:46] It's not just another, quote unquote, "tool" [33:49] that I'm getting a license to and using. [33:51] It's going to then transform the work itself. [33:55] And I think that again, we hear this a lot. [33:58] But to see organizations like yours, the ones [34:00] that Justin's work with, I've seen several that [34:04] are the light bulb turns on. [34:05] There's a lot of iteration, though, [34:07] and there's a lot of I would say, some persistence necessary. [34:10] But once that light bulbs on, it's magical. [34:13] And it's almost you just back away [34:16] and let the folks that are sitting, to your point, [34:18] really close to the user, really close to the translation, [34:22] take the ball, and then you said, create something [34:25] on the fly that can immediately addresses the user's need or use [34:28] case right there. [34:30] Now you may have to productize it and do some more, [34:33] but the point is that light bulb turns on at that moment where [34:36] you're at the peak engagement between the use [34:39] case and the person that's now has [34:41] this power at their fingertips that is not just another tool. [34:44] It's something more than that. [34:47] Absolutely. [34:48] Yeah, and for us, I mean-- [34:50] and look, both of you, I know you personally. [34:52] You're very mission driven. [34:53] You're doing this more than for a livelihood. [34:55] You're doing this for making a dent in health. [34:59] And so for us as well, it's like, OK, it's bottoms up, [35:02] it's top down, but it's inside out. [35:03] So everything we learn inside becomes a product outside. [35:10] And because we are an important, relatively important part [35:15] of the system, we feel almost a responsibility [35:18] saying, hey, if we can solve for this part of UnitedHealth [35:22] group's workflow, well, what if we extend this capability [35:26] across the board. [35:28] And that's why products like Optum Real and Crimson [35:31] and Digital Prime, et cetera as we try [35:34] and solve for these problems, I think the more partners [35:40] we can get in this journey, I think we make a serious dent [35:44] in health care. [35:46] So that's the dream. [35:49] It really does feel like the moment when it can change. [35:52] And thank you so much for joining us. [35:54] And I think the purpose of this was, how can we bring people [35:59] to this mission, to your point, when [36:02] it feels like the forces are aligning [36:04] and the people and the technology and the tools [36:06] to really make things better. [36:07] And so, Sandeep, thank you so much for joining us. [36:10] And we will discuss many more hours on all this in the future. [36:15] I alwasy learn so much from you, Sandeep. [36:16] But I will say the only thing I'm not [36:18] going to take away from this is telling my daughters that I [36:20] should be buying them gifts for Father's Day. [36:22] I'm just letting you know that, of all the pearls you've [36:25] shared with us. [36:26] No, this was fun. [36:27] We could have-- we could have danced all night. [36:29] So this is so good. [36:31] Thank you for what both of you do. [36:33] And always fun to hang with you. [36:35] So appreciate it. [36:36] Thank you.