Protege Warns Medical AI Aces Exams but Fails Real Clinical Tasks
Daisy Wolf, Eva Steinman, Engy Ziedan
Hospital adoption of medical AI based on vendor exam scores risks severe clinical failures because up to 80% of electronic health data is already contaminated and models lack independent, real-time evaluation in live workflows.
AI-assisted summary of Why Medical AI Needs a Referee | Protege's Engy Ziedan — original source. Check the original for full context.
Source published:
Key ideas
- Medical AI exhibits a severe measurement gap: models can score over 92% on static licensing exams while dropping to 45% on real-world clinical tasks, underscoring that benchmark performance does not reflect hospital readiness. (20:00)
- Widespread data contamination undermines benchmark validity, with data partners reporting that up to 80% of accessible electronic health data has already been licensed for model training. (30:00)
- Subtle misalignment in healthcare AI creates an Edgeworth-box conflict where insurer agents automate claim denials and hospital agents maximize revenue recovery, leaving patients without agency in the middle. (10:00)
- Unlike human clinicians who undergo a decade of education, residencies, and state licensure, medical AI models are deployed into clinical workflows without standardized credentials based solely on vendor-reported claims. (32:41)
Detailed notes
## Background Published on August 24, 2026, via web and RSS by Daisy Wolf and Eva Steinman, this episode features Engy Ziedan, co-founder and Chief Scientific Officer of Protege. According to the show description, the conversation explores why medical AI has a profound measurement problem, why acing static licensing benchmarks does not guarantee hospital safety, and how contaminated training data and subtle misalignment necessitate independent, real-world clinical evaluation.
## TL;DR Engy Ziedan details why static multiple-choice medical exams fail to measure clinical competence, noting that models can achieve 92% exam scores while plummeting to 45% on actual clinical tasks. She warns that widespread data contamination—with up to 80% of existing health data already ingested into training sets—causes models to memorize answers rather than reason, forcing Protege to locate completely un-digitized physical pathology slides to build untainted benchmarks. The discussion examines how AI vendors face commercial conflicts of interest when grading their own tools and contrasts the exhaustive credentialing required for human clinicians with the unverified deployment of clinical AI. Ultimately, Ziedan argues that retrospective regulatory frameworks are too slow for agentic models, advocating instead for continuous, real-time independent oversight across specialized clinical subnodes.
## Atmosphere & style The conversation carries an incisive, analytically rigorous, and demystifying tone, anchored by Ziedan’s ability to pair formal health economics models with candid operational reality. Hosts Daisy Wolf and Eva Steinman maintain a brisk, inquisitive dynamic, prompting grounded discussions that cut through vendor hype using memorable real-world examples, like hunting through hospital basement drawers for un-scanned glass tissue slides. The exchange moves from a sharp critique of paper-thin benchmarking practices to a sober, high-stakes warning about automation bias, creating the feel of an urgent, behind-the-scenes strategy briefing on the clinical and economic realities of healthcare AI.
## Bulletin - `20:00` Medical AI exhibits a severe measurement gap: models can score over 92% on static licensing exams while dropping to 45% on real-world clinical tasks, underscoring that benchmark performance does not reflect hospital readiness. - `30:00` Widespread data contamination undermines benchmark validity, with data partners reporting that up to 80% of accessible electronic health data has already been licensed for model training. - `10:00` Subtle misalignment in healthcare AI creates an Edgeworth-box conflict where insurer agents automate claim denials and hospital agents maximize revenue recovery, leaving patients without agency in the middle. - `32:41` Unlike human clinicians who undergo a decade of education, residencies, and state licensure, medical AI models are deployed into clinical workflows without standardized credentials based solely on vendor-reported claims.
## Key takeaways - `00:00` Frontier models face an impending bottleneck in downstream utility driven by the scarcity of specialized, high-quality, real-world training data rather than open internet text. - `05:00` The commercial pivot of foundation model labs from consumer chatbots to enterprise workflow automation requires rigorous independent evaluation to convince risk-averse enterprise buyers. - `06:30` Healthcare AI cannot follow frictionless zero-marginal-cost consumer adoption models because providers incur direct compute expenses, onboarding costs, and severe malpractice liability risks. - `06:30` Information asymmetry in medicine mirrors Akerlof's 'market for lemons,' making it hazardous to license autonomous AI agents based merely on 5,000-question multiple-choice exams. - `10:00` While catastrophic mortality failures are straightforward to define, subtle organizational misalignments—such as suppressing out-of-network referrals to preserve hospital revenue—pose greater detection challenges. - `10:00` Ambient listening tools can mitigate human charting bias by generating objective SOAP notes that remove subjective editorializing and prejudice against patients. - `12:35` The healthcare AI market suffers an evaluation vacuum where 'everyone and no one' watches model safety, as vendors have zero incentive to highlight their own failure modes. - `15:00` Benchmark fragility is widespread: minor changes to prompt wording or multiple-choice answer ordering can drastically shift model rankings in specialized domains like oncopathology. - `15:00` Retrospective quality metrics that review outcomes 6 to 12 months later are inadequate for governing fast-moving, personalized AI agents, necessitating continuous real-time monitoring. - `20:00` Grading AI against human clinicians is confounded by physician-specific habits and treatment beats, meaning a model deviating from a doctor's preference is not inherently incorrect. - `25:00` Protege acts as an independent arbiter across clinical subnodes by running competing models through standardized harnesses to diagnose deficiencies and identify required retraining data. - `27:00` Federal policy and regulation move too slowly to safeguard AI deployment, as demonstrated by quality programs like Medicare readmission penalties taking a decade to evaluate single metrics. - `30:00` To guarantee untainted evaluations, Protege enforces a 'sealed membrane' policy and sourced physical, un-digitized glass pathology slides from hospital drawers. - `32:41` A severe credentialing double standard exists between heavily regulated domestic and international human clinicians and uncertified clinical AI systems. - `34:20` Relying on human-in-the-loop oversight is fundamentally compromised by automation bias, as clinicians rarely possess the time or skepticism to dispute computer-generated clinical outputs.
## Detailed walkthrough ### `00:00` The Core Dilemma of Medical AI and Protege's Genesis
Hundreds of millions of people routinely query generative AI models about their health, yet the healthcare industry faces an unanswered question: who is actually checking whether the outputs delivered to patients are safe, aligned, and clinically accurate? While frontier models frequently ace standard multiple-choice medical board exams, acing static benchmarks does not translate to being safe or competent inside a real-world hospital. Models are fundamentally constrained by the real-world data used to build them, and evaluating their capabilities requires looking past superficial test scores to examine edge cases, subtle misalignments, and catastrophic failure modes.
Daisy Wolf and Eva Steinman introduce Engy Ziedan, co-founder and Chief Scientific Officer of Protege and an assistant professor of healthcare economics at Indiana University whose research has been featured in *The New York Times* and cited by the Centers for Disease Control and Prevention (CDC). Reflecting on how Protege began, Engy recalls sitting in her office as an assistant professor at Tulane University two years after completing her PhD. The COVID-19 pandemic had just begun, and she was urgently trying to write empirical economics papers that required data from literally two days prior—an impossible turnaround at the time. She connected with Bobby, who was working at a data facilitation company and helped her acquire AWS instances, data access, and institutional connections.
After that initial engagement, the two fell out of touch until February 2024, when Bobby sent Engy a straightforward email from a personal Gmail account containing a brief Google Document. In his typical direct style, Bobby posed a foundational question: "What do we hold to be true about the future?" The document set out a clear thesis that frontier foundation models would eventually scrape the open internet clean and find their downstream usefulness bottlenecked by the availability of high-quality, real-world data across specialized domains. That memo became the blueprint for Protege, establishing its mission to source aligned, real-world data and independent evaluations to guide AI training. Today, nearly all major foundation models have utilized Protege’s healthcare data for pre-training and mid-training. The company’s internal operations span diverse multi-modal domains:
- **Endoscopic Video:** Over 100,000 internal endoscopy videos capturing procedures inside human bodies. - **Veterinary Medicine:** Real-world clinical pet and animal care datasets. - **Robotics Manipulation:** 3D physical object suites—such as diverse collections of household mugs—used to train robotic systems to estimate object weight and tactile handling.
> “When I look back at that document, I often read it every few months or so, which is that models are going to be inhibited in their usefulness by the training data available for them.” — Engy Ziedan
---
`05:00` From Consumer Chatbots to Enterprise Workflows
When Protege launched, the team assumed its primary client base would consist of early-stage AI startups trying to build specialized applications. Instead, the world's largest foundation model labs immediately became their modal customers. At first, these labs focused on the broad consumer market, prioritizing whether a conversational bot could interact intelligently with users, maintain safety boundaries, and present reasonably sound medical information without causing immediate harm.
Over time, labs recognized that the largest revenue opportunities lay in enterprise settings, which fundamentally altered their data and evaluation requirements. Rather than simply powering casual conversations, enterprise models must aggregate tasks across knowledge work, automate operational workflows, and potentially take over end-to-end responsibilities. This evolution forced the industry to reconsider how work is structured, recognizing that traditional job descriptions are effectively contrived human bundles of discrete tasks.
```mermaid flowchart TD accTitle: Evolution of AI Deployment Demands accDescr: Diagram illustrating the shift from consumer-facing conversational AI to high-stakes enterprise clinical integration.
subgraph Consumer["Phase 1: Consumer Market"] A["Conversational Safety"] --> B["Basic Alignment"] B --> C["Casual Patient Queries"] end
subgraph Enterprise["Phase 2: Enterprise Integration"] D["Task Aggregation"] --> E["Workflow Automation"] E --> F["Full Clinical Handoff"] end
Consumer -->|"Model Maturity & Revenue Pressure"| Enterprise Enterprise --> G["Requirement for Independent Evals"] ```
Because foundation model labs operate under commercial pressure to market their systems as the industry leader, independent evaluation has become critical. Model builders need objective proof of real-world efficacy rather than self-reported test scores to convince risk-averse enterprise buyers to adopt their technology.
> “Jobs as a collection of tasks, right, are like a contrived human concept.” — Engy Ziedan
---
`06:30` The Economics of Evals and Healthcare's Asymmetry Problem
To explain why evaluations are indispensable for AI, Engy draws on economic theory and the concept of hedonic pricing—the methodology economists use to infer the implicit value of a good based on what individuals are willing to pay for its specific characteristics, such as how local school quality dictates neighborhood home prices. When consumer applications like Uber entered the market, customers never asked for an academic evaluation or benchmark score; they simply booked a ride, observed the convenience and price, and integrated the service into their daily lives until it became a common verb.
AI cannot rely on that frictionless adoption model because the technology does not have a marginal cost of zero. Users and health systems pay real costs: direct compute fees in tokens, staff onboarding and training hours, and the ongoing liability risk that a probabilistic model will fail in a high-stakes scenario. To reach market equilibrium and price these tools correctly, buyers must be able to differentiate truly valuable technology from useless or dangerous implementations.
This dynamic is especially acute in healthcare due to structural information asymmetry. Just as George Akerlof demonstrated in the classic "market for lemons" model of used cars—where sellers possess far more technical knowledge than prospective buyers—healthcare providers possess deep expertise about diseases, clinical risk, and the value of medical interventions that patients cannot easily evaluate on their own. The societal value of clinical expertise is backed by clear empirical evidence: quasi-experimental studies exploiting admissions cutoffs for medical schools have shown that when a family member randomly gains admission to medical school and becomes a doctor, the entire extended family experiences measurable gains in life expectancy.
Deploying autonomous or semi-autonomous AI agents into this asymmetric environment exposes the inadequacy of existing evaluation paradigms. Granting clinical autonomy to an AI agent based solely on a passed benchmark is the regulatory equivalent of suddenly licensing a million new doctors simply because they cleared a 5,000-question multiple-choice exam. By contrast, the approximately one million practicing physicians in the United States undergo nearly a decade of clinical residencies, hands-on apprentice-style supervision, and continuous peer-reviewed testing. Substituting static test results for real-world validation poses immense risks in a sector where errors carry irreversible life-and-death consequences.
- **Non-Zero Marginal Costs:** Operating frontier AI requires accounting for token expenditure, technical integration pipelines, staff workflow adaptation, and the legal and operational probabilities of malpractice. - **The "Doctor in the Family" Evidence:** Empirical health economics demonstrates that having a verified clinician in an extended family network significantly lengthens family members' lifespans, underscoring the high value of genuine, nuanced clinical judgment over static knowledge retrieval. - **Exam Disparity:** Standard AI benchmarks typically evaluate performance across a few thousand static multiple-choice questions, whereas human medical licensure requires undergraduate education, four years of medical school, standardized clinical skills exams, and 3 to 7 years of hospital residency.
> “And without understanding really what is valuable and what's value-less technology, this technology does not have a marginal cost of zero, right? People are paying in tokens. They're paying with onboarding time. They're also paying with the probability that something would go wrong.” — Engy Ziedan
> “When you put agents on top of that, right, it's almost like saying we have a million doctors in America. We're going to hire a million more. Those additional million doctors have taken a lovely 5,000 question exam and passed it. We'll let them go ham and see how it goes.” — Engy Ziedan
`10:00` Catastrophic Failures, Subtle Misalignment, and Ambient Notes
Engy Ziedan draws a distinction between two types of failures in healthcare artificial intelligence: narrow catastrophic failures and pervasive, subtle misalignment. Catastrophic harm—such as direct patient mortality—is relatively straightforward to define and engineer against. The far more difficult challenge is detecting subtle misalignment and bias. Ziedan illustrates this dynamic using an economic concept:
> “You can think of it as like an Edgeworth box where there are two indifference curves that are coming together on a contract curve. One is trying to maximize total revenue and basically minimize payouts, and one is trying to maximize revenue recovered. They are both using agentic tools and the patient is in the middle. The patient has no agency.”
In this scenario, health insurance companies deploy AI to generate medical denials, while hospital systems build or acquire agentic tools to fight prior authorization hurdles and recover revenue. Because there is no active referee evaluating alignment, models can easily optimize for organizational business rules rather than patient welfare. Examples include an AI checking whether a patient owes the hospital money before allowing a prescription, or suppressing out-of-network referrals to protect hospital revenue even when a patient prefers out-of-network care.
Conversely, clinical AI tools can also strip away human bias. Drawing from Protege’s access to hundreds of billions of clinical notes, Ziedan points out that human-written medical charts frequently harbor prejudice, with entries like, *"She looks really disheveled. Her husband is asking good questions."* Phrases like that often signal skepticism toward the patient, leading clinicians to write off severe physical pain as potential mental illness. Ambient listening tools record doctor-patient conversations directly and convert them into objective SOAP notes, eliminating subjective editorializing so subsequent physicians can order appropriate clinical workups. Yet these objective benefits often go unmeasured by standard evaluation frameworks.
```mermaid flowchart TD accTitle: Structural Misalignment in Healthcare AI accDescr: Diagram showing how insurer and hospital AI agents squeeze the patient while ambient tools attempt to capture objective truth.
subgraph Conflict["The Misalignment Arena"] PayerAI["Insurer AI Agents<br/>(Minimize Payouts / Automate Denials)"] HospitalAI["Hospital AI Agents<br/>(Maximize Revenue Recovered / Prior Auth)"] Patient["Patient in the Middle<br/>(No Agency, Vulnerable to Subtle Bias)"] PayerAI -->|"Contract Curve Pressure"| Patient HospitalAI -->|"Contract Curve Pressure"| Patient end
subgraph Solution["Ambient Clinical Documentation"] DrPatient["Doctor-Patient Encounter"] Ambient["Ambient AI Recording"] CleanNote["Objective SOAP Note<br/>(Strips Human Bias & Skepticism)"] DrPatient --> Ambient --> CleanNote end ```
`12:35` Who Watches the Models? The Incentive Gap in Independent Evaluation
When clinicians use AI tools like OpenEvidence at the point of care, the question arises: who ensures the answers generated are safe and accurate? Ziedan responds that the reality is currently “everyone and no one.” Foundation model providers expend significant effort training models on clinical data to improve general alignment and trustworthiness out of the box, but vertical AI builders subsequently fine-tune those foundational models to satisfy specific commercial service-level agreements (SLAs) and business objectives.
Because every vendor issues promotional materials claiming superior performance, healthcare buyers face total uncertainty:
> “Every vendor of an AI tool, every vertical AI builder would have like a little brochure that says I am best. And so the no one is that if everyone says they are best, no one knows who's best. Also, there is no incentive to kind of pinpoint to where your product fails.”
Without an independent evaluator operating at arm's length from vendors, hospitals and patients lack visibility into the failure modes that sit beneath catastrophic outcomes. This confusion is compounded by contradictory academic literature. A paper published in *Nature* argued that general frontier models (such as OpenAI's) outperformed healthcare-specific vertical AI models, while another paper on arXiv reached the opposite conclusion shortly thereafter. Because academic journals incentivize dramatic, controversial contrasts, peer review alone cannot settle real-world utility.
- **Foundational alignment:** Training on medical corpora often enhances general model trustworthiness, but it does not account for downstream enterprise tuning. - **Vendor brochures:** AI vendors have zero incentive to highlight their own failure modes or surface subtle edge cases. - **Academic contradictions:** Competing publications in venues like *Nature* and arXiv demonstrate conflicting findings over whether generalized or specialized medical models perform best.
`15:00` Benchmark Fragility, Contamination, and the Limits of Retrospective Quality Systems
Evaluating models using static benchmarks reveals severe vulnerabilities in practice. At Protege, benchmark results vary wildly based on minor prompt variations or the sequence of answers in multiple-choice questions—especially in complex domains like oncopathology. If shifting the order of multiple-choice answers drastically alters a model's score, it suggests the evaluation benchmark may be contaminated, meaning the model merely memorized the answer keys like an unprepared student rather than mastering clinical reasoning.
This creates severe information asymmetry for hospital procurement teams spending millions of dollars on AI. Furthermore, static evaluations become obsolete as models undergo test-time training, encounter data drift, or evolve into personalized weights tuned to individual physicians:
> “These static question answers or like something in a paper where it was evaluated on a 2024 technology and published in 2026 just won't do. And so you risk not to fearmonger, but you risk a situation where we enter like the opioid epidemic when it peaked in 2010. Only when people started saying, OK, National Task Force, we're calling it an epidemic, we're acting on it, let's move.”
Ziedan contrasts the required paradigm with historical federal programs like value-based purchasing, initiated around 2007, where roughly 80% of government reimbursement to physicians and hospitals is tied to static quality measures (such as penalties for prescribing antibiotics for viral illnesses or tracking falls in skilled nursing facilities). These retrospective frameworks evaluate performance six to twelve months after the fact. While traditional healthcare policy analysts often throw up their hands because healthcare comprises 20% of GDP and 20 million out of 150 million American jobs, retrospective audits cannot govern fast-moving agentic AI.
The necessary path forward is continuous, real-time oversight:
> “The right thing to do is for there to be a watcher all the time. That when my local AI agent as a physician starts to nudge me towards decisions that are maybe profit maximizing, but hurting my patient, I'm alerted.”
Constant monitoring must safeguard against dangerous feedback loops—such as when an AI assistant picks up on a negative remark a nurse makes about a patient, becomes overly eager to recommend misaligned or aggressive steps, and ends up delivering clinical advice that is completely uncalled for.
`20:00` The Failure of Static Benchmarks and Human "Beats"
The discussion kicks off with an examination of why standard benchmarks completely miss the mark when applied to clinical practice. Daisy Wolf highlights a startling reality: AI models can score upwards of 92% on licensing exams while tumbling down to 45% when tasked with real-world clinical tasks. As she puts it, acing a test does not prove an AI is ready for the clinic, just like a top MCAT score doesn't magically make someone a capable surgeon. Engy Ziedan traces how benchmarking got to this point: it began with med school multiple-choice questions where models outscored the average student or resident, and then expanded to massive endeavors where models brag about evaluating over 700,000 question-and-answer pairs reviewed by human physicians.
```mermaid flowchart TD accTitle: Evolution of Medical AI Evaluation accDescr: Progression from static test questions to real-world point-of-care clinical evaluations. subgraph Stage1["1. Academic Testing"] A["Med School Q&A Exams"] -->|"High test score"| B["Passes Licensing Benchmarks"] end subgraph Stage2["2. Scaled Expert Review"] B --> C["Thousands of Doctor-Graded Q&As"] C -->|"Flaw"| D["Captures Physician Bias & Idiosyncrasy"] end subgraph Stage3["3. Real-World Point of Care"] D -->|"Transition"| E["Subnode Evaluation (Point of Care)"] E --> F["Task Success & Clinical Outcomes"] end ```
Engy pushes back on whether these massive Q&A sets matter to someone actually facing high-stakes medical treatment. She poses a direct thought experiment: if you are a patient about to undergo spinal surgery, do you trust a model that passed a 10,000-question exam, or a surgeon who has physically performed 4,000 operations? Patients care about specific, high-risk track records—whether a model has served as a task aggregator in an operating room, whether unwarranted deaths went up, and whether outcomes improved.
Evaluating models against real physicians also runs headfirst into clinical bias and habit. Doctors develop individualized clinical preferences or "beats." Engy cites knee replacements as an example frequently analyzed at Protege: some orthopedic surgeons will only ever perform partial knee replacements, whereas another surgeon will perform a total knee replacement on every single knee they operate on. If an AI model evaluates a patient's chart and recommends the opposite of what the total-replacement doctor chose, traditional evals label the AI wrong. In reality, the physician may have a sticky preference or hysteresis. As models inevitably surpass average clinicians, Engy points out that you cannot simply rely on the law of large numbers by surveying more doctors; instead, evaluation must move directly to high-risk, expert-level point-of-care testing.
- Models can score 92% on licensing exams while falling to 45% on actual real-world clinical tasks. - Major model builders have had physicians review more than 700,000 Q&A pairs, but this general intelligence metric fails to measure specific scenario performance. - “At some point, a task will be so expert, so high risk, that testing at point of care will become really warranted,” says Engy Ziedan.
---
`25:00` Protege as the Independent Arbiter Across Clinical Subnodes
When vertical AI builders create specialized tools within the same clinical "subnodes"—such as ambient scribing, clinical trial matching, nurse scheduling, or oncopathology analyzing H&E tissue slides for immuno-oncology (IO) pairings—they end up acting like "ships in the night." Every founder claims their own model is superior, yet none can verify their competitor's standing. Protege steps into this void as an independent arbiter: they host models in identical subnodes, run them through an identical evaluation harness, and produce comparative reports showing exactly where each model leads or lags.
Serving as this arbiter introduces thorny methodological choices that Protege has to navigate directly: - **Prompt construction:** Deciding standard prompt wording versus allowing models to use specialized prompting. - **Harness customization:** Weighing whether frontier foundation models require custom harness belts versus standard built-in setups. - **Refusal handling:** Figuring out what to do when a model refuses to answer—such as replacing it with a mean value or falling back to a second-best model—and measuring how that decision shifts rank order.
Engy explains that these finalized evaluations are shared directly with the ultimate enterprise buyers, such as hospitals and pharmaceutical companies, so they can make informed procurement decisions. Addressing why model builders shouldn't evaluate their own work, Daisy notes: “It's so obvious why the creators of the models shouldn't be the ones releasing the evals and why people shouldn't be allowed to grade their own homework.”
---
`27:00` Trust, Government Inertia, and Human-Computer Interaction
When questioned on why Protege has the right to serve as this centralized evaluator over anyone else, Engy breaks it down into absolute and comparative advantage: Protege excels in the healthcare data domain with the lowest marginal cost for generating training data and evals. More importantly, when an evaluation reveals a specific deficiency in an AI model, Protege's massive, held-out test sets allow them to immediately diagnose the blind spot and prescribe the exact data required to retrain and reinforce it. To protect credibility, the entire testing methodology is made transparent, participant identities are kept anonymous until the end, and builders are permitted a silent exit if they do not wish to publish their rankings. Engy emphasizes that any perceived bias would trigger an immediate, explosive collapse of platform trust.
Daisy and Engy dismiss the idea that the federal government will step in to solve AI evaluations. Engy points out that government agencies frequently struggle with basic digital infrastructure—reminding the room that the government cannot even facilitate seamless online tax filing without commercial software—and agencies are currently leaning on industry collaborators to help determine how AI should be evaluated. Engy recalls her doctoral dissertation on Medicare's Hospital Readmissions Reduction Program (part of value-based purchasing), which took an entire decade and thousands of papers to answer a single question: does financial penalization reduce readmissions?
Modern AI cannot wait a decade for backward-looking assessments. Current healthcare AI is not outright replacing staff; rather, models are “riding shotgun” with clinicians. Because humans exhibit unpredictable behaviors—ranging from stubborn resistance to unearned overconfidence—retrospective static benchmarks fail to capture how human-computer interaction plays out in live-wire, real-time clinical workflows.
- “We have the right to win here because it kind of naturally evolved is that once we do these evals in the subnode, right, we say once the evaluation is done and we hold this test set that never leaves, if you have a deficiency in an attribute that's now been revealed, right? We can tell you exactly what data would improve the model's performance on that deficiency,” explains Engy Ziedan. - “We think what's more safe is just do it, right? And then let people challenge you on it, but just do it,” Engy says regarding moving forward without waiting for government mandates. - Traditional policy quality evaluations, like Medicare readmission penalties, take a decade to evaluate a single metric, whereas AI deployments need continuous feedback on thousands of clinical dimensions.
`30:00` The Contamination Crisis and the "Sealed Membrane" for Clinical Evals
The hosts bring up the problem of models simply memorizing test answers rather than learning genuine clinical reasoning. Because Protege is plugged directly into live data streams, it is uniquely positioned to evaluate models against clean datasets that have never been sold to model builders. Engy Ziedan reveals that data contamination has sparked intense debate both inside Protege and with external customers. Initially, she assumed sourcing uncontaminated data across hundreds of millions of patients and hundreds of billions of clinical notes would be straightforward, only to discover that the overwhelming majority of existing healthcare data had already been ingested for training. When she asked for data that had never been handed over to anyone, partners told her that up to 80% of what they held had already been licensed out.
To create genuinely independent benchmarks where a model has never encountered a patient before, Protege had to resort to extraordinary measures. In oncopathology, for instance, Protege contacted hospital pathology departments directly and asked them to search their physical pathology drawers for glass slides that had never even been digitized. Engy recalls demanding: “Net new slides. Never before has this patient entered a model.” Even when researchers push back with philosophical arguments—claiming, for example, that if a model is performing worse than random on a given task, a little contamination hardly matters—Engy remains unyielding. She argues that performing worse than random is itself a red flag signaling serious underlying misalignment that demands explanation. To protect the integrity of their assessments, Protege instituted what Engy terms “sealing a membrane”: a strict moratorium dictating that any patient or dataset that has ever touched a pre-training or mid-training pipeline is permanently barred from being used in an evaluation or benchmark.
- Sourcing independent data proved far harder than expected because approximately 80% of readily available electronic health data had already been sold for model pre-training and fine-tuning. - To produce untainted oncopathology benchmarks, Protege sourced physical glass slides that had sat un-scanned in hospital drawers so the digital whole-slide images were completely net-new to any neural network. - Engy rejects researcher dismissiveness around benchmark contamination: “Yeah, but like worse than random could be a detection that there is serious misalignment to like, why is it doing worse than random? We should just care that this patient has not entered training previously.”
---
`32:41` The Credentialing Paradox: Licensing Human Doctors vs. Unleashing Uncertified AI
Engy frames the current regulatory landscape around medical AI as a glaring double standard when compared to the stringent barriers placed on human clinicians. The United States heavily relies on importing physician and nursing talent from abroad—particularly in sectors like long-term care, which depends extensively on nurses educated in developing nations. These healthcare workers are subjected to grueling education, extensive credential verification, and demanding state board examinations before they can touch a patient. Even an American physician fully licensed in one state cannot freely practice in another without jumping through arduous re-examination and reciprocity hoops.
Yet, as Engy points out, frontier AI models are routinely deployed into clinics and health systems without any standardized credentials whatsoever. Often, the sole basis for their deployment is an unsubstantiated marketing claim from the vendor asserting that their AI outperformed a small, arbitrary cohort of internal physicians on an internal test. Engy stresses that health, personal finances, and education represent the foundational pillars of human capital. When models fail or behave unpredictably in these high-stakes domains, it risks triggering widespread, catastrophic public mistrust in artificial intelligence as a whole.
Engy puts the discrepancy in stark terms: “And then we have these AI tools that are unleashed in healthcare systems or in kind of practices with absolutely no credentials, other than the fact that the maker says, compared to some physicians we have, the AI beat it.”
```mermaid flowchart TD accTitle: Human Credentialing vs AI Model Deployment accDescr: A comparison showing the rigorous examination required for human clinicians versus the unregulated claims used to deploy clinical AI models.
subgraph HumanClinicians["Human Clinical Workforce"] A["Imported or Domestic Clinicians"] --> B["Rigorous Study & Clinical Training"] B --> C["Strict State-by-State Board Exams"] C --> D["Certified for Clinical Practice"] end
subgraph MedicalAI["Medical AI Models"] E["AI Model Released by Vendor"] --> F["Self-Reported Vendor Benchmark<br/>('Beat Our Doctors')"] F --> G["Zero Standardized Credentials"] G --> H["Unleashed into Clinical Workflows"] end ```
---
`34:20` The Myth of Self-Regulation and Protecting Public Trust
Daisy and Eva note that the medical industry currently tolerates a loose regime of self-regulation and self-governance under the comforting assumption that AI is merely an assistant. Stakeholders tell themselves that because a licensed physician sits in the loop to review the output, human oversight will effortlessly catch any model hallucination or clinical error. A similar hollow safeguard appears in consumer AI products like ChatGPT, which display ubiquitous disclaimers stating that their answers do not constitute formal medical advice.
In practice, however, these safety nets fall apart because human psychology leans toward automation bias: people naturally trust information produced by computers and rarely possess the time, context, or skepticism needed to dispute it. The hosts express relief that an independent third party like Protege is stepping up to serve as an objective referee grading these models, rather than allowing the AI developers to grade their own work. The episode closes with thanks to Engy for her work safeguarding clinical AI, followed by standard network sign-offs and legal disclosures noting that Andreessen Horowitz (a16z) may maintain investments in the companies discussed on the podcast.
## Notable quotes > `25:00` It's so obvious why the creators of the models shouldn't be the ones releasing the evals and why people shouldn't be allowed to grade their own homework. — Daisy Wolf
> `20:00` At some point, a task will be so expert, so high risk, that testing at point of care will become really warranted. — Engy Ziedan
## Mentioned Daisy Wolf, Eva Steinman, Engy Ziedan, Protege, Indiana University, The New York Times, Centers for Disease Control and Prevention (CDC), Tulane University, Bobby, AWS, Uber, George Akerlof, OpenEvidence, Nature, arXiv, Medicare, Hospital Readmissions Reduction Program, ChatGPT, Andreessen Horowitz (a16z)
## Video description Daisy Wolf and Eva Steinman are joined by Engy Ziedan, co-founder and Chief Scientific Officer of Protege, to discuss why medical AI has a measurement problem, and why scoring well on a benchmark doesn't necessarily mean a model is ready for the hospital. Engy explains why healthcare AI needs independent evaluations that go beyond static exams and measure how models actually perform in real-world clinical workflows. They explore the risks of subtle bias and misalignment, why the same model can rank differently depending on how it's prompted or tested, and what happens as AI becomes more personalized and changes faster than traditional healthcare quality systems can keep up. The conversation also gets into Protege's role as an independent evaluator, how contaminated training data can undermine benchmarks, and why the future of medical AI may require continuous monitoring rather than occasional testing.
[Source](https://mgln.ai/e/1344/afp-848985-injected.calisto.simplecastaudio.com/3f86df7b-51c6-4101-88a2-550dba782de8/episodes/e568464b-7a8f-4739-8a93-ae7df452a987/audio/128/default.mp3?aid=rss_feed&awCollectionId=3f86df7b-51c6-4101-88a2-550dba782de8&awEpisodeId=e568464b-7a8f-4739-8a93-ae7df452a987&feed=JGE3yC0V)