Nebius Learning Engineer Rethinks AI Productivity Trackers

Beyond Adoption: Building AI Judgment in the Flow of Work With Nebius Academy — Ai4

Ai4

Enterprise AI adoption risks degrading core workforce skills unless organizations actively measure and develop human judgment over raw usage volume.

AI-assisted summary of Beyond Adoption: Building AI Judgment in the Flow of Work With Nebius Academy — original source. Check the original for full context.

Source published: 2026-08-16

Detailed notes

## Background Published on YouTube by Ai4 on August 16, 2026, this talk features Sharahn McClung, a Learning Engineer at Nebius Academy. The presentation examines the pitfalls of measuring enterprise AI success purely through usage telemetry, explaining how organizations can cultivate and evaluate true 'AI judgment' directly within daily engineering workflows.

## TL;DR Enterprise AI rollouts often mistake seat count and task speed for genuine human improvement, obscuring the risk of automation bias and long-term skill decay when tools are removed. Sharahn McClung defines real capability as AI judgment across three distinct moves: trusting sound outputs, overriding fluent hallucinations, and thinking unaided. Because traditional L&D courses fail to replicate real-world stakes, judgment must be measured through daily work transcripts and seeded error tests. Tools like Nebius Academy's Evolve Coach demonstrate how tracking 'accept, modify, and reject' behaviors directly within the developer's workflow creates a continuous, defensible capability moat.

## Atmosphere & style The presentation is crisp, analytical, and grounded, cutting through generative AI industry hype with measured cognitive science and empirical workforce studies. McClung speaks with calm, structured authority, guiding the audience from the hidden dangers of uncalibrated adoption toward practical engineering solutions. It feels like an insightful, high-signal masterclass that transforms an uncomfortable warning about workforce deskilling into a clear, actionable operational roadmap.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=jiCfGTEY53g&t=4s) Enterprises frequently mistake AI seat usage for actual skill improvement, leaving up to 40% of prospective productivity gains unrealized due to lagging workforce and talent strategies. - [02:39](https://www.youtube.com/watch?v=jiCfGTEY53g&t=159s) A 2025 METR study revealed that experienced engineers were 19% slower when using AI despite expecting to be 20% faster, illustrating an illusion of productivity driven by offloaded cognitive effort. - [02:39](https://www.youtube.com/watch?v=jiCfGTEY53g&t=159s) Automation bias increases human reliance during model failure modes, causing errors to cluster at model blind spots and resulting in human skill decay once AI tools are removed. - [06:37](https://www.youtube.com/watch?v=jiCfGTEY53g&t=397s) Standard software telemetry cannot distinguish between sharpening competence and cognitive deskilling because high usage appears identical on administrative dashboards. - [06:37](https://www.youtube.com/watch?v=jiCfGTEY53g&t=397s) True AI judgment centers on three human decisions executed before accepting output: trusting accurate results without re-litigation, overriding fluent hallucinations, and recognizing when to think unaided. - [09:31](https://www.youtube.com/watch?v=jiCfGTEY53g&t=571s) AI judgment can be objectively quantified through seeded error detection and evaluative judgment comparisons rather than subjective self-assessment surveys. - [09:31](https://www.youtube.com/watch?v=jiCfGTEY53g&t=571s) Work transcripts reveal critical signals of human verification by capturing whether an employee accepts, modifies, or outright rejects raw AI outputs during authentic tasks. - [13:09](https://www.youtube.com/watch?v=jiCfGTEY53g&t=789s) Standalone training seminars are ineffective for developing judgment because calibration requires the genuine stakes, operational constraints, and pressure of real workflow decisions. - [13:09](https://www.youtube.com/watch?v=jiCfGTEY53g&t=789s) Nebius Academy's Evolve Coach captures in-flow developer telemetry to generate terminal-based AI judgment profiles and managerial competence distribution data without disruptive testing. - [15:53](https://www.youtube.com/watch?v=jiCfGTEY53g&t=953s) As foundation models become commoditized, the primary defensible enterprise moat is the human workforce's ability to rapidly recalibrate trust-and-override thresholds as models evolve. - [15:53](https://www.youtube.com/watch?v=jiCfGTEY53g&t=953s) The defining narrative of enterprise AI adoption shifts from whether workers are augmented versus replaced to whether teams are calibrated versus uncalibrated.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=jiCfGTEY53g&t=4s) The Flawed Assumption of AI Adoption

Sharahn McClung, a Learning Engineer at Nebius Academy with over a decade of background in learning and development and career coaching, introduces the core paradox of corporate artificial intelligence initiatives. While enterprises are aggressively rolling out generative AI tools, the prevailing implementation strategy remains fundamentally flawed. Companies operate under a pattern where they actively "encourage speed, measure usage, and assume improvement." However, unlike output speed or seat count—metrics that are easily tracked on administrative dashboards—actual human skill improvement is merely inferred, creating a dangerous capability blind spot.

```mermaid flowchart LR accTitle: Enterprise AI Adoption Gap subgraph Flawed["Standard Enterprise Adoption"] A["Encourage Speed"] --> B["Measure Usage"] B --> C["Assume Improvement"] end subgraph Reality["The True Performance Gap"] C -. "40% Productivity Left on Table".-> D["Talent Strategy Lag"] end ```

This assumption creates a massive disconnect between technological availability and realized value. Citing EY's 2025 *Work Reimagined* study, McClung emphasizes that organizations are leaving up to 40% of prospective AI productivity gains on the table simply because their talent and workforce strategies have failed to evolve alongside the tools. The software has arrived and hypothetical gains have been mapped out, but real-world enterprise execution continues to stall.

- **Primary Mission**: Establish a framework for building AI judgment in the workflow and identify the behavioral signals separating teams that are getting genuinely *better* from those merely getting *busier*. - **The Measurement Trap**: Speed and seat usage can be counted quantitatively, but cognitive improvement cannot be tracked through standard software telemetry alone.

---

[02:39](https://www.youtube.com/watch?v=jiCfGTEY53g&t=159s) The Invisible Cost, Automation Bias, and Skill Decay

The unchecked adoption of AI introduces invisible costs that actively degrade core human capabilities. A 2025 study by research organization METR evaluated experienced software developers performing authentic programming tasks within codebases they already knew intimately. Heading into the benchmark, these engineers anticipated that AI assistance would make them roughly 20% faster. In reality, they were 19% slower—yet remarkably, post-task reflections showed they still *felt* as though they had worked faster because they had offloaded cognitive effort. This illusion of productivity masks a growing concern: according to EY research, over 40% of professionals actively worry their underlying domain skills are eroding through AI reliance.

```mermaid flowchart TD accTitle: The Mechanics of Automation Bias and Skill Decay subgraph Bias["Automation Bias in High-Stakes Tasks"] E1["Model Returns Error"] --> E2["Human Verification Declines"] E2 --> E3["Errors Cluster at Model Weak Points"] end subgraph Decay["The Support Removal Test"] S1["Proficiency with AI Tool"] --> S2["Support Removed"] S2 --> S3["Performance Drops Below Pre-AI Baseline"] end ```

When models provide accurate outputs, relying on them is sensible and cost-effective. The systemic risk emerges when the system fails. Behavioral research reveals that when an AI produces incorrect recommendations, human reliance does not decrease; in many high-stakes environments, reliance actually climbs as professionals fall victim to "automation bias"—the tendency to trust an automated system over personal judgment. Crucially, these errors do not distribute randomly; they cluster precisely at the model's blind spots, exactly where human-in-the-loop verification is most essential.

Furthermore, this degradation persists long after the individual work session concludes: - **Clinician Skill Degradation**: When AI diagnostic assistants were removed from clinical practitioners who used them routinely, both diagnostic performance and self-reported diagnostic confidence fell below their original pre-AI baseline. - **Problem-Solving Controlled Trials**: Participants in large-scale problem-solving trials outperformed control groups while the AI was active, but once the tool was revoked, their standalone capability was significantly worse than that of peers who had never utilized the AI in the first place. - **The Core Learning Test**: Genuine learning endures when scaffolding is removed, whereas uncalibrated AI usage causes human skill to dissipate, leaving the worker weaker than when they started.

---

[06:37](https://www.youtube.com/watch?v=jiCfGTEY53g&t=397s) Beyond the Proxy: Defining AI Judgment in Three Moves

Because heavy adoption looks identical on a telemetry dashboard whether a team is sharpening its edge or suffering cognitive decay, organizations must abandon seat usage as their primary success proxy. "Usage was an easy count, so we counted it," McClung observes, but counting usage mistook an easily trackable metric for true operational capability. The real differentiator between compounding capability and quiet deskilling is *AI judgment*—the deliberate capacity to decide when to engage the model, when to override it, and when to bypass it altogether.

```mermaid flowchart LR accTitle: The Three Moves of AI Judgment subgraph Judgment["Three Core Moves of AI Judgment"] direction TB M1["1. Trust: Accept sound outputs immediately without wasteful re-litigation"] M2["2. Override: Intercept fluent, plausible hallucinations and manually correct"] M3["3. Think Unaided: Recognize when prompting outpaces direct human reasoning"] end ```

To make AI judgment practical and actionable, McClung breaks it down into three specific cognitive moves:

1. **Trust**: Recognizing a robust, accurate answer, accepting it immediately, and proceeding with work. Spending excessive time re-litigating high-quality outputs introduces unnecessary operational waste. 2. **Override**: Intercepting an answer that is stylistically confident, fluent, and plausible, but factually or logically flawed. This demands rigorous verification to challenge the output rather than passively accepting it. 3. **Thinking Unaided**: Recognizing when reaching for an AI tool is an impediment. McClung shares an anecdote about facing the "fear of a blank page," noting that professionals often spend so much effort structuring and articulating an intricate prompt that they have already done all the necessary analytical thinking themselves, rendering the model redundant.

Crucially, all three moves represent human-driven decisions executed before a single line of generated output is approved or merged into production. These capabilities do not ship as native model features, nor do they improve simply by increasing query volume.

---

[09:31](https://www.youtube.com/watch?v=jiCfGTEY53g&t=571s) Measuring Judgment Through Seeded Errors and Work Transcripts

Moving beyond conceptual definitions, AI judgment can be rigorously measured and scored through concrete evaluation frameworks rather than self-reported surveys. Two primary methodologies make this cognitive process visible:

| Measurement Approach | Mechanism | Evaluative Purpose | |:--- |:--- |:--- | | **Seeded Error Detection** | Deliberately inserting known flaws (an inaccurate data point, an unsound code step, an invalid assertion) into AI outputs during live tasks. | Tests whether an engineer catches the flaw or allows it to pass undetected, providing a clean, reproducible score of real-time over-reliance versus active verification over time. | | **Evaluative Judgment** | Presenting two distinct, highly fluent AI responses—one structurally sound, one subtly flawed—and evaluating if the worker can discern the difference. | Measures expert-level nuance and calibration; requires comparing worker output against worked answer keys, reference standards, or expert rubrics across repeated sessions. |

Rather than subjecting employees to synthetic, standalone exams, organizations can extract these evaluation metrics directly from the artifacts generated during daily work. Every AI interaction leaves a detailed digital transcript containing the initial framing, the iterative prompt refinements, the model's raw response, and the user's ultimate action.

The definitive signal of human judgment lives in whether a practitioner decides to **Accept**, **Modify**, or **Reject** the AI output. On a traditional administrative dashboard, accepting a correct answer and blindly accepting a flawed hallucination register identically as positive utilization. However, in cognitive terms, they represent polar opposites. Transcripts make this visible: modifications provide concrete proof of active critical review, while outright rejections demonstrate real-time model overriding.

---

[13:09](https://www.youtube.com/watch?v=jiCfGTEY53g&t=789s) Building Competence in the Flow of Work with Evolve Coach

Standalone training seminars and isolated development courses are fundamentally ill-suited for cultivating AI judgment. While traditional L&D has its place, judgment is forged under the pressure of real constraints: tight project deadlines, authentic business dependencies, and genuine operational costs when errors slip through. Rehearsing judgment in a simulated environment where nothing is truly at stake strips away the exact tension required to internalize calibration. Training must occur directly "at the point of decision" so that cognitive reinforcement attaches to genuine operational choices.

```mermaid flowchart TD accTitle: Evolve Coach Architecture subgraph Telemetry["In-Flow Telemetry"] W["Real Engineering Tasks"] --> T["Terminal Plugin / Transcript"] end subgraph Feedback["Dual-Sided Visibility"] T --> P["Engineer AI Profile (Trust, Override, Unaided Trends)"] T --> D["Manager Dashboard (Team Competence Distribution)"] end ```

To address this challenge, Nebius Academy created **Evolve Coach**, an integrated AI plugin built to track competence rather than mere consumption without interrupting engineering workflows: - **Zero-Disruption Telemetry**: Functions without scheduled exams, formal assessments, or intrusive training modules by analyzing live developer workflows in the background. - **On-Demand AI Profile**: Developers can view their personal AI profile directly within their terminal environment, visualizing longitudinal trends in how they trust outputs, execute overrides, and engage in unaided problem-solving. - **Managerial Competence Distribution**: Leaders gain single-pane-of-glass visibility into how their teams interact with AI, swapping unreliable self-reported sentiment surveys for empirical competency distributions. - **Cross-Project Skill Portability**: Accelerates onboarding by anchoring new engineers in strong verification and override patterns from day one.

---

[15:53](https://www.youtube.com/watch?v=jiCfGTEY53g&t=953s) The Durable Asset: Continuous Team Recalibration

Looking toward the future of enterprise software and workforce capability, McClung argues that frontier foundation models will inevitably commoditize across the industry. When competing organizations possess access to identical model capabilities, access itself ceases to be a competitive advantage. The only defensible, non-commoditized asset is the collective judgment of the human workforce utilizing those models.

```mermaid flowchart LR accTitle: Compounding Capability Cycle A["Individual Recalibration"] --> B["Sharper Code & Design Reviews"] B --> C["Higher Quality Enterprise Output"] C --> D["Organizational Capability Moat"] ```

The truly durable enterprise competency is not static mastery of today's specific model parameters, but rather how rapidly a team can dynamically recalibrate its trust-and-override thresholds when a new model architecture is deployed. Because models continuously shift under the user, this calibration can only be refined directly within daily work. When individual calibration improves, peer reviews become sharper, technical questioning becomes more rigorous, and higher-integrity work ships to customers. The central narrative of organizational AI transformation is no longer whether workers will be "augmented versus replaced," but whether teams are "calibrated versus not"—and the highest-performing organizations will be those that treat calibration as an unbroken, continuous practice.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=jiCfGTEY53g&t=4s) encourage speed, measure usage, and assume improvement — Sharahn McClung

> [06:37](https://www.youtube.com/watch?v=jiCfGTEY53g&t=397s) Usage was an easy count, so we counted it — Sharahn McClung

> AI judgment: knowing when to trust the model, when to override it, and when to think unaided — Sharahn McClung

> [13:09](https://www.youtube.com/watch?v=jiCfGTEY53g&t=789s) at the point of decision — Sharahn McClung

> [15:53](https://www.youtube.com/watch?v=jiCfGTEY53g&t=953s) augmented versus replaced — Sharahn McClung

> [15:53](https://www.youtube.com/watch?v=jiCfGTEY53g&t=953s) calibrated versus not — Sharahn McClung

## Mentioned Sharahn McClung, Nebius Academy, Ai4, EY, Work Reimagined, METR, Evolve Coach

## Video description Sharahn McClung - Learning Engineer, Nebius Academy

Most enterprises can tell you whether their people use AI — but not whether their people are getting any better at it. That gap matters more than it sounds: recent field studies show that fast adoption can quietly erode the human skill underneath, producing more errors on the tasks where models are weakest and measurable skill decay once the AI is taken away. This talk explores how the real enterprise unlock isn't more adoption — it's AI judgment: knowing when to trust the model, when to override it, and when to think unaided. I will discuss why that judgment is best built during real work rather than in standalone courses, and why it should be measured rather than assumed. Attendees will leave with a practical framework for developing AI competence in the flow of work, along with the signals that distinguish a team that's genuinely improving from one that's merely adopting — or worse, deskilling the very capabilities enterprises are trying to grow.

[Source](https://www.youtube.com/watch?v=jiCfGTEY53g)

中文简报

Nebius学习工程师反思AI生产力追踪指标除非企业主动衡量并培养人类判断力而非单纯关注使用量，否则推进企业级AI落地可能会削弱员工的核心技能。企业在推广AI时常将席位数量和任务速度误判为真正的人员能力提升，从而掩盖了自动化偏差以及工具移除后长期技能退化的风险。Sharahn McClung将真正的能力定义为贯穿三个特定动作的AI判断力：信任合理的输出、纠正看似流畅的幻觉，以及在没有工具辅助时独立思考。鉴于传统的培训与发展（L&D）课程无法复刻现实中的利害关系，判断力必须通过日常工作记录和植入式错误测试来进行衡量。像Nebius Academy的Evolve Coach这类工具表明，直接在开发者的工作流程中追踪“接受、修改与拒绝”行为，能够构建起持续且稳固的能力护城河。企业经常将AI席位使用量误认为是实际的技能提升，由于劳动力和人才战略滞后，导致多达40%的预期生产力提升未能实现。 (0:04) 2025年METR的一项研究显示，经验丰富的工程师在使用AI时速度反而慢了19%，尽管他们此前预计会加快20%，这印证了由认知负担外包所导致的生产力假象。 (2:39) 自动化偏差加剧了人类在模型出现失效模式时的依赖，导致错误集中在模型盲区，并在AI工具被移除后造成人类技能退化。 (2:39) 标准的软件遥测技术无法区分专业技能的精进与认知能力的退化，因为高使用量在管理仪表盘上的表现完全一致。 (6:37) 真正的AI判断力集中体现在采纳输出前做出的三项人为决策：无需反复推敲便能信任准确结果、纠正看似流畅的幻觉，以及明确何时需要独立思考。 (6:37)

Reading summary

Sharahn McClung of Nebius Academy argues that enterprise AI strategies mistakenly equate tool adoption and speed with genuine competence, whereas organizational performance depends on building human AI judgment—the ability to calibrate when to trust, override, or bypass model outputs. The Measurement Trap of AI Adoption (0:04) Enterprises frequently track seat licenses and speed metrics on administrative dashboards, assuming that increased software usage automatically translates to improved human capability. Citing EY's 2025 Work Reimagined study, McClung notes that organizations leave up to 40% of prospective AI productivity gains on the table because workforce development strategies have failed to evolve alongside software deployment. Counting software consumption creates an organizational capability blind spot, mistaking high activity for true operational competence. Automation Bias and Measured Skill Decay (2:39) Uncritical AI reliance introduces significant cognitive costs that traditional metrics overlook. A 2025 METR study evaluated experienced developers working on familiar codebases: while engineers anticipated that AI assistance would make them 20% faster, they were actually 19% slower, yet still felt faster because they had offloaded cognitive effort. Furthermore, EY research indicates that over 40% of professionals worry their baseline skills are degrading. When AI tools generate incorrect answers, human reliance often increases rather than decreases due to automation bias—the tendency to favor automated output over personal judgment. These errors cluster around the model's blind spots. Controlled trials show that when AI tools are removed from routine users—such as clinical diagnosticians or complex problem-solvers—their standalone performance and confidence drop below their pre-AI baselines, showing that uncalibrated usage can lead to measurable deskilling. The Three Moves of AI Judgment (6:37) To prevent cognitive decay, McClung argues organizations must measure AI judgment rather than seat adoption. She breaks judgment down into three actionable human decisions executed before an output is accepted: trusting sound outputs immediately to avoid wasteful re-litigation; overriding fluent, plausible hallucinations through active verification; and thinking unaided when the effort to formulate a complex prompt exceeds the time required to reason through the problem directly. These three moves reflect cognitive decisions made prior to merging code or approving work. They cannot be supplied natively by foundation model updates, nor do they improve simply by increasing query volume. Evaluating Judgment Through Workflow Artifacts (9:31) Isolated seminars and synthetic tests fail to cultivate AI judgment because they lack authentic project constraints, deadlines, and real consequences. Instead, judgment can be measured within live workflows using seeded error detection—deliberately inserting flaws into tasks to test if a worker catches them—and evaluative judgment, which asks practitioners to differentiate between sound and subtly flawed model outputs. Everyday AI interactions generate digital transcripts that reveal whether a user chooses to Accept, Modify, or Reject an output. On standard dashboards, accepting accurate code and blindly accepting an erroneous hallucination register identically as positive usage. However, modifications signal active critical review, while rejections demonstrate real-time model overrides. Nebius Academy operationalizes this in Evolve Coach, an engineering plugin that tracks competence distributions and personal AI judgment profiles without interrupting daily workflows. Because frontier foundation models will inevitably commoditize across the industry, raw technological access provides little lasting advantage. McClung concludes that an enterprise's true competitive moat lies in how rapidly its teams dynamically recalibrate their trust and override thresholds as models evolve, making continuous calibration a core operational discipline.

精炼阅读摘要

Nebius Academy学习工程师Sharahn McClung指出，企业单纯追求AI推行速度与席位用量并默认员工能力会自动提升的策略存在根本误区；随着模型技术走向同质化，企业的真实壁垒在于员工在工作流中形成的“AI判断力”与持续校准能力。采纳陷阱：效率幻觉与技能退化 (0:04) 企业在推进生成式AI时普遍陷入“鼓励速度、统计用量、默认能力提升”的模式。McClung援引安永（EY）2025年《Work Reimagined》研究指出，由于人才与劳动力战略未能同步演进，企业错失了高达40%的潜在AI生产力收益。常规管理后台只能记录席位与调用速度，却无法反映认知层面的实质变化。研究机构METR在2025年针对资深软件工程师的测试显示，工程师在熟悉的代码库中原本预计AI能提速约20%，实际耗时却增加了19%；然而在任务完成后，受试者仍主观认为自己效率更高，暴露出认知卸载带来的虚假生产力感。安永数据同时表明，超过40%的受访专业人士担忧过度依赖AI正在侵蚀自身的核心业务技能。更深层的隐患来自“自动化偏差”（Automation Bias）与长期技能退化。行为研究显示，当AI给出错误建议时，高风险环境下的从业者往往不仅不降低信任，反而因盲从自动化而加剧依赖，且错误往往集中爆发在模型的盲区漏洞。更为关键的是，这种负面影响在工具移除后依然持续：临床医生在脱离日常使用的AI诊断助手后，其诊断准确度与专业自信心均跌落至接触AI前的原始基线以下；而在受控的问题解决实验中，一旦撤除AI辅助，长期依赖工具者的独立解决能力亦显著劣于从未接触AI的对照组，证明未经审慎校准的依赖会导致实质性的去技能化。 AI判断力的三项认知动作 (6:37) 后台遥测指标无法区分团队是在强化技能还是在退化，因此组织必须摒弃以使用频率作为成功标准的传统做法。McClung将核心竞争力定义为“AI判断力”（AI Judgment），即审慎决断何时采纳模型、何时予以纠正以及何时完全规避工具的认知能力。这一判断力包含三项必须在成果合并或交付前完成的人类决策动作。第一项是“信任”（Trust）：敏锐识别模型产出的严谨准确方案并果断采纳，避免在高质量输出上过度推敲而造成流程内耗。第二项是“推翻”（Override）：针对表述流畅自信、看似合理却包含事实谬误或逻辑缺陷的模型幻觉，进行严格审查并手动修正。第三项是“独立思考”（Thinking Unaided）：准确识别人工智能何时会成为思维累赘。McClung以克服“面对空白页的写作恐惧”为例指出，员工有时为了编写和调整复杂的提示词（Prompt），实际上已经在脑海中完成了全部构思与分析，此时强行依赖AI反而降低了工作效率。这三项决策动作无法依靠模型自身算法升级而自动获取，也无法单纯通过提升调用量来训练。量化评估：植入错误与工作转录 (9:31) AI判断力可以通过严谨的操作评估框架进行量化，而非依赖员工主观问卷。McClung提出了两类机制：“植入错误检测”（Seeded Error Detection），即在实时工作任务的AI输出中故意注入已知瑕疵（如失实数据、缺陷代码或无效论断），以此测量从业者是盲目放行还是主动拦截，从而获得对抗过度依赖的量化得分；“评估性判断”（Evaluative Judgment），即向员工呈现两套语言流畅但质量参差的方案，对照专家基准或标准答案，考查其识别细微逻辑缺陷的专业水准。这类评估无需脱离实际业务设置模拟考试，而是可以直接从日常数字工作转录（Work Transcripts）中提取痕迹。每一次人机交互均包含任务框架、迭代提示、原始回复和最终操作。在传统后台中，盲目接受错误输出与正确采纳优质结果同样被计为活跃度，但在转录记录中，用户对模型输出做出的“采纳”（Accept）、“修改”（Modify）与“拒绝”（Reject）构成了截然不同的认知证据：实质性的修改记录证明了批判性审阅的存在，而坚决的拒绝则直接体现了对AI输出的实时纠偏。业务流校准与团队能力壁垒 (13:09) 脱离实际业务的孤立培训无法培育AI判断力，因为判断力的建立高度依赖于真实业务约束——紧张的项目交付期限、客观的系统依赖关系以及错误发生时所需承担的运营代价。在缺乏真实后果的模拟环境中演练，会丧失建立校准敏感度所必需的张力。为此，Nebius Academy开发了集成插件“Evolve Coach”，该工具在开发者终端后台运行，无须中断日常开发即可收集遥测数据。工程师可随时在终端调取个人画像，观察自己在信任、推翻与独立思考上的长期趋势；管理层则能获得团队整体能力的客观分布视图，取代流于形式的主观满意度调研。 McClung强调，随着行业发展，前沿基础模型必然会走向同质化，技术接入本身将不再构成核心壁垒。企业真正持久且不可被商品化的护城河，是团队在使用这些模型时所展现的集体判断力与动态校准能力。当底层模型架构持续更替时，能够快速微调“信任与推翻”阈值的团队，将在代码审查、方案设计与交付质量上持续拉开差距。企业转型的核心已由“被替代还是被增强”转向“是否处于持续校准状态”。该演讲的核心结论在于，前沿模型的普及将使单纯的工具使用权失去竞争价值，唯有在日常工作流中持续进行批判性校准的组织才能沉淀真正的技能资产。但在实践层面，如何在非编程的复杂通用业务中以低摩擦方式捕获行为转录，以及如何在防范过度依赖的同时避免过度怀疑引发的审查效率损耗，仍有待企业在具体执行中进一步探索平衡。

Detailed notes — zh

## Background 该视频由 Ai4 频道于 YouTube 平台发布，主讲人为 Nebius Academy 学习工程师 Sharahn McClung。内容重点探讨了企业在 AI 落地过程中普遍存在的认知盲区，指出过度关注工具使用率正悄然导致员工技能退化，并系统介绍了如何在日常工作流中构建、衡量与校准真正的‘AI 判断力’。 ## TL;DR 企业往往将 AI 的使用频次和提速错觉误认为员工能力的提升，导致高达 40% 的潜在生产力被浪费，并诱发严重的自动化偏差与离线技能衰退。真正的核心竞争力在于‘AI 判断力’，即明确何时信任输出、何时果断推翻、何时脱离工具独立思考。Nebius Academy 提出抛弃脱节的脱产培训，转而利用预设错误检测以及日常工作流交互轨迹（接收、修改、拒绝）来无侵入地量化与沉淀判断力，通过 Evolve Coach 等工具实现人机协同的持续动态校准。 ## Atmosphere & style 演讲风格冷静克制、逻辑严密且高度务实。主讲人以犀利的行业数据和认知心理学实验开篇，打破了技术狂热下的‘效率神话’，营造出清醒审慎的警示感；随后层层递进地展开框架与工具演示，展现出资深人才培养专家对工作流细节的深刻洞察，带给听众兼具危机感与实操路径的前瞻体验。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=jiCfGTEY53g&t=4s) 企业在推行 AI 时普遍存在“鼓励速度、衡量使用量并盲目假设产出提升”的误区；安永调研显示，人才战略滞后导致企业损失高达 40% 的 AI 潜在生产力收益。 - [02:39](https://www.youtube.com/watch?v=jiCfGTEY53g&t=159s) 认知负荷外包会制造高效的假象：METR 研究显示资深开发者使用 AI 后的实际效率下降了 19%，主观上却依然坚信自己比平时更快。 - [04:05](https://www.youtube.com/watch?v=jiCfGTEY53g&t=245s) 自动化偏差（Automation Bias）让人们在模型给出错误答案时反而表现出极高盲从，且在最需要人工审查的高风险环节往往核查得最少。 - [04:05](https://www.youtube.com/watch?v=jiCfGTEY53g&t=245s) 过度依赖 AI 带来了切实的技能衰退风险：多项受控实验证实，当 AI 辅助工具被撤除后，受试人员与专业人士的独立解决问题能力甚至跌破初始基准线。 - [06:56](https://www.youtube.com/watch?v=jiCfGTEY53g&t=416s) 调用频次和使用量仅是容易统计的代理指标，企业真正应该追求并衡量的核心能力是“AI 判断力（AI Judgment）”。 - [06:56](https://www.youtube.com/watch?v=jiCfGTEY53g&t=416s) AI 判断力由三项核心前置决策动作构成：信任正确输出以高效推进、果断推翻看似流畅但暗藏瑕疵的错误、清醒识别何时应当开展无辅助的独立思考。 - [09:25](https://www.youtube.com/watch?v=jiCfGTEY53g&t=565s) AI 判断力可通过预设错误检测（Seeded Error Detection）和深度评价判断（Evaluative Judgment）进行客观量化，而非依赖不可靠的员工主观自评。 - [09:25](https://www.youtube.com/watch?v=jiCfGTEY53g&t=565s) 日常工作流留存的交互轨迹（Transcript）包含关键判断力信号：修改（Modify）证明了批判性思考，拒绝（Reject）则是推翻错误输出的现场证据。 - [13:09](https://www.youtube.com/watch?v=jiCfGTEY53g&t=789s) 判断力无法在脱离业务的孤立培训中习得，只有在承受真实业务后果与交付时间压力的一线工作流中才能有效构建与校准。 - [13:09](https://www.youtube.com/watch?v=jiCfGTEY53g&t=789s) Nebius Academy 的 Evolve Coach 插件通过无侵入式追踪交互行为，为工程师提供个人能力画像并为管理者呈现团队技能图谱。 - [15:53](https://www.youtube.com/watch?v=jiCfGTEY53g&t=953s) 底层大模型能力终将走向商品化与同质化，企业持久的竞争壁垒在于团队在新一代模型到来时以极高速度完成动态重新校准（Recalibrate）的能力。 - [15:53](https://www.youtube.com/watch?v=jiCfGTEY53g&t=953s) 未来的 AI 叙事将从简单的“增强还是替代”转变为“已校准还是未校准”，最终胜出的将是把能力校准融入日常工作流的团队。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=jiCfGTEY53g&t=4s) 企业 AI 采用的致命盲区：假设提升而非真正提升 Sharahn McClung 作为 Nebius Academy 的学习工程师（Learning Engineer），开场介绍了自己在人才培养与职业教练领域拥有超过十年的经验，指导过数千名学员掌握技能与工具。在 Nebius Academy 为企业提供 AI 技能提升服务的过程中，他们观察到了一个非常普遍的企业采用现状：企业在推行 AI 时，“鼓励速度（encourage speed）、衡量使用量（measure usage），并理所当然地假设产出获得了改善（assume improvement）”。这种将“使用量”直接等同于“能力提升”的假设造成了巨大的认知鸿沟。速度和使用频次是显性、可量化的指标，但能力的真正改善却往往只是推断出来的。根据安永（EY）发布的《2025 年重塑工作研究》（Work Reimagined study），由于企业的人才战略未能跟上技术步伐，导致高达 40% 的 AI 潜在生产力收益被白白浪费。工具虽然到位，预期的效能提升也已被构想，但实际交付的结果却往往伴随着未被察觉的代价。 - **核心演讲目标**：提供一套在工作流（Flow of Work）中构建“AI 判断力”的实战框架，并掌握识别团队是在“变得更强”还是“仅仅变得更忙”的核心信号。 - **安永 2025 调研数据**：企业因人才战略滞后，丢掉了高达 40% 的 AI 生产力红利。 --- ### [02:39](https://www.youtube.com/watch?v=jiCfGTEY53g&t=159s) 隐形成本：开发者的速度错觉与技能侵蚀 AI 带来的首要代价是隐蔽的，它直接作用于人类底层的基础技能。METR 在 2025 年进行的一项研究针对了一群最不可能放慢速度的人群——在自己极其熟悉的代码库中处理真实任务的资深开发者。在实验开始前，这些资深工程师预测借助 AI 自己能提速约 20%；然而实际测试结果显示，他们的工作速度反而慢了 19%。更耐人寻味的是，在任务完成后，开发者们主观上依然坚信自己比平时更快。这种认知偏差的产生非常符合直觉：当人们把认知负荷外包给工具时，自身由于没有付出同等强度的脑力劳动，就会产生一种轻快、高效的错觉。同时，根据安永 2025 年的调查数据，超过四成（40% 以上）的员工已经开始担忧自己的专业技能正在因过度依赖 AI 而退化。如果管理层仅仅盯着传统的使用量仪表盘，不仅无法察觉这种深层危机，连员工的自我汇报也无法准确反映真实水平。 --- ### [04:05](https://www.youtube.com/watch?v=jiCfGTEY53g&t=245s) 自动化偏差：最需要人工介入处反而最盲从当模型给出正确答案时，依赖 AI 是廉价且合理的；但真正的灾难发生在模型犯错的时刻。研究表明，当 AI 的建议出现错误时，人类对它的依赖并没有相应下降，在很多场景下反而出现了上升。人们往往以极高的自信跟随着系统走向完全错误的结论，这就是所谓的“自动化偏差”（Automation Bias）——人类倾向于相信机器判断胜过自己的判断。更严峻的是，这种偏差在高风险任务中尤为严重。在那些最需要独立复核的环节，人类往往核查得最少。这导致错误并非随机分布，而是高度集中在模型的固有薄弱点上——而这些薄弱点恰恰是企业设置“人机协同”（Human-in-the-loop）防线的唯一理由。 ```mermaid flowchart TD accTitle: 自动化偏差与技能衰退循环 subgraph S1["单次工作流中的偏差"] A["高风险 / 复杂任务"] --> B["模型输出看似流畅但有隐蔽错误"] B --> C["自动化偏差：人类信任机器超过自身判断"] C --> D["在模型最脆弱处放弃人工审查"] end subgraph S2["离开工具后的能力退化"] E["日常高度依赖 AI 辅助"] --> F["剥离 AI 工具"] F --> G["解决问题表现与自信心跌破初始基线"] end D -.-> E ``` 更深层的影响在单次会话结束后依然在持续蔓延。针对临床医生的实验发现，长期习惯使用 AI 助手的医生在工具被移走后，其诊断表现和专业自信心双双跌落到使用该工具之前的基准线以下；另一项大规模受控实验也显示，受试者在使用工具时表现出色，但在工具被撤除后，其独立解决问题的能力甚至弱于从未使用过 AI 的对照组。“拿走辅助支撑后，学习成果是否依然稳固？”如果工具一走技能就随之消散，这就不是能力提升，而是能力衰退。 --- ### [06:56](https://www.youtube.com/watch?v=jiCfGTEY53g&t=416s) 破解指标陷阱：重新定义“AI 判断力”的三大动作单纯的使用量上升，既可能代表团队正在变得更加敏锐，也同样可能掩盖团队正在走向技能退化。使用量仅仅是一个容易统计的代理指标（Proxy），当企业把代理指标当成终极目标时，就已经误入歧途。真正拉开企业差距的不是团队调用 AI 的频次，而是团队决定“何时该用、何时不该用”的决策水平，即“AI 判断力”（AI Judgment）。 AI 判断力不会随着工具的发放而自动生成，Sharahn McClung 将其拆解为三个具体的关键动作： 1. **信任（Trust）**：识别出高质量回答后直接采纳并快速推进，避免在完全正确的输出上反复纠缠、反复验证而造成效能浪费。 2. **推翻（Override）**：面对流畅、自信但实则暗藏瑕疵的回答时，能够保持警惕，进行严谨核实并果断予以推翻修正。 3. **独立思考（Thinking Unaided）**：清醒地意识到何时伸手使用工具本身就是一种错误。例如面对空白页面的焦虑时，很多人在构思 Prompt 的过程中其实就已经完成了全部的核心思考，此时根本不再需要依赖模型。这三项行动的共同本质在于：它们都是人类在批准或应用任何一行生成内容之前，针对模型做出的前置决策。 --- ### [09:25](https://www.youtube.com/watch?v=jiCfGTEY53g&t=565s) AI 判断力的量化方式与工作流留存信号判断力并非不可捉摸的抽象概念，Nebius Academy 提出了两种可落地的量化评估方法与一种无侵入的数据捕获机制： | 评估维度 | 实施机制 | 核心考察目标与考量难点 | |:--- |:--- |:--- | | **预设错误检测<br>(Seeded Error Detection)** | 在真实任务中故意植入已知的错误（如错误数据、逻辑缺陷、不成立的断言）。 | 观察人员能否敏锐捕捉还是直接放行。捕捉到即代表“信任与推翻”正常运作，漏检则精准捕捉到了过度依赖。评分标准统一，可跨周期横向对比。 | | **深度评价判断<br>(Evaluative Judgment)** | 提供两份表面同样流畅可信、但内涵优劣不同的生成结果。 | 考察人员能否分辨出真正优质的答案并清晰阐述理由。需要外部专家标准答案比对，且依赖长期多次交互中的校准表现。 | 针对评价判断无法通过单次考试测出的问题，Sharahn McClung 指出，真实的工作本身就会留下交互轨迹（Transcript）。在与 AI 的日常交互中，每一次会话都忠实记录了用户的提问、提示词优化、模型返回结果以及人类最终的处理方式——**接收（Accept）、修改（Modify）还是直接拒绝（Reject）**。 - **接收（Accept）**：单纯看使用量，接收正确答案与接收错误答案毫无区别，但在判断力上却是截然相反的两种行为。 - **修改（Modify）**：证明使用者进行了深入阅读并展开了批判性思考。 - **拒绝（Reject）**：是人类在关键时刻执行“推翻”动作的现场抓包。 ```mermaid flowchart LR accTitle: 工作流轨迹中的判断力信号映射 A["工作流交互轨迹"] --> B["提示词构建与迭代"] A --> C["最终操作行为"] C --> D["接收 (Accept)"] C --> E["修改 (Modify)<br>批判性思考证据"] C --> F["拒绝 (Reject)<br>推翻决策现场捕获"] ``` --- ### [13:09](https://www.youtube.com/watch?v=jiCfGTEY53g&t=789s) 告别脱节培训：在真实工作流中实时校准（Evolve Coach）判断力无法在脱离业务的独立培训课程中真正建立。课程可以传授“推翻模型”的概念，但在没有任何真实风险和时间压力的假想环境中，学员只是在“不需要判断力的地方排练判断力”。唯有在面对真实的 Deadline、真实的业务后果和真实的决策关口时，学习才能与决策行为深度绑定。 Nebius Academy 据此研发了名为 **Evolve Coach** 的 AI 插件，其设计原则是不设独立考试、不安排脱产测评、不以“培训”姿态打扰工作： - **工程师端（终端交互）**：工程师可在终端按需查看自己的“AI Profile”，实时了解自身在信任、推翻、独立思考维度的能力画像及历史演变轨迹。 - **管理者端（团队仪表盘）**：一揽子呈现团队在真实使用中的能力分布图谱，彻底替代不可靠的主观自评。 - **业务价值**：沉淀可跨项目复用的判断标准，并帮助新入职工程师从第一天起建立健康的 AI 协同习惯。 --- ### [15:53](https://www.youtube.com/watch?v=jiCfGTEY53g&t=953s) 终局演进：从“增强与替代”到“校准与未校准” 在演讲的尾声，Sharahn McClung 强调，所有企业不久后都将拥有能力相当的底层大模型，模型能力本身会沦为基础设施与入场券（Table Stakes）。不可被同质化和商品化的核心资产，是团队对于模型输出的判断与校准能力。真正持久的竞争优势，绝非员工对今天某特定模型的熟练程度，而是当下一代全新模型出现时，团队能够以多快的速度完成重新校准（Recalibrate）。这种动态校准能力只能在模型行为不断变化的工作流一线中打磨形成。当个人的判断力提升，交付质量就会改善；当这种能力在团队内部复合发酵，代码审查会更尖锐、提出的问题会更深刻、交付的业务成果也更坚实。未来的叙事焦点不再是简单的“被 AI 增强还是被 AI 替代”，而是“完成了人机校准还是处于未校准状态”。最终脱颖而出的团队，绝不是某一阶段突击培训最用力的团队，而是那些将能力校准融入日常、从未停止训练的团队。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=jiCfGTEY53g&t=4s) 鼓励速度（encourage speed）、衡量使用量（measure usage），并理所当然地假设产出获得了改善（assume improvement）。 — Sharahn McClung > [00:04](https://www.youtube.com/watch?v=jiCfGTEY53g&t=4s) 由于企业的人才战略未能跟上技术步伐，导致高达 40% 的 AI 潜在生产力收益被白白浪费。 — Sharahn McClung > [02:39](https://www.youtube.com/watch?v=jiCfGTEY53g&t=159s) 当人们把认知负荷外包给工具时，自身由于没有付出同等强度的脑力劳动，就会产生一种轻快、高效的错觉。 — Sharahn McClung > [04:05](https://www.youtube.com/watch?v=jiCfGTEY53g&t=245s) 当 AI 的建议出现错误时，人类对它的依赖并没有相应下降，在很多场景下反而出现了上升。 — Sharahn McClung > [04:05](https://www.youtube.com/watch?v=jiCfGTEY53g&t=245s) 拿走辅助支撑后，学习成果是否依然稳固？如果工具一走技能就随之消散，这就不是能力提升，而是能力衰退。 — Sharahn McClung > [06:56](https://www.youtube.com/watch?v=jiCfGTEY53g&t=416s) 使用量仅仅是一个容易统计的代理指标（Proxy），当企业把代理指标当成终极目标时，就已经误入歧途。 — Sharahn McClung > [09:25](https://www.youtube.com/watch?v=jiCfGTEY53g&t=565s) 单纯看使用量，接收正确答案与接收错误答案毫无区别，但在判断力上却是截然相反的两种行为。 — Sharahn McClung > [13:09](https://www.youtube.com/watch?v=jiCfGTEY53g&t=789s) 在没有任何真实风险和时间压力的假想环境中，学员只是在“不需要判断力的地方排练判断力”。 — Sharahn McClung > [15:53](https://www.youtube.com/watch?v=jiCfGTEY53g&t=953s) 未来的叙事焦点不再是简单的“被 AI 增强还是被 AI 替代”，而是“完成了人机校准还是处于未校准状态”。 — Sharahn McClung > [15:53](https://www.youtube.com/watch?v=jiCfGTEY53g&t=953s) 最终脱颖而出的团队，绝不是某一阶段突击培训最用力的团队，而是那些将能力校准融入日常、从未停止训练的团队。 — Sharahn McClung ## Mentioned Sharahn McClung, Nebius Academy, Ai4, 安永 (EY), Work Reimagined study, METR, Evolve Coach ## Video description Sharahn McClung - Learning Engineer, Nebius Academy Most enterprises can tell you whether their people use AI — but not whether their people are getting any better at it. That gap matters more than it sounds: recent field studies show that fast adoption can quietly erode the human skill underneath, producing more errors on the tasks where models are weakest and measurable skill decay once the AI is taken away. This talk explores how the real enterprise unlock isn't more adoption — it's AI judgment: knowing when to trust the model, when to override it, and when to think unaided. I will discuss why that judgment is best built during real work rather than in standalone courses, and why it should be measured rather than assumed. Attendees will leave with a practical framework for developing AI competence in the flow of work, along with the signals that distinguish a team that's genuinely improving from one that's merely adopting — or worse, deskilling the very capabilities enterprises are trying to grow. [Source](https://www.youtube.com/watch?v=jiCfGTEY53g)

Source quotes

[0:04] Sharahn McClung: 鼓励速度（encourage speed）、衡量使用量（measure usage），并理所当然地假设产出获得了改善（assume improvement）。 [0:04] Sharahn McClung: 由于企业的人才战略未能跟上技术步伐，导致高达 40% 的 AI 潜在生产力收益被白白浪费。 [2:39] Sharahn McClung: 当人们把认知负荷外包给工具时，自身由于没有付出同等强度的脑力劳动，就会产生一种轻快、高效的错觉。 [4:05] Sharahn McClung: 当 AI 的建议出现错误时，人类对它的依赖并没有相应下降，在很多场景下反而出现了上升。 [4:05] Sharahn McClung: 拿走辅助支撑后，学习成果是否依然稳固？如果工具一走技能就随之消散，这就不是能力提升，而是能力衰退。 [6:56] Sharahn McClung: 使用量仅仅是一个容易统计的代理指标（Proxy），当企业把代理指标当成终极目标时，就已经误入歧途。 [9:25] Sharahn McClung: 单纯看使用量，接收正确答案与接收错误答案毫无区别，但在判断力上却是截然相反的两种行为。 [13:09] Sharahn McClung: 在没有任何真实风险和时间压力的假想环境中，学员只是在“不需要判断力的地方排练判断力”。 [15:53] Sharahn McClung: 未来的叙事焦点不再是简单的“被 AI 增强还是被 AI 替代”，而是“完成了人机校准还是处于未校准状态”。 [15:53] Sharahn McClung: 最终脱颖而出的团队，绝不是某一阶段突击培训最用力的团队，而是那些将能力校准融入日常、从未停止训练的团队。

Source quotes

[0:04] Sharahn McClung: encourage speed, measure usage, and assume improvement [6:37] Sharahn McClung: Usage was an easy count, so we counted it Sharahn McClung: AI judgment: knowing when to trust the model, when to override it, and when to think unaided [13:09] Sharahn McClung: at the point of decision [15:53] Sharahn McClung: augmented versus replaced [15:53] Sharahn McClung: calibrated versus not

Original transcript

[0:04] My name is Sharon McClung and I am a learning engineer at Nebius Academy. And I am really [0:11] excited to present this topic beyond adoption, building AI judgement in the flow of work. [0:18] For over 10 years, I worked in learning and development and as a career coach and I have [0:23] guided thousands of students and clients to finding the right tools and to learning skills, [0:32] maximizing their strengths to elevate their careers, to make a positive impact on their team [0:38] and to advance their careers. And I'm really happy to share the work that we've been doing at Nebius [0:42] Academy with you. So, what I want you to take away today, two things. One, a practical framework [0:52] for how to build AI judgement in the flow of work. And two, understanding the signals so [1:00] that you can tell the difference in between a team whose usage and AI adoption is making them better [1:06] and a team whose usage and AI adoption is making them busier. At Nebius Academy, [1:13] we sell AI upskilling to enterprises and this gives us a very unique vantage point. We get to [1:20] see up close how companies are adopting AI. And for the most part, this is what we see. [1:30] When companies adopt AI, they encourage speed, they measure usage, and they assume improvement. [1:39] The third is very different from the first two. The first two, you can count and measure, [1:43] right? You can see how fast, how much is being produced in the output and you can see um [1:50] if people are using it. But, the third, that's inferred. [1:55] And that's an assumption that is a gap. And according to EY's 2025 Work Reimagined study, [2:04] that gap looks like leaving up to 40% of productivity gains from AI usage on the table [2:12] because companies haven't advanced their talent strategy to keep pace. [2:19] And so, to be clear, the tools have arrived, and gains in productivity have been imagined, [2:25] but the results That's usually where companies come to get us because there is a cost. So, [2:31] I'd like to spend some time to look at the research that shows us what those costs are. [2:39] All right. So, the first one is a cost that is invisible. Um and it's it lands squarely on the [2:46] skills underneath. This is from uh a study by Meter in 2025. Um and consider here that they [2:56] studied the people who were least who least expected to slow down. They were experienced [3:01] developers, and they were doing real tasks in code bases that they knew really well. So, going [3:07] into the study, they thought they'd be about 20% faster. And in actuality, they were 19% slower. [3:14] And even after the fact, they still felt like they thought they had been faster. [3:19] It makes sense, right? When you have a task and [snorts] you export the effort, it may seem like [3:24] you're not expending that effort because you're not, and it might be going faster. So, um this is [3:30] emerging evidence, so let's take that as a strong signal. Now, you might reasonably say, "Fine, [3:36] ask the people doing the work. They'd notice." And in aggregate, they do. Because according to the [3:41] same study, EY 2025, more than four in 10 people worry that their skills are eroding with AI usage. [3:52] So, with usage, if we have a dashboard and we're following usage, it's not going to [3:57] show what the actual problem is. And no amount of self-report is going to uncover it, either. [4:05] So, this is an example of a cost of AI when it's working. Let's look at what it looks like when [4:10] it's not working. So, current research tells us that when the model is right, [4:16] leaning on it is cheap and sensible. That's the deal, right? working as it's intended. [4:22] The problem is what happens when it's wrong. And here the research tells us something [4:26] we would not predict. When AI recommendations are incorrect, our reliance on them does not drop. [4:34] In several settings, it actually climbs. So, we follow the system most confidently into [4:40] exactly the answers that it gets wrong. And that's what researchers call automation bias, [4:46] our tendency to trust the machine over our own judgment. [4:53] And what's interesting is it's worse with high-stake tasks. Tasks where independent [4:58] verification matters most are the tasks where we do it least. So, you put those two things [5:03] together and you get a real understanding of the risk. The errors do not do not land at random. [5:10] They cluster at a model's weak point. And we're precisely the points where the human check was [5:16] the whole reason to keep the human in the loop. So, the tool is least reliable exactly where it is [5:23] we are least likely to challenge it. So, that's an example of the cost inside a single session, [5:29] but it doesn't end when a session ends. This is when the continues after the work is finished. [5:37] And we see the same pattern in two They're very different settings. The first, [5:42] clinicians who used AI assistants routinely were measured once it was taken away, [5:47] both their performance and their confidence dropped below the where they started. Second, a [5:53] large controlled study of people solving problems found the same thing from the other direction. [6:00] They were better with the tool in front of them, but they were worse with the control group once [6:05] it was gone. Better with the tool, weaker without it, and weaker than before they ever touched it. [6:16] So, here's the test worth applying for anything that we're calling learning. [6:22] What happens when the support is taken away? With learning, it holds up. But with this, [6:28] it dissipates. When the tool goes, the skill goes with it, and we're left weaker than we started. [6:37] And this is where it becomes a measurement problem rather than a technology problem because nothing [6:43] on a usage dashboard can tell the two apart. Both of them look like heavy, healthy adoption. [6:52] Which forces the conclusion about what we've been measuring. [6:56] If piling on more adoption is quietly costing us skill, then adoption was never the goal. [7:04] It was just a proxy, and we mistook the proxy for the thing itself. Usage was an easy count, [7:11] so we counted it. Counting it felt like progress, makes sense. But a rising usage line is equally [7:16] consistent with a team that's getting sharper and a team that is slipping. It can't tell [7:22] the two apart. So, on its own, it was not the measurement we really wanted. So, what's the fix? [7:30] AI judgment. [snorts] Not how often a team reaches for AI, [7:36] but how well they decide when to reach for it and or when not to. That's the variable that [7:44] separates a team compounding its skill from one quietly eroding it. And unlike usage, [7:52] it doesn't arrive automatically with the tools. All right, so judgment can sound a little [7:57] amorphous, so let's break it down here. Make it more compre- concrete. AI judgment in three moves. [8:03] One, trust. A good answer. Take it. Move on. Because re-litigating it over and over again, [8:11] make sure that it's right and that it works, is a kind of waste on its own. Two, override. [8:19] Now, that's a challenge. Why? And this was mentioned with our speakers before. [8:24] Nice, confident, fluent answer, it looks right, but it's not, and we have to do [8:29] our diligence to check, to verify if it is correct or not. And lastly, thinking unaided. [8:36] Recognizing when reaching for the tool was the mistake itself. I don't know about you, but I [8:42] personally have fear of a blank page. Does anyone relate to this when you sit down to do something? [8:47] Where to get started, right? And sometimes I find by the time that I write the prompt, [8:52] I've done all of my process and all of my thinking in the prompt, and I don't actually need the tool. [8:57] Thinking unaided. Right? So, notice what all these three things have in common. [9:03] Each is a decision about the model made by a human and before a single line of output [9:12] is approved or used. None of these things ship as a feature of the model, [9:18] and none of them improve just because usage goes up, which is exactly why judgment and not adoption [9:25] is the thing to build. Now, question. Judgment, can you actually measure it? [9:31] I have two examples here of why and how we can measure judgment. And these are um these are [9:38] worked examples that can be worked through to a score. So, the first is seated error detection. [9:44] We take a real task and deliberately plant a known error in what the AI returns. A wrong figure, [9:52] a flawed step, a claim that doesn't hold, and we watch. Does a person catch it? [9:59] Or does it sail past them? Catching it is trust and override working exactly as they should. [10:07] And missing it is over-reliance caught in the moment and on an item we've chosen in advance. [10:14] And because we chose it in advance, we can score it cleanly. The same way every time, [10:21] and we can compare one week against another. Now, the second is deeper evaluative judgment. [10:31] Whether someone can tell a genuinely strong answer from one that merely sounds right. [10:38] So, two responses, both fluent, both plausible, one is sound and one subtly not so much. [10:45] Can they tell which? And can they articulate why? Now, this evaluative judgment is closer to what [10:54] expert actual expert work demands. And it's also harder to measure and it's worth naming why [11:00] because that measurement cost. To score evaluative judgment, you need something from outside the [11:06] session to compare against. You need an expert grade, a worked answer key, a reference standard. [11:13] And there's a second task cost, too, and that's calibration. Learning to trust the model the [11:18] right amount and the habits that come with it. And that doesn't form from a single sitting. It [11:25] surfaces across repeated sessions. So, you cannot do evaluative judgment with a one-off assessment. [11:31] You need to the work itself over time. Now, that may seem like an obstacle, but as it turns out, [11:39] it's not that much of an obstacle because the work already leaves a record. If judgment, [11:46] real judgment, appears in real decisions, then real work is the only place to measure it. [11:56] So, we don't need a separate exam. And with our AI sessions, every session is a transcript. [12:03] It says what the person asked, how they framed or refined a prompt, what the AI returned, [12:11] and what they actually built with it. And the real tell, whether they accepted the answer as is, [12:20] modified it, or rejected it outright. So, that accept, modify, reject, that signal is judgment [12:28] made visible. Think back to the dashboard problem, usage. Accepting a correct answer and accepting [12:36] a wrong answer look identical from a usage count. But, they are opposite acts of judgment. [12:43] And the transcript is what lets us separate them. So, modifying is the evidence of [12:49] someone who read and thought critically, and rejecting is override caught in the moment. [12:56] So, gathered across a team and across time, this stops being anecdotes and becomes something you [13:02] can actually score. So, now that we know what to build, and we know where the signal lives, [13:09] the question is where to build it. And it's not in a standalone course sitting off to the side. [13:15] It's in our projects. And I want to be precise about this because I don't want to knock courses. [13:21] Items not uh we still need courses, we still need learning and development, but in this situation, [13:26] judgment shows up in real decisions, real tasks, real deadlines, a real cost of being wrong, [13:32] and a course can teach the concept of overriding a model, but what it can't do is hand you what you [13:39] need in the moment when overriding the model is exactly actually what matters [13:45] because something real is on the line. Remove the reality and you are merely rehearsing judgement [13:52] in the place where none is required. The real moments live in the flow of work. So, [13:58] the training has to happen there, too. Close enough to the decision [14:03] that the lesson attaches to a real choice and not a hypothetical one, which means learning [14:09] can be something other than the thing that people go to and it becomes something that happened in [14:16] the work continuously at the point of decision as practice. And that is what we are building with [14:26] Evolve Coach. Evolve Coach is an AI plugin that measures AI judgement competence, not consumption. [14:35] And from the work that's already being done, everything I just described is what has gone into [14:40] the design. No exam, no scheduled assessment, nothing that announces itself as training. [14:47] Engineers see their level, we're calling that an AI profile, on demand in terminal, [14:53] and they know how they're trusting, how they're overriding, how they're working unaided, and how [14:59] that's moving over time. It's their signal about their practice. We also have a dashboard. Managers [15:06] see this dashboard that reveals how the team works with AI in one picture. Competence distribution [15:14] across the team from real usage and not from self reports alone. So, with Evolve Coach, [15:22] you can see how your team works with AI help engineers build AI working judgement that carries [15:30] across projects and helps new engineers ramp up with strong patterns from day one. [15:41] And have the QR code here for those of you who clicked on it. We are still looking for a few [15:46] design partners. So if you're interested, please snap away. So I want to leave you with [15:53] why this matters beyond any one tool. Before long everyone will have the same models. [16:02] And the capability itself will become the table stakes. The variable that won't be commoditized [16:08] is whether your team knows when to trust the output. And that's not a property of the tool. [16:14] That's a property of the team using it. And it matters wherever the work happens. [16:20] And the truly durable asset won't be what people know about today's model. It will be [16:28] how fast they recalibrate to the next one. And that's a capability that you can only build in [16:35] the flow of work because that's the only place the model's behavior is actually changing under them. [16:42] And these capabilities compound. When one person's judgement improves, their work gets better. [16:48] When it improves across a team, the team's judgement improves. And the review is sharper, [16:54] the questions are better, and better work makes it out the door. Individual calibration is what [16:59] organizational capability is actually made up. So the story stops being augmented versus replaced [17:06] and becomes calibrated versus not. And the teams that pull ahead won't be the [17:11] ones who trained the hardest. They will be the ones that never stopped training. Thank you.