QualityAI CEO Says Machine-Speed Testing Unlocks Enterprise AI Scale

Operational AI Is The New Competitive Advantage With QualityAI — Ai4

Ai4

Enterprises fail to capture returns from advanced models because linear human review creates an operational trust bottleneck that only continuous automated assurance can resolve.

AI-assisted summary of Operational AI Is The New Competitive Advantage With QualityAI — original source. Check the original for full context.

Source published: 2026-08-16

Detailed notes

## Background Published on YouTube by Ai4 on August 16, 2026, this keynote features Andrew Duncan, CEO of QualityAI. According to the page description, the session addresses why trust has become the defining challenge of the AI era and outlines how organizations can transition from experimentation to deploying AI with confidence at scale using Continuous AI Assurance.

## TL;DR Andrew Duncan argues that enterprise competitive advantage no longer hinges on possessing the most advanced AI models, but on deploying AI with verifiable operational confidence. As autonomous agents embed deeply into business workflows, the widening gap between exponential AI capability and linear human governance creates severe operational risks and stalls value realization. To resolve this trust gap, Duncan details a three-stage Continuous AI Assurance framework—Define, Assure, and Certify—that evaluates systems at machine speed across technical efficacy, business ROI, and safety guardrails. He concludes by urging enterprise leaders to shift their mindset from exploring what AI can do to proving what AI has demonstrated it can be trusted to do.

## Atmosphere & style The presentation carries a polished, authoritative, and pragmatic keynote tone aimed squarely at enterprise executives. Duncan speaks with steady, measured conviction, deconstructing industry hype to address real organizational anxieties around autonomous agent risks. Using evocative analogies like Blade Runner baseline testing and banking compliance models, the talk creates a focused, solutions-oriented atmosphere that transforms daunting governance challenges into a clear, methodical engineering discipline.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=4s) The defining challenge of the current AI era is shifting from raw model intelligence and benchmark capabilities to building verifiable operational trust. - [01:38](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=98s) The greatest enterprise value will be captured not by organizations with the best AI models, but by those capable of deploying AI with confidence at scale. - [01:38](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=98s) Operational AI requires balancing three confidence pillars: technical efficacy, measurable business outcomes, and risk and responsibility boundaries. - [03:38](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=218s) As AI embeds directly into enterprise operating layers and agentic workflows, it exposes organizations to expanded risks including hallucinations, decision failures, model drift, and governance gaps. - [05:39](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=339s) Lack of empirical measurement and governance leads to heavy manual reviews, creating a value blockade that slows adoption and destroys automation economics. - [08:46](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=526s) A structural trust gap is widening because AI capabilities expand exponentially while human oversight capacity grows linearly, necessitating machine-speed AI assurance. - [11:25](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=685s) Continuous AI Assurance relies on a three-stage framework—Define, Assure, and Certify—to continuously measure AI behavior across technical, business, and trust lenses. - [11:25](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=685s) Human accountability remains paramount to establish operational guardrails, define business outcomes, and intervene when confidence thresholds drop. - [13:36](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=816s) Organizations should adopt practical pathways based on maturity, transitioning from foundational AI Readiness Assessments to Managed Continuous Assurance on high-impact workflows. - [16:05](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=965s) Similar to baseline testing in Blade Runner, AI systems need continuous operational baselining, shifting enterprise focus from asking what AI can do to proving what it can be trusted to do. - [17:59](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=1079s) Stakeholder skepticism toward AI is driven primarily by an evidence and knowledge gap rather than inherent bias, and can be resolved through clear, empirical performance data.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=4s) Introduction to QualityAI and the Shift to Trust

Andrew Duncan introduces QualityAI as a highly profitable, billion-dollar quality engineering and AI assurance business with over 25 years of testing heritage. The firm operates across the entire software development life cycle (SDLC) as well as AI engineering and the engineering of AI, collaborating with both providers and enterprise consumers of AI models to drive automation and efficiency. Duncan emphasizes that while QualityAI is a sector-led enterprise providing rare end-to-end testing across the full AI landscape, its ultimate mission is making software and solutions trustworthy—positioning AI as the latest critical frontier in that journey.

Reflecting on the evolution of AI, Duncan notes that the initial era was dominated by raw intelligence, model capabilities, benchmark performance, and scale. AI has made extraordinary progress in architecting, designing, and writing code, generating content, assisting decisions, and running core business processes. However, executive leadership now demands measurable productivity, faster innovation, and sustained return on investment. Having AI capability alone is no longer enough; organizations need empirical evidence indicating where AI can act, under what conditions, and with what level of confidence.

- **QualityAI Profile**: Billion-dollar, highly profitable AI assurance and quality engineering business with a 25-year testing track record. - **Market Shift**: Moving from pure model intelligence and benchmark hype to verifiable business value, operational boundaries, and trust.

---

[01:38](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=98s) Operational AI and the Confidence Imperative

Duncan defines "Operational AI" as the disciplined deployment of AI within enterprise processes backed by clear standards, continuous evidence, and human accountability. While access to powerful AI platforms has broadened and commoditized, organizational confidence in deploying AI at scale remains patchy. Duncan argues that most enterprises currently possess more AI capability than they can effectively convert into business value.

The organizations that realize the greatest value will not necessarily be the ones with the best AI models, but rather those capable of deploying AI with confidence at scale. Duncan breaks down confidence into three distinct pillars: proving technical efficacy, demonstrating intended business outcomes, and maintaining operations within acceptable risk, responsibility, and human intervention thresholds. Enterprises that establish these proofs can move faster, expand use cases, reduce manual reviews, and make sharper risk decisions.

```mermaid flowchart TD accTitle: The Operational AI Framework subgraph Foundation["AI Capability Engine"] A["Enterprise AI Models & Tools"] B["Broad Access & Automation"] end

subgraph Assurance["Continuous AI Assurance"] C["Technical Efficacy Checks"] D["Business Outcome Metrics"] E["Risk, Safety & Boundaries"] end

subgraph Outcome["Operational AI at Scale"] F["Operational Confidence"] G["Sustainable Business Value"] end

Foundation --> Assurance Assurance --> Outcome ```

---

[03:38](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=218s) Enterprise Risk Surfaces and the Agentic Operating Layer

AI has evolved from isolated project sandboxes into a standard enterprise operating layer, embedded directly inside productivity suites, developer environments, customer platforms, and operating systems. Duncan jokes about earlier session discussions where audience members worried about job security, calling AI "the fastest employee in the workplace" due to the unprecedented speed and volume at which it produces code and content. As systems become agentic, AI moves beyond passive assistance to actively orchestrating workflows, triggering actions, and making decisions across multiple systems and platforms.

Operating at this systemic layer exposes organizations to a much broader risk surface, impacting customer relations, workforce dynamics, financials, regulatory compliance, and brand reputation. Duncan points out emerging operational risks, including base and incremental hallucinations, autonomous decision failures, governance gaps, cybersecurity vulnerabilities, model drift, bias, and intellectual property exposure. Because manual review cannot keep up with machine-speed execution, AI must be employed to monitor and assure AI, while keeping final accountability with human leadership.

- **Expanded Threat Vector**: Base and incremental hallucinations, autonomous decision failures, data drift, bias, IP leakage, and governance gaps. - **Operating Discipline**: Transitioning from ad-hoc project governance to continuous enterprise-wide assurance.

---

[05:39](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=339s) The Trust Gap and the Value Blockade

Duncan explains that continuous AI assurance serves as the engine converting raw AI capability into organizational confidence. Trust cannot rely on brand reputation, a successful sandbox pilot, or a one-time pre-deployment test; it requires dynamic, current evidence generated in live operating environments across models, data, workflows, and controls.

When governance and measurement are absent, confidence collapses into distrust, locking enterprise business value behind an unwillingness to let AI run autonomously. Without empirical measurement, organizations default to heavy manual review, which slows adoption and destroys the economic benefits of automation.

```mermaid flowchart LR accTitle: The Trust Deficit Cycle A["Weak Measurement & Governance"] --> B["Lack of Confidence"] B --> C["Lack of Trust"] C --> D["Heavy Manual Review"] D --> E["Slow Adoption & High Costs"] E --> F["Locked Business Value"] ```

---

[08:46](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=526s) The Widening Structural Trust Gap

Duncan presents a chart illustrating a structural divergence: AI systems are expanding rapidly along an exponential curve of capability, autonomy, and business impact, while human governance capacity grows at a flat, linear rate constrained by scarce time, budget, and talent. The widening space between these two curves is the "trust gap."

Every new autonomous agent introduces additional interactions, potential failure points, and decision paths requiring oversight. Duncan explains that machine-speed operations demand machine-speed assurance: AI must continuously evaluate signals and detect anomalies, while human operators retain policy-setting, innovation, and intervention responsibilities. Duncan notes that this dynamic is why QualityAI is experiencing a "tsunami of demand," dispelling advisor skepticism that AI advances would make testing businesses obsolete. Accelerating software creation makes rigorous technical, business, and drift assurance even more critical.

```mermaid flowchart TD accTitle: Machine Speed Governance subgraph Signals["Live Environment Signals"] S1["Agent Workflows"] S2["Data & Prompts"] S3["Model Decisions"] end

subgraph AutoAssurance["Machine-Speed AI Assurance"] M1["Continuous Monitoring"] M2["Change & Drift Detection"] M3["Evidence Assembly"] end

subgraph HumanGov["Human Accountability"] H1["Policy & Guardrails"] H2["High-Risk Interventions"] end

Signals --> AutoAssurance AutoAssurance --> HumanGov ```

---

[11:25](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=685s) The Three Pillars of Continuous AI Assurance

To close the trust gap, Duncan details a three-stage Continuous AI Assurance model:

1. **Define**: Sets and continuously updates operational standards. "Define agents" maintain a live view of business objectives, policies, guardrails, risk tolerances, and success criteria as products and organizational priorities evolve, preventing standards from becoming static documents. 2. **Assure**: Tests live behavior against the defined standards. "Assure agents" ingest production signals, agent behaviors, and workflow outcomes to build cumulative empirical evidence of technical reliability, business value, and risk containment. 3. **Certify**: Evaluates operational readiness to determine whether an AI system can run, scale, or requires human intervention. "Certification agents" synthesize evidence across three evaluation lenses.

Duncan stresses that human accountability remains paramount throughout this process: people establish the core objectives, approve operating boundaries, and step in whenever evidence indicates confidence has dropped below acceptable levels.

---

[13:36](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=816s) Practical Pathways: Maturity Assessment and Managed Assurance

Duncan outlines two implementation approaches based on organizational maturity:

For enterprises earlier in their journey, an **AI Readiness Assessment** benchmarks current maturity, uncovers blind spots and governance risks, and builds a prioritized roadmap across governance, data, technology, and trust before scaling deployments. For mature organizations, **Managed Continuous Assurance** provides real-time validation, proactive risk detection, and continuous confidence scoring—similar to algorithmic AML (anti-money laundering) and KYC (know your customer) scoring in banking.

Duncan advises leaders not to "boil the ocean." Instead, organizations should prioritize business-critical workflows where confidence unlocks measurable returns while mitigating substantial risks. Crucially, leadership must engage stakeholders early to establish and agree upon clear trust criteria and evidence thresholds before initiating deployments.

- **Maturity Spectrum**: - *Emerging*: Readiness assessment, gap/risk identification, sequential foundational roadmap. - *Advanced*: Managed continuous assurance, active anomaly detection, automated confidence scoring. - **Implementation Strategy**: Focus strictly on high-impact, business-critical use cases; establish explicit stakeholder-approved evidence requirements.

---

[16:05](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=965s) Changing the Prompt: From Capability to Proven Trust

In his concluding remarks, Duncan draws an analogy to the film *Blade Runner*, where replicants must undergo a baseline psychological and physiological test every morning—evaluating answers, voice vibrations, and behavioral cues—to verify they remain within operational boundaries. AI systems require the same continuous baselining to prove ongoing technical stability, business alignment, and adherence to safety constraints.

Duncan urges leaders to "change the prompt" from asking *"What can AI do?"* to demanding *"What has AI proven it can do?"* Operational AI rooted in continuous assurance will be the true differentiator separating sustained enterprise success from stalled experimentation.

> "The first era of AI was based on intelligence. The next era is going to focus on trust... Leaders need to change their prompt. They need to move away from asking what can AI do towards asking what has AI proven it can do."

---

[17:59](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=1079s) Q&A: Overcoming Stakeholder Skepticism and Bias

An audience member asks whether stakeholder hesitation toward AI-labeled products stems from inherent negative bias and how organizations can overcome that perception.

Duncan responds that while skepticism is widespread due to alarming headlines, the resistance is driven primarily by a lack of understanding rather than deep-seated bias. Many stakeholders do not yet know what specific evidence or metrics they need to see in order to feel secure. In his experience at QualityAI, unsubstantiated claims—both overly optimistic and overly fearful—stem from misunderstandings of context, which can be resolved by providing clear, empirical evidence of performance and risk controls.

## Notable quotes > [01:38](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=98s) The organizations that realize the greatest value from AI will not necessarily be the ones with the best AI. They will be the ones that can deploy AI with confidence at scale. — Andrew Duncan

> [00:04](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=4s) The challenge is no longer intelligence — it is trust. — Andrew Duncan

> [16:05](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=965s) The first era of AI was based on intelligence. The next era is going to focus on trust. — Andrew Duncan

> [16:05](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=965s) Leaders need to change their prompt. They need to move away from asking what can AI do towards asking what has AI proven it can do. — Andrew Duncan

> [01:38](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=98s) Operational AI is the disciplined deployment of AI within enterprise processes backed by clear standards, continuous evidence, and human accountability. — Andrew Duncan

> [01:38](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=98s) Most enterprises currently possess more AI capability than they can effectively convert into business value. — Andrew Duncan

> [03:38](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=218s) AI is the fastest employee in the workplace. — Andrew Duncan

> [08:46](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=526s) Machine-speed operations demand machine-speed assurance. — Andrew Duncan

## Mentioned Andrew Duncan, QualityAI, Blade Runner, Ai4

## Video description Andrew Duncan - Chief Executive Officer, QualityAI

The organizations that realize the greatest value from AI will not necessarily be the ones with the best AI. They will be the ones that can deploy AI with confidence at scale.”.

AI is advancing at an unprecedented pace. Models are becoming more capable, agents more autonomous, and AI is increasingly writing code, generating content, making decisions, and participating in core business processes. Yet despite billions invested in AI, many organizations remain hesitant to deploy it at scale.

The challenge is no longer intelligence — it is trust.

As AI becomes embedded in enterprise operations, organizations face a new class of risks, including hallucinations, autonomous decision failures, governance gaps, security vulnerabilities, model drift, bias, and intellectual property exposure. At the same time, business leaders are under pressure to accelerate innovation, improve productivity, and realize measurable value from their AI investments.

In this keynote, Andrew Duncan, CEO of QualityAI, will explore why trust is emerging as the defining challenge of the AI era. He will discuss how the enterprise focus is shifting from building AI to operationalizing AI, why quality must evolve from a software engineering discipline into a business assurance capability, and how Continuous AI Assurance can help organizations deploy AI responsibly and confidently at scale.

Attendees will gain practical insights into how leading enterprises are balancing innovation with risk, accelerating adoption while maintaining control, and building the operational confidence required to move AI from experimentation to business-critical operations.

The first era of AI was about intelligence. The next era of AI will be about trust. To unlock the full value of AI, enterprises must change the prompt — from asking what AI can do to proving what AI can be trusted to do.

[Source](https://www.youtube.com/watch?v=GdZ4BCQF4xY)

中文简报

QualityAI首席执行官表示机器速度测试可释放企业AI规模化潜力企业之所以未能从先进模型中获取回报，是因为线性的人工审查造成了运营信任瓶颈，而这一瓶颈唯有通过持续的自动化保障才能解决。 Andrew Duncan认为，企业的竞争优势已不再取决于拥有最先进的AI模型，而在于能否以可验证的运营信任度部署AI。随着自主智能体深度嵌入业务工作流，呈指数级增长的AI能力与呈线性的人工治理之间不断拉大的差距，带来了严重的运营风险并阻碍了价值实现。为弥合这一信任鸿沟，Duncan详细阐述了一个涵盖“定义（Define）、保障（Assure）和认证（Certify）”的三阶段持续AI保障框架，该框架以机器级速度在技术成效、业务ROI以及安全护栏等方面对系统进行评估。他最后敦促企业领导者转变思维模式，从探索AI“能做什么”转向证明AI“已展示出哪些值得信赖的能力”。当前AI时代的决定性挑战正在发生转变，从单纯追求原始模型智能和基准能力，转向建立可验证的运营信任。 (0:04) 捕获最大企业价值的不会是拥有最佳AI模型的组织，而是那些能够充满信心地规模化部署AI的组织。 (1:38) 运营级AI需要平衡三大信任支柱：技术成效、可衡量的业务成果，以及风险与责任边界。 (1:38) 随着AI直接嵌入企业运营层和智能体工作流，组织暴露于更多风险之下，包括幻觉、决策失误、模型漂移以及治理缺口。 (3:38) 缺乏实证衡量与治理会导致繁重的人工审查，进而形成价值阻碍，不仅延缓技术采纳，还会破坏自动化的经济效益。 (5:39)

Reading summary

Andrew Duncan of QualityAI argues that enterprise competitive advantage has shifted from raw model intelligence to 'Operational AI'—the disciplined deployment of AI governed by continuous, empirical assurance and human accountability. The Confidence Imperative in Enterprise Adoption (1:38) Access to foundational AI models has largely commoditized, leaving most enterprises with more raw AI capability than they can safely convert into commercial value. Duncan asserts that organizational value realization does not depend on having the most advanced models, but rather on deploying systems with measurable confidence. This confidence requires verifiable proof across three operational pillars: technical efficacy, tangible business outcomes, and acceptable risk and safety thresholds. Without empirical validation, organizational trust collapses. When leadership lacks clear performance and safety boundaries, enterprises inevitably default to extensive manual reviews. This defensive posture slows adoption, inflates operating costs, and eliminates the productivity and economic gains that automated workflows were intended to provide. Expanding Risk Surfaces in Agentic Operations (3:38) AI systems are migrating out of isolated sandbox pilots and directly into core operating environments, including developer platforms, customer interfaces, and productivity suites. As these systems transition from passive assistants to autonomous agents that orchestrate cross-platform workflows, they introduce substantial systemic exposure across regulatory compliance, workforce stability, brand reputation, and balance sheets. Duncan identifies emerging failure modes that accompany autonomous execution, such as base and incremental hallucinations, autonomous decision failures, data drift, bias, governance gaps, and intellectual property exposure. Because human teams cannot review outputs at the speed of algorithmic generation, traditional ad-hoc governance models fail, necessitating continuous, automated monitoring. Closing the Structural Trust Gap at Machine Speed (8:46) Enterprise governance faces a structural mismatch: AI capability and autonomy expand along an exponential trajectory, while human oversight capacity grows at a linear rate constrained by finite time, budgets, and talent. Duncan terms the widening divergence between these trajectories the 'trust gap,' noting that every added autonomous agent introduces new potential failure points across enterprise workflows. To bridge this divide, organizations must implement machine-speed assurance rather than relying on periodic pre-deployment audits or static brand reputations. Under this model, AI-driven assurance systems monitor real-time production signals, analyze prompt drift, and assemble empirical evidence, while human leadership retains ultimate policy control and the authority to intervene in high-risk scenarios. The Define, Assure, and Certify Architecture (11:25) QualityAI addresses the trust gap through a continuous three-part operational model. First, 'Define agents' establish and dynamically update business objectives, safety policies, and risk boundaries to prevent governance standards from becoming obsolete. Second, 'Assure agents' continuously evaluate live production signals and agent actions to collect ongoing empirical evidence regarding reliability and value delivery. Finally, 'Certification agents' evaluate this evidence through three operational lenses before clearing an agent to act autonomously: a technical lens verifying functional correctness ('Does it work?'), a business lens measuring return on investment ('Does it deliver?'), and a trust and safety lens enforcing guardrails and compliance ('Can we let it run?'). Human operators step in whenever confidence scores drop below defined thresholds. Maturity-Based Implementation and Stakeholder Trust (13:36) Organizations must tailor their assurance strategy to institutional maturity. Early-stage enterprises should conduct baseline readiness assessments across governance, technology, and data to identify blind spots before scaling deployments. Advanced organizations can implement managed continuous assurance, using real-time confidence scoring analogous to anti-money laundering and customer verification controls in banking. Duncan advises against sweeping, enterprise-wide deployments, recommending focused execution on business-critical workflows where automation delivers high returns under clear risk boundaries. Addressing broader stakeholder hesitation, Duncan contends that resistance to AI typically stems from a lack of clear performance metrics rather than deliberate bias, which empirical evidence and transparent controls can resolve. Duncan concludes that enterprises must change their strategic focus from asking what AI can theoretically do to demanding what it has proven it can do within defined boundaries. The primary unresolved challenge remains establishing standardized, cross-industry evidence criteria so stakeholders can objectively verify autonomous operations.

精炼阅读摘要

QualityAI高管Andrew Duncan指出，企业在人工智能领域的竞争壁垒正从单纯的模型能力与基准跑分转向以持续经验证据为支撑的“操作型AI”（Operational AI）。他认为，当前多数企业拥有的AI能力已超出其转化能力，唯有建立涵盖技术有效性、业务价值与风险边界的持续保障机制，才能弥合扩大的信任鸿沟并释放自动化红利。业务价值瓶颈与操作型AI的兴起 (1:38) Duncan指出，早期AI发展由底层模型规模、智能程度与跑分竞赛主导，虽然技术在架构设计、代码生成与内容生产上取得突破，但当前企业管理层更迫切要求看到可量化的生产力提升、更快的创新落地以及明确的投资回报率。Duncan评估认为，多数企业当前拥有的AI能力已超出其能有效转化为实际商业价值的范畴，而阻碍规模化部署的核心阻力在于组织缺乏投产信心。为此，Duncan将“操作型AI”定义为在明确标准、持续证据和人类问责机制支持下，将AI严谨嵌入企业核心流程的运行模式。他主张，未来获取最大商业回报的组织未必拥有最顶尖的模型，而是能在大规模部署中建立充分信心的企业。这一信心依赖于三大支柱的实证闭环：证明技术有效性（系统能否稳定运行）、实现预期业务结果（能否带来可量化价值），以及将运行状态严格控制在可接受的风险、安全与人类干预阈值内。智能体系统性风险与信任鸿沟扩增 (3:38) 随着AI从独立的试验沙盒深入到办公套件、开发者环境和客户交互系统，AI系统正转变为具备跨系统编排、自主触发操作与决策能力的“智能体运行层”。Duncan警告，这一系统性下沉极大拓宽了企业的风险暴露面，可能波及客户关系、劳动力协作、财务运作、法规合规及品牌声誉。具体隐患涵盖基础与增量幻觉、自主决策失灵、模型漂移、算法偏见、数据与知识产权泄露，以及传统治理框架脱节。 Duncan展示的数据模型表明，企业面临着严峻的结构性失衡：AI系统的能力、自主性与影响力正以指数级曲线飙升，而依赖时间、预算和有限人才的人类治理能力仅呈平缓的线性增长，两者之间的差距形成了持续扩大的“信任鸿沟”。他强调，由于人工复核的速度无法匹配机器运行速度，若缺乏机制化度量，组织便会因信任缺失退回重度人工审核，进而拖慢上线周期并直接削弱自动化的经济优势。机器速度治理与三阶段持续保障体系 (11:25) 为了在机器运行速度下实现同等速度的监督，Duncan主张采用“以AI监督AI”的持续保障机制，实时捕获生产环境中的模型、数据与工作流信号，同时由人类把控核心政策边界与高危干预权。QualityAI为此提出了由专门智能体驱动的三阶段闭环架构：第一阶段是“定义”（Define），由定义智能体动态维护业务目标、操作护栏、风险容忍度与成功标准，避免合规要求沦为脱离业务迭代的静态文件；第二阶段是“保障”（Assure），由保障智能体采集生产环境中的实时信号，持续积累关于可靠性与合规性的经验证据。第三阶段为“认证”（Certify），由认证智能体综合多维证据，通过三大评估视角裁定系统是否具备投产、扩容或必须触发人工干预的资格：技术视角检验“它是否有效”，聚焦功能准确性与架构稳定性；业务视角审视“它是否达标”，衡量实际产出与投资回报；信任与安全视角判定“能否放行”，核查风险隔离、合规遵从与伦理标准。Duncan强调，人类在整个体系中拥有最终问责权，负责设定初始目标并在置信度不足时接管系统。成熟度实施路径与循证评估范式转变 (13:36) 在落地策略上，Duncan针对不同成熟度的企业提出了双轨实施路径：对于尚处探索阶段的企业，应通过“AI就绪度评估”排查盲点与治理漏洞，在扩展部署前制定涵盖治理、数据、技术与信任的优先级路线图；对于成熟企业，则建议部署“托管型持续保障”，提供实时验证、主动异常检测和动态置信度评分，类似于金融业在反洗钱（AML）与客户身份识别（KYC）中运用的算法风控模型。他建议管理者切忌盲目追求面面俱到，应聚焦于置信度能直接撬动业务回报且能遏制重大风险的关键流程，并在部署前与利益相关方就证据阈值达成共识。 Duncan援引电影《银翼杀手》中复制人每天接受基线测试以确认其未越界的设定，呼吁管理层改变对AI的考核逻辑——从探究“AI能做什么”转向要求核实“AI已证明它能做什么”。在问答环节，针对外界对AI产品的审慎态度，Duncan澄清称，利益相关方的抵触往往并非出于既定偏见，而是源于对评估指标缺乏了解，只要提供清晰的经验证据与风险控制数据，就能有效消解过度悲观或盲目乐观的情绪。 Duncan明确主张将持续保障作为释放AI商业价值的前提条件，将竞争重心由模型参数转向运行置信度。然而，该架构的有效性高度仰赖用于治理的保障智能体本身的客观性与稳健度；当面对极复杂的长尾场景、深层多智能体级联失效或跨组织合规冲突时，监督算法如何确保自洽，以及人力在有限精力下如何实现实质性接管，仍是该模式在推广中需进一步验证的现实课题。

Detailed notes — zh

## Background 本视频由 Ai4 频道于 2026 年 8 月 16 日在 YouTube 平台发布。视频记录了 QualityAI 首席执行官 Andrew Duncan 的主题演讲，重点探讨了随着企业关注点从构建 AI 转向运营级 AI，信任已成为定义 AI 时代的核心挑战，并系统阐述了如何通过持续 AI 保障（Continuous AI Assurance）实现自信且规模化的业务落地。 ## TL;DR Duncan 指出，AI 时代能捕获最大商业价值的企业并非拥有最强模型的公司，而是能够建立置信度并规模化部署 AI 的组织。随着自主 Agent 深入核心业务流程，AI 爆发式的机器速度与企业缓慢的线性人工治理之间形成了巨大的“信任鸿沟”，导致大量 AI 投资因无法证明受控而被过度审核锁死。为破除这一恶性循环，企业必须以“机器速度的保障匹配机器速度的运营”，借助涵盖“定义、保障、认证”的持续 AI 保障体系与多维评估透镜。最终，企业管理层需转变发问方式，从探索“AI 能做什么”转向验证“AI 已经被证明可以安全做好什么”。 ## Atmosphere & style 整场演讲呈现出典型的高管战略峰会基调，兼具务实深度的工程理性与敏锐的市场前瞻性。Duncan 表达从容笃定，不卖弄技术噱头，而是直击大模型落地受阻的企业痛点，叙事结构逻辑严密且发人深省。演讲中穿插对“AI 是否会终结测试行业”的幽默反讽，并巧妙引入《银翼杀手》的生化人基线测试作为隐喻，让整场专业且严肃的治理议题充满生动鲜活的说服力与掌控感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=4s) QualityAI 专注于软件与 AI 保障（AI Assurance），致力于为 AI 模型提供商与企业消费端提供全链路质量测试，让 AI 解决方案真正值得信赖。 - [01:00](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=60s) AI 发展正从单纯关注参数规模与基准跑分的“智能时代”，全面迈向注重落地边界与运行置信度的“信任与运营时代”。 - [01:00](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=60s) 在 AI 时代获取最大商业价值的企业，不一定是拥有最先进模型的企业，而是能够自信且规模化部署 AI 的企业。 - [01:00](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=60s) 运营级 AI（Operational AI）的定义是将 AI 深度嵌入业务流程，并辅以明确规则、持续证据链以及清晰的人类问责机制。 - [03:38](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=218s) 随着 Agentic 自主智能体深入企业核心架构，增量幻觉、自主决策失控、模型漂移及治理断层等新型风险被成倍放大。 - [05:39](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=339s) 缺乏动态度量与治理会导致企业陷入堆砌人工双重核验、部署成本飙升且落地受阻的恶性循环，锁死 AI 投资回报。 - [05:39](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=339s) 建立企业对 AI 的信任不能依赖一次性上线前测试或概念验证（PoC），而必须依赖生产环境中系统行为的实时、持续证据链。 - [08:46](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=526s) AI 系统的指数级增长与人力治理的线性增长之间形成了“信任鸿沟”，必须用机器速度的自动化保障来匹配机器速度的业务运营。 - [11:25](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=685s) 持续 AI 保障模型包含“定义（Define）”、“保障（Assure）”与“认证（Certify）”三大闭环阶段，将传统软件测试升维为业务保障能力。 - [11:25](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=685s) 评估 AI 运行必须结合三大透镜：技术是否正常工作、业务是否达成预期价值，以及信任与安全边界是否足以支持放行自主运行。 - [13:36](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=816s) 企业落地应分阶段推进：初级阶段通过“AI 就绪度评估”筑牢底线，成熟运营阶段则需建立分级处置风险的“置信度评分”与持续保障体系。 - [13:36](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=816s) 决策者管理 AI 的核心思维提示词必须发生转变：从询问“AI 能做什么”转变为证明“AI 已证明自己能安全做好什么”。 - [17:59](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=1079s) 利益相关者对 AI 的疑虑与偏见通常源于对系统缺乏理解及评估标准缺失，提供客观、量化的运行证据即可有效消解不信任。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=4s) QualityAI 的背景与使命：让软件与 AI 值得信赖演讲伊始，QualityAI 首席执行官 Andrew Duncan 介绍了公司的业务定位与核心能力。QualityAI 是一家年营收达十亿美元、实现高度盈利的质量工程与 AI 保障（AI Assurance）企业，在软件测试领域拥有超过 25 年的深厚积淀。公司的业务不仅贯穿传统软件开发生命周期（SDLC），更深度切入“AI 工程化”（AI Engineering）以及“面向 AI 的工程化”（Engineering of AI），专门与 AI 模型及工具的提供商和企业级应用方合作，推动自动化与运营效率的提升。 Duncan 强调，QualityAI 是一家行业导向型的机构，将深厚的行业洞察与端到端的 AI 质量测试能力相结合。在当前行业中，能够跨越整个 AI 生态系统提供全链路测试的企业极为罕见。公司的根本使命始终是“让软件与解决方案值得信赖（trustworthy）”，而 AI 正是延续这一使命最新、最关键的技术前沿。 - **业务规模与历史**：拥有超过 25 年测试历史，是一家规模达十亿美元且高度盈利的质量工程与 AI 保障企业。 - **服务对象**：同时覆盖 AI 模型/工具的供应端（Providers）与消费端（Consumers）。 - **核心定位**：提供横跨整个 AI 领域的端到端质量保障，致力于消除技术不确定性。 --- ### [01:00](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=60s) 从“智能时代”迈向“信任时代”：何为运营级 AI AI 发展的第一阶段主要由“智能”与“模型认知”所主导，市场普遍聚焦于模型能力、基准测试跑分（benchmark performance）、参数规模以及层出不穷的 AI 创新产品。在过去几年中，AI 的进化速度令人惊叹——它已能独立进行系统架构设计、编写代码、生成内容、辅助决策，并深度参与企业核心业务流程。然而，随着初期狂热退去，企业高管与业务领袖们开始要求看到可量化的生产力提升、更快的创新落地，以及对巨额 AI 投资的持续商业回报。当 AI 真正融入核心关键业务时，仅凭 AI 能力本身已不足以支撑规模化应用。企业必须确切知道 AI 可以在何处运行、在何种条件下触发、具备多大置信度，并且必须拥有切实的证据链作为后盾。因此，“信任（Trust）”正成为 AI 下一时代的分水岭。Duncan 指出，“运营级 AI（Operational AI）”正是指将 AI 嵌入企业运营流程中，并辅以清晰标准、持续证据和明确人类责任的能力。 ```mermaid flowchart LR accTitle: Operational AI Progression subgraph S1["第一阶段：智能时代"] A["模型能力与参数规模"] --> B["基准测试跑分"] B --> C["概念验证与原型开发"] end subgraph S2["第二阶段：信任与运营时代"] D["确定运行边界与条件"] --> E["持续证据与置信度评估"] E --> F["规模化业务价值释放"] end C --> D ``` - **市场核心转变**：从“AI 能做什么”转变为“AI 在什么条件下可以被信任去执行”。 - **企业价值法则**：在 AI 时代获得最大商业价值的企业，不一定是拥有最先进 AI 模型的企业，而是能够“自信且规模化部署 AI”的企业。 - **运营级置信度的三大支柱**：技术运行可靠性、业务预期达成度、风险与责任边界合规性（包括何时必须引入人工介入）。 --- ### [03:38](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=218s) AI 嵌入企业底层架构与新型风险敞口的爆发当前 AI 的普及度正在迅速扩散，已全面渗透进日常办公生产力工具、开发环境、客户交互平台乃至底层操作系统中，俨然成为职场中“效率最高的超级员工”。随着 Agentic（智能体）系统的日益成熟，AI 正从单纯的辅助生成工具演变为能够跨平台、跨环境、跨系统自主协调工作流、触发动作并参与复杂决策的核心组件。然而，这种从外围工具向企业运营核心层的跃迁，导致 AI 的每一次输出与决策都会直接影响客户体验、员工协同、财务收益、合规义务乃至品牌声誉，企业的风险敞口因此被成倍放大。传统的被动式、人工审批治理流程在面对高并发、自适应的智能体时已完全失效。 - **新型 AI 风险矩阵**： - **增量幻觉（Incremental Hallucinations）**：在复杂多轮链路中层层累积放大的幻觉问题。 - **自主决策失控（Autonomous Decision Failures）**：智能体跨系统调度时产生的逻辑失常。 - **安全与合规漏洞**：模型漂移（Model Drift）、系统偏见（Bias）、知识产权（IP）泄露及治理断层。 - **新型治理范式**：面对机器速度的运行，必须依靠“AI 监管 AI”的持续监控与保障机制，同时由人类把控最终的策略设定与问责底线。 --- ### [05:39](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=339s) 治理脱节导致的价值鸿沟：缺乏信任锁死投资回报许多企业虽然手握顶尖的模型和极具前景的应用场景，却因为缺乏统一跨部门、跨系统的治理与度量手段，导致商业价值被牢牢“锁死”。AI 系统的实际表现受到动态提示词、实时输入数据、复杂系统集成以及用户行为变化的持续冲击，这使得量化其实际业务产出变得异常棘手。这种“缺乏度量与治理”引发了企业内部连锁的信任危机：度量不足导致信心缺失，缺乏信心促使管理层堆砌大量低效的人工双重核验，繁重的人工审核进而拖垮部署速度并推高运营成本，最终使原本极具商业潜力的项目变得在经济上不可行。管理层陷入了“投入了高昂智能成本，却因无法证明受控而无法释放商业价值”的尴尬困境。 ```mermaid flowchart TD accTitle: AI Value Gap Vicious Cycle A["缺乏统一治理与动态度量"] --> B["企业对系统输出缺乏置信度"] B --> C["对自主运行缺乏信任"] C --> D["被迫增加重度人工核验"] D --> E["应用落地变慢且成本飙升"] E --> F["AI 投资价值无法规模化释放"] F --> A ``` - **核心矛盾**：AI 技术的演进速度远超企业将其工程化、运营化的速度。 - **信任建立机制**：信任不能单靠供应商声誉、一次性上线前测试或成功的概念验证（PoC），而必须依赖于运行环境中系统行为的“实时、持续证据链”。 --- ### [08:46](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=526s) 剪刀差困境：机器速度的运营必须匹配机器速度的保障 Duncan 结合一张极具冲击力的曲线图，解析了 AI 系统指数级跃升与人类治理能力线性增长之间的结构性矛盾。图中纵轴代表能力与业务影响力，横轴代表 AI 应用规模与时间推移： - **蓝色曲线上升迅速**：代表 AI 系统在自主性、智能化和业务渗透度上的爆发式增长，智能体数量与跨系统交互呈现网状发散。 - **灰色直线增长缓慢**：代表依赖人力、专家经验和有限组织资源的人工治理能力，呈现出缓慢且刚性的线性爬升。这两条曲线之间不断扩大的剪刀差，便是企业面临的**“信任鸿沟（Trust Gap）”**。每增加一个自主 Agent，就意味着引入了指数级增长的交互节点、潜在故障点与合规风险。企图依靠增加编制或人工抽检来填补这一鸿沟无异于缘木求鱼。唯一的解法是以自动化应对自动化，即“用机器速度的保障来匹配机器速度的运营（Machine speed operations require machine speed assurance）”。 Duncan 幽默地提到，许多顾问曾询问他“AI 的爆发会不会终结软件测试与保障行业”，而现实情况恰恰相反，正是因为 AI 开发与交付的速度呈现海啸级爆发，企业对质量验证、合规监控、防漂移等保障服务的需求才迎来了前所未有的繁荣。 --- ### [11:25](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=685s) 持续 AI 保障的三大支柱与评估透镜为了弥合信任鸿沟，QualityAI 提出了“持续 AI 保障（Continuous AI Assurance）”的方法论模型。该模型将传统的软件测试升维为企业级业务保障，主要由三大动态循环阶段构成： ```mermaid flowchart LR accTitle: Continuous AI Assurance Framework subgraph Stages["持续保障三大阶段"] direction TB D["1. 定义 (Define)<br/>动态设定标准与规则"] A["2. 保障 (Assure)<br/>实时测试与证据构建"] C["3. 认证 (Certify)<br/>综合评估与放行决策"] D --> A --> C --> D end subgraph Lenses["三大评估透镜"] direction TB L1["技术透镜：它是否正常工作？"] L2["业务透镜：它是否达成预期价值？"] L3["信任与安全透镜：它能否被放行自主运行？"] end Stages <--> Lenses ``` 1. **定义（Define - 确立动态标准）**：由专门的 Define Agents 动态维护业务目标、合规策略、安全围栏、风险容忍度及衡量指标。标准并非一成不变的死板文档，而是随着产品更迭、新风险出现和组织意图变化而持续更新。 2. **保障（Assure - 持续验证现实）**：由 Assure Agents 在生产环境中不间断捕获运行信号，对模型输出、Agent 行为、业务工作流与既定标准进行实时对齐比对，持续累积可靠性证据链。 3. **认证（Certify - 支撑放行决策）**：由 Certification Agents 汇集所有证据，输出全局就绪度画像，明确界定 AI 可以在哪些环节全自主运行、在哪些场景需扩容、在何种异常状态下必须强制触发人工介入。 - **三大核心透镜**： - **技术透镜**：“系统在技术层面上是否正常工作？” - **业务透镜**：“系统是否确实交付了预期的商业成果？” - **信任与安全透镜**：“系统是否处于安全可控边界内，能否放行其自主运行？” - **人类核心职责**：人类负责顶层战略与业务目标的定义、批准风险边界，并在监控证据显示置信度下滑时果断进行人工干预。 --- ### [13:36](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=816s) 企业落地实操：从就绪度评估到全生命周期管理针对不同发展阶段的企业，Duncan 给出了清晰的落地路径建议，强调企业无需试图“一口吞下一头大象（boil the ocean）”，而应从核心价值节点切入： | 企业成熟度层级 | 推荐切入路径 | 核心工作与关键动作 | |:--- |:--- |:--- | | **初级探索阶段** | **AI 就绪度评估 (AI Readiness Assessment)** | 明确组织现状成熟度；排查治理断层、数据缺陷与安全盲区；输出优先级清晰的演进路线图，在规模化前筑牢技术与流程底线。 | | **成熟运营阶段** | **托管式持续保障 (Managed Continuous Assurance)** | 搭建生产级动态验证体系；构建风险实时检测与响应机制；建立类似金融反洗钱（AML/KYC）的**置信度评分（Confidence Scoring）**模型，对风险信号进行分级处置。 | Duncan 特别引用了经典科幻电影《银翼杀手》（*Blade Runner*）中的生化人基线测试作为绝妙类比：电影中生化人每天早晨都要接受严苛的基线问答（Baseline Test），通过声调振动与反应实时监测其心理与运行状态是否偏离既定轨道。企业对 AI 系统的监管也应如此——绝非“一次上线、永久放任”，而必须实施日常化、动态化的基线比对与行为校准。 - **行动准则**： - 聚焦关键业务流程（Business-Critical Processes），优先在能解锁显著价值且风险可控的场景中验证置信度。 - 必须与关键利益相关者共同确立并达成一致的“信任准则（Trust Criteria）”，明确系统需要呈现哪些具体证据才被允许上线放量。 - **核心提示词转变**：决策者必须将管理提问从“AI **能**做什么？”转变为“AI **已证明自己能安全做好**什么？”。 --- ### [17:59](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=1079s) 问答互动：消除利益相关者对 AI 产品的偏见与疑虑在演讲后的现场提问环节，观众提出了一个现实问题：“在当前环境下，企业利益相关者对标有‘AI 赋能’的产品是否天然存在信任偏见？一旦得知某个方案由 AI 生成，是否会立刻认为其可靠性降低？该如何化解这种潜在偏见？” ``` 观众提问： “在当下的环境中，利益相关者对 AI 是否天然存在信任偏见？如果明确告知对方这是一个 AI 产品，他们是否会主观上觉得它更不可靠？我们该如何克服这种潜在偏见？” Andrew Duncan 回答： “按常理来说，面对满天飞的负面舆论，大家的直觉反应通常是‘确实存在偏见’。但从我们与各行业客户打交道的实际经验来看，其根源往往并非主观恶意或单纯的偏见，而是源于‘缺乏理解’。许多人甚至根本不知道需要看到什么样的证据才能放心地去信任一个系统。行业内充斥着大量关于‘AI 到底灵不灵、值不值得信赖’的断言，但其中许多判断并非基于严谨的事实，而是建立在对业务上下文的误解之上。只要我们能拿出客观、量化的运行证据，这种所谓的偏见就会自然消解。” ``` ## Notable quotes > The organizations that realize the greatest value from AI will not necessarily be the ones with the best AI. They will be the ones that can deploy AI with confidence at scale. — Andrew Duncan > The challenge is no longer intelligence — it is trust. — Andrew Duncan > The first era of AI was about intelligence. The next era of AI will be about trust. To unlock the full value of AI, enterprises must change the prompt — from asking what AI can do to proving what AI can be trusted to do. — Andrew Duncan > [00:04](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=4s) 让软件与解决方案值得信赖（trustworthy） — Andrew Duncan > [01:00](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=60s) 运营级 AI（Operational AI）正是指将 AI 嵌入企业运营流程中，并辅以清晰标准、持续证据和明确人类责任的能力。 — Andrew Duncan > [08:46](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=526s) 用机器速度的保障来匹配机器速度的运营（Machine speed operations require machine speed assurance） — Andrew Duncan > [13:36](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=816s) 决策者必须将管理提问从“AI 能做什么？”转变为“AI 已证明自己能安全做好什么？”。 — Andrew Duncan > [17:59](https://www.youtube.com/watch?v=GdZ4BCQF4xY&t=1079s) 只要我们能拿出客观、量化的运行证据，这种所谓的偏见就会自然消解。 — Andrew Duncan ## Mentioned Andrew Duncan, QualityAI, Ai4, Blade Runner (银翼杀手), Operational AI, Continuous AI Assurance, Define Agents, Assure Agents, Certification Agents ## Video description Andrew Duncan - Chief Executive Officer, QualityAI The organizations that realize the greatest value from AI will not necessarily be the ones with the best AI. They will be the ones that can deploy AI with confidence at scale.”. AI is advancing at an unprecedented pace. Models are becoming more capable, agents more autonomous, and AI is increasingly writing code, generating content, making decisions, and participating in core business processes. Yet despite billions invested in AI, many organizations remain hesitant to deploy it at scale. The challenge is no longer intelligence — it is trust. As AI becomes embedded in enterprise operations, organizations face a new class of risks, including hallucinations, autonomous decision failures, governance gaps, security vulnerabilities, model drift, bias, and intellectual property exposure. At the same time, business leaders are under pressure to accelerate innovation, improve productivity, and realize measurable value from their AI investments. In this keynote, Andrew Duncan, CEO of QualityAI, will explore why trust is emerging as the defining challenge of the AI era. He will discuss how the enterprise focus is shifting from building AI to operationalizing AI, why quality must evolve from a software engineering discipline into a business assurance capability, and how Continuous AI Assurance can help organizations deploy AI responsibly and confidently at scale. Attendees will gain practical insights into how leading enterprises are balancing innovation with risk, accelerating adoption while maintaining control, and building the operational confidence required to move AI from experimentation to business-critical operations. The first era of AI was about intelligence. The next era of AI will be about trust. To unlock the full value of AI, enterprises must change the prompt — from asking what AI can do to proving what AI can be trusted to do. [Source](https://www.youtube.com/watch?v=GdZ4BCQF4xY)

Source quotes

Andrew Duncan: The organizations that realize the greatest value from AI will not necessarily be the ones with the best AI. They will be the ones that can deploy AI with confidence at scale. Andrew Duncan: The challenge is no longer intelligence — it is trust. Andrew Duncan: The first era of AI was about intelligence. The next era of AI will be about trust. To unlock the full value of AI, enterprises must change the prompt — from asking what AI can do to proving what AI can be trusted to do. [0:04] Andrew Duncan: 让软件与解决方案值得信赖（trustworthy） [1:00] Andrew Duncan: 运营级 AI（Operational AI）正是指将 AI 嵌入企业运营流程中，并辅以清晰标准、持续证据和明确人类责任的能力。 [8:46] Andrew Duncan: 用机器速度的保障来匹配机器速度的运营（Machine speed operations require machine speed assurance） [13:36] Andrew Duncan: 决策者必须将管理提问从“AI 能做什么？”转变为“AI 已证明自己能安全做好什么？”。 [17:59] Andrew Duncan: 只要我们能拿出客观、量化的运行证据，这种所谓的偏见就会自然消解。

Source quotes

[1:38] Andrew Duncan: The organizations that realize the greatest value from AI will not necessarily be the ones with the best AI. They will be the ones that can deploy AI with confidence at scale. [0:04] Andrew Duncan: The challenge is no longer intelligence — it is trust. [16:05] Andrew Duncan: The first era of AI was based on intelligence. The next era is going to focus on trust. [16:05] Andrew Duncan: Leaders need to change their prompt. They need to move away from asking what can AI do towards asking what has AI proven it can do. [1:38] Andrew Duncan: Operational AI is the disciplined deployment of AI within enterprise processes backed by clear standards, continuous evidence, and human accountability. [1:38] Andrew Duncan: Most enterprises currently possess more AI capability than they can effectively convert into business value. [3:38] Andrew Duncan: AI is the fastest employee in the workplace. [8:46] Andrew Duncan: Machine-speed operations demand machine-speed assurance.

Original transcript

[0:04] Uh before we get started, I just wanted to give you a quick overview of what is Quality AI, [0:09] what do we do, and why is it relevant to what I'm talking about uh this afternoon. We're a [0:13] highly profitable billion-dollar quality and AI uh quality engineering and AI assurance business [0:19] uh with over a 25-year history in testing. We not only engage across the SDLC, but also in [0:25] AI engineering and the engineering of AI. We're specialists that work with the both providers as [0:32] well as consumers of AI models and tooling in order to drive automation and efficiency. Uh [0:38] we're a sector-led business. Quality engineering is is an important capability when coupled with [0:43] our industry expertise. We do something which is quite rare, which is end-to-end testing across the [0:47] the AI landscape. But ultimately, we're about making software and solutions trustworthy. [0:53] Uh and we see AI as the most recent and important frontier in continuing that mission. [1:00] The first year of AI was dominated by intelligence and understanding. The market focused on model [1:05] capability, got benchmark performance, scale, the growing range of AI products, [1:11] uh and what they could what they could perform. The progress has been extraordinary [1:15] as we've all witnessed over the last several years. AI is now designing, architecting, [1:20] and writing code. It's generating content. It's supporting decisions. It's participating [1:24] in core business processes. However, leaders are now asking for measurable productivity, [1:31] foster innovation, and sustain value from investments already made in the AI space. [1:38] As AI moves into business-critical operations, having AI capability alone is insufficient. [1:43] Organization organizations need to know where AI can act, under what conditions, and on and [1:49] with what confidence, backed up by real evidence. Trust is therefore becoming the defining issue for [1:56] the next era of AI. Operational AI describes the ability to deploy AI within enterprise processes [2:04] and with clear standards, with continuous evidence, and appropriate human accountability. [2:12] Continuous AI creates the confidence required to scale AI confidently. So, what are we seeing out [2:18] there? Well, access to capable AI is broadening. This seems to be obvious. Uh but, confidence in [2:24] using AI at scale remains uneven, patchy. I'd even go further and say most organizations [2:31] already have more AI capability than they are capable of converting into business value. [2:37] AI platforms are becoming widely available, more accessible, and more efficient. However, [2:42] the organizations that realize the greatest value from AI will not necessarily be the [2:47] ones with the best AI. They will be the ones that can deploy AI with confidence at scale, [2:54] and this is what we refer to as operational AI. Confidence is the evidence that AI [3:00] works technically, that it it is providing the intended business results and business outcomes, [3:05] and is operating within acceptable risk and and responsibility boundaries, plus knowing when human [3:12] intervention is required. Organizations that can answer these questions clearly will move faster. [3:18] They will further expand successful use cases. They will further reduce [3:22] unnecessary manual intervention, and they'll make better informed decisions about risk. [3:28] Operational confidence is becoming an important as important as access to AI capability itself. [3:38] So, the market's already moved on. AI now requires an enterprise operating discipline [3:42] rather than an isolated projects governance. AI has become a standard enterprise capability. [3:49] It's appearing inside productivity tools, development environments, [3:53] customer platforms, and even operating systems. AI is becoming the fastest employee in the [3:59] workplace. There was a like a couple of sessions ago someone was asking who's who's worried about [4:03] their job. This is a bit concerning. But it can generate code and content at a pace that changes [4:08] the volume and speed, not just of what it's doing, but also what's happening around it. [4:13] As agentic systems mature, AI is moving beyond assistance. It's now beginning to coordinate [4:19] workflows, trigger actions, and participate in decisions, not just within an environment, but [4:25] across multiple environments, multiple platforms, and multiple systems. [4:29] That progression places AI within the operating layer of an enterprise. Its performance now [4:35] affects customers, employees, financial outcomes, regulatory obligations, and even brand reputation. [4:43] The risk profile and the risk surface has become far broader as a result. [4:48] For example, organizations now need to account for exposures such as incremental hallucinations. We [4:53] know about the base hallucinations, but further incremental hallucinations is an interesting one, [4:58] autonomous decision failure. Governance gaps, security vulnerabilities, we've been reading [5:04] quite a lot about those in the last couple of weeks. Model drift, bias, intellectual property [5:09] property exposure, which we just heard about in the previous session, to name just a few. [5:14] Traditional manual governance cannot keep up with the pace. AI will need to support [5:20] the monitoring and assurance of AI with a human accountability remaining. As AI becomes part [5:29] of the enterprise operation, reliability and trust must become part of a business requirement set. [5:39] So, Continuous AI is the mechanism [clears throat] that converts AI capability into for confidence. [5:46] The challenge is AI confidence. AI capability is everywhere, but the confidence to use it [5:52] is very uneven. Trust cannot rely on reputation or a successful pilot or a one-time test before [5:59] deployment. It now requires current evidence, continual evidence that reflects how the systems [6:06] are behaving in the live operating environment. Continuous assurance provides that evidence. [6:13] It evaluates models, agents, data, workflows, controls throughout the life cycle [6:18] rather than at isolated checkpoints, be them in process or in time. [6:23] That is the basis of operational AI. AI embedded in enterprise processes with clear standards, [6:31] observable behaviors, and evidence-based decisions around autonomy. The combination of speed [6:37] and control matters. Business leaders are under pressure to accelerate adoption while risk, [6:42] technology, and legal teams need confidence that the organization remains in control. [6:47] Operational AI is the outcome of combining capability with confidence. [6:56] I love this slide. It's sort of the things on the right sort of all add up to to to the bottom, but [7:01] essentially business value is locked behind us the lack of confidence to let AI run. Trust failures [7:07] are preventing organizations from converting AI investments into scaled business value. [7:14] AI capability is accelerating faster than many enterprises can operationalize it. [7:19] Organizations may have strong models and and promising use cases, yet still lack a consistent [7:25] way to govern the outcomes across both functions and systems. It's made worse by the tsunami of [7:32] demand for assurance and assurance services brought on by that high level of automation. [7:38] They struggle to measure whether AI is delivering the intended business results, [7:42] especially when performance depends on changing data, changing prompts, changing integration, [7:47] changing user behavior, which is a very modern problem set. These issues are all connected. [7:54] The lack of governance coupled with a lack of true measurable measurability results in [7:58] a lack of confidence. A lack of confidence results in a lack of trust. And who wants to [8:03] implement anything, let alone scale it, if you don't have trust in what you're implementing? [8:09] Further, weak management reduces confidence. I'm sorry, weak measurement reduces confidence. [8:16] Limited confidence increases manual review, heavy manual review slows adoption, [8:21] and it makes things more more uneconomic and more expensive. Leaders therefore face a value gap. [8:28] They've invested in intelligence, but they cannot release its full value until [8:31] the organization can demonstrate control, can trust, continuously. [8:37] Business value will remain constrained until trust can be established and maintained. [8:46] This is an interesting if not scary chart. The structural gap exists between [8:51] rapidly advancing AI systems and the slower human governance capacity. [8:55] So, on the Y axis, we've got capability and impact. On the X axis, we have expansion [9:01] expansion of the AI estate. You can put a time in there as well. The blue curve reflects the [9:07] rate at which AI systems are advancing. In terms of capability, autonomy, and business impact, um [9:14] are all advancing very very quickly indeed, as you can see from the from the the rate of the band. [9:19] The gray line reflects the human capacity to govern. Slowly growing since it depends on [9:26] scarce resources such as time, expertise, and organizational resources themselves. [9:34] This creates the gap, or you see is the dotted line on the right, which we refer to as the trust [9:39] gap, and there have been several talks over the last couple of days that also talk about the trust [9:43] gap. But, every additional agent results in more interactions, more possible failure points, more [9:48] decisions that have to be that need evidence and and and oversight. The governance burden therefore [9:54] grows faster than head count or manual review can reasonably support. Machine speed operations [10:00] require machine speed assurance. AI continuously evaluate signals, detects changes, and assembles [10:08] evidence while people retain responsibility for policy and innovation and intervention. Without [10:15] this operating model, that gap on the right is going to widen as AI becomes more autonomous [10:21] and more deeply embedded in core processes. The trust gap widens rapidly when AI capability scales [10:28] faster than the organization can govern govern it. And I know that for me personally with it within [10:33] quality AI, that is not bad news. It actually is why our our business is in such rude health. [10:39] Is because we have this tsunami of demand for services. Um we I get asked a lot of times um [10:45] from some of our advisors, "Well, this AI thing, it's got to be scary for a testing and assurance [10:49] business." And reality is quite the opposite. Um what is happening is because AI developments and [10:54] the development of AI based solutions and services is accelerating at such a pace, [10:59] it does not circumvent the need to ensure and assure that those those solutions and services are [11:07] basically behaving technically as you'd expect them to behave, [11:10] producing the business results that you expect them to to produce, and most importantly to keep [11:15] produce them as spec and not start drifting or varying the the the equation as the case may be. [11:25] So, we're big believers in continuous AI assurance providing essentially the [11:29] practical model for closing the trust gap on the prior slide. There are three stages of it. [11:34] One is defining or what we call define, which sets the standard. Define agents maintain the [11:39] current view of business objectives, policies, controls, guardrails, risk tolerances, etc., [11:44] as well as success measures. And this is important because organizational intent changes. [11:50] Policies evolve, products change, new risks emerge, and but a standard must remain current. It [11:57] has to keep updated rather than becoming a static document or a static body to which to to get over. [12:04] Assure is the second piece which really tests that reality continuously. [12:08] Assure agents evaluate production production signals, model and agent behavior, [12:13] workflows and outcomes against that standard. And remember, the standard can be very [12:19] The objective is to build evidence over time. This is with the foundation and the core of building [12:25] trust. That evidence shows that whether the system is reliable, again, whether it's delivering the [12:31] value you'd expect and whether the risk remains within acceptable boundaries as it delivers it. [12:36] Certification or certify supports the decisions about when AI can run, [12:41] where it can scale, and where intervention is required. Certification agents bring the [12:47] evidence together into a clear view of operational readiness. Every stage is evaluated through three [12:54] three lenses: technical lens, a business lens, and a trust and safety lens. And the questions [13:00] are fairly straightforward. Does it work? Does it deliver? And And can we let it run? [13:07] This is how quality evolves into business assurance. Technical quality remains essential, [13:13] but the organization also needs evidence of business impact and responsible operation. [13:18] Human accountability remains throughout. People define the objectives, approve the boundaries, [13:23] and act when evidence shows that confidence has fallen. But continuous AI insurance converts [13:29] changing operational signals into current evidence of trust. [13:36] So, where should you begin? Well, this looks like a blatant sales slide, [13:40] but I'm not going to talk to it as a blatant sales slide. It's really a bunch of points as for [13:44] um two types of organization, one which is less mature in in terms of operational AI and one [13:50] which probably is uh quite a long way along that journey. But fundamentally, organizations will [13:55] begin from different levels of maturity, and so we have these two points. The first on the left [14:00] is the AI ready readiness assessment. It basically focuses on establishing maturity, um identifying [14:06] gaps, risks, and blind spots, and defines a prioritized roadmap, where it gives leadership [14:11] a very practical sequence for strengthening governance, data, technology, processes, and trust [14:18] before expanding deployment. And this is something you guys can do with with assistance, not just [14:22] from our organization, but but others, but doing it is very, very important. Establish maturity, [14:28] identify the gaps, risks, and blind spots, and then develop define the prioritized roadmap. [14:34] On the right is is for a more mature organization along the operational AI um um spectrum. It's one [14:40] which relates to managed continuous um assurance. It really has again three main components, [14:46] continuous validation as we've been talking about, risk detection and response. Detection of risk, [14:52] that is something which needs a bit of thought. It doesn't It's not something which is straight-up [14:56] intuitive. It does need thought. It does need ponderance. Um co- confidence scoring [15:01] and evidence. And you'll see a lot of this in the financial and banking communities, [15:04] where particularly in things like AML and KYC, there will be triggers which are then evaluated [15:09] and scored. Is it a serious threat? Is it a non-serious threat? Is it something we need to [15:12] worry about? Is it not something we need to worry about? But a few points as as you go through this [15:17] this these frameworks, depending whether you're left or right. Assess where performance is strong, [15:23] where risk is increasing, and where intervention is required. Do not try and boil the ocean. Do [15:29] not take on every case at once. Start with business-critical processes where [15:35] greater confidence can unlock measurable value and as well as reducing significant risk. [15:42] Trust criteria are really important. You need to meet with your stakeholder set not only establish, [15:49] but also agree trust criteria. These are important because it gives you the or it basically lays out [15:55] what evidence is required for deployments and scale. What do these stakeholders need to see to [16:00] have the trust in what you are recommending is implemented and scaled as solution. [16:05] Lastly, I just I would like to just leave you with a couple of thoughts. [16:09] The first era of AI was based on intelligence. The next era is era is going to focus on trust. [16:15] I can't under emphasize that enough. Enterprises need evidence that AI can operate reliably, [16:23] deliver the intended outcomes, and remain within clear boundaries. This is the foundation of trust. [16:30] If you have trust, you can implement and scale. A broader approach for quality is required, [16:37] extending from software engineering into continuous business assurance. [16:41] This isn't a one-and-done thing anymore. This is a thing which has to keep going and keep [16:45] going and keep being tested. Think about I know those of you who watched Blade Runner, [16:50] um but the the the replicants have to go through a baseline every morning. So, [16:54] they're asked a whole bunch of questions and and they're monitored for monitored for their answers, [16:59] um not just in terms of content, but also in terms of voice vibrations and all the rest of it. [17:03] They're essentially being baselined to make sure that they are still performing technically, [17:07] that they are capable of performing from a business standpoint, and they're staying [17:11] within the boundaries of operation. It's exactly what I'm talking about here. [17:16] Leaders need to change their prompt. They need to move away from asking what can AI do [17:23] towards asking what has AI proven it can do. [17:28] There's a subtle difference, but a very important difference there. [17:32] Continuous AI assurance provides organizations with a structured way to define expectations, [17:38] to test reality, and make evidence-based decisions about where and how AI can run. [17:45] Finally, operational AI is not access to AI alone. It will create sustainable competitive advantage [17:52] for the organizations that embrace it. Thank you. And you could do with time for questions. [17:59] Thank you. Um So, you just spoke about about trustworthiness. I'm curious in this environment, [18:05] is the trustworthiness of AI from a stakeholder perspective to us that's slightly biased? [18:11] Um in your viewpoint of if you tell someone you made this is an AI product versus not, [18:16] do they immediately bias themselves as it as less trustworthy product? And how do you overcome that [18:22] um potential bias? I think by the book and and dealing in an industry that we work, I'd [18:27] say the instant question is yes, it's obviously biased cuz it's all scary stuff out there. But [18:31] but at the end of the day, I I'm not I'm not sure it's based on bias. I think it's based on a lack [18:35] lack of understanding. And and and even to a point where um some some individuals may not actually [18:41] understand or or appreciate what they need to see or be evidenced in order to trust something. [18:47] So, I think you whether that's bias or not, I'm not so sure, but it it's what I've seen most of [18:51] uh and we as an organization deal with it pretty regularly is that there are a lot of [18:55] claims and statements made about sort of whether things are trustworthy or [18:58] whether they're working or not. Um and some of those claims are not [19:02] necessarily based on fact, they're based on misunderstandings of of various contexts.