Panel 1 - 4: Validation Methodologies of Generative AI Responses

Panel 1 - 4: Validation Methodologies of Generative AI Responses — Ai4

Ai4

AI-assisted summary of Panel 1 - 4: Validation Methodologies of Generative AI Responses — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by the Ai4 channel in August 2026, this presentation features Robert Bierman, a lecturer at San Francisco State University. According to the description, the talk examines why generative AI errors make validation a system-level discipline and introduces fourteen validation methodologies across four operational layers to build repeatable verification pipelines.

## TL;DR Robert Bierman explains that large language models generate statistically plausible token sequences rather than verified truths, frequently masking critical logic errors, convention violations, and hallucinations beneath confident prose. To address these vulnerabilities, he presents fourteen validation methodologies organized across four layers: evidence grounding, model-behavior signals, claim and logic validation, and operational safeguards. He illustrates how structural checks and cross-model analysis reveal hidden bugs, prescribing a minimum of two to three independent validation methods depending on operational risk. Finally, he underscores that technical workflows are pivoting from syntax generation to rigorous auditing, emphasizing that automated validation must be paired with genuine human comprehension and accountability.

## Atmosphere & style The session opens with an interactive, humorous room exercise that immediately confronts the audience's tendency to accept polished AI answers at face value. Bierman balances academic precision with engaging, practical software engineering anecdotes, maintaining a steady, analytical, and authoritative delivery. The talk feels like a clear-eyed reality check against uncritical AI adoption, systematically replacing blind trust with structured engineering rigor.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=4s) Large language models are predictive statistical token generators rather than reasoning engines, meaning apparent output confidence is never proof of factual accuracy. - [00:04](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=4s) Relying on superficial plausibility or 'vibe coding' masks severe risks, including hallucinated citations, security vulnerabilities, and logic flaws. - [00:04](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=4s) Validating generative outputs is a system-level discipline and architecture problem rather than a basic fact-checking task tacked onto the end of a workflow. - [03:59](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=239s) Code generated by AI can appear 100% plausible and execute successfully while still critically violating system-level constraints and architectural conventions. - [03:59](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=239s) A robust validation strategy requires combining at least two independent validation methods for low-risk tasks and at least three for mission-critical applications. - [11:06](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=666s) Validation methodologies span four distinct layers: evidence grounding, model-behavior signals, claim and logic validation, and operational safeguards. - [11:06](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=666s) Cross-model validation is effective for identifying discrepancies, but agreement between models does not guarantee ground truth because models can share common error patterns. - [11:06](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=666s) Atomic claim validation deconstructs complex compound responses into isolated assertions to ensure flawed intermediate steps do not corrupt the overall output. - [11:06](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=666s) Regression validation should be implemented as an automated pipeline to maintain consistency in citations, terminology, and constraint adherence across long-term project lifecycles. - [20:33](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=1233s) The primary engineering bottleneck has shifted from manual syntax and boilerplate generation to reviewing, auditing, and validating generated assets. - [20:33](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=1233s) Automated validation does not equal human comprehension; practitioners retain full operational and legal accountability for understanding and deploying AI-generated solutions.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=4s) The Illusion of Plausibility and the Need for AI Validation

The presentation opens with an interactive room exercise: the speaker asks the audience to stand up, instructing anyone who has never used generative AI to sit down, and then telling those who have never accepted a plausible-looking generative response at face value to take a seat as well. Addressing the remaining standing attendees with a grin, he notes that this talk is specifically tailored for them. While large language models present their outputs with supreme confidence—delivering replies that assert, "This is your answer"—users cannot simply take that confidence as proof of truth.

The root of the issue lies in the fundamental architecture of generative systems. Large language models are not intelligent beings capable of genuine understanding; they are predictive statistical models trained to output tokens representing what they calculate the user wants to see next. This design creates well-documented failures across domains, from practicing attorneys facing judicial sanctions for citing hallucinated legal precedents to academic researchers publishing fabricated references. Relying on "vibe coding"—accepting generated code simply because it looks and feels right—is inherently dangerous, because "a broken clock is right twice a day." Validation cannot be treated as a quick fact-check tacked on at the end of a sprint; it must be an integrated, systematic discipline.

- Generative models predict expected token sequences rather than evaluating factual ground truth. - High-profile failures include fabricated legal citations in court filings and hallucinated research references. - Superficial correctness (or "vibe coding") masks critical vulnerabilities, logic errors, and security flaws.

---

[03:59](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=239s) Systematic Failure Modes and the Broken Assembly Code

Validating generative outputs requires recognizing that single-method checks consistently miss critical failure modes. Evaluating a generated SQL query or script solely on whether it satisfies immediate business logic overlooks essential engineering requirements: error handling, cybersecurity vulnerabilities, and "B-path" or beta-path testing (how humans actually misuse systems in edge cases rather than how they should use them). Generative failures stem from distinct systemic vulnerabilities: hallucinations, knowledge staleness caused by training cutoff dates (where asking about rapidly evolving geopolitical conflicts yields conflicting answers depending on the day), prompt bias that skews answers based on phrasing, and structural convention violations.

```mermaid flowchart TD accTitle: Four Layers of Generative AI Validation accDescr: Diagram illustrating the four interconnected validation layers used to catch different categories of generative AI errors.

subgraph L1["Evidence Grounding"] A["Source Triangulation"] B["Citation Verification"] C["RAG & Spec Grounding"] end

subgraph L2["Model Behavior"] D["Self-Consistency"] E["Cross-Model Validation"] F["Uncertainty / Confidence"] end

subgraph L3["Claim & Logic"] G["Atomic Claim Validation"] H["Reasoning Consistency"] I["Rule Constraints & Security"] J["Regression Validation"] end

subgraph L4["Operational Safeguards"] K["Human-in-the-Loop"] L["Adversarial Red Teaming"] M["Temporal Validity Checks"] end

L1 --> L3 L2 --> L3 L3 --> L4 ```

To demonstrate structural failures, the speaker recounts an experiment from his undergraduate computer science course covering low-level systems and assembly language. After prompting ChatGPT to translate a small C program into assembly, the students unanimously declared the generated assembly correct because it could theoretically execute. In reality, it completely violated standard ABI calling conventions and register preservation rules, overwriting scratch and preserved registers indiscriminately. When prompted to fix the code specifically citing the register convention breach, the model generated an entirely new version that violated a different set of register rules. The output was "100% plausible, potentially functional, and absolutely incorrect." Because no single validation technique catches every flaw, best practice requires combining at least two independent methodologies for low-stakes tasks and at least three for mission-critical applications like legal briefs and medical diagnostics.

- **Knowledge Staleness**: Model cutoffs render time-sensitive or contemporaneous queries unreliable without dynamic grounding. - **Prompt Bias**: Subtle variations in phrasing inadvertently steer the model toward uncalibrated or biased responses. - **Register Convention Violation**: The generated assembly appeared functional in isolation but failed system-level calling conventions, producing intermittent runtime bugs. - **Validation Rule of Thumb**: Use a minimum of two independent validation methods for low-risk outputs and three or more for high-consequence tasks.

---

[11:06](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=666s) The Fourteen Validation Methodologies Across Four Layers

To systematically catch errors, validation is structured into fourteen distinct methodologies across four functional layers:

``` +-----------------------------------------------------------------------------------------+ | FOUR VALIDATION LAYERS | +-----------------------------------------------------------------------------------------+ | 1. EVIDENCE GROUNDING | | - Source Triangulation: Verifying claims against authentic external source text. | | - Citation Verification: Confirming cited authors, papers, and journals exist. | | - RAG Grounding: Grounding queries in technical manuals and authoritative specs. | | - Structural Knowledge & Base Grounding: Checking formal syntax and domain bases. | +-----------------------------------------------------------------------------------------+ | 2. MODEL-BEHAVIOR SIGNALS | | - Self-Consistency: Testing stability across prompt reformulations. | | - Cross-Model Validation: Comparing outputs across independent LLMs (e.g., Claude). | | - Uncertainty Quantification: Exposing underlying statistical confidence scores. | +-----------------------------------------------------------------------------------------+ | 3. CLAIM & LOGIC VALIDATION | | - Atomic Claim Validation: Deconstructing complex responses into isolated facts. | | - Reasoning & Consistency Checks: Verifying logic across multi-turn workflows. | | - Rule & Constraint Validation: Enforcing ABI rules, security, and SQL injection. | | - Regression Validation: Running automated recurring checks on terms and citations. | +-----------------------------------------------------------------------------------------+ | 4. OPERATIONAL SAFEGUARDS | | - Human-in-the-Loop: Retaining human accountability and qualitative oversight. | | - Adversarial Red Teaming: Actively attempting to break model logic under stress. | | - Temporal Validity Checks: Monitoring time-bound claims for real-world drift. | +-----------------------------------------------------------------------------------------+ ```

Model behavior methods provide practical diagnostic signals—such as an experiment where ChatGPT and Claude identified different model numbers from an obscure chip photo, allowing the speaker to feed each model's output to the other to debate discrepancies. However, model-versus-model checks remain secondary because two models can easily share identical underlying errors: "If they differ, you know something is wrong. If they are the same, you don't know anything."

Claim and logic validation introduces *Atomic Claim Validation*, breaking complex compound statements into isolated, verifiable components to ensure a flawed intermediate step does not contaminate an otherwise convincing aggregate result. Furthermore, teams should adopt *Regression Validation*—adapting standard software regression testing into an automated pipeline that continuously validates that terminology, document citations, and architectural constraints remain consistent across iterative AI generations.

- **Triangulation vs. Citation**: Triangulation ensures the referenced paper actually asserts what the AI claims it does, whereas citation checks confirm the bibliographic metadata is genuine. - **Cross-Model Dynamics**: Pitting competing models against one another surfaces blind spots, but agreement does not guarantee factual ground truth. - **Regression Validation**: Enforces consistency in labels, entities, and constraint compliance throughout long-horizon project lifecycles.

---

[20:33](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=1233s) From Code Generation to Code Validation and Deep Understanding

The rise of generative AI fundamentally reshapes professional workflows, shifting the primary human skill from generating syntax to reviewing, validating, and auditing generated assets. While industry commentators often claim that junior programming roles will vanish because AI can draft code, the actual engineering bottleneck has shifted: writing boilerplate code is trivial, but determining whether that code is secure, compliant, and architecturally sound requires rigorous evaluation.

Crucially, "a validated result is great, but it does not equate to understanding." Having an AI-generated script pass automated unit tests does not relieve the human practitioner of responsibility. If an engineer deploys code into production without understanding its underlying mechanics, edge cases, and failure boundaries, the application will eventually falter. When systems fail, practitioners cannot point to generative tools as a scapegoat—accountability rests entirely with the human in the loop. Validation must therefore be embraced not as an afterthought or a one-off task, but as an essential, repeatable discipline centered on genuine comprehension.

- **Skill Shift**: Modern technical work moves away from manual syntax writing toward reading, auditing, and validating complex outputs. - **Accountability**: Practitioners retain 100% legal, operational, and organizational responsibility for deployed outputs; "the AI did it" is never an acceptable defense. - **Validation vs. Comprehension**: Rigorous automated validation must be paired with human understanding of the system's internal logic and execution path.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=4s) This is your answer — Robert Bierman

> [00:04](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=4s) vibe coding — Robert Bierman

> [00:04](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=4s) a broken clock is right twice a day — Robert Bierman

> [03:59](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=239s) B-path — Robert Bierman

> [03:59](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=239s) 100% plausible, potentially functional, and absolutely incorrect — Robert Bierman

> [11:06](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=666s) If they differ, you know something is wrong. If they are the same, you don't know anything. — Robert Bierman

> [11:06](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=666s) Atomic Claim Validation — Robert Bierman

> [11:06](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=666s) Regression Validation — Robert Bierman

> [20:33](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=1233s) a validated result is great, but it does not equate to understanding. — Robert Bierman

> [20:33](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=1233s) the AI did it — Robert Bierman

## Mentioned Robert Bierman, San Francisco State University, Ai4, ChatGPT, Claude, C, SQL, ABI, RAG

## Video description Robert Bierman, Lecturer, San Francisco State University

How can we tell whether a polished generative AI response is actually correct?

I will explain why hallucinations, fabricated citations, outdated information, faulty reasoning, and violated constraints make validation a system-level problem rather than a simple fact-checking step. I will present fourteen validation methodologies organized into four layers: evidence grounding, model-behavior signals, claim and logic validation, and operational safeguards. I will also illustrate a practical validation pipeline and explain how to select multiple independent methods based on the risk and consequences of an incorrect response. This talk will provide a repeatable framework for moving from plausible AI output to responsibly validated—and genuinely understood—information.

[Source](https://www.youtube.com/watch?v=d1V7xWoxwQo)

中文简报

专题研讨 1 - 4：生成式 AI 响应验证方法论 Robert Bierman 解释称，大语言模型生成的是在统计上合理的 token 序列，而非经过验证的事实，看似自信的陈述下往往掩盖着关键的逻辑错误、规范违背和幻觉。针对这些脆弱性，他提出了分属四个层级的十四种验证方法：证据锚定、模型行为信号、主张与逻辑验证以及运行安全保障。他阐述了结构性检查与跨模型分析如何揭示隐藏缺陷，并建议根据运行风险至少采用两到三种独立的验证方法。最后，他强调技术工作流正从语法生成转向严格审计，指出自动化验证必须与人类真实的理解及问责相结合。 大语言模型本质是概率性的 Token 预测器而非推理引擎，表面输出的自信度不能作为事实准确性的依据，“凭感觉写代码”（Vibe coding）会掩盖虚构引用和底层逻辑漏洞。 (0:04) AI 生成的代码即使看似合理且能成功执行，仍可能严重违反 ABI 调用约定等系统底层规范；稳健的验证策略要求在低风险任务中至少结合两种独立验证方法，关键任务则需三种以上。 (3:59) 验证方法涵盖证据支撑、模型行为信号、主张与逻辑验证、运营保障四个层级；跨模型对比虽能暴露分歧但一致性无法保证真相，而原子主张验证能将复杂结论拆解以防中间步骤失误污染整体结果。 (11:06) 技术人员的核心能力已从手写代码转向审查、验证与审计，自动化测试通过并不等同于真正理解系统逻辑，人类仍需对部署后的结果承担全部责任。 (20:33)

Reading summary

According to the speaker at Ai4, large language models generate statistical approximations rather than verified truths, meaning surface-level plausibility routinely masks critical structural and factual errors. Technical teams must therefore replace intuitive 'vibe coding' with a disciplined, multi-layered validation framework while maintaining full human accountability. Architectural Pitfalls and the Illusion of Plausibility (0:04) Generative AI systems deliver outputs with supreme confidence, yet their underlying architecture is designed strictly to predict probable token sequences rather than evaluate ground truth. Accepting these outputs at face value—a habit described as 'vibe coding'—proves dangerous because plausibility frequently conceals fatal logic gaps, security vulnerabilities, and outright hallucinations. Documented real-world failures, such as practicing attorneys sanctioned for citing fabricated judicial rulings and academic researchers publishing non-existent references, demonstrate that superficial correctness cannot be mistaken for factual reliability. Hidden Structural Failures and Multi-Method Rules (3:59) Evaluating generated code or queries solely by immediate business logic misses systemic failure modes, such as knowledge staleness from training cutoffs, prompt bias, and structural convention violations. To illustrate this, the speaker cites an experiment where computer science students prompted ChatGPT to translate C code into assembly. While the output appeared plausible and executable, it violated Application Binary Interface (ABI) register preservation rules by indiscriminately overwriting preserved registers. When prompted to fix the fault, the model merely substituted a different set of register violations. Because single-technique evaluations consistently fail to catch such subtle defects, the speaker recommends a concrete operational rule of thumb: practitioners must combine at least two independent validation methodologies for low-stakes tasks, and at least three for mission-critical deployments like medical diagnostics and legal filings. A Four-Layer Framework for Systematic Defense (11:06) To catch errors comprehensively, the presentation organizes fourteen validation techniques across four functional layers: Evidence Grounding, Model-Behavior Signals, Claim and Logic Validation, and Operational Safeguards. Evidence grounding distinguishes between verifying that a cited source exists and conducting source triangulation to ensure the document actually supports the specific claim. Model-behavior signals, such as comparing outputs across independent models like ChatGPT and Claude, help reveal discrepancies; however, the speaker cautions that cross-model agreement does not guarantee accuracy, as different models can share identical underlying errors. Within the logic and operational tiers, atomic claim validation deconstructs complex assertions into isolated facts to stop flawed intermediate steps from corrupting aggregate results. In addition, teams should implement regression validation—adapting software regression suites to continually enforce consistent terminology, constraints, and citations across iterative generations—alongside adversarial red teaming and temporal monitoring. The Shift to Auditing and Human Comprehension (20:33) As generative models automate basic syntax generation, the primary engineering bottleneck shifts from drafting boilerplate code to auditing, validating, and securing AI outputs. Passing automated unit tests is insufficient on its own; deploying code without deeply understanding its underlying mechanics, edge cases, and failure boundaries creates severe operational risk. Crucially, automated validation does not relieve human practitioners of responsibility. Organizations and individual engineers retain complete legal and operational accountability for deployed software, meaning reliance on generative tooling cannot serve as an excuse when systems fail. While structured validation layers significantly reduce generative risks, the presentation concludes that automated tooling cannot substitute for human technical comprehension. Validation must be treated as a repeatable, integrated engineering discipline, with practitioners remaining fully accountable for auditing and understanding every deployed output.

精炼阅读摘要

演讲者指出，大语言模型本质上是基于统计概率预测生成内容的预测模型而非具备真正理解力的智能体，其表现出的高度自信极具欺骗性；因此技术人员绝不能仅凭直觉接受其输出，而必须将多层独立验证确立为一项系统性的工程纪律。 概率生成机制与表面合理性风险 (0:04) 大语言模型在输出答案时往往展现出极高确信度，但这并不代表事实真实性。这类系统是计算并输出统计上符合用户期望的词元序列，并非真正理解客观现实。这种架构缺陷在现实中已引发多起严重后果，例如执业律师在法庭诉状中援引模型虚构的判例而面临司法制裁，以及学者发表包含捏造参考文献的论文。 依赖所谓“直觉编程”（vibe coding）——即仅仅因为生成的代码看起来合理、感觉顺畅便直接采纳——存在巨大安全隐患。正如坏掉的钟表一天也能对两次，表面上的合理性极易掩盖关键的逻辑缺陷、安全漏洞与脆弱性。因此，验证绝不能被当作项目冲刺末尾草草附加的事实核对，必须成为一种内嵌的、系统化的工程纪律。 结构性失效与多重独立验证准则 (3:59) 单一维度的检验手段往往会遗漏系统性错误。例如在评估生成的 SQL 查询或脚本时，仅仅检验其是否符合当下的业务逻辑，会忽视异常处理、网络安全防护以及针对用户非预期操作的“B路径测试”（beta-path testing）。大语言模型常见的系统性失效源自幻觉、因训练数据截止日期导致的知识滞后（如询问地缘政治事件会随提问时间给出自相矛盾的回答）、因提示词措辞变化引发的模型偏见，以及对底层结构约定的破坏。 为展示结构性失效，演讲者列举了计算机系统教学中的实验：让 ChatGPT 将 C 语言程序翻译为汇编代码，学生因代码看似可以运行而误判其正确，但实际上它彻底违反了应用程序二进制接口（ABI）的调用约定与寄存器保护规则，无差别地覆写暂存与保留寄存器。当提示模型依据该规则修复时，它生成的新代码又违反了另一组寄存器规范，表现出“完全看似合理、潜在可运行却绝对错误”的特征。鉴于单一方法无法拦截所有问题，演讲者提出经验准则：低风险任务必须结合至少两种独立的验证方法，而对于涉及法律、医疗等高风险任务，则必须采用三种或更多独立方法。 四大功能层级与十四种验证方法 (11:06) 演讲者提出了分布在四个功能层级中的十四种具体验证方法，以形成全面防线。在证据锚定层，来源三角验证负责核实原文档是否真正包含模型声称的主张，引用核查负责检验文献元数据是否真实，检索增强生成（RAG）与规范锚定则将生成内容约束在权威技术手册与规范之中。在模型行为信号层，可通过自我一致性、不确定性量化以及跨模型比对暴露潜在问题。例如在识别芯片照片型号时，可让 ChatGPT 与 Claude 针对彼此的型号差异展开辩论；但演讲者强调跨模型验证只能作为辅助，因为两款模型可能存在相同的底层盲区，两者一致并不等于符合事实。 在论点与逻辑验证层，团队应推行“原子化断言验证”（Atomic Claim Validation），将复杂的复合陈述拆解为相互独立的细分事实分别核验，防止错误的中间步骤污染整体输出；同时应引入借鉴自传统软件工程的“回归验证”（Regression Validation），建立自动化流水线，确保专业术语、文档引用与架构约束在持续迭代生成中保持稳定。最后在运行保障层，通过保留人类介入（Human-in-the-Loop）确保定性监督与问责，并辅以对抗性红队测试和时间有效性检查以防范现实漂移。 从代码生成转向审查与深度理解 (20:33) 生成式 AI 的普及从根本上重塑了专业人员的技能需求，工程实践的核心能力已从手动编写代码语法转向阅读、审计与验证生成资产。尽管业界经常出现初级程序员岗位将被 AI 取代的论调，但实际的工程瓶颈在于评估生成的样板代码是否安全、合规并具备健全架构，而这一步必须依赖扎实的专业水准。 演讲者特别强调，“通过验证的结果并不等同于深度理解”。一份 AI 生成的代码即使通过了自动化单元测试，也不代表技术人员可以放弃深入研究。若工程师在未完全理解其内部逻辑、边缘工况及失效边界的情况下将其部署至生产环境，系统终将出现故障。在发生生产事故时，使用者绝不能将生成式模型作为借口，人类监督者承担全部法律、运营与组织责任。因此，验证不仅是一种机械化检查，更是一种以彻底理解为前提的工程责任守则。 演讲总结指出，尽管十四种跨层验证方法构筑了系统的防护体系，但自动化验证永远无法免除从业者的理解责任与最终问责。面对模型在知识时效性、底层架构约定及复杂逻辑上的固有限制，工程师必须将多维独立验证与对系统底层机制的深刻掌握相结合，警惕表面合理性所带来的技术盲目。

Detailed notes — zh

## Background 该内容由 Ai4 频道于 2026 年 8 月 15 日发布在 YouTube 平台。内容源自旧金山州立大学讲师罗伯特·比尔曼（Robert Bierman）的主题演讲，主要探讨为何生成式 AI 带来的幻觉、伪造引文与逻辑漏洞需要系统级验证，并提出了涵盖证据溯源、模型行为信号、断言逻辑及操作保障的四层十四种验证方法论。 ## TL;DR 比尔曼指出大语言模型本质仅为概率预测工具，常会生成看似合理却完全错误的“自信幻觉”，使验证工作成为不可或缺的系统工程。为此，他系统阐释了包含证据溯源、模型行为信号、断言与逻辑验证、运维保障在内的四层验证框架，并提出原子断言拆解与跨模型对抗等具体手段。针对高风险场景，他建议组合多种独立验证方法并引入自动化回归测试流水线。最后他强调，行业范式已从“编写代码”转向“阅读与验证代码”，且自动化验证通过绝不等于人类真正理解了底层逻辑，人机协同中的人类责任始终不可替代。 ## Atmosphere & style 演讲开场通过全场起立互动迅速活跃气氛，比尔曼以大学老牌技术极客特有的风趣与犀利，将汇编语言教学翻车和两款大模型互相挑刺的真实经历讲得生动幽默。在轻松诙谐的案例剖析背后，整体叙事充满对“感觉编码”侥幸心理的严厉警示与深刻反思。整场分享节奏明快、架构严谨，带给听众一种既如释重负又倍感工程纪律沉重的专业沉浸感。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=4s) 开场破冰：直面生成式 AI 的“自信幻觉” 演讲一开始，旧金山州立大学讲师罗伯特·比尔曼（Robert Bierman）并没有直接切入枯燥的理论，而是让全场听众起立互动。他向台下抛出两个问题：“在座有谁从未使用过生成式 AI？没用过的请坐下。”全场几乎无人坐下。接着他又问：“有谁在使用生成式 AI 后，看到生成的结果心想‘嘿，这看起来太棒了，我就用这个了’？如果是这样，请继续站着；如果没有，请坐下。”面对依然站立的大批听众，比尔曼笑着说：“很好，今天这场演讲正是为你们准备的。” 比尔曼顺着前一位讲者的议题切入核心：当 AI 给出一个极其笃定、看似权威的回答“这就是你的答案”时，我们究竟该如何判断它的真伪？它真的是正确答案吗？现实中屡见不鲜的翻车案例早已敲响警钟：有律师因在法庭简报中引用 AI 捏造的虚假判例而遭到法院重罚与通报；有学者发现 AI 论文助手编造了根本不存在的参考文献。比尔曼一针见血地指出了大语言模型的底层现实：“AI 并不具备真正的智能，它只是一个预测模型，仅仅是在根据上文预测并输出它认为你接下来想看到的内容。”因此，在日常工作流程中建立一套系统化的验证机制，成为确保 AI 输出可信与可落地的当务之急。 ### [02:14](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=134s) “感觉编码”与浅层验证的致命缺陷 面对一个看似合理（plausible）的模型输出，人们极易陷入认知盲区。比尔曼提到了当下流行的“感觉编码”（vibe coding）——“哦，感觉这段代码挺对的”，但他随即冷峻地反驳：“坏掉的时钟一天也能对两次。”仅仅依靠直觉或事后补救是极其危险的。许多团队抱有一种侥幸心理，认为“等部署到生产环境时再检查”或者“随便跑几个测试用例就行了”，但这种浅层验证根本无法抵御系统性风险。 比尔曼以前一位讲者提到的 SQL 代码生成为例展开剖析：一段由 AI 生成的 SQL 语句，即便语法完全正确，甚至能顺利跑通基础业务逻辑，它依然可能是一段糟糕至极的代码。它是否考虑了异常边界情况？能否抵御 SQL 注入等网络安全攻击？更重要的是，它是否经过了“B 路径测试”（B-path testing）——即真实用户实际使用系统时的偏离路径，而非开发者设想中用户“应当”遵循的理想路径？正因如此，比尔曼强调没有任何单一的检验手段能够彻底锚定真实性，必须构建一个覆盖多个维度的系统性验证框架。 ```mermaid flowchart TD accTitle: 四层生成式 AI 验证框架 accDescription: 展示从证据溯源、模型行为信号、断言与逻辑验证到操作保障的四层架构 subgraph L1["第一层：证据溯源 (Evidence Grounding)"] direction LR A1["数据源三角验证"] --> A2["引文真实性核验"] A2 --> A3["RAG 检索与规范比对"] end subgraph L2["第二层：模型行为信号 (Model-Behavior Signals)"] direction LR B1["自洽性测试"] --> B2["跨模型交叉比对"] B2 --> B3["置信度/不确定性量化"] end subgraph L3["第三层：断言与逻辑验证 (Claim & Logic Validation)"] direction LR C1["原子断言拆解"] --> C2["推理连贯性检查"] C2 --> C3["规则与安全约束校验"] end subgraph L4["第四层：操作保障 (Operational Safeguards)"] direction LR D1["自动化事实核查"] --> D2["回归测试流水线"] end L1 --> L2 L2 --> L3 L3 --> L4 ``` ### [04:48](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=288s) 跨模型交叉验证的趣味与局限 为了系统性捕捉不同类别的错误，比尔曼归纳出了包含 14 种具体验证方法的四层框架：证据溯源（Evidence Grounding）、模型行为信号（Model-Behavior Signals）、断言与逻辑验证（Claim and Logic Validation）以及操作控制/保障（Operational Safeguards）。比尔曼特别指出，在涉及医疗诊断等高危场景时，使用者至少需要同时采用其中三个层级的方法。至于“模型行为信号”，他通常将其视为“可选层”，因为让一个模型去验证另一个模型时，两者完全有可能同时犯错。“但如果它们给出的答案截然不同，你就知道肯定有什么地方出错了；而如果它们给出的答案完全相同，其实你依然什么都无法确定。” 比尔曼分享了一个生动的亲身经历：他曾手头有一张电子芯片的照片，上面印着几串晦涩古怪、并不直接显示型号的编码。他想知道这究竟是哪款芯片，于是分别询问了 ChatGPT 和 Claude，结果两个模型给出了截然相反的型号结论。比尔曼并没有盲目采信任何一方，而是把 ChatGPT 的回答喂给 Claude，再把 Claude 的回答喂给 ChatGPT，质问双方：“为什么对方会得出那个结论，而你给出的又是这个？”两款模型随即开始互相挑刺与辩驳，就像两个争强好胜的小孩在争论谁的衣服更好看一样。这种跨模型对抗确实提供了一种验证视角，但它并不能提供 100% 的确定性。 ### [06:38](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=398s) 幻觉、时效性偏差与汇编语言踩坑实录 在剖析大模型的系统性问题时，比尔曼梳理了几个关键致错因素：幻觉、知识滞后（Knowledge Staleness）、校准失败与提示词偏差。知识时效性问题尤为突出，以地缘冲突为例，“伊朗战争结束了吗？”如果三周前问模型是一个答案，昨天问是第二个答案，今天问可能又变成第三个答案；如果所处理的任务具有强时效性，模型的切断日期和依赖数据就必须接受严格核查。此外，提问方式本身就会诱导输出偏见，“我们提问的方式会直接左右结果，这叫什么？正是偏差（Bias）。” 为了展示 AI 在专业底层结构上的严重漏洞，比尔曼分享了自己在大学计算机系任教时的真实教学案例。作为教授汇编语言和操作系统等底层硬核课程的“老极客”，他曾让 ChatGPT 将一段简短的 C 语言程序转译为汇编代码。代码生成后表面上逻辑通顺、似乎可以运行，比尔曼将代码拿到本科生课堂上问学生：“这段代码正确吗？”全班学生异口同声地回答：“正确。”然而真实情况是“差得十万八千里”。在底层汇编中，函数调用必须严格遵守调用约定（ABI）中关于寄存器保存的规范——哪些属于必须在调用后恢复原值的保留寄存器（preserved registers），哪些属于可被随意覆盖的暂存寄存器（scratch registers）。ChatGPT 生成的代码完全践踏了这一规范；虽然在脱离上下文的独立执行中看似勉强能跑，但只要系统中其他模块依赖这些寄存器，整个程序就会崩溃。更滑稽的是，当比尔曼明确指出它违反了寄存器约定并要求其修正时，ChatGPT 重新生成了一整版全新代码，却又违背了另一组截然不同的寄存器规范。 - 模型的致命特征：生成出的内容往往是“100% 显得合理、潜在具备功能性，却彻底错误”。 - 单一验证手段的局限：没有任何一种孤立的检查方法能够捕获所有类型的系统失效。 - 验证分级准则：比尔曼建议在低风险场景下至少组合 2 种独立验证方法；而在法律简报、医疗诊断等高风险领域，必须强制组合至少 3 种独立方法反复校验。 ### [11:06](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=666s) 验证方法论拆解：从证据溯源到原子断言 比尔曼展示了全场最核心的一张架构胶片，戏称“如果要拍照发推，拍这张就够了”，并逐一拆解各个层级的方法： **第一层：证据溯源（Evidence Grounding）** - **数据源三角验证（Source Triangulation）**：核实引用的来源是否存在，以及原文是否真正包含 AI 声称的内容。比尔曼曾亲自根据 AI 提供的文献检索下载了一篇真实论文，通读后却发现“原文根本不是那个意思”。 - **引文真实性核验（Citation Verification）**：排查虚假文献，或检查是否在真实论文中凭空捏造了合著者。 - **检索增强生成（RAG）与规范比对**：强迫模型去翻阅官方技术手册，例如将寄存器操作强制对齐官方架构白皮书与 ABI 规范文档。 - **结构化知识库对齐（Structural Knowledge Base Grounding）**：验证内容在语法与领域结构标准上的硬性合规。 **第二层：模型行为信号（Model-Behavior Signals）** - **自洽性测试（Self-consistency）**：用微调后的不同提问形式多次测试同一模型，观察结论是否保持稳定。 - **跨模型交叉比对（Cross-model crisscrossing）**：引入 Claude 与 ChatGPT 等不同架构基底的模型交叉校验。 - **置信度与不确定性量化（Confidence and Uncertainty Quantification）**：如果底层接口能够暴露统计概率，输出是处于 99% 的确定性还是仅有 51% 的抛硬币水平，将直接决定输出是否可用。 **第三层：断言与逻辑验证（Claim and Logic Validation）** - **原子断言拆解（Atomic Claim Validation）**：这是一种听起来常识但极易被忽略的工程方法。将 AI 生成的复合性回答粉碎为若干个独立的单项陈述（原子断言），并对每一项逐一独立验证。聚合在一起的复杂陈述在结尾可能看似逻辑自洽，但只要中间夹杂了一个虚假的前提断言，整体方案往往就会在实操中轰然倒塌。 - **推理与前后一致性检查（Reasoning and Consistency Checks）**：在大规模工程开发或长篇科研论证中，监控 AI 是否在多轮对话的前后回答中出现逻辑自我矛盾。 - **规则与约束遵循（Rule Constraints）**：如汇编代码的寄存器约束、业务逻辑边界等硬性契约审查。 - **安全与鲁棒性验证（Security）**：例如生成的 SQL 语句即使业务逻辑无误，也必须审查是否存在 SQL 注入漏洞。 在介绍完逻辑层后，比尔曼将话题引向最后一层——操作保障中的自动化事实核查流水线，并向台下发问：“在座有多少人熟悉自动化回归测试？”由此拉开了将 AI 验证工程化落地的讨论序幕。 ### [16:51](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=1011s) 回归验证与运维保障：人类在回路中的不可替代性 在构建系统化验证体系时，回归验证（Regression Validation）扮演着类似于传统软件工程中回归测试的角色。随着项目推进和 AI 生成内容的不断累积，用户必须预先设定一套严密的判定标准，并在每次内容迭代生成时重新跑一遍验证流程。这包括反复核对所有专业术语是否前后一致、使用的命名标签是否统一、对引用源文档的指代是否存在偏差。若没有这种持续性的回归检查，早期生成的微小偏差很容易在后续迭代中演变成严重的系统性错误。 在所有验证维度中，最为关键的当属“运维保障”（Operational Safeguards），其核心基石正是“人类在回路”（Human in the Loop）。任何自动化工具都无法替代人类的审视、主观解读以及对复杂语境的深度分析能力。最终使用 AI 生成结果并将其投入实际业务的责任完全由人类承担；当系统出现重大事故或产出荒谬结论时，开发者和员工根本不可能对上级推卸说“哦，失败了，那全是 AI 的错”。无论是应对上级问责还是确保系统稳定，责任始终在人。为此，团队还需要引入对抗性红队测试（Adversarial Red Teaming）主动寻找并击穿模型的逻辑漏洞，并配合时效性检查（Temporal Validity Checks）以确保生成内容引用的信息与数据未随时间推移而失效。 - **回归验证的核心关注点**：跨迭代的术语一致性、实体标签统一性、源文档引用准确性。 - **运维保障的关键防线**： - **人在回路（Human in the Loop）**：人类承担最终解读、审核与结果责任。 - **对抗性红队测试（Adversarial Red Teaming）**：采用对抗性 Prompt 和边界条件主动测试并破坏模型输出。 - **时效性检查（Temporal Validity Checks）**：验证生成数据与引用事实是否依然最新且有效。 --- ### [19:00](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=1140s) 验证管线构建与原子主张的应用实践 构建可靠的生成式 AI 系统，绝不能把验证视作事后偶尔为之的“一次性修补”，而必须建立一套成体系、可复用的验证管线（Validation Pipeline）。在此类方法论中，源信息三角交叉验证（Source Triangulation）与原子主张分解（Atomic Claim Validation）尤为实用。原子主张分解法将大段复杂的 AI 生成文本拆解为一个个最小维度的独立事实主张，并逐条核对事实依据。这种方法不仅能够精准捕获细微的幻觉错误，还能自动生成详尽的“审计追踪记录”（Audit Trail），清晰展现系统在哪些逻辑节点曾出现偏差、又是如何被拦截与修正的。 ```mermaid flowchart TD accTitle: Generative AI Validation Pipeline accDescr: A structured pipeline showing evidence grounding, model behavior signals, claim validation, and operational safeguards leading to validated understanding. subgraph Layer1["第 1 层：证据基础与行为信号"] A["生成式 AI 原始输出"] --> B["源信息三角交叉验证"] A --> C["模型行为与不确定性分析"] end subgraph Layer2["第 2 层：主张分解与逻辑核查"] B --> D["原子主张分解 (Atomic Claims)"] C --> D D --> E["生成审计追踪记录 (Audit Trail)"] end subgraph Layer3["第 3 层：运维防线与回归验证"] E --> F["时效性核查与红队测试"] F --> G["回归验证 (Regression Validation)"] end subgraph Layer4["第 4 层：人类闭环"] G --> H["人在回路 (Human in the Loop)"] H --> I["严谨理解与实际业务应用"] end ``` 整套验证框架不仅是一份技术规范，更是企业应对生成式 AI 落地风险的操作手册。通过将原子主张核验、对抗性测试与时间有效性检查串联为流水线，团队能够以制度化的方式排查错误，使 AI 生成的每项输出都具备经得起追溯的技术支撑。 --- ### [20:45](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=1245s) 行业范式转移：从“编写代码”到“验证与真正理解” 当前人工智能的发展正在重塑整个软件工程与知识生产行业的底色。随着 AI 能够轻而易举地自动生成代码，传统初级程序员（Junior Programmer）的岗位与工作模式正在发生巨变。然而，"AI 能写出代码"并不意味着问题已经解决，核心矛盾已经转向：谁来证明这些代码究竟是否正确？行业的工作范式正在从传统的“编写代码”（Writing Code）转向“阅读代码”（Reading Code）与“验证代码”（Validating Code）。 验证不再只是开发流程末尾的一个附加步骤，而是一门必须融入日常工作流的工程纪律（Discipline）。开发者从繁琐乏味的 Python 脚本编写或基础业务编码中解脱出来后，应当将节省下来的时间和精力全力投入到验证机制的设计与评估中。更深层次的警示在于：**一个经过严格验证的结果固然很好，但它绝不等于你真正理解了它（"A validated result is great, but it does not equate to understanding"）。** 即使 AI 生成的代码能够 100% 完美运行并顺利通过各项测试用例，但如果开发者被问及底层运行逻辑时一无所知，整个系统依然极其脆弱。缺乏对输出内容的透彻理解，系统在后续的维护、扩展与复杂现实应用中必将面临崩溃与失控。 - **核心技能重心转移**：由单纯的 Coding（代码编写）全面转向 Reading（代码阅读）与 Validating（逻辑与结果验证）。 - **验证与理解的本质区别**：能够跑通的“100% 正确输出”不代表人类具备了把控能力；没有人类的真正理解，任何 AI 应用落地都潜藏着系统性失效的风险。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=4s) 有谁在使用生成式 AI 后，看到生成的结果心想‘嘿，这看起来太棒了，我就用这个了’？如果是这样，请继续站着；如果没有，请坐下。 — Robert Bierman > [00:04](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=4s) AI 并不具备真正的智能，它只是一个预测模型，仅仅是在根据上文预测并输出它认为你接下来想看到的内容。 — Robert Bierman > [02:14](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=134s) 坏掉的时钟一天也能对两次。 — Robert Bierman > [04:48](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=288s) 但如果它们给出的答案截然不同，你就知道肯定有什么地方出错了；而如果它们给出的答案完全相同，其实你依然什么都无法确定。 — Robert Bierman > [06:38](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=398s) 我们提问的方式会直接左右结果，这叫什么？正是偏差（Bias）。 — Robert Bierman > [06:38](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=398s) 100% 显得合理、潜在具备功能性，却彻底错误 — Robert Bierman > [11:06](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=666s) 如果要拍照发推，拍这张就够了 — Robert Bierman > [16:51](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=1011s) 哦，失败了，那全是 AI 的错 — Robert Bierman > [20:45](https://www.youtube.com/watch?v=d1V7xWoxwQo&t=1245s) A validated result is great, but it does not equate to understanding — Robert Bierman ## Mentioned Robert Bierman, 旧金山州立大学 (San Francisco State University), Ai4, ChatGPT, Claude, Python, C 语言, SQL, RAG (检索增强生成), ABI (Application Binary Interface) ## Video description Robert Bierman, Lecturer, San Francisco State University How can we tell whether a polished generative AI response is actually correct? I will explain why hallucinations, fabricated citations, outdated information, faulty reasoning, and violated constraints make validation a system-level problem rather than a simple fact-checking step. I will present fourteen validation methodologies organized into four layers: evidence grounding, model-behavior signals, claim and logic validation, and operational safeguards. I will also illustrate a practical validation pipeline and explain how to select multiple independent methods based on the risk and consequences of an incorrect response. This talk will provide a repeatable framework for moving from plausible AI output to responsibly validated—and genuinely understood—information. [Source](https://www.youtube.com/watch?v=d1V7xWoxwQo)

Source quotes

[0:04] Robert Bierman: 有谁在使用生成式 AI 后，看到生成的结果心想‘嘿，这看起来太棒了，我就用这个了’？如果是这样，请继续站着；如果没有，请坐下。 [0:04] Robert Bierman: AI 并不具备真正的智能，它只是一个预测模型，仅仅是在根据上文预测并输出它认为你接下来想看到的内容。 [2:14] Robert Bierman: 坏掉的时钟一天也能对两次。 [4:48] Robert Bierman: 但如果它们给出的答案截然不同，你就知道肯定有什么地方出错了；而如果它们给出的答案完全相同，其实你依然什么都无法确定。 [6:38] Robert Bierman: 我们提问的方式会直接左右结果，这叫什么？正是偏差（Bias）。 [6:38] Robert Bierman: 100% 显得合理、潜在具备功能性，却彻底错误 [11:06] Robert Bierman: 如果要拍照发推，拍这张就够了 [16:51] Robert Bierman: 哦，失败了，那全是 AI 的错 [20:45] Robert Bierman: A validated result is great, but it does not equate to understanding

Source quotes

[0:04] Robert Bierman: This is your answer [0:04] Robert Bierman: vibe coding [0:04] Robert Bierman: a broken clock is right twice a day [3:59] Robert Bierman: B-path [3:59] Robert Bierman: 100% plausible, potentially functional, and absolutely incorrect [11:06] Robert Bierman: If they differ, you know something is wrong. If they are the same, you don't know anything. [11:06] Robert Bierman: Atomic Claim Validation [11:06] Robert Bierman: Regression Validation [20:33] Robert Bierman: a validated result is great, but it does not equate to understanding. [20:33] Robert Bierman: the AI did it

Original transcript

[0:04] We've sat through a few presentations here. So,  why don't we all stand up for a second? How many [0:09] here have never used generative AI? If you've  never used generative AI, go ahead and sit down. [0:21] Okay. How many here have used generative AI,  gotten the result and said, "Hey, yeah, that [0:29] looks great. I'll That's That's what I'll use."  If that's you, stay standing. If it's not you, [0:36] you can sit down. Okay, for all those that  are still standing, this talk's for you. [0:45] Our prior speaker talked a lot about,  you know, framework and validation. [0:49] How do we actually understand what we got from  this very confident reply, "This is your answer." [1:07] Is it? [1:11] We've all heard the cases of things like [1:14] lawyers who get fined or cited  for having invalid precedents. [1:22] Uh researchers who have hallucinated references. [1:29] These are things that we need to  understand. AI isn't intelligent. [1:39] It's a predictive model. It's giving  you what it thinks you want next. [1:49] So, how do we ensure that  what we use, how we apply it, [1:58] is something we can trust. How can we ground it? [2:03] And that was the premise in which I came on this  here. And what I came up with was a framework. [2:13] A framework for systematic  integration in your normal workflow. [2:20] Your day-to-day workflow of how to validate AI.  So the question is, is the response correct? [2:31] It's very easy to ask the question [2:33] and the response it gives you  is something it looks plausible. [2:39] It feels right. How many of you ever heard of vibe  coding? Oh, it feels like it's the right code. [2:47] A broken clock is right twice a day. [2:52] So how do we know? How are we  sure? How can we look at and [2:59] take this? So in reality what we come up with,  validation is a separate form. It doesn't [3:10] isn't something you can just say, "Oh  well, I'll just check that answer when we [3:14] put it in production." Or we'll just put it  on and run it and run a couple test cases. [3:23] The prior one talked about SQL is it correct?  Validating, is it correct code? Does it [3:29] meet the business logic? All of that could  be true and it could still be lousy code. [3:35] Does it handle the error conditions?  Does it handle cyber attacks? [3:42] Does it take into account the B path testing? [3:47] The beta path that the way that people might  use it, not the way people should use it. [3:55] So, this is really where [3:59] I came and was thinking about how do we validate?  There is no one single validation that is going [4:06] to give you a grounded answer. So, we have  to look at things like evidence grounding. [4:12] How do we ground that answer? What  is the model's behavior? If I ask [4:20] Chat GPT a question in two slightly different  ways, does it give me the same answer? [4:28] If I ask Chat GPT a question and I ask Claude the  exact same question, do I get the same result? [4:40] Claims and logic and operational controls.  These are the layers that I came up with. [4:48] I came up with 14 distinct methodologies for  validating what generative AI can give us. [5:00] And grouping them into these layers, [5:03] the reason for these layers is each one of these  layers catch different categories of errors. [5:12] And these different categories are what we have  to think about. If you want if you're doing a [5:17] medical diagnosis, and you're relying on the  answer from that medical diagnosis from AI, you [5:25] should be doing at least three of these. Really,  this is the only one I always consider optional [5:31] because we're having one model validate  another model, they can both be wrong. [5:37] But if they differ, you know something is wrong.  If they're the same, you don't know anything. [5:47] But it's really funny. I actually did a very  interesting thing. I had a picture of a chip and I [5:51] needed to know what model chip it was and  of course it only has these weird numbers [5:55] on the chip that don't really  tell you the model number. [5:59] And chat GPT gave me one answer and Claude  gave me a completely different answer. [6:07] And I had to say, "Okay. I gave both the answers  that the other gave back to the other ones. [6:15] And say, "Why did this give me this  and why did this give me that?" [6:20] That's actually one of the forms  you can use for validation. [6:24] But is that 100% reliable? No. So,  the model behavior is an easy one, [6:29] but it doesn't give you an absolute. These are  the three are real solid grounded validations. [6:38] So, why? What What is What's our  systematic problem? Hallucinations. Hey, [6:47] I'm sitting here over This room's full. [6:55] Knowledge staleness. These models, Claude, chat  GPT, they're based upon a certain cutoff date. [7:08] Is the Iran war over? [7:12] Well, if you asked it 3 weeks ago, it would  give you one answer. If you asked it yesterday, [7:18] it would give you another answer. If you ask  it today, it would give you a third answer. [7:24] We have to make sure that if we're doing  something that has contemporaneous meaning, [7:31] is the information that it's  relying on accurate enough? [7:37] The inconsistent answers, the collab the  calibration failures. The way you ask a question [7:46] influences the result. What do we call that?  The way I ask a question influences the result. [7:54] Bias, exactly. Yes. We can  introduce bias into our response [8:03] just based on the way we ask the  question. And structural failures. [8:09] This is one that I had a lot of fun with.  I teach computer science. Yes, I'm a nerd. [8:15] I teach low-level stuff, assembler,  operating systems, the stuff nobody wants. [8:25] Well, in my assembler class, I had a really  little C program, and I asked ChatGPT to give [8:32] me the assembler for this code, and it generated  the assembly. The assembly code could work. [8:42] I asked my class, "Is this code correct?" [8:47] Of course, this is an undergraduate  course, and they all said, "Yes." [8:51] Was it? Not even close. When we talk about  assembler, there's there's certain conventions [8:59] we have to abide by. Certain registers that  have to be saved, preserved when you do a call, [9:07] and certain registers which can be used as  scratch registers. It completely violated that. [9:12] Would the code potentially work? Yes. If nobody  else was relying on certain of the registers [9:20] that it did. So, sometimes it might  work, and sometimes it might not. [9:24] I asked it to fix that, pointing at the fact that  it was violating the register convention, and so [9:30] it generated me a whole new version that violated  a whole different set of the register conventions. [9:39] And so, this is a real situation. It gave  me an answer which was 100% plausible, [9:49] potentially functional, [9:54] and absolutely incorrect. One method of looking at  these things rarely catches every type of failure. [10:03] Just this is the reality. This is what we  have to face. This is one of the things [10:08] I'm actually working on right now  to introduce into our curriculum. [10:15] A pedagogy in which we actually teach  students how to properly use AI. [10:23] This applies to industry as well.  It applies to every one of you here. [10:30] We need a reliable validation, and to do so,  we have to combine different methodologies, [10:36] different layers of this framework. So,  best practice combine at least three. [10:43] I say that here in the pedagogy I'm working  on, I say two when the stakes are lower. [10:51] But three if the If you're doing a legal  brief, you want three. If you're doing a [10:56] medical diagnosis, you want three. You want your  doctor doing three and then doing it three again. [11:06] So, here's how it breaks down.  This one slide actually has it all. [11:11] So, if you're going to take a picture of  slide, this is the one to take a picture of. [11:16] Each one of these are the different methods  of validation. Source triangulation. [11:23] Hey, is that reference real? And is what it  referenced in that reference actually there. [11:35] I had a great one where I was doing some work with  this and I it came up with a reference and it was [11:41] a valid reference. I was able to download  that paper and I read that paper and I went [11:46] that's not what the paper said. The citation  verification, again, making sure yeah, did they [11:53] cite it correctly? Maybe they had the right paper,  but they introduced a whole new author into it. [12:03] Rag. [12:05] Retrieval augmented generation. This is  something that's newer in the models. [12:11] This is a really important aspect. Go look at  the manual. What does the manual actually say? [12:21] Can we actually tie that usage of the  registers to the register specifications and [12:31] and conventions document. Structural  knowledge and base grounding. [12:41] Is this actually syntactically correct? [12:47] Does it meet the actual needs? [12:52] The second layer here, the model behavior. This  is kind of the fun one. I I have a lot of fun [12:57] with this cuz it's just sometimes really fun  to take and div the responses to the opposite [13:04] models and see what they think  of the other models response [13:08] and then you give those  responses back to the first model [13:12] and it says, "Oh, yeah, chat GPT did a good job  of finding this, but you know, they missed that." [13:19] It's It's like you have two kids. You know,  who who who's, you know, outfit looks better. [13:31] There's really a couple different  models here, self-consistency. [13:36] If I ask it the same question, does do I get  the same results? Do I If I ask it slightly [13:41] differently, do I get the same results? The cross  model, cross crisscrossing. And then the hard one, [13:48] confidence and uncertainty quantification.  You usually can't get that, but if you can [13:55] In other words, is the response it's giving  you a 99% certainty or a 51% certainty? [14:05] You know, these models have  underneath that statistical value. [14:11] If you can expose that, that's a great thing.  If it says 99, okay. If it says 51, hmm. [14:21] Claim of logic validation. [14:26] Atomic claim validation. This  is something that really is [14:33] It sounds common sense, but we don't do it. [14:39] Take the problem, take the individual  components of what you asked, [14:44] and is each one of them independently correct? [14:50] If I take a bunch of things in aggregate, it  may look correct at the end, but if one of those [14:57] things in the middle aren't correct, that result  probably is not right. It may work. It may pass, [15:09] but is it truly right? So, breaking it down,  looking at the individual claims being made. [15:17] Reasoning and consistency checks. [15:21] Does it give you the same response  throughout what you asked it to do? [15:30] Or does it sometimes say, "Oh, yeah, here you want  to do this, here you want to do that. Wait wait, [15:33] you just contradicted yourself." When you're doing  a large like development model or preparing large [15:40] documents, research work, and you're asking  AI question after question after question, [15:47] does it maintain consistency throughout that?  Rule constraints. This is that hey, you used the [15:55] wrong registers. Did it abide by the rules that we  need to use for this context, for this response? [16:10] Did it actually meet the requirements that we set? [16:17] Is it a secure response? If we're doing something  like an SQL query, it was one thing to be [16:26] all the way through the business logic correct,  it's another thing if it can be easily injected. [16:33] If your SQL could be easily injected,  that's not any SQL you want to use. [16:42] And automate automated fact-checking pipelines. [16:48] How many here familiar with regression testing? [16:51] Okay? I've got a new one for  you. Regression validation. [16:59] Have a set of things that you say, "This has to  meet this criteria, this criteria, this criteria, [17:06] this criteria." And as you're generating more  and more in this project, every time through, [17:13] go back through that validation.  Are all of terms consistent? [17:20] Are all of the labels you're using consistent?  Are the references to these documents consistent? [17:30] So just like we do regression testing,  now we need to do regression validation. [17:39] And the last one, this is probably the most  important one. The operational safeguards. Why? [17:49] Look at the very first one there. Human in the  loop. Nothing. Nothing substitutes for our eyes, [18:01] for our interpretations, for our ability to  read and analyze this. It's our responsibility [18:12] of what that result is going to do.  If we're going to use that result, [18:17] we can't say, "Oh, it failed.  Oh, that's the AI's fault." [18:21] How many bosses will take that as an answer? No.  It's your fault. So it's your responsibility. [18:31] We have to be responsible and understand these  things. Adversarial red teaming. In other words, [18:38] really trying to break that response. How does  that do? And validity uh temporal validity checks. [18:50] What I was talking about with the  sense of is this up-to-date data? [18:55] Now, I'm going to flip through the next set of  slides here. They break down each one of these. [19:02] The source triangulation. What does it do? [19:05] How would somebody do it? What it's kind  of best for. So I'm just going to I'm not [19:09] going to really talk about these. I'll just  flip through them cuz I want to get through [19:14] to kind of the last slide. We've  talked about each one of these. [19:20] And part of this, I'm sure the decks  are going to be available, right? [19:24] Yeah, so the decks will be available. This is  also a reference deck. This is something you [19:30] can bring to your company and say, "Hey, yeah,  here's what to do. Here's the different methods. [19:37] Here's how we can do each one of  these. How we can get through. [19:47] What do each one of these mean? Yeah, the  atomic claim. I This is one I use a lot. [19:56] I've been doing a lot of work with AI  recently and I've been following this model. [20:04] Because this model's integral in  this new pedagogy I'm working on [20:08] and I've been using the faults that it finds  as an audit trail to show why to use this. [20:20] Human in the loop, never forget that one. [20:25] Your red teaming and finally  the temporal validity checks. [20:33] We're really creating a pipeline here. [20:38] We're saying that this is no longer just  a one-off. Oh, validating this is just a [20:42] one-off thing. No, we want to integrate this. If  we think about how we've talked about the change [20:50] in industries, in programming, junior programmers,  those jobs are going away, right? AI can write it. [20:58] Can it? Who's going to show  if that code's correct? [21:03] We're changing our thought that writing code is  no longer the primary goal but reading the code [21:16] validating the code that is where we're actually  moving to. Can you read it? Can you validate it? [21:25] It's not a step, it's a discipline. We  have to integrate this into our workflow. [21:33] Instead of spending all that time and drudgery  doing this coding, writing a Python script or [21:39] whatever else we're doing, we have to put that  time into how we validate if it's correct. [21:47] And one other very important  point a validated result is great [21:55] but it does not equate to understanding. [22:00] I can have something that works 100%.  Is that something I should just use? [22:08] We're responsible for it. Not only is this  validation framework important but we also [22:17] must understand the output and what we're  using. We need to understand the result. [22:24] So if you're a programmer and it generates  code and the code is 100% perfect [22:30] but if I ask you what's this  code do and you have no idea [22:34] that's not good. You need to read it. You need to  understand it because without that understanding [22:41] application can now falter. Thank you.