Meibel Warns AI Agents Fail in Production on Unstructured Data

Scaling AI with Context, Control and Confidence With Meibel — Ai4

Ai4

Enterprise AI deployments break down because standard frameworks rely on pristine inputs and model self-policing rather than upstream context preparation and deterministic controls to curb compounding multi-turn errors.

AI-assisted summary of Scaling AI with Context, Control and Confidence With Meibel — original source. Check the original for full context.

Source published: 2026-08-19

Detailed notes

## Background Published on YouTube by the Ai4 channel on August 19, 2026, this presentation features Meibel exploring the failure modes that emerge when transitioning artificial intelligence from prototype demos into production. The session description emphasizes that changing data and evolving workflows expose distinct engineering challenges, presenting a practical framework for maintaining context, control, and confidence across production systems.

## TL;DR The talk demonstrates that enterprise AI deployments frequently stall because agent frameworks falsely assume enterprise data is clean and structured, trapping teams in low-level parsing issues rather than solving core business throughput bottlenecks. Through case studies of SpecBooks processing complex construction blueprints and Toffler Associates productizing decades of strategic foresight IP, the speaker illustrates how unstructured PDFs, prompt bloat, and context degradation undermine reliability. Solving these production breakdowns requires multi-level retrieval using Vision-Language Models, strict end-to-end provenance tracking, and deterministic upstream policies rather than relying on LLMs to self-govern. Ultimately, engineering leaders must actively measure full-pipeline data quality to counteract the exponential reliability decay caused by compounding multi-turn agent errors.

## Atmosphere & style The presentation maintains a pragmatic, analytical, and sobering tone that cuts directly through standard generative AI hype with engineering realism. Delivered in a steady, consultative style, it methodically unpacks the unglamorous friction points of production deployment, from mangled PDF layouts to rule-following degradation across multi-turn workflows. It feels like an authoritative architectural post-mortem and field guide, designed to validate the shared frustrations of technical practitioners while steering them toward disciplined systems engineering.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=4s) Enterprise leaders do not face an abstract AI adoption problem, but a business throughput bottleneck requiring faster, more reliable processing of transactions and workflows. - [00:04](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=4s) Off-the-shelf agent frameworks often fail in production because they rely on the flawed assumption that enterprise data is already clean, structured, and model-ready. - [03:00](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=180s) In fragmented enterprise verticals like construction, industry coordination barriers make it impossible to mandate structured data, forcing systems to handle messy, unstructured PDFs. - [03:00](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=180s) Building non-differentiating document-parsing infrastructure and wrestling with prompt libraries diverts critical engineering resources away from core product value. - [03:00](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=180s) Enterprise production pipelines require end-to-end provenance tracking so that every extracted data point can be audited against its original source document. - [08:17](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=497s) Extended, multi-turn interactions frequently suffer from context bloat and prompt expansion, causing model rule-following to degrade and analytical reasoning to drift. - [08:17](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=497s) Both deterministic data extraction and nuanced qualitative reasoning fail at the same root causes: deficient context preparation, lack of deterministic control, and unmonitored output drift. - [12:08](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=728s) Expanding LLM context windows does not solve data quality issues, as bloated context often degrades model accuracy compared to concise, highly relevant information. - [12:08](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=728s) Flattening complex PDFs into raw text strips out crucial layout semantics, tabular structures, and embedded visual assets like charts and diagrams. - [12:08](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=728s) High-fidelity document processing demands multi-level retrieval architectures that synchronize vision-language models (VLMs), structured schemas, and multiple data representations. - [17:25](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1045s) Security, data governance, and access control policies must be enforced deterministically upstream before data reaches the model, rather than relying on LLMs to self-police through prompts. - [17:25](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1045s) Multi-turn agent reliability degrades exponentially due to compounding error rates, meaning even high per-step accuracy quickly drops over a sequence of actions. - [17:25](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1045s) True operational confidence requires evaluating data quality, latency, and drift across the entire pipeline rather than merely inspecting the model's final response.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=4s) The Engineering Trap and Real Business Throughput

When organizations set out to deploy artificial intelligence, their core motivation rarely stems from an abstract technical desire. Across daily conversations spanning energy, pharmaceuticals, construction, and financial services, enterprise leaders never report having "an AI problem." Instead, their fundamental bottleneck is almost always a "throughput problem"—a concrete need to process a specific volume of transactions, documents, or operational activities faster and more reliably. Despite this operational imperative, a significant portion of companies find themselves trapped in an engineering spiral, anxiously worried that they are "behind in AI" while failing to tie their initiatives to measurable business outcomes with clear ROI.

Aaron Aguilar, Head of Growth at Meibel (stepping in for CEO Kevin McGrath), emphasizes that this engineering trap is exacerbated by modern agent-building frameworks. Tooling across the ecosystem—from LangGraph to AWS Bedrock—operates under a foundational, often fatal assumption: that enterprise data is already pristine, structured, and ready for model consumption. That single flawed assumption represents the entire chasm between an impressive prototype demo and a reliable production deployment.

``` +--------------------------------------------------------------------+ | THE PRODUCTION CHASM | | | | [ Prototype / Demo ] ---------> [ Production Failure ] | | * Assumes clean data * Bloated context & drift | | * Promising initial vibes * Rule-following breakdown | | * Zero provenance tracking * Unhandled unstructured PDFs | | | | THE MISSING FOUNDATION | | [ Context Preparation ] -> [ Deterministic Control ] -> [ Quality ]| +--------------------------------------------------------------------+ ```

- Organizations winning with AI anchor their deployments in clearly defined, measurable business throughput rather than generic technology adoption. - Off-the-shelf agent frameworks routinely assume structured, clean data inputs, masking the massive preparatory pipeline required for production scale.

[03:00](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=180s) Case Study: SpecBooks and the Reality of Unstructured Documents

SpecBooks is a vertical SaaS platform serving construction contractors and distributors. The company processes architectural and engineering specification plans into structured formats that allow contractors to produce customer bids and proposals, while also running an integrated marketplace for purchasing materials. On a monthly basis, SpecBooks ingests roughly 100,000 massive, multi-page construction spec plans known as "spec books." Early in their AI journey, their engineering team took a forward-thinking, high-velocity approach: combining AWS native services like Amazon Textract with Amazon Bedrock and custom code to parse and extract data rapidly.

While initial proof-of-concepts showed directionally promising results, the platform quickly collided with the messy reality of the construction industry. Construction workflows operate almost entirely on unstructured PDFs. Due to a complex coordination problem involving thousands of fragmented stakeholders, the industry cannot simply be mandated to transition to structured data. When SpecBooks moved deeper into production, the pipeline suffered from excessive variance, unpredictable extractions, and an inability to trace the provenance of extracted figures back to the original source drawings. The team found themselves burning massive engineering hours and token budgets building non-differentiating document-parsing infrastructure—trying to keep elements from flattening and wrestling with expanding prompt libraries—rather than focusing on their core value proposition: helping contractors generate winning proposals faster.

- SpecBooks processes approximately 100,000 dense, unstructured specification books per month. - The engineering team became trapped spending critical resources on low-level document parsing and prompt management instead of core product differentiation. - Provenance was initially missing, making it impossible to audit or verify where extracted figures originated in the source blueprints.

[08:17](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=497s) Case Study: Toffler Associates and Context Window Degradation

Toffler Associates is a specialized advisory firm with more than 50 years of consulting experience, particularly across the federal sector. Recognizing shifts in the professional services market, the firm set out to productize their intellectual property into a strategic foresight and analysis software platform called "Sign." Designed for scenario planning and red-teaming hypothetical future events, the software needed to encode five decades of proprietary frameworks, analytical methodologies, and structured reasoning directly into the AI's persona and workflow.

Toffler experienced an initial trajectory identical to SpecBooks: rapid early success followed by severe degradation in production reality. As users engaged in extended, multi-turn scenario analyses, expanding prompt libraries and bloated context windows caused model rule-following to collapse. The system drifted away from Toffler’s methodology, failing to reason or "talk like a Toffler associate." While SpecBooks required ultra-precise, zero-variance data extraction and Toffler required deeply nuanced, creative strategic foresight, both failures stemmed from identical underlying systemic gaps: poor context preparation, insufficient operational control, and an inability to monitor and measure output drift at scale.

``` +--------------------------------------------------------------------+ | CONTRASTING PRODUCTION REQUIREMENTS | | | | SPECBOOKS (Construction SaaS) TOFFLER ASSOCIATES ("Sign") | | * 100k spec books / month * 50 years advisory IP | | * Deterministic data extraction * Strategic foresight reasoning| | * Zero-variance tolerance * Nuanced brand tone & voice | | * Provenance to source drawing * Red-teaming scenario plan | | | | SHARED ROOT PROBLEM: Context preparation, deterministic control, | | and compounding multi-turn error. | +--------------------------------------------------------------------+ ```

- Toffler Associates built "Sign" to automate strategic foresight, scenario planning, and red-teaming based on 50 years of consulting IP. - Context bloat during multi-turn interactions caused model rule-following to degrade, stripping the output of the firm's signature analytical rigor. - Both case studies demonstrate that whether the goal is deterministic extraction or nuanced qualitative analysis, unstructured context preparation remains the primary point of failure.

[12:08](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=728s) Context: Relevance, Multi-Level Retrieval, and Provenance

"You cannot build a better agent without better context." While context is the bedrock of production AI, simply stuffing larger volumes of information into an LLM context window often worsens performance. What matters is relevance, structure, and completeness. While structured relational databases and text-to-SQL present relatively clear, narrow engineering paths, unstructured documents present severe multi-modal hurdles. Ingesting a PDF by flattening it into a raw "wall of text" strips out critical layout semantics, garbles complex tables, and discards embedded charts, diagrams, and images.

To preserve the fidelity of complex enterprise documentation, systems require specialized middleware capable of multi-level retrieval. Relying on a single technology—whether a basic vector database, citation graph, or standard RAG pipeline—is insufficient. Production architectures require multiple synchronized representations of document data, pairing Vision-Language Models (VLMs) and layered parsing strategies to interpret structural hierarchies alongside embedded visual assets. In schema-mapping workflows, such as standardizing diverse insurance claim documents into a unified output schema, the system must validate the mapped output while preserving end-to-end provenance so every individual field can be audited against its exact source location.

- More raw context degrades model accuracy; high-performing agents require pre-filtered, highly relevant, and structurally preserved data. - Flattening PDFs into walls of text destroys tables, visual diagrams, and hierarchical document metadata. - Multi-level retrieval combines vision models (VLMs), multi-representation indexing, and structural schema validation. - Full provenance tracking is mandatory in enterprise production to verify extracted data points directly against original source documents.

[17:25](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1045s) Control and Confidence: Deterministic Policies and Error Compounding

Once context is properly structured, controlling agent execution requires reproducible assembly, rigorous version control over prompts and data, and deterministic guardrails. Because foundation models are inherently non-deterministic and notoriously unreliable at strict rule-following, security and data governance policies cannot be delegated to the LLM itself via system prompts. If sensitive data is passed into a model's prompt, there is no guarantee it will be withheld from unauthorized end users. Consequently, strict execution policies and data segregation must be enforced deterministically upstream before the data ever reaches the model layer. Additionally, production architectures must incorporate model fallback strategies to mitigate frequent provider downtime and latency spikes.

``` +--------------------------------------------------------------------+ | THE COMPOUNDING ERROR MULTIPLIER | | | | Turn 1 (99% Acc) ──> Turn 2 (99% Acc) ──> Turn 3 (99% Acc) | | │ │ │ | | ▼ ▼ ▼ | | [ 0.99 ] x [ 0.99 ] x [ 0.99 ] | | | | Turn 4 (99% Acc) ──> Turn 5 (99% Acc) ──> TOTAL FIDELITY | | │ │ │ | | ▼ ▼ ▼ | | [ 0.99 ] x [ 0.99 ] = ~95.1% | | | | Single-step confidence degrades exponentially across agent turns! | +--------------------------------------------------------------------+ ```

Building true confidence in an AI system requires measuring data quality across the entire pipeline rather than merely inspecting final outputs—a mistake equivalent to "putting the cart before the horse." In complex multi-turn agent systems, errors compound rapidly. Even if an agent maintains a 99% success rate on each discrete action, chaining five consecutive actions drops the overall sequence reliability below 95.1%. At enterprise scale, these small error rates compound into severe operational failures. Whether a platform requires exact, deterministic data points (such as SpecBooks extracting a specific part code like "X2R") or nuanced, brand-aligned qualitative reasoning (such as Toffler Associates), scaling successfully demands precise context preparation, deterministic policy controls, and continuous full-chain measurement.

- LLMs cannot be trusted to self-police access control or data privacy; data segregation rules must execute deterministically prior to prompt injection. - Production systems require robust model fallback mechanisms to withstand LLM availability and uptime volatility. - Agent reliability degrades exponentially across multi-step execution paths due to compounding per-turn error rates. - Meibel provides an end-to-end agent building platform designed to prepare unstructured context, enforce deterministic execution controls, and monitor full-chain confidence at scale.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=4s) an AI problem — Aaron Aguilar

> [00:04](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=4s) throughput problem — Aaron Aguilar

> [00:04](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=4s) behind in AI — Aaron Aguilar

> [03:00](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=180s) spec books — Aaron Aguilar

> [08:17](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=497s) Sign — Aaron Aguilar

> [08:17](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=497s) talk like a Toffler associate — Aaron Aguilar

> [12:08](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=728s) You cannot build a better agent without better context. — Aaron Aguilar

> [12:08](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=728s) wall of text — Aaron Aguilar

> [17:25](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1045s) putting the cart before the horse — Aaron Aguilar

## Mentioned Aaron Aguilar, Kevin McGrath, Meibel, Ai4, LangGraph, Amazon Bedrock, Amazon Textract, SpecBooks, Toffler Associates, Sign

## Video description Kevin McGrath - CEO, Meibel

Putting AI into production exposes a different class of problems than most teams expect. Changing data, evolving workflows, and expanding use cases introduce failure modes that don’t appear in demos. This talk explores where production AI breaks down and offers a practical framework for preserving context, control, and confidence as systems evolve.

[Source](https://www.youtube.com/watch?v=E-wz1-Tv2Pw)

中文简报

Meibel警告：AI智能体在生产环境中常因非结构化数据而失效企业级AI部署之所以崩溃，是因为标准框架依赖于完美无瑕的输入和模型自我监督，而非通过上游上下文准备与确定性控制来遏制不断累积的多轮错误。本场演讲表明，企业级AI部署之所以频繁停滞，是因为智能体框架错误地假设企业数据是干净且结构化的，导致团队深陷底层的解析问题，而无法解决核心的业务吞吐量瓶颈。通过SpecBooks处理复杂建筑蓝图以及Toffler Associates将其数十年战略前瞻IP产品化的案例研究，演讲者阐释了非结构化PDF、提示词膨胀和上下文退化是如何破坏可靠性的。要解决这些生产环境中的故障，需要利用视觉语言模型进行多层级检索、实施严格的端到端溯源追踪，并建立确定性的上游策略，而不是依赖LLM自我管理。最终，工程团队负责人必须积极评估全流程的数据质量，以抵消因多轮智能体错误累积而导致的可靠性指数级衰减。企业领导者面临的并非抽象的AI应用落地问题，而是需要对交易和工作流进行更快、更可靠处理的业务吞吐量瓶颈。 (0:04) 现成的智能体框架在生产环境中往往会失效，因为它们依赖于一个有缺陷的假设，即企业数据已经干净、结构化且可直接供模型使用。 (0:04) 在建筑等碎片化的垂直行业中，行业协同壁垒使得强制推行结构化数据变得不可能，迫使系统必须处理混乱、非结构化的PDF文档。 (3:00) 构建缺乏差异化的文档解析基础设施并与提示词库反复纠缠，会分散关键工程资源，偏离核心产品价值。 (3:00) 企业生产流水线需要端到端的溯源追踪，以确保提取出的每一个数据点都能追溯并对照其原始源文档进行审计。 (3:00)

Reading summary

Enterprise AI initiatives frequently stall because standard agent frameworks assume corporate data is already structured and pristine, whereas real organizational bottlenecks center on unstructured document throughput. Aaron Aguilar of Meibel argues that moving from fragile prototypes to dependable production requires pre-model context preparation, deterministic execution controls, and end-to-end quality measurement. The Throughput Problem and Data Assumptions (0:04) Across sectors including energy, pharmaceuticals, construction, and financial services, organizations rarely suffer from an abstract technology deficit. Instead, they face an operational throughput bottleneck—a concrete need to process high volumes of transactions, documents, or workflows with greater speed and reliability. Despite this operational need, enterprise teams often enter an engineering spiral, driven by anxiety over falling behind in AI while failing to tie their projects to measurable business outcomes with clear return on investment. According to Aguilar, off-the-shelf agent-building tooling such as LangGraph and AWS Bedrock introduces a critical vulnerability: these frameworks operate on the assumption that enterprise inputs are already structured, clean, and primed for model consumption. This unaddressed data preparation gap forms the primary barrier preventing promising prototypes from functioning reliably in production. Production Failures in Extraction and Qualitative Reasoning (3:00) Operational breakdowns occur across both deterministic data extraction and nuanced qualitative reasoning. Vertical SaaS provider SpecBooks ingests approximately 100,000 dense, multi-page construction specification books each month. Because fragmented industry workflows prevent mandating structured data, the platform must process messy PDFs. Relying on Amazon Textract, Amazon Bedrock, and custom code, their engineering team was forced to dedicate extensive engineering hours and token budgets to low-level document parsing and prompt management. The system suffered from extraction variance and could not provide provenance tracing figures back to the original architectural drawings. Similarly, consulting firm Toffler Associates attempted to productize 50 years of advisory IP into a scenario-planning and strategic foresight application named "Sign." While early results were promising, extended multi-turn analyses caused context bloat and prompt library expansion, resulting in degraded rule-following and a loss of the firm's structured analytical methodology. Both examples show that whether the task requires zero-variance data extraction or brand-aligned strategic foresight, systems degrade when they lack proper context preparation, operational control, and drift tracking. Multi-Level Retrieval and Provenance Tracking (12:08) Aguilar emphasizes that expanding LLM context windows often reduces performance, as effective context depends on relevance, structure, and completeness rather than raw volume. Ingesting unstructured PDFs by flattening them into continuous text strips out semantic layouts, corrupts complex tables, and discards embedded diagrams and charts. Preserving document fidelity requires multi-level retrieval middleware that combines Vision-Language Models (VLMs) with layered parsing strategies to process structural hierarchies and visual elements in parallel. For complex schema-mapping tasks, such as standardizing diverse insurance claims into uniform schemas, architectures must validate mapped outputs and maintain end-to-end provenance so every individual extracted data point can be audited against its precise location in the original document. Deterministic Governance and Compounding Error Multipliers (17:25) Because foundation models are non-deterministic, enterprise security boundaries and access controls cannot be reliably enforced through system prompts. If sensitive data enters a prompt, the model cannot guarantee it will remain hidden from unauthorized users; consequently, data segregation and execution policies must be enforced deterministically upstream before information reaches the model layer. Production environments also require automated model fallbacks to navigate infrastructure downtime and latency spikes. Furthermore, multi-turn agent systems experience rapid error compounding. Even if an agent achieves a 99% success rate on each independent action, chaining five consecutive actions reduces overall workflow reliability to approximately 95.1% (0.99^5). In enterprise deployments, small per-turn failure rates quickly lead to operational breakdown, necessitating continuous quality measurement across the entire data preparation and execution pipeline rather than inspecting only the final output. Successfully deploying production AI requires resolving the messy reality of unstructured source data through deterministic upstream controls, multi-level visual and structural retrieval, and continuous full-chain pipeline monitoring rather than relying on prompt instructions or model self-policing.

精炼阅读摘要

Aaron Aguilar（Meibel增长主管）指出，企业部署人工智能的核心瓶颈是实际的业务吞吐量问题而非抽象技术诉求；通用智能体框架对纯净结构化数据的脱敏假设，导致原型展示与高可靠性生产环境之间存在巨大断层。业务吞吐量与框架数据假设的断层 (0:04) 企业在能源、医疗、建筑及金融等行业推进AI落地的核心驱动力，通常是为了更快速、更可靠地处理特定体量的交易、合同或日常业务，本质上属于吞吐量瓶颈。然而，许多企业往往陷入盲目的工程焦虑，担心自己在AI赛道掉队，却未能将技术方案与可衡量的投资回报率挂钩。 Aguilar强调，这种工程困境被当前的智能体构建工具（如LangGraph与AWS Bedrock）进一步放大。现有工具普遍预设企业输入的数据已经是清洗完毕、结构整齐且适宜模型读取的状态。这一关键假设忽略了真实生产环境中极度庞杂的数据前置处理流水线，导致大量在演示阶段表现亮眼的原型系统在正式上线时崩溃。落地困境：精准提取与战略推理的失效 (3:00) 建筑垂直SaaS平台SpecBooks每月需解析约10万份高度复杂的多页工程规范书，由于整个建筑行业高度依赖非结构化PDF，无法强制推行数据结构化。其团队初期结合AWS Textract与Bedrock搭建提取方案，但随后在生产中遭遇了数据波动过大、提取结果不可预测以及无法溯源原始图纸等问题，被迫消耗大量工程资源与Token预算去维护低效的底层解析和提示词库。类似的问题同样出现在顾问咨询公司Toffler Associates身上。该公司试图将50年的战略预见方法论产品化为名为“Sign”的推演软件，在多轮推演互动中，上下文窗口迅速膨胀导致模型遵守既定规则的能力彻底退化，丢失了标志性的严谨分析口吻。这两起案例表明，无论是严苛的零容差数据提取还是微妙的战略推演，缺乏充分的上下文准备与漂移监控都是模型失效的共通诱因。上下文保真机制：多层检索与全链路溯源 (12:08) “没有更优质的上下文，就无法构建出更优秀的智能体。”生产环境中若盲目向大语言模型上下文窗口堆砌海量信息，往往会进一步恶化输出质量。将非结构化PDF简单拍平为纯文本会彻底抹除文档的排版语义、打乱复杂表格并丢失图表与图像信息。为了保持企业复杂文档的真实保真度，生产架构需要依赖支持多层检索的专用中间件，结合视觉语言模型（VLM）与分层解析技术，同步保留层级结构与视觉要素。例如在保险理赔文档的标准模式映射中，系统在输出规整数据的同时，必须建立端到端的溯源机制，确保提取出的每一个字段都能精准追溯到源文档的原始位置以便审计。确定性上游控制与多步执行的误差复合 (17:25) 基础模型天然具备非确定性，且难以完全信赖其对系统提示词的严格遵循。因此，企业的数据治理与安全访问权限绝不能交给模型通过提示词进行“自我监督”；一旦敏感数据传入模型，便无法保证其不会泄露给未授权用户，必须在数据进入模型层之前通过确定性逻辑在上游强制执行隔离策略。此外，生产架构必须配备模型回退机制，以抵御模型供应商偶发的停机或延迟激增。在构建系统置信度方面，必须在整个处理流水线全程监测数据质量，而非仅仅检验最终输出。复杂多步智能体存在严重的复合误差现象：即便单步操作准确率达到99%，在连续执行5步后整体成功率也会降至约95.1%。无论业务目标是精准提取特定零件代码还是维持品牌调性的定性推理，规模化部署均依赖精准的上下文准备、确定性规则控制与全程质量监控。 Aguilar的主张表明，企业级智能体要跨越演示到上线的鸿沟，重心必须从后端的提示词调整转向前置的非结构化上下文工程与确定性安全治理；但对于高度非结构化的多方协作行业，如何在不重构原有底层文件生态的前提下彻底降低数据管道的维护成本，仍是各架构演化中需要持续权衡的挑战。

Detailed notes — zh

## Background 该内容由 Ai4 于 2026 年 8 月 19 日发布在 YouTube 平台上。视频简介指出，将 AI 投入生产环境会暴露出一系列未在演示 Demo 中出现的新型失效模式，本次演讲深入探讨了生产级 AI 发生崩溃的核心原因，并提出了一套在系统演进中维护上下文（Context）、控制（Control）与信心（Confidence）的实用框架。 ## TL;DR 演讲指出企业部署 AI 的核心驱动力是解决业务吞吐量瓶颈，但大多陷于非结构化数据难以治理的“工程陷阱”。通过建筑垂直 SaaS SpecBooks 与战略咨询机构 Toffler Associates 的真实案例，分析了文本展平丢失溯源与多轮长上下文导致规则崩溃等生产级问题。演讲提出了三大应对支柱：依托多层级检索与 VLM 还原文档结构以优化 Context，在模型外部实施确定性权限隔离以强化 Control，以及全链路监控多步复合错误率以建立 Confidence。 ## Atmosphere & style 整体演讲风格极其务实且工程导向，摒弃了空洞的行业炒作，直击生产级 AI 落地时的残酷现实。叙述节奏沉稳紧凑、逻辑严密，讲者以冷静客观的视角剖析从 Demo 的惊艳到进入复杂生产环境后遭遇各种瓶颈的落差。听感上充满实战踩坑经验的共鸣，并以严密的工程架构逻辑为开发者与业务决策者带来了化解不确定性的踏实感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=4s) 企业部署 AI 的核心驱动力是解决业务吞吐量（Throughput）瓶颈，而非单纯的技术追求；缺乏可量化 ROI 的项目极易陷入“工程陷阱”。 - [00:04](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=4s) 市面上主流 Agent 平台往往默认数据已就绪，但现实中非结构化数据的治理与上下文输入才是从 Demo 走向生产级系统的最大卡点。 - [03:00](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=180s) SpecBooks 案例表明，简单使用文本展平与基础 API 解析非结构化 PDF 会导致表格/图像结构丢失、无法溯源及准确率严重漂移。 - [08:17](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=497s) Toffler Associates 案例表明，多轮长对话会导致 Context 膨胀与模型规则遵循能力崩溃，使资深方法论无法稳定产品化输出。 - [12:08](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=728s) 单纯堆砌数据不等于提供高质量上下文，系统需要构建中间件并结合视觉语言模型（VLM）等多层级检索技术还原版面结构与元数据血缘。 - [17:25](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1045s) 在生产环境中，系统必须通过多模型回退策略、版本追踪和可量化的错误容忍阈值来对非确定性 LLM 施加确定性控制。 - [17:25](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1045s) 数据安全与访问权限必须在数据进入 Context 之前通过系统代码强制隔离，绝不能依赖 Prompt 指令来做安全防护。 - [20:05](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1205s) 在多步骤 Agent 工作流中，微小的单步错误会因复合效应被急剧放大，因此必须对流水线的每一步输入与输出建立端到端的置信度度量。 - [20:05](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1205s) 无论是追求低方差的精确提取还是高灵活性的创造性推演，规模化落地的核心都在于上下文准备（Context）、确定性控制（Control）与全链路置信度度量（Confidence）。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=4s) 企业落地 AI 的初衷与陷入的“工程陷阱” 企业究竟为什么要去部署 AI？Meibel 的增长负责人 Aaron Aguilar 指出，每天与他们交流的企业没有一家会说“我们遇到了 AI 问题”，他们普遍面临的其实是**吞吐量问题（Throughput problem）**——即需要处理更多数量的特定业务对象（X），或者完成更多繁琐的业务活动。然而，当前大量企业在落地 AI 时陷入了“工程陷阱（Engineering trap）”。许多团队仅仅因为觉得“自己在 AI 领域落后了”而盲目跟风，却未能找准真正能为业务带来可衡量投资回报率（ROI）的具体商业痛点。如果无法明确具有量化结果的业务场景，这些企业往往无法成为当今 AI 浪潮中的赢家。作为 Meibel 的增长负责人，Aaron 代替未能到场的 CEO Kevin McGrath 进行分享。他每天都在与能源、制药、建筑、金融服务等各行各业的企业沟通，发现这些企业在构建 AI 智能体时面临着极其一致的共同挑战：**如何将正确的业务上下文（Context）输入给模型，并在工作流或项目输出中精确呈现**。像 LangGraph、Amazon Bedrock 等现有的 Agent 构建平台，往往都预设了一个前提——“你的数据已经被清洗干净且已为 AI 做好了准备”。然而，正是这个不切实际的假设，构成了“好看的演示 Demo”与“真正可扩展的生产级系统”之间的巨大鸿沟。 - 企业部署 AI 的核心驱动力是解决吞吐量瓶颈，而非单纯的技术追求。 - 缺乏明确商业目标与可量化指标的 AI 项目极易沦为工程内耗。 - 市面主流平台默认数据已就绪，但现实中非结构化数据的治理才是 Demo 走向生产的最大卡点。 --- ### [03:00](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=180s) 案例一：SpecBooks 在建筑领域的文档上下文泥潭建筑行业是一个严重依赖非结构化文档的领域。软件服务商 SpecBooks 为建筑垂直领域的承包商和分销商提供服务，核心业务包括解析规格计划（Spec plans）、生成商务提案以及运营产品交易市场。他们每月需要处理大约 100,000 份极其厚重的“规格书（Specbooks）”。在初期探索中，SpecBooks 采用了非常典型的技术栈：AWS 原生 Textract、Amazon Bedrock 以及 Claude 代码生成工具。一开始项目进展迅速，初步的 Demo 表现也符合预期且前景看似十分光明。然而，一旦进入真实业务细节，问题便集中爆发。建筑行业涉及多方协作，几乎不可能强制全行业推行统一的数据结构化标准，海量的 PDF 成为唯一的数据载体。SpecBooks 很快遇到了严重的方差问题，输出结果经常出现无法向原始文档溯源（Provenance）的偏差，无法在大规模运行时保障并衡量准确率。他们陷入了维护日益庞大的 Prompt 库、应对各种复杂文档 Schema 的 engineering trap 中。团队耗费了巨额的工程人力和 Token 成本去研究“如何更好地解析文档、如何防止表格和元素被压平（Flattened）”，但这部分工作并不是他们的核心业务壁垒。对于 SpecBooks 而言，真正的商业价值在于让客户更快生成提案（解决吞吐量问题），而在底层数据准备上的过度消耗极大地拖慢了核心产品的迭代步伐。 ```mermaid flowchart LR accTitle: SpecBooks 早期架构问题 subgraph S1["输入层"] A["厚重非结构化 PDF<br/>(月均10万份)"] end subgraph S2["早期朴素方案"] B["Textract 简单解析"] C["单层文本展平 (Flattening)"] D["Bedrock / Claude API"] end subgraph S3["生产端瓶颈"] E["表格/图像结构丢失"] F["缺乏溯源能力 (Provenance)"] G["高方差 & 准确率漂移"] end A --> B B --> C C --> D D --> E D --> F D --> G ``` --- ### [08:17](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=497s) 案例二：Toffler Associates 的战略推演与规则崩溃与垂直建筑 SaaS 截然不同，拥有超过 50 年咨询经验的联邦咨询机构 Toffler Associates 则面临着另一种知识转化挑战。为了应对咨询行业的快速变革并保持竞争优势，Toffler 计划将自身沉淀了半个世纪的内部知识、推演方法论和分析框架产品化，打造了一款名为 **Sign** 的战略前瞻与情景分析工具，主要用于红蓝对抗（Red teaming）和假设未来事件模拟。 Toffler 的 AI 落地历程与 SpecBooks 如出一辙：起初充满热情，Demo 效果令人惊艳，但当他们试图将更庞大的方法论上下文输入模型、让团队与客户深入多轮对话时，系统迅速在生产环境中发生恶化。随着 Context 窗口越来越臃肿，模型开始出现“规则遵循能力崩溃（Rule following breakdown）”。AI 不再按照 Toffler 资深顾问的逻辑和行话进行专业分析，分析质量出现不可控的降级，完全无法直接交付给终端客户使用。这两个看似截然不同的案例证明：无论是追求极低方差的精确数据提取（如 SpecBooks），还是追求具有特定思维逻辑的创造性输出（如 Toffler），其本质都在于如何准备上下文、如何对 AI 进行确定性控制，以及如何在大规模运行中监控输出质量。 - **业务目标**：将 50 年咨询智库方法论提炼为战略前瞻产品 Sign。 - **失效模式**：多轮长对话导致 Context 膨胀，模型遗忘系统指令与推理规则。 - **核心痛点**：输出内容偏离顾问风格，无法达到企业级客户交付标准。 --- ### [12:08](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=728s) Context 优化的核心：多层级检索与文档结构还原 “如果不改善 Context，你就绝不可能构建出更好的 Agent。”Aaron 强调，更多的数据并不等同于更好的 Context；盲目堆砌 Context 往往导致更差的模型表现，关键在于**相关性（Relevance）与完整性（Completeness）**。相较于路径相对清晰的 SQL 关系型数据，非结构化文档的治理要困难得多。如果只是简单地将 PDF 文档压平为纯文本墙（Wall of text）直接喂给 LLM，文档中的表格、图表、图形等视觉与层级信息都会严重丢失或变形。要解决非结构化文档的规模化理解问题，企业必须构建一层专门的中间件（Middleware），依靠**多层级检索（Multi-level retrieval）**架构而非单一技术（如单纯的向量数据库或基础 RAG）。系统需要先利用解析器与视觉语言模型（VLM）等分层策略，理解文档的版面结构，提取嵌套在图像和图表内部的数据，并建立可追溯至原始文档的元数据血缘（Provenance）。以保险单据为例，面对格式各异的原始扫描件或电子档，系统必须能精准识别文档结构，将其映射到统一的标准 Schema 中，并完成严格的输出校验。 ```mermaid flowchart TD accTitle: 多层级上下文准备架构 subgraph Input["原始非结构化资产"] Doc1["扫描件 / 图像"] Doc2["复杂排版 PDF"] Doc3["包含图表的报告"] end subgraph Middleware["上下文准备中间件 (Meibel)"] Parser["VLM 视觉与版面结构解析"] MultiRep["多重数据表征提取<br/>(表格/文本/图像)"] Prov["数据源溯源锚定 (Provenance)"] end subgraph OutputTarget["结构化投递"] Schema["统一目标 Schema 映射"] AgentCore["Agent 运行时决策环境"] end Input --> Parser Parser --> MultiRep MultiRep --> Prov Prov --> Schema Schema --> AgentCore ``` - **文本展平的弊端**：直接转文本会彻底破坏表格的行列关系和图表的语义。 - **多技术融合**：结合视觉模型（VLM）、多层级向量表征与图谱，实现多维度解析。 - **全链路溯源**：在生产环境中，每项提取出的指标必须能够精确定位到原始文档的特定页码与坐标区域。 --- ### [17:25](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1045s) Control：确定性策略与非确定性模型的工程隔离在完成文档上下文的理解后，构建 Agent 的核心挑战转向了**控制（Control）**。在非确定性（Non-deterministic）的 LLM 世界中，系统必须具备确定性且可复现的管理机制。这包括对 Prompt 版本、数据流水线版本的严格追踪，以及明确设定业务对错误率的容忍阈值。一旦错误率超出预设容忍度，系统必须能主动告警并执行预设的降级与补救措施；同时，鉴于目前基础大模型可用性（Availability）和稳定性的波动，多模型无缝回退（Fallback）机制也是生产环境的必备底座。更关键的一点在于**执行策略（Execution policies）**的物理隔离。业界必须认清一个事实：LLM 天生不擅长稳定遵守规则，如果将敏感数据直接放在 Context 中却指望 Prompt 指令让模型“不准透露”，这在安全和隐私要求极高的企业级场景下完全不可行。因此，数据访问权限的控制必须在**数据输入模型之前**由确定性的系统逻辑强制执行。只有目标用户被授权查看的数据才能被组装进 Context，杜绝依赖 AI 自身的“自觉性”来做安全防护。 - **版本与容错控制**：对提示词和数据流实施版本管理，设定量化错误容忍度与告警机制。 - **模型高可用策略**：支持跨模型 Provider 的自动降级与负载回退。 - **前置安全隔离**：在模型推理外部通过代码强制实施权限管控，禁止依赖 Prompt 做数据屏蔽。 --- ### [20:05](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1205s) Confidence：复合错误率监控与 Meibel 的平台定位 AI 系统落地的最后一个支柱是**信心（Confidence）**，即对全链路输入与输出的端到端可度量性。很多团队往往把注意力全部放在最终输出的质量上，这种做法属于“本末倒置（Putting the cart before the horse）”。如果无法量化每一阶段输入数据的纯净度，就根本无法确保输出的可靠性。特别是在多步执行的 Agent 工作流中，**复合错误率（Compounding error）**具有毁灭性的放大效应：假设 Agent 每一步动作的准确率都是 99%，在经过 5 轮交互推演后，整体准确率就已经大幅跌落，不再是 99%。针对不同的业务需求，系统对信心的衡量标准截然不同：SpecBooks 需要的是绝对的准确性与极低方差，例如提取出的“X2R”型号参数必须在输出中原样再现；而 Toffler 需要的是深度符合其咨询逻辑与思考风格的创造性输出。尽管业务形态两极分化，但背后的底层解法完全一致：**正确准备上下文（Context）、在大规模运行时施加确定性控制（Control）、并对流水线的每一个环节建立置信度度量（Confidence）**。Aaron 最后总结道，Meibel 正是一个为此而生的 Agent 构建平台，旨在为企业提供从数据上下文准备、规模化运行到全流程可控评估的完整基础设施。 | 维度 | SpecBooks（建筑垂直 SaaS） | Toffler Associates（战略咨询产品 Sign） | |:--- |:--- |:--- | | **业务核心诉求** | 提升每月 10 万份规格书的提案生成吞吐量 | 将 50 年推演方法论产品化，支持战略前瞻与红蓝对抗 | | **输出特性定义** | 绝对精确、极低方差（如特定零件编码精准提取） | 具备创造性且严格符合内部思维框架的分析报告 | | **早期失效原因** | 缺乏溯源能力，复杂表格/图像数据在展平时丢失 | 多轮推演导致 Context 膨胀，模型规则遵循能力崩溃 | | **统一解决路径** | 依托 Meibel 进行多层级文档结构还原与精准映射 | 依托 Meibel 确定性执行策略隔离与多步置信度监控 | ## Notable quotes > [00:04](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=4s) 企业没有一家会说“我们遇到了 AI 问题”，他们普遍面临的其实是吞吐量问题 — Aaron Aguilar > [00:04](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=4s) 如何将正确的业务上下文（Context）输入给模型，并在工作流或项目输出中精确呈现 — Aaron Aguilar > [00:04](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=4s) 你的数据已经被清洗干净且已为 AI 做好了准备 — Aaron Aguilar > [03:00](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=180s) 团队耗费了巨额的工程人力和 Token 成本去研究“如何更好地解析文档、如何防止表格和元素被压平”，但这部分工作并不是他们的核心业务壁垒 — Aaron Aguilar > [08:17](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=497s) 随着 Context 窗口越来越臃肿，模型开始出现“规则遵循能力崩溃（Rule following breakdown）” — Aaron Aguilar > [12:08](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=728s) 如果不改善 Context，你就绝不可能构建出更好的 Agent — Aaron Aguilar > [12:08](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=728s) 更多的数据并不等同于更好的 Context；盲目堆砌 Context 往往导致更差的模型表现，关键在于相关性（Relevance）与完整性（Completeness） — Aaron Aguilar > [17:25](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1045s) LLM 天生不擅长稳定遵守规则，如果将敏感数据直接放在 Context 中却指望 Prompt 指令让模型“不准透露”，这在安全和隐私要求极高的企业级场景下完全不可行 — Aaron Aguilar > [20:05](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1205s) 很多团队往往把注意力全部放在最终输出的质量上，这种做法属于“本末倒置（Putting the cart before the horse）” — Aaron Aguilar > [20:05](https://www.youtube.com/watch?v=E-wz1-Tv2Pw&t=1205s) 复合错误率（Compounding error）具有毁灭性的放大效应：假设 Agent 每一步动作的准确率都是 99%，在经过 5 轮交互推演后，整体准确率就已经大幅跌落 — Aaron Aguilar ## Mentioned Kevin McGrath, Aaron Aguilar, Meibel, LangGraph, Amazon Bedrock, SpecBooks, AWS Textract, Claude, Toffler Associates, Sign ## Video description Kevin McGrath - CEO, Meibel Putting AI into production exposes a different class of problems than most teams expect. Changing data, evolving workflows, and expanding use cases introduce failure modes that don’t appear in demos. This talk explores where production AI breaks down and offers a practical framework for preserving context, control, and confidence as systems evolve. [Source](https://www.youtube.com/watch?v=E-wz1-Tv2Pw)

Source quotes

[0:04] Aaron Aguilar: 企业没有一家会说“我们遇到了 AI 问题”，他们普遍面临的其实是吞吐量问题 [0:04] Aaron Aguilar: 如何将正确的业务上下文（Context）输入给模型，并在工作流或项目输出中精确呈现 [0:04] Aaron Aguilar: 你的数据已经被清洗干净且已为 AI 做好了准备 [3:00] Aaron Aguilar: 团队耗费了巨额的工程人力和 Token 成本去研究“如何更好地解析文档、如何防止表格和元素被压平”，但这部分工作并不是他们的核心业务壁垒 [8:17] Aaron Aguilar: 随着 Context 窗口越来越臃肿，模型开始出现“规则遵循能力崩溃（Rule following breakdown）” [12:08] Aaron Aguilar: 如果不改善 Context，你就绝不可能构建出更好的 Agent [12:08] Aaron Aguilar: 更多的数据并不等同于更好的 Context；盲目堆砌 Context 往往导致更差的模型表现，关键在于相关性（Relevance）与完整性（Completeness） [17:25] Aaron Aguilar: LLM 天生不擅长稳定遵守规则，如果将敏感数据直接放在 Context 中却指望 Prompt 指令让模型“不准透露”，这在安全和隐私要求极高的企业级场景下完全不可行 [20:05] Aaron Aguilar: 很多团队往往把注意力全部放在最终输出的质量上，这种做法属于“本末倒置（Putting the cart before the horse）” [20:05] Aaron Aguilar: 复合错误率（Compounding error）具有毁灭性的放大效应：假设 Agent 每一步动作的准确率都是 99%，在经过 5 轮交互推演后，整体准确率就已经大幅跌落

Source quotes

[0:04] Aaron Aguilar: an AI problem [0:04] Aaron Aguilar: throughput problem [0:04] Aaron Aguilar: behind in AI [3:00] Aaron Aguilar: spec books [8:17] Aaron Aguilar: Sign [8:17] Aaron Aguilar: talk like a Toffler associate [12:08] Aaron Aguilar: You cannot build a better agent without better context. [12:08] Aaron Aguilar: wall of text [17:25] Aaron Aguilar: putting the cart before the horse

Original transcript

[0:04] Why do companies even go down the path of deploying AI? You know, we have companies uh that [0:10] we speak with on a daily basis that are coming to us and none of them are saying we have an AI [0:16] problem. They're all saying we have a throughput problem. We have a a certain number of X that we [0:24] need to process more of or activities that we need to get done. And what we are seeing today is that [0:34] a lot of companies are getting stuck in an engineering trap. Uh you know they're [0:41] they're looking at uh at AI and saying I'm behind in AI. Um they're trying to understand [0:48] what is it that uh that would really move the needle for them to drive ROI for their business. [0:55] and if they aren't able to identify that business problem uh that has a measurable outcome, [1:03] they're typically not the companies that are winning with AI today. [1:08] So, a bit about myself. I'm Aaron Aguilard. I'm head of growth for Maybel. Our [1:14] CEO Kevin McGrath was scheduled for this session, but uh he wasn't able to be here. [1:19] Um so I'm having these types of conversations with uh with different organizations across [1:26] just about every vertical from energy to uh pharma to construction financial services [1:34] and these conversations uh they all have a common thread. um what we're working through today. Um [1:43] I'm going to share a couple examples from two customers of ours, SpecBooks and Tooffler. [1:49] Two very different businesses, uh each in completely different industries [1:55] with very different problems that they're solving for. uh but there's a commonality across them and [2:00] I would say a commonality across just about every customer that we uh work with and that is really [2:08] around the problem of uh you know how do we get the right context about our business both to the [2:17] model and as an output uh in in whatever workflow or whatever project that they're building. [2:25] Uh there's you know every every platform out there um you know for building agents [2:33] you know from Langraph Bedrock uh you name it uh they they all start with [2:38] a certain assumption that your data is already clean and it's prepared for AI [2:44] and that assumption is the whole distance between having a demo that looks nice um you know has [2:51] impressive results initially uh and something that you can actually scale and deploy in production. [2:57] Now, a bit of a disclaimer. I'm going to spend about 10 minutes here talking [3:02] about uh something that's a little bit deeper than uh the the title of this uh of this talk. So, [3:09] we're talking about scaling AI. Um, so I'm going to spend a little time though on a lab on a layer [3:15] below that talking about documents and why getting the right context to the model is both [3:24] a significant challenge but it's the one worth solving in order to achieve exactly what the topic [3:30] of this uh uh this entire talk is about uh how to scale AI. Uh so I'm going to do that first in the [3:40] context of uh Specbooks. Um Specbooks is a uh SAS product that serves a construction uh vertical. [3:51] They're working with distributors and contractors. They build spec plans. Uh they take they process [3:58] them into um into a format that allows their customers to produce a proposal. They also have [4:07] a marketplace um where people can buy and sell products on their platform and they are processing [4:15] about a 100,000 of these really beefy documents uh called a specbook um on a on a monthly basis. [4:24] uh they are they started early on in their journey with uh the typical approach you know [4:31] AWS uh native uh uh textract uh using bedrock um clog code to to build as many things as they can [4:39] moving really fast very forward thinking uh uh approach um and you know it was working they were [4:47] they were getting they were getting good results and they were seeing something that directionally [4:51] uh looked very promising Um but you know the construction industry it runs on unstructured [4:59] documents. It's a you know it's a a truth of many businesses more than anyone might have uh [5:08] have expected. Um you know starting an AI company never expected to see as many PDFs uh and and [5:16] unstructured documents as I have uh these days but it's it's sort of the lifeblood of most industries [5:22] and especially in construction where there is a bit of a coordination problem. you know, [5:27] even if they wanted to digitize and they wanted to to start working with more structured data, [5:32] uh it would require coordination across so many different stakeholders that no one could actually [5:40] push the industry to to make that change. So as a result they started to see that [5:48] when you start to get into the details despite directionally very promising results uh initially [5:53] they were uh seeing too much variance too many uh things that they couldn't trace back [6:00] why am I getting these types of results uh how how what what even is the right approach to [6:08] uh ensure that we are getting accuracy and are able to measure that at scale. Um they they didn't [6:20] have any way to trace back the providence all the way back to the original document. Uh they also [6:28] had a uh a lot of complexity just around what is the best way to retrieve all of this information [6:37] and how do we manage as we start to iterate through the the growing prompt libraries. [6:44] um the different uh the the different uh schemas that each one of these documents contained [6:53] and all of the different types of data that were embedded within those unstructured documents. [7:01] So you know they despite their initial excitement and uh the the growing complexity that they [7:08] realized they continued on and uh you know started to understand that over time they were spending a [7:15] tremendous amount of effort on engineering things that weren't necessarily differentiators for them. [7:22] the even even the tokens that they were spending were building solutions that were not particularly [7:30] differentiating. They were just trying to solve this engineering trap that they had fallen into. [7:35] How do we parse a document better? How do we make sure elements are not flattened? How do we [7:41] ensure that we are getting preparing the context be uh from our data even before we start to [7:48] retrieve that context and and provide it to the model uh and you know execute on the the [7:54] parts of the application that were actually differentiating for them which were the things [7:58] that would drive uh their customers to be able to produce a proposal faster. At the end of the day, [8:05] anything that didn't lead to their customers being able to get more outputs faster, that [8:11] throughput problem that we were talking about, that that wasn't differentiating for them. Uh [8:17] so I I'll also share with you a a similar example but from a completely different uh perspective. [8:26] Tofler Associates. Uh Toffler Associates was uh they they are in the federal sector. uh they have [8:34] over 50 years of consulting experience. They are um in the in the business of uh providing [8:44] consulting right now. They understand that their business is changing very rapidly and in order to [8:51] remain competitive they also needed to produce a product and that product they called sign. [8:56] It's a strategic foresight and analysis uh uh tool that their customers can use for scenario [9:02] planning um and uh for red teaming uh different uh hypothetical future events. Now they have [9:12] 50 years of internal knowledge uh all different types of methodologies and frameworks [9:17] all of which that they need to to make sure are reflected back in that product [9:21] that they're building. Um their initial uh their initial journey looked very similar to specbooks. [9:28] Again completely different product but ultimately they started with a lot of enthusiasm [9:34] got initial results that looked very promising but with [9:39] the ongoing increase of uh of of context that was pushed to the model. the uh more they they they [9:49] uh saw their customers or their team iterating through uh each of these conversations, [9:56] rule following started to break down uh bloated context windows uh different limitations that that [10:04] uh came up in the actual production reality caused that uh analysis to degrade. it wasn't speaking or [10:12] talking like a Tooffler associate and it wasn't uh it wasn't providing uh anything close to [10:21] the level of quality that they would require to put this in front of a customer. So [10:29] what we're doing here what we're what we're looking at here are two different customers [10:34] uh that that had a very similar problem. They they both needed to be able to define [10:41] what good looks like. And those two definitions were completely different. [10:46] But the answer to solving for each of those uh those outputs and and getting it exact, [10:53] it was the same same problem and the same solution that they needed to drive towards. [10:59] And that was primarily how do we prepare context for the model? How do we ensure that we control [11:08] our AI uh in accordingly so that uh we are calling the right tools so that we are calling the right [11:15] context um so that we are uh ensuring that the right information is even available to the model [11:23] in the first place. Uh and then last but not least how do we measure outputs? How do we ensure that [11:30] once we are getting high quality highfidelity outputs uh that we can monitor that at scale [11:36] because everything has a a certain rate of error. Everything is going to require uh ongoing [11:43] maintenance, ongoing uh analysis of the outputs so that you are sure that things do not drift [11:51] and that when you get an output that is not aligned with your uh with your tolerance [11:56] that you're able to flag it proactively and you're not getting that back from a customer who's saying [12:01] hey you know your product told me X and that's you know totally wrong. [12:08] So you can build a better agent uh you you cannot build a better agent without better context. [12:15] That's if there's one takeaway from this uh from this talk today, it's the [12:22] problems that underly most deployments [12:26] usually come back to a context optimization, improving the way your data is prepared, uh the [12:32] way it is assembled and retrieved. uh and those those are the underlying issues that plague [12:39] most projects that are in that sort of middle space where they haven't figured it out yet. [12:46] So, uh you know, more context is not necessarily the answer despite uh everything that we've [12:53] discussed so far. Uh context is key, but more of it is not necessarily the answer. [13:01] What is really important is the relevance and the completeness. So more context I you know [13:10] despite being able to lead to worse results if the information you're providing to the model is [13:16] true is is actually prepared in a way that the the model can understand and is all relevant [13:22] that often will drive better outcomes. uh we want to we want to talk a little bit about [13:30] the hard part of preparing that context because there there are many different data types [13:35] and for example uh you know relational text SQL uh it it's it's an important problem it's not [13:43] without its challenges but it's a narrower problem and let's say the path is a little bit clearer [13:48] uh when we're talking about unstructured documents uh that's a that's a point where [13:54] we are seeing um that that's the point where we are seeing uh a lot more challenge uh and a lot [14:02] more issues with um how do you prepare that that context? What we're talking about here is uh you [14:09] you you have many different uh uh ways to go about preparing context. Um the [14:16] main thing to understand is that when you flatten a PDF or when you flatten a document and you send [14:21] a wall of text at an LLM, uh that typically uh will result in a lot of uh information being [14:29] missed. Um you can also end up with information being malformed. If you have tables, if you have [14:35] graphs, if you have images in those documents, those things are not necessarily going to [14:40] be reflected back to the to the model properly. So what is required is actually [14:47] a uh a middleware right. So we have plenty of uh things that I'm sure everyone in this room has [14:51] heard of uh you know from vector DBs to citation graphs uh rag gentic rag all these things um but [14:59] what we're talking about uh in order to properly reflect back uh at scale unstructured documents to [15:07] to the model is solving for multi-level retrieval. uh no one technology is is the fix all. You [15:16] actually need to have multiple representations of your data and you need to pre-process and prepare [15:22] them in ways that uh will actually allow you to uh first understand uh the structure of those [15:29] documents and all the information that's contained within that as well as uh understanding the um the [15:35] data that's trapped inside of images within those documents and graphs within those documents etc. [15:41] Um so you also are um going to need to handle structure and providence of uh of those documents. [15:52] Um being able to uh pull structure and being able to trace back all the way to the original source. [16:01] um where did this data come from when it gets into a production environment [16:06] when you start to rely on these outputs is going to be a critical task uh for you. So [16:12] um this example that you see here is just an insurance example. you have a doc where you have [16:18] a standard uh schema um and you need to you need to pull from you know this this schema from from [16:27] uh potentially a variety of different schemas into a standard structured output. Um you know [16:34] in order to do that you need to understand how how the um uh h what's the structure of the document [16:40] in the first place. You need to understand the schema that um that you're mapping it to. [16:45] Um and you need to also be able to validate that the output from that uh is the exact you know [16:53] exactly what you're looking for. It's uh it's a really hard problem. It requires many technologies [17:00] uh you know from parsing VLMs uh layered strategies uh you know there's no one answer uh [17:07] and because these documents come in every shape and size from scanned to images to digitized [17:14] um you you need to have a a a variety of different techniques to approach this. Uh now [17:25] that brings us to to control. Uh control you know once you've handled how do we understand [17:34] the documents now it's time to start building an agent. Uh what is it that [17:39] uh we are actually going to um uh to do? You need to be able to define that and actually control um [17:47] the entire assembly process. you know, keep uh keep track of uh different version and um [17:56] different versions of your um of your prompt of your uh of your data [18:02] um and have a reproducible process. Uh so being able to do that um consistently um and you know [18:10] even though we're in a uh non-deterministic world, you're going to want to have some [18:17] way of scoping out error and understanding what's your tolerance for error as well. Um, so being [18:23] able to say this is this is the rate of tolerate of error that we're able to tolerate and um [18:29] and these are the ways that we are going to flag when uh that uh rate of error is is unacceptable [18:36] and exactly what we're going to do about it. Um you also are going to want to have uh ways [18:42] to fall back to other models. We all know that uh at the moment LLM availability is is not uh [18:50] the greatest and being able to fall back whenever um whenever you need to is is very important. [18:58] Um and last but not least uh execution policies are very very critical in order for [19:05] uh you to be able to ensure that the data that you want to present to the LLM is presented and [19:11] everything else is not. Uh we all know that LLMs are not great at rule following. Uh and and if [19:18] data is presented to an LLM, uh that is not going to be consistently withheld. Uh so you can't rely [19:26] on the AI to to do that enforcement if you have customers that are um relying on you to [19:34] protect their data and only provide the right information to the right user or at the right [19:39] time. um you need to have some deterministic way of uh segregating data that should not be [19:46] seen by the model uh from from what you're trying to pre present to it as well as uh ensuring that [19:54] all the other rules that are very critical to your workflow to your process are being [19:58] enforced consistently and this is something that has to happen even before AI is involved [20:05] and last but not least that brings us to confidence that's the ability to actually measure [20:10] each input as well as output. Uh people are very focused on on the quality of the outputs [20:19] and uh you know looking at uh the outputs is uh sort of putting the cart before the horse if you [20:25] haven't uh measured what are you putting in in the first place. So uh being able to measure each [20:33] stage across the entire uh the entire chain from input to output and understand um you [20:40] each turn that the the agent takes what is the rate of confidence in that action [20:46] is very very important. You know, it's sort of a a classical uh um formula of uh if you [20:53] have 99% uh accuracy on, you know, the first the first action that the agent takes, 99% [21:00] accuracy on the second uh action, third action, fourth action, those uh you know, [21:06] after five turns, you don't have 99% accuracy anymore, right? Those uh rates of error compound. [21:13] So, uh being able to uh understand at at each step how much uh the [21:21] that error might impact downstream actions is is very very important to do especially at scale. [21:29] And so that brings us back to our customers, MacBooks and and uh and Toffler. These again two [21:37] very different uh two two very different customers with a completely different uh definition of what [21:43] good looks like. One of which really wants to have uh accuracy and very low variance in every output. [21:51] if they um extract a specific uh data point, you know, X2R from uh from their specbook, [22:01] it needs to be reproduced exactly that way in the output. Whereas software is looking for [22:08] much more of a um much more creative output that needs to be grounded in their methods, [22:15] in their reasoning, in their in their analysis. um and and and it needs to look and talk like them. [22:22] Uh so that's the uh uh those are two very very different uh things that they're trying [22:28] to accomplish and yet they're able to solve it through the same approach preparing their data uh [22:34] properly and making sure that they are applying the right controls [22:38] consistently in a way that's scalable and then measuring every every piece of the chain. [22:43] Uh so a little bit about Maybel I'll I'll leave you with. We are an agent build platform that [22:50] uh provides exactly that um everything that you need to build, run at scale as well as prepare [22:56] uh all of that context in exactly the ways that we've been discussing here today. [23:01] Um so with that, thank you very much for your time and attention and uh have a great rest of the day.