LlamaIndex Builds Hybrid Document Stack as Context Bottlenecks AI

上下文成AI核心瓶颈，LlamaIndex打造混合文档技术栈 企业自动化受阻并非由于模型推理能力不足，而是传统遗留文档难以被机器识读；这使得专用混合解析器与确定性工作流变得不可或缺，从而规避成本高昂且易生幻觉的前沿多模态API。 Jerry Liu 指出，在超过10万亿页面向人类阅读的原生内容中，提供丰富且相关的文档上下文是企业落地AI的主要瓶颈。他详细介绍了 LlamaIndex 如何从基础 RAG 演进为多层级文档上下文平台，涵盖了专用工具 LlamaParse、基于 Rust 的开源提取器 LightParse 以及 ParseBench 评测套件。演讲展示了混合检索与确定性 DAG 工作流如何解决发票和理赔处理等高风险企业业务流程中的随机性问题。展望未来，Liu 概述了下一代系统的核心要求，重点强调了面向智能体的原生文档格式、24/7 全天候后台任务周期，以及用于智能体持续优化的持久化决策轨迹日志记录。 企业 AI 落地的主要瓶颈已从模型本身的能力转向深层上下文的供给，全球约 10 万亿页散落在 PDF 和 Office 文档中的非结构化数据成为智能体自动化运转的核心资产。 (0:04) 针对 PDF 排版难以机器理解及直接调用通用 VLM 成本极高的痛点，LlamaIndex 推出了极速开源 Rust 提取器 LightParse，并发布了包含 2,000 页真实复杂场景评测基准 ParseBench。 (9:46) 智能体文档搜索已从原始的 top-k 向量召回演进为工具调用驱动的混合检索，通过将 BM25 关键词检索与稠密语义搜索整合至智能体 harness 中实现多步推理。 (16:47) 针对发票审批、理赔和 KYC 等容错率极低的企业任务，采用有向无环图（DAG）定义确定性工作流，可消除大语言模型的随机性并保障全流程审计合规。 (18:10) 文档智能体的未来在于构建人机双向可读写的原生文档格式，并通过持久化存储底层的决策轨迹（Decision Traces），推动智能体迈向 24/7 全天候自主运行。 (19:38)

LlamaIndex co-founder Jerry Liu argues that the primary bottleneck for enterprise AI agents is no longer raw model intelligence, but unlocking the estimated 10 trillion pages of unstructured knowledge trapped across legacy enterprise document formats through a dedicated context infrastructure. Overcoming the Limits of Legacy Document Parsing (5:38) Jerry Liu explains that document ingestion remains fundamentally difficult because formats like PDF were engineered for 2D visual coordinate displays rather than semantic, machine-interpretable text streams. Consequently, traditional heuristic scrapers and shallow machine learning models break down when encountering multi-layer rate sheets, complex balance sheets, or unbordered tables. While modern vision-language models (VLMs) offer zero-shot visual comprehension, relying solely on generic frontier VLM APIs introduces severe financial and operational hurdles. Liu emphasizes that high-volume enterprise pipelines processing between 10 million and 1 billion pages annually face costs reaching up to $100 million at standard 10-cent-per-page API rates. Beyond cost, naive API calls lack visual layout bounding boxes, exhibit uncalibrated confidence scores, and risk hallucinating numbers across dense financial tabular data. Tiered Parsing Architectures and Standardized Evaluation (9:46) To navigate the trade-offs between cost, speed, and accuracy, LlamaIndex advocates a multi-tiered parsing architecture. In this design, a lightweight, Rust-based engine called LightParse executes fast, local first-pass text extraction directly inside active agent loops, escalating only visually complex pages or dense tables to heavy, post-trained layout models like LlamaParse. To establish rigorous evaluation standards in an industry lacking ground-truth metrics, LlamaIndex introduced ParseBench, an open-source benchmark containing 2,000 human-audited pages across financial services, insurance, legal, and government administration. Downstream from OCR, structured extraction harnesses map parsed documents to strict schemas, generating auditable bounding boxes and calibrated confidence scores required for high-stakes enterprise compliance. Hybrid Search and Deterministic Execution Pipelines (16:47) Document retrieval has shifted beyond the naive 2023 Retrieval-Augmented Generation (RAG) paradigm of static text chunking and basic top-k vector similarity. Modern agentic search equips frontier coding harnesses, such as Claude or OpenAI's Codex harness, with dynamic toolsets that combine BM25 keyword matching with dense semantic embeddings across document repositories containing tens of thousands of PDFs. Simultaneously, Liu notes that high-volume enterprise operations like invoice reconciliation, insurance claims processing, and Know Your Customer (KYC) verifications cannot tolerate stochastic model behavior. Rather than relying on unconstrained prompts, enterprises require deterministic Directed Acyclic Graph (DAG) workflows. These step-by-step pipelines provide inspectable control to run mission-critical automations predictably at scale. Agent-Native Document Formats and Continuous Agents (19:38) Looking forward, Liu identifies the document representation layer and agent execution horizons as the next frontiers in agent infrastructure. Because standard Markdown lacks rich visual layout fidelity and HTML presents parsing overhead, Liu forecasts the emergence of hybrid, agent-native document formats engineered for bidirectional reading, editing, and redlining by both humans and autonomous systems. Furthermore, Liu argues that agent harnesses must expand beyond five-minute task sprints toward continuous, 24/7 background execution that reflects sustained human knowledge work. Achieving this shift requires persisting step-by-step 'decision traces' at the data layer, equipping autonomous agents with historical audit trails to self-improve over time and react to enterprise event triggers. Liu concludes that enterprise agent value requires building specialized context infrastructure across parsing, hybrid search, and deterministic execution; however, establishing cost-effective vision OCR at scale, standardized agent-human document formats, and reliable 24/7 autonomous loops remain ongoing technical challenges.

LlamaIndex 创始人 Jerry Liu 在 Ai4 大会上指出，随着前沿模型推理能力与开发工具趋于成熟，释放企业 AI 价值的核心瓶颈在于向模型提供高质量的上下文；构建兼顾解析、提取、检索与工作流的文档上下文层，是实现企业级知识工作自动化的关键基础设施。 海量非结构化文档是企业智能的关键瓶颈 (0:04) Jerry Liu 认为，前沿模型在代码编写与数学推理方面的能力已快速提升，Anthropic 的 Claude Code 与 OpenAI 的 Codex 等框架也使开发者调度架构日益标准化，当前企业价值转化的主要制约因素已变为能否向模型输入丰富且相关的上下文。现实中绝大多数人类知识工作依然沉淀在非结构化文档中，全球约有超过 10 万亿页内容以 PDF、Word、PPTX、Excel 和 HTML 等格式分散在邮件附件、SharePoint、OneDrive 或 Amazon S3 中。无论是自动起草投资备忘录、准备会议简报，还是处理发票核对与客户工单，AI Agent 都必须先深入理解这层庞大的文档上下文。 传统格式解析困境与分层解析架构方案 (5:38) 文档 OCR 技术虽已发展数十年，但 PDF 等传统格式最初是为 2D 图形显示与字形渲染而设计，缺乏原生语义流，导致在处理包含复杂损益表、数万项费率清单或无边框表格时极易失效。虽然通用视觉语言模型（VLM）具备多模态零样本解析能力，但如果直接调用前沿模型 API 逐页处理，不仅推理成本极高（例如每年处理数亿页的场景成本可能高达数千万至上亿美元），还存在幻觉风险且缺少版面边界框（bounding box）与置信度校准。 针对此类工程痛点，LlamaIndex 提出了结合轻量引擎与重型模型的分层解析机制。团队推出了基于 Rust 构建且免费开源的文本提取器 LightParse，通过 CLI、Python、Node 和 MCP 协议直接嵌入 Agent 执行循环，在初步解析时快速提取纯文本；遇到高度密集的图表或复杂版面时，再分流升级至专用的 LlamaParse 引擎进行深层 OCR 与版面还原。为解决行业缺乏语义评估标准的问题，团队建立了包含 2,000 页人工审核文档的基准测试 ParseBench，覆盖金融、法律和保险等严苛领域。 从朴素向量检索演进至工具化智能文档搜索 (16:47) Jerry Liu 回顾指出，2023 年典型的朴素检索增强生成（RAG）大多依赖简单的文本切块、向量存储与 top-k 相似度匹配，但这种粗粒度机制难以应对复杂的长文档。现代 Agent 搜索架构已转变为由编码 Agent 驱动的动态工具调用模式，为前沿模型提供包括 BM25 关键词匹配、密集向量语义检索及细粒度提取在内的多元接口。 LlamaParse 为此构建了针对大规模非结构化文档的索引支持，能够单次吞吐 10,000 份 PDF 并直接输出标准检索端点。通过将版面 OCR 与下游推理框架相结合，该架构能够为保险理赔、复杂商业合同等业务生成带有版面边界框与置信度评分的结构化 JSON 数据，保证了企业在进行人工审计时的可溯源性。 企业落地确定性工作流与长期自主智能体探索 (18:10) 对于发票处理、理赔核算与反洗钱/KYC 审查等业务，企业无法接受大语言模型固有的随机性。为此，LlamaIndex 采用有向无环图（DAG）将任务建模为严格可控的分步管道，支持通过代码或声明式自然语言配置，使任务按照预定逻辑稳定执行。这种方式既保留了模型的理解能力，又消除了未受约束的 Prompt 带来的不可控波动。 面向未来发展，Jerry Liu 提出了两项重要演进方向：一是探索超越纯文本 Markdown 和 HTML 的人机双向可读写、可视化的 Agent 原生文档格式；二是突破当前 5 分钟左右的任务生命周期，构建能够 24 小时后台持续运行的长周期 Agent。实现这一目标需要在数据层持久化记录完整的决策轨迹（decision traces），使智能体能够根据企业事件动态触发，并在持续执行中迭代自我改进。 当前针对企业文档的 Agent 落地仍面临文档格式标准不统一、长期运行稳定性不足以及高并发下推理成本高昂等现实挑战。建立具备自我纠错和全流程追踪能力的底层数据层，将是决定长周期智能体能否真正替代人类承担复杂知识工作的核心问题。

## Background 本视频由 Ai4 频道于 2026 年 8 月 14 日发布在 YouTube 平台上。视频记录了 LlamaIndex 联合创始人兼 CEO Jerry Liu 的主题演讲，探讨了如何克服企业海量非结构化文档处理的瓶颈，通过前沿文档 OCR、Agent 编排以及长周期自动化工作流为 AI Agent 构建核心文档上下文层。 ## TL;DR Jerry Liu 指出当前大模型落地知识工作的核心制约在于缺乏精准完备的企业文档上下文，为此 LlamaIndex 演进出了涵盖多模态解析、语义存储与工作流编排的 Agent 原生基础设施。针对 PDF 底层解析复杂与大模型多模态调用成本高昂的痛点，他介绍了业界评测基准 ParseBench，并提出了结合纯 Rust 轻量解析器 Light Parse 与深度视觉引擎 LlamaParse 的混合帕累托策略。此外，演讲还阐述了从朴素 RAG 到智能体搜索与确定性 DAG 工作流的转变，并展望了未来支持全天候持续运行和决策轨迹回溯的文档 Agent 愿景。 ## Atmosphere & style 整场演讲节奏明快、技术干货密集，洋溢着行业顶级 AI 大会务实且极客的技术交流氛围。Jerry Liu 的表达自信从容且不乏幽默感，面对投屏演示的小插曲也能巧妙化解，全程通过清晰的架构推导和深入浅出的工程拆解牢牢抓住观众注意力。从底层二进制字形解析的痛点剖析到宏观长周期 Agent 的未来构想，演讲给人一种既极其接地气又充满前瞻洞察的沉浸式体验。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=j96gok4nZlg&t=4s) 上下文而非模型智商是当前 AI Agent 落地的核心瓶颈，全球超过 10 万亿页的非结构化文档构成了企业知识工作的基石。 - [02:53](https://www.youtube.com/watch?v=j96gok4nZlg&t=173s) LlamaIndex 确立了面向 Agent 的文档上下文层三大核心支柱：解析与 OCR 层、语义与存储层，以及端到端工作流编排。 - [05:38](https://www.youtube.com/watch?v=j96gok4nZlg&t=338s) 直接单次调用前沿多模态大模型解析海量文档面临成本极其高昂、密集复杂表格易产生幻觉，以及缺乏空间坐标定位与置信度评分三大致命痛点。 - [10:18](https://www.youtube.com/watch?v=j96gok4nZlg&t=618s) LlamaIndex 开源了首个面向复杂生产级文档的评测基准 ParseBench，并强调根据精度、成本与延迟的帕累托曲线制定混合解析策略。 - [14:05](https://www.youtube.com/watch?v=j96gok4nZlg&t=845s) 推出基于 Rust 的开源极速解析器 Light Parse 作为第一道初筛工具，与深度多模态解析 LlamaParse 构成双层混合处理流水线。 - [14:05](https://www.youtube.com/watch?v=j96gok4nZlg&t=845s) Schema 驱动的结构化提取将 OCR 与顶层推理结合，确保抽取的字段具备原始页面边界框锚定（Bounding Box Grounding）与置信度评分。 - [16:47](https://www.youtube.com/watch?v=j96gok4nZlg&t=1007s) 检索技术从写死逻辑的朴素 Top-K RAG 演进为智能体搜索（Agentic Search），将 BM25 与向量搜索封装为工具供前沿模型自主规划调度。 - [18:10](https://www.youtube.com/watch?v=j96gok4nZlg&t=1090s) 针对发票核对、保险理赔与 KYC 等高确定性场景，必须结合 DAG 有向无环图工作流编排以消除随机性并保证全流程可审计。 - [19:38](https://www.youtube.com/watch?v=j96gok4nZlg&t=1178s) 下一代文档智能体将从数分钟的即席交互跃迁至 24/7 长周期持续运行，亟需探索兼顾人机协同的新型文档格式并持久化存储决策轨迹（Decision Traces）。 - [21:31](https://www.youtube.com/watch?v=j96gok4nZlg&t=1291s) LlamaIndex 的核心使命是作为专注文档理解的 AI 模型实验室，为各类前沿智能体解锁高质量、全链路的企业级文档上下文层。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=j96gok4nZlg&t=4s) AI Agent 时代的上下文挑战与文档数据价值 在 AI 4 大会的现场，LlamaIndex 联合创始人兼 CEO Jerry Liu 登台开场。他首先感叹会场规模之庞大，直言这是他在各类会议中心见过“最大的演讲厅之一”。Jerry 迅速切入演讲核心主题——为 AI Agent 构建文档上下文层（Document Context Layer）。他指出，当前大模型与 Agent 的推理、编码和数学能力正在呈指数级飞速进化，像各类前沿模型以及标准化的 Agent 运行框架（如 Claude Code、Codex 等）都在变得极其强大；然而，企业与开发者在实际应用 AI 自动化自身工作流时，面临的最大瓶颈并非模型智商，而是能否为模型提供精准、完备的上下文（Context）。 正如 Jerry 所强调的：“归根结底，如果模型对你的业务一无所知，它们就无法做出任何有价值的事情。”所谓的上下文不仅涵盖企业的操作手册（Runbooks）或联网搜索信息，更关键的是全球范围内庞大未被充分利用的文档数据。现实世界中存在着超过 10 万亿（10 trillion+）页由人类生产的原生内容，广泛分布在 PDF、Word、PowerPoint、Excel 和 HTML 之中。这些文档深锁在文件系统、邮件附件、SharePoint、OneDrive 或 S3 对象存储里，构成了日常知识工作的绝对基石。无论是撰写投资备忘录、准备客户沟通、回复邮件还是处理发票对账，让 AI Agent 具备无缝摄取、解析这些非结构化文档内容的能力，是其真正承担知识工作的必要先决条件。 - 全球有超过 10 万亿页的人类原生文档（PDF、DOCX、PPTX、XLSX、HTML 等）支撑着主要的知识工作。 - 核心论点：“上下文就是一切”（Context is everything），提供高质量上下文是释放大模型潜力的关键瓶颈。 - 典型的文档依赖型知识工作包括：邮件回复、电话会前准备（Call Prep）、投资备忘录生成、发票核对（Invoice Reconciliation）等。 --- ### [02:53](https://www.youtube.com/watch?v=j96gok4nZlg&t=173s) 从 RAG 到 Agent 原生文档上下文基础设施 Jerry 回顾了 LlamaIndex 的发展历程。三年前，LlamaIndex 最早以开源 RAG 框架的形态进入大众视野；而随着技术演进与企业真实需求的浮现，团队将核心使命聚焦并深化为打造业界顶尖的“文档上下文层基础设施”。Jerry 将这套面向 Agent 的文档平台拆解为三大核心支柱： ```mermaid flowchart TD accTitle: LlamaIndex Agent 原生文档上下文层架构 accDescription: 展示从底层解析到语义存储再到端到端工作流的完整文档上下文层 subgraph Parsing["1. 文档解析与 OCR 层 (Parsing & OCR)"] RawDocs["原始文档 (PDF / Word / PPT / Excel)"] --> Engine["多模态解析引擎 / Light Parse"] Engine --> CleanData["结构化 Markdown / JSON / 坐标边界框"] end subgraph Storage["2. 语义与存储层 (Semantic & Storage Layer)"] CleanData --> Retrieval["检索端点 (Grep / 关键词 / 向量嵌入)"] Retrieval --> Collab["版本管理 / 权限控制 / 人机协作"] end subgraph Workflows["3. 端到端工作流 (End-to-end Workflows)"] Collab --> Pipeline["分类 -> 切分 -> 提取 -> 校验 -> 路由"] Pipeline --> AgentApps["下游 AI Agent 知识工作与自动化应用"] end ``` 在功能层面上，这套架构不仅需要提供精确提取 Markdown、JSON 和边界框（Bounding Box）的 OCR 能力，还要提供丰富的检索接口，支持关键字搜索、向量嵌入（Embeddings）和正则抓取（Grep）。此外，随着端到端工作流的复杂化，企业往往不再盲目追求单一顶配前沿模型，而是寻求混合模型策略，在成本与精度的帕累托曲线（Pareto Curve）上寻求最佳平衡点。Jerry 展望未来，文档平台将全面实现“Agent 原生化”：提供兼顾人类与机器双向可读的文档格式，支持细粒度的文档版本控制、在线编辑、权限管控与人机协同。 - **解析层（Parsing Layer）**：负责将最复杂的排版、图表、表格转化为高质量、干净的 Markdown/JSON。 - **语义与存储层（Semantic & Storage Layer）**：提供多模态检索端点、权限管理以及文档版本跟踪。 - **工作流编排（Workflows）**：支持在文档上执行分类（Classify）、切分（Split）、提取（Extract）、验证（Validate）和路由（Route）。 --- ### [05:38](https://www.youtube.com/watch?v=j96gok4nZlg&t=338s) 为什么文档 OCR 与 PDF 解析极其困难？ Jerry 深入剖析了文档解析（Document OCR）背后的技术痛点。尽管 OCR 技术已经发展了二三十年，但将 PDF 转换为结构良好的机器可读文本依然是一个极度棘手的工程难题。PDF 格式在设计之初就不是为了机器阅读而生，其内部存储的本质上是屏幕显示坐标、字形（Glyphs）和二进制图片数据，缺乏结构化的排版语义。当遇到包含密集表格的研究报告或单页拥有上万项条目的费率表（Rate Sheet）时，纯代码规则很难还原其逻辑结构；表格往往仅由分散的线条与孤立文本拼凑而成。 即便是结构化程度稍高一些的 Office 格式（如 Word 的 OOXML、PowerPoint 或 Excel），解析难度依然不容小觑。解析器必须准确计算真实分页、识别由线条手动绘制的伪表格，并精准保留格式标记，如删除线（Strike-throughs）、斜体、红线修订（Redlines）与版本痕迹。在生成式 AI 出现之前，行业依赖基于启发式规则的硬编码模板或针对单一文档类型微调的传统机器学习模型，泛化能力极差。 虽然当前顶尖的前沿多模态大模型（如 Gemini、GPT 系列）具备截图识字能力，但 Jerry 指出直接依赖单次 Prompt 调用多模态 API 面临三大致命缺陷： 1. **成本极其高昂**：若直接向多模态大模型喂入数百万页图片，API 费用将达到天文数字，"会让你直接破产"； 2. **边缘场景与幻觉**：面对极度复杂的密集表格时，纯视觉模型极易漏读或产生幻觉； 3. **缺乏空间接地与置信度校准**：模型无法返回精确的文本空间坐标（Bounding Boxes），无法提供校准的置信度评分来触发人工复核（Human-in-the-loop）。 为了从根本上解决这些问题，LlamaIndex 将自身定位为专注文档理解的“AI Agent 模型实验室”，深入 PDF 底层渲染引擎提取全部原生数据，并结合后训练（Post-trained）的多模态模型与 Agent 推理机制，实现对任意复杂文档的鲁棒解析。 --- ### [10:18](https://www.youtube.com/watch?v=j96gok4nZlg&t=618s) 评测基准 ParseBench 与帕累托权衡 为了衡量文档解析在实际业务场景中的真实语义质量，LlamaIndex 发布了业界首个综合评测基准 **ParseBench**（已在 Hugging Face、Kaggle 和 parsebench.ai 完全开源）。Jerry 指出，学术界以往缺乏针对真实生产级复杂文档的评测标准。ParseBench 收集了 2,000 页经过严格人工核验的高难度页面，覆盖金融服务、保险、政府公文和法律四大重点垂直领域。 | 评测维度 | 具体测试场景与挑战 | |:--- |:--- | | **密集复杂表格** | 资产负债表、复杂嵌套报表、无边框表格的数据对齐与提取 | | **非结构化图表** | 无清晰标签的折线图、柱状图数据点解析 | | **文本样式与修订标记** | 精准捕获红线修订（Redlines）、斜体、删除线、高亮标注等排版元数据 | | **阅读顺序线性化** | 多栏排版、侧边栏、插入图文混排时的正确阅读流（Reading Order）重构 | 围绕解析技术的实际落地，Jerry 详细阐述了精度、成本与延迟三者之间的“帕累托曲线”（Pareto Curve）权衡策略： - **高精度敏感场景**：在金融、法律等对数字准确性要求绝对严苛的领域，用户愿意为多步 Agentic 推理和交叉校验支付额外算力成本； - **超大规模吞吐场景**：当企业每年需处理 1000 万至 10 亿页文档时，单页解析成本必须被压至极低，否则单是解析费就高达上亿美元； - **极低延迟交互场景**：当用户向 AI 助手一次性上传上百份文件时，系统必须在数秒内完成处理，而不是让用户等待数分钟或数小时的多模态大模型慢速推理。 --- ### [14:05](https://www.youtube.com/watch?v=j96gok4nZlg&t=845s) 轻量开源解析 Light Parse 与 Schema 驱动的文档提取 针对低延迟与低成本的需求，Jerry 重点介绍了 LlamaIndex 推出的开源纯 Rust 解析器 **Light Parse**。他将其定义为一款“极速、免费且表现惊艳”（Fast, free, and surprisingly good）的文本提取工具。Light Parse 支持 Python、Node.js 以及命令行 CLI，并能作为技能（Skill）或 MCP（Model Context Protocol）插件直接挂载进各类 Agent 运行时。它不调用任何多模态大模型，核心定位是作为 Agent 在执行任务时的第一道初筛工具（First pass）；当 Agent 发现某特定页面存在复杂图表或复杂表格时，再动态调用 LlamaParse 进行深度视觉解析，从而兼顾了极致的速度与最终解析质量。 ```mermaid flowchart LR accTitle: Agent 文档处理双层混合解析流程 accDescription: 展示先由轻量解析器初筛，必要时再触发高精度深度 OCR 的两阶段工作流 Doc["输入文档"] --> Step1["第一道初筛: Light Parse<br/>(纯 Rust / 极速 / 免费提取)"] Step1 --> Judge{"页面是否包含<br/>复杂图表/表格?"} Judge -- "常规文本页" --> FastOutput["直接输出给 Agent 上下文"] Judge -- "高难度页面" --> Step2["第二道深度解析: LlamaParse<br/>(后训练 VLM + Agentic 推理)"] Step2 --> AccurateOutput["输出带 Bounding Box 的<br/>结构化 Markdown / JSON"] ``` 在解析基础之上，Jerry 进一步介绍了**文档提取（Document Extraction）**功能。文档提取本质上是“文档 OCR + 顶层推理层”，其目标是根据用户预定义的结构化 Schema，从复杂的非结构化纸质单据（如各类发票、保险索赔单、投资协议、施工图纸等）中精准抽取出目标字段。相比直接调用 LLM 的传统方式，专用的提取 Harness 能确保输出字段与原始页面坐标之间具备精确的定位回溯（Bounding Box Grounding），并附带准确的置信度评分，使企业级流水线具备坚实的审计能力与合规性。Jerry 在现场演讲中还幽默地提到投屏右侧本应播放一段演示视频，尽管屏幕显示为空白，但这并不妨碍大家理解通过定义 Schema 自动抽取结构化数据的直观价值。 - **Light Parse 特性**：基于 Rust 编写、速度极快、开源免费、支持 MCP，专注于作为 Agent 工作流的第一道轻量抽取层。 - **结构化提取（Document Extraction）**：基于 Schema 驱动，融合 OCR 与推理，输出自带源文件坐标锚定（Grounding）与置信度评分的结构化结果。 - **企业价值**：彻底摆脱了过去针对每种版式单独训练专有小模型的繁琐流程，为海量合同、单据自动化提供了高鲁棒性的解决方案。 ### [16:47](https://www.youtube.com/watch?v=j96gok4nZlg&t=1007s) 从朴素 RAG 到智能体搜索：赋能多样化检索工具 在文档智能处理的体系中，除了深度解析与结构化提取，另一个核心支柱是文档搜索，这直接对应了当下各类智能体搜索（Agentic Search）的工作负载。回想 2023 年，行业中普遍构建的还是相对简陋的朴素 RAG（Naive RAG）方案：基本流程就是把文档切块（Chunking），塞进向量数据库，然后执行 Top-K 相似度检索。 如今，整体技术栈已经逐渐固化并有所升级，但架构本身并没有变得过度复杂。现代智能体搜索的核心范式，是为代码智能体或前沿大模型框架赋予一系列基础工具（如 BM25 关键词检索、语义向量搜索等），然后让智能体自主在海量文档集上规划检索。LlamaParse 平台内置了索引能力，例如支持一次性索引 10,000 份 PDF 文档，并对外暴露 BM25、语义搜索等开箱即用的检索端点。开发者无论选用 OpenAI Codex、Anthropic Claude Code 还是其他前沿模型 Harness，都能将复杂的推理与调度任务交由智能体处理，同时让智能体拥有超越单纯 Top-K 检索的多维文档阅读与操作能力。 ```mermaid flowchart LR accTitle: 智能体搜索架构演进 subgraph DocLayer["文档数据层"] A["海量文档 (如 10,000+ PDF)"] --> B["LlamaParse 统一索引"] B --> C1["BM25 检索端点"] B --> C2["语义向量检索端点"] end subgraph AgentLayer["智能体框架 (Harness)"] D["前沿模型 (Claude / Codex)"] D --> E["工具调用 & 规划推理"] end C1 <--> E C2 <--> E ``` - **从固定流水线到工具化调用**：告别写死逻辑的 Top-K 检索，将 BM25 与向量搜索包装为工具供智能体动态调用。 - **开箱即用的海量文档索引**：LlamaParse 支持对数以万计的 PDF 文档建立统一索引并提供检索 API。 - **模型框架解耦**：无缝对接 Claude Code、Codex 等主流智能体框架，将复杂的推理卸载给前沿大模型。 --- ### [18:10](https://www.youtube.com/watch?v=j96gok4nZlg&t=1090s) 企业级确定性工作流：DAG 编排与业务自动化 随着通用智能体能力的迅速提升，企业场景中的知识工作却仍然面临一个现实瓶颈：许多核心业务环节对“确定性”有着极其严苛的要求。在发票报销处理（Invoice Processing）、保险理赔核赔（Claims Processing）以及金融客户身份认证（KYC）等任务中，企业绝不能容忍不可控的随机性（Stochasticity），而是必须确保系统严格按照“第一步、第二步、第三步”依序执行，且每一个中间环节都必须具备可视化与可审查性（Inspectable）。 因此，将业务流程建模为有向无环图（DAG）的工作流（Workflows）依然不可或缺。LlamaIndex 在平台底层构建了完善的核心能力，允许开发者通过代码或自然语言来显式表达这类工作流。即便通用对话式交互已经普及，人类专家仍需要通过提示词或操作手册（Runbooks）来约束系统。只有将工作流的可靠性与智能体的灵活性相结合，企业才能在保持高容错与可审计的前提下，真正实现规模化运营。 - **确定性 vs 随机性**：核心金融与合规场景（KYC、理赔、发票）必须消除不可控的随机输出。 - **DAG 任务编排**：支持以可视、可修改、可审查的有向无环图形式定义业务流。 - **双模表达支持**：允许开发者通过编写代码或直接使用自然语言定义复杂的工作流逻辑。 --- ### [19:38](https://www.youtube.com/watch?v=j96gok4nZlg&t=1178s) 下一代文档智能体：长周期运行、原生格式与自进化 展望未来发展方向，核心命题依然是“如何为智能体提供最高质量的文档上下文”。这一链条始于解析、提取、搜索与工作流，但随着智能体从“被动阅读”迈向“主动生成与编辑”，全新的技术机遇正在涌现——包括智能体原生文档格式、文档版本控制、精准局部编辑以及长周期（Long-running）运行机制。 目前智能体最擅长生成 Markdown 格式，便于人类快速阅读与审阅，但 Markdown 作为视觉呈现层存在明显局限；HTML 在智能体生成内容中也很流行，但业界仍然缺少一种让“人类与智能体都能高效读写、检查和呈现”的理想文件格式接口。此外，智能体的任务跨度（Task Horizon）是一个根本性的开放课题：接入工具后的智能体不应仅仅停留在完成 5 分钟的单点即席任务，更应该在后台持续运作，模拟知识工作者的全天候日常业务。要实现这种 24/7 的不间断自动化，不仅需要提升智能体 Harness 的工程设计，还必须在数据层持久化存储智能体的“决策轨迹”（Decision Traces），让智能体能够根据事件触发器持续运行，并在历史决策中实现自我进化与迭代。 - **格式演进瓶颈**：Markdown 视觉表现力不足，亟需探索兼顾人机协同与渲染表现力的新一代文档格式接口。 - **任务跨度延展**：从“5 分钟即席交互”跃迁到“24/7 持续运行”，端到端接管长流程知识工作。 - **决策轨迹回溯（Decision Traces）**：在数据层记录智能体的推理与决策链条，为长周期运行和自主迭代提供数据支撑。 --- ### [21:31](https://www.youtube.com/watch?v=j96gok4nZlg&t=1291s) 总结与展会交流 演讲最后总结道，如果整个分享只能带走一个核心要点，那就是：“我们全力以赴所做的一切，就是为你喜爱的智能体解锁最高质量的文档上下文。” 对于拥有海量复杂非结构化文档的企业，演讲者邀请参会者前往 1561 号展位进行深入交流，并提到当晚将与数据抓取平台 Nimble 以及机器学习平台 Comet 联合举办线下交流晚宴。 - **核心使命**：为各类 AI 智能体打通企业级文档上下文层（Document Context Layer）。 - **线下活动**：展位号 1561，当晚与 Nimble 和 Comet 联合举办交流晚宴。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=j96gok4nZlg&t=4s) 归根结底，如果模型对你的业务一无所知，它们就无法做出任何有价值的事情。 — Jerry Liu > [00:04](https://www.youtube.com/watch?v=j96gok4nZlg&t=4s) 上下文就是一切 — Jerry Liu > [02:53](https://www.youtube.com/watch?v=j96gok4nZlg&t=173s) 文档上下文层基础设施 — Jerry Liu > [05:38](https://www.youtube.com/watch?v=j96gok4nZlg&t=338s) 会让你直接破产 — Jerry Liu > [14:05](https://www.youtube.com/watch?v=j96gok4nZlg&t=845s) 极速、免费且表现惊艳 — Jerry Liu > [14:05](https://www.youtube.com/watch?v=j96gok4nZlg&t=845s) 文档 OCR + 顶层推理层 — Jerry Liu > [19:38](https://www.youtube.com/watch?v=j96gok4nZlg&t=1178s) 如何为智能体提供最高质量的文档上下文 — Jerry Liu > [21:31](https://www.youtube.com/watch?v=j96gok4nZlg&t=1291s) 我们全力以赴所做的一切，就是为你喜爱的智能体解锁最高质量的文档上下文。 — Jerry Liu ## Mentioned Jerry Liu, LlamaIndex, LlamaParse, Light Parse, ParseBench, Ai4, Claude Code, Anthropic, OpenAI Codex, Gemini, GPT, Hugging Face, Kaggle, parsebench.ai, Rust, Model Context Protocol (MCP), Nimble, Comet, SharePoint, OneDrive, Amazon S3 ## Video description Jerry Liu - CEO and Co-Founder, LlamaIndex 90% of enterprise data is unstructured, and a lot of that lives in the form of documents - PDFs, pptx, docx, etc. - powering a majority of enterprise knowledge work.AI agents have the potential to simultaneously better make sense of docs and also automate human workflows over them. Existing document AI use cases have either been long-running but constrained (e.g. a batch invoice processing through RPA), or non-deterministic but short-horizon (e.g. ChatGPT deep research over a collection of existing files). Going forward there's a huge opportunity to build long-running agents that can understand, reason over, and edit documents at scale.In this session, we'll walk through core advances in document OCR and agent orchestration that have given rise to initial use cases around doc extraction workflows and general-purpose chatbots (and share real-world customer use cases). We will then articulate some of the remaining gaps needed to achieve "long-running" document work automation. [Source](https://www.youtube.com/watch?v=j96gok4nZlg)

[0:04] Jerry Liu: 归根结底，如果模型对你的业务一无所知，它们就无法做出任何有价值的事情。 [0:04] Jerry Liu: 上下文就是一切 [2:53] Jerry Liu: 文档上下文层基础设施 [5:38] Jerry Liu: 会让你直接破产 [14:05] Jerry Liu: 极速、免费且表现惊艳 [14:05] Jerry Liu: 文档 OCR + 顶层推理层 [19:38] Jerry Liu: 如何为智能体提供最高质量的文档上下文 [21:31] Jerry Liu: 我们全力以赴所做的一切，就是为你喜爱的智能体解锁最高质量的文档上下文。

[0:04] Jerry Liu: context is everything [0:04] Jerry Liu: the primary bottleneck in unlocking enterprise value is no longer raw model capability. Instead, it is the user's ability to supply the model with rich, relevant context. [0:04] Jerry Liu: Without deep context, artificial intelligence cannot perform meaningful, customized work. [0:04] Jerry Liu: An estimated 10 trillion pages of human-native content reside across PDFs, Word documents, PowerPoint presentations, Excel spreadsheets, and HTML files. [5:38] Jerry Liu: Document OCR has existed for two to three decades, yet it remains fundamentally difficult because formats like PDF were engineered for graphical display rather than machine comprehension. [5:38] Jerry Liu: PDFs store content purely as 2D display coordinates and visual glyph shapes rather than semantic, machine-readable text streams. [5:38] Jerry Liu: While modern Vision-Language Models (VLMs) enable one-shot parsing across diverse layouts, simple LLM API calls create prohibitive costs, hallucinated outputs, loss of visual grounding, and a lack of calibrated confidence scores. [9:46] Jerry Liu: High-volume workloads processing 10 million to 1 billion pages per year cannot afford generic 10-cent-per-page VLM calls, which would total up to $100 million in OCR compute alone. [16:47] Jerry Liu: Document search serves as the natural foundation for any agentic search workload. [18:10] Jerry Liu: While generalized reasoning agents continue to advance, enterprise knowledge work still demands a high degree of determinism for core operational tasks. [18:10] Jerry Liu: Modeling these procedures as Directed Acyclic Graphs (DAGs)—where execution flows strictly from step one to step two to step three—provides the necessary control, reliability, and visual inspectability. [19:38] Jerry Liu: Rather than completing single ad-hoc tasks within a 5-minute window, the goal is building autonomous agents that operate continuously in the background on a 24/7 basis, mirroring the sustained output of knowledge workers.

[0:04] Hey everyone. Great to be here. It's my first  time at the AI 4 conference and this room is [0:09] absolutely massive. It's like probably one of  the biggest rooms I've I've seen at a conference [0:15] conference center. So the main goal of  today's talk is basically building the [0:19] document context layer for AI agents and I  think part of the goal of the talk is really [0:23] to kind of impress upon you the importance of  context even as agents get really really good [0:28] and specifically what's actually needed to  focus on a specific piece of that context layer, [0:33] which is the vast pool of unstructured document  based data that exists in the world today [0:38] that powers a lot of like knowledge work  out there today. For everyone just joining, [0:42] I'm Jerry, co-founder CEO of LlamaIndex. It's  great to be here and today the talk title is [0:47] building the modern document context layer for  AI agents. So we'll just get kind of into it. [0:53] You know, for for me context is is everything. You  know, models and agents are getting exponentially [1:00] better. You know, like with Fable 5, Opus 5, the  new GPT models, all the open weight models are [1:06] getting really good at agentic reasoning, coding  and math. And the harnesses are getting better, [1:11] too, right? Like applications are standardizing  around stuff like Claude code, the Codex harness. [1:17] But the biggest barrier to the ability for you to  actually use AI models to power your own workflows [1:23] is your own ability to provide context to the  models. At the end of the day, if the models [1:28] don't know anything about you, they're not going  to be able to do anything useful. And so when you [1:33] think about like the value that you provide to the  AI in order to help accelerate your own workflows, [1:38] we think about that in terms of context.  This includes runbooks, this includes like [1:42] web search context and for us because our core  mission of the company is around like documents, [1:48] this includes the vast pool of document  context that exists in the world today. [1:53] There are 10 trillion plus pages of human  native content that's spread up spread [1:59] out within like PDFs, Word documents, you know,  PowerPoints, Excel sheets, HTML files, and others. [2:06] We see the volume of these pages processed by  AI agents exponentially increasing over time. [2:12] Just think about all the massive pools of like  document-based data that exists within your file [2:17] system that are locked up as web attachments,  email attachments, or exist within your [2:21] SharePoint, OneDrive, S3, so on and so forth.  The ability for agents to actually ingest this [2:28] content is basically a necessary condition  for them to do any sort of knowledge work. [2:32] Whether you're responding to an email, doing call  prep, being able to generate an investment memo, [2:38] being able to kind of like do  some invoice reconciliation, [2:42] a lot of knowledge work depends on the vast  pool of unstructured documents out there today. [2:47] So, for us, we really think about how do  you build an agent-native document platform? [2:53] For those of you who might be familiar with  LlamaIndex, you know, I'm the co-founder and CEO, [2:58] and we have a very strong team that's evolved our  mission over the past few years. You might have [3:02] seen us as an open-source rag framework when  we first started 3 years ago. And since then, [3:07] we've really narrowed and evolved our mission  towards building best-in-class infrastructure [3:12] at the document context layer. This includes a few  core components. This includes the parsing layer, [3:19] which is basically document OCR, being able to  read PDFs into even the hardest PDFs out there, [3:25] and PowerPoints, and Word documents into  clean, well-structured content that agents [3:29] can understand. This includes, you know, the  semantic and storage layer as well. How do you [3:34] actually offer the right retrieval endpoints,  permissioning, versioning, so that agents and [3:40] humans can actually collaborate, operate on, and  interact over documents? And this also includes [3:45] kind of more end-to-end document workflows,  especially as, you know, more horizontal [3:50] harnesses are coming out, but people are starting  to become interested in using the right blend of [3:56] models actually satisfy a task instead of just  using the latest frontier model, being able to [4:00] actually help climb a workflow over documents,  satisfying a specific point on the Pareto curve of [4:06] cost and accuracy becomes increasingly important  for basically any task that you want to solve. [4:12] So, today, you know, the core capabilities that we  have within the document context layer. And again, [4:17] keep in mind this is basically what you needed  to actually this as context to any sort of agent [4:22] technology work. This includes document OCR,  you know, being able to actually convert to PDF [4:28] into well-structured markdown, JSON, bounding  boxes and others. This includes, you know, [4:33] document extraction and semantic search. If  you index an entire collection of documents, [4:38] having the right retrieval end points to be able  to grep, search by keyword, search via embeddings, [4:44] and do all sorts of different operations to  pull out the right context that you need. [4:48] And this also includes some initial capabilities  towards being able to build like end-to-end [4:53] workflows over your documents, being  able to classify, split, extract, [4:57] validate, route, and do a lot of other types  of operations on top of document-based data. [5:03] We'll probably lightly touch on kind of the future  territory that's still yet unexplored because [5:08] I basically see a world where a lot of software,  including ours, is basically being an agent native [5:14] way for, you know, both agents and humans to be  able to interact with various sources of data. [5:19] So, some of the additional product features that  you need for kind of a document platform include [5:23] being able to invest in agent native  document formats that are both human and [5:27] also machine readable, document versioning,  editing, permissions, and collaboration [5:31] in a way that both agents and also humans can  understand. So, let's start with document OCR. [5:38] If you followed our Twitter, I don't know, just  like the company at all over the past like few [5:42] years, you might have seen that as we have evolved  from rag to really our current mission and focus, [5:47] We started talking a lot about the importance of  document OCR, which is basically the translation [5:52] of your PDFs into well-structured markdown  and JSON. Um it's been around for 20 years. Um [5:57] you know, why why do we decide to focus on it  now? It's because it's a very hard problem. [6:01] Um the representations of a lot of these documents  are not natively machine-readable. PDFs store [6:07] contents as like display coordinates and not  necessarily as machine-readable text. And so they [6:13] basically require kind of complex algorithms  to actually pull information out of these [6:17] containers in a way that you can actually use  as interpretable plain text contacts for agents. [6:24] PDF parsing is especially hard. Um if you have,  you know, a complex research report with like [6:29] charts and tables, if you have like uh just like a  giant rate sheet with like a 10,000 items within a [6:34] given page, you can see that, you know, the  way that text is drawn on the documents um [6:39] is basically not very easy for a machine to  actually interpret what's going on. You know, [6:44] tables are drawn as just a mixture of lines and  text. Um you know, I think text are represented [6:48] as glyph shapes. Images are of course represented  as like binary bits. Um and so there's no ability [6:55] inherent ability for someone to just like pull  out a holistic representation of a document [7:00] in a way that again is plain text and  that you can just feed to an LM endpoint. [7:05] Even like other plain text formats uh like  slightly more plain text formats like Word and uh [7:10] uh PowerPoints um and Excel um are are also  hard to parse out. You know, Word documents [7:15] are OOXML. Um they carry like more structured  information than a PDF, similar to HTML pages. [7:21] Um but you still need kind of core features  to be able to determine pagination. Being [7:26] able to actually map out proper table  structures, especially if you kind of [7:30] drew the lines on there instead of actually  representing it as a Word table. And again, [7:34] representing it uh and being able to preserve  stuff like formatting, uh uh strike-throughs, [7:39] italics, um redlines, versioning, and others.  And so, uh, the this industry itself has been, [7:47] you know, kind of going on for around 20 to  30 years. Prior to Johnny I, uh, people used [7:52] both like heuristic-based approaches where you  fit like deterministic templates onto various [7:57] types of documents that were basically the same  in in format, but just like varying in content. [8:02] Um, people also fit machine learning models,  like carefully tuned machine learning models, uh, [8:07] prior to LLMs and VLMs to really classify and be  able to detect a certain class of document types. [8:14] The advantage of the new age of kind of like, uh,  VLMs and LLMs and transformer-based architectures [8:20] is that instead of having to spend effort trying  to like, you know, uh, train a new model for every [8:25] new subtype of documents, there's really like just  one-shot approaches now, um, including what we do, [8:31] which can process just like any document  of any complexity, and then use agentic [8:35] reasoning plus post-trained VLMs towards  representing the document in the right format. [8:41] I think the, um, obviously a lot  of the latest frontier models, uh, [8:45] carry these capabilities. Um, if you use  like, I don't know, Gemini 3.5 Flash, if [8:50] you use GPT-5.6 Soul, if you use Fable, they all  have like native image reading capabilities. So, [8:56] the default path is you screenshot a page, feed  it to a frontier model, and you get back the text. [9:01] The issue is, uh, one, it's like super expensive,  um, if you actually want to run this across like [9:06] millions of documents, you're actually going  to go bankrupt. Um, and then the second is, um, [9:11] you're going to have a lot of edge cases. It's not  going to be able to read the most complex tables. [9:16] Um, actually, maybe even more importantly,  besides the hallucinations and missing content, [9:20] um, it lacks a grounding. You basically don't have  bounding boxes, uh, over your documents. Um, you [9:25] also don't really necessarily have cal- calibrated  confidence scores that allow, you know, you'd [9:30] actually flag if it thinks, you know, this content  needs additional human review. There's just a lot [9:35] of metadata and annotations are generally useful  for processing documents that are hard to do if [9:40] you try to DIY it with an LLM API. And in general,  I think our entire focus as a company these days [9:46] has basically become a very focused like AI  agent like model lab that specializes in document [9:51] understanding. We post train models. Um, we have  a focus that just like get really really deep into [9:57] the PDF like engine um and make use of all the  text that's available within these file formats. [10:04] Um, and then you know, we create open source  libraries. If you've checked out light parse, you [10:07] know it's a very fast and cheap uh text extractor  to also be able to train our own models. Um, and [10:13] inters- like stitch together different techniques  to parse document pages of any complexity. [10:18] Um, the just a quick note about the product  itself. Um, our core product today is called Llama [10:23] Parse. It's basically represents our entire vision  um of document infrastructure for agents and [10:29] contains, you know, of course document parsing,  OCR, and also document extraction, search, and [10:34] other types of workflows. And for us represents  the core foundational module to actually process [10:38] and structure all your document-based context  regardless of whatever agentic application you're [10:43] building on top of it. Um, the benchmarking piece  itself is hard, too, because um you know, there's [10:49] actually haven't been that many modern attempts  towards really understanding how both frontier [10:54] models and OCR solutions do in terms of real-world  documents um in terms of semantic quality. Um, [11:00] you know, for us in order to actually build stuff  that's really good, we also need to build the [11:05] right comprehensive benchmarks that reflect the  real-world use cases that our customers are using [11:09] our stuff for, whether it's financial services,  legal, insurance, manufacturing, construction. Um, [11:15] you know, parse bench represents our first attempt  at it. It contains 2,000 human verified pages [11:21] across three or four domains. It's financial  services, insurance, government, legal. [11:26] Um, and it measures kind of the ability for  models and OCR tools to read some of the common [11:32] elements within a page uh including, you know,  dense tables. Uh you know, if you have a really [11:36] dense kind of like formatted table like a balance  sheet for instance or like unlabeled line charts. [11:42] Um if you're able to detect stuff like red  lines, italics, strikethroughs, highlights, [11:46] uh be able to to to actually linearize the page  into the proper reading order so that humans can [11:50] actually understand what's going on. And this  is a fully open benchmark uh that's available [11:56] on Hugging Face, Kaggle. Um you know, I think  we're iterating on subsequent versions of this [12:00] and also more vertical specific uh iterations of  this too uh targeted towards specialized domains. [12:06] Um but it's kind of important for us to understand  because, you know, as frontier models themselves [12:10] get better and better, um they we want to make  sure that obviously we're able to make use of the [12:15] latest visual understanding capabilities of some  of these models but continue to distill and make [12:20] sure that we're able to cover all the different  points on the Pareto curve of accuracy and cost [12:24] no matter how complex the document pages  are. Um you know, we benchmark against uh [12:28] the OpenAI and Tropic, Google models. We  also benchmark against other specialized [12:33] OCR providers from startups to hyperscalers. Um  and you know, the benchmark is fully available on [12:40] uh Hugging Face, Kaggle and it's also on  parsebench.ai um if you want to take a look. Um [12:46] in general, I think there's a lot of opportunities  to continue pushing this Pareto curve of accuracy, [12:50] cost, and latency. Um you can see while all the  frontier models are obviously uh starting to hill [12:55] climb on reasoning benchmarks like coding, math,  um you know, kind of like general knowledge work, [13:00] uh we're investing a lot on just basically pushing  the Pareto curve of visual understanding of like [13:04] document-based data. And so, this actually maps  to a few different use cases depending on where [13:09] you're at the Pareto curve. If you care about  accuracy, if you're in a very sensitive use case [13:14] where the numbers have to be absolutely correct  and you need to extract out the right information, [13:18] um that's probably something where you're  willing to pay a little bit more money to [13:21] use agentic reasoning validation to make sure you  extract out the right information from documents. [13:26] If you care about like scale, if you're processing  like 10 million to a billion documents a year, [13:31] you don't want to, you know, pay like 10 cents  per page, otherwise you're going to be paying like [13:35] $100 million just on doc parsing. And so there's  ways of actually being able to get this content [13:40] a lot cheaper and faster. Um the kind of frontier  that I think we're pushing on that is still like [13:46] has a lot of room to grow is like low latency. If  a user uploads like 100 documents to Cloud Code or [13:53] to your AI agents, how do you actually process it  in a fast and efficient manner instead of waiting [13:58] like minutes or hours just for a kind of like  higher quality VLN based processing over the data. [14:05] The other kind of dimension of this is  Light Parse, which we call fast, free, [14:09] and surprisingly good parsing. It is  the best free and open source parser [14:13] out there. It is completely Rust based. It  supports, you know, kind of Python and Node. [14:19] It's available via CLI. You can plug it in as  a skill or MCP into your favorite agent, too. [14:25] And our main goal is basically to just be the best  like free text extractor out there. It's not This [14:30] doesn't use VLNs at all. This is actually designed  to be used in the agent loop as a first pass. [14:35] And then afterwards, if the agent actually needs  to dive deeper into a specific page of content, [14:40] it will use either Lama Parse or some other,  you know, OCR solution to be able to read the [14:44] text off the page. I think in general we're going  to see a lot more tools like this that, you know, [14:51] besides using kind of like document processing  in offline fashion, as like agents with knowledge [14:57] work take off and gain adoption across a variety  of different industries, actually plugging this [15:01] in as a skill in the agent loop so that you're  able to guarantee almost like high accuracy, [15:07] high quality, and also auditability over your data  is going to be increasingly important. So besides [15:13] parsing, I've talked to D Cena about parsing.  There's also stuff like document extraction. [15:19] You know, document extraction is basically schema  based like structured outputs over any documents. [15:24] Um, it's document OCR plus a reasoning layer on  top so that you actually extract out the relevant [15:30] items that you care about from a document instead  of just parsing. Um, obviously the frontier models [15:35] can do this, too. Um, you just, you know, feed  all the text into an LLN and then ask it to output [15:39] structured outputs. The main thing is, um, it  oftentimes helps to have a specialized harness do [15:45] this and we're going to come out with benchmarks  relatively soon that basically highlight the [15:49] advantage of our solution versus kind of a naive  one-shot call to an LLN. But, if you think about [15:54] any kind of paperwork heavy use case within the  enterprise, you know, whether it's invoices, uh, [15:59] claims, or you're setting up like data pipelines  over just a a massive pool of like contracts, [16:03] investment memos, um, you know, uh, like, you  know, construction diagrams or blueprints, [16:09] um, you want to make sure that you have really  high accuracy data that's also well-calibrated, [16:14] uh, to the, um, with like the right levels of  grounding, uh, with the right bounding boxes back [16:19] to the source, and also well-calibrated  to the confidence level of the parser. [16:24] Um, and so, you know, uh, there's there's no video  in this. This is I know there's like a blank, uh, [16:28] screen on the right. It's supposed to be a video.  But, basically, um, you can imagine you, the user, [16:33] define some sort of schema, um, you input the  schema along with the document and extracts out [16:37] like structured output based on that. It used to  be prior to join AI, you would have to actually, [16:42] uh, kind of, um, actually train a specialized  model to extract out information. Um, [16:47] these days we have like an extremely fine-tuned  harness that's really, really good at just going [16:52] over your pages and doing deep document extraction  over again any pages of any document complexity. [16:58] The last piece here is document search, uh, which  is obviously a use case for any kind of agentic [17:03] search type workloads. Um, I think back in 2023  when everyone's building like naive rag solutions, [17:09] it was basically, uh, the steps  around top K chunking, or sorry, [17:13] chunking and then putting into a vector  database and doing like top K retrieval. Um, [17:17] I think these days uh the stack has maybe  solidified and improved a little bit, but [17:21] actually hasn't gotten more complex. The overall  uh architecture of like agent search today is, [17:27] you know, you basically give uh coding agent  harness access to a set of tools like BM25, [17:32] semantic search, um and then you basically apply  it over a collection of documents. Um, we have [17:37] some inbuilt capabilities within Llama Parse  that let you index, let's say like 10,000 PDFs, [17:42] and then we offer a set of retrieval endpoints  like BM25, semantic search over that. That's very [17:48] um like easy to plug in to whatever frontier model  harness that you want to use. So, whether you're [17:52] using Codex, Claude, Code, and others, you're  able to actually use that, offload a lot of [17:57] complexity and reasoning to the agent harness,  but still equip it with a diversity of tools to [18:02] read and operate over these documents, and search  it in various ways beyond like top-k retrieval. [18:10] The last piece here is basically around document  workflows. Um, I think basically, you know, [18:15] as generalized agents get better and better, um  there is a certain set of like use cases within [18:21] uh you know, the enterprise and just like various  aspects of knowledge work that still need a level [18:25] of determinism. Um, I think it's still helpful  to think about, you know, workflows as a concept [18:31] uh where you basically map it out as a DAG,  so to speak, where you follow like step one, [18:35] step two, step three. Um, and you know, the  reason is you want some level of determinism. [18:40] You don't really want to have stochasticity in  this, and you want the ability to kind of be [18:44] able to actually modify the steps in a visual and  inspectable way. Um, this includes anything that, [18:50] you know, can be modeled out as a workflow from  invoice processing to claims processing to KYC. [18:56] Um, and I think we've built a lot of the  core capabilities uh within the platform [19:01] to actually let you express a workflow through  code or natural language. Um, and I think we see [19:06] workflows still having a place to and like power  a a of different types of enterprise operations, [19:12] even as uh these generalized chat interfaces  exist across a variety of different [19:18] like platforms. Because even as humans input their  own prompts, runbooks into these generals agents, [19:25] you still need to have some level of processes  where you have some level of reliability, [19:30] determinism, and you know, being able  to actually run that thing at scale. [19:38] So, a quick note on what's next. In terms of  just a lot of the things that we think about [19:45] are basically around how do we provide just the  highest quality document context to agents. And it [19:51] starts with document parsing, document extraction,  document search, and workflows. I think there's [19:56] still as agents are not only reading more content,  but actually starting to generate more content, [20:01] increasing opportunities around you know, agent  native document formats, document versioning, [20:06] document editing, and also being able to create  kind of longer running workflows over this data. [20:12] So, you know, agents are well tuned towards  generating markdown formats today. You know, [20:16] they're easily readable and revealable by humans.  But, you know, there's kind of some downsides [20:22] with markdown as a visual representation layer.  HTML is also a format that I think seems popular [20:28] among agent generated content. But, I still think  there's some emerging opportunities to be able [20:33] to have a great file format interface that both  agents and humans can read, write, and inspect. [20:40] And I think you know, as the models and harnesses  get better and as agents are able to not only do [20:46] ad hoc tasks, but actually complete end-to-end  processes around knowledge work, there's an open [20:51] question around the task horizon of all these  like document agents. If you actually plug in [20:56] these tools into these harnesses, can it not only  just do a certain task within 5 minutes for you, [21:01] but actually continuously run in the background  and basically model a lot of operations that like [21:06] knowledge workers typically do and even run on a  24/7 basis. Um I think this includes just being [21:12] able to actually improve the harness engineering  around whatever agents actually kind of plugged [21:16] into these tools, but also you need to be able  to store these decision traces at the data layer, [21:21] so that agents can actually self-improve  over these decisions over time, [21:25] and actually run continuously according  to a variety of different event triggers. [21:31] So, that's basically it. Um you know, I think if  uh you took away one thing from the talk, it's [21:35] basically we care a lot about how do you unlock  document context for your favorite agents today. [21:41] Um if you do have a lot of documents, uh come  talk to us. We're at booth um 1561. Um and we [21:47] also have a dinner tonight that we're hosting with  uh Nimble and also uh Comet. Um and so, thank you.