Snapshot AI CEO Says Naive RAG Fails Enterprise AI Over Context Limits

Context Is a Lie What RAG Gets Wrong—and What Comes Next With Snapshot AI — Ai4

Ai4

Enterprise AI pipelines fail when stuffing raw documents into prompts because transformers operate as next-token autocompleters lacking conceptual comprehension, demanding a shift toward structured knowledge graphs and internal neural memory.

AI-assisted summary of Context Is a Lie What RAG Gets Wrong—and What Comes Next With Snapshot AI — original source. Check the original for full context.

Source published: 2025-08-29

Detailed notes

## Background Published on YouTube by Ai4 on August 29, 2025, this presentation features Kirimgeray Kirimli, CEO of Snapshot AI, addressing the core limitations of modern retrieval systems. According to the description, the talk dissects why naive RAG and brute-force context stuffing fail to deliver genuine understanding, advocating instead for structured, persistent memory architectures like Raptor and Memory Mosaics that enable models to reason rather than merely regurgitate.

## TL;DR Kirimgeray Kirimli argues that naive Retrieval-Augmented Generation and ever-expanding context windows fundamentally bottleneck AI intelligence by confusing statistical next-token prediction engines with raw, unstructured data. Because transformers struggle with token distraction and lack hierarchical concept comprehension, feeding massive enterprise backlogs directly into prompts leads to degraded performance and high latency. While production systems currently rely on GraphRAG and agentic node-traversal to make data digestible, next-generation solutions like Memory Mosaics embed context directly into internal memory modules. Ultimately, achieving production-grade accuracy across mission-critical domains requires moving beyond surface tokenization and naive prompt-stuffing toward genuine conceptual reasoning.

## Atmosphere & style The presentation maintains a pragmatic, incisive, and technically grounded tone that deliberately undercuts prevailing industry hype around massive context windows. Kirimli speaks with the measured clarity of an enterprise builder, balancing deep theoretical critiques of transformer decoders with tangible high-stakes analogies like field turbine maintenance. The session feels like an eye-opening architectural debrief, trading speculative enthusiasm for a rigorous, disciplined look at the middleware engineering required to make AI reliable.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=I367mYw8guI&t=4s) Naive RAG acts as the backbone for modern AI tools but forms a critical architectural bottleneck because it lacks higher-level conceptual understanding of the data it retrieves. - [00:04](https://www.youtube.com/watch?v=I367mYw8guI&t=4s) Brute-force context stuffing into large prompt windows produces muddled results and token confusion rather than actionable intelligence. - [02:18](https://www.youtube.com/watch?v=I367mYw8guI&t=138s) Standard transformer architectures lack dual-tier comprehension, preventing them from simultaneously tracking granular low-level details and overarching structural models. - [02:18](https://www.youtube.com/watch?v=I367mYw8guI&t=138s) Because LLMs remain statistical next-token prediction engines, scaling context length increases token distraction and actively degrades reasoning performance. - [04:45](https://www.youtube.com/watch?v=I367mYw8guI&t=285s) Enterprise data causes RAG latency to spike due to quadratic attention costs, while surface-level token matching fails to synthesize actionable business insights across all major model providers. - [06:58](https://www.youtube.com/watch?v=I367mYw8guI&t=418s) Next-generation architectures like Memory Mosaics transition context handling from external vector databases directly into in-network, parallel internal memory modules. - [09:06](https://www.youtube.com/watch?v=I367mYw8guI&t=546s) GraphRAG overcomes context degradation by structuring enterprise datasets into discrete nodes traversed by agents using search algorithms like BFS and DFS. - [09:06](https://www.youtube.com/watch?v=I367mYw8guI&t=546s) Partitioning information into tightly bounded, semantically coherent segments leverages the fact that LLMs achieve near-perfect synthesis on small contexts. - [12:41](https://www.youtube.com/watch?v=I367mYw8guI&t=761s) Bridging the gap between 70–90% proof-of-concept accuracy and the 100% precision demanded by mission-critical enterprise deployments requires extensive, deterministic middleware engineering. - [16:39](https://www.youtube.com/watch?v=I367mYw8guI&t=999s) Standard tokenizers blind models to conceptual equivalence across different scripts and representations, creating disjoint token IDs for identical semantic entities. - [16:39](https://www.youtube.com/watch?v=I367mYw8guI&t=999s) Pure transformer scaling has reached architectural limits, necessitating a shift beyond next-token autoregression toward conceptual reasoning, internal memory, and dynamic ingestion.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=I367mYw8guI&t=4s) The Context Problem and the Limits of Naive RAG

Every modern AI tool relies on Retrieval-Augmented Generation (RAG) under the hood. Whether developers are using ChatGPT to conduct background research or Cursor to autocomplete and generate code, RAG serves as the foundational retrieval backbone. However, this reliance exposes a critical architectural bottleneck. Recent foundational model releases fail to deliver expected leaps in intelligence not simply because of training scale, but because RAG lacks a higher-level conceptual understanding of the data it retrieves.

Foundational model marketing frequently touts context windows reaching a billion tokens, but the reality of the underlying transformer architecture tells a different story. In a standard transformer, every single token in the context window must be processed during inference. Naive RAG operates simply by running a basic vector similarity search across an external database, pulling matching text chunks, and stuffing them into the prompt as raw text. This brute-force injection creates severe downstream issues: connecting a vast enterprise codebase or an entire Jira backlog directly to an LLM produces muddled results rather than actionable intelligence.

``` Naive RAG (Context Stuffing): +-------------+ Vector Search +-----------------+ | User Query | --------------------> | Vector Database | +-------------+ +-----------------+ | | | +-----------------------------+ v v (Stuffs raw top-k chunks) +-------------------------------------------------------+ | Prompt: [System] + [100k Tokens of Chunks] + [Query] | +-------------------------------------------------------+ | v (Suffers from "Lost in the Middle" & Token Confusion) +-------------------------------------------------------+ | Standard Transformer (Next-Token Autocomplete Engine) | +-------------------------------------------------------+ ```

[02:18](https://www.youtube.com/watch?v=I367mYw8guI&t=138s) Why More Context Confuses Autocomplete Models

The current push toward agentic AI attempts to bridge these retrieval shortcomings, yet modern workflows often exacerbate the problem by shoving excessive context into prompts. Instead of enhancing the output, longer context windows frequently degrade model performance. At their core, Large Language Models remain statistical next-token prediction engines—massive matrices trained on vast corpuses of human text designed to calculate probabilities rather than synthesize concepts.

Academic benchmarks, including studies on Long-to-RAG and the FRAMES evaluation dataset, demonstrate that feeding an LLM excessive raw context actively harms its performance. The standard transformer architecture inherently lacks dual-tier comprehension: it cannot parse granular, low-level details while simultaneously maintaining an overarching, high-level structural model of a system. When fed an entire codebase or large function libraries, the model struggles to build coherent conceptual clusters across tokens.

- **Token Confusion:** As prompt length scales, token distraction increases, leading to hallucination and poor reasoning over long horizons. - **Next-Token Limitations:** Predicting the next token sequentially prevents models from understanding structural dependencies across disparate data fragments. - **Architectural Omission:** Current models lack native mechanisms to group tokens into semantic, hierarchical clusters.

[04:45](https://www.youtube.com/watch?v=I367mYw8guI&t=285s) The Failure of RAG on Enterprise Data

Ingesting massive enterprise context through standard RAG introduces severe computational and functional bottlenecks. When an LLM evaluates a prompt packed with retrieved context, inference latency spikes drastically because every single newly generated token must calculate attention over every preceding token. Despite this heavy compute cost, the model possesses no true semantic comprehension of the text; it merely performs token-level matching across the prompt.

``` Standard Transformer Context Scaling Cost: - Inference Time: O(N) generation steps where each step attends to Prompt Length (P) - Context Attention: Scales quadratically O(P^2) or linear-attention bounds - Semantic Depth: Surface token matching without hierarchical abstractions ```

Large language models function effectively as a "fancy autocomplete" when handling concise, bounded inputs. Summarizing an isolated 1,000-word newspaper article yields accurate, coherent output. However, attempting to dump disorganized, enterprise-scale data into a RAG pipeline and expecting the system to synthesize actionable business insights causes the architecture to fail completely. This bottleneck is not exclusive to OpenAI's GPT models—it is an intrinsic constraint shared across Google Gemini and Anthropic Claude due to their reliance on foundational transformer decoders.

[06:58](https://www.youtube.com/watch?v=I367mYw8guI&t=418s) Memory Mosaics and the Shift to Internal Memory Modules

To overcome the architectural ceiling of external RAG, modern research is shifting toward internal, persistent memory designs. A prominent example is the "Memory Mosaics" architecture, which evolves historical memory networks and was highlighted in research published at ICLR 2025 by Meta Research. Rather than bolting an external vector database onto the outside of a transformer, Memory Mosaics integrates context directly into the neural network's internal processing graph.

Under this paradigm, every prompt token is routed into a dedicated, parallel internal memory module rather than passed as static prompt context. Each token directly alters the internal calculations of the network alongside the pre-trained weights. This structure provides a path toward near-infinite context retention without degrading next-token generation mechanics. While experimental architectures like Memory Mosaics, HippoRAG, and MemoRAG reflect how biological brains synthesize working memory, no major production-scale foundational model has fully transitioned away from pure transformer decoders yet.

[09:06](https://www.youtube.com/watch?v=I367mYw8guI&t=546s) GraphRAG and Agentic Chunking Strategies

While deep architectural replacements remain in academic pipelines, current production systems address context limits using GraphRAG and agentic decomposition. Both paradigms leverage a simple principle: since LLMs achieve near-perfect synthesis over small documents, enterprise data must be partitioned into tightly bounded, semantically coherent segments before processing.

In GraphRAG, raw enterprise datasets—such as project management spaces like Jira—are converted into structured knowledge graphs. Instead of stuffing an entire backlog into a vector database, the data is partitioned into discrete nodes representing sprint boards, functional epics, and individual tickets. An autonomous agent navigates the graph using standard algorithmic traversal techniques like breadth-first search (BFS) or depth-first search (DFS). The model inspects a single node, extracts the pertinent signals, carries that compressed state forward, and evaluates the adjacent node in context.

``` GraphRAG Iterative Traversal: [Raw Backlog] ---> [Graph: Sprint Nodes & Ticket Edges] | v (BFS / DFS Traversal) +-------------+ | Read Node A | ---> Extract Findings (Small Summary) +-------------+ | | v +------------> +-------------+ | Read Node B | ---> Extract Findings +-------------+ | v Final Output ```

- **Digestible Node Sizing:** Each graph node is scoped to fit well within the model's peak comprehension window without triggering context rot. - **Recursive State Passing:** Intermediate insights are distilled into compact context states passed sequentially to subsequent queries. - **Selective Traversal:** The system executes targeted graph lookups rather than performing wide, fuzzy similarity matches across millions of unrelated vector embeddings.

[12:41](https://www.youtube.com/watch?v=I367mYw8guI&t=761s) The Brittle Reality of Enterprise Agentic Workflows

Translating open-ended business requirements into reliable agentic workflows requires extensive manual engineering. At Snapshot AI, developing executive-facing features—such as the "Raised Flags" module designed to highlight engineering team anomalies—requires deconstructing broad user intents into granular, deterministic inspection paths. Engineers must explicitly model every operational dimension: code quality regressions, process friction, and organizational misalignments. Building this middleware layer between foundational LLMs and enterprise data mirrors orchestrating "infinite McKinsey analysts" manually.

While a 70% or 90% accurate system might serve as an impressive proof of concept (POC), enterprise production deployments demand 100% precision. In industrial settings, failures carry direct financial and safety consequences. For example, field engineers maintaining 100-foot-tall wind turbines frequently work in high-stakes environments using translation systems—such as translating Japanese maintenance manuals into Spanish on-site. If an AI system operating on a multi-million-dollar turbine achieves only 95% accuracy, the 5% margin of error represents unacceptable operational failure.

``` Enterprise AI Gap: +-------------------------------------------------------+ | Proof-of-Concept Accuracy: ~70% - 90% (Standard RAG) | | - Unacceptable for mission-critical infrastructure | +-------------------------------------------------------+ | v (Requires Heavy Agent Engineering) +-------------------------------------------------------+ | Production Enterprise Target: 100% Accuracy | | - Turbines, engineering orgs, medical, legal data | +-------------------------------------------------------+ ```

[16:39](https://www.youtube.com/watch?v=I367mYw8guI&t=999s) Moving Beyond Next-Token Prediction to Conceptual Reasoning

Fixing enterprise AI requires fundamental breakthroughs in conceptual representation and dynamic context ingestion. At the conceptual level, tokenizers completely blind models to semantic equivalences across representation formats. For instance, recent research on phonemes highlights that spelling a concept like "Facebook" in Latin, Cyrillic, or Chinese script yields entirely disjoint token IDs. The LLM processes them as unrelated entities because it lacks an abstraction layer that maps disparate surface tokens to a unified conceptual entity before execution.

``` The Tokenizer Concept Blindspot: "Facebook" (Latin) ---> Token IDs: [ 3452, 8912] --+ "Фейсбук" (Cyrillic) ---> Token IDs: [12903, 44201] --+-> Mapped as completely "脸书" (Chinese) ---> Token IDs: [58219, 21903] --+ different concepts | v Need: Upstream Conceptual Mapping Layer ----------------------+ ```

Second, ingesting dynamic enterprise context should mirror the training process itself rather than relying on ephemeral prompt-stuffing. While consumer chat platforms cannot afford the compute to run pseudo-training loops on every user prompt, enterprise applications with high accuracy requirements can comfortably trade higher inference compute for reliable ingestion. Pure transformer scaling has reached the end of its architectural lifecycle; advancing enterprise autonomy requires moving past next-token autoregression toward internal memory, dynamic reasoning, and true conceptual synthesis.

## Notable quotes > Most AI systems today fake context by stuffing in documents and hoping relevance will emerge. — Kirimgeray Kirimli

> [00:04](https://www.youtube.com/watch?v=I367mYw8guI&t=4s) Every single token in the context window must be processed during inference. — Kirimgeray Kirimli

> [00:04](https://www.youtube.com/watch?v=I367mYw8guI&t=4s) Connecting a vast enterprise codebase or an entire Jira backlog directly to an LLM produces muddled results rather than actionable intelligence. — Kirimgeray Kirimli

> [02:18](https://www.youtube.com/watch?v=I367mYw8guI&t=138s) At their core, Large Language Models remain statistical next-token prediction engines—massive matrices trained on vast corpuses of human text designed to calculate probabilities rather than synthesize concepts. — Kirimgeray Kirimli

> [02:18](https://www.youtube.com/watch?v=I367mYw8guI&t=138s) The standard transformer architecture inherently lacks dual-tier comprehension: it cannot parse granular, low-level details while simultaneously maintaining an overarching, high-level structural model of a system. — Kirimgeray Kirimli

> [04:45](https://www.youtube.com/watch?v=I367mYw8guI&t=285s) Large language models function effectively as a 'fancy autocomplete' when handling concise, bounded inputs. — Kirimgeray Kirimli

> [06:58](https://www.youtube.com/watch?v=I367mYw8guI&t=418s) Rather than bolting an external vector database onto the outside of a transformer, Memory Mosaics integrates context directly into the neural network's internal processing graph. — Kirimgeray Kirimli

> [09:06](https://www.youtube.com/watch?v=I367mYw8guI&t=546s) Since LLMs achieve near-perfect synthesis over small documents, enterprise data must be partitioned into tightly bounded, semantically coherent segments before processing. — Kirimgeray Kirimli

> [12:41](https://www.youtube.com/watch?v=I367mYw8guI&t=761s) Building this middleware layer between foundational LLMs and enterprise data mirrors orchestrating 'infinite McKinsey analysts' manually. — Kirimgeray Kirimli

> [12:41](https://www.youtube.com/watch?v=I367mYw8guI&t=761s) If an AI system operating on a multi-million-dollar turbine achieves only 95% accuracy, the 5% margin of error represents unacceptable operational failure. — Kirimgeray Kirimli

> [16:39](https://www.youtube.com/watch?v=I367mYw8guI&t=999s) Pure transformer scaling has reached the end of its architectural lifecycle; advancing enterprise autonomy requires moving past next-token autoregression toward internal memory, dynamic reasoning, and true conceptual synthesis. — Kirimgeray Kirimli

## Mentioned Kirimgeray Kirimli, Snapshot AI, ChatGPT, Cursor, OpenAI, GPT, Google Gemini, Anthropic Claude, Meta Research, ICLR 2025, Memory Mosaics, HippoRAG, MemoRAG, Raptor, GraphRAG, Long-to-RAG, FRAMES evaluation dataset, Jira, McKinsey

## Video description Kirimgeray Kirimli, CEO, Snapshot AI

Abstract

Everyone’s using RAG, but no one’s asking the real question—does the model actually understand what matters? Most AI systems today fake context by stuffing in documents and hoping relevance will emerge. In this talk, I’ll break down why naive retrieval architectures fall short, and how new approaches like Raptor and Memory Mosaics force us to rethink context as structured, persistent, and goal-aware memory. You’ll see where today’s assumptions collapse—and how to design systems that reason instead of regurgitate. If you care about building intelligent AI, this conversation is overdue.

[Source](https://www.youtube.com/watch?v=I367mYw8guI)

中文简报

Snapshot AI CEO：朴素RAG因上下文限制无法支撑企业级AI 直接将原始文档塞入提示词会导致企业级AI管线失效，因为Transformer本质上是缺乏概念理解的下一词自动补全器，这促使技术必须转向结构化知识图谱与内部神经记忆。 Kirimgeray Kirimli认为，朴素检索增强生成（RAG）和不断扩大的上下文窗口从根本上制约了AI智能，因为原始、非结构化的数据会干扰基于统计的下一词预测引擎。由于Transformer难以应对Token注意力分散且缺乏分层的概念理解能力，直接将海量企业积压数据塞入提示词会导致性能下降和高延迟。虽然当前的生产系统依赖GraphRAG和智能体节点遍历来使数据更易于处理，但诸如Memory Mosaics等下一代解决方案已开始将上下文直接嵌入内部记忆模块。归根结底，要在关键任务领域达到生产级精度，必须超越表层的Token切分与朴素的提示词堆砌，迈向真正的概念推理。朴素 RAG（Naive RAG）依靠向量检索直接向提示词中堆砌原始文本，因缺乏高层概念理解而极易导致上下文混乱，已成为现代 AI 工具的核心架构瓶颈。 (0:04) 由于 Transformer 依赖统计型的下个 Token 预测机制，盲目拉长上下文不仅会因注意力计算引发严重的推理延迟，还会加剧 Token 干扰并劣化推理质量。 (2:18) 像 Memory Mosaics 这样的前沿架构正尝试摆脱外挂向量数据库，转而将上下文直接路由至神经网络内部的并行持久化记忆模块中处理。 (6:58) GraphRAG 将庞杂的企业数据重构为图谱节点，利用智能体通过广度或深度优先搜索（BFS/DFS）进行遍历与状态压缩，充分发挥模型在处理短上下文时的综合优势。 (9:06)

Reading summary

Snapshot AI argues that naive Retrieval-Augmented Generation (RAG) and expanded context windows fail in enterprise settings because transformer decoders function as next-token predictors lacking hierarchical conceptual comprehension. Addressing these shortcomings requires shifting from raw prompt-stuffing toward structured graph traversals and integrated, in-network memory architectures. Context Stuffing Degrades Transformer Performance (2:18) Standard transformer architectures from OpenAI, Anthropic, and Google rely on statistical next-token prediction rather than conceptual synthesis. Academic benchmarks, including Long-to-RAG and the FRAMES dataset, demonstrate that injecting massive raw context directly into prompts causes token distraction, hallucinations, and degraded reasoning over long horizons. Beyond the architecture's inability to reconcile granular details with high-level structural models, packing external context triggers significant computational penalties. Inference latency increases substantially because each newly generated token must calculate attention over every preceding prompt token with quadratic or bounded linear scaling costs, yielding superficial token matching rather than true semantic comprehension. Tokenizer Disconnects Inhibit Conceptual Synthesis (16:39) Foundational models are structurally limited by tokenizers that obscure semantic equivalences across different surface representations. For example, phoneme research shows that writing a brand concept like 'Facebook' in Latin, Cyrillic, or Chinese scripts produces entirely disjoint token IDs. Lacking an upstream abstraction layer to map these representations to a single conceptual entity, the model treats them as unrelated targets. The speaker asserts that pure transformer scaling has reached the end of its architectural lifecycle. Because next-token autoregression cannot infer structural dependencies across fragmented text, enterprise applications require conceptual mapping layers and dynamic context ingestion that mirrors pre-training rather than temporary prompt-stuffing. GraphRAG and Agentic Node Traversal (9:06) Because large language models reliably synthesize small, bounded documents, current production systems address context rot using GraphRAG and agentic decomposition. Enterprise backlogs such as Jira are restructured into knowledge graphs where discrete nodes represent sprint boards, epics, and individual tickets. Autonomous agents navigate this structure using breadth-first or depth-first search algorithms, inspecting tightly scoped nodes that fit within the model's optimal comprehension window. The agent extracts intermediate findings, compresses the state, and carries it forward to adjacent nodes, replacing broad, fuzzy similarity matches across millions of raw vector embeddings with targeted traversals. Deterministic Engineering and High Stakes Precision (12:41) Bridging open-ended enterprise data with foundational models requires extensive manual middleware engineering. Snapshot AI notes that implementing executive tools, such as its 'Raised Flags' anomaly detector, demands manually coding deterministic paths for operational dimensions like code regressions, process friction, and organizational misalignments—a process likened to orchestrating consulting analysts by hand. While proof-of-concept workflows often settle for 70% to 90% accuracy, industrial enterprise deployments require near-total precision. For mission-critical tasks, such as field engineers maintaining 100-foot wind turbines with translated technical manuals, a 5% error margin carries catastrophic physical and financial risks that standard RAG cannot mitigate. Replacing External Retrieval With In-Network Memory (6:58) To transcend the limits of external vector stores, recent research focuses on embedding persistent memory directly inside neural networks. Meta Research's 'Memory Mosaics' architecture, presented at ICLR 2025, alongside experimental systems like HippoRAG and MemoRAG, routes input tokens into parallel internal memory modules instead of passing them as passive prompt context. By enabling context tokens to directly alter the network's internal processing graph alongside pre-trained weights, in-network memory offers a path toward near-infinite context retention without degrading generation mechanics. However, no commercial foundation model has yet fully transitioned from pure transformer decoders to these integrated memory designs. While GraphRAG and deterministic agentic decomposition provide actionable workarounds for enterprise reliability today, achieving scalable intelligence requires moving beyond next-token autoregression toward internal memory modules and conceptual mapping layers. Whether foundation model providers will successfully replace transformer decoders in production remains an open industry challenge.

精炼阅读摘要

Snapshot AI与产业分析指出，依赖简单向量检索的“朴素RAG”与持续堆叠上下文长度的方案已经遭遇底层架构瓶颈；要实现高精度的企业级自主智能，AI系统必须摆脱纯粹的统计学“下一词预测”，向内生记忆网络、图结构切片以及概念层抽象演进。朴素RAG与上下文堆叠的结构性困境 (0:04) 尽管主流模型厂商宣传其上下文窗口已达数百万乃至十亿词元（tokens），但在底层Transformer架构下，推理阶段的每一次计算都必须对上下文中的全部词元进行注意力计算，导致计算成本与推理延迟急剧上升。当前多数AI工具所依赖的朴素检索增强生成（Naive RAG），仅仅通过向量相似度检索将外部文档切片以原始文本形式暴力塞入提示词，极易引发“迷失在中间”（Lost in the Middle）与词元干扰现象。学术界针对Long-to-RAG以及FRAMES基准数据集的研究表明，向模型直接注入过量原始上下文会显著损害其推理性能。大语言模型本质上依然是统计性的下一词预测引擎，缺乏双层理解能力：模型无法在解析微观细节的同时维持对系统宏观架构的全局把握，当面对海量代码库或无序的企业工单时，表面词元匹配无法转化为概念级的深度综合。从外部挂载转向网络内生记忆架构 (6:58) 为打破外挂式RAG的架构天花板，前沿研究正转向构建模型内部的持久记忆模块。Meta Research在ICLR 2025发表的研究展示了一种名为“Memory Mosaics”的架构方案：该方案不再将外部向量数据库作为补丁外挂于Transformer，而是将上下文直接整合到神经网络的内部计算图中，使输入词元直接路由至专用的并行内部记忆模块，与静态预训练权重共同改变网络计算路径。根据讨论，这种类似生物大脑工作记忆机制的设计，能够在不损害下一词生成机制的前提下实现近乎无限的上下文留存。尽管Memory Mosaics、HippoRAG与MemoRAG等架构展示了可行路径，但目前主流生产级基础模型尚未完全脱离传统的纯Transformer解码器结构。 GraphRAG与结构化图遍历的过渡方案 (9:06) 在底层模型架构彻底迭代前，当前工业界主要依靠GraphRAG和智能体切片技术来规避上下文退化。由于基础模型在处理简短、边界清晰的文本时表现最为精准，GraphRAG将非结构化的企业数据（例如Jira项目积压工单）转化为由冲刺看板、功能史诗和独立任务单构成的结构化知识图谱节点。在执行任务时，自主智能体运用广度优先搜索（BFS）或深度优先搜索（DFS）等经典算法遍历图谱。模型单次仅深入审视符合理解窗口的单一节点，提炼关键信号并形成高度压缩的中间状态，再将其传递至相邻节点进行递进评估，以此替代全量向量数据的模糊相似度匹配，确保检索兼具微观精度与上下文结构。概念抽象断层与工业级高精度要求 (12:41) 企业级应用对确定性的严苛要求与概念理解盲区构成了当前落地的核心阻碍。在概念抽象层面，现有词元分析器（Tokenizer）割裂了跨表征的语义同一性，例如“Facebook”在拉丁字母、西里尔字母和中文（脸书）下会被分配为完全无关的词元ID，使模型无法在底层将异构表达映射至统合的实体概念上。在工程实践方面，Snapshot AI在研发面向管理层的“异常标旗”（Raised Flags）功能时发现，将宽泛的业务意图转化为可靠的智能体工作流需要耗费大量人力去手工编排中介逻辑。概念验证（POC）阶段通常能够容忍70%到90%的准确率，但在工业生产环境——例如百英尺高的风力涡轮机现场进行多语言维护手册即时翻译——5%的失误率足以造成严重的运营与安全事故。为此，讨论主张企业级场景应突破传统的提示词注入，通过在推理端引入类似微调训练的动态计算回路，以更高的推理算力换取绝对的输出确定性。该来源的核心结论在于：单纯依靠参数规模与长提示词注入的Transformer演进路线已达极限。企业级自主智能若要突破概念验证阶段的脆性，必须从底层的词元自回归逻辑，向内生持久记忆网络、结构化知识图谱与概念级抽象映射彻底转型。

Detailed notes — zh

## Background 该视频由 Ai4 频道于 2025 年 8 月 29 日发布在 YouTube 平台上，主讲人为 Snapshot AI 首席执行官 Kirimgeray Kirimli。视频简介指出，当前多数 AI 系统仅通过机械塞入文档来“伪造上下文”，本场演讲旨在剖析朴素 RAG 架构的局限，并探讨如何利用 Raptor 与 Memory Mosaics 等方法将上下文重构为结构化、持久化且具备目标意识的记忆系统。 ## TL;DR 演讲者指出，基于 Transformer 的大模型本质上仍是“高级自动补全”，盲目扩大上下文窗口反而会引入更多噪音并损害推理质量。针对企业级复杂数据，现有方案主要通过 Graph RAG 与智能体（Agentic）递归遍历将庞大数据拆解为可消化的语义节点，但这极度依赖人工对业务逻辑的深度拆解。由于工业与关键企业场景对准确率有着近乎 100% 的极致苛求，仅靠词汇表层的模式匹配已无法满足需求。行业必须超越传统外挂式 RAG，转向具备动态信息摄取与原生记忆能力的架构，而 Transformer 架构本身也已步入生命周期的末期。 ## Atmosphere & style 整场演讲呈现出冷静、犀利且极具反思性的技术批判基调，主讲人以一线开发者的务实视角无情戳破了行业内关于“超长上下文窗口”的营销泡沫。通过将高管决策困境、100 英尺高空风电机维修的生死考验与底层 Tokenizer 分词缺陷串联在一起，叙事层层深入，既充满对企业级落地严苛现实的清醒洞察，又透露出呼唤底层架构范式转移的紧迫感与前瞻性。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=I367mYw8guI&t=4s) 现有 AI 系统与 RAG 架构的核心瓶颈在于缺乏对概念的“高层次理解”，盲目增加上下文长度往往适得其反，导致模型更加困惑。 - [03:12](https://www.youtube.com/watch?v=I367mYw8guI&t=192s) Transformer 底层仍是基于概率的“下一个 Token 预测器”（高级自动补全），处理海量上下文时推理成本与计算延迟剧增，且极易引入语义噪音。 - [07:03](https://www.youtube.com/watch?v=I367mYw8guI&t=423s) Memory Mosaics 等内建记忆架构将上下文转化为神经网络内部的并行计算模块，超越了传统外挂式 RAG 粗暴拼接文本的局限，更贴近人脑长期记忆机制。 - [09:06](https://www.youtube.com/watch?v=I367mYw8guI&t=546s) 在底层架构革新前，Graph RAG 与 Agentic 递归检索通过将多维企业数据拆解为图节点并增量遍历，有效解决了长上下文消化不良的问题。 - [12:44](https://www.youtube.com/watch?v=I367mYw8guI&t=764s) 当前构建 Agentic AI 仍高度依赖人工设计，开发者必须像“拥有无限麦肯锡分析师”一样逐层穷尽业务逻辑，才能将模糊需求转化为智能体的具体动作。 - [14:42](https://www.youtube.com/watch?v=I367mYw8guI&t=882s) 企业级工业场景与消费级 AI 的容错率存在本质差异，在涉及高价值资产与人身安全的关键任务中，95% 的准确率依然不可接受，必须追求 100% 的确定性。 - [16:39](https://www.youtube.com/watch?v=I367mYw8guI&t=999s) Transformer 缺乏跨表征的高层概念抽象能力，多语种或不同字形书写的同一实体会被分词器割裂为独立 Token，本质上只是在做字符模式匹配而非概念理解。 - [18:27](https://www.youtube.com/watch?v=I367mYw8guI&t=1107s) 现有的上下文拼接机制无法让新知识被深度同化，未来突破需要引入类似模型训练的动态摄取过程，而非盲目去卷数十亿超长上下文窗口。 - [18:27](https://www.youtube.com/watch?v=I367mYw8guI&t=1107s) 基于标准 Transformer 的架构演进已接近其生命周期末期，解决企业级复杂推理难题必须依赖具备结构化记忆组织与动态认知的新范式。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=I367mYw8guI&t=4s) 现有 AI 模型的上下文困境与 RAG 的核心瓶颈无论是使用 ChatGPT 进行深度资料调研，还是在 Cursor 中借助 AI 编写代码，当下几乎所有主流 AI 应用在底层都高度依赖检索增强生成（RAG）。然而，这种对 RAG 的普遍依赖恰恰构成了当今 AI 系统的最大局限。即便大家对最新一代前沿模型（例如 GPT-5 级别的演进）抱有极高期待，其实际表现却往往未能实现预期的跨越式突破。这背后的核心症结不仅在于 RAG 机制本身的粗糙，更在于现有模型普遍缺乏对复杂概念更深维度的“高层次理解”（higher understanding of a concept）。目前行业内充斥着宣称支持海量甚至“数十亿 Token”（billion token context window）超长上下文窗口的模型，但从底层 Transformer 架构来看，每次输入时模型依然必须机械地逐一处理每一个 Token。典型的传统 RAG 流程往往极其朴素：系统先在向量数据库中进行一次简单的向量检索，随后极少做深层结构化处理，就直接把捞出来的文档当作普通的 Prompt 塞给模型。这种做法在真实工业场景中弊端尽显。以企业研发场景为例，如果开发者将整个大型代码库或者包含成百上千任务的 Jira Backlog 全量灌入 ChatGPT，期望它指出系统架构中的隐患或具体缺陷，最终往往只能得到令人失望的结果。实践证明，当前很多系统尝试通过拼命增加上下文来改善表现，但结果往往适得其反——"传入的上下文越长，模型反而变得越困惑"。 --- ### [03:12](https://www.youtube.com/watch?v=I367mYw8guI&t=192s) Transformer 架构的本质：“高级自动补全”与长上下文衰减归根结底，现有主流大语言模型的本质仍然只是一个在人类全量文本矩阵上训练出来的“下一个 Token 预测器”。面对用户输入的整段代码或大量函数定义，模型只是在概率空间中猜测后续输出。正因如此，无论厂商如何宣传其超长上下文容量，无节制地堆砌信息最终只会大幅增加模型的噪音干扰。包括 Long-to-RAG 以及 FRAMES 等在内的多项学术研究均已证实：在很多复杂任务中，盲目塞入更多上下文实际上正在损害模型的输出质量与推理准确性。从计算过程来看，当海量上下文被强行喂给大模型时，推理延迟（inference time）会呈线性乃至二次方急剧上升，且生成的每一个新 Token 都必须回溯依赖前面所有 Token 的注意力计算。更致命的是，模型对这些词汇缺乏真正的概念理解与拓扑归纳能力；它或许能在字面上匹配某些词项，却缺乏一种能将 Token 聚类为高层语义模块（clusters）的底层架构机制。从本质上讲，大模型目前更像是一个极其精致的“高级自动补全”（fancy autocomplete）。如果仅给它一篇一千字左右的新闻报道并要求做摘要，它几乎每次都能完成得无懈可击；但一旦面对庞杂的企业级多维数据，只是简单粗暴地将其塞进现有的 RAG 应用中并奢望 AI 自己理清脉络，系统便会彻底崩溃。这一瓶颈根植于所有基于标准 Transformer 构建的模型（包括 ChatGPT、Google Gemini 和 Claude 等）之中。 --- ### [07:03](https://www.youtube.com/watch?v=I367mYw8guI&t=423s) 超越传统 RAG：Memory Mosaics 与网络内建记忆架构为了从根本上打破这一桎梏，学术界与工业界正在探索超越传统外挂式 RAG 的全新架构，其中最引人注目的方向之一便是 **Memory Mosaics（记忆马赛克）**。该思路最早脱胎于 Memory Networks（记忆网络），并在 Meta Research 发表的一篇被 ICLR 2025 收录的研究论文中得到了系统性阐述。 ```mermaid flowchart TD accTitle: 传统外挂 RAG 与 Memory Mosaics 架构对比 subgraph Traditional["传统外挂 RAG 模式"] A1["外部上下文数据"] -->|向量检索召回| A2["文本拼接进 Prompt"] A2 --> A3["Transformer 统一顺序处理<br/>(推理变慢、语义缺乏聚类)"] end subgraph MemoryMosaics["Memory Mosaics 架构"] B1["Prompt 输入 Token"] --> B2["网络内建记忆模块<br/>(Parallel Memory Module)"] B1 --> B3["已训练神经网络骨干"] B2 <-->|并行双向参与计算| B3 B3 --> B4["结构化与持久化语义输出"] end ``` 传统 RAG 仅仅是生硬地“螺栓固定”（bolted on）在 Transformer 外部的补丁；而在 Memory Mosaics 体系中，上下文直接成为了神经网络内部的原生组成部分： - **内建模块化处理**：输入 Prompt 中的每一个 Token 都会在网络内部的独立并行模块中进行专门计算，而非作为外部文本生硬填入。 - **全局参数级协同**：Prompt 中的每个 Token 都能直接参与下一步输出的整体神经网络计算。 - **原生超长记忆**：该结构拥有与主网络并行的动态记忆模块，使得长程上下文能够被真正结构化持久存储。尽管类似于 Memory Mosaics、HippoRAG 或 MemoRAG 等内建记忆架构更贴近人类大脑处理长期记忆的方式，代表了后 Transformer 时代的发展方向，但目前主流的商业基础模型尚无一例原生采用此类架构。 --- ### [09:06](https://www.youtube.com/watch?v=I367mYw8guI&t=546s) 现阶段的破局思路：Graph RAG 与智能体（Agentic）分块递归检索在底层基础模型架构尚未发生范式转移之前，业界应对长上下文与企业数据困境的主要方案集中在 **Graph RAG（图谱 RAG）** 与 **智能体（Agentic）递归检索** 上。这两种方法的核心逻辑高度一致：既然模型在处理有限篇幅（如 1000 字以内）的高相关性文本时表现极佳，那么系统就应当主动将庞大数据集拆解为大模型易于消化且语义关联紧密的小块（digestible and relevant chunks）。 ```mermaid flowchart LR accTitle: Graph RAG 与 Agentic 递归处理流程 subgraph Ingestion["数据结构化阶段"] D["企业级多维数据<br/>(例如 Jira Backlog)"] -->|分块与拓扑关联| G["图谱节点网络<br/>(Sprint/项目/关联任务)"] end subgraph Traversal["智能体遍历与提炼"] G --> N1["节点 A 抽取关键上下文"] N1 -->|携带上一节点提炼信息| N2["节点 B 深度遍历计算"] N2 -->|递归缩小搜索空间| N3["精准最终答案"] end ``` 以企业内部的 Jira 项目管理数据为例，Graph RAG 与 Agentic 方法的具体实施逻辑包括： - **图节点构建**：将不同 Sprint 迭代看板、史诗任务（Epics）和子项目拆分成独立的图节点（Graph Nodes），建立节点间的关联边，而非把整个 Backlog 粗暴扔进向量库。 - **图遍历检索**：AI Agent 利用广度优先搜索（BFS）、深度优先搜索（DFS）等标准图遍历算法逐个访问节点。 - **增量信息流转**：Agent 在每个节点只处理 LLM 负荷范围内的文本量，提炼出核心信息后，带着上一阶段的精简结论继续跳转至下一节点进行递归分析。 - **自适应动态查询**：Agentic 架构由 AI 动态决策“下一步应该深入调取哪一部分数据集”，通过多轮小步快跑的循环交互，不断收敛信息，从而在不撑爆上下文的前提下得出高精度结论。 ### [12:44](https://www.youtube.com/watch?v=I367mYw8guI&t=764s) Agentic AI 的现实困境：必须像“拥有无限麦肯锡分析师”一样手动拆解业务当前构建 Agentic AI（智能体 AI）的方法本质上仍然极其依赖人工设计。AI 目前还不具备自主理解复杂业务目标的能力；要构建一个行之有效的 Agentic 架构，开发者必须先从根本上明确企业用户真正需要解答的问题。在 Snapshot AI，团队面临的核心挑战就在于此。例如，系统内部设计了一个名为“红旗预警”（Raised Flags）的模块，其目标是为主管工程研发的高管（engineering executive）高亮指出所有他们可能需要关心的潜在风险。然而，“高管需要知道什么”是一个极其模糊的需求。为了将其转化为 Agentic AI 能够执行的操作，团队必须把业务逻辑逐层深挖、穷尽列举：工程管理者可能想了解团队协作中的人际摩擦、流程执行中的卡点，或是底层代码库（codebase）中潜藏的技术债。开发者必须假设自己“拥有无限的预算和无限的麦肯锡分析师”（"infinite budget and infinite McKinsey analysts"），预先人工梳理出所有分析步骤，然后在基座模型与企业数据之间构建转换层，把这套流程严密映射为智能体的一系列具体动作。这种高度定制化的方案在 Snapshot AI 这样聚焦特定垂类场景的产品中确实奏效，但也凸显了当前企业级 AI 落地的沉重负担。 ```mermaid flowchart TD accTitle: Agentic AI 业务逻辑拆解与落地流程 subgraph ProblemAnalysis["1. 业务问题拆解 (麦肯锡式人工梳理)"] A["高管宏观诉求: 风险预警"] --> B["团队协同隐患"] A --> C["研发流程阻塞"] A --> D["代码库架构缺陷"] end subgraph AgentTranslation["2. 中间层动作转化"] B & C & D --> E["Agentic 调度与映射逻辑"] end subgraph Execution["3. 系统执行与数据交互"] E --> F["基座大模型 (Foundational Model)"] E --> G["企业数据源 (Enterprise Data)"] end ``` - 企业应用若只做到 70% 的准确率，或许能做一个很酷的概念验证（PoC），但绝不可能达到生产就绪（production ready）的标准。 - 企业级场景要求系统具备持续一致的稳定性与确定性，容不得模糊推理。 --- ### [14:42](https://www.youtube.com/watch?v=I367mYw8guI&t=882s) 企业级 AI 的准确率底线：100 英尺风机上的生死考验针对企业级 AI 对准确率的极端苛求，演讲者分享了另一家智能体应用公司的真实交付案例。该客户是一家拥有大量大型风力发电机组（windmills）的企业，其风机设备的使用维护手册全部为日文编写，而现场负责抢修运维的工程师普遍讲西班牙语。当技术人员顶着恶劣环境爬上 100 英尺（约 30 米）高的风电机顶端排除机械故障时，唯一的依赖就是手持终端上的企业级 AI 应用。在这种场景下，95% 或 99% 的准确率不仅不够用，甚至可能带来毁灭性的灾难。工程师身处 100 英尺高空，一旦 AI 给出错误的诊断或操作指导，就可能直接损坏价值数百万美元的风电设备，甚至威胁人身安全。演讲者强调：“在那种高空中，你必须拥有 100% 的准确率。”无论是维修风力涡轮机、清洗企业业务数据，还是梳理工程研发组织架构，开发者在设计企业级智能体系统时都必须对每一个边缘用例（use case）保持极致谨慎。 - 消费级 AI 的容错率与工业/企业级 AI 存在本质不同，关键任务场景对幻觉与偏差是零容忍的。 - 在涉及高价值资产与物理安全的运维环境中，95% 的模型表现被明确定义为“非常糟糕”（"very bad"）。 --- ### [16:39](https://www.youtube.com/watch?v=I367mYw8guI&t=999s) Transformer 的根本局限：缺失高层概念抽象与跨表征认知要彻底解决准确率和理解力的问题，必须从底层架构反思现有技术的缺陷。现有的 Transformer 架构完全聚焦于“预测下一个 Token”（"guessing the next token"），其底层并不具备真正的高层抽象思维（higher-level thinking）与概念聚类能力。演讲者以近期关于音素（phonemes）表征的前沿论文为例：如果分别用西里尔字母（Cyrillic）、拉丁字母（Latin）和中文汉字来拼写同一个品牌名称（例如 "Facebook"），对人类而言，无论其书写形式如何变化，发音和背后所指代的事物都是完全一致的概念；但在大语言模型中，经由分词器（Tokenizer）切分后，这三种形式会被编码为完全不同的 Tokens，被模型视作毫无关联的独立实体。这揭示了当前架构级的一大空白——大模型缺乏一个能够独立于文字表象、统一处理高维概念与深层语义抽象的“高阶逻辑模块”。 - 现存 LLM 无法在符号底层将多语种、多表征的同义概念自发归一化，本质上是在做字符模式匹配而非概念理解。 - 仅靠在词汇（words/tokens）层面做关联运算，模型无法建立起理解复杂世界所必需的高阶逻辑思想体系。 --- ### [18:27](https://www.youtube.com/watch?v=I367mYw8guI&t=1107s) 呼唤类似训练的动态摄取机制与 Transformer 时代的黄昏除了概念抽象的缺失，现行上下文传递（context passing）机制也是巨大的瓶颈。基座模型之所以聪明，完全归功于其庞大的预训练数据；而用户在运行时传入的上下文信息，对模型而言是“外来的陌生异物”（foreign）。现有的直接拼接方式极为受限，真正理想的架构应当在传入海量代码库或企业资料时，触发一种“类似模型训练的动态摄取过程”（training-like process），使新知识能够深度同化到模型的推理结构中。虽然对于 ChatGPT、Claude 这类面向大众的商业对话产品而言，若每次交互都执行类似训练的深度同化，推理成本与能耗将让企业迅速破产；但对于追求极致准确的企业级场景，企业完全愿意为 100% 的确定性买单。演讲者断言，未来的突破绝不是盲目去卷类似新版 Gemini 那样 20 亿（2 billion）超大上下文窗口，而是需要模型具备动态信息摄取、结构化记忆组织与真正推理的能力。演讲在最后给出了极具冲击力的预判：“我们似乎已经走到了 Transformer 架构生命周期的最末期（'the very end of the lifetime of the transformer architecture'），我们需要全新的范式。” - 通用大模型设计初衷是服务日常交互（用户甚至可以在上面随意分享个人医疗档案），但其浅层的上下文机制无法胜任庞大工业级数据的深层推理。 - 堆砌 Context Window 长度只是治标不治本的工程妥协，无法替代对信息进行结构化动态组织与深度推理的新架构演进。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=I367mYw8guI&t=4s) 传入的上下文越长，模型反而变得越困惑 — Kirimgeray Kirimli > [03:12](https://www.youtube.com/watch?v=I367mYw8guI&t=192s) 高级自动补全 — Kirimgeray Kirimli > [07:03](https://www.youtube.com/watch?v=I367mYw8guI&t=423s) 螺栓固定 — Kirimgeray Kirimli > [12:44](https://www.youtube.com/watch?v=I367mYw8guI&t=764s) 拥有无限的预算和无限的麦肯锡分析师 — Kirimgeray Kirimli > [14:42](https://www.youtube.com/watch?v=I367mYw8guI&t=882s) 在那种高空中，你必须拥有 100% 的准确率 — Kirimgeray Kirimli > [14:42](https://www.youtube.com/watch?v=I367mYw8guI&t=882s) 非常糟糕 — Kirimgeray Kirimli > [16:39](https://www.youtube.com/watch?v=I367mYw8guI&t=999s) 预测下一个 Token — Kirimgeray Kirimli > [18:27](https://www.youtube.com/watch?v=I367mYw8guI&t=1107s) 类似模型训练的动态摄取过程 — Kirimgeray Kirimli > [18:27](https://www.youtube.com/watch?v=I367mYw8guI&t=1107s) 我们似乎已经走到了 Transformer 架构生命周期的最末期，我们需要全新的范式 — Kirimgeray Kirimli ## Mentioned Kirimgeray Kirimli, Snapshot AI, Ai4, ChatGPT, Cursor, GPT-5, Transformer, Jira, Long-to-RAG, FRAMES, Google Gemini, Claude, Memory Mosaics, Memory Networks, Meta Research, ICLR 2025, HippoRAG, MemoRAG, Graph RAG, McKinsey ## Video description Kirimgeray Kirimli, CEO, Snapshot AI Abstract Everyone’s using RAG, but no one’s asking the real question—does the model actually understand what matters? Most AI systems today fake context by stuffing in documents and hoping relevance will emerge. In this talk, I’ll break down why naive retrieval architectures fall short, and how new approaches like Raptor and Memory Mosaics force us to rethink context as structured, persistent, and goal-aware memory. You’ll see where today’s assumptions collapse—and how to design systems that reason instead of regurgitate. If you care about building intelligent AI, this conversation is overdue. [Source](https://www.youtube.com/watch?v=I367mYw8guI)

Source quotes

[0:04] Kirimgeray Kirimli: 传入的上下文越长，模型反而变得越困惑 [3:12] Kirimgeray Kirimli: 高级自动补全 [7:03] Kirimgeray Kirimli: 螺栓固定 [12:44] Kirimgeray Kirimli: 拥有无限的预算和无限的麦肯锡分析师 [14:42] Kirimgeray Kirimli: 在那种高空中，你必须拥有 100% 的准确率 [14:42] Kirimgeray Kirimli: 非常糟糕 [16:39] Kirimgeray Kirimli: 预测下一个 Token [18:27] Kirimgeray Kirimli: 类似模型训练的动态摄取过程 [18:27] Kirimgeray Kirimli: 我们似乎已经走到了 Transformer 架构生命周期的最末期，我们需要全新的范式

Source quotes

Kirimgeray Kirimli: Most AI systems today fake context by stuffing in documents and hoping relevance will emerge. [0:04] Kirimgeray Kirimli: Every single token in the context window must be processed during inference. [0:04] Kirimgeray Kirimli: Connecting a vast enterprise codebase or an entire Jira backlog directly to an LLM produces muddled results rather than actionable intelligence. [2:18] Kirimgeray Kirimli: At their core, Large Language Models remain statistical next-token prediction engines—massive matrices trained on vast corpuses of human text designed to calculate probabilities rather than synthesize concepts. [2:18] Kirimgeray Kirimli: The standard transformer architecture inherently lacks dual-tier comprehension: it cannot parse granular, low-level details while simultaneously maintaining an overarching, high-level structural model of a system. [4:45] Kirimgeray Kirimli: Large language models function effectively as a 'fancy autocomplete' when handling concise, bounded inputs. [6:58] Kirimgeray Kirimli: Rather than bolting an external vector database onto the outside of a transformer, Memory Mosaics integrates context directly into the neural network's internal processing graph. [9:06] Kirimgeray Kirimli: Since LLMs achieve near-perfect synthesis over small documents, enterprise data must be partitioned into tightly bounded, semantically coherent segments before processing. [12:41] Kirimgeray Kirimli: Building this middleware layer between foundational LLMs and enterprise data mirrors orchestrating 'infinite McKinsey analysts' manually. [12:41] Kirimgeray Kirimli: If an AI system operating on a multi-million-dollar turbine achieves only 95% accuracy, the 5% margin of error represents unacceptable operational failure. [16:39] Kirimgeray Kirimli: Pure transformer scaling has reached the end of its architectural lifecycle; advancing enterprise autonomy requires moving past next-token autoregression toward internal memory, dynamic reasoning, and true conceptual synthesis.

Original transcript

[0:04] Um, thank you very much for coming. So [0:07] today I want to talk about context and [0:10] the context problem in a lot of the [0:13] existing AI models. [0:16] So I'm imagining that pretty much uh all [0:19] of you have used some form of AI by now [0:22] and a lot of the problems that we face [0:25] with existing AI models actually have a [0:28] lot to do with rag because like any AI [0:31] application that you use today it relies [0:34] heavily on rag whether you're using chat [0:39] GPT to do research or whether you're [0:42] using cursor to write code it relies on [0:46] rag [0:47] and that's where the issue is and that's [0:49] where the limitation is. [0:52] Now, if you actually think about um [0:56] improving the models, if you look at the [0:58] chat GPT's most recent chat GPT5, [1:01] the reason why it doesn't have the the [1:05] improvements that we were all hoping [1:07] for, the reason why it's not as the [1:09] reason why it's not as smart as we were [1:11] hoping for is actually partially because [1:13] of rag and partially because it's [1:15] actually lacking what we could call um [1:18] the higher understanding of a concept. [1:21] And that's what I wanted to talk about [1:22] today. [1:25] Now, um the biggest problem that we're [1:27] facing with these existing models is [1:29] that they actually uh struggle with the [1:32] context handling. You could there are a [1:35] lot of these uh new models out there [1:37] that they claim [1:39] to have a billion token context window. [1:42] But the reality is if you actually look [1:44] at the transformer architecture, [1:47] the transformer architecture has to like [1:50] process every single one of these tokens [1:52] every time you um pass. And the way rag [1:57] works is it'll do a [2:00] simple in most cases it'll do a vector [2:04] search on a database and then it might [2:08] may do something uh smart but rarely [2:11] ever and then it'll just pass that as if [2:14] you're writing a standard prompt. [2:18] Now that's problematic in many ways but [2:20] uh it also means that if you want an AI [2:24] agent or an AI application or let's say [2:26] uh you connect your codebase or you [2:28] connect your um entire Jira backlog to [2:33] chat GPT [2:35] you would hope to get some good answers. [2:38] You would hope to be like okay this is [2:40] my codebase tell me what's wrong. [2:43] However the reality is it's far from [2:45] that. And in order to be able to do that [2:48] at the moment, in order to be able to [2:51] understand [2:53] um better, there are a lot of things [2:56] that we do like the agentic AI. However, [3:00] at the moment, all we're doing is that [3:02] we pass a lot of context and instead of [3:05] helping it, the longer the context we're [3:08] passing, the more confused the model [3:10] gets. [3:12] And the reason that uh for that is [3:14] because all of these models essentially [3:17] are built um just as like guessing the [3:21] next token and nothing else. They are [3:23] there to guess the next token based on [3:26] what you're giving it. It's a massive [3:28] matrix that has been trained on every [3:31] single text that's available to human. [3:34] And then whatever you do, you just give [3:37] it a [3:40] uh you just give it a set uh of text, [3:43] whether that's a code base or whether [3:44] that's a bunch of functions. And then [3:47] you're asking it to guess the next [3:49] token. And the more you give, the more [3:52] information that you give and uh [3:54] regardless of whatever they claim about [3:56] the context, you're actually confusing [3:59] it more. [4:01] The way to get around this at the moment [4:06] and this is um the way to get around [4:09] this at the moment is by using some of [4:11] the agentic uh approaches which we'll [4:13] discuss later but to go deeper [4:17] on this like there are a lot of uh [4:19] studies including long to rag and frames [4:23] where they show that giving more context [4:25] is actually hurting in many ways and if [4:28] you look at uh The way the transformer [4:32] architecture is built, it actually lacks [4:35] the kind of understanding that we need [4:38] that would be able to extract the higher [4:40] level understanding as well as a um [4:42] lower level understanding [4:45] to give another example. For instance, [4:47] if we actually were to walk through how [4:51] um like step by step, how would be how [4:56] would you ingest the um a large piece of [4:59] ch um information large context into uh [5:03] an LLM? The problem that you're going to [5:05] have there is that this LLM is going to [5:09] um process every single token that you [5:12] give it and then first of all the [5:15] inference time is going to be extremely [5:16] slow and every single token that it [5:20] generates as a response is going to have [5:23] to depend on the previous tokens. But [5:26] here's the problem that you're facing [5:28] there. It doesn't actually have an [5:30] understanding of these words. It might [5:33] be able to match some of the words or [5:35] some of the um the tokens together, but [5:38] it still lacks that higher level [5:40] architectural um piece where you need to [5:45] be able to you need a model. You need an [5:49] architecture that's able to tag these [5:52] tokens more than just one after another [5:54] but in clusters. [5:58] And that's where we're uh missing. And [6:00] that's why uh rag applications at the [6:03] moment fail in addition to the context [6:05] problem we're facing. [6:09] As I've said, it's a fancy autocomplete. [6:13] It'll be good if I give a single [6:16] newspaper article [6:18] and if I ask the AI to summarize that [6:20] for me or a limited piece of text, it [6:24] can do a great job. But if I have an [6:26] enterprise data and I just shove it uh [6:30] to some rack application and hope that [6:32] the AI will figure it out, that's where [6:34] it fails miserably. And this is [6:38] something that a lot of these uh AI [6:40] companies have been trying to address, [6:42] whether that's chat GPT, Google Gemini [6:46] uh or [6:48] um Claude. [6:49] But again, it goes back to the GPT [6:52] architecture that they're using. that is [6:54] the main limiting factor there. [6:58] To address these um there are several [7:00] different approaches that people are [7:02] trying. [7:03] One of my favorites actually is uh the [7:05] memory mosaics. Memory mosaics is [7:08] something that uh have been developed [7:10] for a while. It started with the memory [7:12] networks idea and one of the most recent [7:16] papers from meta research was actually [7:18] about memory mosaics that was um [7:21] published in ICLR 2025. [7:24] And the idea is that it's still a neural [7:27] network, but rather than using rag, [7:31] which is essentially something that they [7:33] bolted on to the transformer [7:35] architecture, [7:37] that context is actually a part of the [7:40] network. So every time for every single [7:43] token that comes in your initial prompt, [7:47] it gets processed in a separate module [7:51] within the network rather than being [7:54] something outside the network. [7:56] Therefore, every single pro um token in [7:59] your um prompt becomes a part of the [8:04] overall calculation [8:06] for what comes next or what's the output [8:08] and it's an entirely parallel module to [8:11] the trained portion of the network that [8:14] allows you to have a much longer [8:18] context window and therefore that [8:20] context actually gets stored. that I [8:23] believe is the future. Might not [8:25] necessarily be the um memory mosaics or [8:28] hippo rag or uh memorag whatever the [8:31] name is but an architecture that [8:33] actually has [8:36] um the prompt processing and every [8:39] single token processing within the [8:41] network rather than being outside. I [8:43] feel like that seems to be the way like [8:45] how our brains work and that seems to be [8:48] the future for um the architectures [8:51] beyond the transformer models. [8:54] But obviously we're probably quite far [8:56] from there and there is no there's not a [8:58] single transformer there's not a single [9:01] foundational model out there um that [9:04] actually uses an architecture like that. [9:06] for the existing solutions. Uh a lot of [9:09] people try to do something called a [9:11] graph frag [9:13] um and also the general agentic [9:17] approaches. What these two approaches [9:19] have in common is that as I mentioned [9:21] earlier, if I was to give AI a limited [9:25] piece of text and ask it to summarize or [9:27] tell me whatever it is in that say like [9:30] thousandword document, AI will almost [9:34] every time give me the perfect answers [9:37] about whatever I'm asking for because [9:39] it's a long enough text that I can [9:41] handle that the AI can handle. And the [9:44] other two approaches the graph rag or [9:47] the agentic approaches actually take [9:48] that principle [9:50] on the graph rag side what they do is [9:53] that if you're dealing with an [9:55] enterprise level data the first thing [9:57] you do is to create a graph you try to [10:02] put that data you split that data into [10:04] digestible chunks [10:07] digestible and relevant chunks and then [10:10] you make you draw a graph out of it. So [10:13] whenever you need to process that [10:15] enterprise data, let's take the example [10:17] of Jira tickets. Uh I'm assuming a lot [10:21] of people are familiar with um Jira or [10:23] some form of a project management here. [10:26] The idea is that you would actually take [10:28] um different sprint boards and you uh [10:30] put them into different chunks and [10:32] different projects and each one of these [10:34] would be um a different graph node. Then [10:38] rather than passing your entire [10:41] Jiraa backlog into um AI or putting it [10:44] on a vector database, you'd have your AI [10:48] agents going one by one um processing [10:52] each single graph node and then going [10:54] down or could use uh any of the standard [10:58] uh bread first search or depth first [11:00] search or any um graph traversal [11:03] algorithm. And the idea would be that [11:05] you would look at one um one graph node. [11:08] You process that and you want to make [11:10] sure that that's actually a amount that [11:14] you can process and it's not too big and [11:16] that it's big enough for the LLM to [11:17] handle. You get what you want from that [11:19] node. Then you pass it on to the next [11:23] node. You process that next node with [11:26] the information from that previous node [11:29] and so on. And this way you can go [11:33] through the entire data set only by [11:36] looking at what you need while keeping [11:39] the information from the previous [11:40] graphs. That's essentially graph rag. On [11:44] the agentic side, again the same idea [11:48] potentially uh in certain applications [11:50] slightly more smarter. But what you do [11:53] is you would ask the AI this is my data [11:57] set which one do I pick and then you [12:01] pass that data set you process it the [12:03] prompt then you ask again and you ask [12:06] again similar to the graph uh approach [12:09] but the principle is the same. You want [12:12] to make sure that you give a relevant [12:15] enough chunk of your data set to get an [12:18] accurate answer. Then you follow up. You [12:22] get the first answer which is hopefully [12:23] a much smaller uh chunk of information [12:26] that the initial um pro uh context you [12:29] gave. Then you go on to the next step. [12:32] You search the data further and do that [12:35] recursively until you get an answer. [12:39] Um [12:41] there's a however a problem with that [12:44] approach. [12:45] Once you have [12:48] uh the agentic approach as I've [12:49] mentioned it's very manual [12:52] AI is still not able to um [12:55] understand on its own. If I wanted to [12:57] build an agentic approach then I need to [13:00] identify the problems. I this is also [13:03] like uh what we do at snapshot as well [13:07] for um the biggest chunk uh of the work [13:10] that we do is first we need to [13:12] understand what is the question that the [13:16] enterprise user needs to be answered. [13:19] Once you understand that question [13:22] then you have to really dig down. For [13:25] instance, we have a module that would [13:27] that we call raised flags. That raised [13:29] flags module uh is to highlight any [13:34] information that an engineering [13:35] executive might want to know about. [13:38] However, that's a very vague question. [13:41] And in order to be able to convert that, [13:44] translate that into an agentic AI, we [13:47] actually have to go deep uh deep down [13:50] and actually ask every single thing. [13:53] What would an engineer and leader want [13:54] to know? They might want to know about [13:57] problems in their teams. They might want [13:59] to know about um issues uh in processes. [14:03] They might want to know about problems [14:05] in codebase. And you have to define [14:07] every single one of these as if you [14:09] would do it manually as if you had [14:11] infinite budget and infinite Mckenzie [14:13] analysts and as if they could do all of [14:15] these. Then you need to go back to the [14:19] other side. The part that sits between [14:22] your foundational model and your data [14:25] and you need to translate these [14:27] processes into agentic AI actions. [14:32] I mean it works in our product but our [14:35] product is a um very niche very um small [14:39] application very focused product. [14:42] However, for every single one of these [14:44] enterprise AI applications, you have to [14:46] do the same because otherwise you might [14:49] get a enterprise AI product that's 70% [14:53] accurate. It's a great PC, but it's not [14:56] production ready. Because at Enterprise [14:58] AI, you need it to work all the time. [15:02] Um, one of the examples that I was [15:05] thinking about was we've we also have [15:08] another company that does a lot of [15:10] Gentic AI. um applications. And one of [15:13] our clients was a big um [15:17] company that has a lot of windmills. And [15:21] apparently the windmills manuals were in [15:23] Japanese and a lot of the engineers were [15:26] Spanish speakers. And they would go up [15:28] the uh these uh windmills, god knows [15:31] how, like 100 ft tall, and then they [15:34] would have to, you know, fix the [15:35] windmill. [15:37] And they go up there, they open the [15:40] Enterprise AI app and ask a question [15:43] about whatever technical problem they [15:46] have. And if your Enterprise AI app is [15:49] 95% accurate, [15:52] that's very bad. It has to be 100% [15:55] accurate when you're up there at 100 ft. [15:57] Um, and that's what I mean. You need [16:00] 100% accuracy. [16:03] 95 or 99% might not be enough if you're [16:06] breaking a multi-million dollar windmill [16:09] by mistake. [16:11] So like every single one of these [16:13] applications whether you're fixing uh [16:15] windmills or whether you're fixing um [16:17] enterprise data or whether you're fixing [16:20] engineering organization data right now [16:23] if you want to build an enterprise level [16:26] application you have to be very careful [16:28] and you have to think about every uh [16:31] potential use case and then build an [16:33] agentic approach. Hopefully that's going [16:36] to change. [16:39] Um in order to change this there are two [16:42] main things that we need to do. [16:45] One of them on the um concept side as [16:51] I've said you have the transformer uh [16:55] architecture where it's entirely focused [16:58] on um guessing the next token but it [17:01] doesn't have that higher level thinking [17:04] doesn't have that um higher level [17:06] grouping of concept because one of the [17:10] most recent I'll give that example as [17:12] well uh one of the most recent papers um [17:15] that I've read was about something [17:18] called phonms um a very interesting [17:20] paper but the core idea of the paper was [17:24] that I can write the same word in [17:27] cerillic or Latin alphabet or in a [17:30] Chinese alphabet let's say uh I'm [17:33] writing Facebook [17:35] and to us when you read that out it's [17:37] the same exact word but when you [17:40] actually write that out um in these [17:42] three different alphabets and then you [17:45] convert that into tokens. These three [17:47] words are completely different to the AI [17:51] to any of these existing LLMs. And there [17:54] needs to be a higher logic within the [17:56] LLM to be able to understand this to be [17:59] able to [18:02] see that these are actually the same [18:04] exact thing. It's the same concept. And [18:06] that is an architectural architectural [18:09] level thing that we're missing. [18:11] There almost needs to be a higher level [18:15] module within the GPT architecture that [18:17] could cover these concepts and not just [18:21] words concepts but actually like higher [18:23] level ideas. [18:25] That's one of the things that we're [18:26] missing. The other things that uh that [18:28] we're missing is the passing of the [18:31] context. As I've mentioned earlier, um [18:34] when you pass the context, it just [18:37] you're so limited. You're so limited in [18:40] what you can give and the model is very [18:43] smart because of the data that was [18:45] trained with. But the information that [18:49] you pass afterwards, not so much. It's [18:53] foreign to the model and there needs to [18:56] be a training like process for the model [18:59] to almost ingest that. [19:02] So that every time you pass on um a [19:05] massive code base, it needs to replicate [19:08] a training like process. Obviously the [19:10] existing models don't have that because [19:12] chat GPT is already spending a lot of [19:15] energy and imagine every single time you [19:17] pass something it'll be like a training [19:19] like process they would probably be [19:21] bankrupt. [19:23] But for an enterprise application where [19:25] you need [19:27] accuracy and accuracy is key you can [19:30] afford that. For instance, let's go back [19:33] to the windmill example. If I need 100% [19:36] accuracy, then I want every single one [19:39] of these uh documents that I've passed, [19:43] I want the AI to spend as much as it [19:46] needs so that I could get gain on the [19:48] accuracy side. And this goes for every [19:51] other uh agentic application or um [19:54] enterprise application as well where we [19:56] could change the costs. [19:58] All of these models that we have whether [20:00] that's chat GPT claud or like the [20:02] commercially available stuff they're [20:04] good for human use. I actually um share [20:07] everything. I share my medical uh [20:09] records with these but for larger [20:12] information that might not be the case. [20:17] Therefore um we need models that reason [20:21] and that reasoning is not going to [20:23] happen with the existing um techniques [20:27] we have. We need something a little bit [20:28] more dynamic in ingesting the [20:31] information and we need something that's [20:33] a little bit more intelligent in the way [20:36] it structures them within the model. [20:40] Therefore, um [20:42] it's not necessarily going to be the two [20:44] billion context window that the new [20:47] Gemini is going to have, but it's going [20:49] to be a different shift. [20:53] It seems to me that we're reaching the [20:56] very end of the lifetime of the [20:59] transformer architecture [21:02] and we need something else. [21:05] Thank you very much.