AI Engineer
Embeddings function as cached compute, allowing upfront codebase indexing to drastically undercut the recurring token and latency costs of unindexed agent grepping.
AI-assisted summary of RAG is dead, right?? — Kuba Rogut, Turbopuffer — original source. Check the original for full context.
Source published:
Key ideas
- Despite widespread tech discourse claiming 'RAG is dead' in late 2025 and early 2026, Google search volume for RAG hit a major inflection point in mid-2025 and surged to all-time highs. (0:07)
- Cursor reported a 23.8% increase in answer accuracy on its Composer model after adding semantic search, alongside a 2.6% increase in code retention and a 2.2% drop in dissatisfied user requests in production A/B tests. (4:43)
- Embeddings function as cached compute, presenting a direct economic tradeoff between upfront indexing (Cursor) and repeated runtime discovery costs like unindexed filesystem grepping (Claude Code). (6:08)
- Citing Google's Jeff Dean, expanding LLM context windows does not replace retrieval; architectures require staged retrieval to sift trillions of candidate tokens down into the right millions for processing. (9:44)
Detailed notes
## Background Published by the AI Engineer channel on YouTube on June 9, 2026, this presentation features Kuba Rogut, a deployed engineer at turbopuffer. The video addresses the popular discourse declaring retrieval-augmented generation obsolete, presenting real-world performance metrics from Cursor and architectural comparisons with Claude Code to demonstrate why iterative, multi-primitive retrieval remains vital.
## TL;DR Kuba Rogut challenges the social media consensus that 'RAG is dead,' pointing to record-breaking search volume and Cursor's double-digit accuracy gains from integrating semantic search into its composer workflows. He argues that while naive single-turn vector lookups are indeed outdated, modern RAG has evolved into iterative agentic retrieval that combines vector search, BM25, grep, and structured filters. Contrasting Claude Code's unindexed filesystem grepping with Cursor's pre-computed Merkle-tree indexing, Rogut frames embeddings as cached compute that amortizes costs over repeated queries. Ultimately, citing Google's Jeff Dean, he demonstrates that expanding LLM context windows still necessitates staged retrieval to filter down trillions of candidate tokens to the specific million needed for inference.
## Atmosphere & style The presentation is punchy, pragmatic, and delivered with the confident humor of an engineer confronting theoretical social media hype with cold production telemetry ('So take that Twitter'). Rogut speaks briskly to an engaged live audience, utilizing crisp benchmark charts, terminal traces, and architectural comparison tables to ground the discussion in practical trade-offs. The atmosphere feels like an insider systems post-mortem, replacing hand-waving debates with clear mental models around compute amortization and storage tiering.
## Bulletin - [00:07](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=7s) Despite widespread tech discourse claiming 'RAG is dead' in late 2025 and early 2026, Google search volume for RAG hit a major inflection point in mid-2025 and surged to all-time highs. - [04:43](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=283s) Cursor reported a 23.8% increase in answer accuracy on its Composer model after adding semantic search, alongside a 2.6% increase in code retention and a 2.2% drop in dissatisfied user requests in production A/B tests. - [06:08](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=368s) Embeddings function as cached compute, presenting a direct economic tradeoff between upfront indexing (Cursor) and repeated runtime discovery costs like unindexed filesystem grepping (Claude Code). - [09:44](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=584s) Citing Google's Jeff Dean, expanding LLM context windows does not replace retrieval; architectures require staged retrieval to sift trillions of candidate tokens down into the right millions for processing.
## Key takeaways - [00:07](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=7s) Social media claims that RAG is dead are contradicted by real-world developer demand, as evidenced by Google search volume surging in mid-2025. - [01:39](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=99s) The perceived failure of RAG stems from a narrow definition limited to naive, single-turn vector search rather than a broader retrieval ecosystem. - [01:39](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=99s) Comprehensive retrieval incorporates vector search, full-text algorithms like BM25, regex, glob, grep, and structured metadata filtering. - [02:08](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=128s) Agentic search is an iterative reasoning process where agents dynamically inspect context, issue targeted queries, and refine searches rather than executing single-step lookups. - [03:15](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=195s) Cursor optimizes team-wide codebase indexing by using cryptographic Merkle trees to detect structural similarities, re-indexing only modified files. - [04:43](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=283s) Semantic search integration in Cursor improved answer accuracy by 12.5% to 13.5% across evaluated models and nearly 24% on its Composer model. - [05:20](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=320s) Production A/B testing at Cursor revealed that modest single-digit metric improvements represent substantial qualitative gains because semantic search triggers selectively. - [06:08](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=368s) Claude Code's abandonment of vector search in favor of agentic grep represents an architectural tradeoff between zero upfront indexing and higher per-session compute costs. - [06:55](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=415s) Embeddings serve as cached compute: paying an upfront cost to chunk and index pays dividends when amortized across repeated team queries. - [07:30](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=450s) Unindexed grep discovery scales poorly across multi-developer environments where identical context queries consume thousands of tokens per run. - [08:38](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=518s) Single-turn top-k vector search is obsolete for advanced applications, giving way to multi-turn agentic retrieval that actively feeds agent reasoning loops. - [09:44](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=584s) Jeff Dean emphasized that massive context windows require staged retrieval to act as a lightweight sieve, filtering trillions of tokens into the right million. - [10:20](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=620s) Growing foundation model context windows complement rather than eliminate retrieval, providing a necessary filter to make ultra-large models economically viable.
## Detailed walkthrough ### [00:07](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=7s) Introduction and the "RAG is Dead" Narrative
Kuba Rogut introduces himself as a deployed engineer at turbopuffer, a full-text and vector search database built from first principles directly on top of object storage. Addressing a packed room, he dives into the prevalent social media sentiment that retrieval-augmented generation has run its course, pointing out that by late 2025 and early 2026, tech discourse on X (formerly Twitter) became saturated with claims that "RAG is dead" and that agentic file search is all developers need.
```mermaid flowchart LR accTitle: Public Perception vs Search Reality accDescr: Contrasting social media narratives claiming RAG is dead with the mid-2025 spike in Google search interest. subgraph Narrative["Social Media (X / Twitter)"] A["Claims: 'RAG is dead'"] --> B["Agentic file search replaces RAG"] end subgraph Reality["Google Search Trends"] C["2023: Initial AI surge"] --> D["2024: Plateau / cooldown"] D --> E["Mid-2025: Inflection point / massive spike"] end ```
Rogut counters the online narrative by presenting real-world search behavior. While Google search volume for RAG climbed during the initial generative AI boom in 2023 and plateaued across 2024, it reached a dramatic inflection point in mid-2025, surging to all-time highs. Kuba sharply dismisses the online consensus against the data, quipping: “So take that Twitter.”
[01:39](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=99s) Defining RAG vs. Agentic Search
Rogut clarifies the terminology, explaining that the apparent death of RAG is merely the death of an oversimplified definition. Many developers reduce RAG to naive vector search—simply generating embeddings for a text corpus, performing a single vector similarity query, and feeding the resulting chunks directly into an LLM. At turbopuffer, retrieval is treated as a far broader collection of mechanisms, encompassing vector search, full-text search algorithms like BM25, traditional filesystem utilities like grep and glob, regular expressions, and structured metadata filters.
```mermaid flowchart TD accTitle: Expanded Agentic Retrieval Architecture accDescr: How an iterative agent uses a wide array of search tools rather than a single vector search step. User["Agent Query"] --> Tools subgraph Tools["Retrieval Tooling Suite"] T1["Vector Search"] T2["Full-Text (BM25)"] T3["Grep / Glob / Regex"] T4["Metadata Filtering"] end Tools --> Loop{"Sufficient context found?"} Loop -- "No" --> Tools Loop -- "Yes" --> LLM["LLM Generation"] ```
At the same time, the industry’s understanding of "agentic search" has been colloquially reduced to unindexed filesystem exploration via grep, popularized by command-line coding tools like Claude Code. Rogut reframes agentic search not as mindless grepping, but as an iterative, tool-rich reasoning process. Describing this model, Kuba states: “what we actually believe it is and you know kind of the definition we want to we want to give it is it's really giving the agents a set of tools to kind of progressively and iteratively find and reason over context.” Instead of a single retrieval step, an agent reads files, inspects the current state, identifies missing context, issues new queries across its tooling suite, and iterates until it has accumulated the exact context needed to complete the task.
[03:15](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=195s) How Cursor Indexes Codebases with Merkle Trees and turbopuffer
To demonstrate effective agentic search in production, Rogut highlights Cursor, an early turbopuffer customer that published a detailed engineering breakdown in early 2026 on codebase indexing. When a developer opens a repository or checks out a new branch, Cursor automatically parses, chunks, and embeds the files to enable fast semantic search across the project.
Indexing large repositories from scratch is computationally prohibitive, especially within teams where dozens of engineers frequently touch the same codebase. To avoid re-chunking, re-embedding, and re-uploading identical files, Cursor uses Merkle trees—cryptographic hash trees—to compute structural similarities between codebases across a team. If the trees show substantial overlap, Cursor copies over the pre-existing index data and limits parsing and embedding exclusively to the modified files, relying on turbopuffer to ensure the operation remains fast and secure.
- Merkle trees compute file and directory differences across developer environments to prevent redundant index runs. - Only diffed and updated files are re-chunked and re-embedded, drastically reducing overhead. - turbopuffer manages and isolates multi-tenant codebase index storage securely in object storage.
[04:43](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=283s) The Measured Impact of Semantic Search in Cursor
Addressing why a development tool would take on the engineering overhead of continuous background indexing, Rogut cites Cursor’s published benchmarks and internal evaluations. On their internal Cursor context benchmark, adding semantic search produced a 12.5% to 13.5% average gain in answer accuracy across evaluated models. The impact was especially pronounced on Cursor's Composer model (prior to Composer 2), which saw a 23.8% surge in accuracy once backed by semantic retrieval tools.
```mermaid flowchart TD accTitle: Cursor Retrieval Benchmark Gains accDescr: Benchmark accuracy and production A/B testing improvements from adding semantic search. subgraph ContextBenchmark["Internal Context Benchmark"] M1["Average across models: +12.5% to 13.5%"] M2["Composer Model: ~24% increase in accuracy"] end subgraph ProductionAB["Online A/B Testing"] P1["Code retention in large codebases: +2.6%"] P2["Dissatisfied user requests: -2.2% drop"] end ```
In live production, Cursor validated these improvements through an online A/B test. The results demonstrated a 2.6% increase in code retention within large codebases and a 2.2% decrease in dissatisfied user requests. While single-digit percentage improvements might appear modest at first glance, Rogut emphasizes that the semantic search tool is not triggered on every single prompt. Because the tool only fires when relevant to the request, a 2% to 3% shift across all production traffic represents a substantial qualitative leap.
[06:08](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=368s) Embeddings as Cached Compute: Claude Code vs. Cursor
Rogut examines the alternative architectural approach championed by Boris Cherny, the creator of Claude Code. Cherny noted publicly on X that early iterations of Claude Code experimented with local vector databases and RAG, but the team ultimately abandoned them in favor of running agentic grep commands directly against the user's filesystem. Rather than treating this as proof that RAG has failed, Rogut frames the divergence through a core turbopuffer thesis: embeddings are essentially cached compute.
```mermaid flowchart TD accTitle: Per-Session Discovery vs Cached Index Traces accDescr: Comparing Claude Code iterative grepping against Cursor indexed retrieval. subgraph ClaudeCode["Claude Code: Per-Session Grep (No Index)"] direction TB C1["User Query: 'How metadata filtering works'"] --> C2["Grep codebase"] C2 --> C3["Read file contents"] C3 --> C4["Assess relevance & repeat loop"] C4 --> C5["Cost: ~6,000+ tokens consumed per session"] end subgraph Cursor["Cursor: Cached Compute (Indexed)"] direction TB K1["Upfront Index: Parse, embed, store in turbopuffer"] K1 -. "One-time cost".-> K2["Runtime Tool Call: 'How is metadata filtered?'"] K2 --> K3["Lightweight semantic search"] K3 --> K4["Immediate targeted context (low tokens, fast execution)"] end ```
To illustrate the tradeoff, Rogut compares the execution trace of a Claude Code session with a Cursor session answering the query: "How does metadata filtering work?":
| Characteristic | Claude Code (Agentic Grep) | Cursor (Indexed Semantic Search) | |:--- |:--- |:--- | | **Indexing Step** | None. Operates directly on raw local files. | Upfront parsing, chunking, and embedding into turbopuffer. | | **Context Discovery** | Iterative loop: agent greps, reads full files, assesses context, and re-greps. | Single lightweight tool call querying the pre-computed index. | | **Token & Time Cost** | High per session (e.g., 6,000+ tokens for a single sub-step); paid repeatedly on every query. | Negligible runtime token overhead; one-time indexing cost amortized over thousands of queries. | | **Multi-User Scaling** | 10 developers asking the same architectural question repeat the full grep loop 10 times. | All developers leverage the shared cache; answers return nearly instantaneously. |
Kuba summarizes the dynamic succinctly: “So obviously there's this like upfront cost of parsing a code base embedding it and making it available but this is like a one-time cost and then at runtime the agent can just query something like how is metadata filtered?” He notes that within turbopuffer's own engineering team, several developers who previously relied on Claude Code migrated to Cursor because the combination of pre-computed semantic search and Composer 2 drastically reduces latency and token consumption.
[08:38](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=518s) The Evolution Toward Agentic Retrieval
Rogut closes the architectural comparison by underscoring that simple, single-turn RAG—embedding chunks, querying top-k matches once, and blindly stuffing them into a context window—is indeed obsolete for advanced use cases. While that rudimentary pattern dominated the landscape in 2023 and early 2024, production systems have transitioned to multi-turn agentic retrieval. By providing autonomous agents with a rich suite of complementary retrieval primitives, modern systems achieve major performance breakthroughs over naive context stuffing.
[09:09](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=549s) The Shift to Iterative, Agentic Retrieval
Retrieval is no longer a naive, single-step operation where an application fires off a solitary query to an external vector database and blindly consumes whatever returns. Instead, modern AI agents run multi-step reasoning loops, dynamically executing semantic search, full-text search, or other filtering techniques as required at each step. These agents pull in strictly what is necessary for their immediate task, analyze the newly acquired information, and then issue follow-up searches based on their updated context. As Kuba Rogut explains: “It's becoming super iterative and these agents are really understanding what they're searching and searching to understand more in a sense.” This cyclical loop transforms retrieval from an isolated lookup into an active reasoning tool that unblocks entirely new software products.
```mermaid flowchart TD accTitle: Iterative Agentic Retrieval Loop accDescr: Diagram showing an agent iteratively reasoning, dispatching search queries, and filtering context down to relevant tokens.
subgraph AgentLoop ["Agent Reasoning Loop"] A["Agent Reasoning Step"] B{"Search Strategy"} C["Semantic / Vector Search"] D["Full-Text / Filter Search"] E["Fetch Targeted Context"] end
A --> B B -->|"Concept match"| C B -->|"Exact/Keyword match"| D C --> E D --> E E -->|"Analyze & refine"| A ```
- Modern architectures treat retrieval not as a single database call, but as an ongoing conversation between reasoning agents and storage layers. - Agents adapt their search mechanisms on the fly, alternating between vector indexes and full-text search depending on the nature of the missing context. - Information retrieval feeds directly back into the agent's reasoning loop, allowing it to formulate sharper, downstream queries.
[09:44](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=584s) Jeff Dean on Staged Retrieval and Context Windows
Even as foundation models expand their raw context windows to handle massive token counts, Kuba Rogut points out that sheer capacity does not eliminate the necessity of retrieval. Referencing an interview with Google's Jeff Dean regarding Gemini's massive context windows, Rogut emphasizes that expanding context sizes to astronomical levels still requires intermediate mechanisms to sift through raw data. Jeff Dean captured this core architectural principle by observing: “what you really need is stage retrieval, like a lightweight mechanism to narrow down these trillion tokens into essentially millions at a time.” In short, you do not need a trillion tokens dumped into a prompt at once; you need the right million.
At Turbopuffer, customers store and index trillions of tokens, making staged reduction critical in practice. Rather than saturating a context window with unfiltered corpus data, the retrieval system's role is to act as that lightweight sieve—filtering down trillions of candidate tokens to the specific 10,000, 100,000, or million tokens that directly resolve the user's intent. Rogut concludes the talk by underscoring that context window growth complements rather than replaces indexing, inviting the audience to discuss technical details and thanking attendees as the presentation wraps up.
- Jeff Dean highlighted that even with hypothetical trillion-token context windows, staged retrieval remains indispensable to filter massive corpuses into digestible subsets. - High-scale systems at Turbopuffer manage trillions of tokens by relying on lightweight pre-filtering down to targeted sets of 10,000, 100,000, or 1,000,000 tokens before populating context windows. - Retrieval acts as a precision filter that makes ultra-large context models economically and computationally feasible.
## Notable quotes > [09:44](https://www.youtube.com/watch?v=UM6sFg_jdlE&t=584s) what you really need is stage retrieval, like a lightweight mechanism to narrow down these trillion tokens into essentially millions at a time. — Jeff Dean
## Mentioned Boris Cherny, Claude Code, Cursor, Composer, Composer 2, Gemini, Google, Jeff Dean, Kuba Rogut, turbopuffer, X (formerly Twitter)
## Video description Cursor added semantic search and measured a 24% increase in answer accuracy on their composer model, a 2.6% gain in code retention in large codebases, and a 2.2% drop in dissatisfied user requests. Those numbers look small until you factor in that semantic search does not fire on every query. Meanwhile Google search volume for RAG hit a new inflection point in mid 2025 and went through the roof. The Twitter "RAG is dead" discourse and the actual usage curve are moving in opposite directions.
Kuba Rogut's argument is that the problem was never retrieval, it was the narrow definition of it. RAG is not just a vector search call. It is vector search, full text search, glob, regex, and filters used iteratively by an agent that keeps searching until it has what it needs. He contrasts Claude Code (grep per session, no index, repeat cost every run) with Cursor (one time upfront indexing, lightweight tool calls at runtime). Claude Code's approach is not wrong, it is a deliberate tradeoff. The frame that clarifies it: embeddings are cached compute, and whether to cache depends on query volume. Jeff Dean's version: you do not need a trillion tokens at once, you need the right million.
Speaker info: - https://www.linkedin.com/in/kubarogut/ - https://x.com/rogutkuba
Timestamps: 0:00 Introduction to the "RAG is dead" discourse 1:12 Google search volume trends for RAG 1:39 Defining RAG vs. Agentic Search 3:15 Cursor's indexing and semantic search approach 6:10 Contrasting Claude Code (grep) vs. Cursor (indexed) 6:40 The concept of embeddings as cached compute 8:38 The shift from simple RAG to Agentic Retrieval 9:44 Jeff Dean on context windows and stage retrieval
---
Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/UM6sFg_jdlE-rag-is-dead-right-kuba-rogut-redacted-username
Join our upcoming AI Engineer conferences (see https://ai.engineer landing page):
- AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney
---
[Source](https://www.youtube.com/watch?v=UM6sFg_jdlE)