Vespa AI Pitches Multimodal Tensors to Replace Basic Vector Databases
Ai4
Standard 1D vector embeddings degrade enterprise AI agent accuracy by stripping spatial and structural context, requiring native in-database tensor search to prevent reasoning errors.
## Background
Published on YouTube by the Ai4 conference channel on August 29, 2025, this presentation features Vespa.ai's Bonnie Chase (Director of Product Marketing) and Harini Gopalakrishnan (General Manager, Industry). The session explores how enterprises must rethink Retrieval-Augmented Generation (RAG) and AI agent workflows as fundamental search problems by adopting high-dimensional tensor frameworks and in-engine custom ranking.
## TL;DR
The speakers outline why traditional enterprise RAG systems falter when restricted to basic 1D vectors, brittle document chunking, and high-latency external reranking pipelines. Vespa addresses these bottlenecks through native multi-phase ranking computed directly where data resides, using high-dimensional tensors that unify text, tables, images, and biological data structures. Through a live life-sciences demonstration using the ColPali vision-language model, they show subsecond retrieval across complex scientific PDFs, extracting data directly from embedded chart arrows and clinical acronyms. The talk also illustrates broader high-dimensional applications, including AlphaFold protein structure embeddings and histopathology slide search, emphasizing that reliable agentic workflows require computing search directly within the data storage engine.
## Atmosphere & style
The talk maintains a crisp, authoritative, and pragmatic tone that balances high-level architectural strategy with granular engineering reality. Chase and Gopalakrishnan exhibit a smooth, collaborative dynamic, fluidly trading off between enterprise platform scalability and specialized scientific domain challenges. The pacing feels grounded and assured, heightened by live, unedited query demonstrations on raw PDF research documents that showcase practical capability rather than theoretical hype.
## Key takeaways
- [00:00](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=0s) Agentic AI workflows rely on asking questions, making decisions, and executing actions, meaning systems fail if the underlying search and retrieval is inaccurate or stale.
- [03:22](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=202s) Moving from prototype chatbots to enterprise RAG exposes critical bottlenecks: multimodal document complexity, custom relevancy tuning, and error propagation across multi-step agent reasoning chains.
- [06:40](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=400s) Production platforms like Perplexity and Spotify achieve web-scale performance by leveraging custom in-engine indexing and ranking rather than relying solely on larger foundation models.
- [10:40](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=640s) Executing multi-phase ranking natively inside the storage engine eliminates the high latency and infrastructure costs of transferring large candidate sets across network boundaries to external rerankers.
- [10:40](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=640s) Standard 1D vectors flatten spatial, temporal, and hierarchical context, whereas multidimensional tensors preserve fine-grained visual layouts and time-based coordinates.
- [12:57](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=777s) A modular three-part RAG architecture decouples unstructured data, high-dimensional knowledge bases, and foundation models, enabling seamless runtime updates and model swapping.
- [12:57](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=777s) Vision-language models like ColPali index PDF pages as multi-vector patch tensors, enabling direct visual lookup and data extraction from charts and flowcharts without brittle OCR pipelines.
- [15:55](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=955s) Tensor-based search engines can resolve semantic medical concepts and extract embedded chart metrics directly from raw PDFs with subsecond latency.
- [17:50](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1070s) High-dimensional tensor indexing accommodates AlphaFold 3D protein structures and localized histopathology slide searches to accelerate drug discovery and clinical cohort matching.
- [17:50](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1070s) Composite multi-vector tensors allow disparate data modalities—such as macromolecular structures, chemical molecular graphs, and research text—to be unified into single queryable records.
- [19:48](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1188s) Next-generation RAG systems must compute where data resides, treat high-precision search as the core engine of agentic workflows, and move from flat vectors to tensors.
## Detailed walkthrough
### [00:00](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=0s) RAG Reimagined: Reframing RAG as a Search Problem
Bonnie Chase kicks off the session alongside Harini Gopalakrishnan to explore how enterprises need to rethink Retrieval-Augmented Generation (RAG) by fundamentally treating it as a search problem. While ChatGPT sparked widespread excitement around generative AI and standard RAG workflows, the paradigm is rapidly shifting from passive text retrieval to agentic systems that actively make decisions and execute actions. These agentic workflows require three core capabilities: asking the right questions, making sound decisions, and taking informed actions.
```mermaid
flowchart TD
accTitle: Agentic Decision Loop Dependent on Retrieval
subgraph CoreEngine ["Search & Retrieval Engine"]
A["Accurate, Timely Information"]
end
subgraph AgenticLoop ["Agentic System Workflows"]
B["1. Ask the Right Questions"]
C["2. Make Good Decisions"]
D["3. Take Informed Actions"]
end
A --> B
B --> C
C --> D
```
If the underlying information retrieval mechanism is flawed or stale, the downstream agentic loop completely disintegrates. In high-stakes applications—such as a healthcare assistant querying outdated clinical trial registries or an e-commerce agent recommending out-of-stock products—simple vector search fails. Basic single-embedding vector databases struggle under complex business rules and cannot adequately handle rich multimodal datasets, underscoring why modern AI needs more than basic similarity matching.
---
[03:22](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=202s) The Enterprise Bottlenecks: Scale, Multimodality, and Relevancy
Harini Gopalakrishnan draws on her background working with multimodal data and enterprise AI across tech and pharmaceutical organizations like Sanofi to identify why standard chatbot proof-of-concepts fail to transition smoothly into production. While building a prototype over thousands of documents is trivial, four major hurdles emerge at true enterprise scale:
```mermaid
flowchart LR
accTitle: Four Enterprise RAG Bottlenecks
subgraph Hurdles ["Enterprise Hurdles"]
direction TB
H1["1. Scale Beyond Text"]
H2["2. Multimodal PDFs & Posters"]
H3["3. Relevancy Tuning"]
H4["4. Long Reasoning Propagation"]
end
subgraph Solutions ["Requirements"]
direction TB
S1["Tensors (Vectors of Vectors)"]
S2["Vision-Language Embeddings"]
S3["In-Database Custom Ranking"]
S4["High-Precision First-Step Retrieval"]
end
H1 --> S1
H2 --> S2
H3 --> S3
H4 --> S4
```
1. **Scale Beyond Text:** Handling massive corporate corpora without relying on fragile chunking pipelines and brittle preprocessing scripts.
2. **Multimodal Complexities:** Extracting meaning from real-world scientific PDFs, conference posters, tables embedded inside graphs, and complex diagrams that traditional Optical Character Recognition (OCR) and Natural Language Processing (NLP) fail to parse.
3. **Relevancy Tuning:** Rather than forcing users to "prompt engineer the heck of it," the retrieval engine itself must understand context and rank documents dynamically. Harini highlights Perplexity’s approach: their "secret sauce" is not training a massive proprietary foundation model from scratch, but rather indexing external content with custom relevancy scoring.
4. **Error Propagation in Long Reasoning Agents:** In multi-step agentic execution (such as Model Context Protocol / MCP chains), retrieving an incorrect answer on step one compounds error across subsequent reasoning steps.
---
[06:40](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=400s) Four Core Capabilities of Vespa and the Engine's Pedigree
To address these enterprise challenges, Harini outlines four foundational capabilities offered by Vespa, before passing the presentation back to Bonnie to provide background on the platform's history.
- **Multimodal Tensor Handling:** Moving past single flat vectors to native tensors (arrays of vectors/matrices) that can encode protein structures, chemical formulas, image patches, and tables into unified representations.
- **Precision Retrieval and Relevancy Tuning:** Built-in support for multi-phase search and native machine learning ranking algorithms directly inside the storage engine.
- **Enterprise Security and Deployment Fitment:** Deployable on AWS, Azure, GCP, and on-premises environments, ensuring sensitive data never leaves enterprise virtual private clouds (VPCs).
- **Cost Efficiency and ROI:** Storing data as compressed tensors avoids storing petabytes of raw unindexed imagery/PDFs, while reducing separate cross-network reranking calls saves engineering overhead and Amazon EC2 runtime.
Bonnie highlights that although Vespa spun out of Yahoo as an independent startup at the end of 2023, the underlying technology has matured over more than a decade in high-throughput production environments. Today, Vespa powers search, recommendations, and feed personalization for major web-scale platforms including Yahoo Finance, Spotify, Perplexity, and Vinted. For example, Perplexity scaled its index from 1.5 billion toward 10 billion documents on Vespa, while Spotify indexes every individual word spoken across millions of podcast episodes so users can locate specific episodes from half-remembered spoken phrases.
---
[10:40](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=640s) Multi-Phase Ranking and High-Dimensional Tensors vs. 1D Vectors
Bonnie explains why the conventional architecture of pairing a separate vector database with an external reranking service creates latency bottlenecks and ballooning infrastructure costs. Transferring large candidate sets across network boundaries for ranking degrades performance. Vespa resolves this by executing multi-phase ranking natively where the data lives:
```mermaid
flowchart TD
accTitle: Vespa In-Database Multi-Phase Ranking
subgraph StorageEngine ["Vespa Data Engine (Local Storage & Compute)"]
direction TB
Corpus["Indexed Tensor & Document Corpus"]
Stage1["First-Phase Ranking: Fast, Low-Cost ML Models Narrow Candidates"]
Stage2["Second-Phase Ranking: High-Accuracy, Expressive Models Re-Score"]
Final["Top Relevant Results Fed to LLM"]
Corpus --> Stage1
Stage1 --> Stage2
Stage2 --> Final
end
```
A critical differentiator is Vespa’s native support for multidimensional tensors versus traditional one-dimensional vector embeddings:
- **The Limitation of 1D Vectors:** Standard vectors flatten complex documents, video streams, or visual layouts into a single point in space, stripping away spatial, temporal, and hierarchical context.
- **The Tensor Advantage:** Tensors allow multi-dimensional representations. For video retrieval, this enables finding the exact timestamp of an event without custom hacks; for complex documents, it preserves fine-grained visual-spatial relationships.
---
[12:57](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=777s) The Three-Part Paradigm and ColPali Demo in Life Sciences
Harini outlines the modern three-pillar architecture for enterprise RAG: keeping the raw unstructured data, the foundation models (embedding and generation models like Qwen or ColBERT), and the central knowledge base distinct. Decoupling these components ensures that enterprises can update their knowledge corpora continuously at runtime and hot-swap open-source vision-language models as state-of-the-art architectures evolve.
```mermaid
flowchart LR
accTitle: Modern Three-Part RAG Architecture
Data["1. Unstructured & Structured Corpus"] --> KB["2. High-Dimensional Knowledge Base (Vespa)"]
Models["3. Modular Models (ColPali, Qwen, ColBERT)"] <--> KB
KB --> Context["Rich Multimodal Context at Runtime"]
```
Harini demonstrates this architecture applied to complex life sciences literature—clinical trial filings, scientific posters, and drug brand documents—indexed using **ColPali**, an open-source vision-language late-interaction model:
- **Patch Embeddings as Tensors:** ColPali treats each PDF page as an image, breaking it into localized pixel patches (e.g., $30 \times 30$ grids). Each page is stored within Vespa as a multi-vector tensor.
- **MaxSim Precision Lookup:** Utilizing late-interaction operators like MaxSim, the system matches queries against specific visual tokens. When searching for the efficacy of *Warfarin*, the system locates the exact table row on the page and visualizes the match with highlighting.
- **Contextual Concept Matching:** When queried for a cost comparison between obesity GLP-1 medications (*Semaglutide* vs. *Tirzepatide*), the engine pinpoints a chart expressing drug cost-effectiveness as an ICER (Incremental Cost-Effectiveness Ratio) metric—even though the literal word "cost" never appeared in the text.
- **Complex Poster Extraction:** An exact query regarding weight loss percentage targets and extracts data embedded directly inside flowcharts and visual arrows on scientific conference posters without requiring manual text extraction or OCR pre-processing.
[15:55](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=955s) Multimodal PDF Retrieval and Complex Life Sciences Use Cases
The live demonstration highlights the precision of vision-language models (VLMs) and tensor-based indexing when querying rich, visual documents. In the demo, the system successfully extracts information embedded directly inside text labels and arrows within a complex chart, accurately comparing weight-loss percentages across cohorts. The setup required minimal engineering overhead: the team took ColPali, tensorized the document representations, ingested them into Vespa, and applied basic relevance tuning. Designed simply as a proof-of-concept rather than an over-engineered production pipeline, the retrieval query was configured to retrieve and summarize the top three most relevant results—a parameter that can easily be scaled to top 50 or weighted by metadata such as recency to prioritize newer research.
```mermaid
flowchart TD
accTitle: Multimodal Document Retrieval Flow
subgraph Ingestion["1. Multimodal Document Ingestion"]
A["Original Research PDF"] --> B["ColPali / VLM Tokenizer"]
B --> C["High-Dimensional Multi-Vector Tensor"]
C --> D["Vespa Tensor Index Engine"]
end
subgraph QueryExecution["2. Subsecond Retrieval & Ranking"]
E["Complex Life Science Query\n(e.g., MACE % reduction)"] --> F["Custom Ranking & Relevancy Tuning\n(Top-K, Recency Boosting)"]
D --> F
F --> G["Visual Grounding & Chart Parsing"]
G --> H["Generated Synthesized Answer"]
end
```
In a medical query test, the system parsed the clinical acronym "MACE" (Major Adverse Cardiovascular Events) directly out of an embedded graph, correctly calculating the percentage reduction for patients on a specific diet regimen. Because the live system queries raw PDF pages directly and renders subsecond responses without edited recordings, it demonstrates the readiness of high-dimensional search architectures for production RAG pipelines. This approach naturally extends beyond standard business charts to intricate life sciences workloads, such as querying antibody structures against protein targets, isolating sub-segments in protein embeddings paired with annotation metadata, or executing fine-grained visual similarity searches in retail catalogs.
- The system accurately extracted embedded percentages directly from graphical arrow annotations and summarized clinical trial weight-loss outcomes.
- Clinical chart parsing correctly interpreted the domain acronym MACE (Major Adverse Cardiovascular Events) from a raw visual plot.
- Relevancy ranking easily accommodates configurable top-$N$ parameters (e.g., top 3 to top 50) and recency boosting without complex re-indexing.
- All multimodal document retrievals operate at subsecond latency on unedited live PDF data.
---
[17:50](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1070s) Beyond Text: Protein Embeddings, Histopathology, and High-Dimensional Tensors
Expanding RAG into scientific discovery requires moving far past simple text and flat vectors into high-dimensional tensor representations. Highlighted by the Nobel Prize awarded for DeepMind's AlphaFold protein-folding transformer, modern AI models can map entire macromolecular structures into dense embeddings. By indexing these representations directly into tensor engines, researchers can search across diverse data modalities to accelerate oncology research and novel drug discovery pipelines. Similarity in these systems transcends standard vector Euclidean distances, allowing multi-attribute and geometric matching across complex biological data.
```mermaid
flowchart LR
accTitle: High-Dimensional Multi-Modal Tensor Fusion
subgraph DataModalities["Multimodal Modalities"]
P["Protein Structures\n(AlphaFold Dense Embeddings)"]
H["Histopathology Slides\n(Pixel-Level Cancer Subsections)"]
D["Multimodal Documents\n(Charts, Tables, Page Layouts)"]
end
subgraph FusionEngine["Unified High-Dimensional Representation"]
P & H & D --> T["Single Composite Tensor\n(Multi-Vector Space)"]
T --> V["Vespa Engine Retrieval\n(Custom Distance & Similarity Metrics)"]
end
subgraph Outcomes["Clinical & Research Applications"]
V --> R1["Drug Discovery & Antibody Targeting"]
V --> R2["Diagnostic Progression & Historical Cohort Matching"]
end
```
In digital pathology, high-dimensional tensors allow pixel-level indexing of whole-slide histopathology images. Instead of relying solely on rigid supervised classification models, clinicians can extract a localized tissue segment of interest (such as a suspected malignant region) and instantly query historical repositories: "Have I seen this image before from my corpus before, and then see what the prognosis or diagnosis or progression was?" Furthermore, a single composite tensor can unify multiple disparate modalities—binding a protein structure, textual literature, chemical molecular graphs, and slide imagery together—enabling holistic, cross-domain queries critical for health-tech and biopharma applications.
- AlphaFold transformer embeddings allow dense vector and tensor representations of 3D protein structures for novel drug discovery.
- Histopathology slide search enables sub-region retrieval to identify historical patient matches and disease progression patterns without heavy bespoke classifiers.
- Multi-page document architectures generate page-level tensors that natively encapsulate layout, typography, charts, and embedded graphics.
- Composite multi-vector tensors combine proteins, molecular structures, and descriptive text into unified queryable records.
---
[19:48](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1188s) Architecting Intelligent Agents and Key Takeaways for Next-Gen RAG
Building truly intelligent enterprise agents demands an infrastructure that combines structured analytics with high-performance unstructured search. An agentic or Model Context Protocol (MCP) architecture must orchestrate structured Text-to-SQL data fabrics alongside multimodal, tensor-based unstructured retrieval engines.
To conclude the session, the speakers outline three foundational takeaways for architects building scalable, next-generation RAG systems:
1. **Compute Where the Data Resides:** High-throughput, low-latency RAG requires evaluating complex ranking, filtering, and tensor mathematics directly within the storage engine rather than shuttling massive vector payloads across the network.
2. **Search is the Fundamental Core:** Retrieval quality across multimodal assets determines agent reliability; framing RAG as an advanced search problem is essential.
3. **Move Beyond Flat Vectors to Tensors:** Traditional 1D vector embeddings have reached their architectural ceiling in multi-attribute, multimodal environments. Engineers should explore tensor formalism to represent multidimensional relationships natively.
> "When you're wanting to build a RAG system that's accurate, low latency, and high scale, you want to be able to compute where the data resides. You need to remember that search is still the core problem, and you need to be able to search across modalities. And remember, vectors have a limit. Start looking at tensors."
## Notable quotes
> As AI agents and retrieval-augmented generation (RAG) become more advanced, search & retrieval has moved from a backend tool to the engine driving intelligent decisions.
> Agentic workflows depend on asking the right questions, making smart decisions, and taking actions—all of them depend on accurate information retrieval.
> Traditional methods like basic vectors and matrices fall short, especially in complex fields like life sciences, drug discovery or in e-commerce. That’s where high-dimensional data structures come in.
> [03:22](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=202s) prompt engineer the heck of it — Harini Gopalakrishnan
> [17:50](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1070s) Have I seen this image before from my corpus before, and then see what the prognosis or diagnosis or progression was? — Harini Gopalakrishnan
> [19:48](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1188s) Compute Where the Data Resides
> [19:48](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1188s) Search is the Fundamental Core
> [19:48](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1188s) Move Beyond Flat Vectors to Tensors
> [19:48](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1188s) When you're wanting to build a RAG system that's accurate, low latency, and high scale, you want to be able to compute where the data resides. You need to remember that search is still the core problem, and you need to be able to search across modalities. And remember, vectors have a limit. Start looking at tensors.
## Video description
Bonnie Chase, Director of Product Marketing, Vespa.ai
Harini Gopalakrishnan, General Manager, Industry, Vespa.ai
Abstracts
As AI agents and retrieval-augmented generation (RAG) become more advanced, search & retrieval has moved from a backend tool to the engine driving intelligent decisions.
Agentic workflows depend on asking the right questions, making smart decisions, and taking actions—all of them depend on accurate information retrieval. This is why we need to rethink how RAG and AI agents are designed - we need to frame it as a search problem.
Traditional methods like basic vectors and matrices fall short, especially in complex fields like life sciences, drug discovery or in e-commerce. That’s where high-dimensional data structures come in.
In this talk, Vespa will introduce a new approach using high-dimensional data structures (vision-language models) with a flexible tensor framework and accuracy in retrieval with custom ranking. We’ll show how these tools power better search—and how you can start using them today.
Harini Gopalakrishnan and Bonnie Chase argue that enterprise RAG and autonomous agent workflows fail when treated as simple similarity matching, requiring organizations to reframe retrieval as a search problem powered by high-dimensional tensors and in-database multi-phase ranking.
Failure of Flat Vectors in Agentic Workflows (0:00)
Gopalakrishnan and Chase state that the transition from simple conversational chatbots to autonomous agentic systems exposes critical vulnerabilities in standard retrieval architectures. Agentic workflows—such as Model Context Protocol chains—depend on a sequential loop of querying, decision-making, and execution. When the underlying search mechanism returns incomplete, stale, or poorly ranked data, errors compound across each subsequent reasoning step, causing multi-step workflows to fail.
Representing Complex Modalities Through Multidimensional Tensors (17:50)
Conventional one-dimensional vector embeddings flatten complex documents, video streams, and scientific data into a single point in vector space, discarding essential spatial, hierarchical, and temporal context. Chase and Gopalakrishnan contend that enterprise scale requires native tensors—multidimensional arrays of vectors—that can encode structured relationships directly. In video retrieval, tensors enable systems to isolate exact timestamps, while in document retrieval, they preserve visual layouts without brittle chunking pipelines.
In scientific applications, high-dimensional tensors allow direct indexing of DeepMind's AlphaFold protein embeddings and pixel-level histopathology whole-slide images. Gopalakrishnan notes that composite multi-vector tensors can bind protein structures, textual research, and molecular graphs into a single queryable entity, allowing researchers to conduct geometric similarity searches across disparate modalities without building bespoke supervised classifiers.
In-Database Multi-Phase Ranking Over External Rerankers (10:40)
Standard architectures that separate vector databases from external reranking services create severe latency bottlenecks and inflate infrastructure costs by transferring large candidate datasets across network boundaries. Chase asserts that scalable enterprise retrieval requires computing relevance directly within the storage engine. Vespa uses a multi-phase ranking model where lightweight machine learning models narrow candidates locally before expressive, high-accuracy models score the final subset.
To substantiate the scalability of in-database compute, the speakers highlight enterprise deployments on Vespa, including Perplexity expanding its index from 1.5 billion toward 10 billion documents with custom relevance scoring, and Spotify indexing spoken words across millions of podcast episodes for subsecond natural-language phrase lookups.
Multimodal Document Grounding Using Vision-Language Embeddings (12:57)
Enterprise documents such as clinical trial filings and scientific posters frequently embed critical data inside charts, tables, and visual annotations where standard optical character recognition and text-based parsers fail. Demonstrating an implementation using ColPali, an open-source vision-language late-interaction model, Gopalakrishnan illustrates how PDF pages are converted into localized pixel-patch tensors and matched via MaxSim operations to identify precise visual tokens.
During live testing on unedited research literature, the system matched queries comparing obesity drug costs to charts displaying Incremental Cost-Effectiveness Ratios (ICER), despite the word 'cost' being absent from the text. The engine also extracted clinical trial weight-loss percentages directly from visual arrows in conference posters and interpreted the medical acronym MACE (Major Adverse Cardiovascular Events) from raw plot data, returning results with subsecond latency.
The speakers conclude that next-generation enterprise architectures must combine structured analytics like Text-to-SQL with in-database, tensor-based unstructured search engines. However, the system's operational viability hinges on maintaining high-precision initial retrieval, as unmitigated first-step errors continue to present a severe vulnerability for multi-step autonomous agent reasoning.
## Background
该内容由 Ai4 于 2025 年 8 月 29 日在 YouTube 平台发布，由 Vespa.ai 的产品营销总监 Bonnie Chase 与行业总经理 Harini Gopalakrishnan 共同主讲。视频围绕“重塑 RAG”展开，探讨了在智能体（AI Agent）工作流快速普及的背景下，如何将检索增强生成重新定义为核心搜索问题，并利用高维张量框架与自定义排序机制突破传统向量数据库在生命科学、医药及电商等复杂行业的落地瓶颈。
## TL;DR
演讲指出智能体系统的自主决策与行动高度依赖底层精准检索，而传统单一向量数据库在处理复杂排版、多模态图表及长推理链时极易引发错误级联。Vespa 基于十余年工业级沉淀及 Perplexity、Spotify 的超大规模生产实践，提出了“原生多阶段重排”与高维张量（Tensor）架构，使计算直接发生在数据存储层以降低延迟与成本。通过将数据层、高维知识库与推理模型层解耦，并结合 ColPali 视觉语言模型，系统能够实现针对临床文献复杂表格、科研海报及像素级病理切片的亚秒级高精度召回。演讲最后提炼了构建下一代企业级 RAG 的三大原则：就地计算、坚守搜索核心以及全面转向张量化表征。
## Atmosphere & style
整场分享洋溢着严谨务实且极具前瞻性的硬核工业技术氛围。两位讲者配合默契，Bonnie 从宏观架构与海量高并发业务痛点切入，剖析传统拼凑型方案的缺陷；Harini 则深入生命科学垂直场景，通过未剪辑的实时实测直观呈现复杂多模态图表与专业缩写的秒级定位。叙述节奏紧凑干练，既有对传统单向量物理极限的犀利反思，又充满将前沿视觉语言模型工程化落地的实战底气与专业说服力。
## Key takeaways
- [00:00](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=0s) RAG 的本质是一个搜索问题，精准及时的信息检索是驱动智能体工作流（正确提问、明智决策、精准行动）的绝对基石。
- [03:22](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=202s) 企业级 RAG 在实际落地中面临多模态版面解析、相关性精细调优、长推理链错误级联以及合规与算力成本控制四大核心挑战。
- [08:26](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=506s) Vespa AI 是源自雅虎十余年工程验证的原生张量搜索引擎，深度融合了十亿级大规模倒排索引与高维张量计算能力。
- [09:25](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=565s) Perplexity 支撑百亿级文档检索以及 Spotify 对播客逐字索引的工业级实践，证明了低延迟、高吞吐检索对现代 AI 系统的关键性。
- [10:40](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=640s) 将多阶段重排（Multi-phase Ranking）直接下沉至数据存储层就地计算，可消除数据在存储与模型间的网络搬运，显著降低延迟与算力成本。
- [10:40](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=640s) 传统一维单向量表征多模态数据存在物理局限，而高维张量（Tensors）能原生保留复杂版面、时空拓扑与精确时间戳等丰富上下文。
- [12:57](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=777s) 推行数据层、高维知识库与模型推理层的三层解耦范式，企业无需盲目堆砌模型参数，即可实现知识库与最新前沿模型的即插即用。
- [12:57](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=777s) 结合 ColPali 等视觉语言晚期交互模型，将文档页面按 Patch 转换为张量存入 Vespa，可省去繁琐的 OCR 清洗工程并实现端到端检索。
- [14:50](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=890s) 在生物医药文献实测中，张量晚期交互架构能在亚秒级内跨越复杂表格、缩写图表和海报流程图，实现精准的视觉高亮召回。
- [18:55](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1135s) 多模态张量框架支持将像素级病理切片、蛋白质三维空间结构与文献文本统一表征，重塑了临床相似病例诊断与靶点抗体匹配流程。
- [19:48](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1188s) 构建生产级智能体系统的三大核心原则：计算必须贴近数据存储层、搜索始终是 RAG 的本质、数据表征必须从单一向量转向多维张量。
## Detailed walkthrough
### [00:00](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=0s) 重新构想 RAG：将检索增强生成定义为搜索问题
来自 Vespa.ai 的产品营销总监 Bonnie Chase 与行业总经理 Harini Gopalakrishnan 共同展开了关于“重塑 RAG（RAG Reimagined）”的主题分享。演讲的核心主张是将检索增强生成（RAG）从本质上重新定义为一个“搜索问题”。随着 ChatGPT 的爆发，整个行业掀起了利用 AI 和 RAG 构建应用的浪潮；然而，当前技术演进的焦点正在发生转移——AI 系统已不再局限于简单地检索信息，而是正在演变为能够自主决策并执行动作的智能体系统（Agentic AI / Agents）。
```mermaid
flowchart LR
accTitle: Agentic Workflow Foundation
subgraph Core["检索底座 (Search & Retrieval)"]
R["准确、相关、及时的信息检索"]
end
subgraph Agent["智能体工作流 (Agentic Workflow)"]
Q["提出正确的问题"] --> D["做出明智的决策"]
D --> A["采取有效的行动"]
end
R --> Q
```
智能体工作流的稳定运行依赖于三个关键能力：提出正确的问题、做出优质的决策以及执行精准的行动。这一切的根本基石是“准确、相关且及时的信息检索”。一旦底层的检索环节发生故障，整个上层系统就会彻底崩溃。Bonnie 举例说明了检索错误在现实中的严重后果：
- **医疗健康助手**：若检索到底层过时或错误的临床试验数据，将直接向患者或医生输出具有误导性的危险建议。
- **电商智能体**：若向用户推荐了已经缺货（out-of-stock）的商品，业务闭环便瞬间失效。
当前市场上绝大多数基础 RAG 系统都依赖于初级的向量数据库（Vector Databases），仅依靠单一嵌入（Single Embeddings）进行简单的相似度计算。在面对医疗健康（Healthcare）或金融（Financial Systems）等规则严苛、涉及海量多模态数据的复杂企业级场景时，这种单一模式便会迅速瓦解。因此，搜索绝不能仅仅被视作企业内部的一个传统后台工具，它必须具备深度上下文整合能力，以确保检索结果的绝对精准。
---
### [03:22](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=202s) 企业级落地痛点与构建 RAG 系统的四大核心挑战
Harini Gopalakrishnan 结合自身在赛诺菲（Sanofi）等制药企业处理多模态数据以及长期在科技行业服务企业 AI 的经历，指出了企业从 POC（概念验证）走向大规模生产落地时必然遭遇的瓶颈。很多企业虽然能轻松构建出处理几千份文档、几百万向量的初级聊天机器人，但面对复杂的现实生产环境，往往会在以下四个维度面临挑战：
```mermaid
flowchart TD
accTitle: Four Core Challenges in Enterprise RAG
subgraph P1["1. 规模化与多模态数据"]
direction TB
A1["超越纯文本分块"] --- A2["解析含复杂图表的 PDF/海报"]
end
subgraph P2["2. 相关性调优与精准排序"]
direction TB
B1["告别复杂 Prompt Engineering"] --- B2["多阶段匹配与业务规则结合"]
end
subgraph P3["3. 长推理链中的错误传播"]
direction TB
C1["推理智能体 (Reasoning Agents)"] --- C2["首步检索失误引发连锁错误"]
end
subgraph P4["4. 企业级合规与计算成本控制"]
direction TB
D1["VPC / 本地私有部署"] --- D2["原生张量结构降低算力开销"]
end
P1 --> P2 --> P3 --> P4
```
- **多模态与复杂版面处理能力（Multimodal Data Scaling）**：企业数据绝不仅是扁平的纯文本，绝大多数高价值信息存在于学术会议海报、论文 PDF 中，往往嵌入了“图表内含表格、表格内含图像”的嵌套多模态结构。普通向量数据库擅长做纯文本 NLP 分块，但难以解析这种复杂的复合视觉-文本数据。
- **相关性调优（Relevancy Tuning）**：当企业脱离 MVP 阶段后，相关性排序将成为分水岭。以 Perplexity 为例，其核心壁垒并非自研基础大模型，而是通过索引全网外部内容，建立极其强大的相关性检索与排序体系。系统不应强求终端用户通过复杂的提示词工程（Prompt Engineering）去迁就检索器，而是应该能直接理解用户意图并给出精准排序。例如在生命科学领域查询“针对特定国家的司美格鲁肽（Semaglutide）等减重药物”时，检索系统必须能原生结合地理、监管规则进行精细重排。
- **长推理链中的错误级联（Reasoning Agents & Error Propagation）**：在基于 MCP 等长链路推理智能体（Reasoning Agents）的工作流中，第一步检索若返回了错误信息，该错误会在后续的第三步、第四步中被成倍放大，导致最终结果完全偏离。
- **企业合规与成本效益（Enterprise Fitment & Cost / ROI）**：强监管行业对数据出境或调用第三方 API 极度敏感。此外，存储海量图像与原始 PDF 的成本极高。通过将多模态数据转化为张量（Tensor）表示，并利用原生多阶段排序（Multi-phase Ranking）减少昂贵的重排 EC2 运行开销，才能在保障精度（Precision）的同时实现极佳的投资回报率（ROI）。
---
### [08:26](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=506s) 经过十余年工业级验证的张量搜索引擎：Vespa AI
针对上述挑战，Bonnie 阐释了 Vespa 的技术定位及其在 AI 生态中的差异化优势。Vespa 并非伴随大模型浪潮新近成立的通用包装层，而是原本作为雅虎（Yahoo）内部的核心搜索架构运行了十余年（例如支撑 Yahoo Finance 等核心产品），并在 2023 年底正式拆分独立运营。
市场上常见的方案往往是两类系统的生硬拼凑：一类是以 Elasticsearch 为代表、源自 Lucene 体系的传统倒排索引搜索引擎；另一类则是近年涌现的新型向量数据库。这两类产品都在试图向对方的领地延伸扩展。然而，Vespa 从诞生之初便将二者深度融合，是一个原生支持十亿级（Billion-scale）大规模数据与张量计算的信息检索及 AI 应用构建平台：
- **原生张量数据库（Native Tensor Database）**：超越单一向量（Single Vector），原生支持张量（Tensor，即向量数组或高维矩阵）。不论是图像、蛋白质结构、化学分子式还是 PDF 表格，都能在统一的张量空间中完成高效编码与检索。
- **原生多阶段搜索与机器学习排序（Multi-phase Search & ML Ranking）**：在引擎底层原生集成高精度匹配与机器学习重排，摆脱了外部多组件调用的高昂延迟与资源开销。
- **灵活的企业级部署架构**：支持在全球三大公有云（AWS、Azure、Google Cloud）及企业本地私有化环境（On-prem / VPC）内无缝部署，确保高价值敏感数据资产不外流。
- **头部工业级客户背书**：除了长期支撑雅虎业务外，Vespa 目前是 Perplexity（AI 搜索底座）、Spotify（推荐与搜索）以及 Vinted 等众多处理海量高并发业务平台的底层核心引擎。
### [09:25](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=565s) 超大规模数据检索：从 Perplexity 到 Spotify 的生产实践
在构建现代 RAG（检索增强生成）与智能体系统时，检索与重排早已不再是简单的后端辅助工具，而是决定决策准确性的核心引擎。Bonnie 强调，在深入架构细节之前，必须正视企业级应用所面临的真实数据规模与严苛要求。以 Vespa 的典型企业客户为例，AI 搜索引擎 Perplexity 在项目起步阶段就为 1.5 亿（1.5 billion）份文档建立了索引，其扩展目标更是高达 100 亿（10 billion）文档；而全球流媒体巨头 Spotify 面对数以百万计的活跃用户，不仅管理着庞大的音乐库，还将播客音频中的“每一个词”都进行了全文本索引。这意味着即使听众记不清播客节目的确切名称，仅凭脑海中记得的一句对白或片段内容，系统也能精准定位并秒级召回对应的音频片段。
对于驱动自主行动的 AI Agent 而言，“低延迟、高精度与海量数据吞吐”构成了不可妥协的三要素。如果底层检索系统无法在毫秒级时间内提供准确无误的上下文切片，上层智能体就无法做出正确的业务决策与行动执行。
- **Perplexity 规模**：初始索引量达 15 亿文档，目标扩展至 100 亿级超大规模语料。
- **Spotify 深度索引**：对海量播客实现逐字（every single word）级深度索引，支持任意对话内容的逆向模糊定位。
- **Agent 核心诉求**：极低延迟与高精度的语义召回是智能体执行确定性动作的前提。
---
### [10:40](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=640s) 原生多阶段重排与高维张量：突破传统向量数据库的瓶颈
Bonnie 指出，传统向量数据库或拼凑型检索系统面临着严重的架构缺陷——检索与重排（Ranking）往往割裂在完全独立的系统之间。在这种割裂模式下，系统必须通过网络在不同服务间反复搬运大量候选数据来计算相关性评分，这不仅带来了巨大的网络延迟与算力浪费，更推高了基础设施成本。Vespa 的解决思路是将“多阶段重排（Multi-phase Ranking）”直接下沉到数据存储的原生位置：首先在存储层利用轻量级、低成本的机器学习模型对海量候选集进行快速粗筛与过滤，随后仅将精简后的头部候选结果提交给高精度、大算力的复杂深度学习模型进行精细化打分重排，在严格控制成本的同时实现极高的相关性精度。
```mermaid
flowchart TD
accTitle: Vespa Native Multi-phase Ranking Architecture
subgraph StorageLayer["Vespa 数据原生存储与计算层"]
A["海量文档 / 张量索引库"] --> B["第一阶段：轻量快速 ML 模型<br/>(大规模粗筛过滤)"]
B --> C["候选结果子集 (Top-K)"]
C --> D["第二阶段：高精度复杂 ML 模型<br/>(原位精细重排)"]
end
D --> E["高精度上下文切片<br/>(毫秒级返回 Agent/LLM)"]
```
除了原位重排，传统向量数据库在表达复杂数据结构时也已“撞上物理瓶颈”。普通向量仅支持一维表示，强行将复杂的多模态信息（如长文本结构、高分辨率图像局部、长视频时序）压缩为单一片段时，会剥离掉绝大多数关键上下文。高维张量（Tensors）实质上是多维向量框架，能够原生保留多维拓扑与时空上下文。例如在视频检索场景中，传统方案往往需要打各种“黑客式补丁”（hack things together）才能勉强定位，而高维张量能够原生支持在海量视频帧中直接检索并输出某个关键事件发生的“精确时间戳”。
- **数据与计算共存**：消除检索数据库与重排模型之间的网络数据搬运，从根源上降低延迟与成本。
- **多阶段打分机制**：先用低开销算法过滤海量噪音，再用高精度模型精排 Top-N，兼顾吞吐与精确度。
- **张量（Tensor）突破一维局限**：支持保留时序、空间和复杂业务属性的多维上下文，无需拼接繁琐的折中方案。
---
### [12:57](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=777s) RAG 的三层解耦范式与 ColPali 视觉语言模型
Harini Gopalakrishnan 接过话筒，首先解答了业界最常见的疑问：“为什么不能直接用一个万能的基础大模型（Foundational Model）解决所有问题？”她指出，随着 RAG 架构的演进，必须建立清晰的“三层解耦范式”：
1. **数据层（Data Corpus）**：包含非结构化文档、表格、研究海报以及结构化数据源。
2. **高维知识库（Knowledge Base）**：作为整个系统的核心中枢，负责在运行时动态注入丰富上下文。
3. **推理模型层（Model Layer）**：包括嵌入模型、视觉语言模型（VLM）及生成摘要模型。
```mermaid
flowchart LR
accTitle: Three-Paradigm RAG Architecture
subgraph DataTier["1. 数据层"]
D1["非结构化语料<br/>(PDF/图像/海报/研报)"]
D2["结构化数据<br/>(Text-to-SQL/数据库)"]
end
subgraph CenterTier["2. 核心中枢"]
KB["高维张量知识库<br/>(Vespa 知识中枢)"]
end
subgraph ModelTier["3. 模型推理层"]
M1["嵌入/交互模型<br/>(ColPali / ColBERT)"]
M2["前沿大语言模型<br/>(Qwen / LLM)"]
end
DataTier --> KB
KB <--> ModelTier
```
将模型与知识库解耦带来了两大颠覆性优势：其一，企业无需不断追求堆砌模型的本体参数规模，小模型配合智能高维知识库即可在运行时获得卓越表现；其二，底层知识库可随业务实时更新，而模型层可即插即用——今天接入 Qwen，明天切换至 ColBERT 或各类新型 VLM。
为验证这一架构在极端复杂场景下的威力，Harini 展示了基于生命科学临床试验文献的实时检索演示。系统直接摄入临床试验报告、排版复杂的科研 PDF、海报及品牌宣传文档，利用开源的视觉语言晚期交互模型（Late Interaction Model）**ColPali** 将每个 PDF 页面切分为多维图像块（如 30×30 的图像 Patch 像素块），并将每个图像块的稠密表征整体转化为高维张量存入 Vespa。
---
### [14:50](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=890s) 复杂文档视觉检索实例与生命科学多模态前沿
Harini 在未剪辑的现场实测中输入了一系列极具挑战性的生物医药专业查询：
- **复杂表格内靶向检索**：查询抗凝药“华法林（Warfarin）”的有效性数据，系统不仅从海量论文深处的复杂表格中召回了数据，更借助晚期交互的最大相似度算子（MaxSim），在原始 PDF 页面上精准用高亮框标出了华法林所在的那一行表格单元格。
- **图表语义理解**：针对当前热门减重药物司美格鲁肽（Semaglutide）与替尔泊肽（Tirzepatide）的“成本效益分析”查询，文档原文中甚至从未出现“Cost（成本）”一词，但 Vespa 与 ColPali 准确识别了摘要折线图中的专业指标词“ICER”（增量成本效果比），直接定位并高亮召回该图表。
- **海报图形内容提取**：针对减重百分比的查询，数据深埋在科研海报内带箭头的图形流程图文本中，系统无需任何前置 OCR 工程清洗，直接抓取图形内的具体百分比并生成精准对比摘要。
- **缩写图表解析**：查询心血管不良事件指标“MACE（主要不良心血管事件）”时，系统瞬时在柱状图与曲线图中定位该缩写并提炼出受试者的风险下降百分比。
"整个检索过程完全是亚秒级（subsecond）的，我们没有任何经过后期剪辑的演示成分。" Harini 强调，该原型没有复杂的胶水代码工程，仅仅是将 ColPali 张量化后存入 Vespa 并微调了相关性。
除了文档与图表，张量检索正在向更深度的生命科学与工业前沿渗透。Harini 提到了 DeepMind 凭借蛋白质三维结构预测模型 AlphaFold 斩获诺贝尔化学奖的里程碑事件，AlphaFold 的核心同样是 Transformer 架构。通过类似模型将蛋白质立体空间结构转化为稠密张量嵌入，并与临床元数据、注释信息联合索引，研究人员可以在毫秒内完成靶点抗体匹配、蛋白质子结构相似度搜索、以及肿瘤病理切片（Histopathology Slides）与癌变区域图像的细粒度多模态对比（如 Landing AI 的病理分割检索实践）。这种能力同样适用于零售电商中的细粒度以图搜图（如精准匹配特定款式与材质的红色高跟鞋），彻底重塑产业级多模态智能。
### [18:55](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1135s) 多模态张量在生命科学与复杂文档检索中的落地应用
在生命科学与医疗健康领域，高维张量检索展现出了重塑诊断与研究流程的巨大潜力。以病理图像分析为例，系统可以直接对图像的每一个像素进行索引，并标记相似的癌细胞特征。当新患者就诊或医生评估疾病进展情况时，无需每次都运行繁琐复杂的图像分类与目标检测大模型，而是可以直接将其转化为检索问题——即在历史知识库中查询“我以前在语料库里见过类似的图像吗？”（"have I seen this image before from my corpus before"），从而快速调取相似病例的既往病程、诊断结果与预后表现。
```mermaid
flowchart TD
accTitle: 多模态张量统一检索架构
subgraph MultiModalInput["多模态实体输入"]
A["病理切片 / 像素图像"]
B["PDF / 图表 / 幻灯片"]
C["蛋白质结构 / 分子式"]
end
subgraph TensorEngine["Vespa 张量计算引擎"]
D["统一页面 / 实体张量构建"]
E["跨模态联合相关性计算"]
end
subgraph Outputs["下游智能决策"]
F["预后 / 诊断结果比对"]
G["关联多模态问答"]
H["智能体上下文生成"]
end
A --> D
B --> D
C --> D
D --> E
E --> F
E --> G
E --> H
```
对于企业中普遍存在的复合型长文档（如包含图表、PowerPoint 幻灯片和扫描件的 PDF），张量框架同样提供了优雅的解决方案。系统无需拆碎排版，而是为每一个文档页面构建复杂的长效张量（long acting tensor），直接实现跨图文的高亮查询与定位。更重要的是，在生物医药研发场景下，张量框架能够将蛋白质结构、分子数据与科研文献文本统一嵌入到单个张量结构中。研究人员只需提出一个跨领域问题，系统便能同时贯通分子性质、生物结构与文本上下文进行联合召回，这对于医疗科技与生命科学企业而言至关重要。
- **像素级病理索引**：通过检索历史库中相似的癌细胞形态特征，直接复用既往的预后与病程数据，简化临床辅助诊断。
- **页面级统一张量**：无论是统计图表、幻灯片还是复杂排版的行业 PDF，单页即可表示为高维张量，无需割裂结构化与非结构化视觉元素。
- **多实体联合表征**：将文本、蛋白质立体结构及化学分子结构压缩进统一的张量空间，支持多维度跨模态穿透查询。
---
### [19:48](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1188s) 构建智能体架构与下一代 RAG 的三大核心原则
要构建真正具备生产可用性的智能体（Agentic Systems），必须将分析型数据系统与非结构化数据处理紧密结合。基于在数据仓库领域的深厚经验（如前 Snowflake 的实践），结构化分析中的 Text-to-SQL 与企业数据编织（Fabric）依然不可或缺；但更决定智能体上限的，是以高度智能化的方式处理海量非结构化数据。系统必须深度融合多模态相关性算法与张量数据表示，并通过新兴的智能体框架或 MCP（Model Context Protocol，模型上下文协议）架构将分析系统与检索链路无缝串联。
在总结分享时，Bonnie Chase 为现代 RAG 与检索架构的设计提出了三项必须牢记的核心原则：
1. **计算必须贴近数据（Compute where the data resides）**：为了实现高准确度、低延迟与高扩展性，避免在网络中来回搬运海量高维数据，必须在数据存储层就地完成张量运算与打分。
2. **搜索始终是 RAG 的核心本质**：无论是何种智能体工作流，精准召回信息才是决策基石，系统必须原生具备跨越文本、视觉等多种模态的联合搜索能力。
3. **传统单向量表征已达瓶颈，必须转向张量（Tensors）**：简单向量与矩阵无法承载复杂的企业级现实关系。Vespa 团队为此专门撰写了关于“张量形式化”（Tensor Formalism）的学术论文，深入探讨如何利用弹性张量突破传统向量检索的维度限制。
演讲最后强调，以张量为底座、计算存储一体化的新型检索架构，不仅是破解 RAG 幻觉与性能瓶颈的必经之路，更是未来整个企业级 AI 系统的核心基础设施。
## Notable quotes
> As AI agents and retrieval-augmented generation (RAG) become more advanced, search & retrieval has moved from a backend tool to the engine driving intelligent decisions.
> Agentic workflows depend on asking the right questions, making smart decisions, and taking actions—all of them depend on accurate information retrieval. This is why we need to rethink how RAG and AI agents are designed - we need to frame it as a search problem.
> Traditional methods like basic vectors and matrices fall short, especially in complex fields like life sciences, drug discovery or in e-commerce. That’s where high-dimensional data structures come in.
> [14:50](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=890s) 整个检索过程完全是亚秒级（subsecond）的，我们没有任何经过后期剪辑的演示成分。 — Harini Gopalakrishnan
> [18:55](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1135s) have I seen this image before from my corpus before — Harini Gopalakrishnan
> [19:48](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1188s) Compute where the data resides — Bonnie Chase
> [19:48](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1188s) 搜索始终是 RAG 的核心本质：无论是何种智能体工作流，精准召回信息才是决策基石，系统必须原生具备跨越文本、视觉等多种模态的联合搜索能力。 — Bonnie Chase
> [19:48](https://www.youtube.com/watch?v=YLSpw6rJcXw&t=1188s) 传统单向量表征已达瓶颈，必须转向张量（Tensors）：简单向量与矩阵无法承载复杂的企业级现实关系。 — Bonnie Chase
## Mentioned
Bonnie Chase, Harini Gopalakrishnan, Vespa.ai, Vespa, Yahoo, Yahoo Finance, Perplexity, Spotify, Vinted, Sanofi, Snowflake, Landing AI, DeepMind, AlphaFold, ColPali, ColBERT, Qwen, ChatGPT, Elasticsearch, Lucene, AWS, Azure, Google Cloud, MCP (Model Context Protocol)
## Video description
Bonnie Chase, Director of Product Marketing, Vespa.ai
Harini Gopalakrishnan, General Manager, Industry, Vespa.ai
Abstracts
As AI agents and retrieval-augmented generation (RAG) become more advanced, search & retrieval has moved from a backend tool to the engine driving intelligent decisions.
Agentic workflows depend on asking the right questions, making smart decisions, and taking actions—all of them depend on accurate information retrieval. This is why we need to rethink how RAG and AI agents are designed - we need to frame it as a search problem.
Traditional methods like basic vectors and matrices fall short, especially in complex fields like life sciences, drug discovery or in e-commerce. That’s where high-dimensional data structures come in.
In this talk, Vespa will introduce a new approach using high-dimensional data structures (vision-language models) with a flexible tensor framework and accuracy in retrieval with custom ranking. We’ll show how these tools power better search—and how you can start using them today.
[Source](https://www.youtube.com/watch?v=YLSpw6rJcXw)
Source quotes
As AI agents and retrieval-augmented generation (RAG) become more advanced, search & retrieval has moved from a backend tool to the engine driving intelligent decisions.
Agentic workflows depend on asking the right questions, making smart decisions, and taking actions—all of them depend on accurate information retrieval. This is why we need to rethink how RAG and AI agents are designed - we need to frame it as a search problem.
Traditional methods like basic vectors and matrices fall short, especially in complex fields like life sciences, drug discovery or in e-commerce. That’s where high-dimensional data structures come in.
[14:50] Harini Gopalakrishnan: 整个检索过程完全是亚秒级（subsecond）的，我们没有任何经过后期剪辑的演示成分。
[18:55] Harini Gopalakrishnan: have I seen this image before from my corpus before
[19:48] Bonnie Chase: Compute where the data resides
[19:48] Bonnie Chase: 搜索始终是 RAG 的核心本质：无论是何种智能体工作流，精准召回信息才是决策基石，系统必须原生具备跨越文本、视觉等多种模态的联合搜索能力。
[19:48] Bonnie Chase: 传统单向量表征已达瓶颈，必须转向张量（Tensors）：简单向量与矩阵无法承载复杂的企业级现实关系。
Source quotes
As AI agents and retrieval-augmented generation (RAG) become more advanced, search & retrieval has moved from a backend tool to the engine driving intelligent decisions.
Agentic workflows depend on asking the right questions, making smart decisions, and taking actions—all of them depend on accurate information retrieval.
Traditional methods like basic vectors and matrices fall short, especially in complex fields like life sciences, drug discovery or in e-commerce. That’s where high-dimensional data structures come in.
[3:22] Harini Gopalakrishnan: prompt engineer the heck of it
[3:22] Harini Gopalakrishnan: secret sauce
[17:50] Harini Gopalakrishnan: Have I seen this image before from my corpus before, and then see what the prognosis or diagnosis or progression was?
[19:48] Compute Where the Data Resides
[19:48] Search is the Fundamental Core
[19:48] Move Beyond Flat Vectors to Tensors
[19:48] When you're wanting to build a RAG system that's accurate, low latency, and high scale, you want to be able to compute where the data resides. You need to remember that search is still the core problem, and you need to be able to search across modalities. And remember, vectors have a limit. Start looking at tensors.
Original transcript
[0:00] All
[0:04] right, awesome. Hello everyone. Thank
[0:07] you so much for joining us today. My
[0:09] name is Bonnie and I'm with Vespa AI and
[0:11] I'm joined by my colleague Harini. And
[0:14] today we're going to be talking about
[0:15] rag reimagined and ultimately this is
[0:18] really about reframing how we think
[0:20] about rag as a search problem. So I'll
[0:23] be sharing a little bit of context
[0:25] setting talking a little bit about how
[0:28] the Vespa perspective differs a little
[0:30] bit from what you see in today's vector
[0:32] databases and Harini will be digging
[0:34] into some industry examples starting
[0:37] with life sciences and we'll also share
[0:39] a few other examples as well.
[0:44] So, just to kind of kick it off and set
[0:47] the stage a bit, you know, I want to
[0:49] start by talking a little bit about the
[0:51] shift in AI and kind of what we've been
[0:53] seeing. And obviously, we know chat GPT
[0:57] kicked off a bunch of really cool things
[0:58] that everybody got everybody interested
[1:00] in leveraging AI, interested in
[1:03] retrieval augmented generation or rag.
[1:06] And now we're seeing the shift that's
[1:07] moving from just retrieving information
[1:10] to really making decisions and taking
[1:12] action with that information. Now these
[1:15] systems which we call agentic AI or
[1:17] agentic systems or just agents they
[1:20] really need to do three things well.
[1:22] They need to be able to ask the right
[1:24] questions. They need to be able to make
[1:27] good decisions and they need to be able
[1:29] to make informed actions. Now, all of
[1:32] this relies on having accurate,
[1:35] relevant, and timely information
[1:37] retrieval. And that really is the core
[1:39] of the issue that we're talking about
[1:41] today. If that information retrieval is
[1:43] broken, then the rest of the system
[1:45] breaks as well.
[1:48] Now, when we think about uh retrieval,
[1:52] this really is more about search. So, if
[1:55] you get that wrong in the beginning,
[1:57] that's where that starts to fail. So
[1:59] when you think about a health assistant
[2:02] for example, if that health assistant is
[2:04] getting outdated trial information, it's
[2:07] going to be giving the wrong information
[2:08] to those users. An e-commerce agent, if
[2:11] they're giving the recommendations for
[2:14] um outofstock products, again that
[2:17] system is broken. So we want to make
[2:19] sure that those systems are are starting
[2:22] with the right foundation, which is
[2:24] search and retrieval.
[2:26] many systems today that we see with
[2:28] basic rag they they start with a vector
[2:30] database that goes off of simple
[2:33] similarity uh single embeddings and when
[2:37] it comes to more advanced complex
[2:39] systems like what you need in healthcare
[2:42] or financial systems that's where it
[2:44] really starts to break down because
[2:45] those simple vector databases can't
[2:47] really manage the complex rules they're
[2:50] not able to handle the multimodal data
[2:52] that's required and so that's where we
[2:54] really see it start breaking down.
[2:58] So again, this goes back to Rag as a
[3:00] search problem. Search is more than
[3:03] just, you know, the backend system that
[3:06] we've kind of gotten used to in the
[3:07] enterprise. This is really about having
[3:10] that additional context, being able to
[3:11] pull all of that information together so
[3:14] that we're able to retrieve the right
[3:15] information.
[3:18] So I'm going to let her talk a little
[3:19] bit more about why this is relevant for
[3:21] the enterprises.
[3:22] >> Yeah. So uh just for context right we
[3:25] quickly Bonnie is going to cover what's
[3:26] Vespa AI but just for context it's been
[3:28] it's spun out of Yahoo it's been around
[3:30] with Yahoo and it's actually been the if
[3:32] you use Yahoo Finance ever all of the
[3:34] data comes from the Vespa back end and
[3:36] now since it's spun off it's one of the
[3:38] backends that Spotify uses Perplexity
[3:40] had a great talk yesterday so it's also
[3:42] the back end for Perplexity so if you go
[3:44] to Vespa AI and Google for Perplexity
[3:46] you'll see the group blog on how it's
[3:47] being used so with that context being
[3:49] said uh I used to work with AI and
[3:52] enterprises a lot in my last two gigs in
[3:54] tech and I was also in Xanophi so worked
[3:56] in pharma and multimodal data all my
[3:58] life uh four things whenever I
[4:01] encountered chat bots right everybody in
[4:03] every industry has built a chatbot now
[4:05] the problem is not about building a
[4:07] basic chatbot it's about how does it
[4:09] scale so people have been able to bring
[4:12] thousands of probably documents probably
[4:14] millions of embeddings but does it scale
[4:16] that's the number one problem how do you
[4:18] actually make it work for a corpus
[4:20] that's multimodal that's not just text
[4:22] and chunking and complex pre-processing.
[4:25] Second is every time I've talked to
[4:27] customers, it comes down to PDFs or
[4:29] posters in conferences that they go to
[4:32] uh where there's images inside a table
[4:34] inside a graph and you want the model to
[4:36] be able to interpret that not just
[4:38] extract the text and do NLP. Most vector
[4:41] databases including like bedrocks or the
[4:43] world that connect do very well when the
[4:45] data is text but the question is how do
[4:47] you scale beyond that right? So that's
[4:48] the second problem with scale. Third is
[4:51] something probably enterprise haven't
[4:53] hit on yet. It's called relevancy
[4:54] tuning. Probably because you haven't
[4:56] asked really good or accurate questions
[4:58] yet because you're still in the MVP
[5:00] phase. But the complexities of the world
[5:03] are unique because of the main factor
[5:05] called relevancy tuning. Like if you
[5:07] look at how publicity works, right? They
[5:09] haven't built their own unique
[5:10] foundational model. Maybe from
[5:12] yesterday's talk it looks like they did
[5:13] something but their biggest secret
[5:14] source is they leverage a lot of the
[5:16] open source model. the in the secret
[5:18] source is in they index all of the
[5:20] external content and how they actually
[5:22] create a relevant information based on
[5:24] your question. You don't ask the user to
[5:26] go prompt engineer the heck of it to ask
[5:28] a question the way you want the answer
[5:30] but your question should be interpreted
[5:33] and the database that supports the
[5:34] knowledge corpus should be able to
[5:37] create a relevancy rank based on what is
[5:39] asked that's the secret source. So if
[5:42] all of you building these rack systems,
[5:44] if you're not getting the most relevant
[5:45] document, your point in life sciences,
[5:47] you're asking a question about obesity
[5:49] drug like summer diet and you want to
[5:51] get it for a specific country, you
[5:53] should be able to use that in your
[5:54] ranking and that's one of the reasons
[5:56] why Vespa differs and you'll see a demo
[5:58] of where that comes in as well. Last
[6:00] point, reasoning agents. Everybody's
[6:03] talking about MCP long reasoning agents.
[6:05] If you get the wrong answer in your
[6:07] first question, that's going to get
[6:08] propagated all the way to the fourth
[6:09] step. So as you build these log
[6:11] reasoning agents the biggest thing is
[6:13] going to be the topic point above which
[6:15] is getting the relevant answer in the
[6:16] first and how do you make that relevant
[6:19] how can you customize relevancy it's not
[6:21] just by doing a simple semantic uh
[6:23] scoring you know there's something more
[6:24] beyond that and uh hopefully in the demo
[6:26] and after the demo you can talk and we
[6:28] are also having a boot so please feel
[6:29] free to stop by um with that I'm going
[6:33] to pass on to Bonnie to explain a bit
[6:34] about Vesper while we come back to see
[6:36] some of the life science or the
[6:37] enterprise use cases but I want to leave
[6:40] you with these four things that is a
[6:41] teaser for what best properties in right
[6:43] first is multimodal data handling the
[6:46] important thing is you got to think
[6:47] beyond a single vector it has to be a
[6:49] tensor which is an array of vectors so
[6:52] whether it's an image a protein model a
[6:54] chemistry structure or a table inside a
[6:56] PDF you need to be able to embed all of
[6:58] that as a vector of vectors is a native
[7:01] tensor database so you can actually
[7:02] encode all of it into a single texture
[7:04] and do search on that you're not doing a
[7:06] search by search for every single vector
[7:08] point Second, precision retrieval,
[7:11] relevancy tuning. That can be its own
[7:13] topic, but there are machine learning
[7:14] models that can also help you with
[7:16] ranking. Vespa can natively support
[7:18] that. Uh it's called multiase search. Um
[7:20] but just leaving it out there. Uh Bonnie
[7:23] might give you a good teaser. Third is
[7:25] enterprise fitment. I work with
[7:26] regulated industries. You know how
[7:28] important it is not to take the data out
[7:30] or send it to a model outside. Um so
[7:33] runs on all three clouds and on prem. So
[7:36] the data that you would parse it remains
[7:38] with you in your VPC or in your own
[7:40] prem. Last is cost efficiency. ROI is
[7:43] the name of the game. Uh everybody's
[7:45] looking to optimize cost, quality and
[7:46] performance and one of the indexes goes
[7:48] out of balance. U there are ways to do
[7:51] it, right? Storing the data as a tensor
[7:53] already reduces your cost because you're
[7:55] not keeping pabytes of images or pabytes
[7:57] of PDF. You're already converting it to
[7:59] numbers. If you ask the right question
[8:01] or if you reduce the reranking step,
[8:03] you're also reducing engineering cost
[8:05] and EC2 runtimes. It gets into some more
[8:07] technicalities of how it's implemented,
[8:09] but overall the multiase ranking that
[8:11] Vespa natively offers should bring the
[8:13] ROI down without affecting the quality
[8:16] or precision of the result. That being
[8:18] said, uh just a teaser of what Vespa's
[8:20] four key capabilities are, but Bonnie's
[8:22] going to run through uh some of this in
[8:24] detail and we'll come back to seeing a
[8:25] demo in action.
[8:26] >> Yeah. And and so I'll I'll share a
[8:28] little bit about Vesa just because as
[8:30] Harini said, you know, Vespa spun out of
[8:32] Yahoo at the end of 2023. So we are a we
[8:35] are technically a startup now, but the
[8:36] product has been around for over 10
[8:38] years. And when you think about the
[8:40] space right now with Rag and what people
[8:42] are looking for, we see a lot of people
[8:44] combining search engines with vector
[8:46] databases. So you have those lucine
[8:48] based engines like elastic search that
[8:50] really came from a search perspective.
[8:52] You have those vector databases that
[8:54] started from a vector perspective and
[8:56] now they're trying to layer in and add
[8:58] on all these additional capabilities
[9:00] that have been core to Vespa for over 10
[9:02] years.
[9:04] So Vespa is a platform for building AI
[9:07] applications that really depend on
[9:09] largecale data and information
[9:11] retrieval. So we're talking billioncale
[9:14] search. Um we're talking platforms like
[9:17] Perplexity, Spotify, Vintage. These are
[9:20] some of the customers who use Vespa for
[9:23] their uh search and personalization
[9:25] capabilities today. Now, I won't dig
[9:28] into each one of these boxes, but there
[9:31] are two boxes that I will dig into that
[9:33] are relevant to the RAG conversation
[9:35] today, and that is the um the the
[9:38] ranking and the tensor operations.
[9:42] Before I do that, you know, we've been
[9:44] talking about some of these uh customers
[9:46] that we've been working with. You've
[9:47] seen perplexity here today. And when
[9:50] we're talking largecale data, we're
[9:52] talking, you know, Perplexity, for
[9:54] example, they index 1.5 billion
[9:56] documents when they started with a goal
[9:58] of 10 billion. When you think about
[10:00] Spotify and the the number of users, the
[10:04] millions of users that they have and all
[10:06] of the songs and podcasts that they
[10:08] have, they've actually indexed every
[10:10] single word of every single podcast so
[10:13] that when you don't remember the name of
[10:15] a podcast, for example, and you search
[10:17] by something that you heard within the
[10:19] podcast, you can still get the results.
[10:21] So here again, we're talking about low
[10:24] latency, we're talking about high
[10:25] accuracy, and we're talking about
[10:27] largecale data. And that's really what
[10:29] these rag systems need, especially when
[10:31] it comes to um agents that require the
[10:35] right information in order to take the
[10:37] right action.
[10:40] So, one of the key areas is this
[10:42] advanced retrieval and ranking. And what
[10:45] we're seeing today with some of these um
[10:47] these other systems, vector databases or
[10:49] whatever it is, a combination of the two
[10:51] that you're using is that the retrieval
[10:54] and ranking happens in different
[10:56] systems. And at Vespa, we see that as a
[10:59] core problem because you're sending data
[11:01] across these systems to be able to um to
[11:05] be able to even get relevancy. So again,
[11:08] you're adding latency, you're adding
[11:09] cost. Um we also think it's really
[11:12] important to be able to do the
[11:14] multi-phase ranking. So that multi-phase
[11:17] ranking happens where the data resides,
[11:19] not in a separate system. And it does
[11:21] the lowerc cost machine learning models
[11:23] first to narrow down those results. and
[11:26] then you use those more expensive models
[11:28] on a smaller set of data to provide that
[11:31] increased accuracy. So just a little bit
[11:34] about the the retrieval and and ranking
[11:36] there. I won't go into too much detail,
[11:38] but you'll see it on the slide. The next
[11:41] piece here is the advanced tensor
[11:43] support. And I think you will all be
[11:45] hearing a lot more about tensors in the
[11:48] coming years because this is where
[11:49] vector databases really hit a wall. Um,
[11:52] you know, tensors really are
[11:54] multi-dimensional vectors. Vectors work
[11:57] with onedimensional data. So, it strips
[12:01] away a lot of context. So, if you're
[12:03] trying to embed whether it's text,
[12:05] images, or videos, you're only getting
[12:07] one dimension of that data set. So, for
[12:11] example, you want to be able to search
[12:14] within a video. and you want to not just
[12:16] search for something within a video, but
[12:18] get that exact timestamp where that
[12:20] piece of information resides. You can't
[12:22] do that with a vector database. You have
[12:24] to kind of hack things together, add a
[12:26] little bit more details, and kind of
[12:28] force it to to be able to do something
[12:30] with that. But that's where tensors come
[12:32] in because you can really specify those
[12:34] very specific complex business needs um
[12:38] in order to bring those results and
[12:40] bring that context into into that
[12:43] search. Now I think it would be really
[12:46] helpful for you to see kind of this in
[12:48] action. So I will have Harini kind of
[12:50] she'll walk you through a specific
[12:51] example using life sciences um where you
[12:54] can see how all this comes together.
[12:57] >> So just before we get into that demo and
[13:00] also tensors mean a lot more in the life
[13:02] science arena but what we always get
[13:05] this question is like but can't a
[13:07] foundational model do that so I want to
[13:08] set the stage by saying that as you
[13:11] evolve your rack there has to be a three
[13:12] paradigm. The first one is your data
[13:16] which is your corpus of unstructured.
[13:17] Now you can also have structured data
[13:19] with text to SQL you know co-pilots
[13:21] operating on it but there is a large
[13:23] stream of information in unstructured
[13:25] data which is what companies are trying
[13:26] to tackle. So you have to have the data
[13:29] separate from the model which could be a
[13:31] embedding model which is also your
[13:32] summarization model and most important
[13:35] is a centerpiece which is your knowledge
[13:36] base. Right? So your model doesn't have
[13:39] to be intelligent by itself. You can
[13:40] have a small model now but have your uh
[13:43] intelligent knowledge base of your
[13:44] unstructured corpus recite and add the
[13:47] context at runtime which is what rag is
[13:49] all about right. So separating that does
[13:52] two things. You can as you get more and
[13:54] more data your knowledge base can be
[13:56] updated and if you have a
[13:57] highdimensional database you can do more
[13:59] cooler questions with it and as the
[14:01] state-of-the-art model evolves you can
[14:02] plot it can be quen today it can be um
[14:05] cober tomorrow you can play with vision
[14:07] language models is more possibility so
[14:09] having these three things are extremely
[14:11] important and not just about making the
[14:12] model more and more intelligent. So if
[14:14] you can hit play um for this demo to
[14:17] work thanks. So what I've done is we
[14:20] poked a bunch of clinical trial
[14:21] documents, complex PDFs and posters and
[14:24] brand uh PDFs threw it into uh vision
[14:27] language and late interaction model.
[14:29] It's called Calpali. It's open source
[14:31] that basically converts each of this PDF
[14:34] into patches basically treats it like an
[14:36] image and because each page has hundreds
[14:39] of images because you're actually
[14:41] creating it like a pixel let's say a 30
[14:43] by 30 patch of each page each of that is
[14:46] embedded. So each page becomes a tensor
[14:48] of multiple dimensions. So now when I
[14:51] ask a question, I'm asking a very
[14:52] complex question like find me warerin in
[14:55] one of the documents right and if what
[14:56] is efficacy data for that it actually
[14:58] found warfarein in a table and using the
[15:01] ability of something called maxim I can
[15:03] also highlight exactly in which page of
[15:05] which document in which table I found
[15:06] warfarein. It's not something that you
[15:08] can do easily with a regular vector
[15:10] model or with just a onedimensional
[15:13] database. Now I'm asking it for it a
[15:15] question like provide me cost analysis
[15:17] for two obesity drugs out there
[15:19] saglutire and triceptide if everybody's
[15:21] on weight loss u it found the
[15:23] information in a chart in one of the
[15:25] abstracts or one of the papers that was
[15:26] out there again it doesn't even say cost
[15:29] in fact it says a term called ICER the
[15:31] late interaction model can understand
[15:33] the context all of this is tensorized in
[15:35] vesper it finds the relevant information
[15:37] serves the exact page and highlights
[15:38] where it found it and you just use a
[15:40] genuine to give an answer that a user
[15:42] can understand that another pattern
[15:44] right we are now asking for an exact
[15:46] weight loss percentage that information
[15:48] was inside a poster in a couple of
[15:51] arrows we just hit run and we were just
[15:53] curious to see whether it would perform
[15:55] interestingly it found the percentage
[15:57] that was inside the text and inside the
[15:59] arrows and it was able to summarize the
[16:01] two comparison of the weight loss uh we
[16:04] didn't do any engineering we just took
[16:06] coali tensorized it threw it into
[16:08] westper did some tweaking on relevancy
[16:11] but that's it wasn't a complex build
[16:13] because we just built a demo. We weren't
[16:14] building a solution. Um and here we
[16:17] fine- tuned it to top three. So
[16:19] basically we said find the top three
[16:20] relevant results in summarize. You can
[16:21] make it top end like top 50 and you can
[16:23] also do recency. You can say the most
[16:25] recent one is more important. Last one I
[16:28] I was curious like mace is a medical
[16:29] term that actually stands for um
[16:31] cardiovascular adverse events. Right? It
[16:34] found mace inside a graph and told me
[16:35] what was the percentage reduction if
[16:37] somebody was on a cell diet. And uh
[16:40] again we just threw in the PDF. What you
[16:41] see is an actual PDF and actual term in
[16:44] the PDF and the graph that is summarized
[16:46] by the VLM and VP and all of this
[16:48] retrieval as you can see subsecond we
[16:50] are not playing around with edited
[16:52] demos. So pretty powerful as you think
[16:54] of scaling up rag. These are the kind of
[16:56] use cases we would love you to process
[16:58] and uh love to chat with you on this. Uh
[17:01] so stop by our booth, stop talk to us
[17:02] after the session, right? And we'd love
[17:04] to feed more complex information as
[17:06] well. Not um not just charts and PDF.
[17:09] I'm looking to see if we can do medical
[17:10] images like cancer segments,
[17:12] hystopathology slides, protein
[17:14] embeddings. uh combine the protein
[17:16] embedding find similar proteins or
[17:18] sub-segment in the protein with uh say
[17:20] another metadata that has annotations
[17:22] with it. Um so next slide. So these are
[17:26] some of the questions in life sciences
[17:28] we would aim to answer right find
[17:30] antibodies given a structure of a
[17:32] protein or find me images that are
[17:34] similar to that image. In retail it's a
[17:36] very common problem. Uh you want a shoe
[17:38] a red shoe you want to find similar red
[17:40] shoes. How do you tune relevancy? And
[17:42] Bonnie has a use case exactly that talks
[17:44] to it. Um I don't know how you're doing
[17:46] on time. So we have minutes. Yeah. Five
[17:50] minutes. Uh I'm going to skip. Uh before
[17:52] I give to Bonnie on closing, just want
[17:54] to say what are the tensors, right? So
[17:55] it's not just about text images inside
[17:57] PDF etc. Uh last year the Nobel Prize
[18:01] for chemistry was actually won for by
[18:03] deep mind uh for actually creating
[18:06] alphafold the protein folding algorithm.
[18:08] I think it was one of the first
[18:09] generative AI Nobel Prize awards even
[18:12] though it was given for chemistry. So
[18:14] what that lets you do is alpha fold is a
[18:16] transformer. Um you can actually use a
[18:18] similar transformer to take the
[18:20] structure of a protein convert into a
[18:21] dense embedding throw it into vector and
[18:24] you know u today the keynote I love how
[18:26] Jeffington talked about finding the next
[18:28] drug by using different modalities of
[18:31] the data to accelerate you know cancer
[18:33] treatments to ancillate drug new
[18:34] modalities of drug exactly what you need
[18:36] throw the modality in throw the metadata
[18:38] in and that's how you actually find
[18:40] similar patterns and you can define
[18:42] similarity in many ways not just dist
[18:44] distances same thing with medical image
[18:46] this actually an exact use case that
[18:48] landing AI was trying to build. Take a
[18:51] segment that's actually you see a actual
[18:53] pathology segment within the small
[18:55] subsection where you see one and two is
[18:57] probably the area of interest which is
[18:58] cancer. um again was in the keynote
[19:00] today. You can actually index every
[19:02] pixel and uh by labeling similar cancer
[19:06] cells, you can actually find if a new
[19:08] patient comes in or you're looking at
[19:09] disease progression instead of doing a
[19:11] complex classification or image
[19:13] detection, you can just find hey, have I
[19:15] seen this image before from my corpus
[19:17] before and then see what the prognosis
[19:18] or diagnosis or progression was.
[19:22] U last is the multimodal PDF we talked
[19:24] about brand documents. You can create a
[19:26] tensor per page. Doesn't matter whether
[19:27] it's a chart, graph, powerpoint, PDF.
[19:30] Create a complex long acting tensor and
[19:31] you can ask any questions to query it
[19:33] and highlight it and they all come
[19:36] together. I can embed in a single tensor
[19:38] the protein, the text, the molecule and
[19:40] ask a question that connects all three
[19:42] which is really really important for
[19:44] companies that are in health tech and
[19:45] life science space. Uh I'm going to skip
[19:48] through these slides. This is just a
[19:49] last slide on how do you really build an
[19:51] intelligent agent, right? You need an
[19:53] analytical system. I worked for
[19:55] Snowflake before so I've talked about
[19:56] this a lot. Text to SQL is very
[19:58] important. Fabric of the world are
[20:00] important. Unstructured data is
[20:02] extremely important and what you need to
[20:04] do is process this unstructured data in
[20:05] a smart way where you can leverage
[20:07] relevancy and multimodal and tensor
[20:09] information and you can string them
[20:10] together with an agent or MCP
[20:12] architecture. Um so remember those three
[20:16] things and I'll let Bonnie finish up
[20:18] with the e-commerce centers.
[20:19] >> Yeah, I mean at at the end of the day
[20:21] I'll I'll kind of summarize. I mean, you
[20:22] you all get these slides and you'll be
[20:24] able to to click through some of these
[20:25] links, but really when we think about
[20:27] rag today, I want you to come away with
[20:29] three things. One, when you're wanting
[20:32] to build a rag system that's accurate,
[20:34] low latency, and high scale, you want to
[20:37] be able to compute where the data
[20:38] resides. You need to remember that
[20:40] search is still the core problem, and
[20:42] you need to be able to search across
[20:44] modalities.
[20:45] And remember, vectors have a limit.
[20:48] Start looking at tensors. If you're
[20:50] curious, more curious about tensors,
[20:53] Vesa wrote a paper, some engineers at
[20:55] Vesba wrote a paper on tensor formalism.
[20:57] Um, and we really do see this as the
[20:59] future of RAG and the the future of AI
[21:02] systems. Thank you all.
[21:03] >> Thank you all.
