From Search to Synthesis: How We Built AI That Thinks Like a Research Team With Perplexity AI

From Search to Synthesis: How We Built AI That Thinks Like a Research Team With Perplexity AI — Ai4

Ai4

AI-assisted summary of From Search to Synthesis: How We Built AI That Thinks Like a Research Team With Perplexity AI — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 29, 2025, this session features Tony Wu, VP of Engineering and Research at Perplexity AI. According to the page description, the talk examines the architectural principles behind building autonomous, multi-modal research agents like Deep Research and Labs to replace traditional search workflows with expert-level synthesis.

## TL;DR Tony Wu outlines how Perplexity AI transitioned from conventional answer engines to autonomous research agents with products like Labs and the Comet browser. He details Perplexity's post-training infrastructure, LOTUS, which combines supervised trajectory fine-tuning with on-policy reinforcement learning guided by user Elo pairwise preferences. Wu also explains Perplexity's heterogeneous serving stack, highlighting the Iris routing system that decouples prefill from decode phases to optimize KV-cache reuse. The session concludes with technical discussions on mitigating context bloat through compaction versus sub-agent delegation, selecting open-weight embedding models, and using evaluator models for trajectory reward modeling.

## Atmosphere & style The presentation carries the polished, measured, and pragmatic tone of a frontier systems engineering breakdown. Wu speaks with understated authority, methodically walking through concrete architectural diagrams rather than relying on marketing hype. The atmosphere shifts into an engaging, technical dialogue during the Q&A, where Wu candidly discusses the operational trade-offs, bottlenecks, and failure modes of production-scale agentic AI.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=hs49d_LTkRc&t=4s) Perplexity's core product ecosystem spans direct answer search (Ask), exhaustive investigations (Deep Research), autonomous artifact generation (Labs), and agentic web browsing (Comet). - [01:40](https://www.youtube.com/watch?v=hs49d_LTkRc&t=100s) Agentic systems fundamentally shift research paradigms by compressing dozens of disjointed searches and 40 to 50 hours of human labor into autonomous 10-minute agent runs. - [06:01](https://www.youtube.com/watch?v=hs49d_LTkRc&t=361s) Perplexity built LOTUS (Learning Optimization and Tuning System) to support rapid researcher experimentation and arbitrary model architectures, including dense, MoE, GQA, and MLA setups. - [06:01](https://www.youtube.com/watch?v=hs49d_LTkRc&t=361s) The primary competitive bottleneck in reinforcement learning scaling is the cluster speed and compute efficiency of generating environment rollout steps. - [07:58](https://www.youtube.com/watch?v=hs49d_LTkRc&t=478s) Transforming conversational LLMs into autonomous agents begins with supervised fine-tuning (SFT) on curated multi-step traces to ground structured tool-calling syntax and execution loops. - [09:54](https://www.youtube.com/watch?v=hs49d_LTkRc&t=594s) Human preference data is collected through side-by-side product comparisons and modeled via a chess-style Elo rating framework to provide grounded reward distributions. - [09:54](https://www.youtube.com/watch?v=hs49d_LTkRc&t=594s) Perplexity trains agent policies predominantly on-policy using explicit Kullback-Leibler (KL) divergence penalties rather than standard surrogate clipping heuristics to stabilize training. - [12:25](https://www.youtube.com/watch?v=hs49d_LTkRc&t=745s) Inference serving handles a heterogeneous stack, routing traffic with fast classifiers for specialized UI widgets while using Expert, Tensor, and Pipeline Parallelism for trillion-parameter models. - [14:24](https://www.youtube.com/watch?v=hs49d_LTkRc&t=864s) Perplexity developed Iris to disaggregate generative inference, intelligently routing requests to dedicated prefill nodes for KV-cache prefix reuse before transferring state to decode nodes over InfiniBand. - [15:10](https://www.youtube.com/watch?v=hs49d_LTkRc&t=910s) Future engineering priorities center on scaling reinforcement learning for Comet's complex web actions and implementing infinite context memory to eliminate multi-turn conversational drift. - [16:05](https://www.youtube.com/watch?v=hs49d_LTkRc&t=965s) Embedding systems rely exclusively on open-weight architectures fine-tuned with in-house contrastive learning on proprietary search distributions rather than proprietary third-party APIs. - [18:14](https://www.youtube.com/watch?v=hs49d_LTkRc&t=1094s) To manage context bloat in long-running agent loops, Perplexity favors in-trajectory compaction over sub-agent delegation, avoiding compounding errors caused by poorly specified subtask prompts. - [20:26](https://www.youtube.com/watch?v=hs49d_LTkRc&t=1226s) Dense reward signals for complex, multi-turn agent trajectories are generated by secondary evaluator judge models that grade the entire execution trace against the initial objective.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=hs49d_LTkRc&t=4s) Introducing Perplexity and the Core Product Ecosystem

Tony Wu introduces himself and sets the stage for how Perplexity built what he describes as "the world's leading AI answer engine." Perplexity was founded roughly three years prior and has grown to 250 global employees, spearheaded by its three co-founders. Wu highlights that the company's defining mission has remained constant since day one: "to cater to the world's curiosity by rewriting the fabric of the internet." The company's vision centers around four core consumer and enterprise products:

- **Ask:** Perplexity’s flagship search and answer engine that replaces traditional search link pages with direct, synthesized answers. - **Deep Research:** Launched earlier in the year, designed to conduct exhaustive, multi-step investigations to help users thoroughly master a topic at length. - **Labs:** Shipped mid-year, functioning as an agentic project builder and AI code creator that generates complete artifacts, tools, and apps. - **Comet:** Perplexity’s latest release, billed as the world's first truly agentic browser designed to partner with users by actively reasoning and executing tasks on their behalf.

Wu shares his personal background, noting that he studied at Stanford and held engineering roles at Uber, Meta, and OpenAI before leaving OpenAI six months prior to lead the AI organization at Perplexity. His organization oversees end-to-end AI development, spanning post-training infrastructure, inference optimization, product-focused AI engineering behind Deep Research and Labs, as well as classical machine learning systems responsible for core ranking and recommendations.

---

[01:40](https://www.youtube.com/watch?v=hs49d_LTkRc&t=100s) Compressing Days of Work into Minutes: Demos of Labs and Comet

To illustrate what agentic AI looks like in practice, Wu plays a promotional showcase of Perplexity Labs, an autonomous research agent built to tackle deep, open-ended tasks. He grounds the motivation behind Labs in the everyday friction of knowledge work. In conventional search paradigms, complex tasks demand massive amounts of cognitive overhead and time. For example, planning a trip to Japan or traveling to a conference in Las Vegas often requires firing off 50 to 60 disparate Google queries across airline sites, hotels, reviews, and travel guides, eating up 40 to 50 hours of human labor.

Perplexity Labs fundamentally alters this workflow by consolidating dozens of disjointed search sessions and multi-day research loops into an autonomous, 10-minute agent run that outputs fully synthesized artifacts. Wu follows this up with a presentation of Comet, the company's fully agentic web browser, and rolls a clip from NVIDIA's GTC Paris where Jensen Huang spotlights Perplexity on stage to showcase the frontier of agentic AI.

---

[06:01](https://www.youtube.com/watch?v=hs49d_LTkRc&t=361s) LOTUS: The Post-Training Infrastructure for Agentic Models

Turning under the hood, Wu explains that enabling models to run autonomous, multi-step trajectories requires a massive engineering effort across both high-throughput infrastructure and reinforcement learning (RL). To power this, Perplexity built an internal post-training platform called **LOTUS**, a backronym for the **Learning Optimization and Tuning System**. The team designed LOTUS to meet two primary demands: model architecture flexibility and rapid researcher velocity.

Because the open-source model ecosystem moves at breakneck speed, LOTUS was architected from the ground up to support arbitrary architectures without engineering rewrites. It natively accommodates dense models, sparse Mixture-of-Experts (MoE) networks, and modern attention variants such as Grouped-Query Attention (GQA) and Multi-Head Latent Attention (MLA). For researchers, LOTUS provides a lightweight, modular sandbox to hack quickly, iterate on supervised fine-tuning (SFT), and experiment with reinforcement learning algorithms like PPO (Proximal Policy Optimization) and GRPO (Group Relative Policy Optimization).

```mermaid flowchart TD accTitle: LOTUS Post-Training Architecture

subgraph INGEST["Model Ingestion & Parallelism"] direction TB M["Open-Source Base Models<br/>(Llama, DeepSeek, Qwen, Kimi)"] P["Distributed Strategies<br/>(FSDP, TP, EP)"] M --> P end

subgraph LOTUS["LOTUS Framework Engine"] direction TB A["Attention & Model Agnostic<br/>(Dense, MoE, GQA, MLA)"] R["Researcher Experimentation<br/>(Rapid Prototyping & Hacking)"] A --- R end

subgraph OPT["Training & Alignment Loop"] direction TB SFT["Supervised Trajectories<br/>(Token & Batch Loss)"] RL["Reinforcement Learning<br/>(PPO, GRPO with KL Penalty)"] SFT --> RL end

INGEST --> LOTUS LOTUS --> OPT ```

- **Distributed Parallelism Support:** LOTUS supports Fully Sharded Data Parallel (FSDP), Tensor Parallelism (TP), and Expert Parallelism (EP) to handle large parameter scales efficiently. - **Cross-Ecosystem Deployment:** The system has been leveraged to post-train and deploy leading open-weights model families, including Kimi, Qwen, DeepSeek, and Llama. - **Rollout Step Efficiency:** Wu emphasizes that algorithmic design is only half the battle; the real competitive advantage lies in how fast and cheaply the cluster can generate environment rollout steps to reach designated RL checkpoints.

---

[07:58](https://www.youtube.com/watch?v=hs49d_LTkRc&t=478s) Transitioning from Conversational Chat to Agentic Trajectories via SFT

Transforming an LLM from a typical conversational chatbot into an autonomous agent introduces profound systems and modeling challenges. Rather than generating text solely for human reading, an agent must translate conversational intent into valid, deterministic tool executions and API function calls. Perplexity frames this setup as a classical reinforcement learning problem where an agent's policy $\pi(a | s)$ maps the environment state $s$—defined by the incoming context and running chat history—to an action $a$, represented by the emitted tokens that specify either thoughts, answers, or structured function calls.

```mermaid flowchart LR accTitle: Agent Trajectory Rollout and Optimization Loop

subgraph TRAJ["Trajectory Rollout"] direction TB S["State Context (s)"] --> PI["Policy π(a|s)"] PI --> ACT["Emitted Action (a)<br/>(Tool / Function Call)"] end

subgraph EVAL["Preference & Reward"] direction TB ACT --> ENV["Execution & User Facing"] ENV --> PREF["Side-by-Side ELO Rating<br/>(Pairwise Wins / Losses)"] PREF --> REW["Reward Formulation"] end

subgraph OPT["Policy Optimization"] direction TB REW --> LOSS["KL-Divergence Loss"] LOSS --> GRAD["On-Policy Gradient Update"] GRAD --> PI end ```

The prerequisite to successful RL alignment is seeding the model with high-quality representations of what a multi-step agent trajectory looks like. Perplexity uses Supervised Fine-Tuning (SFT) over curated, ideal end-to-end agentic traces. Because an agentic trajectory—interleaved with environment tool calls and tool responses—still preserves the standard autoregressive structure of token generation, Perplexity applies the same token-level cross-entropy loss functions used in conversational LLMs:

- The model calculates standard cross-entropy loss at the token level across the valid action steps. - Individual token losses are accumulated to compute total trajectory-level sample loss. - Sample losses are subsequently aggregated over the batch to perform gradient updates, stabilizing the base model's tool-calling syntax and reasoning cadences before introducing reinforcement learning.

---

[09:54](https://www.youtube.com/watch?v=hs49d_LTkRc&t=594s) Preference Modeling, Elo Pairwise Scoring, and On-Policy RL

Once models have established strong trajectory fundamentals through SFT, Perplexity turns to reinforcement learning to refine the subtle qualities of answers—especially formatting, tone, thoroughness, and synthesis quality. To gather high-fidelity human preference signals at scale, Perplexity incorporates side-by-side answer comparisons directly into its consumer product. Users frequently encounter an interface prompt revealing two candidate completions side by side, allowing them to select the superior answer.

Perplexity treats these user preference pairs using an Elo rating framework identical to the scoring system used in competitive chess. When one candidate completion wins out over another, its Elo rating adjusts accordingly, yielding a mathematically grounded probability distribution over which trajectory or model variant performs better.

Equipped with these Elo-backed preference models, the AI team runs on-policy reinforcement learning loops to maximize cumulative rewards across agentic executions:

- **Policy vs. Environment:** The agent samples multi-step trajectories in an on-policy regime, deciding when to search the web, execute code, or make downstream tool calls. - **KL Divergence over Clipping:** Wu notes that while standard PPO relies heavily on surrogate clipping heuristics, Perplexity's RL formulation deviates by utilizing an explicit Kullback-Leibler (KL) divergence penalty to constrain policy drift and maintain training stability. - **On-Policy Rationale:** Perplexity trains predominantly on-policy because modern open-weights foundation models are already remarkably capable, meaning that sampling directly from the current policy distribution produces rich, high-reward trajectories without requiring off-policy replaying schemes.

[12:25](https://www.youtube.com/watch?v=hs49d_LTkRc&t=745s) Serving Heterogeneous Models: From Lightweight Classifiers to Trillion-Parameter MoEs

Deploying models to production and turning them into real-world consumer products requires handling vastly different inference workloads. At Perplexity, the serving stack hosts everything from lightweight embedding models and fast classifiers up to massive, one-trillion-parameter large language models. The small models perform critical front-line triage; for example, intent classifiers determine where to route specific queries so the interface can display specialized experiences. A sports-related search surfaces interactive sports widgets, while a travel-related query activates a dedicated suite of travel tools and booking experiences.

```mermaid flowchart TD accTitle: Perplexity Serving Architecture and Parallelism subgraph Routing["Front-Line Triage"] Q["User Query"] --> C["Classifiers & Embeddings"] C -->|"Sports Intent"| SW["Sports Widgets"] C -->|"Travel Intent"| TW["Travel Suite"] C -->|"Complex Generation"| LLM["Trillion-Parameter LLM"] end

subgraph Parallelism["Distributed Inference Techniques"] LLM --> DP["Data Parallelism<br/>(Request routing across nodes)"] LLM --> EP["Expert Parallelism<br/>(MoE MLPs sharded across nodes)"] LLM --> TP["Tensor Parallelism<br/>(Split weight matrices)"] LLM --> PP["Pipeline Parallelism<br/>(Layer stages across GPUs)"] end ```

While smaller embedding and classification models require minimal scaling optimizations, serving massive models demands sophisticated distribution techniques across clusters of GPUs:

- **Dynamic Batching:** Grouping incoming token streams to maximize compute saturation on hardware during inference. - **Data Parallelism:** Distributing query throughput by routing requests effectively across independent compute nodes. - **Expert Parallelism (EP):** For Mixture of Experts (MoE) architectures, separating standard attention layers from feed-forward networks (MLPs) and sharding individual expert MLPs across dedicated nodes. - **Tensor and Pipeline Parallelism (TP/PP):** Slicing large weight matrices intra-node and segmenting layer pipelines inter-node to fit massive models across GPU memory boundaries.

---

[14:24](https://www.youtube.com/watch?v=hs49d_LTkRc&t=864s) Disaggregated Prefill and Decode with Iris

To optimize inference efficiency at scale, Perplexity decouples the two phases of generative inference: the compute-bound prefill step and the memory-bandwidth-bound decode step. Rather than forcing a single GPU to handle both phases with suboptimal hardware utilization, Perplexity developed an internal routing infrastructure called **Iris**.

```mermaid flowchart LR accTitle: Disaggregated Prefill and Decode with Iris subgraph Ingestion["Ingestion & Routing"] Req["Incoming User Request"] --> Iris["Iris Front-End Service"] end

subgraph DisaggregatedServing["Disaggregated Compute"] Iris -->|"Prefix-aware routing<br/>(Maximizes KV reuse)"| PFN["Dedicated Prefill Nodes"] PFN -->|"Transfers KV cache<br/>via InfiniBand"| DCN["Dedicated Decode Nodes"] DCN --> Gen["Token Generation Stream"] end ```

Iris operates as a front-end routing service rather than a database. It inspects incoming requests and directs them to the optimal prefill node that maximizes prefix matching and reuse of the existing Key-Value (KV) cache. Once the prefill phase completes, the resulting KV cache states are transferred over high-speed InfiniBand directly to dedicated decoding nodes to stream the final tokens back to the user.

---

[15:10](https://www.youtube.com/watch?v=hs49d_LTkRc&t=910s) The Perplexity Roadmap: Comet RL Scaling and Infinite Context

Looking forward, Perplexity is focusing engineering efforts on advancing autonomous agent workflows and conversational memory:

- **Scaling Comet:** Further developing Comet by scaling out reinforcement learning infrastructure to run high-fidelity RL workloads on complex, multi-step agent trajectories. - **Infinite Context:** Implementing mechanisms to overcome conversational drift across long session threads. Users frequently pivot between diverse topics within a single conversation; infinite context will allow the model to seamlessly retrieve and reference facts established dozens of turns prior without degrading output quality.

---

[16:05](https://www.youtube.com/watch?v=hs49d_LTkRc&t=965s) Q&A: Clarifying Iris, Data Stores, and Open Embedding Choices

Opening the floor to audience questions, Tony Wu first clarified an audience member's misconception regarding the storage layer: **Iris is strictly an intelligent routing layer for KV cache prefix matching**, not a key-value data store. Perplexity relies on industry-standard platforms like Databricks and Snowflake for general analytics and data warehouse infrastructure.

Addressing another question about model selection for embeddings (such as BERT derivatives versus models like Qwen or proprietary options), Tony explained their core requirements:

- **Openness for Domain Tuning:** Models must be open-weight to allow in-house contrastive learning and fine-tuning on Perplexity's specific search and retrieval query distributions, ruling out closed proprietary APIs like OpenAI embeddings. - **Empirical Architecture Selection:** Perplexity tests a wide range of architectures—including legacy architectures like BERT and newer alternatives like Qwen—choosing winners strictly based on empirical benchmark evaluations and internal retrieval signal.

---

[18:14](https://www.youtube.com/watch?v=hs49d_LTkRc&t=1094s) Q&A: Taming Context Bloat in Long-Running Agent Loops

An audience member asked how Perplexity prevents context bloat in single-agent loops capped at standard context windows (such as 128k tokens) without losing critical details, and whether they rely on multi-agent decomposition. Tony contrasted two core architectural approaches:

```mermaid flowchart TD accTitle: Strategies for Context Window Management subgraph Compaction["Strategy 1: Trajectory Compaction"] A["Raw Search Results"] --> B["Summarize & Discard Raw Dumps"] B --> C["Inject Compact State into System Prompt"] C --> D["Continue Single Trajectory"] end

subgraph SubAgents["Strategy 2: Sub-Agent Delegation"] M["Parent Agent"] -->|"Formulates subtask query"| S["Sub-Agent"] S -->|"Independent context execution"| S S -->|"Returns distilled answer"| M end ```

1. **In-Trajectory Compaction:** When an agent conducts an exploratory search loop, retaining raw web scrapes rapidly exhausts context space. The model compacts prior turn outputs into concise, synthesized summaries, inserts them back into the system message or history, and drops the raw intermediary dumps before proceeding down the trajectory. 2. **Sub-Agent Delegation:** Tasks can be partitioned out to temporary sub-agents operating within clean, independent context windows. However, Tony highlighted a crucial failure mode: "one of the drawbacks to this is that you're relying on the original agent to formulate the request effectively." If the parent agent under-specifies requirements or miscommunicates context, small misunderstandings compound into compounding errors as tasks traverse multiple agent boundaries.

---

[20:26](https://www.youtube.com/watch?v=hs49d_LTkRc&t=1226s) Q&A: Reinforcement Learning and Trajectory-Level Reward Modeling

The session concluded with an inquiry into where reinforcement learning is applied within Perplexity, specifically regarding recommendation systems (like suggested follow-up queries) versus agent behaviors:

- **Search Suggestions vs. Agents:** Perplexity does not currently train query suggestion models using RL; reinforcement learning is reserved for training complex agent search trajectories. - **Trajectory Reward Modeling:** For multi-turn tasks (such as an agent traversing LinkedIn to retrieve contact data or send messages), rewards are generated by secondary evaluator models. These judge models inspect the initial prompt alongside the final execution trace to grade whether the trajectory achieved the desired objective, providing dense, scalable reward signals for RL training.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=hs49d_LTkRc&t=4s) the world's leading AI answer engine — Tony Wu

> [00:04](https://www.youtube.com/watch?v=hs49d_LTkRc&t=4s) to cater to the world's curiosity by rewriting the fabric of the internet. — Tony Wu

> [16:05](https://www.youtube.com/watch?v=hs49d_LTkRc&t=965s) Iris is strictly an intelligent routing layer for KV cache prefix matching — Tony Wu

> [18:14](https://www.youtube.com/watch?v=hs49d_LTkRc&t=1094s) one of the drawbacks to this is that you're relying on the original agent to formulate the request effectively. — Tony Wu

## Mentioned Perplexity AI, Tony Wu, Ask, Deep Research, Labs, Comet, Stanford, Uber, Meta, OpenAI, Google, NVIDIA, Jensen Huang, GTC Paris, LOTUS, Llama, DeepSeek, Qwen, Kimi, BERT, Iris, Databricks, Snowflake, LinkedIn, InfiniBand

## Video description Tony Wu, VP of Engineering, Research, Perplexity AI

Abstracts

Hook: What if AI could replace not just individual searches, but entire research workflows—conducting dozens of queries, reading hundreds of sources, and delivering expert-level reports in minutes instead of hours?

Context: As enterprises struggle to extract actionable insights from overwhelming data volumes, traditional search tools fall short of delivering the comprehensive analysis that strategic decisions require. Perplexity's recent launch of Deep Research and Labs represents a fundamental shift from reactive information retrieval to proactive knowledge synthesis, transforming how organizations approach research-intensive tasks.

Key Takeaways: Attendees will discover the architectural principles behind building autonomous research agents that iterate through complex reasoning loops, and learn practical strategies for implementing multi-modal AI systems that generate structured deliverables—from interactive dashboards to comprehensive reports—while maintaining accuracy across diverse enterprise use cases.

Call to Action: Join this session to understand how your organization can move beyond basic AI chatbots to deploy research-grade AI systems that amplify human expertise across finance, healthcare, and strategic planning functions.

[Source](https://www.youtube.com/watch?v=hs49d_LTkRc)

中文简报

从搜索走向综合：我们如何在 Perplexity AI 打造具备研究团队思维的 AI Tony Wu 概述了 Perplexity AI 如何借助 Labs 和 Comet 浏览器等产品，从传统问答引擎转型为自主研究智能体。他详细介绍了 Perplexity 的后训练基础设施 LOTUS，该系统将监督式轨迹微调与基于用户 Elo 成对偏好引导的同策略强化学习相结合。此外，Wu 还解释了 Perplexity 的异构服务技术栈，重点介绍了通过解耦预填充与解码阶段以优化 KV 缓存复用的 Iris 路由系统。会议最后探讨了多项技术议题，包括通过压缩与子智能体委托缓解上下文膨胀、选择开源权重嵌入模型，以及利用评估器模型进行轨迹奖励建模。 Perplexity 核心产品生态涵盖 Ask、Deep Research、Labs 以及智能体浏览器 Comet，致力于将传统搜索中耗时 40 至 50 小时的繁杂流程压缩为 10 分钟的自主代理运行。 (1:40) 团队自研后训练基础设施 LOTUS，原生支持 MoE、GQA 与 MLA 等模型架构，并通过极速生成环境 rollout 步数大幅降低强化学习扩展的技术壁垒与计算成本。 (6:01) 从聊天助手演进为智能体需依托高质量多步轨迹进行 SFT 规范工具调用，再结合产品内双答案盲测的国际象棋式 Elo 评分构建偏好奖励，并采用显式 KL 散度惩罚保障 On-policy 训练稳定。 (9:54) 生产环境部署了自研路由服务 Iris 以实现 Prefill 与 Decode 阶段的解耦与 KV 缓存前缀复用，并结合专家并行（EP）、张量并行（TP）与流水线并行（PP）支撑万亿参数模型的低延迟推理。 (14:24)

Reading summary

Perplexity is shifting the search paradigm from static link retrieval to autonomous multi-step synthesis by coupling open-weight foundation models with specialized post-training and inference infrastructure. According to AI lead Tony Wu, the company trains its systems using structured trajectory fine-tuning, Elo-rated on-policy reinforcement learning, and a disaggregated prefill-and-decode serving layer. Autonomous Research and Agentic Product Workflows (1:40) Wu characterizes conventional search as structurally inefficient for open-ended knowledge work, noting that complex tasks like travel planning frequently require 50 to 60 queries and 40 to 50 hours of human labor. Perplexity addresses this friction through an ecosystem comprising its flagship Ask engine, Deep Research, Labs, and the Comet agentic browser. Rather than returning web links, these systems execute multi-step research loops to synthesize answers and generate software artifacts, compressing multi-day investigations into autonomous runs that average roughly 10 minutes. Infrastructure and Trajectory Seeding via LOTUS (6:01) To adapt open-weight model families such as Llama, DeepSeek, Qwen, and Kimi, Perplexity built an internal post-training platform called LOTUS (Learning Optimization and Tuning System). Wu states that LOTUS was engineered to support dense models, Mixture-of-Experts (MoE), and modern attention patterns like Grouped-Query Attention (GQA) and Multi-Head Latent Attention (MLA) without code rewrites. The platform utilizes distributed parallelism—including FSDP, Tensor Parallelism, and Expert Parallelism—to maximize the throughput of environment rollout steps during training. Before reinforcement learning can be applied, models undergo Supervised Fine-Tuning (SFT) over curated agentic traces to establish reliable tool calling and API execution. Because multi-step trajectories preserve standard autoregressive token structures, Perplexity calculates token-level cross-entropy loss across action steps. These token losses accumulate into trajectory-level sample losses and batch-level gradient updates, stabilizing tool syntax and reasoning patterns before reward optimization begins. Elo-Based Preference and On-Policy Reinforcement Learning (9:54) To refine output formatting, tone, and synthesis depth, Perplexity gathers real-world human preferences through live side-by-side completion prompts in its consumer product. User selections are tracked using an Elo rating framework identical to competitive chess, providing a mathematical probability distribution over trajectory quality. Using these preference signals, the engineering team executes on-policy reinforcement learning via PPO and GRPO. Unlike standard implementations that rely on surrogate clipping heuristics, Perplexity introduces an explicit Kullback-Leibler (KL) divergence penalty to constrain policy drift and ensure stability. Wu notes that training remains predominantly on-policy because modern base models already generate high-reward trajectories natively. For extended multi-turn tasks, secondary evaluator judge models evaluate complete execution traces to assign trajectory-level reward signals. Disaggregated Serving and Prefill-Decode Decoupling (14:24) Perplexity's production stack hosts diverse inference workloads ranging from fast embedding models and intent classifiers up to one-trillion-parameter MoEs. Lightweight classifiers perform initial triage, routing sports or travel queries to dedicated interactive widgets. Massive models rely on dynamic batching, data parallelism, and expert parallelism to shard feed-forward layers across GPU clusters. To maximize hardware efficiency, Perplexity built an internal routing service named Iris that decouples compute-bound prefill from memory-bandwidth-bound decode operations. Iris functions as an intelligent front-end routing layer rather than a key-value database, directing requests to prefill nodes that maximize Key-Value (KV) cache prefix matching. After prefill completion, the resulting KV cache states are transferred via InfiniBand to dedicated decoding nodes that stream tokens back to the user. Context Window Compaction Versus Sub-Agent Delegation (18:14) Managing context bloat during extended single-agent loops capped at standard context limits presents an ongoing systems challenge. Wu outlines two distinct mitigation strategies: in-trajectory compaction and sub-agent delegation. In compaction, intermediate raw search results are summarized and reinjected into the prompt history while raw web dumps are discarded to free up tokens. Delegating tasks to sub-agents provides clean, independent context windows, but Wu cautions that it introduces compounding risks: if the parent agent under-specifies instructions or misinterprets context, errors compound across agent boundaries. Perplexity's forward roadmap focuses on expanding reinforcement learning for the Comet agent and developing infinite context mechanisms to prevent degradation across long, multi-topic conversational threads. While Perplexity's infrastructure demonstrates how open-weight foundation models can be tuned into autonomous research agents, long-horizon trajectory reliability remains constrained by context-window trade-offs and the risk of error propagation in multi-agent task delegation.

精炼阅读摘要

Perplexity AI 人工智能组织负责人 Tony Wu 阐述了团队如何通过自研后训练平台、在策略强化学习与计算解耦推理架构，将基础大模型改造为能够执行复杂多步研究任务的自主智能体系统。 LOTUS后训练框架与高效算力扩展 (6:01) Tony Wu 介绍，Perplexity 创立约三年且拥有 250 名员工，其产品矩阵已从旗舰问答引擎 Ask 延伸至深度研究工具 Deep Research、智能体项目构建器 Labs 以及自主浏览器 Comet。为了支撑 Labs 将原本耗费 40 至 50 小时、需要 50 到 60 次搜索的人工调研压缩至 10 分钟自主流程，团队构建了内部后训练基础设施 LOTUS（Learning Optimization and Tuning System）。该系统的核心目标是兼顾模型架构的通用性与研究员的迭代速度。 由于开源模型生态迭代迅速，LOTUS 在设计上原生兼容稠密模型、混合专家网络（MoE）以及分组查询注意力（GQA）和多头潜在注意力（MLA）等注意力机制变体，并支持完全分片数据并行（FSDP）、张量并行（TP）与专家并行（EP）。该平台已被用于部署与微调 Llama、DeepSeek、Qwen 和 Kimi 等主流开源权重模型。Wu 强调，后训练的竞争壁垒不仅在于算法设计，更取决于集群以多低廉的成本与多快的速度生成环境交互回滚步数（rollout steps），以驱动强化学习检查点的迭代。 从SFT轨迹到Elo偏好驱动的强化学习 (7:58) 为了让大语言模型从生成文本的对话机器人转变为能够执行 API 与工具调用的自主智能体，Perplexity 将其抽象为强化学习中的策略模型 $\pi(a|s)$，其中状态 $s$ 为当前上下文与历史，动作 $a$ 为包含思维链与结构化函数调用的输出标记。模型首先基于经过筛选的端到端理想轨迹执行监督微调（SFT）。由于智能体轨迹在本质上仍保留自回归标记生成形式，团队采用标准的标记级交叉熵损失函数，逐样本累积并按批次聚合执行梯度更新，借此固化工具调用语法与多步推理节奏。 在完成基础轨迹拟合后，团队引入强化学习来优化回答格式、语气与综合质量。Perplexity 直接在消费级产品端嵌入双候选结果对比界面，收集真实用户的胜负偏好，并引入国际象棋的 Elo 积分算法建立偏好概率模型。Wu 指出，在实际强化学习循环中，Perplexity 主要采用在策略（On-Policy）采样，并放弃了标准 PPO 常用的裁剪启发式策略，转而引入显式 KL 散度惩罚项来约束策略漂移。对于多轮复杂任务（例如在领英上抓取联系人），系统利用独立的裁判模型对照初始提示词与执行轨迹进行评判打分，以此生成高密度的奖励信号，而推荐系统中的后续提问建议当前未采用强化学习训练。 异构模型服务与Iris解耦推理架构 (12:25) 在生产推理层面，Perplexity 的服务栈需要协同处理参数规模差异极大的异构模型，涵盖轻量级意图分类器、嵌入模型乃至达到万亿参数级别的超大 MoE 模型。前端轻量分类器负责执行流量分流，例如将体育查询定向至互动小部件，或将旅行需求接入专用规划套件；而针对万亿参数模型，系统则综合运用动态批处理、数据并行、专家并行、张量并行与流水线并行技术，以应对 GPU 集群的显存与计算瓶颈。 针对生成式推理中计算密集的预填充（Prefill）阶段与显存带宽受限的解码（Decode）阶段存在的硬件利用率失衡，Perplexity 开发了名为 Iris 的前端路由系统。Wu 明确澄清，Iris 是一套请求路由层而非键值数据库（通用数仓仍依托 Databricks 和 Snowflake）。Iris 能够解析传入请求并将其路由至最契合前缀匹配的预填充节点，最大限度复用现存键值缓存（KV Cache），随后通过高速 InfiniBand 网络将 KV 缓存状态直接转送至专用解码节点进行流式输出。此外，Perplexity 在嵌入模型选型上坚持使用 Qwen 或 BERT 衍生等开源架构，以支持针对内部搜索分布进行对比学习微调，而非采用封闭的专有 API。 轨迹压缩与子智能体调用的上下文管理 (18:14) 针对单智能体在标准上下文窗口（如 128k 标记）下执行长周期循环时面临的信息膨胀难题，Tony Wu 对比了两种架构路径。第一种是轨迹内压缩（In-Trajectory Compaction），即智能体在网页探索循环中丢弃抓取的原始网页冗余数据，将阶段性成果提炼为简短摘要重新注入系统提示或历史中，从而在单轨迹内维持精简状态继续执行。 第二种机制是子智能体分发（Sub-Agent Delegation），即将子任务切分给在独立、干净上下文环境中运行的临时子智能体。然而 Wu 指出该方案存在显著的工程隐患：系统高度依赖父级智能体精准表达需求的能力。一旦原始智能体对子任务的描述不明确或遗漏关键上下文，误差会在跨智能体边界的流转中逐级放大，造成级联式的执行失败。 Tony Wu 总结表示，Perplexity 未来的研发重心将围绕 Comet 智能体轨迹的大规模强化学习训练，以及旨在解决长会话跨话题漂移的“无限上下文”技术展开；但在长程多智能体架构中如何彻底避免任务委派的误差累积，以及如何在保障模型架构开放性的同时进一步压低回滚训练成本，仍是系统演进中需要持续探索的问题。

Detailed notes — zh

## Background 该内容由 Ai4 于 2025 年 8 月 29 日发布在 YouTube 平台上，主讲人为 Perplexity AI 人工智能工程副总裁 Tony Wu。视频展示了 Perplexity 如何从传统的被动信息检索转向主动的知识综合系统，系统介绍了 Deep Research 与 Labs 等产品背后的自主研究智能体架构、后训练基础设施及企业级应用实践。 ## TL;DR Perplexity AI 人工智能副总裁 Tony Wu 深入拆解了将数十小时多轮搜索压缩进 10 分钟自主智能体运行的完整技术实现。演讲详细介绍了支持稠密与 MoE 架构的自研统一后训练框架 Lotus，以及通过用户真实 A/B 偏好、Elo 评分机制和同策略强化学习（On-policy RL）进行 Agent 对齐的算法闭环。在基础设施层面，团队展示了服务万亿参数模型的分布式推理方案与自研解耦 Prefill-Decode 的 Iris 调度系统，并在问答环节深入探讨了长轨迹上下文压缩、子智能体误差传递及开源 Embedding 的对比微调实践。 ## Atmosphere & style 整场演讲呈现出高密度的硬核工程范儿与务实的技术深度，Tony Wu 的讲解逻辑严密、节奏紧凑，直击系统设计的核心难点。叙述从直观的产品形态自然推进至底层的强化学习建模与分布式并行架构，生动展现了顶尖 AI 实验室在算法与基建协同上的高速迭代节奏。问答环节中针对上下文膨胀与子智能体复合误差等现实工程权衡的坦诚解答，让现场交流兼具前瞻技术洞见与深厚的实战底蕴。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=hs49d_LTkRc&t=4s) Perplexity 团队规模约 250 人，其使命是通过重写互联网结构来满足全球好奇心，并已构建起 Ask、Deep Research、Labs 以及 Comet 浏览器四大核心产品矩阵。 - [01:40](https://www.youtube.com/watch?v=hs49d_LTkRc&t=100s) Perplexity Labs 与 Comet 将传统需要执行 50-60 次搜索、耗费数十小时的多轮复杂检索与规划工作流，压缩至大约 10 分钟的自主 Agent 运行周期。 - [06:01](https://www.youtube.com/watch?v=hs49d_LTkRc&t=361s) 为了统一支撑复杂的智能体训练，Perplexity 自研了名为 Lotus 的后训练框架，全面支持 Dense 与 MoE 架构，兼容 GQA 与 MLA 注意力机制，并整合了 FSDP、TP 与 EP 等并行策略。 - [06:01](https://www.youtube.com/watch?v=hs49d_LTkRc&t=361s) 后训练的核心竞争力不仅在于训练算法本身，更在于能否以极高的吞吐与算子效率生成强化学习 Checkpoint 所需的大规模 Rollout 轨迹数据。 - [08:00](https://www.youtube.com/watch?v=hs49d_LTkRc&t=480s) 将对话模型转变为自主 Agent 时，系统将交互流程抽象为标准的 RL 决策过程，第一阶段通过监督微调（SFT）在 Token 级别灌输结构化工具调用的理想执行轨迹。 - [08:00](https://www.youtube.com/watch?v=hs49d_LTkRc&t=480s) 在偏好对齐阶段，Perplexity 利用产品端 A/B 真实用户对比数据建立 Elo 评分机制，并采用引入 KL 散度约束的同策略（On-policy）强化学习来稳定策略更新。 - [12:25](https://www.youtube.com/watch?v=hs49d_LTkRc&t=745s) 在超大模型服务方面，Perplexity 针对万亿参数模型构建了跨 GPU 的分布式推理体系，综合运用动态批处理、数据并行、专家并行、张量并行及流水线并行技术。 - [14:24](https://www.youtube.com/watch?v=hs49d_LTkRc&t=864s) 为了提升推理能效，Perplexity 自研了前端调度系统 Iris，实现 Prefill 与 Decode 阶段的物理硬件解耦，并根据 KV Cache 前缀匹配最大化复用缓存，通过 InfiniBand 高速互联进行传输。 - [16:02](https://www.youtube.com/watch?v=hs49d_LTkRc&t=962s) 在嵌入模型选型上，Perplexity 坚决采用 BERT、Qwen 等开源基底，因闭源 API 无法支持基于自研搜索数据分布的对比学习预训练和精细微调，所有选型严格以严密的 Eval 评测为准。 - [18:15](https://www.youtube.com/watch?v=hs49d_LTkRc&t=1095s) 针对长轨迹 Agent 上下文超限问题，团队提出两种方案：通过上下文压缩消除冗余检索碎片，或通过子智能体委托拆分任务，但后者需要防范多层委托引发的复合误差。 - [20:26](https://www.youtube.com/watch?v=hs49d_LTkRc&t=1226s) Perplexity 的强化学习重心聚焦于优化 Agent 的端到端执行轨迹，通过独立的 Judge 评判模型对长流程任务完成度进行对齐打分并提供奖励标量信号。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=hs49d_LTkRc&t=4s) Perplexity 的使命、团队与核心产品矩阵 Tony Wu 开场向大家介绍了 Perplexity 以及团队如何将其打造为全球领先的“AI 答案引擎”（AI answer engine）。Perplexity 成立于大约三年前，目前在全球拥有约 250 名员工。公司的核心使命是“通过重写互联网的结构来满足全世界的好奇心”（rewriting the fabric of the internet to cater to the world's curiosity）。 截至目前，Perplexity 已经构建起四大核心产品体系： 1. **Ask**：即大众熟知的基础 AI 搜索引擎产品。 2. **Deep Research**：今年早些时候推出的深度研究功能，支持对复杂主题进行长篇幅、全方位的深度拆解与理解。 3. **Labs**：今年年中上线的产品，内置 AI 代码创建器（AI code creator），能够让用户直接构建项目并生成交互式组件（artifacts）。 4. **Comet**：最新推出的下一代全自主智能体浏览器（agentic browser），能够真正作为用户的思维伙伴协助完成各类复杂操作。 Tony 随后介绍了自己的背景。他毕业于斯坦福大学（Stanford），此前曾先后在 Uber、Meta 和 OpenAI 工作。大约六个月前，他离开 OpenAI 加入 Perplexity，目前担任 Perplexity 的人工智能副总裁（VP of Engineering / AI Organization Lead）。他带领的 AI 团队职责涵盖 Post-Training（后训练）、推理优化（Inference）、前沿 AI 产品孵化（如 Research 与 Labs），以及包括排序（Ranking）和推荐（Recommendations）在内的经典机器学习（Traditional/Classical ML）领域。 --- ### [01:40](https://www.youtube.com/watch?v=hs49d_LTkRc&t=100s) 从多轮搜索到智能体压缩：Perplexity Labs 与 Comet 演示 在现场，Tony 通过宣传视频和产品演示向观众直观展示了 Labs 和 Comet 的核心能力。针对传统互联网工作流中的痛点，Tony 阐述了 Perplexity 的长远愿景：许多人在日常工作或生活中需要执行大量的重复性搜索，比如规划一次前往日本或拉斯维加斯的旅行，通常需要进行 50 到 60 次 Google 检索来分别查找机票、酒店和行程，前后累积耗费 40 到 50 个小时。而 Perplexity Labs 作为一个深度搜索智能体（Deep Search Agent），能够把长达数十小时的多轮检索与分析流程压缩至一个大约 10 分钟的自主 Agent 运行周期中。 随后展示的 Comet 浏览器进一步强调了端到端的全自主执行能力。现场还播放了一段英伟达（NVIDIA）CEO 黄仁勋（Jensen Huang）在 GTC 巴黎大会上的演示片段，黄仁勋在演讲中以 Perplexity 构建的应用为例，向现场观众展示了新一代 Agentic AI 的实际形态。 --- ### [06:01](https://www.youtube.com/watch?v=hs49d_LTkRc&t=361s) 统一后训练基础设施：Lotus 框架的架构设计 为了支撑这些复杂的智能体行为，底层的后训练（Post-Training）与强化学习（RL）基础设施至关重要。Tony 详细介绍了 Perplexity 内部自主研发的后训练框架——**Lotus**（Learning Optimization and Tuning System 的首字母缩写）。 Lotus 的核心设计目标是保证高度的**架构灵活性**与**研究员迭代效率**。随着各类开源大模型的快速涌现，Lotus 需要兼顾各种不同的模型架构与注意力模式： - **模型结构支持**：全面兼容稠密模型（Dense models）与稀疏混合专家模型（Sparse MoE models）。 - **注意力机制**：原生支持分组查询注意力（GQA, Grouped Query Attention）以及多隐层注意力（MLA, Multi-Head Latent Attention）。 - **并行训练范式**：支持 FSDP（Fully Sharded Data Parallel）、张量并行（Tensor Parallelism, TP）以及专家并行（Expert Parallelism, EP）。 - **算法生态与快速实验**：让研究人员能够快速进行原型验证与实验，灵活支持监督微调（SFT）以及 PPO、GRPO 等多种前沿强化学习算法。 借助 Lotus 框架，团队能够高效部署与微调市面上各类主流开源模型，包括 Kimi、Qwen、DeepSeek 以及 Llama。Tony 强调，后训练框架的核心威力不仅在于算法本身，更在于能否以极高的吞吐与效率生成达到 RL Checkpoint 所需的训练步数（rollout steps）。 ```mermaid flowchart TD accTitle: Lotus Post-Training Architecture accDescr: Lotus framework supporting diverse base models, parallelisms, and training algorithms to produce agentic models. subgraph BaseModels["基础开源模型 (Base Models)"] M1["Dense / Sparse MoE<br/>(Llama, Qwen, DeepSeek, Kimi)"] M2["Attention: GQA / MLA"] end subgraph LotusCore["Lotus 后训练框架 (Optimization & Parallelism)"] P1["并行策略: FSDP / TP / EP"] P2["高效 Rollout 生成与算子优化"] end subgraph TrainingAlgorithms["训练算法与目标 (Algorithms)"] A1["监督微调 (SFT)"] A2["强化学习 (PPO / GRPO)"] end subgraph Output["最终交付"] Agent["自主 Agentic 模型<br/>(Deep Research / Labs / Comet)"] end BaseModels --> LotusCore LotusCore --> TrainingAlgorithms TrainingAlgorithms --> Output ``` --- ### [08:00](https://www.youtube.com/watch?v=hs49d_LTkRc&t=480s) 从 Chat 模型到 Agent：轨迹建模、SFT 与人类偏好对齐 将传统的对话型 LLM（Chat LLM）转变为能自主调取外部工具的 Agent，需要经历系统性的建模与后训练重构。Agent 的本质是将特定的对话输出转化为结构化的工具调用（Tool calls）和函数调用（Function calls）。Perplexity 将这一过程建模为一个标准的强化学习（RL）决策过程： - **状态（State, $s$）**：由对话中传入的完整上下文（Context）表示。 - **动作（Action, $a$）**：由模型自回归输出的 Token 序列及对应工具指令构成。 - **策略（Policy, $\pi(a|s)$）**：给定当前状态下输出动作的概率分布。 要让开源模型掌握 Agent 能力，第一阶段是通过监督微调（SFT）向模型灌输理想的智能体轨迹（Agentic trajectories）。由于智能体轨迹在本质上仍然保持着自回归（Autoregressive）特性，团队可以在 Token 级别计算交叉熵损失，聚合为单个样本损失，再聚合为批次（Batch）损失进行梯度更新。 在 SFT 之后，团队进入人类偏好对齐阶段。Tony 指出，在 Agent 回答问题的新场景下，风格（Style）和回答质量的偏好极其关键： - **偏好数据收集**：Perplexity 在产品端向真实用户展示 A/B 两个并排的回答候选，由用户点击选出更优答案，从而持续沉淀成对偏好数据（Preference pairs）。 - **Elo 评分机制**：借鉴国际象棋中的 Elo 积分体系，将模型或轨迹之间的对抗胜负转化为胜率概率模型，量化评估不同 Agent 轨迹的优劣。 ```mermaid flowchart LR accTitle: Preference-Driven RL Training Loop accDescr: The closed loop from environment action execution to Elo reward formulation and on-policy gradient updates. subgraph Rollout["Agent 环境交互 (On-Policy Rollout)"] Policy["当前 Policy 模型"] -->|"生成 Action / Tool Call"| Env["执行环境 / Web 检索"] Env -->|"返回 State / 结果"| Policy end subgraph Reward["偏好评估与奖励"] Env -->|"完整 Trajectory"| Pref["双选偏好数据<br/>& Elo 评分模型"] Pref -->|"计算 Reward 信号"| RewardFn["奖励函数 (KL 散度正则)"] end subgraph Optimization["梯度更新"] RewardFn -->|"Loss / 梯度计算"| Update["更新 Policy 参数"] Update -->|"更新权重"| Policy end ``` 在强化学习训练循环中，Agent 在环境中执行动作、发起网络检索并调用工具，系统根据偏好模型计算累积奖励（Cumulative reward）。与传统使用 Clip 截断机制的标准 PPO 不同，Perplexity 的方案引入了 KL 散度约束来稳定策略更新。此外，Perplexity 的强化学习主要采用 **On-policy（同策略）** 训练，因为作为底座的开源预训练模型本身已经具备相当强大的通用能力，无需借助复杂的 Off-policy 机制即可实现稳定收敛与能力跃升。 ### [12:25](https://www.youtube.com/watch?v=hs49d_LTkRc&t=745s) 模型部署与万亿参数规模的并行推理架构 在将算法转化为可用产品的过程中，模型服务（Model Serving）与底层推理部署是至关重要的一环。在 Perplexity 的架构中，需要支持多种不同类型和尺度的推理任务：从轻量级的嵌入模型（Embeddings）和分类器（Classifiers），到规模庞大的大语言模型。其中，分类器在搜索体验中扮演着“请求分流”的核心角色——当用户搜索体育赛事时，系统会自动路由并呈现专属的体育小组件；当检测到旅游查询时，则会调出成套的旅游产品与服务。 针对较小的分类器和嵌入模型，工程团队无需进行过于复杂的优化。但当部署规模达到万亿参数级别（1-trillion parameter models）的超大模型时，必须结合一系列高级并行化技术来跨 GPU 进行分布式处理： ```mermaid flowchart TD accTitle: 万亿参数模型服务并行化技术体系 subgraph ServingTechniques["万亿参数分布式推理架构"] direction TB A["请求调度与批处理 (Batching)"] --> B["数据并行 (Data Parallelism)"] B --> C["专家并行 (Expert Parallelism / MoE)"] C --> D["张量并行与流水线并行 (TP & PP)"] end ``` - **批处理（Batching）与数据并行（Data Parallelism）**：确保请求能够被高效打包并正确路由至具体的计算节点。 - **专家并行（Expert Parallelism）**：针对混合专家模型（Mixture of Experts, MoE），将注意力层（Attention layers）与多层感知机（MLP）中的各个专家模块切分（shard）到不同的节点上进行分布式计算。 - **张量并行（Tensor Parallelism）与流水线并行（Pipeline Parallelism）**：将单层算子和模型不同层深拆分到多个 GPU 上，协同承载万亿参数模型的计算负荷。 ### [14:24](https://www.youtube.com/watch?v=hs49d_LTkRc&t=864s) 解耦式 Prefill-Decode 与 Iris 调度系统 为了进一步提升大规模推理效率，Perplexity 采用了“Prefill 与 Decode 解耦”（Disaggregated Prefill-Decode）架构。Prefill（提示词预填充）阶段和 Decoding（逐 token 解码）阶段对算力与内存带宽的需求特征截然不同，采用专用硬件分别处理能够实现资源利用的最大化。 为此，工程团队自研了名为 **Iris** 的前端调度服务架构： ```mermaid flowchart LR accTitle: Iris 解耦式推理流架构 subgraph Ingress["调度层"] REQ["用户请求"] --> IRIS["Iris 调度网关"] end subgraph DisaggregatedServing["解耦推理层"] IRIS -->|"最大化前缀复用"| PRE["专用 Prefill 节点<br/>(KV Cache 计算)"] PRE -->|"InfiniBand 高速传输 KV 元素"| DEC["专用 Decode 节点<br/>(Token 持续生成)"] end ``` - **KV Cache 命中最优化**：Iris 并非底层键值数据库，而是一个智能路由层，能够识别并把请求发送到能够“最大化复用 KV Cache 前缀”的 Prefill 节点。 - **高速网络互联**：通过 InfiniBand 高速网络将 Prefill 阶段生成的 KV 元素快速同步至 Decode 节点，确保后续 token 生成的高吞吐与低延迟。 ### [15:10](https://www.youtube.com/watch?v=hs49d_LTkRc&t=910s) 未来演进路线：Comet 规模化与无限上下文 在产品与技术演进方面，Perplexity 将持续优化 Deep Research 系统，并重点推进以下方向： 1. **扩大 Comet 平台规模**：在 Comet 上扩展运行高保真度（High-Fidelity）的强化学习（RL）工作负载。 2. **探索无限上下文（Infinite Context）**：解决用户在单一对话线程内连续搜索多个不同主题时的跨度引用问题，让模型能够跨越久远的对话历史精准检索和引用背景信息。 ### [16:02](https://www.youtube.com/watch?v=hs49d_LTkRc&t=962s) 现场问答：基础设施定位与嵌入模型选型 在 Q&A 环节，现场观众首先询问了关于推理数据基础设施与 Iris 的定位。演讲者澄清：**Iris 是一个前端调度路由服务**，负责根据 KV 缓存前缀匹配来分发请求，并不是存储底层的 Key-Value 数据仓库；至于企业数据基础设施，Perplexity 主要采用 Databricks 和 Snowflake 等成熟的云平台。 针对开源嵌入模型（如 BERT、Qwen 等）与闭源商业模型（如 OpenAI Embeddings）的选型考量，团队分享了以下核心准则： - **模型必须保持开源（Open）**：闭源的商业 Embedding API 无法满足对特定搜索数据分布（Distribution）进行对比学习（Contrastive Learning）自研预训练和精细微调（Fine-tuning）的需求。 - **以实验评估为准（Empirical Evaluation）**：团队持续在 BERT 系列、Qwen 等开源基底上做试验，最终选型完全取决于严密的 Eval 评测信号反馈。 ### [18:15](https://www.youtube.com/watch?v=hs49d_LTkRc&t=1095s) 现场问答：长轨迹智能体的上下文防膨胀策略 针对观众提出的“单智能体（Single Agent）在持续循环调用中如何防止上下文窗口爆炸（Context Bloat）”的问题，演讲者指出了业界模型目前常见的 128k token 上下文上限，并给出了两套核心工程解法与权衡： | 方案 | 工作机制 | 优势与局限分析 | |:--- |:--- |:--- | | **上下文压缩（Compaction）** | 将前序搜索轨迹中的原始检索结果进行摘要压缩，转化为紧凑表示后重新放回 System Message。 | **优势**：消除了冗长无用的原始网页搜索碎片，保留核心事实延续后续探索轨迹。 | | **子智能体委托（Sub-agents）** | 将独立子任务委托给拥有独立 Context Window 的子智能体执行，主智能体仅接收汇总返回。 | **权衡**：高度依赖主智能体在初始阶段精准提炼任务需求和输入上下文；一旦传递偏差，多层子智能体间会产生**复合误差（Compounding Errors）**。 | ### [20:26](https://www.youtube.com/watch?v=hs49d_LTkRc&t=1226s) 现场问答：强化学习（RL）在推荐与 Agent 中的应用 观众提问强化学习在推荐应用（如相关问题生成）以及 Agent 奖励函数（Reward）设计上的具体实践。演讲者澄清：目前像“相关问题推荐”这类型功能尚未引入强化学习，Perplexity 的 RL 技术主要集中在**优化 Agent 的执行轨迹（Agent Trajectories）**上。 在 Agent RL 奖励建模的具体落地中，系统通常借助独立的评判模型（Judge Model）对用户的原始查询与整条执行轨迹进行对齐打分，判定最终目标是否达成。例如在 Comet Agent 执行跨平台信息抓取、自动化交互等复杂多步任务时，外部评估模型会依据任务完成度给出标量奖励信号，以此指导策略模型的迭代优化。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=hs49d_LTkRc&t=4s) 通过重写互联网的结构来满足全世界的好奇心 — Tony Wu > [00:04](https://www.youtube.com/watch?v=hs49d_LTkRc&t=4s) rewriting the fabric of the internet to cater to the world's curiosity — Tony Wu > [01:40](https://www.youtube.com/watch?v=hs49d_LTkRc&t=100s) 把长达数十小时的多轮检索与分析流程压缩至一个大约 10 分钟的自主 Agent 运行周期中 — Tony Wu > [06:01](https://www.youtube.com/watch?v=hs49d_LTkRc&t=361s) 后训练框架的核心威力不仅在于算法本身，更在于能否以极高的吞吐与效率生成达到 RL Checkpoint 所需的训练步数 — Tony Wu > [08:00](https://www.youtube.com/watch?v=hs49d_LTkRc&t=480s) 风格和回答质量的偏好极其关键 — Tony Wu > [16:02](https://www.youtube.com/watch?v=hs49d_LTkRc&t=962s) Iris 是一个前端调度路由服务，负责根据 KV 缓存前缀匹配来分发请求，并不是存储底层的 Key-Value 数据仓库 — Tony Wu > [16:02](https://www.youtube.com/watch?v=hs49d_LTkRc&t=962s) 闭源的商业 Embedding API 无法满足对特定搜索数据分布进行对比学习自研预训练和精细微调的需求 — Tony Wu > [18:15](https://www.youtube.com/watch?v=hs49d_LTkRc&t=1095s) 高度依赖主智能体在初始阶段精准提炼任务需求和输入上下文；一旦传递偏差，多层子智能体间会产生复合误差 — Tony Wu > [20:26](https://www.youtube.com/watch?v=hs49d_LTkRc&t=1226s) Perplexity 的 RL 技术主要集中在优化 Agent 的执行轨迹上 — Tony Wu ## Mentioned Perplexity AI, Tony Wu, Jensen Huang (黄仁勋), NVIDIA, OpenAI, Meta, Uber, Stanford University, Databricks, Snowflake, Google, Ask, Deep Research, Labs, Comet, Lotus, Iris, Kimi, Qwen, DeepSeek, Llama, BERT ## Video description Tony Wu, VP of Engineering, Research, Perplexity AI Abstracts Hook: What if AI could replace not just individual searches, but entire research workflows—conducting dozens of queries, reading hundreds of sources, and delivering expert-level reports in minutes instead of hours? Context: As enterprises struggle to extract actionable insights from overwhelming data volumes, traditional search tools fall short of delivering the comprehensive analysis that strategic decisions require. Perplexity's recent launch of Deep Research and Labs represents a fundamental shift from reactive information retrieval to proactive knowledge synthesis, transforming how organizations approach research-intensive tasks. Key Takeaways: Attendees will discover the architectural principles behind building autonomous research agents that iterate through complex reasoning loops, and learn practical strategies for implementing multi-modal AI systems that generate structured deliverables—from interactive dashboards to comprehensive reports—while maintaining accuracy across diverse enterprise use cases. Call to Action: Join this session to understand how your organization can move beyond basic AI chatbots to deploy research-grade AI systems that amplify human expertise across finance, healthcare, and strategic planning functions. [Source](https://www.youtube.com/watch?v=hs49d_LTkRc)

Source quotes

[0:04] Tony Wu: 通过重写互联网的结构来满足全世界的好奇心 [0:04] Tony Wu: rewriting the fabric of the internet to cater to the world's curiosity [1:40] Tony Wu: 把长达数十小时的多轮检索与分析流程压缩至一个大约 10 分钟的自主 Agent 运行周期中 [6:01] Tony Wu: 后训练框架的核心威力不仅在于算法本身，更在于能否以极高的吞吐与效率生成达到 RL Checkpoint 所需的训练步数 [8:00] Tony Wu: 风格和回答质量的偏好极其关键 [16:02] Tony Wu: Iris 是一个前端调度路由服务，负责根据 KV 缓存前缀匹配来分发请求，并不是存储底层的 Key-Value 数据仓库 [16:02] Tony Wu: 闭源的商业 Embedding API 无法满足对特定搜索数据分布进行对比学习自研预训练和精细微调的需求 [18:15] Tony Wu: 高度依赖主智能体在初始阶段精准提炼任务需求和输入上下文；一旦传递偏差，多层子智能体间会产生复合误差 [20:26] Tony Wu: Perplexity 的 RL 技术主要集中在优化 Agent 的执行轨迹上

Source quotes

[0:04] Tony Wu: the world's leading AI answer engine [0:04] Tony Wu: to cater to the world's curiosity by rewriting the fabric of the internet. [16:05] Tony Wu: Iris is strictly an intelligent routing layer for KV cache prefix matching [18:14] Tony Wu: one of the drawbacks to this is that you're relying on the original agent to formulate the request effectively.

Original transcript

[0:04] All right. Hello everyone. Uh I'm Tony. [0:06] Uh here I today I'm here to talk a [0:09] little bit about Perplexity, how we've [0:11] built um you know the world's leading AI [0:14] answer engine. Uh and so I'll start off [0:16] with like what is Perplexity? So we uh [0:18] as a company were founded uh about 3 [0:20] years ago. Uh we've got 250 global [0:23] employees. Uh you'll see the three [0:25] founders on the right. And our mission [0:27] has always been to cater to the world's [0:29] curiosity by rewriting the fabric of the [0:32] internet. And we have four products. We [0:34] have the ask product which folks know as [0:36] our search engine. Uh and then earlier [0:39] this year we built the deep research [0:41] product uh which allows you to go [0:42] in-depth and understand a topic uh at [0:45] length. Um we shipped a third product uh [0:48] in the middle of this year called labs [0:49] which allows you to create artifacts and [0:52] builds projects uh using an AI code [0:54] creator. And finally, uh, most recently, [0:57] we shipped Comet, which is, uh, the [0:59] world's first browser that actually [1:01] partners with you to to think on your [1:02] behalf and help you do things. Um, so [1:04] those are the four products. [1:06] And I'll talk a little bit about myself. [1:08] So I, uh, went to Stanford. Uh, I [1:11] previously worked at Uber, Meta, and [1:13] OpenAI. And about six months ago, I left [1:15] OpenAI to join Perplexity. And I [1:17] currently lead the AI organization at [1:19] Perplexity. And, uh, my team focuses on [1:22] post training. Uh we have a team [1:24] dedicated to inference. Uh we build out [1:26] a se series of AI products like research [1:28] and labs. Uh and we also uh focus on [1:31] traditional machine learning. Uh and we [1:34] have a lot more like classical ML [1:36] techniques like ranking uh and [1:38] recommendations. [1:40] Uh cool. So let's uh quickly show you uh [1:43] perplexity labs. Our PR folks have put [1:45] together an awesome video to to showcase [1:47] what Labs is. Uh so yeah, if you could [1:50] hit play on the video for me. Heat. [1:51] Heat. [1:53] [Music] [2:21] [Music] [2:33] Heat. Heat. [2:38] Heat. Heat. [2:44] [Music] [3:01] Heat. Heat. [3:02] [Music] [3:34] Well, yeah. So, that's uh Perplexity [3:35] Labs. It's a uh it's a deep search agent [3:38] that we uh [3:41] that that we built. The video is loaded [3:43] from YouTube, I suppose. Um [3:46] and uh I think our our long-term vision [3:49] for the future of search and work is if [3:52] you think about all of the different [3:53] workflows that uh you go through, um a [3:56] lot of them involve multiple Google [3:58] searches across many hours. You're uh [4:00] let's say you're planning a trip. You're [4:01] you're traveling to Japan. uh that [4:03] typically involves like 50 60 different [4:06] Google queries to search for flights, [4:08] hotels, if you're planning the trip to [4:10] come to Vegas. And what Labs allows you [4:12] to do is take all of those uh different [4:14] queries um the the 40 to 50 hours that [4:17] you're spending and compress it all into [4:18] like a 10-minute agent uh run. Um so [4:22] that's the that's the product that we [4:23] built. Um the the latest uh product that [4:26] we shipped is uh Comet. It's the new [4:28] browser uh fully agentic. Uh, so yeah, [4:30] we also have a video for for comment to [4:32] show you guys what that looks like. [4:37] Uh, yeah, if folks can uh Thank you. [4:47] [Music] [4:55] Heat. [4:58] [Music] [5:07] Heat. [5:13] [Music] [5:28] pull up the clip of Jensen demoing [5:30] Perplexity Labs. [5:33] >> I've pulled up a YouTube video showing [5:35] Jensen demoing Perplexity Labs at GTC [5:38] Paris. It should be at that moment [5:40] >> to formulate what is now a Gent AI. [5:43] Let's take a look at one example. Let me [5:45] show you something. It's built on [5:47] perplexity. [5:50] [Music] [5:50] [Applause] [5:53] [Music] [6:01] Okay, cool. So uh now that we've shown [6:04] some of the the products that Perplexity [6:06] has shipped uh I'm going to talk a [6:08] little bit about uh what is all the work [6:10] that we need to do both on the [6:11] infrastructure side the RL side in order [6:14] to produce models that are capable of of [6:16] doing these things. Uh so we'll start [6:18] with Lotus. So so we built this internal [6:20] post training framework called Lotus. Uh [6:22] it's a backronym stands for the learning [6:24] optimization and tuning system. And the [6:27] reason we set out to do this is uh as [6:29] you guys all know uh there's many many [6:31] models that are coming out all the time [6:33] and we wanted to build a post- training [6:35] framework that was flexible so it can [6:36] handle uh many different types of models [6:38] dense models uh sparse models uh models [6:41] that have different types of attention [6:43] patterns grouped query attention multi [6:45] latent attention um as well as uh are [6:48] are like really flexible to to be able [6:50] to support all of these um and we also [6:53] optimized for researcher flexibility. We [6:55] wanted our researchers to be able to go [6:57] in and hack very quickly, run [6:59] experiments rapidly and uh run a variety [7:02] of different algorithms whether it's RL [7:03] algorithms like GRPO or PO uh be able to [7:06] do supervised fine-tuning on them and uh [7:09] I think this is sort of the this forms [7:11] the foundation to us being able to like [7:13] advance these models uh to be able to [7:15] exercise these like agentic trajectories [7:17] right and one of the key uh important [7:20] things is not just it's not just [7:22] important for us to pick the right [7:23] algorithm but for us to be able to [7:25] optimize them uh efficiently to be able [7:27] to run uh the the uh a lot of the power [7:30] that comes from this is not just uh the [7:32] algorithm itself but it's how [7:34] efficiently can you generate uh the [7:35] steps needed uh to hit a specific RL [7:38] checkpoint. So, so we built this out uh [7:41] it supports uh all the different [7:42] parallelisms uh FSTP, tensor parallel, [7:45] expert parallel uh and uh we we've uh [7:49] been able to use it to deploy all sorts [7:51] of different open source models uh from [7:53] Kimmy to QN to DeepC to Llama. Um and uh [7:58] the the the next step so so after you [8:00] you have this like uh really powerful [8:01] post trading framework the next thing [8:03] you need to do is take what are [8:05] traditional LLMs uh which specialize in [8:07] chat and convert them into an actual [8:09] agent right and uh this involves uh a [8:12] lot of complexity. Uh at at at its heart [8:16] what the agent is able to do is to take [8:18] uh specific chat output and translate [8:20] into tool calls and function calls. Uh [8:23] so so what we do is we model uh the uh [8:26] the chat uh and the agent as an RL [8:28] problem. So you have effectively a [8:30] policy. It takes the state uh the state [8:33] is represented by the context that comes [8:36] in through the chat and we represent [8:37] that as uh tokens that are being [8:39] outputed uh via action. And of course [8:41] these are all stochastic models and so [8:43] they so a policy pi takes a given state [8:46] and action and outputs a probability. [8:50] And uh so for us to take uh one of these [8:53] like typical open- source models and [8:55] make them really effective uh to serve [8:58] as an agent um the first step you need [9:00] to do is uh give them an ideal [9:01] representation of what an agent [9:02] trajectory actually looks like. And so [9:04] SFT supervised fine-tuning is the way in [9:06] which we do that right so uh we [9:08] essentially collect these ideal uh [9:11] agentic trajectories. Um and uh the the [9:14] good news is that um whether you're [9:16] modeling uh an agent as uh like a chat [9:19] model which is like outputting tokens or [9:22] uh you're combining all of these chat [9:24] outputs as actual trajectories uh it [9:26] maintains the auto reggressive nature of [9:27] it. And so you can use the the same SFT [9:30] algorithms that are that are used um you [9:32] know to do uh preference optimization uh [9:35] with chat models uh to to do this with [9:37] agents. And so um this is sort of like [9:39] how how you typically do SFT. uh you [9:42] compute uh a loss at the token level and [9:44] then you can aggregate it to to get a [9:46] single sample and then you can aggregate [9:47] it against to get the uh to get the loss [9:50] at the batch level and that's that's [9:51] sort of how how we um start [9:54] uh and then so from there uh so so after [9:57] we've sort of like done some SFT on the [9:59] model uh we we collect preferences to to [10:02] understand uh answers as you know like [10:04] um there's uh when it comes to this this [10:07] new world for for for agents answering [10:09] your questions uh a lot of things like [10:11] style and preference matter a lot. And [10:13] so what we do is we actually show uh our [10:15] users multiple uh answers. Um so you'll [10:18] notice on Perplexity sometimes you can [10:19] actually click a button to pop up two [10:21] different answers, one on the left and [10:22] one on the right. We ask users to to [10:24] select which one they like better and [10:26] we're able to formulate uh and capture [10:28] all of these uh these preference pairs. [10:31] Uh and from these uh preference pairs uh [10:33] what we we can do is uh we borrow a page [10:35] from the ELO playbook. So ELO is what uh [10:38] people use to evaluate uh for example [10:41] chess players. Um as you win more [10:43] matches in chess you your ELO goes up. [10:45] Um and uh given these like wins and [10:47] losses you can actually formulate it uh [10:50] as a probability uh of like one model [10:53] being better than the other or one [10:55] trajectory being better uh than the [10:56] other. Uh and from there um we we can [10:59] take all of these preferences uh and we [11:01] can actually run some reinforcement [11:03] learning on them. So uh I mentioned [11:05] before that we take uh a we we model uh [11:09] a uh a user uh trajectory or or or a [11:12] model answer uh as a as a reinforcement [11:14] learning problem. Right? So we we have a [11:17] policy um the policy specifies what the [11:19] probability is, right? Uh and then in RL [11:22] what we do is we try to find uh the [11:23] optimal set of model parameters that [11:25] actually maximize the cumulative [11:26] rewards, right? Uh and so uh the we our [11:30] our approach differs a little bit from [11:32] the traditional PO method in which uh [11:34] we'll we'll use uh something like a kale [11:36] divergence instead of clipping uh but [11:38] ultimately we're able to to um do [11:40] utilize these these preference pairs in [11:42] order to uh u improve our improve our [11:45] our our agents and uh from there uh [11:48] essentially we we run a more traditional [11:51] RL training loop right so uh you'll have [11:53] uh this agent it'll go uh take a bunch [11:55] of actions it might search for things it [11:57] might make some tool calls. Uh we are [11:59] able to like formulate the reward. The [12:01] reward comes from the preferences that I [12:02] mentioned earlier and then we're able to [12:04] like uh feed those as losses to update [12:07] the gradients to in order to like update [12:09] uh our our policy. Uh most of what we do [12:12] is on policy learning uh and it the [12:14] reason is because most of the models [12:16] that we are postraining on top of are [12:17] already pretty pretty powerful. So [12:19] there's no need for us to to to go off [12:21] policy. [12:23] Uh cool. So um finally I'll talk a [12:25] little bit about serving our model. So [12:27] what do we need to do to actually deploy [12:29] these models and uh turn them into [12:31] actual products and and let people use [12:32] them. Uh so uh at Perplexity we uh serve [12:35] a lot of different there's a lot of [12:37] different inference types that we we [12:38] will have to serve. Um there's small [12:40] models like embeddings, there's [12:41] classifiers uh that we use. Uh [12:44] classifiers are extremely useful for uh [12:46] determining uh whether or not a specific [12:48] query needs to be routed to a specific [12:50] place. Right? So if you for example [12:52] search for a sports query uh we will [12:54] show you sports widgets. If you search [12:56] for travel, uh there's a whole set of [12:58] travel products that we'll we'll uh show [12:59] you. Um that all comes from our [13:01] classifiers. And of course, we have [13:02] these large language models that we've [13:04] also deployed uh as well. Um and uh the [13:07] different uh uh depending on the size of [13:10] the model and what is what it's actually [13:12] used for uh we will actually uh employ [13:14] different techniques in order to [13:15] optimize them for for scale. [13:18] Uh so uh obviously uh for the smaller [13:22] models like classifiers and bettings [13:24] there's not too much optimizations that [13:25] we need to do. Uh but when it comes to [13:27] serving these larger models we we serve [13:28] as large as one trillion parameter [13:30] models. Um uh there's there's a series [13:33] of techniques that are necessary for us [13:35] to do that. Um the most common obviously [13:37] like there's batching that's pretty [13:39] commonly understood when it comes to [13:40] inference. Um and then of course there's [13:42] a series of parallelism data [13:43] parallelism. Uh so making sure that we [13:46] uh route the the correct requests to the [13:48] specific uh nodes. Um expert parallelism [13:51] that uh involves uh having separate [13:54] experts. Um so if folks are familiar [13:56] with the mixture of experts model uh um [13:59] typically uh what you'll do what you [14:01] have is uh these attention layers as [14:03] well as these fully connected layers the [14:05] these MLPs. Um each of these MLPS will [14:07] be sharted to specific nodes and uh [14:10] we'll will support uh expert [14:11] parallelism. to to support this large [14:13] model uh across different nodes. Uh [14:14] tensor parallelism, pipeline [14:16] parallelism, these are other techniques [14:17] that we also use in order to be able to [14:19] take these large trillion parameter [14:20] models and and be able to distribute [14:22] them across different GPUs. [14:24] Uh cool. We uh the other thing that [14:26] we've been working on is uh disagregated [14:29] pre-filled decode. Uh so the high level [14:31] idea here is that you have different [14:32] GPUs. uh these GPUs um are uh it's [14:36] really best to optimize uh and have [14:38] dedicated hardware uh to handle the [14:40] pre-code pre-fill step as opposed to the [14:42] decoding step, right? And so uh what [14:45] we've done is we've built um [14:46] infrastructure called Iris which [14:48] actually comes in and optimizes uh which [14:51] specific node to send a request to for [14:53] prefilling. It uh essentially identifies [14:55] like which uh pre-filled node uh [14:58] maximizes the reuse of a KV cache, [15:00] right? And uh we're able to through [15:03] Infiniband send those KV elements to the [15:05] decode step which is necessary uh in [15:07] order to generate all of the different [15:08] tokens. [15:10] Uh and so yeah, what's next for us? Uh [15:12] so we'll continue to make uh [15:14] improvements to deep research. Um uh I [15:17] demoed Comet. Uh we're going to scale [15:18] that out and uh continue to like run [15:22] very uh high high fidelity workloads for [15:25] RL on comet. Um, we're also [15:27] experimenting with infinite context, uh, [15:29] which is the ability to, um, often when [15:31] folks do a search, uh, they will be [15:33] searching for many different things [15:35] within a a single conversation thread, [15:37] right? And, uh, we'd like the ability to [15:39] be able to like reference things from, [15:41] uh, from many many conversations ago as [15:43] you're as you're searching. Uh, and uh, [15:46] yeah, that's what's next in store for [15:48] us. Um, cool. [15:50] Uh so yeah, I will take any questions uh [15:53] depending on how much time there is and [15:56] uh if folks if I if we run out of time [15:58] I'll also be outside if folks want to [15:59] come ask me questions. [16:02] >> Cool. Yeah, I saw your your hand go up [16:03] first. [16:05] >> Tony great. Thank you. [16:08] Question about your post training and [16:11] inference data infrastructure. You [16:13] mentioned that your value is built on [16:16] and how to scale it. [16:18] >> Yeah. So, Iris is not our key value [16:20] store. So, Iris is our front-end service [16:22] uh that actually routes uh uh different [16:25] requests based to the uh appropriate [16:28] pre-filled node depending on maximizing [16:30] the prefix of the KV cache. Yeah. So, [16:32] Iris is not a data store. [16:36] >> Yeah. I didn't talk a little bit about [16:38] our data infrastructure too much. Um [16:40] it's all pretty standard stuff like data [16:41] bricks, snowflakes. We Yeah, we leverage [16:43] uh those platforms. Yeah. Yeah. Cool. [16:47] five minutes. Um any any other [16:49] questions? Yeah, go ahead. [16:54] [Music] [16:56] >> Have you ever tried anything else and [16:58] why you [17:01] >> um sorry the um can you repeat the [17:02] question a little bit? [17:07] >> Have you tried anything else? [17:11] >> Thank you. [17:11] >> Why choose this two over the others? for [17:14] example, Quinn or um some Google model. [17:18] >> Yeah, we've uh we've experimented with [17:20] uh Quen as well. Um I think for [17:22] embedding models uh we actually have [17:24] also uh experimented with our own like [17:26] we've done our own contrast of learning [17:28] to uh uh pre-train some of our own [17:30] embedding models. Um the uh the BERT [17:34] models um so uh essentially yeah all all [17:38] the models that that we use uh for [17:39] embedding uh there's a couple [17:41] characteristics that are important. one, [17:42] they they generally have to be open. So [17:44] that sort of like limits us from like [17:46] using like for example the open AI [17:47] embeddings, right? Um and it allows us [17:50] to like fine-tune them for our specific [17:52] distribution. Uh and uh I think BERT has [17:56] been around for for a while. In fact, I [17:59] think we we've we've just experimented [18:00] with with a lot a lot of them. And uh [18:03] yeah, I I don't know if I can speak to [18:05] like a specific reason we have or [18:07] haven't used uh them. Most of them are [18:08] are empirical. like retest them uh based [18:11] on some evals and and we get signal [18:13] there. Yeah. [18:14] >> Cool. Yeah. Go ahead. [18:15] >> Uh hi. So uh the slide that you showed [18:18] after your uh supervised fine-tuning and [18:21] RL, right? Like it looked like a single [18:23] agent system, right? So how do you [18:25] prevent like a context bloat up in a [18:28] single agent system where the single LLM [18:29] is going through a loop continuously and [18:32] you know uh how do you do that or do you [18:34] have a multi- aent system where each LLM [18:36] is kind of trying to each LLM invocation [18:38] is trying to do a subtask then you [18:40] collate them then you start another [18:41] thread something like that. Yeah. [18:43] >> Yeah. So um that's a great question. And [18:45] so most most models uh they uh I think [18:48] they cap out at like uh somewhere like [18:50] the more powerful models I think cap out [18:52] around like 128k token context. Um so [18:56] there's a couple techniques. Um there's [18:58] uh you can obviously do like compaction. [19:00] So let's say that you have a specific [19:02] agent context um from the previous [19:05] turns, right? you can uh compact them [19:07] and then uh uh put those back into like [19:10] the system system message so that uh [19:12] there's it's like a summarization of [19:14] what you've done. So for example, let's [19:15] say you um you have this like trajectory [19:18] where you are searching for a bunch of [19:19] things. Um you have those search results [19:21] but now you're going and exploring [19:23] something else or you're um searching [19:25] for something uh different, right? Those [19:27] the the the first search that you did [19:29] the results from there uh you you often [19:31] don't need those raw search results. You [19:33] often just need like a summary of what [19:35] those are. So you can sort of like [19:37] compact them into like uh um a more [19:40] efficient representation uh and then [19:41] continue the trajectory. Um the other [19:43] thing um that you can do there's some [19:45] trade-offs between um creating what are [19:48] effectively like sub aents, right? So uh [19:50] if you uh don't want to like uh add to [19:52] the sort of like context window of the [19:54] existing agent, uh you can sort of [19:55] delegate to like a sub agent, have that [19:57] sub agent with its own context window do [19:59] do things on your behalf and then come [20:01] back with results. Um there's pros and [20:03] cons to that. Uh I think one of the the [20:05] drawbacks to this is that you're relying [20:07] on the original uh agent to formulate [20:10] the request effectively and essentially [20:13] like provide the right amount of context [20:15] to the sub agent so that it can go off [20:17] and and if there's if it's even a little [20:18] bit off um those uh errors can sort of [20:22] compound as you have more and more like [20:24] sub aents. Yeah. [20:26] Cool. Yeah. Go ahead. [20:33] >> Yeah. Yeah, if you have a microphone. [20:35] >> So I am curious about the RL [20:39] about the RL and the recommendation [20:42] applications in uh perexity. The so [20:46] first question is uh what is the [20:48] specific uh rewards for the LL agent and [20:52] the second question is uh how can you uh [20:56] train the the recommendation model to uh [21:00] give uh some uh uh questions to the [21:04] users. [21:06] >> Yeah. So I think for most of our [21:08] recommend when you say recommendation [21:09] you mean like the related queries like [21:11] related questions where okay yeah I [21:13] think for for most of those we we are as [21:15] of yet are not using RL reinforcement [21:17] learning for that it's the agent [21:19] trajectories that are that are um [21:21] leveraging RL for for now um yeah does [21:24] that make sense [21:25] >> okay so how about the agent [21:31] [Music] [21:32] to train some [21:35] angels [21:36] So what is the speed? [21:39] >> Yeah, that's a good question. Uh I [21:41] probably can't go into too too much [21:43] detail around that, but you can imagine [21:44] how to formulate uh the right reward, [21:46] right? So um you can have uh you could [21:49] have different models uh take a look at [21:51] a specific user query and score a [21:53] trajectory to determine like whether or [21:55] not the the actual end uh goal was [21:58] achieved. Right? So, if you ask it to go [22:00] um like like let's say you go ask your [22:01] comment agent to go um go through [22:04] LinkedIn and uh send a bunch of messages [22:06] to people or or fetch a bunch of [22:08] information or whatever. Um yeah, you [22:10] can you can leverage like a separate [22:12] model to actually like grade uh whether [22:14] or not the the the trajectory was [22:15] successful. Yeah, at a high level that [22:17] that's kind of those are some techniques [22:18] that you can do.