CMU Web Agents Beat OpenAI Models by Scaling Actions Over Reasoning

Towards Specialized Web Agents Using Production-Scale Workflow Data With Carnegie Mellon University — Ai4

Ai4

Fine-tuning open-source models on authentic human workflows and expanding test-time interaction steps outperforms prompt scaffolding and internal reasoning tokens.

AI-assisted summary of Towards Specialized Web Agents Using Production-Scale Workflow Data With Carnegie Mellon University — original source. Check the original for full context.

Source published: 2025-08-28

Detailed notes

## Background Published on YouTube by Ai4 on August 28, 2025, this talk features Carnegie Mellon University researcher Junhong Shen discussing specialized web agents. The session focuses on moving beyond general-purpose models like GPT-4 by fine-tuning open-source LLMs on production-scale workflow data encompassing over 250 domains and 6 billion tokens.

## TL;DR Junhong Shen outlines why traditional scaffolding around general-purpose LLMs falls short for browser navigation and presents specialized training on grounded human workflows as a superior alternative. He highlights ScribeAgent, a fine-tuned open-source model trained on 20,000 real-world user trajectories that achieves state-of-the-art results on WebArena. Shen then investigates test-time reasoning in multi-step settings, demonstrating that allocating compute to environmental actions ('act more') offers greater information advantages than merely expanding per-step deliberation ('think more'). Finally, he presents Test-Time Interaction (TTI), a reinforcement learning framework using Gemma 3 12B that progressively expands the action horizon to foster rich exploration and error recovery.

## Atmosphere & style The presentation carries an engaging, forward-looking academic energy, opening with Shen's enthusiastic note about adding brand-new research findings on agent scaling right before the talk. Shen speaks with clear, methodical authority, dissecting the structural limitations of chatbot-centric architectures with lucid diagrams and measured reasoning. The pacing moves smoothly from critique to practical fine-tuning results and paradigm-shifting reinforcement learning concepts, making the viewer feel plugged directly into the bleeding edge of agent research.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=Llm5SdguQOY&t=4s) Unlike conversational chatbots, digital web agents operate in a continuous perception-action cycle, processing multimodal observations (HTML, accessibility trees, screenshots) to execute browser actions. - [04:16](https://www.youtube.com/watch?v=Llm5SdguQOY&t=256s) Scaffolded systems built around general-purpose frontier LLMs face major bottlenecks because base models lack native pre-training on structured web contexts and grounded interaction. - [04:50](https://www.youtube.com/watch?v=Llm5SdguQOY&t=290s) Specializing the foundation model directly through post-training on web interaction data yields greater performance headroom than complex prompting templates or external evaluator modules. - [07:27](https://www.youtube.com/watch?v=Llm5SdguQOY&t=447s) ScribeAgent was trained via supervised fine-tuning on roughly 20,000 authentic production workflow trajectories captured across hundreds of web domains. - [08:50](https://www.youtube.com/watch?v=Llm5SdguQOY&t=530s) Fine-tuning open-source models on real human demonstrations enabled even compact 7B models to outperform large proprietary models like OpenAI's o1-mini on web benchmarks. - [12:34](https://www.youtube.com/watch?v=Llm5SdguQOY&t=754s) Injecting test-time reasoning tokens enables web agents to deliberate over constraints and compare alternatives before committing to irreversible actions. - [14:20](https://www.youtube.com/watch?v=Llm5SdguQOY&t=860s) Multi-step web environments introduce an orthogonal axis for scaling test-time compute: scaling environmental interaction steps ('act more') rather than just per-step reasoning tokens ('think more'). - [15:10](https://www.youtube.com/watch?v=Llm5SdguQOY&t=910s) Executing more actions provides an empirical information advantage by retrieving live feedback from the external world, facilitating exploratory navigation and backtracking. - [16:15](https://www.youtube.com/watch?v=Llm5SdguQOY&t=975s) The Test-Time Interaction (TTI) reinforcement learning framework progressively expands the maximum step horizon during training to incentivize non-greedy exploration and error recovery. - [17:10](https://www.youtube.com/watch?v=Llm5SdguQOY&t=1030s) Applying TTI to the open-weights multimodal Gemma 3 12B model achieved state-of-the-art performance across competitive open web agent benchmarks.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Llm5SdguQOY&t=4s) Introduction and the Agentic Interaction Loop

Junhong Shen, a PhD student at Carnegie Mellon University, opens his talk by sharing recent research on developing specialized web agents through large language models (LLMs) and real-world data. He notes with a smile that his presentation title differs slightly from the conference poster outside because of a last-minute addition: a brand-new research development that he believes is "potentially leading to a new paradigm in agent scaling" and will likely become an active area of exploration across the research community in the coming months.

To ground the discussion, Shen draws a fundamental distinction between conversational chatbots and agentic systems. While chatbots operate in closed text exchanges, an agent continuously interacts with an external environment. The environment emits observations, which the LLM processes to formulate and issue concrete actions. Once executed in the environment, these actions alter the state and produce fresh observations. This iterative perception-action cycle forms the backbone of digital agents across varied environments such as IDEs, video games, mobile operating systems, and web browsers.

```mermaid flowchart LR accTitle: Web Agent Interaction Loop subgraph Environment ["Browser Environment"] WebState["Web Page State (HTML, A11y Tree, Screenshots)"] end subgraph Agent ["LLM Web Agent"] History["Task Goal & Action History"] ActionGen["Action Determination (Click, Type, Hotkeys)"] end WebState -->|"Observations"| History History --> ActionGen ActionGen -->|"Executable Commands"| WebState ```

In web navigation specifically, the agent begins with an explicit task goal—such as booking a flight or reserving a hotel room—along with a starting URL. At each step, the agent inspects the page state via modalities like raw HTML, browser-specific accessibility trees, or visual screenshots. It weighs these observations alongside its historical trail of prior actions to decide what to do next, generating discrete commands such as clicking links, typing text strings, or triggering keyboard shortcuts (e.g., Ctrl+C, Ctrl+V). The agent repeats this loop until the overarching task reaches completion.

[04:16](https://www.youtube.com/watch?v=Llm5SdguQOY&t=256s) The Limitations of Scaffolding and General-Purpose LLMs

The dominant paradigm for building web agents relies on taking proprietary, general-purpose frontier models—such as GPT-4o or Claude—and constructing elaborate scaffolding on top of them. Developers design intricate workflow charts that decompose user tasks into multi-stage pipelines and augment the base model with external modules. These include memory components designed to store and retrieve past successful trajectories, as well as evaluator modules that validate and score candidate actions before they are executed in the browser.

Shen emphasizes that despite their architectural complexity, these scaffolded systems suffer from a major bottleneck: general-purpose LLMs are inherently unspecialized for web-native execution. Because frontier models are pre-trained predominantly on general web text rather than grounded browser workflows and structured HTML layouts, their out-of-the-box navigation ability leaves a substantial performance gap when compared to human users. Complex scaffolding cannot fully compensate for a foundation model that lacks native grounding in web interaction.

Instead of endlessly tweaking prompting templates or chaining external modules around an off-the-shelf model, Shen and his collaborators argue that the greatest headroom for improvement comes from specializing the LLM itself before adding scaffolding. This hypothesis is supported by recent results on WebArena, a premier academic benchmark for web agents, where the leaderboard is led by models post-trained and fine-tuned on specialized web interaction datasets rather than raw general-purpose models wrapped in prompt engineering.

[07:27](https://www.youtube.com/watch?v=Llm5SdguQOY&t=447s) ScribeAgent: Specializing Open-Source Models on Production Workflow Data

To build a genuinely specialized web agent, the CMU team partnered with Scribe, an AI workflow documentation startup. Scribe provides a browser extension that records user interactions on the web—such as an HR employee navigating enterprise software during an employee onboarding process—and automatically transforms those clicks and keystrokes into structured, step-by-step guides with AI-generated titles and descriptions. This platform captures authentic, real-world human workflows across hundreds of diverse web domains.

The researchers sampled a subset of roughly 20,000 real-world user trajectories from Scribe's production data to investigate how human demonstrations could augment LLMs for web tasks. The training methodology relies on straightforward Supervised Fine-Tuning (SFT): each recorded workflow is formatted into a standardized trajectory sequence pairing environmental inputs with the corresponding human action. When applied across various open-source base models, SFT on this golden dataset produced dramatic performance jumps. The gains scaled predictably with both model capacity and the volume of training tokens.

- **Model Variants:** The team trained two primary open-source models, releasing them as ScribeAgent Small (7B parameters) and ScribeAgent Large (32B parameters). - **Internal Benchmark Performance:** Even the compact 7B ScribeAgent outperformed large proprietary models such as OpenAI's o1-mini on internal web evaluation benchmarks. - **Academic Benchmark Success:** ScribeAgent achieved state-of-the-art results on the competitive WebArena benchmark, demonstrating that fine-tuning on authentic human trajectories establishes a strong, durable performance foundation.

Shen highlights that while ScribeAgent established top performance using standard direct-response models, the machine learning landscape underwent a swift transformation heading into late 2024 with the emergence of test-time reasoning models that deliberate and generate thinking traces prior to emitting their final browser actions.

[12:34](https://www.youtube.com/watch?v=Llm5SdguQOY&t=754s) Reasoning Paradigms and Test-Time Compute for Web Agents

Recent breakthroughs in reasoning models demonstrate that generating internal reasoning traces or "thinking tokens" before producing a final answer dramatically boosts model capability on complex, single-turn tasks such as mathematics and competitive coding. Motivated by these successes, the research team investigated how this reasoning paradigm could be adapted to agentic environments. Early iterations of agents like ScribeAgent operated via a direct mapping: the model ingested a webpage observation and immediately outputted a concrete browser action. The new approach explores whether prompting or fine-tuning the agent to output explicit thinking traces before taking an action can substantially improve task completion.

In practical web agent scenarios, this mirrors human rumination. For instance, when tasked with booking a hotel for a specific date, an agent should not merely click the "book" button on the first hotel it sees. Instead, it benefits from pausing to reason through multiple constraints: comparing prices, analyzing location proximity, evaluating amenities, and synthesizing user reviews. Injecting intermediate reasoning tokens allows the model to deliberate on whether a particular option genuinely satisfies the user's criteria before committing to an irreversible web action.

```mermaid flowchart TD accTitle: Test-Time Interaction (TTI) Reinforcement Learning Loop subgraph EnvLoop ["Online Trajectory Collection"] A["Agent Deployed in Web Environment"] --> B["Interact Across Webpages (Horizon / Max Steps Expands over Training)"] B --> C["Collect Trajectories with Exploration & Backtracking"] end subgraph UpdateLoop ["Policy Filtering & Training"] C --> D["Filter Successful Trajectories"] D --> E["Update Multimodal Model (e.g., Gemma 3 12B)"] E --> A end ```

---

[14:20](https://www.youtube.com/watch?v=Llm5SdguQOY&t=860s) The "Think More vs. Act More" Trade-Off in Multi-Step Environments

While scaling per-step reasoning tokens is the conventional approach borrowed from single-turn benchmarks, agentic tasks introduce a fundamental structural difference: they are intrinsically multi-step and interactive. In web tasks, the agent continuously acts within and receives feedback from a dynamic external environment. This realization opens up a completely orthogonal axis for scaling test-time compute: scaling the number of interaction steps rather than merely scaling the internal reasoning tokens generated at each step. This exploration forms the core premise of the team's research on the trade-offs between "thinking versus doing."

Given a fixed test-time compute budget, developers face a strategic choice between allocating tokens to per-step deliberation ("think more") or allowing the agent to execute more environmental actions ("act more"). The speaker emphasizes that acting more provides a distinct advantage that per-step reasoning alone cannot offer: it enables the retrieval of fresh, ground-truth information from the external world. By clicking through multiple search results, inspecting alternative pages, and navigating deep into booking flows, the agent gathers real data, explores alternative paths, and backtracks when hitting dead ends instead of hallucinating on stale observations.

- **Per-step Scaling ("Think More"):** Allocates compute budget toward generating extended internal reasoning chains before emitting an action on the current webpage. - **Interaction Scaling ("Act More"):** Allocates compute budget toward executing a higher volume of actions, facilitating active web exploration, diverse query formulation, and multi-page information synthesis. - **Information Advantage:** Acting directly in the environment uncovers latent state information and real-world feedback that static per-step ruminations cannot produce.

---

[16:15](https://www.youtube.com/watch?v=Llm5SdguQOY&t=975s) Test-Time Interaction (TTI) via Horizon-Scaled Reinforcement Learning

To evaluate whether scaling environmental interaction genuinely benefits agents, the team introduced Test-Time Interaction (TTI), a specialized reinforcement learning (RL) framework designed to incentivize exploration and multi-step problem-solving. TTI functions through an iterative two-stage online RL loop: the agent is first deployed into the live web environment to roll out interaction trajectories, after which successful trajectories are filtered out and used to update the policy.

The distinguishing feature of TTI lies in how it manages the agent's interaction horizon. As training proceeds, the framework progressively increases the maximum allowable steps per task. Rather than forcing the agent into brittle, greedy shortcuts, expanding the horizon populates the training dataset with increasingly long, rich, and sophisticated trajectories. This curriculum encourages advanced behavioral patterns such as speculative exploration, cross-checking, and error recovery.

Using DeepMind's open-source multimodal Gemma 3 12B model, the resulting agent established state-of-the-art results across several competitive open-source and open-data web agent benchmarks. In closing, the speaker reiterated that as reasoning models proliferate, researchers must look beyond chatbot-style token scaling. Because interaction is the defining attribute of autonomous agents, scaling test-time interaction represents one of the most promising frontiers for building capable, production-grade web agents.

- **TTI Mechanism:** Employs an online trajectory collection loop paired with success-filtering and supervised policy updates. - **Expanding Horizon Curriculum:** Dynamically raises the maximum step threshold throughout training to reward exploratory and non-greedy behavioral patterns. - **Benchmark Performance:** Fine-tuning the Gemma 3 12B multimodal open-weights model achieved top-tier performance on established web agent benchmarks under fully open data and model settings.

## Notable quotes > Large Language Model (LLM) agents are rapidly improving to handle increasingly complex web-based tasks. — Junhong Shen

> general-purpose LLMs are not specifically trained to understand specialized web contexts such as HTML, and they often struggle with long-horizon planning. — Junhong Shen

> We explore an alternative approach that fine-tunes open-source LLMs using production-scale workflow data collected from over 250 domains corresponding to 6 billion tokens. — Junhong Shen

> [00:04](https://www.youtube.com/watch?v=Llm5SdguQOY&t=4s) potentially leading to a new paradigm in agent scaling — Junhong Shen

> [00:04](https://www.youtube.com/watch?v=Llm5SdguQOY&t=4s) While chatbots operate in closed text exchanges, an agent continuously interacts with an external environment. — Junhong Shen

> [04:16](https://www.youtube.com/watch?v=Llm5SdguQOY&t=256s) general-purpose LLMs are inherently unspecialized for web-native execution. — Junhong Shen

> [04:16](https://www.youtube.com/watch?v=Llm5SdguQOY&t=256s) Complex scaffolding cannot fully compensate for a foundation model that lacks native grounding in web interaction. — Junhong Shen

> [04:16](https://www.youtube.com/watch?v=Llm5SdguQOY&t=256s) the greatest headroom for improvement comes from specializing the LLM itself before adding scaffolding. — Junhong Shen

> [14:20](https://www.youtube.com/watch?v=Llm5SdguQOY&t=860s) scaling the number of interaction steps rather than merely scaling the internal reasoning tokens generated at each step. — Junhong Shen

> [14:20](https://www.youtube.com/watch?v=Llm5SdguQOY&t=860s) acting more provides a distinct advantage that per-step reasoning alone cannot offer: it enables the retrieval of fresh, ground-truth information from the external world. — Junhong Shen

> [14:20](https://www.youtube.com/watch?v=Llm5SdguQOY&t=860s) Acting directly in the environment uncovers latent state information and real-world feedback that static per-step ruminations cannot produce. — Junhong Shen

> [16:15](https://www.youtube.com/watch?v=Llm5SdguQOY&t=975s) Because interaction is the defining attribute of autonomous agents, scaling test-time interaction represents one of the most promising frontiers for building capable, production-grade web agents. — Junhong Shen

## Mentioned Junhong Shen, Carnegie Mellon University, Ai4, GPT-4, GPT-4o, Claude, OpenAI o1-mini, Scribe, ScribeAgent, ScribeAgent Small, ScribeAgent Large, WebArena, Google DeepMind, Gemma 3 12B, Test-Time Interaction (TTI)

## Video description Junhong Shen, Researcher, Carnegie Mellon University

Abstracts

Large Language Model (LLM) agents are rapidly improving to handle increasingly complex web-based tasks. Most of these agents rely on general-purpose, proprietary models like GPT-4 and focus on designing better prompts to improve their planning abilities. However, general-purpose LLMs are not specifically trained to understand specialized web contexts such as HTML, and they often struggle with long-horizon planning. We explore an alternative approach that fine-tunes open-source LLMs using production-scale workflow data collected from over 250 domains corresponding to 6 billion tokens. This simple yet effective approach shows substantial gains over prompting-based agents on existing benchmarks.

[Source](https://www.youtube.com/watch?v=Llm5SdguQOY)

中文简报

CMU网页智能体通过扩展动作而非推理击败OpenAI模型在真实人类工作流程上微调开源模型并扩展测试时交互步数，其表现优于提示脚手架与内部推理Token。 Junhong Shen阐述了为何围绕通用LLM构建的传统脚手架在浏览器导航中表现乏力，并提出在真实人类工作流程上进行专门训练是更优的替代方案。他重点介绍了ScribeAgent，这是一款在20,000条真实世界用户轨迹上训练的微调开源模型，在WebArena上取得了最先进的成果。随后，Shen探讨了多步设定下的测试时推理，证明将算力分配给环境动作（“多行动”）比单纯扩展每一步的深思熟虑（“多思考”）能带来更大的信息优势。最后，他介绍了测试时交互（TTI），这是一种使用Gemma 3 12B的强化学习框架，能够逐步扩展动作视界，以促进充分探索与错误恢复。相比于在通用闭源大模型上堆叠复杂的脚手架框架，研究表明直接利用真实网页交互工作流对开源基础模型进行后训练专门化，能提供更大的性能提升空间。 (4:16) CMU 团队利用 Scribe 平台的约 2 万条真实生产环境工作流轨迹训练出 ScribeAgent，其 7B 参数模型在多项网页基准测试中甚至击败了 OpenAI 的 o1-mini 等大型专有模型。 (7:27) 在多步骤网页任务中，测试时计算扩展开辟了新维度：除了单步内生成推理 Token（多思考）外，通过扩展与外部环境交互步数（多行动）能获取新鲜反馈并支持探索与回溯，形成关键的信息优势。 (14:20) 团队提出的测试时交互（TTI）强化学习框架结合逐步拉长步数上限的课程机制，基于开源 Gemma 3 12B 多模态模型在开源网页智能体评测中取得了前沿 SOTA 表现。 (16:15)

Reading summary

Junhong Shen of Carnegie Mellon University argues that advancing web agents requires specializing underlying foundation models on authentic workflow data and scaling environmental interactions, rather than relying solely on complex prompt scaffolding or internal reasoning tokens. Limitations of Scaffolded General-Purpose Models (4:16) Conventional web agent development commonly applies complex scaffolding—such as multi-stage pipelines, memory components, and action-evaluator modules—around proprietary general-purpose frontier models like GPT-4o or Claude. Shen contends that this scaffolding cannot compensate for foundation models that lack native grounding in web environments, HTML structures, and browser workflows. Because these general models are pre-trained predominantly on static web text, their navigation capabilities lag behind human performance. Supervised Specialization on Human Trajectory Data (7:27) To ground models in real-world browser usage, CMU collaborated with Scribe to train agents on approximately 20,000 recorded user trajectories capturing human workflows across diverse enterprise web domains. Using supervised fine-tuning (SFT) to map environmental observations directly to human actions, the researchers developed ScribeAgent Small (7B parameters) and ScribeAgent Large (32B parameters). Performance gains scaled systematically with model size and token volume. The Thinking Versus Doing Trade-Off (14:20) While modern reasoning models boost performance on single-turn tasks by expanding internal 'thinking tokens' before answering, agentic web navigation introduces a trade-off under fixed test-time compute budgets. Developers can allocate budget to per-step deliberation ('think more') or to executing a higher volume of actions in the environment ('act more'). Shen points out that acting more gives agents an information advantage: direct interaction allows models to retrieve fresh external feedback, inspect alternative pages, and backtrack from dead ends instead of hallucinating over static observations. Test-Time Interaction via Expanding Horizons (16:15) To operationalize interaction scaling, the research team introduced the Test-Time Interaction (TTI) framework, which trains models using an online reinforcement learning loop that filters successful rollout trajectories to update policy weights. TTI employs an expanding-horizon curriculum that progressively raises the allowable step limit per task, teaching the model behaviors such as speculative exploration and error recovery rather than forcing brittle, greedy actions. Applied to DeepMind's open-source multimodal Gemma 3 12B model, this approach established state-of-the-art results on competitive open-source web agent benchmarks. Shen concludes that because dynamic interaction fundamentally distinguishes agents from conversational chatbots, scaling test-time interaction and horizon lengths offers a more effective path to production-grade web agents than relying exclusively on single-turn reasoning token scaling.

精炼阅读摘要

卡耐基梅隆大学（Carnegie Mellon University）研究学者沈俊宏（Junhong Shen）提出，通用大语言模型即使经过复杂外部支架的封装，在网络原生执行中仍存在性能瓶颈；开发高性能网络智能体（Web Agents）的关键在于利用真实生产工作流数据对模型本体进行专业化微调，并在测试期计算中扩大环境交互探索步骤（“多行动”），而非仅依赖单步内部思考链（“多思考”）。智能体闭环与通用脚手架的局限 (4:16) 沈俊宏指出，网络智能体与封闭文本对话的聊天机器人具有本质区别，其核心在于与外部浏览器环境构成的“感知-行动”持续交互闭环。智能体需要接收包含原始HTML、无障碍功能树（accessibility tree）以及视觉截图的多模态环境观测，结合任务目标与历史动作序列，生成点击、文本输入或键盘快捷键等离散操作，以逐步推进并改变页面状态。当前行业构建网络智能体的主流范式，普遍是在GPT-4o或Claude等通用前沿专有模型之上搭建复杂的外部脚手架系统，例如设计多阶段任务分解流水线、轨迹记忆检索模块以及候选动作评估器。然而，研究团队强调这种重度依赖外部脚手架的架构存在明显的性能瓶颈。通用基础模型主要基于常规网页文本进行预训练，缺乏对底层结构化HTML布局和浏览器原生交互逻辑的充分理解，开箱即用的网页导航能力与人类存在显著差距。复杂的外部提示词工程或外挂模块无法根本弥补基础模型缺乏具身接地的短板；学术基准测试WebArena的评测榜单也表明，排名前列的方案均依赖于专业网页交互数据集的后训练微调，而非单纯依靠通用模型外包脚手架。生产工作流微调与ScribeAgent (7:27) 为了实现网络智能体的专业化，卡耐基梅隆大学团队与企业工作流文档平台Scribe展开合作。Scribe通过浏览器插件记录真实员工在企业软件中的点击与按键操作，并利用AI自动生成结构化的步骤指南。研究人员从其生产环境中抽样整理出约20,000条真实人类交互轨迹，作为监督微调（SFT）训练集。这些真实轨迹将环境输入与人类操作精确配对，在开源基座模型上进行训练后表现出显著的性能跃升，且性能提升幅度与模型容量及训练Token总量呈现明确的正相关缩放规律。基于该数据方案，团队训练并发布了ScribeAgent Small（7B参数）和ScribeAgent Large（32B参数）两款开源模型。在团队内部的网页任务评估基准测试中，仅有70亿参数的ScribeAgent Small表现超过了OpenAI的o1-mini等大规模闭源专有模型；而在权威学术基准WebArena上，ScribeAgent也取得了顶尖水准（state-of-the-art）。沈俊宏指出，这证明基于真实人类轨迹数据的专业化微调，能够为网页智能体打下坚实且持久的性能基石。测试期计算中多思考与多行动的权衡 (14:20) 2024年底具备内部推理链（thinking tokens）的测试期推理模型兴起后，研究团队进一步探讨了推理范式在智能体领域的应用。早期ScribeAgent采取网页观测到操作指令的直接映射机制，而引入测试期显式思考链后，智能体可以在不可逆操作前进行多约束权衡。例如在酒店预订场景中，智能体能够暂停并综合考量价格、地理位置、便利设施与用户评价，从而避免由于盲目点击首个搜索结果而导致的任务失败。然而，沈俊宏强调多步交互环境与单轮问答任务存在根本不同：网络任务天然具有多步性和动态反馈机制。在测试期计算预算固定的前提下，系统面临着分配计算资源给单步内部推演（“多思考”）还是外部操作执行（“多行动”）的战略抉择。研究表明，“多行动”具备单步思考无法企及的优势——它能够从外部真实世界检索最新客观信息。通过点击多个搜索结果、浏览替代页面以及在遇到死胡同时主动回溯，智能体能依靠真实环境反馈修正路线，有效避免基于静态陈旧观测而产生的模型幻觉。地平线扩展强化学习与TTI框架突破 (16:15) 为了系统性利用环境交互带来的扩展收益，该团队提出了测试期交互（Test-Time Interaction, TTI）强化学习框架。TTI采用在线两阶段强化学习闭环：首先将智能体部署至真实网页环境中展开交互并采集行为轨迹，随后筛选出成功完成任务的高质量样本，用于更新多模态策略模型。 TTI的核心机制在于对单任务最大允许交互步数（即交互地平线 Horizon）实行递增课程训练。随着训练进程推进，系统逐步放宽任务交互的步数上限，从而在训练集中积累更长、更复杂且具备充分探索特性的样本。这种机制摒弃了迫使模型寻找脆弱、贪婪捷径的传统方式，促使智能体学会在复杂场景中展开推测性探索、跨页面交叉核对与错误恢复等高级行为。基于DeepMind开源的多模态模型Gemma 3 12B，该系统在全开源数据和模型设置下刷新了多项权威网络智能体评测的顶尖成绩。沈俊宏总结认为，交互是自主智能体区别于传统对话系统的本质属性，网络智能体研究不应局限于单轮对话式的思考Token堆叠，而应重点扩展测试期交互尺度。不过，该方法目前依赖在线轨迹过滤与策略更新机制，未来如何在更为开放、高风险和长流程的生产环境中平衡探索成本、安全性与执行效率，仍是该技术走向实际工业落地有待解决的关键课题。

Detailed notes — zh

## Background 该视频由科技会议组织 Ai4 于 2025 年 8 月 28 日发布在 YouTube 平台，演讲者为卡耐基梅隆大学（CMU）研究员沈俊宏（Junhong Shen）。视频内容主要围绕 Web Agent 的演进展开，探讨了如何摆脱对通用大模型及繁琐提示工程的依赖，转而通过真实生产级工作流数据微调开源模型，并探索测试时环境交互的新型扩展范式。 ## TL;DR 沈俊宏首先剖析了传统 Web Agent 依赖通用闭源模型与外部脚手架时面临的泛化瓶颈，展示了通过与 Scribe 合作提取两万条人类真实网页操作轨迹所微调出的 ScribeAgent，该模型在基准测试中展现出超越前沿闭源模型的表现。随后，演讲重点探讨了将推理模型的测试时计算扩展迁移至智能体领域的挑战，深入对比了“单步深思”（Think more）与“多步行动”（Act more）的取舍。最后，团队提出了 TTI（Test-Time Interaction）强化学习框架，通过动态扩大步数上限激励智能体在真实网页中进行深度探索与试错回溯，并在 Gemma 3 12B 开源基准上刷新了 SOTA 成绩，指明了扩展多步环境交互这一全新方向。 ## Atmosphere & style 整场演讲呈现出干练、严谨且充满前沿探索张力的学术科技报告氛围。演讲者开场坦言在最后一刻为了加入最新研究成果而临时变更课题，为整场分享注入了一种紧贴科研最前沿的临场感与兴奋感。叙述逻辑紧凑而富有启发性，从工程微调实践自然过渡到底层思考与行动权衡的深度思考，让观众能够直观感受到智能体技术正在从“单步直觉反应”迈向“深思熟虑与自主环境探索”的范式跃迁。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=Llm5SdguQOY&t=4s) 大模型智能体（LLM Agent）与传统对话聊天机器人的根本分水岭在于必须与外部环境进行动态交互并形成感知-决策-行动的闭环。 - [01:59](https://www.youtube.com/watch?v=Llm5SdguQOY&t=119s) 基于通用大模型搭配复杂脚手架和提示工程的传统 Web Agent 方案存在结构冗余，且未能从底层解决模型对 HTML 与网页语义理解不足的问题。 - [01:59](https://www.youtube.com/watch?v=Llm5SdguQOY&t=119s) 提升 Web Agent 性能的关键在于对大模型本体进行专业化训练，学术基准如 WebArena 上领先的方案均采用了专业化后训练的路线。 - [07:27](https://www.youtube.com/watch?v=Llm5SdguQOY&t=447s) CMU 团队与 Scribe 合作构建了包含 2 万条人类真实网页操作轨迹的数据集，通过有监督微调打造了专精网页任务的 ScribeAgent 系列开源模型。 - [07:27](https://www.youtube.com/watch?v=Llm5SdguQOY&t=447s) 微调后的 ScribeAgent 展现出明确的 Scaling 效应，仅 7B 规格的小尺寸模型在内部评测中即可超越 o1-mini 等闭源前沿模型。 - [11:52](https://www.youtube.com/watch?v=Llm5SdguQOY&t=712s) 推理模型（Reasoning Models）的兴起催生了将测试时计算扩展（Scaling Test-Time Compute）引入智能体每步决策的新范式。 - [14:20](https://www.youtube.com/watch?v=Llm5SdguQOY&t=860s) 智能体任务开辟了除每步推理 Token 外的新扩展维度，即在给定计算预算下权衡“多思考”（单步深度推理）与“多行动”（增加环境交互步数）。 - [14:20](https://www.youtube.com/watch?v=Llm5SdguQOY&t=860s) 在环境中实际采取多步行动能为智能体带来全新外部信息，赋予其试错、广泛探索与回溯修正的能力，突破了贪心决策的局限。 - [16:15](https://www.youtube.com/watch?v=Llm5SdguQOY&t=975s) TTI 强化学习框架通过在训练循环中动态放宽智能体最大允许步数，有效抑制短期捷径行为并激励复杂长程探索策略的形成。 - [17:35](https://www.youtube.com/watch?v=Llm5SdguQOY&t=1055s) 基于开源 Gemma 3 12B 底座与 TTI 方案的组合，在多个公开 Web Agent 基准测试中创下了开源体系下的 SOTA 记录。 - [17:35](https://www.youtube.com/watch?v=Llm5SdguQOY&t=1055s) 将 Scaling Law 的研究重心从单纯的单步 Token 生成转向多步环境交互扩展（Scaling Interaction），是突破高阶智能体长程规划瓶颈的关键路径。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Llm5SdguQOY&t=4s) 演讲开场与核心主题：利用真实工作流数据构建专业 Web Agent 来自卡耐基梅隆大学（Carnegie Mellon University, CMU）的博士生沈俊宏（Junhong Shen）开启了本场分享。他首先向观众解释，现场展示的幻灯片标题与会场外部海报略有出入，这是因为他在最后一刻决定加入团队最新的一项研究成果。他认为这项新工作极为关键，很可能为智能体扩展（Agent Scaling）开启全新的范式，并预计学术界与工业界的研究人员在接下来的几个月内会积极投入该方向的研究。 ```mermaid flowchart LR accTitle: Web Agent 交互循环流程图 subgraph Environment ["外部 Web 环境 (Browser)"] A["观察输入 (Observation) HTML / 可访问性树 / 截图"] end subgraph AgentSystem ["LLM Web Agent"] B["大模型处理与决策"] end subgraph ActionExec ["执行阶段"] C["产生 Action Click / Type / Key Combos"] end A --> B B --> C C --> Environment ``` 在介绍深入技术前，演讲者厘清了大语言模型（LLM）智能体与传统聊天机器人（Chatbots）的本质区别。两者的关键分水岭在于：Agent 任务必须与**外部环境**进行动态交互。环境向 Agent 提供观察输入（Observations），Agent 对这些输入进行解析后做出决策并发出动作指令（Actions），动作在环境中执行后又会反馈新的观察，如此周而复始构成闭环。对于数字智能体而言，外部环境可以是集成开发环境（IDE）、游戏世界、移动端操作系统，或是最为常见的网络浏览器。本次演讲的核心便聚焦于 **Web Agent（网络智能体 / 网页导航任务）**。 --- ### [01:59](https://www.youtube.com/watch?v=Llm5SdguQOY&t=119s) Web Agent 的问题形式化定义与传统范式瓶颈演讲者详细拆解了 Web Agent 在每个步骤所面临的输入与输出形式： - **观察空间（Observation）：** 包含用户的最终目标（Task Goal，如订机票、预订酒店）、起始网址（Starting URL），以及当前网页状态。网页状态的表达方式包括原始 HTML 源码、浏览器特有的可访问性树（Accessibility Trees）以及网页截图（多模态视觉输入）。 - **动作空间（Action）：** 包括点击按钮、点击超链接、向文本框输入字符序列、按下组合快捷键（如 `Ctrl + C`、`Ctrl + V`）等。模型生成的输出会被转换成可在浏览器中直接执行的代码或控制指令。 - **历史上下文（History）：** Agent 会保留过去已执行的动作序列，明确自身当前所处的流程阶段。针对 Web Agent 的主流实现方案，演讲者指出了当前基于通用大模型（如 GPT-4o、Claude 等前沿闭源模型）搭配提示工程与复杂脚手架（Scaffoldings / Agent Frameworks）的根本局限： 1. **结构冗余与泛化难题：** 传统方法依赖极其复杂的多阶段工作流拆解，辅以外部记忆系统（Memory）或动作评估器（Evaluator），但这种外挂模块的方式并未从底层改善模型对网页语义的理解。 2. **缺乏针对性定制与专业化训练：** 通用前沿大模型在预训练时并未针对 HTML 代码、浏览器结构及网络交互数据进行专门优化，导致其在零样本或少样本提示下的表现与人类水平存在显著差距。 3. **性能提升的关键在于模型本体：** 演讲者强调，与其把精力全部花在繁琐的外挂脚手架上，不如直接提升 LLM 模型自身的网页理解与操作能力。在学术界广泛认可的 Web 评测基准 **WebArena** 榜单上，排名前列的方案无一例外都采用了基于专业网页数据微调/后训练的“专业化智能体”（Specialized Agents）。 --- ### [07:27](https://www.youtube.com/watch?v=Llm5SdguQOY&t=447s) ScribeAgent：从真实工作流到微调专业 Web 模型为了让模型学会人类真实的网页操作逻辑，CMU 团队与 AI 工作流文档生成初创公司 Scribe 展开了深度合作。Scribe 是一款浏览器插件产品，用户只需点击录制并在网页上完成日常操作，软件即可自动捕获操作流并利用 AI 提炼生成图文并茂的步骤指南（如用于 HR 新员工入职培训或业务 SOP 文档）。这种产品特性天然积累了大量由真实人类操作产生的“黄金交互轨迹”（Golden Trajectories）。 | 维度 / 特征 | 细节描述 | |:--- |:--- | | **合作背景** | 与工作流自动化工具 **Scribe** 合作，获取高价值真实人类操作行为数据 | | **数据集规模** | 抽取并重构了包含 **20,000 条轨迹（20k Trajectories）** 的真实工作流子集 | | **训练方法** | 采用有监督微调（Supervised Fine-Tuning, SFT），将轨迹格式化为标准的输入观察与目标动作对 | | **模型尺寸规格** | 分别训练开源基础模型，构建 **ScribeAgent-7B（Small）** 与 **ScribeAgent-32B（Large）** | | **实验核心结论** | 1. 随着训练数据量增加，模型表现呈现明确的 Scaling 效应； 2. 经过微调后，即使是 7B 小尺寸开源模型，在内部评测上的表现也能超越 o1-mini 等闭源前沿模型； 3. 在 WebArena 基准测试中夺得半年前的 SOTA（最先进）成绩。 | --- ### [11:52](https://www.youtube.com/watch?v=Llm5SdguQOY&t=712s) 范式转移的开端：推理模型（Reasoning Models）的崛起在总结 ScribeAgent 的突破后，演讲者话锋一转，解释了为何要特别强调这项成果诞生于“半年前”。因为从 2024 年夏季开始，整个大模型领域的技术格局发生了剧烈震荡：在 2024 年夏季之前，业界使用的模型大多是“直接输入用户 Prompt、直接输出最终动作/回答”的单步直觉型模型；然而最近几个月内，**推理模型（Reasoning Models）** 迅速席卷学术界与工业界。这类模型在输出最终答案或执行动作之前，会显式生成一段长思考过程（Chain-of-Thought / Reasoning Tokens），从而在复杂规划与逻辑推理任务中展现出降维打击般的解决能力，这也为 Web Agent 的后续架构进化带来了全新的探索空间。 ### [12:34](https://www.youtube.com/watch?v=Llm5SdguQOY&t=754s) 推理范式向智能体的迁移与测试时计算扩展随着推理模型在编程和数学等复杂单轮任务中展现出强大实力，研究团队开始思考如何将这种“生成多步推理轨迹（Reasoning Tokens）以提升最终表现”的范式引入到智能体（Agent）场景中。以团队此前构建的 ScribeAgent 为例，其初始形态非常直接：模型在观察到网页当前状态后，立即输出下一步操作动作（Action）。然而，在新的范式设想下，模型在做出决策前应当先输出一段思考链（Thinking Traces），通过在决策前进行更深度的内部推演来提升任务成功率。这种思路本质上是将测试时计算扩展（Scaling Test-Time Compute）引入智能体系统。例如在预订特定日期的酒店任务中，用户往往不希望智能体在看到某个选项时就急躁地点击预订按钮，而是期望它在行动前能够“多琢磨琢磨”（"Ruminate more / think more"）。智能体需要在内部权衡不同维度的约束：这个选项是否真的是最优解？价格是否合理？地理位置如何？用户评分和评价反馈是否达到预期？这种在每一步行动前分配更多 Token 进行深思熟虑的机制，构成了传统的每步推理扩展思路。 ```mermaid flowchart LR accTitle: Agent Per-Step Reasoning Process subgraph S1["传统模式"] A["观察网页状态"] --> B["直接输出动作"] end subgraph S2["推理增强模式"] C["观察网页状态"] --> D["生成思考轨迹 (价格/位置/评价推演)"] D --> E["输出决策动作"] end ``` --- ### [14:20](https://www.youtube.com/watch?v=Llm5SdguQOY&t=860s) “多思考”还是“多行动”：探索环境交互的新扩展轴尽管单步深度推理具有明显优势，但单轮数学问题与智能体任务之间存在一个根本性差异：智能体任务是多步骤的，必须与外部环境展开持续互动。这一特性开辟了一条全新的扩展维度——除了扩展单步推理的 Token 数量（Per-Step Reasoning Tokens），还可以扩展智能体与环境交互的动作步数（Interaction Steps）。研究团队在其名为《Thinking versus Doing》的新论文中，重点探讨了在固定总计算预算（Compute Budget）下，“想得更多”与“做得更多”（"Think more versus act more"）之间的权衡与取舍。团队强调，向“增加行动步数”这一新方向进行扩展具有不可替代的价值，因为**在环境中实际采取行动能够获取全新的外部信息**，而这是单步空想完全无法实现的。依然以酒店预订为例，如果智能体仅仅停留在当前页面深思，它所掌握的信息量是受限的；但如果允许它点击进入多个不同页面、发起多次检索，它执行的动作越多，获取到的可用备选方案就越丰富，从而可以在更全局的视野下挑选最优解。这种机制赋予了智能体更丰富的探索能力（Exploration）以及在遇到死胡同或次优解时的回溯能力（Backtracking）。 - **计算预算分配权衡**：在给定的总推理算力限制下，算力既可以用于“单步深思”（生成更多思考 Token），也可以用于“多步交互”（执行更多环境操作）。 - **外部信息增益**：单纯在内部推理无法凭空产生新数据，而多步交互通过翻页、检索等操作持续从真实网页拉取新上下文。 - **行为模式丰富度**：多步行动支持智能体执行试错、广泛探索与回溯修正，突破了贪心决策（Greedy Selection）的局限。 --- ### [16:15](https://www.youtube.com/watch?v=Llm5SdguQOY&t=975s) TTI 强化学习框架：逐步扩大时域以鼓励复杂探索为了验证增加交互步数是否能切实使智能体受益，研究团队设计并训练了专门激励智能体在环境中进行多步探索的算法体系，即测试时交互框架（Test-Time Interaction, TTI）。TTI 采用了一种两阶段的强化学习（RL）闭环：第一阶段将智能体部署在真实环境中收集交互轨迹（Trajectories）；第二阶段筛选出其中成功的轨迹，并用其对智能体策略进行更新。 ```mermaid flowchart TD accTitle: TTI Reinforcement Learning Loop subgraph Loop["TTI 强化学习闭环"] A["智能体部署到真实环境"] --> B["采集多步交互轨迹"] B --> C["筛选成功轨迹"] C --> D["更新策略模型"] D -->|推进训练轮次| E["逐步提高最大允许步数 (Max Steps)"] E --> A end ``` TTI 机制的核心亮点在于其动态调整时域的设计：随着训练的持续推进，框架会主动增加智能体执行任务所允许的最大步数上限（Horizon / Max Steps）。这意味着在更新策略时，训练数据集中所包含的成功轨迹会变得越来越长、内容越来越丰富。这种逐步放宽步数限制的机制，能够有效抑制智能体依赖短期捷径的贪心行为，促使其自发学会更复杂、更具深度的长程行为模式与探索策略。 --- ### [17:35](https://www.youtube.com/watch?v=Llm5SdguQOY&t=1055s) Gemma 3 开源基准评测与智能体交互扩展的未来在实证评估方面，研究团队在一系列主流开源网页智能体基准测试（Web Agent Benchmarks）上对 TTI 进行了严谨检验。实验采用了 DeepMind 推出的多模态开源模型 Gemma 3 12B 作为基础底座。评测结果表明，基于该开源模型与 TTI 方案的组合，在多个公开基准上成功刷写了开源模型与开源数据体系下的最优性能记录（SOTA）。演讲最后总结道，当前学术界与工业界虽然有大量工作集中在优化智能体模型架构本身，但随着大模型推理（Reasoning）与扩展规律（Scaling Laws）的全面普及，研究者必须重新审视智能体究竟还有哪些维度可供扩展。智能体任务与传统单轮对话机器人（Chatbot）最核心的区别，正是这套“多步行动与环境交互”的机制。因此，扩展行动数量与交互频次（Scaling Interaction），不仅是当前突破智能体长程规划瓶颈的关键路径，也是未来智能体前沿研究中最具前景的发展方向。 - **模型底座**：采用开源多模态大模型 Gemma 3 12B 进行训练与验证。 - **基准表现**：在多个开源 Web Agent 基准测试中，取得了同类开源模型与公开数据条件下的最高水平。 - **核心结论**：智能体区别于聊天机器人的本质在于“行动”，将 Scaling Law 的研究重心从单纯的单步 Token 生成转向多步环境交互，是通往高阶通用智能体的必由之路。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=Llm5SdguQOY&t=4s) 这项新工作极为关键，很可能为智能体扩展（Agent Scaling）开启全新的范式，并预计学术界与工业界的研究人员在接下来的几个月内会积极投入该方向的研究。 — Junhong Shen > [01:59](https://www.youtube.com/watch?v=Llm5SdguQOY&t=119s) 与其把精力全部花在繁琐的外挂脚手架上，不如直接提升 LLM 模型自身的网页理解与操作能力。 — Junhong Shen > [07:27](https://www.youtube.com/watch?v=Llm5SdguQOY&t=447s) 经过微调后，即使是 7B 小尺寸开源模型，在内部评测上的表现也能超越 o1-mini 等闭源前沿模型。 — Junhong Shen > [11:52](https://www.youtube.com/watch?v=Llm5SdguQOY&t=712s) 这类模型在输出最终答案或执行动作之前，会显式生成一段长思考过程（Chain-of-Thought / Reasoning Tokens），从而在复杂规划与逻辑推理任务中展现出降维打击般的解决能力。 — Junhong Shen > [12:34](https://www.youtube.com/watch?v=Llm5SdguQOY&t=754s) Ruminate more / think more — Junhong Shen > [14:20](https://www.youtube.com/watch?v=Llm5SdguQOY&t=860s) 在环境中实际采取行动能够获取全新的外部信息，而这是单步空想完全无法实现的。 — Junhong Shen > [14:20](https://www.youtube.com/watch?v=Llm5SdguQOY&t=860s) Think more versus act more — Junhong Shen > [16:15](https://www.youtube.com/watch?v=Llm5SdguQOY&t=975s) 这种逐步放宽步数限制的机制，能够有效抑制智能体依赖短期捷径的贪心行为，促使其自发学会更复杂、更具深度的长程行为模式与探索策略。 — Junhong Shen > [17:35](https://www.youtube.com/watch?v=Llm5SdguQOY&t=1055s) 智能体区别于聊天机器人的本质在于“行动”，将 Scaling Law 的研究重心从单纯的单步 Token 生成转向多步环境交互，是通往高阶通用智能体的必由之路。 — Junhong Shen ## Mentioned Carnegie Mellon University, Junhong Shen, Scribe, ScribeAgent, ScribeAgent-7B, ScribeAgent-32B, WebArena, GPT-4, GPT-4o, Claude, o1-mini, Gemma 3, Google DeepMind, Thinking versus Doing, TTI (Test-Time Interaction) ## Video description Junhong Shen, Researcher, Carnegie Mellon University Abstracts Large Language Model (LLM) agents are rapidly improving to handle increasingly complex web-based tasks. Most of these agents rely on general-purpose, proprietary models like GPT-4 and focus on designing better prompts to improve their planning abilities. However, general-purpose LLMs are not specifically trained to understand specialized web contexts such as HTML, and they often struggle with long-horizon planning. We explore an alternative approach that fine-tunes open-source LLMs using production-scale workflow data collected from over 250 domains corresponding to 6 billion tokens. This simple yet effective approach shows substantial gains over prompting-based agents on existing benchmarks. [Source](https://www.youtube.com/watch?v=Llm5SdguQOY)

Source quotes

[0:04] Junhong Shen: 这项新工作极为关键，很可能为智能体扩展（Agent Scaling）开启全新的范式，并预计学术界与工业界的研究人员在接下来的几个月内会积极投入该方向的研究。 [1:59] Junhong Shen: 与其把精力全部花在繁琐的外挂脚手架上，不如直接提升 LLM 模型自身的网页理解与操作能力。 [7:27] Junhong Shen: 经过微调后，即使是 7B 小尺寸开源模型，在内部评测上的表现也能超越 o1-mini 等闭源前沿模型。 [11:52] Junhong Shen: 这类模型在输出最终答案或执行动作之前，会显式生成一段长思考过程（Chain-of-Thought / Reasoning Tokens），从而在复杂规划与逻辑推理任务中展现出降维打击般的解决能力。 [12:34] Junhong Shen: Ruminate more / think more [14:20] Junhong Shen: 在环境中实际采取行动能够获取全新的外部信息，而这是单步空想完全无法实现的。 [14:20] Junhong Shen: Think more versus act more [16:15] Junhong Shen: 这种逐步放宽步数限制的机制，能够有效抑制智能体依赖短期捷径的贪心行为，促使其自发学会更复杂、更具深度的长程行为模式与探索策略。 [17:35] Junhong Shen: 智能体区别于聊天机器人的本质在于“行动”，将 Scaling Law 的研究重心从单纯的单步 Token 生成转向多步环境交互，是通往高阶通用智能体的必由之路。

Source quotes

Junhong Shen: Large Language Model (LLM) agents are rapidly improving to handle increasingly complex web-based tasks. Junhong Shen: general-purpose LLMs are not specifically trained to understand specialized web contexts such as HTML, and they often struggle with long-horizon planning. Junhong Shen: We explore an alternative approach that fine-tunes open-source LLMs using production-scale workflow data collected from over 250 domains corresponding to 6 billion tokens. [0:04] Junhong Shen: potentially leading to a new paradigm in agent scaling [0:04] Junhong Shen: While chatbots operate in closed text exchanges, an agent continuously interacts with an external environment. [4:16] Junhong Shen: general-purpose LLMs are inherently unspecialized for web-native execution. [4:16] Junhong Shen: Complex scaffolding cannot fully compensate for a foundation model that lacks native grounding in web interaction. [4:16] Junhong Shen: the greatest headroom for improvement comes from specializing the LLM itself before adding scaffolding. [14:20] Junhong Shen: scaling the number of interaction steps rather than merely scaling the internal reasoning tokens generated at each step. [14:20] Junhong Shen: acting more provides a distinct advantage that per-step reasoning alone cannot offer: it enables the retrieval of fresh, ground-truth information from the external world. [14:20] Junhong Shen: Acting directly in the environment uncovers latent state information and real-world feedback that static per-step ruminations cannot produce. [16:15] Junhong Shen: Because interaction is the defining attribute of autonomous agents, scaling test-time interaction represents one of the most promising frontiers for building capable, production-grade web agents.

Original transcript

[0:04] Hi everyone, my name is Chinho and I'm [0:06] currently a PhD student at Carnegie [0:08] Melon University and today I want to [0:11] share some of our recent work on [0:12] developing specialized web agents using [0:15] LLMs and uh real world data. You might [0:19] notice my title here is slightly [0:20] different from what's shown uh outside [0:22] on the poster and that's because at the [0:25] very last minute I decided to add [0:27] something uh which is our very recent [0:30] like a research work that sorts of I [0:34] think it's important because I think [0:35] it's potentially leading to a new [0:37] paradigm in agent scaling and uh we're [0:41] expecting researchers to like actively [0:43] working on it maybe in the next few [0:45] months. So that's why I really want to [0:46] share uh this new work with you guys. [0:50] Okay, cool. So let's start with some [0:52] basics. So what are uh LM agents? From [0:56] my understanding, I think a key [0:57] difference between uh chat bots and [1:00] agents is that for agentic tasks, there [1:03] are there are external environments that [1:07] agents need to interact with more [1:09] specifically. So the environment will [1:12] produce some uh observations like inputs [1:17] and the LMS or the agent itself will try [1:19] to process these inputs and in turn [1:22] issue some actions which will be [1:24] executed in the environment. [1:26] So this loop goes uh on and on [1:30] iteratively and that's what constitutes [1:32] the agentic tasks. [1:35] So for example for digital agents the [1:37] environment can take form as like [1:40] ideides or uh game uh worlds or uh [1:44] mobile uh like and lastly this uh [1:47] browser stuff. So in my talk I'll be [1:50] mainly focusing on web use or web agents [1:54] or um web navigation tasks. [1:59] Okay. So more specifically for web [2:02] agents. The observation [2:04] mostly consists of a task goal which you [2:08] want your agent to complete like help [2:10] you to book a flight or book a hotel and [2:13] then you will have a starting URL where [2:16] so so that the agent knows like where to [2:18] start and where to uh where to start uh [2:21] so that it can complete the tasks and [2:24] then the most important part is actually [2:26] the web page uh like at each stage you [2:28] can observe what's going on that can [2:31] take several forms. forms such as like [2:33] HTMLs or um accessibility trees which [2:36] are unique to some uh to each like [2:39] browser and also screenshots and this is [2:43] like a multimodel input. [2:47] Okay, so the agent will take a look at [2:50] all of these observations. What it will [2:52] do next is to try to [2:55] determine an action which will be [2:57] executed uh in the browser. And [3:00] typically we consider uh actions like [3:03] clicking a button, clicking a link or um [3:06] you type a sequence of characters into a [3:09] text box uh or you press some keyboard [3:12] combinations like uh control ctrl ctrl + [3:15] v this type of uh actions [3:19] and for example like your may output [3:22] something like this what's shown in the [3:24] text box and this can be translated into [3:28] like a command and or um a line of code [3:32] that can be actually executed in the [3:35] browser. [3:38] Okay, so to summarize what is web agent? [3:42] Web agent is basically you have a task [3:44] and then the agent will observe uh [3:47] everything on the web page and note also [3:49] that it has most of the time it has [3:52] access to the past actions that it has [3:55] performed so that it knows like where uh [3:59] at what stage it is at like right now. [4:02] And then what the agent will do is that [4:05] it will produce an action and this loop [4:09] will go on and on until the task is [4:12] completed. [4:16] Okay. So um then that's a problem [4:20] formulation. Let's now focus on the [4:23] agent itself like how is the agent [4:25] instantiated? I think um traditionally [4:29] or the simplest way of instantiating an [4:32] web agent is that you take some general [4:34] purpose LMS like GB uh 405 or like cloud [4:38] or any of the frontier models and then [4:40] what you do is that you build uh [4:43] scaffoldings or frameworks on top of it [4:45] like by scaffolding or frameworks I'm uh [4:50] referring to the this kind of like super [4:52] complicated workflow charts where you [4:55] can You potentially decompose the task [4:59] into several stages. And you might also [5:02] have external modules to help you [5:04] improve the task success rate. Like um [5:08] uh for instance uh some of the agent [5:10] workflows has like a memory system where [5:12] you can store past successful experience [5:15] into it and then like retrieve it to [5:17] help you complete the current task. or [5:19] um some workflows might have evaluator [5:23] uh modules which assess the quality of [5:26] your proposed action before you actually [5:29] execute it in the uh environment. [5:33] Okay, so this is kind of the dominant [5:35] paradigm uh that I know of. However, a [5:38] key limitation of this paradigm is that [5:42] everything is not customized. Like [5:45] you're relying heavily on the general [5:47] purpose model on the pre-training [5:48] knowledge of the model and you're not [5:50] allowed to or it's not that easy to uh [5:55] customize your agent into and make it a [5:59] specialized uh like a system for your [6:02] specific use case. [6:05] And also actually because of the because [6:08] of the fact that the general purpose [6:10] models they're not trained specifically [6:11] on web data they are most of the time [6:14] not very good. Um so the performance gap [6:18] between like frontier models without any [6:21] specialization and human performance is [6:23] actually quite large. [6:28] Okay. So then our question becomes like [6:32] how do we improve this thing like as a [6:34] researcher we don't want to just depend [6:36] on GBD4 or like cloud but we actually [6:39] want to improve the whole performance of [6:41] the web agent. So that's why we think [6:44] the main [6:47] like a point uh or like the large room [6:50] of improvement actually lies in how we [6:52] can improve the LM itself before [6:55] augmenting it with any external modules. [6:58] And just a teaser um of how this might [7:01] work. So this this is a leaderboard of a [7:04] very widely used um web agent benchmark [7:07] in academia which is called like web [7:09] arena. And as you can see, the top [7:11] performing agents right now are all [7:14] specialized agents, meaning that they're [7:15] fine-tuned or like post-trained on [7:17] specialized web data data instead of you [7:20] just take a general purpose model and [7:21] then try to add some like prompting [7:23] techniques uh on top of it. Okay. Yeah. [7:27] And then like the first work I want to [7:29] talk about today is actually this scribe [7:32] agent work. And uh so scribe agent why [7:37] is it called like this? It's because we [7:39] collaborate this like work with a [7:42] company uh called Scribe, which is a uh [7:46] AI workflow documentation startup. [7:49] >> Scribe turns any process into a [7:51] step-by-step guide instantly. To capture [7:54] a Scribe, click the extension and select [7:56] start capture. Then just walk through [7:58] your process and Scribe will [8:00] automatically create a step-by-step [8:01] guide. After capturing your process, you [8:05] can add any customizations you like. You [8:07] can use AI to generate the title, [8:09] description and icon. [8:12] >> Okay, so this software uh is basically [8:15] like a browser extension which records [8:18] whatever things you do on the web and [8:21] then translate it into a step-by-step [8:23] guide. It's like this is very helpful [8:25] for instance uh if you're an HR or you [8:27] want to show some new employees on how [8:29] to complete some uh onboarding tasks [8:33] and it's actually uh because of the [8:35] nature of this software scribe has a [8:37] large amount of real world data or real [8:41] world uh workflow data which consists of [8:43] um authentic use cases and like [8:47] authentic trajectories that human [8:49] actually like how humans uh would do in [8:53] or to perform a specific task and uh we [8:56] decided to collaborate with them because [8:58] we're really interested in seeing how we [9:00] can leverage this golden trajectories [9:03] produced by real humans and to make it [9:05] improve uh to use it to improve like LMS [9:08] in web navigation. So more specifically [9:11] we take a subset of their data set and [9:13] these are some statistics. Uh for [9:16] instance there are like 20k trajectories [9:19] and uh we our goal is to see how this [9:23] data set uh consists of real world data [9:26] can be used to augment uh LMS. [9:31] Okay. So as for how we actually do the [9:35] training is super easy. It's just [9:37] supervised finetuning. So we take the [9:39] data set and then we translate or [9:42] rewrite each uh piece of example into a [9:45] like a trajectory form consisting of the [9:48] input and the output action and then we [9:51] just use this uh golden data set to [9:54] fine-tune [9:56] open source models and what we actually [9:59] found is that like first of all we tried [10:01] a bunch of models and for all of them [10:06] fine-tuning or supervised fine tuning [10:08] ing on this data set just significantly [10:11] improves the performance of LMS uh in [10:14] this web navigation tasks and moreover [10:16] as the model size grows uh you also see [10:19] a larger improvement [10:22] and there's also this data scaling part [10:25] where we find that like you use more [10:28] data you get better performance that's [10:30] that's very like intuitive so ultimately [10:32] we choose this like two uh models uh to [10:36] develop uh scribe agent one is like a 7B [10:39] parameter and the other is a 32B [10:41] parameters. So we just train them on uh [10:44] using all the data like we have and then [10:47] develop like scribe agent small and [10:49] scribe agent large [10:52] and so these are some performance on the [10:54] internal data that we use to evaluate uh [10:57] the performance of web agents. As you [10:59] can see, even though we use open-source [11:01] models and even though we use the 7 [11:03] billion models, after fine-tuning, the [11:06] performance is uh better even than like [11:09] 01 mini uh which I think shows the power [11:13] of uh model specialization. [11:16] Okay. And remember this benchmark that I [11:18] uh just I mentioned before like web [11:21] arena is super widely used right now in [11:24] academia for evaluating web agent [11:26] performance and we basically show that [11:29] scribe agent achieves the [11:31] state-of-the-art performance on this [11:33] benchmark and this result is actually [11:37] like half a year ago. So yeah, I think [11:42] the leaderboard has been updating [11:44] throughout, but like scrape agent still [11:46] remains a very strong performance on top [11:49] of it. Okay, so [11:52] why am I saying it's like a half year [11:54] ago? It's actually because things [11:56] changed a lot in the past half years. So [11:59] before like let's say like summer 2024 [12:04] what people usually use like the models [12:07] they're straightforward in the sense [12:08] that the model just directly gives you [12:11] an answer uh to the question or the user [12:15] prompt that you input. However um during [12:18] the past few months there's a new class [12:20] of models that's [12:23] been super popular and that's like the [12:26] reasoning models. It's more specifically [12:28] it's like before outputting the actual [12:31] answer you um like the model generate [12:34] multiple reasoning like traces reasoning [12:36] tokens to help it improve the final [12:40] outcome. And this reasoning model has [12:42] been shown to be super powerful for more [12:45] complic complicated task like coding or [12:48] like math um reasoning. [12:51] And then that's why we begin to think [12:54] okay can we also apply this reasoning [12:56] paradigm to agentic settings not just to [12:59] like mass single term mass uh problems. [13:04] So this means that scribe agent we [13:09] want to improve it because the original [13:11] form of it is like a it observes a web [13:14] page and it directly outputs an uh [13:17] action. But right now what we're [13:20] thinking is can we also make the model [13:22] uh agent output some thinking traces and [13:25] does that probably helps it improve uh [13:29] the final task performance. [13:33] Okay. So this is the paradigm of uh [13:37] scaling test time compute in agentic [13:39] settings. For example, if you want your [13:41] agent to help you book a hotel uh at a [13:45] specific date, what you can or like what [13:48] you want the agent to do perhaps is that [13:51] before it actually clicks the book uh [13:54] button for a specific hotel, you might [13:57] want it to like ruminate more like think [13:59] more. Okay, is this one the final like [14:03] option that I really want to uh book? [14:06] like think about the price, think about [14:08] the location and think about the [14:12] uh like the quality and user reveals [14:14] everything like this. [14:17] Uh okay. [14:20] However, this is on the left side. This [14:23] is how people traditionally think [14:26] reasoning or like scaling can be applied [14:29] to aentic settings. But [14:32] as I said before, a key difference [14:34] between single term mass problem and [14:36] agentic task is that agentic tasks are [14:39] multi-step, which means that you get to [14:41] interact with the uh environment for [14:44] multiple steps. [14:47] And this leads to a new scaling axis [14:50] apart from uh this scaling per step [14:53] reasoning which is you can also scale [14:56] the number of interaction steps. And [14:58] this is a this is what our new work uh [15:01] thinking versus doing this new research [15:03] paper is focusing on. [15:06] Okay. So this just to help you better [15:08] understand. So if you have a total [15:10] compute budget for agentic task, you can [15:12] either scale the per step thinking [15:14] tokens or you can scale the number of uh [15:17] steps [15:19] that the agent performs to complete the [15:21] task. And that's like the think more [15:24] versus act more um trade-off. [15:29] Okay. So why do we perhaps want to scale [15:33] along this new direction? I think the [15:35] main reason is that if you have the [15:38] chance to act more in the environment, [15:39] you actually get to collect new [15:41] information and that's something like a [15:43] scaling per step scaling cannot give you [15:47] basically uh like if you for the same [15:51] hotel booking task if you click on [15:53] multiple pages or do multiple searches [15:56] uh the more actions you perform the more [15:59] kind of options there are that's uh [16:02] available and you can sort of pick which [16:05] one is better [16:07] and this enables like reacher uh reacher [16:10] action like exploration or backtracking. [16:15] Okay. So how do we then see if this [16:20] benefits agents or not? Uh we actually [16:23] train agents to incentivize them to [16:25] explore to do more steps in the [16:28] environment uh to help it uh perform [16:30] better. And that's this TTI work uh or [16:34] like test time interaction. Yeah, TTI [16:38] stands for test time interaction. It's [16:39] an RL framework that's like uh two step [16:43] uh on the first step you collect some [16:44] trajectory uh by deploying the agent in [16:47] the actual environment and in the second [16:49] stage you filter out the successful [16:52] trajectories and use that use them to [16:54] update your agents. This is how like uh [16:56] traditional online RL works. But what's [17:00] uh specific about TDI is that as [17:03] training goes on, it actually increases [17:06] the horizon or the max steps allowed um [17:11] for the agent to perform the task. That [17:13] means that as you go on like as training [17:15] goes on the training data set the agent [17:19] um [17:20] used to update will consist of more like [17:23] longer and longer trajectories, richer [17:25] and richer trajectories. This will [17:27] encourage it to show more complicated uh [17:30] behavior patterns instead of just a [17:33] greedy one. [17:35] Okay. So yeah, this basically shows uh [17:38] the performance of TTI and um we also [17:43] evaluated on a bunch of open-source [17:45] benchmarks uh web agent benchmarks and [17:49] we use this Gemma 3 uh [17:54] 12V model uh which is uh like uh [17:58] multimodel open source model that deep [18:00] mind uh like serves. We use this open [18:05] source model to actually achieve uh the [18:08] best open source open data performance [18:10] on a bunch of web agent benchmarks. [18:13] Yeah. So this I think kind of shows why [18:18] we want to like study agent scaling at [18:22] this point. Uh there are so many works [18:24] that have been done for like improving [18:27] the agent itself. However, as this [18:29] reasoning model uh gets more and more [18:32] popular, as scaling gets more and more [18:34] popular, we really need to see like [18:36] okay, what access we can still scale at [18:40] this moment. And um yeah a key [18:44] difference between agentic task and the [18:46] traditional like uh chatbot setting is [18:49] exactly this action part like multi-step [18:52] part and that's why I think scaling the [18:55] number of actions scaling interaction is [18:58] um very like promising direction for [19:02] like future agent work. [19:04] Cool. Yeah, that's it. Thank you guys [19:07] for listening and um happy to take [19:09] questions also. [19:10] [Applause]