Toloka AI Sees Shift to High-Signal Trajectory Data for AI Agents
Ai4
Enterprise adoption of autonomous agents requires shifting human data production from simple labels to complex, multi-day trajectories that push model capabilities.
## Background
Published on YouTube by the Ai4 channel on August 27, 2025, this video features a talk by Albert Chun, Business Development Director at Toloka AI. According to the page description, the presentation explores how the transition from classical ML to autonomous agents is transforming data production through taxonomy development, synthetic data generation blending human expertise with AI, realistic environments, red teaming, and scalable evaluation frameworks.
## TL;DR
Albert Chun explains why top AI research labs and tech giants are investing billions in high-caliber human data to transition from basic LLMs to autonomous agents. He traces the ten-year evolution of data tasks from sub-second image labeling to hours-long, multi-step trajectory collections that capture human inner monologues across computer use, tool use, and coding. Rather than relying on static single-turn responses, effective agent training requires sandboxed evaluation and multi-hop function calling. Ultimately, Chun argues that the AI data strategy must shift from merely seeking high-quality annotations to generating high-signal data through increasingly complex, high-difficulty tasks.
## Atmosphere & style
The presentation is delivered with an authoritative, analytical, and forward-looking tone, blending historical perspective with practical enterprise strategy. Drawing on his past experience as a history teacher and his current role consulting for frontier AI labs, Chun uses clear analogies—such as bicep curl weight progression—to make abstract data engineering concepts engaging and grounded. The narrative creates an energetic, highly professional vibe that balances high-level industry trends with actionable advice for building defensible AI workflows.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=4s) Major tech companies like Meta, OpenAI, and Google are investing heavily in human-annotated data, viewing custom data as the key driver for model differentiation and competitive moats.
- [02:48](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=168s) Toloka AI operates with a dual-tier workforce combining over 10,000 domain experts and a crowd network of more than 200,000 global annotators.
- [03:20](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=200s) Human data tasks have evolved historically from sub-second image labeling to multi-step reasoning, safety guardrails, and days-long agent trajectory evaluations.
- [07:11](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=431s) Technical teams define autonomous agents as multi-hop function calling and tool use, moving beyond simple single-turn chatbots.
- [07:35](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=455s) Closing the accuracy gap between off-the-shelf models (~80%) and enterprise reliability (90–95%+) requires training models on specialized multi-step trajectory data.
- [08:50](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=530s) Capturing human annotators' "inner monologue" during multi-step actions helps models understand the underlying rationale behind complex decisions.
- [12:02](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=722s) The primary high-value use cases for AI agents are Computer Use (UI/desktop interaction), Tool Use (API calls), and end-to-end Coding.
- [17:42](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=1062s) The industry paradigm is shifting from 2024's focus on "high quality data" to 2025's requirement for "high signal data," which tests models on increasingly difficult and complex tasks.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=4s) Introduction & The Billions Flowing into Human Data
Albert Chun, Director of GenAI Data at Toloka AI, opens his presentation by introducing his background in the human data space, where he has been working since 2023. Over that time, he has consulted for leading frontier research labs including OpenAI, Meta, Nvidia, Cohere, and Amazon, and now works with Toloka clients such as Anthropic, Poolside, Microsoft, and Shopify.
He highlights the massive shift happening in AI infrastructure, citing Meta's high-profile $14 billion investment into Scale AI, an incumbent in human data. Chun poses a fundamental question to the audience: why are the smartest and most advanced AI companies putting billions into human data providers? Drawing from conversations earlier that day at the conference with leaders from OpenAI, Google, Amazon, and Apple, he notes that every major player is actively using high-caliber human data to refine model outputs and drive agentic workflows. For founders, CEOs, AI engineers, and product managers alike, custom data is the ultimate driver of model differentiation, moat creation, and state-of-the-art performance.
- **Speaker:** Albert Chun, Director of GenAI Data at Toloka AI.
- **Client & Partner Footprint:** OpenAI, Meta, Nvidia, Cohere, Amazon, Anthropic, Poolside, Microsoft, Shopify, Hugging Face, ServiceNow, MIT, and Carnegie Mellon.
- **Key Industry Move Cited:** Meta's $14B investment in Scale AI.
- **Core Value Proposition of Data:** Provides customization, defensibility, and a competitive moat.
---
[02:48](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=168s) Who is Toloka? Scale, Crowd, and Historical Evolution
Chun introduces Toloka AI, a human data provider that has been operating for a decade and recently raised $72 million in funding backed by Jeff Bezos and the CTO of Shopify. Toloka supports enterprise clients and top research institutions using a two-tiered network of human intelligence: a curated network of over 10,000 domain experts (including RevOps specialists, data analysts, chiefs of staff, and software engineers) and a global crowd of over 200,000 active annotators.
Reflecting on his background as a former history teacher, Chun provides a historical perspective on how human data tasks have evolved over the last ten years:
In the early days of traditional machine learning (ads, search engines, basic computer vision, robotics), data labeling tasks were simple and took fractions of a second to a few seconds—such as clicking on swimming pools on a map or identifying stop signs covered in snow or glare (referencing Ilya Sutskever's famous stop sign rant at Tesla). Today, data production has grown exponentially more complex, spanning chain-of-thought (CoT) reasoning, AI safety and guardrails, voice/audio interfaces (Cartesia, ElevenLabs, Eleven), and synthetic image/video models (Sora, Stability AI, Moonvalley).
- **Funding & Backers:** $72 million recent investment round involving Jeff Bezos and the CTO of Shopify.
- **Human Talent Scale:** 10,000+ specialized domain experts; 200,000+ active global crowd annotators.
- **Historical Shift:** Task durations transformed from fraction-of-a-second clicks to hours or days of deep, multi-step work.
---
[07:11](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=431s) The Shift to Autonomous Agents & Trajectory Collection
Moving into the core topic, Chun unpacks what autonomous agents actually represent under the hood. Recalling a conversation with a founding member of Google Brain, Chun points out that while terms like "agentic workflows" sell well, technical teams understand agents as "multi-hop function calling and tool use." As organizations move beyond simple off-the-shelf chatbots, out-of-the-box accuracy (around 80%) falls short for production enterprise workflows. To achieve 90% or 95%+ reliability, companies must collect high-quality trajectory data.
To train an agent effectively, human annotators do not just evaluate single-turn responses; they execute and label the entire end-to-end trajectory of a complex task (e.g., qualifying a lead in Salesforce across 25 clicks). Crucially, annotators write out their "inner monologue"—explaining *why* each action or decision was made. Chun compares this to educational pedagogy: teaching models the underlying rationale makes their behavior sticker, more accurate, and more reliable when deployed in dynamic environments.
- **Technical Definition:** Autonomous agents = multi-hop function calling + tool use.
- **Accuracy Gap:** Moving from ~80% off-the-shelf baseline accuracy to 90–95%+ enterprise production standards requires specialized RL and trajectory data.
- **Inner Monologue Annotation:** Capturing human rationale at every step of a multi-click trajectory so the agent learns step-by-step logic.
- **Sandboxing:** Building isolated container environments for agents to execute actions (e.g., inside Salesforce) safely and reproducibly during evaluation and red teaming.
---
[12:02](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=722s) Future Horizons: Days-Long Turns and Key Agent Use Cases
Looking ahead, Chun highlights how the scope of data annotation turns is ballooning from seconds to hours, days, and eventually weeks or months. Where early generative demonstrations showed Claude building a simple app like *Flappy Bird* in a single prompt, future agent evaluation will involve multi-program enterprise software development and complex workflows spanning days or longer.
Chun identifies the three most valuable and expensive agent use case categories across the industry:
1. **Computer Use:** Interacting directly with desktop applications or legacy systems lacking APIs—filling out spreadsheets, adjusting system settings, or navigating multi-page web forms.
2. **Tool Use:** Calling APIs across modern tech stacks to automate routine enterprise workflows.
3. **Coding:** End-to-end software development, test generation, and multi-file code editing.
For teams getting started, Chun urges a practical strategy: begin with a tight, single-step tool call or function use case, evaluate the output, and iteratively bridge the gap from 7/10 to 9/10 performance before attempting complex multi-agent architectures.
- **Task Duration Trajectory:** Evolving from single turns to week-long or month-long continuous trajectory generation.
- **Enterprise SaaS Integration Stack:** Salesforce, Zendesk, Google, Slack, Zoom, Notion, Trello, Asana, Snowflake, Stripe, Plaid, PayPal, Shopify, eBay.
- **Multi-Agent Flow Example:** Meeting transcription parser → entity extractor → PII filter → API recommender → execution agent → natural language summary generator.
---
[17:42](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=1062s) From "High Quality" to "High Signal" Data
Chun closes his talk by reframing the conversation around data strategy. While "high quality" and "expert data" were the buzzwords of 2024, the frontier of AI data production in 2025 and beyond belongs to **high-signal data**. High-signal data means presenting models with increasingly difficult, complex problems that push them to the limits of their capabilities.
To illustrate this, Chun uses a weightlifting analogy:
> "If I showed you up here, I was doing a five-pound curl with perfect form, high quality, you would not be impressed. But if I had a 300-pound weight and I did it with perfect form without breaking a sweat, you would... all be impressed. Why? Because high quality isn't everything. It's how difficult is the task that you're doing the high-quality work in."
He advises companies to approach agent development incrementally—"eat the elephant one bite at a time"—by continuously escalating problem difficulty to unlock true ROI and build defensible product value.
- **Industry Shift:** 2024 ("High Quality Data") → 2025+ ("High Signal Data").
- **Analogy:** Perfect form on a 5 lb bicep curl vs. perfect form on a 300 lb curl—the true test is difficulty.
- **Takeaway:** Continual evaluation against harder benchmarks justifies training ROI and builds long-term product defensibility.
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=4s) why are the smartest and most advanced AI companies putting billions into human data providers? — Albert Chun
> [07:11](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=431s) multi-hop function calling and tool use — Albert Chun
> [17:42](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=1062s) If I showed you up here, I was doing a five-pound curl with perfect form, high quality, you would not be impressed. But if I had a 300-pound weight and I did it with perfect form without breaking a sweat, you would... all be impressed. Why? Because high quality isn't everything. It's how difficult is the task that you're doing the high-quality work in. — Albert Chun
## Video description
Albert Chun, Business Development Director, Toloka AI
Abstracts
The shift from classical ML to autonomous agents is fundamentally changing how we create training data. While traditional ML used fixed datasets with human labels, agents need entirely new data approaches supporting variable inputs, outputs, and complex interactions.
This talk explores how agents are revolutionizing data production through taxonomy development, synthetic data generation blending AI with human expertise, and realistic environment creation. Drawing from Toloka's experience with 200k+ contributors and leading AI institutions, we'll show how human-AI collaboration is evolving for agent development.
Key insights include why agents need different evaluation frameworks than traditional models, the critical role of red teaming in agent data pipelines, and practical strategies for scaling high-quality, safety-aligned datasets. We'll share real examples of organizations successfully navigating this transformation as agents become increasingly autonomous.
Albert Chun, Director of GenAI Data at Toloka AI, argues that developing enterprise-grade autonomous agents requires shifting human data production from rapid, simple labeling to complex, high-signal trajectory collection. He contends that bridging the gap between off-the-shelf model performance and enterprise reliability depends on capturing human reasoning across increasingly challenging, multi-step workflows.
Escalating Complexity in Human Data Production (2:48)
Citing major capital movements—including Meta's $14 billion investment in Scale AI and Toloka's $72 million funding round backed by Jeff Bezos and Shopify's CTO—Chun emphasizes that frontier AI companies treat custom human data as their primary source of differentiation and defensibility. Over the past decade, data tasks have transformed from sub-second image labeling, such as categorizing search results or identifying snow-covered stop signs, into deep cognitive engagements involving chain-of-thought reasoning, safety guardrails, and multimodal synthesis. To service these demands, Toloka deploys a two-tiered network comprising over 200,000 crowd annotators alongside 10,000 specialized domain experts.
Closing the Reliability Gap with Trajectory Data (7:11)
Chun defines autonomous agents technically as systems capable of multi-hop function calling and tool use. While off-the-shelf foundational models typically offer around 80% baseline accuracy, enterprise deployment demands 90% to 95% or higher reliability. Achieving this jump requires moving past single-turn evaluations to end-to-end trajectory collection. Human annotators perform complete multi-step tasks in isolated sandbox environments, such as executing a 25-click lead qualification workflow in Salesforce. Crucially, annotators record their 'inner monologue' at each step, teaching the model the underlying rationale and pedagogical logic needed to make decisions in dynamic production environments.
Expanding Horizons Across Computer Use, Tools, and Code (12:02)
Chun identifies three major categories dominating agent development: computer use for navigating desktop interfaces and legacy software without APIs, tool use for orchestrating API calls across enterprise stacks like Slack and Salesforce, and end-to-end coding across multi-file codebases. The duration of data generation turns is expanding accordingly, shifting from single prompts to continuous workflows that span hours, days, and eventually weeks. Rather than immediately building multi-agent architectures, Chun advises engineering teams to start small by optimizing a single-step tool call and iteratively refining its reliability.
Prioritizing High-Signal over Baseline Quality (17:42)
Chun asserts that the frontier of AI data strategy is moving beyond the baseline standard of 'high quality' toward 'high-signal' data. High signal requires confronting models with difficult, complex tasks that push the boundaries of their current capabilities, rather than demonstrating flawless execution on simple tasks. Using a weightlifting analogy, Chun notes that completing a five-pound bicep curl with perfect form is unimpressive, whereas maintaining form on a 300-pound lift demonstrates true capability. For enterprises, continually testing models against escalating difficulty is critical to proving training ROI and securing defensible product value.
Chun concludes that teams must approach agent deployment incrementally by mastering single tool integrations before attempting multi-agent flows. The presentation highlights the growing demand for days-long trajectory collection, though it leaves unresolved how enterprises will manage the steep economic and operational overhead of sustaining specialized domain expert networks for such extended tasks.
精炼阅读摘要
Toloka AI生成式AI数据总监Albert Chun指出，顶尖科技公司正向人类数据提供商投入数十亿美元，因为自主智能体若要从约80%的基础准确率跃升至90%至95%以上的企业级可靠标准，必须依赖捕捉推理逻辑的端到端轨迹数据与高难度“高信号”数据。
资本投向与数据标注的历史演进 (0:04)
Albert Chun指出，Meta向Scale AI投资140亿美元，以及Toloka近期完成由Jeff Bezos与Shopify首席技术官支持的7200万美元融资，均反映出顶尖实验室正将高质量人类数据视为模型差异化和技术护城河的核心驱动力。Toloka目前依托超过1万名专业领域专家和全球20多万活跃标注者提供服务。Chun回顾指出，过去十年间数据标注任务发生了质的转变：从早期传统机器学习中只需花费几分之一秒的简单标注（如地图上点击游泳池或识别被雪覆盖的停车标志），发展到当前涵盖思维链（CoT）推理、模型安全护栏、语音与图像生成，以及需耗费数小时乃至数天的复杂多跳智能体数据生成。
轨迹数据与“内心独白”标注机制 (7:11)
针对自主智能体的技术本质，Chun援引与Google Brain创始成员的交流指出，智能体在技术底座上即为“多跳函数调用与工具使用”（multi-hop function calling and tool use）。目前基础模型开箱即用的准确率通常仅在80%左右，无法满足企业工作流90%甚至95%以上的生产可靠性要求。为弥补这一差距，开发团队必须采集高水准的端到端轨迹数据。在此类标注中，人类专家不仅在隔离的容器沙盒环境中完整执行跨多步操作的任务（例如在Salesforce中经历25次点击完成销售线索核验），还必须同步写下每一步操作的“内心独白”（inner monologue），通过阐释决策原因来教会模型底层逻辑，进而提升其在动态环境下的稳定度。
三大核心场景与渐进式落地路径 (12:02)
Chun归纳了当前行业中最具价值且投入最高的智能体三大应用场景：第一是计算机使用（Computer Use），即在缺乏API的情况下直接接管桌面应用、老旧系统操作或复杂网页表单填写；第二是工具使用（Tool Use），即跨越Salesforce、Slack、Stripe等企业级SaaS技术栈调用API完成自动化任务；第三是软件编码（Coding），覆盖端到端系统开发、测试生成以及多文件代码库修改。针对具体落地，Chun建议企业避免一开始就构建庞杂的多智能体架构，而是应采取渐进策略，先从单一明确的工具调用或函数切入，通过严密评估将性能由7分逐步打磨至9分。
从“高质量”转向“高信号”数据 (17:42)
Chun预测，AI数据竞争的焦点正从2024年的“高质量专家数据”转变为2025年及以后的“高信号数据”（high-signal data）。他以举重作比喻指出，动作规范地完成5磅杠铃弯举并不能证明实力，唯有在承重300磅时依然维持标准动作才能展现真正的能力；同理，高品质只是基础前提，模型真正需要的是能够逼近其能力极限的高难度复杂问题。企业唯有持续用更高难度的基准评估模型，才能支撑模型训练的投资回报率，构建长期的竞争防御壁垒。
Chun强调了端到端轨迹与高难度问题对智能体可靠性的决定性作用，但从单轮交互走向跨越数天乃至数周的多步骤轨迹采集，在真实工业落地中也对沙盒容器环境建设和专业标注成本提出了更高的要求。
Detailed notes — zh
## Background
该视频由行业会议频道 Ai4 于 2025 年 8 月 27 日发布在 YouTube 上。视频记录了 Toloka AI 业务拓展总监 Albert Chun 的演讲，主要介绍从传统机器学习向自主智能体（Agents）转型过程中数据生产范式的深刻变革，涵盖数据分类法演进、人机协作合成数据、沙箱环境构建及红队评估框架等核心议题。
## TL;DR
演讲揭示了前沿科技巨头重金押注人类专家数据的底层逻辑——高度定制且复杂的专有数据是 AI 产品的真正护城河。数据生产模式已从早期秒级的简单 CV 标注，跃迁至当前智能体跨越数小时至数天的长周期轨迹（Trajectory）采集与“内心独白”（Inner Monologue）归因标注。针对电脑操作（Computer Use）、工具调用（Tool Use）和代码编写等落地场景，开发者应在受控沙箱中从小处着手推进多智能体协作。最后，讲者指出行业数据叙事已从单纯的“高质量”演进为追求高推理密度的“高信号数据”（High Signal Data）。
## Atmosphere & style
整场演讲风格鲜明犀利、接地气且极富说服力。讲者巧妙融合了历史教师的叙事视野与顶尖数据服务的一线实战经验，用词干脆直接、充满幽默调侃，善用“300磅哑铃弯举”等形象比喻将抽象的数据工程概念化繁为简。现场节奏紧凑且充满思想碰撞，既有对行业泡沫的冷静审视，又为落地 Agent 数据闭环提供了极具实操性的务实指引。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=4s) 顶尖 AI 机构与前沿实验室之所以重金押注人类数据公司，核心在于开箱即用的基础模型无法构筑壁垒，高度定制化、复杂的专家数据才是 AI 产品的核心护城河。
- [02:48](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=168s) 数据标注正在经历深刻的范式转移：从早期传统计算机视觉的秒级简单标注，演进到大模型的分/小时级推理与代码生成，最终迈向自主智能体时代横跨数天甚至数周的长周期动作序列。
- [02:48](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=168s) 智能体（Agent）在技术本质上等同于“多跳函数调用（Multihop Function Calling）与工具使用（Tool Use）”的工程组合。
- [08:01](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=481s) 基础模型开箱即用带来的 80% 准确率无法满足企业严肃业务需求，在长尾边缘场景下，唯有依赖定制化数据工程将准确率拉升至 95% 以上才能赢得客户信任。
- [08:01](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=481s) 智能体训练要求从单轮对话评测转向全生命周期的“轨迹采集与标注（Trajectory Collection & Annotation）”，系统记录完整的任务执行动作链。
- [08:01](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=481s) 在步骤级标注中引入“内心独白（Inner Monologue）”，要求专家写明每一步操作背后的思考原因（Why），可显著提升模型在泛化场景下的推理稳健性。
- [08:01](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=481s) 企业在落地自主 Agent 时必须构建受控且可复现的沙箱仿真环境（Sandbox），并配合红队测试（Red Teaming）与多维 Eval，确保在生产环境中的行为安全。
- [13:20](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=800s) 电脑操作（Computer Use）、工具调用（Tool Use）与代码编写（Coding）是当前最具商业价值、也是最昂贵的智能体三大落地方向。
- [13:20](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=800s) 电脑操作 Agent（Computer Use）的核心价值在于替代或增强无公开 API 的传统桌面软件与复杂跨系统录入操作。
- [13:20](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=800s) Agent 落地切忌“试图煮沸整座大海（Don't boil the ocean）”，应从小处着手，先针对单一工具或单次函数调用进行快速验证评估，再迭代优化。
- [13:20](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=800s) 复杂业务场景应通过多智能体协同（Multi-Agent Systems）实现分工协作，由不同模型分别承担内容解析、实体提取、PII 脱敏、API 推荐及最终执行。
- [17:45](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=1065s) 数据范式正从“高质量数据”升级为“高信号数据（High Signal Data）”，数据的核心区分度不再仅看动作是否标准，而取决于任务本身的难度、复杂性与推理密度。
- [17:45](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=1065s) 唯有利用具备高难度与高推理密度的“高信号数据”攻克复杂商业难题，才能为客户提供高溢价，实现数据工程与模型研发的高投资回报率（ROI）。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=4s) 为什么顶级 AI 机构对人类数据重金押注？
Albert Chun（Toloka AI 的生成式 AI 数据总监）登台开场。他回顾了自己自 2023 年起在人类数据领域的从业经历，期间曾作为顾问服务过 OpenAI、Meta、NVIDIA、Cohere 和 Amazon 等前沿实验室；如今在 Toloka，他带领团队与 Amazon、Anthropic、Poolside、Microsoft、Shopify 等行业巨头展开深度合作。Albert 抛出了一个直击痛点的问题：Meta 为何要对传统人类数据巨头 Scale AI 进行高达数十亿美元级别的巨额投资？当今全球最顶尖、技术最前沿的公司，为什么要将数百亿美元砸向“人类数据”公司？他们究竟洞察到了什么普通人尚未注意到的趋势？
即使在大会当天，Albert 与来自 OpenAI、Google、Amazon 和 Apple 的高管领袖交流时，所有人无一例外都在探讨如何利用人类数据来优化模型和智能体工作流（Agentic Workflows）。台下的观众涵盖了创始人、CEO、AI 工程师、产品经理、企业客户以及 GTM 人员。Albert 强调，前沿公司之所以不惜成本投入，核心原因在于：**数据能够带来定制化（Customization）、构筑竞争护城河（Moat）与防御性（Defensibility）**。当模型获得经过专家标注、高度复杂且多元的高质量数据时，其输出表现才能达到世界一流的前沿水准（State-of-the-Art）。Albert 幽默地调侃道：“大家可能会想：‘你大老远跑到拉斯维加斯，就是为了跟我们讲这个最显而易见的常识吗？’ 没错，确实如此。”
- **演讲者背景**：Albert Chun（Toloka AI 生成式 AI 数据总监），曾任历史教师。
- **合作客户群**：OpenAI、Meta、NVIDIA、Cohere、Amazon、Anthropic、Poolside、Microsoft、Shopify 等。
- **核心论点**：单纯依靠开箱即用模型无法形成壁垒，高度定制、复杂且由专家标注的高质量数据才是 AI 产品的核心护城河。
---
### [02:48](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=168s) Toloka 的十年演进与数据标注的范式转移
Albert 介绍了 Toloka 的背景。Toloka 已经在数据领域深耕约 10 年，近期获得了由 Jeff Bezos 以及 Shopify CTO 领投的 7200 万美元投资。公司不仅服务顶级企业客户，还与 MIT、卡内基梅隆大学（CMU）等前沿科研机构，以及 Hugging Face、ServiceNow、Poolside 等公司展开技术合作。Toloka 拥有遍布全球的 10,000 名跨职能专家网络（涵盖营收运营 RevOps、数据分析师、幕僚长 Chief of Staff 以及核心技术工程人才），同时维持着一支规模达 200,000 名活跃众包标注者的庞大群体，每天推动模型解决复杂的企业与消费端挑战。
```mermaid
flowchart TD
accTitle: 数据复杂度的演进历程
subgraph S1["早期传统 AI (秒级)"]
A["简单 CV 标注<br/>(圈出水池/识别停顿牌)"]
end
subgraph S2["生成式大模型 (分/小时级)"]
B["思维链推理 (CoT)<br/>复杂代码生成<br/>AI 安全与护栏"]
end
subgraph S3["自主智能体时代 (天/周/月级)"]
C["多跳函数调用<br/>轨迹采集与内心独白<br/>沙箱多轮交互"]
end
S1 --> S2 --> S3
```
结合历史教师的视角，Albert 梳理了人类数据从过去到现在的巨大跨越：
1. **传统 AI 时代（秒级任务）**：最初的数据标注极其简单乏味，单个任务耗时仅几分之一秒或数秒。例如在计算机视觉中圈出卫星地图上的游泳池，或是像 Ilya Sutskever 曾经在特斯拉极力强调的“识别被雪覆盖、模糊或有反光的 Stop Sign（停止标志）”。
2. **生成式 AI 与多模态时代（分钟/小时级任务）**：演进到包含思维链（Chain of Thought）推理的文本交互；像 Cursor、Windsurf、Claude for Code 背后支撑的数百万行复杂代码生成数据；心理咨询等场景下的 AI 安全对齐（Trust & Safety）与护栏（Guardrails）构建；以及在 Cartesia、ElevenLabs 推动下的语音音频标注，和 Sora、Stability AI、Moonvalley 等驱动的视频图像多模态数据。
3. **自主智能体时代（天/周级长周期任务）**：复杂任务已从“几秒钟的一问一答”演变为横跨数小时、24 小时乃至一周的连续动作序列。例如让 Claude 完整开发一个 Flappy Bird 游戏或企业级应用，其背后的复杂轨迹数据需要伴随模型演进被全流程记录。
- **标注颗粒度跃升**：任务耗时从早期传统 AI 的 `< 1 秒` 跃升至当前 Agent 场景下的 `数小时至数周`。
- **Agent 本质定义**：引用 Google Brain 创始成员的说法——“所谓的智能体（Agents），本质上就是多跳函数调用（Multihop Function Calling）与工具使用（Tool Use），只是换了个更具商业吸引力的营销包装。”
---
### [08:01](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=481s) 轨迹采集、内心独白与仿真沙箱评估
谈及企业在落地大模型时的困境，Albert 直言：现在每个人都在调用 ChatGPT 或 Claude 做 PoC（概念验证），但投资者和董事会必然会拷问：“你和隔壁竞品有何区别？你的护城河在哪里？” 基础模型开箱即用（Out of the box）的表现看似不错，但一旦遇到长尾边缘场景（Edge cases），80% 的准确率根本无法让拥有 20 年行业经验的企业客户买单；客户会毫不犹豫地流向准确率达 90%、95% 甚至 97%、能够完美去除敏感隐私信息（PII）且语气精准的定制化方案。
为了让模型在复杂的智能体工作流中落地（例如开完会后自动进入 Salesforce 填写复杂线索字段，或通过繁琐的 25 次点击去调取一份 RevOps 报表），Toloka 采用了全新的数据工程范式：
```mermaid
flowchart LR
accTitle: 智能体轨迹采集与评估流
A["执行完整任务轨迹<br/>(Trajectory)"] --> B["步骤级标注与归因<br/>(内心独白 Inner Monologue)"]
B --> C["受控沙箱容器<br/>(Sandbox Environment)"]
C --> D["红队攻防与安全性评估<br/>(Red Teaming & Eval)"]
```
- **轨迹采集与标注（Trajectory Collection & Annotation）**：不再只评估单轮对话的 Prompt-Response，而是对整个任务执行的全轨迹进行系统级标注。
- **标注内心独白（Inner Monologue）**：在标注每一个操作步骤时，要求专家写明“为什么采取这个动作”的底层思考逻辑。Albert 强调，这正如教学理念一样，向模型传授“背后的原因（Why）”，能让策略在后续泛化中更加稳固、准确与可靠。
- **受控沙箱与测试环境（Sandbox & Test Environment）**：企业不能冒险让未经严密验证的自主 Agent 直接在生产环境的 Salesforce 中运行。因此必须为 Agent 构建可复现、受控的沙箱容器（Sandbox Environment），配合红队测试（Red Teaming）与多维 Eval，确保其交互安全与行为边界。
---
### [13:20](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=800s) 智能体三大核心场景与多 Agent 系统落地
Albert 建议开发者和企业关注最具价值也最昂贵的三大智能体方向：**电脑操作（Computer Use）、工具调用（Tool Use）以及代码编写（Coding）**。
```mermaid
flowchart LR
accTitle: 典型多智能体串联协作流程
subgraph MA["多 Agent 协同流程"]
direction LR
N1["会议纪要解析 Agent"] --> N2["实体提取 Agent"]
N2 --> N3["PII 数据脱敏分类器"]
N3 --> N4["API 推荐与排序 Agent"]
N4 --> N5["执行 API 调用"]
N5 --> N6["自然语言结果反馈"]
end
```
在具体场景的落地策略上，Albert 提出了具体建议：
1. **电脑操作 Agent（Computer Use）**：专门针对**没有公开 API** 的老旧桌面软件、Excel/Google Sheets 录入、跨网站日历调度及系统偏好配置等场景，提供辅助或增强能力。
2. **从小处着手，避免好大喜功**：Albert 极力告诫开发者切忌“试图煮沸整座大海（Don't try to boil the ocean）”。建议先选定**单个用例、单次函数调用（Single Function Call）或单一工具**进行最小可行测试，做快速评估（Vibe Check），观察当前结果是 7 分还是 8 分，再专注于如何通过数据优化将其从 8 分拉升至 9 分或 10 分。
3. **多智能体协同（Multi-Agent Systems）**：在复杂场景下，可通过模型管理模型或同级模型交接。例如一个模型负责解析会议记录，下游模型提取关键实体，分类器过滤 PII 敏感信息，排序模型评估推荐 API，最终由执行 Agent 调用接口并将结果以自然语言反馈用户。
4. **工具调用 Agent（Tool Use）**：针对企业内繁杂的 SaaS 堆栈（Slack、Notion、Salesforce、Snowflake、Stripe、Zendesk 等），训练模型精准更新特定字段，在每一个动作节点明确“什么是优质表现（What good looks like）”，进而沉淀为专有业务壁垒。
---
### [17:45](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=1065s) 数据范式的未来：从“高质量数据”迈向“高信号数据”
在演讲尾声，Albert 纠正了一个行业常见的认知偏差：“如果你去问数据标注行业的从业者，所有人都会告诉你未来属于‘专家数据’或‘高质量数据’。但我告诉你，**高质量数据已经是 2024 年的旧叙事了（That was so 2024），未来的核心是‘高信号数据’（High Signal Data）**。”
Albert 借用健身训练做了一个生动的类比：
> “如果我站在这里，用极其标准的姿势、完美的形式完成了一次 5 磅的哑铃弯举，哪怕质量再高，也没有人会觉得了不起；但如果我举起的是 300 磅的重量，依然保持完美动作且面不改色，所有人都会为之震撼。”
- **高信号数据的核心逻辑**：单纯的高质量并不足以拉开差距，关键在于**任务本身的难度与复杂性**。企业必须用难度递增、具备高推理密度的复杂问题去挑战和训练模型。
- **商业闭环与 ROI 证明**：唯有解决极高难度的问题，才能为企业客户提供更高的商业价值；客户才愿意支付更可观的溢价，从而在财务上真正覆盖模型训练与数据工程的高昂 ROI。
- **结语指引**：“一口一口把大象吃掉（Eat the elephant one bite at a time）”。从易到难逐步提升任务难度与信号密度，这正是整个 AI 智能体行业正在发生的深刻演进。
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=4s) 大家可能会想：‘你大老远跑到拉斯维加斯，就是为了跟我们讲这个最显而易见的常识吗？’ 没错，确实如此。 — Albert Chun
> [02:48](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=168s) 所谓的智能体（Agents），本质上就是多跳函数调用（Multihop Function Calling）与工具使用（Tool Use），只是换了个更具商业吸引力的营销包装。 — Albert Chun
> [08:01](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=481s) 你和隔壁竞品有何区别？你的护城河在哪里？ — Albert Chun
> [13:20](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=800s) 试图煮沸整座大海（Don't try to boil the ocean） — Albert Chun
> [17:45](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=1065s) 高质量数据已经是 2024 年的旧叙事了（That was so 2024），未来的核心是‘高信号数据’（High Signal Data）。 — Albert Chun
> [17:45](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=1065s) 如果我站在这里，用极其标准的姿势、完美的形式完成了一次 5 磅的哑铃弯举，哪怕质量再高，也没有人会觉得了不起；但如果我举起的是 300 磅的重量，依然保持完美动作且面不改色，所有人都会为之震撼。 — Albert Chun
> [17:45](https://www.youtube.com/watch?v=1vpgvrb4z-Q&t=1065s) 一口一口把大象吃掉（Eat the elephant one bite at a time）。 — Albert Chun
## Mentioned
Albert Chun, Toloka AI, OpenAI, Meta, NVIDIA, Cohere, Amazon, Anthropic, Poolside, Microsoft, Shopify, Scale AI, Google, Apple, Jeff Bezos, MIT, 卡内基梅隆大学 (CMU), Hugging Face, ServiceNow, Ilya Sutskever, 特斯拉 (Tesla), Cursor, Windsurf, Claude for Code, Cartesia, ElevenLabs, Sora, Stability AI, Moonvalley, Google Brain, Salesforce, Slack, Notion, Snowflake, Stripe, Zendesk
## Video description
Albert Chun, Business Development Director, Toloka AI
Abstracts
The shift from classical ML to autonomous agents is fundamentally changing how we create training data. While traditional ML used fixed datasets with human labels, agents need entirely new data approaches supporting variable inputs, outputs, and complex interactions.
This talk explores how agents are revolutionizing data production through taxonomy development, synthetic data generation blending AI with human expertise, and realistic environment creation. Drawing from Toloka's experience with 200k+ contributors and leading AI institutions, we'll show how human-AI collaboration is evolving for agent development.
Key insights include why agents need different evaluation frameworks than traditional models, the critical role of red teaming in agent data pipelines, and practical strategies for scaling high-quality, safety-aligned datasets. We'll share real examples of organizations successfully navigating this transformation as agents become increasingly autonomous.
[Source](https://www.youtube.com/watch?v=1vpgvrb4z-Q)
Source quotes
[0:04] Albert Chun: 大家可能会想：‘你大老远跑到拉斯维加斯，就是为了跟我们讲这个最显而易见的常识吗？’ 没错，确实如此。
[2:48] Albert Chun: 所谓的智能体（Agents），本质上就是多跳函数调用（Multihop Function Calling）与工具使用（Tool Use），只是换了个更具商业吸引力的营销包装。
[8:01] Albert Chun: 你和隔壁竞品有何区别？你的护城河在哪里？
[13:20] Albert Chun: 试图煮沸整座大海（Don't try to boil the ocean）
[17:45] Albert Chun: 高质量数据已经是 2024 年的旧叙事了（That was so 2024），未来的核心是‘高信号数据’（High Signal Data）。
[17:45] Albert Chun: 如果我站在这里，用极其标准的姿势、完美的形式完成了一次 5 磅的哑铃弯举，哪怕质量再高，也没有人会觉得了不起；但如果我举起的是 300 磅的重量，依然保持完美动作且面不改色，所有人都会为之震撼。
[17:45] Albert Chun: 一口一口把大象吃掉（Eat the elephant one bite at a time）。
Source quotes
[0:04] Albert Chun: why are the smartest and most advanced AI companies putting billions into human data providers?
[7:11] Albert Chun: multi-hop function calling and tool use
[17:42] Albert Chun: If I showed you up here, I was doing a five-pound curl with perfect form, high quality, you would not be impressed. But if I had a 300-pound weight and I did it with perfect form without breaking a sweat, you would... all be impressed. Why? Because high quality isn't everything. It's how difficult is the task that you're doing the high-quality work in.
Original transcript
[0:04] So, my name is Albert Chun. I'm the
[0:06] director of Genai data over at Toka.
[0:08] I've been working in human data since
[0:10] 2023. Um, I got a chance to partner with
[0:14] uh multiple frontier labs um on a
[0:17] consulting basis uh from OpenAI, Meta,
[0:21] Nvidia, Coher um and Amazon. Um, and now
[0:25] I get to work uh at Toka with clients
[0:28] like Amazon, Anthropic, Poolside,
[0:31] Microsoft, Shopify, and more. Um, and
[0:34] why is this important? So, I think most
[0:39] of you here probably heard the news
[0:41] about Meta's 14 billion investment into
[0:45] scale AI, one of the incumbents in the
[0:48] human data space. And the question is
[0:51] why would some of the smartest
[0:54] uh why would one of the smartest most
[0:55] frontier companies in the world invest
[0:58] in a company billions of dollars into a
[1:01] a human data company? What do they know
[1:04] that the rest of us don't?
[1:07] Even just today because we have such
[1:09] high quality guests here, I got to speak
[1:11] from leaders at OpenAI, Google, Amazon,
[1:15] and Apple. And they were all talking
[1:18] about how they were using human data to
[1:22] improve their models and their agentic
[1:24] workflows.
[1:26] So I know because some of you are here,
[1:30] I imagine some of you know the
[1:32] importance of uh that caliber of data
[1:35] that improves model outputs and maybe
[1:38] some of you don't and that's okay and
[1:40] that's why you're here. Um
[1:44] one of the things um the reasons to
[1:47] really pay attention to this. So I
[1:49] imagine in a room this size we have
[1:52] founders, CEOs, we have AI engineers,
[1:55] product managers, maybe you're an
[1:57] enterprise client, maybe you're in maybe
[1:59] you're in go to market or what have you.
[2:01] Um but some of the things to understand
[2:03] about data, the reason why so many of
[2:05] these companies are investing hundreds
[2:08] of billions of dollars into the space is
[2:11] because data provides customization.
[2:14] data provides a moat and defensibility
[2:18] and it provides when you get enough
[2:22] diverse data that is high quality,
[2:25] highly complex and it's expertly
[2:28] annotated.
[2:30] What do you think would happen to the
[2:31] model outputs? It'd be world class, be
[2:34] state-of-the-art. It'd be frontier.
[2:36] You're saying, "Hey, did you come all
[2:38] the way to Las Vegas to tell us the most
[2:40] common sense thing?" Yeah, kind of. And
[2:43] so this is what uh we'll jump into here.
[2:46] So
[2:48] who are we Toka? So in case you haven't
[2:52] heard of us, we've been around for about
[2:53] 10 years. We just uh got a recent
[2:56] investment from Jeff Bezos and the CTO
[2:59] of Shopify uh to a tune of $72 million.
[3:02] Uh we work with enterprise clients and
[3:06] Frontier Research Labs at MIT and
[3:08] Carnegie Melon. some of the uh clients I
[3:10] had mentioned previously including
[3:12] hugging face service now Microsoft
[3:15] poolside etc
[3:17] we have
[3:19] to the tune of 10,000 experts from
[3:21] around the world in different functions
[3:24] like the ones that are in your company
[3:27] quite frankly the revops the data
[3:30] analysts the chiefs of staff the
[3:32] engineering the technical talent we also
[3:36] have a crowd that is the size of most
[3:39] Midwestern towns, 200,000 active um
[3:44] crowd annotators who are
[3:47] every single day moving models forward
[3:50] and solving more and more complex
[3:53] enterprise challenges and consumer
[3:55] problems.
[3:59] So over the last 10 years when we've
[4:01] been working here, we started off in
[4:03] traditional AI working in ads and search
[4:06] engines, robotics, voice, etc.
[4:10] And
[4:12] some of the work, as you can imagine, in
[4:14] that data labeling space, candidly, the
[4:17] complexity was so simple. The tasks, the
[4:20] turns, they were fractions of a second,
[4:22] sometimes seconds long, never over a
[4:25] minute long. And we're going to get into
[4:27] just what some of that looks like
[4:30] today.
[4:33] And so just want to share a little bit
[4:35] in case just a little bit of primary. I
[4:37] used to be a history teacher. So this is
[4:39] I like stuff like this. Um I want to
[4:42] talk a little bit about where we were
[4:44] where we are and where we will be in
[4:46] human data. So initially we were at just
[4:50] very blas data just labeling. You know,
[4:53] you may have heard of computer vision
[4:54] tasks like bringing up a map and
[4:57] identify the pool. Or maybe you heard of
[5:00] uh uh I think it was Ilia's epic stop
[5:04] sign rant um about identifying stop
[5:07] signs and how difficult that could be
[5:09] when he was at Tesla and that the stop
[5:11] sign covered in snow? Is it half blurry?
[5:14] Is there glare? There's a lot of there's
[5:16] a lot of interesting use cases that help
[5:18] build up the foundations of some of the
[5:20] greatest technologies that we have
[5:21] today. But it's gotten much more complex
[5:24] since then. Um, it's gone into text and
[5:27] reasoning as we all know how every
[5:29] single turn that we now put almost
[5:32] embeds chain of felt reasoning as an
[5:33] example. We've seen it in creative work
[5:37] in coding. When you think about cloud
[5:39] for code, when you think about cursor,
[5:42] wind surf and all of these different
[5:44] applications, we get to see the end
[5:46] result of the hundreds of thousands of
[5:49] millions of rows of coding data that
[5:51] candidly people like Teloka provides
[5:54] these foundation model companies.
[5:58] >> We'll see them in AI safety, trust and
[6:02] safety like we heard uh previously
[6:04] around the therapy use case. My
[6:06] goodness, there there are so many use
[6:09] cases there. I remember we partnered
[6:11] with one of the state-of-the-art AI
[6:14] safety labs over the last couple of
[6:16] years in building out really robust
[6:18] guard rails for AI because of just the
[6:21] amount of responsibility that's required
[6:24] there when we have so much power to
[6:25] leverage. We see it in voice especially.
[6:28] I know that everyone here, we've seen it
[6:31] on the agenda about how audio is taking
[6:35] over, right? We're seeing it take over
[6:37] IVRs, call centers, customer support,
[6:40] um, go to market motions, etc. In the
[6:43] likes of Cartisian, Levvenl Labs,
[6:45] Element, you name it. We certainly work
[6:48] with all those modalities or in images
[6:51] like with uh entrance like Moon Valley
[6:56] incumbents like stability AI or Sora or
[6:59] what have you. Again, you're talking
[7:01] about
[7:04] billions of dollars worth of data to get
[7:06] it to the level of refinement that we
[7:08] are at now.
[7:11] And one of the pieces that we see here
[7:13] is in autonomous agents. It's funny. I
[7:17] was working with uh one of the um uh
[7:22] founding members of Google Brain on this
[7:24] and he was like Albert agents is what it
[7:27] really should be called is multihop
[7:29] function calling and tool use but that
[7:32] doesn't sell as well as agentic
[7:34] workflows or agents. Um but essentially
[7:37] like what does it look like to deploy
[7:40] these agents
[7:42] in potentially anywhere the spectrum of
[7:46] uh lowrisisk use cases like customer
[7:48] support and high-risisk cases like
[7:51] financial data or healthcare in very
[7:54] highly regulated industries.
[7:57] And so to share a little bit here so
[8:01] what does this look like? Um, right now
[8:05] everyone here has used chatbt or claude
[8:10] or what have you. That's obvious. And
[8:12] maybe some of you even have tested it
[8:14] out, done a pilot, a PC, maybe you've
[8:16] put it into a workflow or so. Um, but
[8:20] the question that I always ask people
[8:23] and candidly board members will ask you,
[8:26] investors will ask you is what makes it
[8:28] different from the next guy? Where's the
[8:31] defensibility and where's your moat?
[8:34] What's the differentiator?
[8:36] And candidly, some of these models, they
[8:38] have gotten so good
[8:40] out of the box. Some people it's
[8:42] indistinguishable, right? Seemingly
[8:45] indistinguishable until you start
[8:47] getting into edge cases until you talk
[8:48] to someone with real insight knowledge.
[8:50] You have customers who's been in the
[8:52] space for 20 years and you're saying,
[8:53] "Guys, 80% accuracy ain't going to cut
[8:56] it."
[8:58] and they're going to migrate over to the
[8:59] customer to the provider who's getting
[9:02] 85% accuracy or 90 or 95 or 97%. Or they
[9:07] get in the right tone, they have
[9:09] filtered out the PII data. They've
[9:11] they've made it different from out of
[9:13] the box. And again, we all know if you
[9:15] get something out of the box, you're
[9:17] just like your competitors.
[9:20] And so this is the weight and the
[9:22] reasons uh why we have uh so many
[9:26] different use cases in uh reinforcement
[9:28] learning.
[9:30] The other part here is uh in trajectory
[9:33] collection and annotation. So this is
[9:36] where we start to get into these agentic
[9:38] workflows. So, for example,
[9:41] yeah, like we all ask Claude to write a
[9:45] nicer email than the one we're about to
[9:47] send,
[9:49] but we also want a model to
[9:53] go into Salesforce after a meeting and
[9:57] fill out all those annoying fills to
[10:00] fields to qualify a lead, as an example.
[10:03] Or that one annoying report that your
[10:04] supervisor keeps asking for yesterday,
[10:07] like you need to pull up that report. it
[10:09] takes 25 clicks. you need to call
[10:10] someone from, you know, RevOps team to
[10:12] say, "Hey, how do I do this again?"
[10:14] Like, you want them to solve that,
[10:15] right? And so, one of the things we
[10:18] would do is, well, why not
[10:22] if instead of tracking and getting datas
[10:25] per turn and saying what good looks
[10:27] like, why not say that for an entire
[10:29] trajectory of a task?
[10:32] And then why not after you go and label
[10:35] the steps and say, "Hey, this is how you
[10:37] actually do it and I'm going to get the
[10:39] model to do it." Why not annotate it and
[10:42] imagine sharing your inner monologue,
[10:44] the reasons why you do each step so that
[10:47] they actually do it for the right
[10:48] reasons next time.
[10:51] You're starting to see where my
[10:53] background in education really helps in
[10:55] training models. It's just, you know,
[10:56] the when you share the why about
[10:58] something, it gets a bit more stickier.
[11:00] It gets a bit more accurate. it gets a
[11:02] bit more helpful, it gets better
[11:04] outcomes. And then some of the other
[11:06] things that we do are uh is evaluations
[11:10] and red teaming.
[11:12] So an example here is defining what good
[11:15] looks like, defining what these
[11:17] different pieces are. Um red teaming as
[11:19] we heard um and I'd say the the latest
[11:23] piece is around context and environment.
[11:27] We all know that as these agents get
[11:30] deployed as an example, how do you know
[11:32] how it's going to engage in your
[11:35] customer's environment
[11:38] or in your end users's environment?
[11:40] Not quite sure, right? So, if you're
[11:42] going to set off an autonomous agent in
[11:45] Salesforce, better make sure that thing
[11:47] doesn't mess everything up. And so we
[11:50] would create a test environment for
[11:52] agents to almost a bit of a sandbox, a
[11:55] container in order for us to see the
[11:58] results in a more controlled setting and
[12:00] a more reproducible setting.
[12:02] I'm going to jump along here. The place
[12:05] that we see this going is, as I
[12:06] mentioned earlier, some of our turns
[12:08] were about seconds long. It's become
[12:10] minutes long. I know personally some of
[12:12] our the turns that and some of the uh
[12:15] flows that I've worked through have gone
[12:16] into hours, eight hours, 24 hours, week
[12:20] long. Think about the use case here.
[12:24] Hey, we've all heard what was it a few
[12:26] months ago, hey Claude, create Flappy
[12:28] Bird or whatever, whatever the app is.
[12:30] That required someone to create that
[12:33] level of app per turn. And again to the
[12:36] tunes of hundreds of thousands of
[12:38] examples of creating all these different
[12:40] kinds of games and applications, it
[12:43] starts to take quite a bit. And we
[12:45] actually imagine a world where these
[12:48] turns could take days, weeks, months,
[12:51] even a year. If you could consider an
[12:55] enterprise a a a product that has
[12:58] enterprise polish or imagine um
[13:02] multi-program uh applications
[13:05] it the amount of data complexity and
[13:08] challenges that are required um we would
[13:11] be tracking that right alongside them.
[13:15] I'm going to go ahead and jump ahead to
[13:17] a couple of slides here.
[13:20] We've covered some of these things and
[13:22] let's get to the fun stuff. So I have an
[13:26] eye for time. So
[13:30] uh there are three major use cases to
[13:32] consider um for agent use cases. It
[13:35] would be computer use, tool use and
[13:38] coding. It's most expensive problems out
[13:41] there. Snap away please. Yes.
[13:45] So let's think about computer using
[13:47] agents first. So imagine this. Anything
[13:51] without an API,
[13:53] you would use something for a computer
[13:56] use agent. So imagine that maybe it's an
[13:59] archaic software that you have that's
[14:00] only on the desktop and that you would
[14:02] want an agent to use your computer with.
[14:05] Or maybe it's to input fields into
[14:08] Google Sheets or Excel as an example. Or
[14:12] maybe it's to schedule an event in your
[14:13] calendar. Uh navigate a website. Maybe
[14:16] it's to change some settings on your
[14:17] computer, dim the light, whatever it
[14:19] might be. And then imagine assistive use
[14:22] cases or augmented use cases as
[14:25] different customer profiles, right? Like
[14:27] the the world starts to really open up
[14:30] beyond the uh hoham chatbot use case
[14:33] that we have
[14:36] been used to. And so
[14:39] right now, less on the words and more
[14:41] less on the text and more on the words
[14:43] here. Um there are I I I feel like I'm
[14:48] in a group full of people who are either
[14:49] already doing this or on the precipice
[14:51] of doing this. And so one one thing I
[14:53] would actually offer is to consider one
[14:56] use case for you to deploy an agent in
[15:00] single use case, one function call, one
[15:02] tool use. Don't get too complicated.
[15:05] Don't try to solve the whole don't try
[15:07] to boil the ocean. Just try one and then
[15:10] see what it looks like at the action
[15:12] level. do a quick eval, give a vibe
[15:14] check. What was the result? Was it 100
[15:15] out of was it 10 out of 10? My guess is
[15:17] no. But how do you get from the seven
[15:20] out of 10 to 8 out of 10? The 8 to nine.
[15:23] And that's where really things start to
[15:25] get really interesting.
[15:30] And then of course you could always get
[15:31] into multi- aent systems where you have
[15:34] different we heard it a little bit
[15:35] earlier about a model managing another
[15:38] model. You may have peer models doing
[15:40] hand uh function call handoffs or here
[15:42] I'll give you a quick example. Maybe you
[15:44] have a model that parses your meeting
[15:46] transcriptions.
[15:48] And then maybe you have that meeting
[15:50] transcription model start to parse out
[15:52] all the entities that are important. Or
[15:54] maybe you have another classifier model
[15:55] that scrubs out filters out all the PII
[15:57] data. And then maybe that hands off to
[15:59] another model that does executes uh gets
[16:03] a rinker and recommends different APIs
[16:05] to call on. And maybe after that you get
[16:08] the actual execution step in that API
[16:12] and then maybe it goes back to a natural
[16:14] query to say hey I did this right it's
[16:20] I know it sounds simple it's I know it's
[16:22] harder to actually do but it's candidly
[16:25] not as bad as we all think.
[16:28] So let's move on here to
[16:33] again I for time no one's heling me
[16:36] down. Okay, there we go. All right, so
[16:38] uh tool use agents. So, um a piece here
[16:42] that I would consider is imagine the
[16:44] tools here. I was just talking to some
[16:46] business owners today, some people in
[16:47] private equity. They're saying, "Hey,
[16:49] Albert, like how do I reduce costs
[16:51] across my port code as an example, my
[16:53] use case?" And so, think about all the
[16:55] APIs that are out there. Think about
[16:56] your own tech stack. Google, Slack,
[16:59] Zoom, Notion, Trello, Asauna, Snowflake,
[17:02] Stripe, Plaid, PayPal, what have you,
[17:04] Shopify, eBay, Salesforce, Zenesk. Did I
[17:07] cause any stress yet? Right? Like
[17:10] imagine if you had a tool using agent to
[17:14] do again just one workflow. Update this
[17:17] one field, try it out, test it out, and
[17:20] see where it goes. And I will just share
[17:22] again at each of those steps to show
[17:25] what the model what good looks like. The
[17:28] most important thing you have to train
[17:31] the model to do plus whatever comes out
[17:34] of the box to get to the defensibility
[17:38] portion of your value proposition.
[17:42] Um,
[17:45] that time went by much faster than I
[17:47] thought. I imagine that that stoked a a
[17:51] great deal of questions. Um, but where I
[17:54] where we do see the world going in the
[17:56] future, everyone ask anyone in human
[17:59] data, you know what everyone is going to
[18:00] tell you? They're going to say it's all
[18:02] about expert data or high quality data.
[18:06] I'll tell you right now that was so
[18:07] 2024.
[18:09] Where we're going at is in high signal
[18:12] data. And what that means is you need to
[18:15] challenge the model with increasingly
[18:17] difficult problems to solve because why?
[18:20] Then you provide increasing value to
[18:22] your client because why? Then they pay
[18:24] increasing money for the pro problem you
[18:27] solved that then justifies all the ROI
[18:30] for the training and all of this all of
[18:32] this work that you've been doing, right?
[18:34] And so in other words, let me provide an
[18:36] analogy. If I showed you up here, I was
[18:40] doing a five pound curl with perfect
[18:42] form, high quality, you would not be
[18:45] impressed. But if I had a 300lb weight
[18:48] and I did it with perfect form without
[18:50] breaking a sweat, you would I that's a
[18:53] dream. Yeah. That you would all be
[18:55] impressed. Why? Because high quality
[18:57] isn't everything. It's how difficult is
[19:00] a task that you're doing the high
[19:02] quality work in. And so everything
[19:05] starts with steps. Start with solving
[19:07] the easier problems.
[19:09] Eat the elephant one bite at a time. And
[19:12] this is where the entire industry starts
[19:15] to innovate and evolve. Thank you so
[19:17] much.
