## Background
Published on YouTube by Ai4 on August 28, 2025, this presentation features Shelby Heinecke, Senior AI Research Manager at Salesforce. The session covers how Salesforce's latest research—spanning small action models like xLAM to the MCPEval benchmarking framework—is advancing AI agents toward real-world readiness across modalities with faster decisions, smarter actions, and seamless evaluation.
## TL;DR
Shelby Heinecke outlines Salesforce's strategy for evolving autonomous AI agents into fast, multimodal, and enterprise-ready systems. She showcases lightweight small action models such as xLAM-1B ('Tiny Giant') that match much larger models in tool-calling while enabling low-latency, privacy-compliant edge deployment. The talk also introduces TACO, a multimodal action framework that resolves visual reasoning tasks via specialized tool calls, alongside MCPEval, an automated synthetic benchmarking suite for agents using the Model Context Protocol (MCP). Heinecke emphasizes Salesforce's commitment to open-source collaboration by providing models, datasets, and evaluation tools directly to the research and developer community.
## Atmosphere & style
The presentation carries a polished, authoritative keynote tone that seamlessly connects theoretical research with practical enterprise deployment. Heinecke delivers her insights with clarity and momentum, using concrete failure cases and architectural diagrams to demystify complex agent workflows. It feels informative, forward-looking, and developer-friendly, reinforced by a genuine enthusiasm for open-source contributions.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=4s) The next evolution of AI agents focuses on lower latency, multimodal reasoning, and enterprise-grade reliability over compute-heavy, text-only systems.
- [00:04](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=4s) Salesforce Research provides open-source checkpoints, code, and synthetic datasets on Hugging Face to accelerate community agent development.
- [01:31](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=91s) Salesforce's research pipeline bridges fundamental innovation and enterprise deployment, directly powering the Atlas reasoning engine inside Agentforce.
- [01:31](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=91s) Research is extending into on-device AI to run capable language and action models locally on smartphones and edge hardware.
- [04:13](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=253s) Small Language Models (SLMs) address compute bottlenecks, reduce latency, and ensure strict enterprise data privacy by running on local infrastructure.
- [04:13](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=253s) Salesforce's xLAM-1B ('Tiny Giant') demonstrates that specialized small action models can outperform general models ten times their size on tool-calling benchmarks.
- [10:14](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=614s) Frontier vision-language models often fail complex visual reasoning when relying only on internal weights rather than dynamic tool execution.
- [10:14](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=614s) TACO leverages over 270,000 synthetic chain-of-thought and action trajectories to enable multimodal agents to use specialized vision and code tools for visual problem-solving.
- [13:54](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=834s) Standardizing tool interfaces with Model Context Protocol (MCP) requires dedicated evaluation frameworks to verify an agent can reliably coordinate complex multi-tool workflows.
- [13:54](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=834s) MCPEval automatically generates high-coverage synthetic test scenarios across selected MCP servers to benchmark and diagnose agent failure modes before production release.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=4s) The Next Phase of AI Agents: Faster, Multimodal, and Battle-Tested
Opening the third day of the conference, Shelby Heinecke reflects on how the AI landscape has evolved, observing that while the previous years belonged to the rise of large language models, "this year is the year of AI agents." However, she argues that the industry has only begun scratching the surface of what autonomous systems can achieve. Salesforce Research is laying the groundwork for a new generation of agentic AI designed to overcome the friction points of current deployments—specifically addressing high latency, rigid text-only interfaces, and questionable real-world reliability.
The next evolutionary leap focuses on three core pillars: making agents substantially faster by moving away from compute-heavy architectures, expanding their perceptual reasoning across modalities like video and audio, and ensuring they are thoroughly battle-tested for enterprise environments. Emphasizing open collaboration, Heinecke notes that every innovation, model checkpoint, and dataset discussed is fully open-sourced on Hugging Face with accompanying QR codes, allowing practitioners to immediately test and integrate the breakthroughs into their own systems.
- **Speed and Efficiency:** Reducing agent response times through lightweight action-specialized architectures.
- **Multimodal Capabilities:** Equipping agents to navigate and reason over images, audio, and video alongside text.
- **Enterprise-Grade Evaluation:** Validating multi-step execution rigor before customer-facing deployment.
- **Open-Source Delivery:** Releasing code, weights, and synthetic trajectories directly to the community.
---
[01:31](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=91s) Research at Salesforce: Innovating from Atlas to On-Device Models
Salesforce Research operates as a dedicated innovation powerhouse with over a decade of history, having been founded back in 2014. Heinecke leads a team focused on anticipating customer needs three to nine months ahead of the product cycle. Their operating model bridges foundational science and practical applications: publishing academic papers and releasing open-source prototypes that ultimately graduate into enterprise products.
```mermaid
flowchart LR
accTitle: Salesforce Research to Production Pipeline
subgraph Research["Salesforce AI Research"]
R1["10+ Years R&D<br/>(Founded 2014)"] --> R2["Academic Papers &<br/>Open Source Prototypes"]
end
subgraph Frontiers["Emerging Directions"]
F1["Small Action Models (SLMs)"]
F2["On-Device AI (Mobile/Edge)"]
F3["Multimodal Agents"]
end
subgraph Production["Enterprise Deployment"]
P1["Agentforce"]
P2["Atlas Reasoning Engine"]
end
R2 --> Frontiers
Frontiers --> Production
```
Heinecke highlights how foundational research transitions into flagship features, citing the Atlas reasoning engine—the core decision-making brain of Salesforce’s Agentforce platform—which was developed directly within her research organization. Looking past 2025 into 2026, her team is expanding beyond traditional cloud architectures into on-device AI, exploring how language and action models can run directly on consumer hardware like smartphones without sacrificing reasoning capability.
---
[04:13](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=253s) Small Action Models and the "Tiny Giant" (xLAM)
Highlighting a widely circulated NVIDIA research paper titled *"Small Language Models Are the Future of Agentic AI"*, Heinecke agrees wholeheartedly with the premise that smaller models represent the future of autonomous systems. Deploying massive frontier models for every sub-step of an agentic workflow consumes excessive GPU compute and introduces latency bottlenecks. Small language models (SLMs) feature significantly fewer parameters, translating to fewer mathematical operations, faster inference, reduced deployment costs, and the versatility to run on local laptops, on-prem servers, phones, smartwatches, or drones. Furthermore, local deployment provides strict data privacy and governance, ensuring sensitive user information never leaves enterprise boundaries.
| Model Category | Example Models | Parameter Count | Primary Use Case & Trade-offs |
|:--- |:--- |:--- |:--- |
| **Traditional Frontier LLMs** | DeepSeek-V3, Mistral Large | 123B – 671B+ parameters | General reasoning; high GPU compute, significant latency, costly cloud hosting |
| **Salesforce Small Action Models** | xLAM-1B ("Tiny Giant"), xLAM-3B, xLAM-8B | 1B – 8B parameters | Specialized tool-calling & action execution; ultra-low latency, edge/local deployment, beats models 10× its size |
To make small models viable for autonomous execution, Salesforce developed Action Models—networks explicitly optimized to translate natural language intents into structured API calls and executable commands. Heinecke introduces Salesforce's flagship "Tiny Giant", the **xLAM-1B** model, alongside 3B and 8B counterparts. Despite its 1-billion-parameter footprint, xLAM matches or outperforms models more than ten times its size on agentic tool-calling benchmarks. Heinecke emphasizes that the secret behind this outsized performance lies in the meticulously curated synthetic training datasets, which Salesforce has also made fully open source.
---
[10:14](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=614s) TACO: Multimodal Action Models with Chain-of-Thought and Action
Current frontier multimodal models frequently fail at seemingly elementary visual reasoning tasks when forced to rely purely on internal weights. Heinecke presents a concrete failure case: showing GPT-4o an image of a motorcycle race and asking for the specific number on the vehicle ridden by the motorcyclist in the yellow suit, which the model misidentifies. Rather than forcing a vision-language model to resolve visual ambiguities from memory, Salesforce empowers models to use specialized tools to inspect, crop, and compute visual data dynamically.
This methodology is realized in **TACO**, a multimodal action model named as an anagram for its reasoning framework: **T**hought and **A**ction (Chain-of-Thought and Action). To train TACO, the research team generated over 270,000 synthetic image-text trajectories. When given a complex visual query—such as counting the exact number of mascots in an image—TACO breaks the objective down into a sequence of tool calls, invoking a computer vision model to detect objects and executing a deterministic Python function to tally them, achieving robust visual accuracy.
- **Structured Trajectories:** Over 270,000 synthetic reasoning-and-action trajectories created for multimodal fine-tuning.
- **Tool-Augmented Vision:** Offloads specialized visual sub-tasks to dedicated CV models and Python execution environments rather than raw text generation.
- **Open Availability:** TACO model weights and datasets are released under open-source licenses.
---
[13:54](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=834s) MCP Eval: Benchmarking Enterprise Agents on the Model Context Protocol
Transitioning an agent from an impressive demo to a production-ready enterprise asset requires rigorous, automated benchmarking. The rapid adoption of Anthropic’s Model Context Protocol (MCP) has made it seamless for AI agents to connect with diverse external services and software tools, ranging from Slack to YouTube and Instagram. However, Heinecke notes that simply providing an agent with a large toolbox does not mean it knows how to use those tools correctly: "It's like if I gave you a toolbox right now, do you know how to use those tools necessarily? Maybe, maybe not."
```mermaid
flowchart LR
accTitle: MCPEval Workflow
subgraph Step1["Step 1: Synthetic Dataset Generation"]
A["Select MCP Servers<br/>(e.g., Slack, GitHub, CRM)"] --> B["MCPEval Engine"]
B --> C["Automated Synthetic Test Trajectories"]
end
subgraph Step2["Step 2: Execution & Reporting"]
C --> D["Run Agent Against MCP Servers"]
D --> E["Comprehensive Evaluation Report"]
end
```
To solve the bottleneck of manually writing comprehensive test cases for complex multi-tool interactions, Salesforce built **MCPEval**. The open-source evaluation framework operates in two streamlined phases:
1. **Automated Test Generation:** The user selects the target MCP servers, and MCPEval synthetically generates realistic, high-coverage evaluation scenarios and expected execution paths.
2. **Execution & Diagnostic Reporting:** The agent is evaluated against the generated test suite, producing granular diagnostic reports that reveal failure modes, tool-calling errors, and compliance gaps.
This framework allows enterprise teams to systematically refine agent policies and guarantee operational readiness before releasing autonomous systems into production environments.
[16:51](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=1011s) Open-Source Tool Releases and Closing Remarks
Shelby concludes the presentation by highlighting the ongoing release of their evaluation benchmarks and frameworks. She invites developers, practitioners, and researchers to explore these open-source tools to benchmark and refine their own AI agents, ensuring they are robust and fully equipped for real-world readiness in the next phase of agent development.
She closes her talk by expressing gratitude to the audience: "Thank you all so much."
- Salesforce is actively rolling out these evaluation frameworks and open-source tooling to help builders improve agent capabilities.
- Developers are encouraged to leverage these tools to prepare their agentic systems for complex, real-world deployment.
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=4s) this year is the year of AI agents. — Shelby Heinecke
> [04:13](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=253s) Small Language Models Are the Future of Agentic AI — Shelby Heinecke
> [13:54](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=834s) It's like if I gave you a toolbox right now, do you know how to use those tools necessarily? Maybe, maybe not. — Shelby Heinecke
> [16:51](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=1011s) Thank you all so much. — Shelby Heinecke
## Video description
Shelby Heinecke, Senior AI Research Manager, Salesforce
Abstracts
AI agents are shifting from simple responders to systems that can reason, plan, and act, across modalities. I’ll share how our research at Salesforce, from xLAM to MCPEval, is pushing agents toward real-world readiness: faster decisions, smarter actions, and seamless evaluation. If you're building the next generation of agents, this is what’s coming next.
Salesforce Research argues that the next phase of agentic AI requires shifting away from compute-heavy frontier models toward lightweight action models, tool-augmented multimodal reasoning, and automated evaluation frameworks.
Small Action Models for Latency and Deployment (4:13)
Deploying frontier language models with hundreds of billions of parameters across every step of an agentic workflow creates significant GPU compute demands and latency bottlenecks. Shelby Heinecke of Salesforce Research endorses the view that small language models represent the future of autonomous systems, pointing out that smaller parameter footprints reduce mathematical operations, lower operational costs, and allow execution on local devices such as laptops, smartphones, and edge hardware, while preserving enterprise data governance.
xLAM Benchmarks and Synthetic Training Data (4:13)
To adapt compact architectures for tool execution, Salesforce created its xLAM family of action models, spanning 1B, 3B, and 8B parameters. According to Heinecke, the 1-billion-parameter xLAM-1B—dubbed the 'Tiny Giant'—matches or outperforms models more than ten times its size on tool-calling benchmarks by focusing specifically on converting natural language into structured API calls. Salesforce attributes this performance to carefully curated synthetic training datasets, which have been released open-source alongside model weights.
Multimodal Tool Calling with TACO (10:14)
Current frontier vision-language models often struggle with fine-grained visual tasks when relying solely on internal weights, as seen when GPT-4o misidentifies details like motorcycle race numbers. To overcome this limitation, Salesforce developed TACO (Chain-of-Thought and Action), a framework that equips agents with external programmatic tools to crop, inspect, and calculate visual inputs dynamically.
Automated Agent Benchmarking via MCPEval (13:54)
While Anthropic’s Model Context Protocol (MCP) standardizes integrations between agents and external tools, access to tools does not ensure correct execution. To replace manual test creation, Salesforce introduced MCPEval, an open-source evaluation suite. The system first generates synthetic test scenarios across targeted MCP servers such as Slack or GitHub, then runs the agent through these paths to produce diagnostic reports detailing tool-calling failures and compliance gaps prior to production deployment.
Salesforce Research is directing its pipeline toward 2026 by expanding into on-device models that run directly on consumer hardware. To accelerate broader enterprise adoption, the team has open-sourced its model checkpoints, synthetic datasets, and evaluation harnesses on Hugging Face.
## Background
本视频由 YouTube 频道 Ai4 于 2025 年 8 月 28 日发布。视频内容为 Salesforce 高级 AI 研究经理 Shelby Heinecke 的主题演讲，主要探讨了 AI Agent 从简单响应系统向跨模态推理、规划与行动系统的演进，并介绍了 Salesforce 在 xLAM、TACO 及 MCPEval 等推动智能体走向企业级落地的前沿开源研究成果。
## TL;DR
Shelby Heinecke 系统阐述了下一代企业级 AI Agent 的三大核心演进方向：极致响应速度、跨模态动作交互与严苛的实战评估体系。演讲首先解析了小型行动模型（如参数量仅为 1B 的 xLAM 系列）如何在降低延迟与成本的同时实现卓越的函数调用能力；接着介绍了结合思维链与外部工具调用的多模态动作模型 TACO，展示了突破纯文本局限的视觉推理路径；最后针对 MCP 协议带来的工具激增挑战，开源了自动化评测框架 MCPEval，为智能体的企业级投产就绪度提供了完备的量化评估方案。
## Atmosphere & style
整个演讲节奏明快、条理清晰，洋溢着前沿技术研究者兼具工程落地务实感的专业自信。Shelby Heinecke 采用循序渐进的叙事架构，以生动的比喻与直观的案例拆解复杂架构，既展现了 Salesforce AI Research 十年沉淀的技术底蕴，又通过全面的开源分享传递出强烈的社区开放精神与极富前瞻性的科技视野。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=4s) AI Agent 正在从单纯的文本响应器转变为具备推理、规划与执行能力的系统，其下一代演进聚焦于极低延迟、多模态交互和企业级实战评测。
- [01:31](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=91s) Salesforce AI Research 团队超前布局前沿技术基建，研发了支撑 Agentforce 运作的底层大脑——Atlas 推理引擎，并持续开源相关核心成果。
- [04:13](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=253s) 小型行动模型（SAM）具备低能耗、超低延迟和边缘泛在部署等优势，能在本地端侧完全保障敏感提示词与企业机密数据的隐私安全。
- [07:13](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=433s) 行动模型专为生成 API 调用等可执行动作而优化；开源的 10 亿参数轻量模型 xLAM-1B 在函数调用等 Agent 任务中表现媲美甚至超越参数大其 10 倍的模型。
- [09:56](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=596s) 传统多模态大模型在特定细粒度视觉推理上仍存在局限，克服这一瓶颈的关键在于赋予模型工具调用能力，将其打造为按需调度外部工具的动作模型。
- [12:05](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=725s) 开源多模态动作模型 TACO 结合了思维链与动作规划架构，通过 27 万条高质量合成轨迹训练，掌握了自主拆解复合任务并分步调用 CV 检测模型与代码的能力。
- [13:54](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=834s) MCP（模型上下文协议）虽为智能体连接各类企业软件提供了标准化通道，但工具接口的增多也对智能体的调用策略与执行鲁棒性提出了严峻考验。
- [15:24](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=924s) 开源评估框架 MCPEval 通过全自动合成贴近真实场景的测试集，量化评测智能体在 MCP 工具调用中的表现并生成就绪报告，加速企业级稳定投产。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=4s) AI Agent 新纪元：更快、多模态与实战评估
在会议的第三天，Salesforce 高级 AI 研究经理 Shelby Heinecke 开场指出，如果说前几年是 LLM（大语言模型）的爆发之年，那么当下正是属于 AI Agent（智能体）的时代。然而，目前市面上所见到的 Agent 应用仅仅触及了未来潜力的冰山一角。Salesforce AI Research 正在开展大量前沿研究，为下一阶段的 AI Agent 奠定坚实基础。
下一代 AI Agent 将展现出三大核心变革：首先是**更快的响应速度**，彻底告别冗长的等待时间；其次是**多模态交互能力**，突破当下仅能处理和生成文本的局限，让 Agent 能够自如理解并操作视频、音频和图像等复杂数据；最后是**经过严苛实战检验的可靠性**，通过全新开发的企业级评估体系，确保 Agent 在复杂的实际业务场景中稳定落地。最关键的是，本次分享中涉及的研究成果、代码及模型权重均已在 Hugging Face 等平台完全开源，开发者可通过现场提供的二维码立即获取并应用于自身系统。
```mermaid
flowchart TD
accTitle: 下一代 AI Agent 的三大核心演进方向
subgraph S1["速度与效率"]
A["Small Action Models\n(轻量级行动模型)"] --> D["极低延迟 / 边缘部署"]
end
subgraph S2["感知与执行"]
B["Multimodal Action Models\n(多模态行动模型)"] --> E["跨图文/音视频交互与推理"]
end
subgraph S3["质量与信任"]
C["Enterprise Eval Frameworks\n(企业级评估体系)"] --> F["严格测评 / 达到企业投产标准"]
end
```
### [01:31](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=91s) Salesforce AI Research 的创新底座与技术愿景
Shelby Heinecke 介绍了自己所带领的团队定位。该研究团队的核心使命是超前预判客户在未来 3 个月、6 个月乃至 9 个月后的真实需求，提前攻关下一代 AI 技术。团队的研发流程涵盖从前沿论文发表、技术原型开源到最终赋能核心产品的全链路。除了 AI Agent，团队的研究方向还覆盖端侧 AI（On-device AI，例如在手机端本地运行 LLM）以及小型语言模型（SLM）等尖端领域。
Salesforce 的研发体系具备深厚的历史积累，其 AI Research 部门已成立超过 10 年。自 2014 年成立以来，该团队一直是推动 Salesforce 产品技术革新的核心引擎。从 2022 至 2023 年的一系列 LLM 基础创新，到 2025 年正式发布的旗舰产品 **Agentforce**，其核心大脑——**Atlas 推理引擎（Atlas Reasoning Engine）** 正是由 AI Research 团队一手打造。团队目前已将技术视野投向 2025 年底乃至 2026 年，全力布局下一代 Agent 的核心技术基建。
- **组织角色**：Salesforce AI Research 拥有 10 年以上研发沉淀，负责为核心业务线输送颠覆性创新。
- **代表性成果**：研发了支撑 Agentforce 运作的底层大脑——Atlas 推理引擎。
- **前沿探索**：重点布局端侧 AI、小模型架构、行动模型及智能体评估。
---
### [04:13](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=253s) 小型行动模型（Small Action Models）的崛起
在详细拆解具体技术时，Heinecke 首先聚焦于“小型行动模型”。近期 NVIDIA 一篇题为《Small Language Models are the Future of Agentic AI》的爆火论文印证了这一技术趋势。相比动辄消耗海量 GPU/CPU 算力的大参数模型，参数量较小的小模型展现出了无可比拟的工程与商业优势：
1. **高能效与低成本**：参数量减少意味着更少的计算量，大幅降低了推理成本，提升了部署的可持续性。
2. **超低延迟**：由于前向传播中的数学运算大幅减少，模型能够实现极速响应。
3. **泛在部署能力**：小模型不仅能在云端实现超快响应，还能直接下沉部署到个人笔记本、本地私有服务器（On-prem）、智能手机、智能手表乃至无人机等各种边缘设备上。
4. **数据隐私与自主掌控**：当模型完全私有化部署在本地或专属环境中时，所有提示词与敏感数据无需回传至第三方服务商。这在构建需要访问个人隐私文档或企业机密数据的定制化 Agent 时至关重要。
```mermaid
flowchart LR
accTitle: 小型语言模型在 Agent 中的核心价值
subgraph In["输入端"]
P["用户私密指令 / 任务"]
end
subgraph Core["小型行动模型 (SLM/SAM)"]
M1["参数量少 & 算力消耗低"]
M2["超低推理延迟"]
end
subgraph Out["落地部署形态"]
D1["云端超快接口"]
D2["手机 / 笔记本 / 边缘端侧"]
D3["100% 数据私有化保障"]
end
P --> Core
Core --> Out
```
---
### [07:13](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=433s) 解密 Action Model 与开源力作“Tiny Giant” (xLAM)
所谓行动模型（Action Model），是指专门针对**生成可执行动作（Executable Actions）**进行深度优化的模型。这是 Agent 区别于传统文本生成 LLM 的核心所在。例如，当用户提问“*今天下午我的日历上有哪些会议？*”时，行动模型能够准确理解意图并将其转化为系统可直接调用的 API 查询代码，从而驱动外部系统完成真实任务。
针对“能否将负责执行核心动作的模型做小”这一关键挑战，Salesforce AI Research 给出了肯定答卷，并推出了名为 **Tiny Giant** 的轻量级行动模型系列——**xLAM-1B**（10 亿参数级别），同时提供 3B 和 8B 等不同规格。
与当前动辄数百亿、上千亿参数的主流大模型（如 6710 亿参数的 DeepSeek-V3、1230 亿参数的 Mistral Large）相比，仅有 10 亿参数的 xLAM-1B 体积仅为其几十分之一乃至几百分之一，却在函数调用（Function Calling）和各类 Agent 任务中，击败了参数规模比它大 10 倍以上的模型。Heinecke 强调，小模型之所以能爆发出如此强悍的性能，关键在于其经过了针对性的高质量数据微调与训练策略。相关论文、模型架构以及训练数据集已全套在社区完全开源。
| 模型名称 | 参数规模 | 典型应用与定位 |
|:--- |:--- |:--- |
| **DeepSeek-V3** | 671B（6710 亿） | 超大规模通用混合专家（MoE）模型，依赖高昂算力集群 |
| **Mistral Large** | 123B（1230 亿） | 典型的大规模旗舰级云端语言模型 |
| **xLAM-1B (Tiny Giant)** | **1B（10 亿）** | 专为 Agent 行动/函数调用设计的超轻量模型，性能媲美 10 倍以上大模型 |
| **xLAM-3B / 8B** | **3B / 8B（30亿 / 80亿）** | 兼顾多任务推理与极致效率的高性能行动模型系列 |
### [09:56](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=596s) 迈向多模态动作模型：突破纯文本的局限
在确立了小型模型是推动下一阶段 AI 智能体发展的核心动力之后，技术演进的下一个必然方向是拓展智能体的感知与行动边界。现实世界中的任务绝不仅限于文本交互，智能体需要学会处理更丰富的数据形态。为此，团队开始研发多模态动作模型（Multimodal Action Models）。未来的交互不再仅仅是输入一行简单的文本提示词（例如“我今天的日程安排是什么？”），而是可以直接传入视频、图像或音频等多模态信息；而多模态动作模型不仅能够理解这些输入，还能引用图像甚至生成多模态内容并执行相应动作。
然而，当前的多模态大模型在应对现实任务时仍存在显著短板。尽管人们每天使用的尖端模型在许多多模态基准上表现优异，但它们在面对一些“看似非常简单”的任务时依然频频受挫。演讲者展示了一个典型案例：向模型输入一张赛车图片并提问“身穿黄色骑行服的摩托车手所骑的摩托车上显示的是几号？”对人类而言，一眼就能锁定穿黄色衣服的人并读出号码；但几个月前将该任务输入 GPT-4o 以及其他开源多模态模型时，它们均给出了错误答案。
克服这一瓶颈的关键在于赋予模型使用工具的能力，将其转变为真正的“动作模型”。模型不应仅凭自身参数中存储的内部知识硬抗所有复杂推理，而应学会按需调用更专业、更聪明的外部模型或工具来辅助求解。
```mermaid
flowchart LR
accTitle: Multimodal Action Model Execution Flow
subgraph Input["输入阶段"]
A["多模态输入<br/>(图像 / 视频 / 音频 / 文本)"]
end
subgraph Agent["动作模型推理 (TACO)"]
B["思维链与动作规划<br/>(Chain of Thought & Action)"]
end
subgraph Tools["专业工具调用"]
C["CV 目标检测模型<br/>get_objects"]
D["代码执行 / Python<br/>count_objects"]
end
subgraph Output["最终输出"]
E["精准执行结果<br/>(如：检测出 4 个目标)"]
end
A --> B
B --> C
C --> D
D --> B
B --> E
```
---
### [12:05](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=725s) 开源多模态动作模型 TACO 与合成轨迹训练
为了践行这一理念，团队开源了多模态动作模型 TACO。这个名字巧妙地取自“思维与行动”（Thought And Action）的字母构词，其核心机制正是结合了思维链（Chain of Thought）与动作执行轨迹。为了训练该模型掌握复杂的工具调用推理链，研究团队合成了超过 270,000 条高质量的“图像-文本”动作轨迹数据。
演讲者举了一个具体的训练示例：给定一张包含多个角色的复杂照片，提问“这张照片中有多少个 Salesforce 吉祥物？”多模态大模型往往无法直接精准数清，而 TACO 学习到的解决路径是拆解任务并分步调用工具：首先调用名为 `get_objects` 的高性能计算机视觉（CV）模型来定位并提取所有目标，接着调用 Python 编写的计数函数 `count_objects` 计算数量，最终返回精准的答案“4”。通过在 27 万多条此类合成轨迹上进行训练，TACO 掌握了在面对多模态输入时自主规划、分步调取合适工具的能力，从而彻底攻克了传统端到端多模态模型难以解决的视觉推理难题。目前 TACO 已完全开源，供社区开发者探索与应用。
- **TACO 命名渊源**：由 Chain of Thought and Action（思维链与动作）衍生而来的易记缩写。
- **训练数据规模**：构建了超过 270,000 条合成的多模态交互与工具调用轨迹。
- **协同执行范式**：模型作为中央调度器，按需串联专用 CV 检测模型与代码工具完成复合任务。
---
### [13:54](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=834s) 企业级落地的瓶颈与 MCP 协议带来的工具洪流
构建速度更快、支持多模态的智能体只是第一步，要让智能体真正走出实验室并在企业级业务中稳定运行，面临着完全不同维度的工程挑战。"在自己的电脑上让智能体跑通 5 个演示样例固然很棒，但要让它在企业环境中稳定、可靠地工作，则是另一回事。"为此，业界亟需更加严密、精细的评估框架。
在当前的企业级智能体架构中，一个至关重要的组成部分是 MCP（模型上下文协议，Model Context Protocol）。MCP 的核心价值在于建立标准化的连接管道，让智能体能够极其便捷地与各类软件、企业应用和服务（例如 Slack、YouTube、Instagram 等）进行通信。简而言之，MCP 为智能体打开了前所未有的工具库大门，使其理论上具备了在各种系统内自主采取行动的巨大潜力。
然而，“理论上可行”与“实际上好用”之间存在着巨大的鸿沟。演讲者打了个生动的比方：如果现在直接递给你一个塞满各种专业工具的工具箱，你未必真的知道每一样工具该在何时、以何种方式正确使用。智能体也是如此——仅仅给它配置工具接口，并不等于它掌握了正确的调用策略与鲁棒的执行逻辑，必须通过系统化的评测来检验其真实能力。
---
### [15:24](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=924s) 开源评估框架 MCPEval：合成测试数据与就绪报告
为了填补智能体工具使用评测的行业空白，团队打造并开源了评测框架 MCPEval。评估智能体面临的最大痛点在于缺乏真实场景的测试数据，无论是人工手动编写测试用例还是设计评估标准都极其耗时费力。MCPEval 通过全自动的合成评估机制彻底解决了这一难题。
```mermaid
flowchart TD
accTitle: MCPEval Workflow
subgraph Setup["1. 配置阶段"]
A["接入待测 Agent"]
B["选择目标 MCP Servers<br/>(Slack / GitHub / CRM 等)"]
end
subgraph Gen["2. 数据合成阶段"]
C["MCPEval 引擎<br/>自动生成针对性合成测试集"]
end
subgraph Eval["3. 评测与产出阶段"]
D["执行工具调用评测"]
E["生成企业就绪度评估报告<br/>(Enterprise Ready Report)"]
end
A --> C
B --> C
C --> D
D --> E
```
在 MCPEval 的工作流程中，开发者只需将自己的智能体接入框架，并勾选该智能体被授权访问的 MCP 服务器。第一步，框架会自动针对这些特定的 MCP 工具生成高度贴合实际调用场景的合成评估数据集；第二步，框架运行评估测试并生成详尽的性能报告，量化分析智能体在调用这些 MCP 服务时的成功率、规划合理性与边界处理能力。开发者据此可以持续针对薄弱环节进行调优迭代，直至智能体真正达到企业级部署的严苛标准。
演讲最后总结道，下一代 AI 智能体的演进趋势清晰可见：模型体积将更加轻量小巧、推理响应更加敏捷迅速、全面支持多模态动作交互，并且伴随着更加健全严格的评估生态。随着 xLAM、TACO 以及 MCPEval 等一系列开源成果的发布，开发者已拥有了完备的工具链来加速构建面向未来的企业级 AI 智能体。
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=4s) 如果说前几年是 LLM（大语言模型）的爆发之年，那么当下正是属于 AI Agent（智能体）的时代。 — Shelby Heinecke
> [01:31](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=91s) 超前预判客户在未来 3 个月、6 个月乃至 9 个月后的真实需求，提前攻关下一代 AI 技术。 — Shelby Heinecke
> [04:13](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=253s) Small Language Models are the Future of Agentic AI — NVIDIA
> [07:13](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=433s) 今天下午我的日历上有哪些会议？ — Shelby Heinecke
> [07:13](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=433s) 生成可执行动作（Executable Actions） — Shelby Heinecke
> [09:56](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=596s) 身穿黄色骑行服的摩托车手所骑的摩托车上显示的是几号？ — Shelby Heinecke
> [12:05](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=725s) Thought And Action — Shelby Heinecke
> [12:05](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=725s) 这张照片中有多少个 Salesforce 吉祥物？ — Shelby Heinecke
> [13:54](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=834s) 在自己的电脑上让智能体跑通 5 个演示样例固然很棒，但要让它在企业环境中稳定、可靠地工作，则是另一回事。 — Shelby Heinecke
> [13:54](https://www.youtube.com/watch?v=yst5Ce_SbAM&t=834s) 如果现在直接递给你一个塞满各种专业工具的工具箱，你未必真的知道每一样工具该在何时、以何种方式正确使用。 — Shelby Heinecke
## Mentioned
Shelby Heinecke, Salesforce, Salesforce AI Research, Agentforce, Atlas 推理引擎 (Atlas Reasoning Engine), xLAM (Tiny Giant), TACO, MCPEval, MCP (Model Context Protocol), NVIDIA, DeepSeek-V3, Mistral Large, GPT-4o, Hugging Face, Slack, GitHub, Python
## Video description
Shelby Heinecke, Senior AI Research Manager, Salesforce
Abstracts
AI agents are shifting from simple responders to systems that can reason, plan, and act, across modalities. I’ll share how our research at Salesforce, from xLAM to MCPEval, is pushing agents toward real-world readiness: faster decisions, smarter actions, and seamless evaluation. If you're building the next generation of agents, this is what’s coming next.
[Source](https://www.youtube.com/watch?v=yst5Ce_SbAM)
Source quotes
[0:04] Shelby Heinecke: 如果说前几年是 LLM（大语言模型）的爆发之年，那么当下正是属于 AI Agent（智能体）的时代。
[1:31] Shelby Heinecke: 超前预判客户在未来 3 个月、6 个月乃至 9 个月后的真实需求，提前攻关下一代 AI 技术。
[4:13] NVIDIA: Small Language Models are the Future of Agentic AI
[7:13] Shelby Heinecke: 今天下午我的日历上有哪些会议？
[7:13] Shelby Heinecke: 生成可执行动作（Executable Actions）
[9:56] Shelby Heinecke: 身穿黄色骑行服的摩托车手所骑的摩托车上显示的是几号？
[12:05] Shelby Heinecke: Thought And Action
[12:05] Shelby Heinecke: 这张照片中有多少个 Salesforce 吉祥物？
[13:54] Shelby Heinecke: 在自己的电脑上让智能体跑通 5 个演示样例固然很棒，但要让它在企业环境中稳定、可靠地工作，则是另一回事。
[13:54] Shelby Heinecke: 如果现在直接递给你一个塞满各种专业工具的工具箱，你未必真的知道每一样工具该在何时、以何种方式正确使用。
Source quotes
[0:04] Shelby Heinecke: this year is the year of AI agents.
[4:13] Shelby Heinecke: Small Language Models Are the Future of Agentic AI
[4:13] Shelby Heinecke: Tiny Giant
[13:54] Shelby Heinecke: It's like if I gave you a toolbox right now, do you know how to use those tools necessarily? Maybe, maybe not.
[16:51] Shelby Heinecke: Thank you all so much.
Original transcript
[0:04] So, we made it to day three, and I'm
[0:06] guessing day one, day two, you've heard
[0:09] a lot about AI agents, right? This year
[0:12] is the year of AI agents. Last year is
[0:14] the year of LLM, maybe the year before,
[0:16] but agents all around, right? So, what I
[0:19] want to tell you about today is what's
[0:22] what we have in store for AI agents
[0:24] because a lot of what you've seen is
[0:26] just scratching the surface on what's
[0:28] possible. Everyone, we have a lot of AI
[0:30] research that's going to lay the
[0:32] groundwork for a next phase of AI agents
[0:34] that's coming. And the next phase of AI
[0:36] agents is going to be faster. Tired of
[0:39] waiting for your agents? They're going
[0:40] to be faster. I'm going to tell you why.
[0:42] They're going to be multimodal. A lot of
[0:44] agents today can only read and handle
[0:47] text data and generate text. But we need
[0:50] agents that can navigate our videos, our
[0:52] audio, our images, and so on. So, we've
[0:56] got some innovations around multimodal
[0:57] agents. that I'm going to tell you about
[0:58] today. And finally, we need agents that
[1:01] are battle tested, right? We've deployed
[1:04] agents across industries, have shown
[1:06] great success, but we're realizing that
[1:08] we need to evaluate our agents a little
[1:10] bit better. So, we've got some
[1:11] innovation there to share with you
[1:12] today, too. Really awesome stuff. And
[1:15] the best part, everything I'm going to
[1:16] share with you today is completely open
[1:18] source, everyone. So, I'm going to share
[1:19] QR codes all in hugging face, all
[1:22] completely open source. you can check it
[1:23] out on your computers as you're
[1:25] listening and you can you can apply it
[1:27] to your agents right away. So, let's go
[1:29] ahead and dive in.
[1:31] So, a little bit about me everyone. Why
[1:33] am I talking about this? So, as as you
[1:35] heard, I'm a senior AI research manager.
[1:38] I lead an AI research team at Salesforce
[1:41] and our role, we're always thinking
[1:43] about what the customer is going to need
[1:44] three, six, nine months ahead of time,
[1:47] if not more, right? We're always
[1:49] thinking about the the next AI we need
[1:51] to build. And so as we're doing that,
[1:54] we're open sourcing a lot. That's why
[1:56] the the work I'm tell showing you today
[1:58] is open source. You can check it out
[2:00] yourself. We write a lot of papers on
[2:02] our findings. So we contribute to the
[2:04] cutting edge. We lead the cutting edge
[2:06] when it comes to AI agents. And
[2:08] ultimately, of course, we want to we
[2:10] deploy to product. So the papers in open
[2:13] source are are basically prototypes that
[2:15] we're releasing on the way to getting it
[2:18] into product. Okay. And so my team
[2:21] focuses on AI agents, but not only AI
[2:24] agents, there's some other cutting edge
[2:26] directions. Won't talk about them too
[2:27] much today, but on device AI is
[2:30] something we're thinking about. So
[2:31] imagine LLMs on your phones, for
[2:33] example. My team also focuses on small
[2:36] language models. I'll talk about that
[2:37] today. That's going to be key for the
[2:39] next phase of AI agents and more. So
[2:42] happy I only have 20 minutes here today,
[2:45] but happy to sync offline. Feel free to
[2:47] connect on LinkedIn and we can talk more
[2:48] out after the talk.
[2:52] A little bit about Salesforce.
[2:55] In case you didn't know, Salesforce has
[2:56] had a research org for over 10 years,
[2:59] everyone. And it's the innovation
[3:01] powerhouse of Salesforce. So again, it's
[3:04] this it's this org, this innovation org
[3:06] that is always thinking ahead and that
[3:08] is propelling the the product forward.
[3:11] started back in 2014, so a while ago.
[3:14] Fast forward to 2022, 2023, you'll see a
[3:17] lot of LLM based innovations and then
[3:20] afterwards, a lot of LLM things going
[3:22] into product. Fast forward to today,
[3:24] 2025.
[3:26] If you all in case you all aren't aware,
[3:28] we have released agent force and some of
[3:31] the key innovations that make agent
[3:33] force what it is. For example, the Atlas
[3:35] reasoning engine, the brain of Agent
[3:37] Force was developed by the AI research
[3:39] team. So, what I'm going to tell you
[3:42] today is we're looking past 2025 or
[3:45] maybe we're looking towards the end of
[3:47] 2025 into 2026. What have we been
[3:49] working on and what is this and what
[3:52] groundwork we're laying for the next
[3:55] batch of agents.
[3:58] So, here we are. Here's the road map for
[3:59] today, everyone. I want to tell you
[4:00] about three key innovations. And again,
[4:03] this is three key open source
[4:05] innovations that are going to power the
[4:07] next AI agents you're going to see
[4:09] across the industry. The first
[4:13] the first is small action models. I'm
[4:16] going to get into what an action model
[4:17] is, but you may have seen from other
[4:20] talks, small models are going to be key.
[4:22] We're going to dive into that. But these
[4:24] small action models are going to not
[4:25] only make our agents better performing,
[4:26] they're going to make it faster. Imagine
[4:28] not waiting as long for your agent to
[4:30] get the work done.
[4:31] Then I'm going to talk about multimodal
[4:34] action models. Again, having fast agents
[4:37] is nice, but we want agents that can
[4:39] operate beyond text. The future is not
[4:42] just limited to text. We're going to
[4:43] have audio, video, as I mentioned
[4:45] earlier. We need agents that are going
[4:46] to be smart and be able to reason about
[4:48] that about those types of data, too. So,
[4:49] we've got some key innovations around
[4:51] multimodal action models. And finally,
[4:54] we're going to have even though we'll
[4:56] have faster agents, we'll have
[4:57] multimmodal agents, it doesn't matter if
[5:00] it's not performing for enterprise,
[5:01] right? We have enterprisegrade agent
[5:04] eval frameworks that we've developed
[5:06] that are going to help to make sure that
[5:08] your agents are ready for deployment,
[5:10] ready for enterprise use. So, let's dive
[5:12] in everyone. Let's start with those
[5:14] small action models. Who here saw this
[5:16] viral paper? This was a very viral paper
[5:20] from Nvidia recently. Okay, good. a good
[5:22] portion of people and it's titled small
[5:25] language models are the future of
[5:26] agentic AI and I couldn't agree more
[5:29] there's so many reasons for that first
[5:32] you know when we when we think about
[5:34] deploying agents that are powered by
[5:36] LLMs you know they consume GPU CPU and
[5:39] so on smaller models they have less
[5:42] parameters remember parameters are the
[5:44] number of weights in the deep neural
[5:45] network small models simply have less
[5:47] parameters which means they consume less
[5:49] compute so these are going to be much
[5:52] more sustainable, much more affordable
[5:53] to deploy. So small models are
[5:55] beneficial for the fact that they
[5:57] consume less compute. Now they're also
[6:00] low latency. In other words, they're
[6:01] fast. And again, that's because they
[6:03] have fewer parameters, which means fewer
[6:06] mathematical operations that need to be
[6:08] done to get an output. They're a lot
[6:10] faster.
[6:12] And because of both of those, they can
[6:14] be deployed almost anywhere. That's the
[6:16] beautiful thing, right? They can be
[6:18] deployed in the cloud, which is useful.
[6:20] You want fast, you want fast interaction
[6:22] on the cloud, but they can also be
[6:24] deployed on your personal laptop. They
[6:26] can be deployed on prim. They can be
[6:28] deployed on your phone if it's small
[6:30] enough, on a smartwatch, on a drone, on
[6:33] devices, right? So small models are
[6:36] opening the door to agentic experiences
[6:39] on all devices. So that's pretty
[6:40] powerful. And finally, this is a great
[6:44] benefit with a small model. You're in
[6:46] control. You own the model. you whatever
[6:49] you pass in the model doesn't go
[6:51] anywhere. It does not go to a third
[6:52] party. That's powerful especially as we
[6:54] think of personalized or customized
[6:58] agents, right? Imagine an agent that
[7:00] you'll want to navigate your personal
[7:02] documents. You do not want that sitting
[7:05] on some thirdparty server. You want your
[7:07] data to be safe. A small model is going
[7:10] to get you there.
[7:13] Now, not just any small model I want to
[7:14] talk about today. I want to talk about
[7:15] action models. Now you may have heard of
[7:19] this term large action models. Now an
[7:22] action model is a model that is
[7:24] specialized for generating an executable
[7:27] action. This is the heart of what an
[7:30] agent can do. This is why an agent is
[7:31] different from an LLM. An L an agent can
[7:34] actually take action. So what I mean by
[7:36] that is imagine we have this prompt.
[7:38] What meetings are on my calendar this
[7:40] afternoon? an action model can take that
[7:43] and generate the correct API call so
[7:46] that the system can execute it. In other
[7:49] words, it empowers that action. That's
[7:51] the key.
[7:53] And so the big question is if these are
[7:55] so critical for AI agents and they're
[7:57] currently powered by large models, can
[8:00] we take this important piece of an agent
[8:02] and make it smaller?
[8:05] And the good news is yes, we can. And
[8:07] we've shown that we can do it even by
[8:10] even with beating models more than 10
[8:12] times its size. So one of the things
[8:13] that we've developed at Salesforce is
[8:15] called what we've called the tiny giant.
[8:19] Tiny giant. It is a 1 billion parameter
[8:22] model everyone. So I'm going to put that
[8:24] in perspective for you in a second. But
[8:25] 1 billion parameters is a really really
[8:27] small model. And what we've done is
[8:28] we've shown that when it comes to taking
[8:30] action, when it comes to function calls,
[8:31] when it comes to agentic tasks, a 1B
[8:34] model trained properly, that's the key.
[8:37] We fine-tune this properly, trained
[8:39] properly, can beat models more than 10
[8:42] times its size. So we can pack a lot of
[8:44] punch in a small size. So I want to put
[8:46] that into perspective for you so we're
[8:48] all on the same page on how powerful
[8:49] this is. When I say 1 billion
[8:52] parameters, look how small that is. You
[8:55] see here, we see here very common LLMs
[8:58] that you I'm sure you all are familiar
[9:00] with and have tried. Deepseek V3 671
[9:03] billion parameters. Mistro large 123
[9:07] billion parameters. The point here is
[9:09] that a lot of LLMs, the typical
[9:11] run-of-the-mill LLM that you're used to
[9:13] has hundreds of billions of parameters
[9:14] in size, requires tons of GPUs to run.
[9:17] What I'm saying is that our tiny giant
[9:20] XL SLAM 1B, we also have a three billion
[9:23] parameter model and 8 billion parameter
[9:25] model are all a fraction of the size but
[9:28] can perform just as just as well on
[9:30] agentic tasks.
[9:33] So as I mentioned, great news is this is
[9:36] completely open source. We even open
[9:37] sourced uh a lot of the training data.
[9:39] So you can see um how we you see what
[9:42] our training data looks like. Check
[9:44] follow feel free to follow the link. You
[9:46] can even see we've written some papers
[9:48] on how we generated the training data
[9:50] because for a model this size to perform
[9:53] so well the big secret was the training
[9:56] data. That was a secret. So doing that
[9:58] is everything. So feel free to check it
[10:00] out.
[10:04] Awesome. Awesome. So there we have it.
[10:06] So small models are key. I think we're
[10:08] all on the same page there. That's what
[10:09] we're going to be seeing coming for our
[10:11] next uh phase of AI agents. Now let's
[10:14] talk about expanding the scope of agents
[10:16] though. It agents need to navigate more
[10:18] than text.
[10:20] That's why we started thinking about
[10:21] multimodal action models. We've started
[10:24] developing those already. So now instead
[10:27] of just passing in a prompt like this, a
[10:29] text prompt. What's on my agenda for
[10:31] today? I'm passing in multimodal
[10:33] information as well. Maybe I'm passing
[10:34] in a video, maybe I'm passing in an
[10:36] image, audio. And now I'm now my
[10:39] multimodal action model is able to
[10:41] generate multimodal action. Maybe it's
[10:43] referencing an image. Maybe it's could
[10:45] be possibly even generating multimodal
[10:47] information.
[10:49] So a big question though,
[10:51] where do today's multimodal still stand?
[10:54] Multimodal models still stand. So we
[10:57] know that a lot of models that we use
[10:59] every day are great at multimodal tasks.
[11:01] But what I wanted to share is that they
[11:04] still struggle on seemingly simple
[11:07] tasks. Right? We're seeing this even
[11:09] with LLMs too. I want to show you this
[11:11] example with multimodal models. So in
[11:13] this example, check check out this
[11:14] picture. And this in this picture,
[11:17] imagine we're asking what number is
[11:19] displayed on the motorcycle ridden by
[11:21] the motorcyclist wearing a yellow suit.
[11:23] So for us, if we look at this, we can
[11:26] see the answer immediately, right? We
[11:27] can spot the person with yellow and and
[11:29] know the and know their number. However,
[11:32] this was this was something that we
[11:33] passed into GPT40, so a few months ago,
[11:37] and another open-source multimodal
[11:39] model. And the point here that I want
[11:41] you to see is that they would get these
[11:43] wrong. They would get this wrong.
[11:46] However, one way we can overcome this is
[11:50] if we if we empowered these models to be
[11:52] able to use tools. So if we made them
[11:55] action models, so they didn't have to
[11:57] rely on their internal knowledge to
[11:59] solve this. Maybe they can call a
[12:00] smarter model to help them solve this.
[12:02] That's exactly what our open-source
[12:05] multimodal action model does.
[12:08] We've called it taco. And taco the
[12:11] there's a there's a reason for that
[12:13] name. We call it taco because it reasons
[12:15] with chain of thought. So a lot of you
[12:17] here have probably heard of chain of
[12:19] thought. It reasons with chain of
[12:21] thought and action. So that's how and
[12:24] taco is an anagram of that. So what
[12:27] we've done is in order to train this
[12:29] model, we've created over 270,000
[12:32] synthetic
[12:34] image text trajectories. So, let me give
[12:36] you an example what that looks like.
[12:39] So, here's an example. Here's a picture.
[12:41] Here's a task. How many Salesforce
[12:43] mascots are in this photo? Easy for us
[12:46] to do, of course. But again, we saw that
[12:48] multimodal models couldn't sometimes
[12:50] struggle with these tasks. So, what
[12:52] we've done is we've created these chain
[12:54] of thought and action trajectories as
[12:56] you can see here. the the act the
[12:58] correct actions to take would be to call
[13:00] get objects which could be some kind of
[13:03] better performing CV model for getting
[13:05] objects. Then we could call a function
[13:07] called count objects which could be a
[13:09] Python function who whatever function
[13:10] right and then we would turn and then
[13:13] the output would be the final result
[13:15] which would be four. So what I want you
[13:17] to see here is that this is just one
[13:19] example we that we synthetically
[13:22] created. We created 270,000
[13:25] if not more examples like this, took
[13:29] this, trained the model on this. So now
[13:33] the model when it sees images, when it
[13:35] sees multimodal information, it can
[13:37] reason in this train of thought and
[13:39] action way. It can take the correct
[13:40] actions to ultimately get these complex
[13:43] challenges correct.
[13:46] So Taco is completely open source as I
[13:48] mentioned. Feel free to check it out. QR
[13:50] code right there. Thanks so much
[13:52] everyone.
[13:54] Awesome. So towards the end of the road.
[13:58] We've so so far we've talked about fast
[14:00] agents. We need that. We've talked about
[14:01] multimodal agents. We need that. That's
[14:03] coming. We need them to be
[14:05] enterprisegrade though. That's one thing
[14:06] we've learned over the past couple of
[14:08] years, right? That getting an agent to
[14:10] work on your computer and show it on
[14:11] five examples is great, but getting it
[14:13] to work reliably for enterprise is
[14:16] another is another challenge, right? So
[14:18] we need to think about more
[14:20] sophisticated
[14:21] evaluation frameworks and that's what I
[14:24] want to tell you about next.
[14:27] So one of the newest components as we
[14:30] think about enterprise agentic systems
[14:32] is MCP
[14:34] model pro model context protocol and it
[14:38] what MCP does it enables agents to
[14:41] easily talk to apps talk to software
[14:45] talk to services in a very very easy
[14:48] way. So, long story short, MCP enables
[14:51] the agent to access lots and lots of
[14:54] tools, more tools than it ever had
[14:55] access to before. And that's great. That
[14:58] means our agent can take action in all
[14:59] these different ways. It could go in
[15:01] Slack. It can go in YouTube. It can
[15:03] navigate Instagram. And that's that's
[15:05] great. That's very powerful in theory.
[15:08] In theory. But the but the problem is
[15:10] that just because an agent has access to
[15:11] tools doesn't mean it knows how to use
[15:13] it, right? It's like if I gave you a
[15:16] toolbox right now, do you know how to
[15:18] use those tools necessarily?
[15:20] Maybe, maybe not. We have to test and
[15:22] see.
[15:24] That's exactly why we we created MCP
[15:27] Eval. We saw this gap. We saw this gap
[15:30] and we knew we needed to fill that gap
[15:32] of performance. So, MCP Eval is a
[15:36] totally open source framework where you
[15:38] plug in your agent, you select the MCP
[15:41] servers your agent could have access to,
[15:44] and in step one, it it automatically
[15:47] creates evaluation data for those MCP
[15:50] servers. That's the key. Coming up with
[15:53] real examples for testing agents is
[15:55] hard. It's it's challenging. You either
[15:58] ask someone to do it or you come up with
[16:00] clever ways to do it do it
[16:02] synthetically. That's what we've done.
[16:03] So step one, you just select the the MCP
[16:05] servers. It comes up with great
[16:07] synthetic examples that that utilize
[16:09] those tools.
[16:12] Now once you have the that evaluation
[16:14] data set generated, you're ready to
[16:16] evaluate and you generate a report. So
[16:18] now you can actually measure your agents
[16:19] performance on the MCP servers that
[16:21] you've given it access to. And from
[16:23] there, at that point, you can continue
[16:25] to improve it and get your get your
[16:26] agent enterprise ready.
[16:30] So MCP Eval is completely open source.
[16:32] Feel free to check it out.
[16:35] Thanks everyone.
[16:39] Awesome. Awesome. So that wraps that
[16:41] wraps it up everyone. So as you can see
[16:43] the future of agents, smaller, faster,
[16:46] it's going to be multimodal
[16:48] and you're going to see more and more
[16:49] agent evals. We need a lot of agent
[16:51] evals. We're starting to release all of
[16:53] these. So check out these open source
[16:57] tools today to improve your agents and
[16:58] get them ready for the next phase.
[17:02] Thank you all so much.
