Startups Trade Control for Speed on Claude Agents, Seek Cost Levers

Claude

Founders argue that building proprietary agent infrastructure stalls early product validation, but scaling high-fanout workflows requires managed platforms to deliver batch pricing and tiered model routing to remain viable.

AI-assisted summary of How founders build on Claude Managed Agents — original source. Check the original for full context.

Source published: 2026-09-08

Key ideas

Startups like Wispr, Actively, and Pendo launched production-ready agent features in as little as one day to two weeks by adopting Claude Managed Agents instead of engineering proprietary orchestration harnesses. (0:07)
Wispr guards user trust by utilizing Managed Agents' Outcomes feature with isolated context windows, validating briefs against objective rubrics and suppressing outputs entirely if candidate text fails verification. (2:14)
Actively employs a two-tier architecture where a supervisory 'Watchtower' agent dynamically writes code to fan out analytical queries across 500 dedicated per-account agents to aggregate executive pipeline forecasts. (18:43)
Founders cite cost attribution and the lack of native asynchronous batch processing as primary friction points, noting that scheduled batch execution could slash non-urgent background agent costs by 50% to 75%. (28:48)

Detailed notes

## Background Published on YouTube by Claude on September 8, 2026, this panel features three founders from Wispr, Actively, and Pendo discussing how they build and scale autonomous agent systems using Claude Managed Agents. The discussion focuses on practical production realities, including outcome verification, org-wide memory, code sandboxing, evaluation strategies, and the operational calculus of build versus buy.

## TL;DR Founders from Wispr, Actively, and Pendo share how adopting Claude Managed Agents enabled them to ship complex features—such as pre-meeting briefs, cross-account sales aggregations, and self-healing telemetry pull requests—in days to weeks rather than months. They detail critical architecture patterns, including decoupling worker agents from rubric-driven verifiers in clean context windows, sandboxing dynamic code execution, and managing long-term memory across individual accounts versus organizational workflows. The group tracks the evaluation lifecycle from early 'vibes-based' iteration to regression suites derived from real customer traces, emphasizing the difficulty of evaluating mutable, stateful tools. Finally, they weigh the economics of managed platforms against proprietary harnesses, calling for finer-grained cost levers, batch pricing tiers, and sandbox pre-warming to mitigate latency.

## Atmosphere & style The panel carries a candid, pragmatic, and unpretentious engineering tone, feeling like an authentic peer-to-peer post-mortem among founders trading notes from the production trenches. Conversations move briskly through technical nuance without marketing gloss, with speakers laughing off common industry headaches like messy Salesforce schemas and the sheer absurdity of attempting to mock an entire customer Slack workspace. The dynamic is collaborative and grounded, balancing enthusiasm for rapid prototyping speed with hard-nosed scrutiny around token attribution, unexpected stylistic model quirks, and production costs.

## Bulletin - [00:07](https://www.youtube.com/watch?v=hm8NzEd5io0&t=7s) Startups like Wispr, Actively, and Pendo launched production-ready agent features in as little as one day to two weeks by adopting Claude Managed Agents instead of engineering proprietary orchestration harnesses. - [02:14](https://www.youtube.com/watch?v=hm8NzEd5io0&t=134s) Wispr guards user trust by utilizing Managed Agents' Outcomes feature with isolated context windows, validating briefs against objective rubrics and suppressing outputs entirely if candidate text fails verification. - [18:43](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1123s) Actively employs a two-tier architecture where a supervisory 'Watchtower' agent dynamically writes code to fan out analytical queries across 500 dedicated per-account agents to aggregate executive pipeline forecasts. - [28:48](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1728s) Founders cite cost attribution and the lack of native asynchronous batch processing as primary friction points, noting that scheduled batch execution could slash non-urgent background agent costs by 50% to 75%.

## Key takeaways - [12:13](https://www.youtube.com/watch?v=hm8NzEd5io0&t=733s) Building a custom agent harness is only justifiable when low-level execution mechanics constitute the core differentiating IP of the product; otherwise, doing so is premature optimization. - [02:14](https://www.youtube.com/watch?v=hm8NzEd5io0&t=134s) In proactive assistant workflows, delivering a false-positive brief damages customer trust far more than showing no information at all. - [05:16](https://www.youtube.com/watch?v=hm8NzEd5io0&t=316s) Decoupling verifier agents into isolated context windows prevents evaluation bias from contaminated worker reasoning traces and enables authentic quality gating. - [06:52](https://www.youtube.com/watch?v=hm8NzEd5io0&t=412s) Actively overcomes inconsistent Salesforce schemas and deprecated fields by querying users interactively to learn custom definitions and persisting them into organization-wide memory. - [10:38](https://www.youtube.com/watch?v=hm8NzEd5io0&t=638s) Pendo offloaded sandbox container management to Managed Agents to securely inspect customer source code and submit automated pull requests without exposing internal infrastructure to arbitrary execution vulnerabilities. - [16:03](https://www.youtube.com/watch?v=hm8NzEd5io0&t=963s) Offloading undifferentiated mechanics like prompt caching and context window compaction frees engineering teams to focus entirely on user retention and qualitative output. - [16:03](https://www.youtube.com/watch?v=hm8NzEd5io0&t=963s) Early agent iteration benefits from qualitative, vibes-based testing to establish product stickiness before formalizing quantitative offline eval suites. - [17:49](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1069s) Actively deploys a hybrid memory model, retaining proprietary control over long-lived, multi-year account histories while relying on Managed Agents for shared organizational rules and rep preferences. - [21:43](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1303s) Pendo pairs source code sandboxing with live application telemetry to create self-healing software loops that detect conversion funnel drops and autonomously submit remediation pull requests. - [26:41](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1601s) Evaluating agents against mutable state environments like live Slack channels is inherently brittle, as external state drift breaks offline assertions unless teams undertake complex mocking. - [28:13](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1693s) When transitioning between foundational model generations, teams should prioritize detecting new stylistic quirks and distinct failure modes over micro-optimizing prompts for marginal intelligence gains. - [31:13](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1873s) High-fanout architectures require Anthropic to provide tiered pricing controls, enabling a frontier model orchestrator to delegate sub-tasks to cheaper, lightweight models across hundreds of accounts. - [33:33](https://www.youtube.com/watch?v=hm8NzEd5io0&t=2013s) Claude Managed Agents provide the highest leverage for workflows that demand peak frontier reasoning, can tolerate multi-minute runtimes, and prioritize rapid feature validation over low-level infrastructure control.

## Detailed walkthrough ### [00:07](https://www.youtube.com/watch?v=hm8NzEd5io0&t=7s) How Wispr, Actively, and Pendo Deploy Managed Agents

The conversation opens with three founders explaining how they have integrated Claude Managed Agents into their core products across different business domains. Sahaj kicks off by describing how Wispr approaches meetings. While almost everyone in the industry narrowly focuses on what happens during a meeting, Wispr views meetings as a three-stage journey: preparation before the conversation, facilitation during the call, and execution on follow-ups afterward. Using Managed Agents, Wispr built pre-meeting briefs to ensure users walk into discussions prepared and post-meeting task automation to handle the downstream work automatically.

Mihir explains that Actively provides autonomous agents for enterprise sales teams. Actively's baseline architecture assigns a dedicated agent to every single customer account, continuously accumulating memory and determining the next steps to advance deals. However, sales leaders and representatives also need macro-level cross-account intelligence—answering questions like which top five accounts a rep should prioritize each morning or which deals a sales manager should include in next quarter's forecast. To solve this, Actively built "Watchtower," an aggregation, roll-up, and cross-account reporting agent requiring completely different tools from the single-account agents. Leveraging Managed Agents, the team built and shipped Watchtower in just two weeks, turning it into one of their most-used features and an underlying engine across the product.

Rounding out the panel, Todd describes Pendo's product analytics platform, which instruments customer applications to track user behavior and provide product improvement insights. As software development cycles have accelerated, maintaining accurate instrumentation has become painful because code changes cause analytics tracking to drift away from reality. To fix this, Pendo introduced an agent sitting directly in front of their platform. Using Managed Agents, this agent snapshots customer codebases, correlates what the code is doing with actual telemetry in the running application, and proactively surfaces insights alongside automated code fixes.

```mermaid flowchart TD accTitle: Claude Managed Agents Across Wispr Actively and Pendo accDescr: High-level overview of how Wispr, Actively, and Pendo implement Managed Agents across meetings, sales workflows, and codebase telemetry.

subgraph Wispr["Wispr: Meeting Lifecycle"] W1["Pre-Meeting Briefs"] --> W2["Live Call Support"] --> W3["Post-Meeting Task Automation"] end

subgraph Actively["Actively: Sales Intelligence"] A1["Per-Account Agents"] --> A2["Watchtower Engine"] A2 --> A3["Cross-Account Roll-Ups & Forecasting"] end

subgraph Pendo["Pendo: Analytics Drift"] P1["Codebase Snapshot"] --> P2["Runtime Application Telemetry"] P2 --> P3["Proactive Insights & Code Fixes"] end ```

- **Wispr's meeting paradigm**: Expands beyond live audio transcription to automated pre-meeting briefs and post-call task execution. - **Actively's Watchtower**: Shipped in two weeks on Managed Agents to support cross-account roll-ups, rep daily prioritization, and quarterly pipeline forecasting. - **Pendo's telemetry agent**: Bridges codebase snapshots with production analytics to detect tracking drift and submit fixes.

---

[02:14](https://www.youtube.com/watch?v=hm8NzEd5io0&t=134s) Outcome Verification and Rubric-Driven Briefs at Wispr

When asked about the features within Managed Agents that matter most, Sahaj highlights "Outcomes." In Wispr's pre-meeting briefs, the agent synthesizes who the user is meeting, why they are meeting, what prior agreements were established, and contextual notes from previous interactions weeks prior. However, when an agent hallucinates or confuses identities, the feature rapidly becomes counterproductive. If the agent matches the wrong person—such as conflating two individuals named Tom—the user clicks the profile, reads false pre-context, and enters the meeting completely disoriented.

Because false positive information is actively destructive, Wispr relies on Managed Agents' Outcomes feature to evaluate outputs against an objective rubric. The Outcomes system functions as an independent evaluator, validating whether the brief hit the target criteria and verifying source attribution before anything is presented in the user interface. If the generation fails the rubric, Wispr intentionally suppresses the brief entirely: showing no brief at all is far better for product trust than presenting incorrect pre-reads.

The rubric itself evaluates both objective facts and human consumption ergonomics. Objectively, it verifies deterministic data points, such as checking whether the retrieved LinkedIn profile matches the exact person attending the calendar event. Ergonomically, it scores readability and scan-ability. A rambling five-page document is useless right before a meeting. The rubric forces the agent to organize information in a strict progressive structure—first identifying who the person is, then explaining why the meeting is happening, and finally outlining action items. The agent hill-climbs against these UX-centric rubric criteria until the brief is verified or discarded.

- **The false positive penalty**: Delivering inaccurate pre-context (e.g., pulling the wrong person named Tom) damages user trust more than displaying no pre-read at all. - **Objective rubric verification**: Validates exact matches between calendar attendees and fetched external profiles like LinkedIn. - **UX and readability scoring**: Penalizes overly dense text to prevent multi-page briefs, enforcing a scannable format ordered by attendee identity, meeting purpose, and expected outcomes. - **Hill-climbing against rubrics**: Allows the agent loop to iterate against explicit presentation criteria prior to serving output to users.

---

[05:16](https://www.youtube.com/watch?v=hm8NzEd5io0&t=316s) Decoupled Evaluators and Independent Context Windows

Addressing how verification works in practice, Sahaj explains the operational runtime and ground-truth mechanics of Wispr's briefs. Wispr runs its brief generation roughly 24 hours before a scheduled meeting. This extensive buffer provides ample time for the system to run agent workflows, pull external information, and complete evaluation loops without tight latency pressures.

```mermaid flowchart LR accTitle: Wispr Decoupled Verifier Architecture accDescr: Diagram illustrating Wispr's separation of the worker agent trace from the verifier agent using independent context windows.

subgraph Generation["Generation Window"] A["Calendar & Email Data"] --> B["Worker Agent"] B --> C["Candidate Brief"] end

subgraph Verification["Independent Verification Window"] C --> D["Verifier Agent"] E["External Sources (LinkedIn)"] --> D D -->|Pass| F["Render to User"] D -->|Fail| G["Suppress Output"] end ```

The critical architectural advantage in Managed Agents is the strict separation between the worker agent that performs the task and the verifier agent that assesses the work. Rather than evaluating candidate text inside the worker agent's contaminated execution trace, Outcomes uses an entirely independent context window. Just as a human sales rep reviews an email thread, calendar invite, and LinkedIn profile to intuitively spot mismatches—noticing when something "does not look like the right person"—the independent verifier agent examines the primary sources and candidate output with clean reasoning.

This isolation guarantees that quality assessment is an authentic deployment gate. Wispr does not always demand that the agent loop continuously until perfection is reached; instead, they take advantage of the verifier's judgment to silently swallow unverified insights. Sahaj summarizes their core operating philosophy for proactive assistance: "Help when you can and do not cause harm or annoyance."

- **24-hour lead time**: Brief workflows spin up a full day prior to calendar events, removing real-time generation latency constraints. - **Clean context verification**: The verifier agent operates in an isolated context window, avoiding bias from the worker agent's chain-of-thought traces. - **Heuristic mismatch detection**: Mirroring human intuition, the verifier compares calendar invites, email exchanges, and LinkedIn metadata to identify subtle identity mismatches. - **Graceful suppression**: The platform drops briefs when confidence falls below the rubric threshold rather than risking annoying or confusing the user.

---

[06:52](https://www.youtube.com/watch?v=hm8NzEd5io0&t=412s) Org-Wide Memory, Salesforce Messiness, and Watchtower's Agent Fan-Out

Mihir details how Actively approaches memory and cross-account reasoning in sales environments. Enterprise sales data is notorious for idiosyncratic nomenclature and inconsistent business logic: every company calculates deal ownership differently, runs pipeline forecasting according to custom criteria, and uses heavily customized CRMs. Mihir notes that virtually every company's Salesforce setup contains "30 different fields that mean the same thing that were deprecated," making standardized prompting useless out of the box.

Instead of requiring long, manual administrative setup during onboarding, Actively uses interactions with the sales team to learn these unique concepts dynamically. When a user first asks a query—such as asking which deals should be forecasted for the upcoming quarter—the system asks clarifying questions, maps the company's internal definitions, and stores that logic in an organization-wide memory store. Once stored, those learned rules persist across the entire organization, improving answers for every team member without manual configuration.

```mermaid flowchart TD accTitle: Actively Watchtower Fan-Out Execution Flow accDescr: Flowchart showing how Watchtower processes complex queries by writing code, filtering accounts, fanning out tasks to per-account agents, and rolling up results.

Q["User Query: 'Forecast deals for next quarter'"] --> W["Watchtower Agent"] M[("Org-Wide Memory")] <-->|Business Logic & Definitions| W W --> C["Dynamic Code Execution & Tool Calling"] C --> F["Candidate Filter (e.g., 500 Deals)"]

subgraph FanOut["Fan-Out Architecture"] F --> A1["Account Agent 1"] F --> A2["Account Agent 2"] F --> A3["Account Agent N"] end

A1 --> R["Roll-Up Aggregation"] A2 --> R A3 --> R R --> Out["Consolidated Forecast Report"] ```

Beyond memory, Managed Agents enables Watchtower to handle deep workloads across hundreds of accounts through dynamic code generation and programmatic tool calling. When a manager asks complex forecasting questions across 500 deals, Watchtower does not attempt to evaluate everything in a single context window. Instead, it generates and executes code to filter the candidate pool, programmatically fans out specific analytical tasks to the dedicated agents running on individual customer accounts, and rolls up the findings into a consolidated answer. For an account executive managing a book of 100 accounts, this creates an operational structure where a personal team of 100 dedicated agents works in parallel under the orchestration of Watchtower.

- **Messy CRM schemas**: Solves the common enterprise hurdle of deprecated, duplicated Salesforce fields without requiring tedious manual onboarding configuration. - **Clarification and org-wide memory**: Dynamically learns company-specific sales definitions (e.g., deal stages, ownership boundaries) via user Q&A and persists them across the whole company. - **Dynamic code generation**: Watchtower writes and executes code at runtime to handle complex programmatic filtering over hundreds of accounts. - **Fan-out orchestration**: Divides macro-level questions by delegating account-level deep dives to individual per-account agents and aggregating the responses back up. - **Account executive agent fleets**: An account executive with 100 assigned accounts is effectively backed by a team of 100 persistent, dedicated agents feeding into Watchtower.

[09:17](https://www.youtube.com/watch?v=hm8NzEd5io0&t=557s) Cross-Account Memory and the Two-by-Two Matrix

The conversation opens with an exploration of how agents collect and process intelligence over time. When an account executive interacts with the system, the platform passively gathers details about individual user habits and broader organizational workflows. The host points to Actively's Watchtower product, asking how it synthesizes distributed memory across an entire fleet of agents—a background consolidation process likened to "dreaming," where the system combs through fragmented learnings to surface actionable strategic insights.

The memory architecture breaks down into two distinct layers. First, every customer account has a persistent agent that runs continuously throughout the account's entire lifecycle. This agent establishes an exhaustive historical log while extracting foundational concepts that can later be rolled up. Second, there are organizational concepts that emerge only when analyzing activity across multiple accounts—such as company-wide forecasting mechanics or internal operational norms. While cross-account insights may not be relevant to an individual sales rep managing a single prospect, they are vital for Watchtower's executive overview.

```mermaid flowchart TD accTitle: Two-by-Two Memory Architecture for Sales Agents accDescr: Flow showing how account-specific memory and cross-account memory feed into individual interactions and the Watchtower product.

subgraph AccountLevel["Account-Level Scope"] direction TB A1["Persistent Account Agent"] -->|"Tracks full lifecycle"| A2["Account Context & Deal History"] end

subgraph OrgLevel["Cross-Account Scope"] direction TB B1["Org-Wide Processes"] B2["Forecasting Conventions"] B3["Field Definitions & Preferences"] end

subgraph ProductSurfaces["Product Consumption"] direction TB W["Watchtower (Executive Overview)"] Rep["Account Executive Workflow"] end

A2 -->|"Informs directly"| Rep A2 -->|"Extracts roll-up concepts"| B1 B1 --> W B2 --> W B3 --> W ```

This model forms a conceptual two-by-two matrix spanning account-level versus cross-account memory on one axis, and user-level versus organization-level memory on the other. Capturing intelligence across all four quadrants allows the platform to enrich different layers of the product dynamically.

- **Account vs. Cross-Account:** Individual account agents maintain perpetual context for specific deals, while cross-account processes aggregate high-level operational trends across the entire pipeline. - **User vs. Org Level:** The system separates bespoke rep preferences from rigid organizational rules and forecasting models.

---

[10:38](https://www.youtube.com/watch?v=hm8NzEd5io0&t=638s) Sandboxing Code Snapshots and Pull Requests at Pendo

Todd turns to Pendo's implementation of sandboxing within Claude Managed Agents. In Pendo's architecture, the agents snapshot customer source code to detect tracking issues, recommend fixes, and propose code updates. Sandboxing is non-negotiable in this workflow; pulling third-party code into an execution environment exposes proprietary logic, secrets, and environment tokens. Securing this pipeline requires tight control over which tools can touch the codebase, what data payloads get passed into the model, and what terminal actions are authorized. The ultimate output of an agent interaction is often creating a pull request and pushing it directly to a customer's git repository.

Building and maintaining custom sandboxing infrastructure quickly became a distraction. While Pendo initially experimented with a homegrown sandbox, defending against every edge case and security vulnerability proved exhausting. Security and isolation were not their core value proposition—the sandbox simply needed to work flawlessly and earn immediate trust from both self-serve SMBs and rigorous enterprise security teams. Offloading this layer to Managed Agents allowed Pendo to reach enterprise-grade isolation without stalling product development.

Pendo relies on two distinct operational cadences for its agent sandboxes: - **Overnight Batch Sessions:** Scheduled cron jobs run during off-peak hours to crawl source code and analyze macro-level anomalies, such as sweeping product funnel changes or broken tracking logic across massive repositories. - **Interactive Real-Time Sessions:** Low-latency queries triggered when an end user asks a direct question. The agent inspects the code in real time, treating the repository as the authoritative ground truth to generate an accurate, verified answer.

---

[12:13](https://www.youtube.com/watch?v=hm8NzEd5io0&t=733s) The Build vs. Buy Calculus: Core Competency vs. Harnesses

The host frames the broader architectural debate: when should an engineering team roll its own agent harness versus adopting an off-the-shelf platform like Managed Agents? Wispr's founder argues that the decision hinges entirely on whether the execution harness itself represents the core competency of the product. When Wispr developed its "briefs" feature, the execution harness was merely an implementation detail. End users cared about the quality and relevance of the synthesized meeting briefs, not the custom plumbing underneath. Conversely, if Wispr were building a real-time, voice-driven Jarvis system, the proprietary harness might represent the foundational IP and capabilities of the company.

The calculation comes down to trading control against cost and speed: if an infrastructure component does not directly define the differentiated quality of your product, you should buy it. Todd reinforces this logic from the vantage point of an early-stage startup. In the initial phases of customer discovery and validation, the singular mandate is reaching product-market fit and delivering a 10x improvement over legacy solutions. Founders cannot afford to spend their time masquerading as infrastructure engineers when they need to iterate on user value.

- **Differentiating IP:** Custom harnesses are worth building only if the low-level orchestration mechanics create an unreplicable product experience. - **Future Re-evaluations:** Companies may eventually outgrow managed solutions if granular operating costs or bespoke performance profiles necessitate deep, fine-grained control, but building early is almost always premature optimization. - **Execution Speed:** Speed of validation trumps architectural perfection during early iteration loops.

---

[14:15](https://www.youtube.com/watch?v=hm8NzEd5io0&t=855s) From Prototypes to Scale: Days to Ship, Weeks to Roll Out

The founders compare notes on their rollout timelines. Todd recounts how Pendo previously ran an internal agent harness, but the cracks showed quickly under real customer loads. Rather than undertaking an extensive, painful rewrite to address lurking edge cases, Pendo ran a head-to-head bake-off against Claude Managed Agents. The managed implementation took only a few days to stand up in development, and within a few weeks, Pendo migrated its entire production workload over to the managed harness.

Wispr experienced a similarly dramatic acceleration when constructing its meeting assistant. The team shipped a functioning v1 prototype in a single day. Over the subsequent weeks, they expanded the underlying user base by 100x to 1,000x, spending just a few additional days tuning the infrastructure. The vast majority of their engineering energy was spent figuring out what users actually wanted through rapid cycles of internal dogfooding, external user testing, and prompt tweaking.

When iterating inside Managed Agents, the primary development surfaces involve refining system prompts, tuning custom tool definitions, and perfecting multi-step orchestration loops:

```mermaid flowchart LR accTitle: Wispr Orchestration and Information Retrieval Loop accDescr: Sequential flow showing how Wispr iterates over search, retrieval, ranking, and surfacing to produce meeting briefs.

S["Search"] --> R["Retrieval"] R --> RK["Ranking"] RK --> M["Multi-Source Orchestration"] M --> SF["Surfacing to User"] ```

- **Search and Retrieval:** Calibrating when the agent queries specific sources versus drawing on active context. - **Ranking and Pruning:** Determining which snippets of retrieved data are relevant before passing them into the reasoning prompt. - **Cross-Source Orchestration:** Aligning data pulled across heterogeneous workspaces into a single, cohesive brief.

---

[16:03](https://www.youtube.com/watch?v=hm8NzEd5io0&t=963s) Vibes-Based Iteration, Claude Code, and Prompt Caching

Actively saw a similarly rapid ramp-up with Watchtower: the feature went from an initial whiteboard concept to paying customers in just two weeks, with the team dogfooding working builds within the first seven days. By outsourcing harness mechanics like context window compaction, thread persistence, and prompt caching to Managed Agents, the team focused exclusively on the qualitative output: when an account executive walks into an upcoming meeting, what is the single most critical insight they must see? Using structured outcomes features allowed the team to enforce strict guarantees on information architecture while iterating rapidly on prompts.

This sparks an aside about the lifecycle of agent evaluation. The host and founders reference the early development of Claude Code, which initially launched without formal quantitative evals, relying instead on subjective, "vibes-based" iteration. The panel agrees that in the earliest stages of an agentic product, obsessive evaluation suites are counterproductive: "If the thing is not sticky, no amount of evaluation is going to make the thing sticky. You got to figure out how to make it sticky first." Once intuitive, vibes-based testing yields genuine user retention, teams must quickly construct rigorous evals to avoid breaking functionality during future iterations.

- **Managed Infrastructure Wins:** Critical cost-saving mechanics like prompt caching work out of the box without manual cache-key management. - **Context Compaction:** The managed harness automatically compacts context windows behind the scenes, freeing developers from token housekeeping. - **Vibes to Evals Progression:** Use qualitative, intuitive testing to discover an engaging, sticky user loop, then lock it down with quantitative evals to scale safely.

---

[17:49](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1069s) Managing Long-Lived Account Memory vs. Org-Level Context

The conversation circles back to the practical realities of production memory systems. Memory is notoriously tricky to implement cleanly without performance degradation or hallucinations. The Actively founder details why they maintain a hybrid model between bespoke internal storage and Claude Managed Agents' native memory features.

For core account-level intelligence, Actively retains proprietary control. Because an account agent persists indefinitely across a multi-year sales lifecycle, the context accumulated on an account is deep, highly detailed, and prone to corruption if left unpruned. To keep this perpetual record queryable and performant, Actively builds custom indexing layers and validation checks around account history. In contrast, they leverage Managed Agents' built-in memory capabilities for the fluid, shared context that surrounds the organization: mapping what specific CRM fields mean, internal business conventions, and individual rep working preferences.

[18:43](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1123s) Managed Memory and the Multi-Agent "Watchtower" Architecture

For software companies building AI tools, commoditized backend plumbing needs to function reliably without requiring bespoke reinvention. As one founder explains, their core mission is fundamentally to "make tokens useful for salespeople," which depends on user experience, product surface area, and thoughtful workflows rather than maintaining infrastructure. Claude Managed Agents allowed them to stand up a production system in just two weeks, and several months later, they have felt zero need to migrate off the platform because offloading managed memory freed their team to concentrate on features that drive immediate customer value.

The technical setup relies on a two-tier agent topology. At the base layer, individual account agents run outside of Claude Managed Agents on proprietary infrastructure, each maintaining independent memory for a specific customer account. Above this fleet sits a supervisory Claude Managed Agent known as the "Watchtower." If an account executive (AE) managing 100 accounts asks, *"What are the five accounts I need to work on today?"*, Watchtower dynamically synthesizes two separate inputs: it writes code on the fly to fan out queries against the account-level agent memories, and it combines those findings with higher-level user preferences and organizational playbooks stored directly in Claude's managed memory. This architecture naturally scales from individual ICs up to sales leadership; a sales manager overseeing five AEs effectively monitors 500 accounts through the exact same system.

```mermaid flowchart TD accTitle: Watchtower Cross-Account Query Flow accDescr: An AE queries the Watchtower agent, which combines organizational memory with account memories via fan-out code execution.

subgraph UserLayer ["User Interface"] AE["Account Executive / Sales Leader"] end

subgraph WatchtowerSystem ["Claude Managed Agent: Watchtower"] WT["Watchtower Agent"] MM[("Managed Memory\n(Org Priorities & AE Preferences)")] end

subgraph AccountFleet ["Independent Account Agents"] direction LR A1[("Account 1 Memory")] A2[("Account 2 Memory")] AN[("Account N Memory")] end

- **Speed to delivery**: The entire Watchtower cross-account agent was built and operational within two weeks. - **Hierarchical scaling**: An individual AE monitors roughly 100 accounts, while a manager overseeing five reps uses the Watchtower to monitor 500 accounts simultaneously. - **Dynamic code execution**: The Watchtower handles cross-fleet query orchestration by programmatically writing code to execute fan-out calls across separate account storage layers.

---

[21:43](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1303s) Codebase Sandboxing, Continuous Instrumentation, and Self-Healing Software

Todd describes how Pendo uses sandboxed managed agents to inspect customer source code side by side with live application telemetry. After attempting to build an internal harness across multiple engineering teams, Pendo adopted Claude Managed Agents and went live in under two weeks. By offloading the underlying sandbox execution and container management, the engineering team only needed minor tuning to align the setup with their internal operational controls.

Operating with direct source repository access, the platform orchestrates continuous workflows across the entire development lifecycle: 1. **Pull Request UX Reviews**: Whenever an engineer opens a pull request, a sandboxed agent reviews code diffs and applies a lightweight UX critique to intercept poor interface decisions before they reach production. 2. **Automated Analytics Instrumentation**: Once code merges, the agent assesses the changes and autonomously submits a follow-up pull request injecting missing telemetry tracking into the codebase. 3. **Overnight Degradation Analysis**: Background jobs monitor production metrics, identifying conversion funnel dips or gradual month-over-month telemetry degradations. If a metric declines, the agent inspects merged commits from up to four weeks prior and drafts a remediation patch. 4. **Agentic Conversation Analysis**: The system monitors real-time user-agent conversations, flags unexpected error spikes, and suggests runtime prompt modifications.

Todd summarizes the ultimate architectural objective as creating "a self-healing piece of software." Rather than relying purely on pre-deployment checks, the platform observes how real users navigate interfaces, connects that telemetry directly to source code changes, and proposes automated fixes to help development teams ship continuously without friction.

---

[24:22](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1462s) The Progression of Evals: From Vibes to Live Production Traces

The founders debate the realities of evaluating agentic systems, contrasting early-stage iteration with production rigor. Initial prototyping always starts at "Phase zero," which is purely vibes-based. While quick internal dogfooding helps get an initial product off the ground, relying solely on internal company testers quickly leads to overfitting on an unrepresentative sample. A curated eval set that looks great internally can completely miss the real distribution of actual users, leaving teams hill-climbing on metrics that customers do not care about.

```mermaid flowchart LR accTitle: Evolution of Agent Evaluation Strategy accDescr: Four-phase progression of agent evals from informal vibes to segmented production test suites.

P0["Phase 0: Vibes-Based\n(Internal Dogfooding)"] P1["Production Telemetry\n(Thumbs Up / Down Signals)"] P2["Real Trace Extraction\n(Customer Interviews & Queries)"] P3["Segmented Offline Suites\n(Regressions, Tasks & Rubrics)"]

P0 --> P1 P1 --> P2 P2 --> P3 ```

To combat abstraction drift, one founder describes digging directly into customer queries and interviewing the users to learn what they actually intended to accomplish before writing a test case. True evaluation requires capturing broad production signals—such as explicit thumbs-up and thumbs-down ratings—and organizing them into distinct test cohorts: - **Regression suites** to catch breakages on previously solved user tasks. - **Edge-case partitions** to stress-test complex reasoning and corner cases. - **Task-specific cohorts** tailored to specific actions assigned to the agent. - **Unified rubrics** designed to align offline evaluation scoring with real-world user outcomes.

---

[26:41](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1601s) The Stateful Eval Trap and Preparing for Model Migrations

Evaluating agents that interface with mutable state remains an unsolved challenge. When an agent possesses live memory or integrates with third-party tools via the Model Context Protocol (MCP)—such as reading from a live Slack workspace—running deterministic offline evals becomes fragile. Because external state drifts constantly outside the system's control, test assertions inevitably fail unless engineering teams undertake the massive effort of mocking out entire third-party ecosystems. One founder laughs off the concept of fully mocking an enterprise workspace, noting that *"Mocking a customer's Slack I'm not getting in that territory. That's not too tractable."* While trajectory evals can score whether an agent interacts with its memory correctly, verifying exact end-state results on stateful operations remains difficult across the industry.

When handling major foundational model migrations (such as upgrading when an Opus-class model drops), the founders caution against hyper-optimizing prompts to the idiosyncrasies of one specific model family. An engineering team needs an established eval suite to benchmark the new model immediately, but they must also watch out for new "gotchas." Different models fail in entirely different ways; for example, newer model releases sometimes exhibit distinct stylistic shifts, introducing unmistakable "AI telltale signs" and unnatural prose quirks that require re-tuning prompt guardrails.

[28:13](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1693s) Model Failure Modes and Prompt Optimization

The discussion opens on the quirks and stylistic traps of newer model families. Rather than fussing over minor prompt tweaks to squeeze out an incremental gain, the founders emphasize the necessity of guarding against distinct regressions and bad stylistic habits—such as an overabundance of em dashes or convoluted, unreadable sentence structures. As models evolve, chasing prompt tweaks to extract the "last 1%" from a specific model generation matters far less than discovering and guarding against known, idiosyncratic failure modes. New model releases inevitably introduce new failure behaviors that teams must actively hedge against, whereas raw model intelligence will naturally improve with subsequent generations.

- Teams prioritize understanding model-specific failure modes over micro-optimizing prompts for diminishing returns. - Model updates frequently introduce stylistic quirks (such as excessive em dashes and unnatural sentence structures) that require targeted guardrails.

---

[28:48](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1728s) The Realities of Cost, Control, and Attribution

Cost remains top of mind for any team building production agents. As one founder remarks, "nobody working on agents right now is not worried about cost to some degree," noting that disregarding cost signals an unsustainable business model. The strategic question founders repeatedly ask is when the tradeoff between cost and engineering control justifies moving off Claude Managed Agents to build an in-house harness. While the infrastructure markup on Managed Agents is low relative to the CPU overhead provided, the platform currently lacks native abstractions like batch processing. Running scheduled tasks 24 hours in advance via batch mode could yield instant 50% to 75% cost reductions. For now, the pragmatic approach is to stay mindful and spend well while delivering rapid product value, deferring hyper-optimization until the platform either rolls out these capabilities or the product hits massive scale.

Attribution is an equally acute pain point. Founders must pinpoint exactly which sub-agent or call within the harness is driving spend. During massive evening batch runs where large volumes of customer data are processed, costs can spike quickly. Teams can implement quick engineering fixes—such as running jobs less frequently or trimming the payload size—without re-architecting the system. However, as agentic workflows mature, teams require granular telemetry to track attribution across every node in the execution tree.

- Managed Agents offers a low infrastructure markup, but the absence of native batch mode prevents teams from capturing estimated 50%–75% cost savings on non-urgent jobs. - Fine-grained telemetry is critical to trace which specific agent invocations and loop iterations accumulate budget during high-volume batch workloads. - Early scaling relies on lightweight interventions (pruning payload sizes and reducing job frequency) rather than full harness rewrites.

---

[31:13](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1873s) Where Anthropic Can Improve: Cost Levers and Latency UX

When asked what Anthropic should improve across Claude and Managed Agents, the founders pinpoint two main areas: cost levers and latency user experience. Frontier models like Opus excel when deployed on tasks that demand peak intelligence and are relatively price-insensitive. However, products often follow a pattern where a central frontier agent must orchestrate or fan out across hundreds of parallel workers. In Actively's Watchtower feature, for instance, a lead agent coordinates discovery across 500 separate customer accounts; deploying frontier intelligence across every individual account check would be prohibitively expensive. In its current form, Claude Managed Agents can feel like an "all or nothing" proposition, lacking the dials needed to swap in cheaper models or run background tasks cheaply.

```mermaid flowchart TD accTitle: Tiered Multi-Agent Architecture accDescr: High-intelligence central orchestrator fanning out to lightweight workers across accounts.

subgraph Orchestration ["Frontier Orchestration Layer"] A["Main Coordinating Agent<br/>(Claude Opus / Frontier Intelligence)"] end

subgraph FanOut ["Distributed Execution Layer"] B["Account Worker 1<br/>(Cheaper, Low-Cost Model)"] C["Account Worker 2<br/>(Cheaper, Low-Cost Model)"] D["Account Worker 500...<br/>(Cheaper, Low-Cost Model)"] end

A -->|"Coordinates query"| B A -->|"Coordinates query"| C A -->|"Coordinates query"| D ```

To bridge this gap, founders want more flexible cost levers: configurable effort settings, multi-agent cost tiers, and asynchronous flex/batch modes that could cut expenses in half. Beyond pricing, latency remains a hurdle for interactive products. When agents take minutes on deep reasoning tasks, users tolerate variable latency; but for responsive, user-facing features, teams want lower-level levers like pre-warming sandboxes to eradicate cold-start delays.

- **High-intelligence vs. high-fanout:** Orchestrators require frontier reasoning, while fan-out jobs (such as scanning 500 accounts) need lightweight, low-cost models. - **Desired cost levers:** Anthropic should surface multi-tier model selection within harnesses, compute-effort sliders, and asynchronous batch/flex processing. - **Latency UX:** Teams building responsive tools need platform controls to pre-warm execution sandboxes and eliminate cold-start overhead.

---

[33:33](https://www.youtube.com/watch?v=hm8NzEd5io0&t=2013s) When to Build In-House vs. Leverage Managed Agents

The panel wraps up by distilling the exact framework for choosing between a managed platform and rolling a custom agent harness. If building a feature viability test or a non-standard interface—like an always-on, voice-controlled Jarvis "chief of staff"—demands a 90% cost drop or ultra-low latency, building an in-house harness becomes necessary. Outside those edge cases, managed agents excel in two primary scenarios: long-running tasks that demand frontier reasoning where teams are price-insensitive, and experimental validation where time-to-learn outweighs runtime infrastructure cost.

Mihir and the founders outline four key criteria that dictate when to rely on Claude Managed Agents: 1. **Frontier intelligence requirements:** Tasks where peak cognitive capability is strictly required. 2. **Price insensitivity:** Workflows where the underlying business value dwarfs raw model API costs. 3. **Tolerance for longer runtimes:** Complex jobs that legitimately take several minutes to reason through. 4. **Validating viability:** Fast-moving prototyping where the true operational cost of an experiment is measured in engineering weeks, making managed infrastructure the fastest path to production.

## Notable quotes > [05:16](https://www.youtube.com/watch?v=hm8NzEd5io0&t=316s) Help when you can and do not cause harm or annoyance. — Sahaj

> [06:52](https://www.youtube.com/watch?v=hm8NzEd5io0&t=412s) 30 different fields that mean the same thing that were deprecated — Mihir

> [16:03](https://www.youtube.com/watch?v=hm8NzEd5io0&t=963s) If the thing is not sticky, no amount of evaluation is going to make the thing sticky. You got to figure out how to make it sticky first.

> [18:43](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1123s) make tokens useful for salespeople — Mihir

> [18:43](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1123s) What are the five accounts I need to work on today?

> [21:43](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1303s) a self-healing piece of software. — Todd

> [26:41](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1601s) Mocking a customer's Slack I'm not getting in that territory. That's not too tractable.

> [28:48](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1728s) nobody working on agents right now is not worried about cost to some degree

> [31:13](https://www.youtube.com/watch?v=hm8NzEd5io0&t=1873s) all or nothing

## Mentioned Claude, Claude Managed Agents, Anthropic, Wispr, Sahaj, Actively, Mihir, Watchtower, Pendo, Todd, Salesforce, LinkedIn, Claude Code, Model Context Protocol (MCP), Slack, Claude Opus

## Video description We sat down with three founders to chat about what they learned building and scaling agents with Claude Managed Agents. Wispr shipped the first version of their meeting assistant in a day and scaled it in a few weeks. Actively built a cross-account sales agent in two weeks. Pendo’s analytics platform now reads customers' codebases and suggests fixes.

They get into verifying agent work with outcomes, running memory in production, sandboxing, evals, cost, and deciding what to build yourself.

Learn more about Claude Managed Agents: https://platform.claude.com/docs/en/managed-agents/overview

[Source](https://www.youtube.com/watch?v=hm8NzEd5io0)

中文简报

初创公司舍弃控制权换取Claude Agent落地速度，并寻求成本调控杠杆创始人指出，自研Agent基础设施会延误早期产品验证，但要扩展高扇出工作流，托管平台必须提供批量定价和分层模型路由以保持可行性。 Wispr、Actively和Pendo的创始人分享了采用Claude托管Agent如何让他们在数天至数周（而非数月）内上线复杂功能，例如会前简报、跨账户销售聚合以及自愈式遥测PR。他们详细阐述了核心架构模式，包括在干净的上下文窗口中将工作Agent与基于规则的验证器解耦、沙箱化动态代码执行，以及在个人账户与组织工作流之间管理长期记忆。创始人们梳理了从早期“凭直觉”迭代到基于真实客户追踪日志构建回归测试集的评估历程，强调了评估可变、有状态工具的难度。最后，他们权衡了托管平台与自研编排框架的经济性，呼吁平台提供更细粒度的成本调控手段、批量定价层级以及沙箱预热以降低延迟。 Wispr、Actively和Pendo等初创公司通过采用Claude托管Agent而非自研专有编排框架，在短短一天到两周内就上线了生产级Agent功能。 (0:07) Wispr利用托管Agent的Outcomes功能与隔离的上下文窗口来维护用户信任，依据客观规则验证简报，若候选文本未通过验证则完全拦截输出。 (2:14) Actively采用两层架构，由主管级的“Watchtower”Agent动态编写代码，将分析查询扇出分发给500个专用的单账户Agent，以汇总高管层销售管道预测。 (18:43) 创始人指出成本归因问题以及缺乏原生异步批处理是主要痛点，并提到计划批处理执行可使非紧急后台Agent成本降低50%至75%。 (28:48)

Reading summary

Founders from Wispr, Actively, and Pendo state that adopting Claude Managed Agents allowed them to launch production products in days by offloading execution harnesses, sandboxing, and context management. However, sustaining these systems requires decoupled verification to prevent hallucinations, hybrid memory to navigate messy enterprise data, and new platform cost levers for high-fanout workloads. Deploying Domain Agents Across Complex Workflows (0:07) Wispr, Actively, and Pendo use Managed Agents to address continuous business workflows rather than single-turn chat tasks. Sahaj explains that Wispr expands meeting assistance beyond live transcription into a three-stage lifecycle, running 24-hour pre-meeting preparation briefs and automated post-meeting task execution. Todd describes Pendo's analytics platform, which places a sandboxed agent in front of customer codebases to correlate runtime telemetry with application code, automatically opening pull requests to fix instrumentation drift. Actively applies this approach to sales intelligence through "Watchtower," an executive engine that aggregates deal data across hundreds of accounts. Mihir reports that Actively built and shipped Watchtower in two weeks, using it to automate daily account prioritization for sales representatives and cross-account pipeline forecasting for managers. Decoupled Verification and Graceful Output Suppression (2:14) Wispr relies on Managed Agents' "Outcomes" feature to evaluate generated briefs against objective rubrics before surfacing them to users. Sahaj emphasizes that false positives—such as conflating two contacts with the same name—destroy user trust. To ensure quality, Wispr isolates the generation agent from the verifier agent in separate context windows. This separation prevents the verifier from inheriting the worker's internal reasoning bias when checking candidate text against primary sources like calendar invites and LinkedIn profiles. The evaluation rubric balances factual accuracy against readability, rejecting multi-page drafts in favor of a concise sequence outlining attendee identity, meeting purpose, and expected outcomes. When a brief fails to meet the rubric, Wispr silently suppresses it, operating on the principle that delivering no context is far better than providing inaccurate information. Hierarchical Orchestration and Two-Tier Memory Systems (6:52) Enterprise environments frequently present fragmented schemas. Mihir notes that corporate Salesforce setups routinely contain dozens of deprecated, redundant fields that break standard prompts. Instead of requiring manual administrative onboarding, Actively dynamically learns company-specific definitions through sales team interactions and stores them in an organization-wide memory store that persists across all users. To scale cross-account reasoning, Actively uses a two-tier architecture. Proprietary per-account agents maintain granular deal histories outside Managed Agents, while a supervisory Watchtower agent runs on Claude Managed Agents. When an account executive requests priorities across 100 accounts—or a manager reviews 500—Watchtower writes and executes dynamic code to fan out programmatic queries across the account fleet, combining account-level data with organizational rules stored in Claude's managed memory. The Build-Versus-Buy Infrastructure Calculus (12:13) The founders argue that engineering teams should only build custom harnesses if orchestration mechanics represent their core intellectual property. Todd notes that Pendo initially experimented with an in-house sandbox, but securing customer source code, handling secrets, and defending against execution vulnerabilities quickly drained engineering capacity. Transitioning to Managed Agents delivered enterprise-grade container isolation within days. Wispr similarly built a functioning meeting brief prototype in one day, scaling its user base by up to 1,000-fold over subsequent weeks with minimal infrastructure changes. The panel agrees that early product iteration must prioritize rapid qualitative testing to achieve user retention before teams invest in formal evaluation suites, prompt fine-tuning, or proprietary infrastructure. Cost Levers, Stateful Evals, and Fleet Scaling (28:48) Despite operational gains, founders note distinct infrastructure limitations around cost and evaluation. Mihir points out that deploying frontier models across high-fanout tasks—such as scanning 500 accounts simultaneously—is economically unsustainable. The founders urge Anthropic to introduce multi-tier model execution within harnesses so a frontier orchestrator can delegate tasks to cheaper workers, alongside pre-warmed sandboxes to eliminate user-facing latency. Cost attribution and scheduling remain additional challenges. Without native batch processing in Managed Agents, non-urgent workloads like 24-hour pre-meeting briefs cannot capture estimated 50% to 75% batch discounts. Furthermore, evaluating agents that interact with live, mutable third-party environments like customer Slack workspaces remains technically fragile, forcing teams to rely on trajectory evaluations and customer trace reviews rather than deterministic offline tests. While Claude Managed Agents accelerates time-to-market by offloading backend harness complexity, scaling agentic fleets exposes unresolved trade-offs between frontier reasoning and cost control. As agent architectures expand from rapid validation to high-volume production, platform success depends on introducing granular cost attribution, multi-tier execution, and batch processing for asynchronous enterprise workloads.

Source quotes

[5:16] Sahaj: Help when you can and do not cause harm or annoyance. [6:52] Mihir: 30 different fields that mean the same thing that were deprecated [16:03] If the thing is not sticky, no amount of evaluation is going to make the thing sticky. You got to figure out how to make it sticky first. [18:43] Mihir: make tokens useful for salespeople [18:43] What are the five accounts I need to work on today? [21:43] Todd: a self-healing piece of software. [26:41] Mocking a customer's Slack I'm not getting in that territory. That's not too tractable. [28:48] nobody working on agents right now is not worried about cost to some degree [31:13] all or nothing

Original transcript

[0:07] All you guys are using [0:08] Managed Agents in different ways. [0:10] Maybe, Sahaj, I'd love to hear about how you're using Managed Agents [0:12] in your new meeting product. [0:14] Absolutely. [0:15] For us, we think about meetings in terms of the full journey of the meetings. [0:19] There's what happens before it, during it, and after it. [0:21] Almost everybody confines their thinking to literally just the period during it. [0:25] But the place where agents become the most interesting [0:27] are thinking about how do we help you prepare [0:29] for the conversation that you're about to have? [0:31] How do you make sure what you actually want to get done [0:33] happens in that conversation? [0:34] Then how do you execute on the things that are supposed to happen after? [0:38] We've been using Managed Agents to build and design a lot of new features [0:41] that we're building, both on the briefs side [0:43] of getting you prepared, [0:44] and on the tasks and task automation side of handling your work after for you. [0:49] Then, Mihir, how about yourself? [0:52] Our product is agents for sales teams, and the core of the product is this idea [0:56] of an agent on every account that's always working [0:58] and building memory and figuring out what to do to progress it. [1:00] But there are a lot of workloads for sales teams [1:02] that are not just restricted to one account. [1:05] If I'm a sales rep, it's like, what are the top five accounts [1:07] I should go after today? [1:08] Or if I'm a manager, it's like, what deals should I forecast for the next quarter? [1:12] So on. [1:13] We wanted to build, we call it Watchtower. [1:14] We wanted to build this product that's cross-account reporting [1:17] and aggregation and roll-ups. [1:19] That agent is fairly different than the shape of our per-account agent. [1:23] It needs a bunch of different tools and features and so on. [1:26] We spun that up in Managed Agents in 2 weeks. [1:28] That is now one of our most used features [1:30] and also powers a lot of other parts of the product. [1:33] Then Todd, rounding it out. [1:35] We have a product analytics solution that integrates into applications [1:38] and captures what users are actually doing in it [1:40] so you can feed back data on how to make it better. [1:43] As people have moved faster and faster, as we can imagine, [1:46] it gets harder and harder to instrument those applications and understanding [1:49] where that analytics drifts from what's going on. [1:52] We built a new agent which sits in front of our platform. [1:55] What it does is it integrates the codebase [1:57] and connects what the codebase is doing to what's actually happening in reality. [2:01] We're using Managed Agents to actually snapshot a version of the codebase, [2:06] compare it to what's actually happening in the application, [2:09] start proactively sending insights and even fixes to the application. [2:13] Nice. [2:14] I'd love to hear more about the details and the features you're actually using. [2:18] We were talking a little bit earlier [2:19] about outcomes being really important to you. [2:22] Tell me a little bit about how you think about outcomes, [2:24] and why it's such an important feature in Managed Agents. [2:27] Let's talk about briefs as a feature for what we're doing. [2:29] Briefs are meant to surface to you who is the person you're about to talk to? [2:33] Why are they relevant? What did you agree to talk about? [2:36] What's important to get out of the meeting, for example, [2:38] pre-context from what you might have forgotten from weeks ago. [2:41] Briefs are awesome when they're correct. [2:43] If they actually tell you who you're talking to correctly. [2:46] If you're talking to a person named Tom, [2:47] and it gives you information about the wrong Tom, [2:49] it's actually almost destructive because the moment you end up [2:52] in the meeting, you click to learn more about them, and you end up confused. [2:57] Being able to correctly identify whether the agent produced the right outcome [3:01] and literally choose not to show the user anything at all, [3:05] if it didn't, is way better than giving the user false positive information. [3:10] Outcomes is super helpful because we can literally just take the agent, [3:14] and have a complete independent assessor of whether the agent actually produced [3:18] the right outcome in that case in question, [3:20] and then make sure the attribution of sources is actually correct. [3:24] This means that when we show things to our users, [3:27] they're correct all the time. [3:29] It also helps drive feedback loops. [3:30] The more you can assess that on the fly, [3:32] the more you can bake into the initial agent loop, [3:35] making sure it gets it right from the get-go. [3:37] But it guarantees, essentially, good product experience. [3:40] Maybe specify a little bit. [3:41] An outcome is basically supplying a rubric to the agent [3:44] and allowing the agent to iterate until the rubric is satisfied. [3:47] In this case, how do you structure and set up the rubric? [3:51] For the rubric, we're looking at all of the different things [3:53] that are important to the end user to determine whether or not [3:58] the pre-read was effective. [4:00] This depends a lot on the type. [4:03] For example, there are aspects of it where it's, did you literally just pull [4:07] the correct LinkedIn for the right person? [4:10] That's a very objectively verifiable thing that you can actually assess against. [4:15] There are aspects where it is about the way it's presented to the user. [4:19] That information ought to be presented to you [4:21] in a way that's easily scannable and readable. [4:24] The rubric will also assess for things like that, [4:26] where you're actually making sure it's consumable. [4:29] Otherwise, you get a five-page pre-read before a meeting, [4:31] nobody is going to look at that. [4:32] Even if it's structured in the wrong order [4:37] of it doesn't first tell you who the person is, [4:39] then why you're meeting them, then what to do, even then it's hard. [4:43] We'll look at each of the things that, from a UX perspective, [4:46] we want the user to experience, turn that into part of the rubric, [4:50] and then hill climb against that rubric. [4:52] That makes a lot of sense. [4:53] One clarification, though, is basically this is at runtime. [4:58] Basically, if I have a meeting, I guess, [5:00] say it's in an hour, your agent might kick off [5:02] some period of time prior to that. [5:04] Spin up, create this, for example, I guess it would be a brief, [5:09] and that is graded against the outcomes iteratively [5:13] in some time period before the meeting. [5:15] -Is that right? -Yeah. [5:16] But then how do you inject ground truth? [5:17] You said you can verify the LinkedIn, but you have to go fetch that. [5:21] How are you actually doing verification when external information is being met? [5:25] Absolutely. [5:27] We run these, for example, 24 hours before the conversation. [5:29] It gives you a pretty long time window over which things can be assessed. [5:33] Cool. [5:34] In terms of verification here, I think the interesting thing [5:37] is agents that do the work and agents that evaluate the work [5:40] tend to just have decoupled assessments of whether things are correct or not. [5:44] Even an agent that is doing independent work can look at the LinkedIn [5:47] and can look at the email and can look at the calendar invite, [5:50] and make its own judgment of, was this probably correct or not? [5:56] Even I, when I show up to cold email, cold meeting, I can look up the LinkedIn [6:00] and be like, "This does not look like the right person. [6:02] I'm not sure why, but it doesn't look right." [6:05] This is the value of having something independent from the trace [6:09] doing the work, evaluating the work. [6:12] With Managed Agents, we actually use an independent context window, [6:14] independent verifier agent, and that's baked into outcomes. [6:16] That's the thing that's helpful about it. [6:17] Otherwise, it would just be part of the agent loop itself. [6:20] That's good, and it's going to be part of it. [6:22] But the fully independent clean context window [6:24] means that you can assess the quality of the work as part of deploying it. [6:30] For us, we actually just use it more simply as we don't necessarily [6:33] have to iterate until it's perfect. [6:36] You can just choose not to tell people something when you don't know [6:39] if your information is correct. [6:40] That's actually fine in our situation [6:43] when you're building something that's proactively helpful. [6:47] Help when you can and do not cause harm or annoyance. [6:51] That's really cool. I want to ask you about memory. [6:54] We were talking earlier about your Watchtower product [6:56] and how it's using memory. [6:58] I'm really interested in this topic. [7:00] Maybe tell me a little bit about memory and dreaming, [7:02] and how you're thinking about these things. [7:04] One of the things that's hard about this problem is there are so many concepts [7:08] within a company that are specific to that company. [7:10] For example, how a sales team defines ownership or how they do their forecasting [7:14] or whatever, how their deals work, et cetera. [7:18] That's especially hard because a lot of the source data is pretty messy. [7:22] Everybody's Salesforce has 30 different fields [7:25] that mean the same thing that were deprecated, whatever. [7:28] The idea is if someone asks a question, if I ask a question [7:31] about what are the deals I should forecast for next quarter? [7:34] That's a pretty, A, an important question to be able to answer correctly. [7:38] There's a lot that goes into that. [7:40] The way that we figure that out is, one, as people interact with the product, [7:45] we learn those concepts and then store them in org-wide memory. [7:49] That then enables us to give better and better results for everyone else. [7:53] That's one of the things that saves us, [7:55] A, from having to go in, and do a bunch of config at the beginning. [7:58] We don't have to do that, but then the system can just very quickly [8:01] pick up those concepts and get interaction. [8:03] We'll do some things where when you first ask a question, [8:06] we'll ask user question and clarify and then save those. [8:10] That's one thing. [8:11] Then the second thing that we do, not just memory, but the second thing [8:16] we do that I think Managed Agents [8:17] has been pretty helpful for is some of these queries [8:21] or use cases are pretty complicated [8:23] and require pretty deep work across a bunch of accounts. [8:26] If I have 500 deals and I want to do some deeper work, [8:29] I can do some basic filtering, but maybe the system wants to go deeper. [8:34] It'll actually write code to be able to answer the question, [8:36] and it can even do programmatic tool calling. [8:38] It can go and narrow down a candidate set and then fan out more work [8:43] to independent agents that work on every single account, [8:46] and roll up feedback back. [8:48] It's both the concept of learning all of these org-wide business logic [8:53] in a way that generalizes across every team is different. [8:56] Then also being able to write code to be able to handle [8:59] arbitrarily really complex queries [9:01] across relatively large sets of accounts and things like that. [9:05] You were telling me earlier about the setup of your product. [9:07] It's basically, I'm an account executive and I have some number of accounts. [9:11] Let's say it's 100, you spin up an agent for each account. [9:14] That's right. [9:15] Basically, a team of agents that's servicing me [9:17] as an account executive or salesperson. [9:21] As I'm interacting, like you said, you're learning memory. [9:24] You're learning things about maybe my preferences or org-wide preferences. [9:27] But then this new product is you have this memory [9:31] collected across this team of agents, and it seems like it's doing "dreaming" [9:38] where it's looking across all those memories and suggesting things. [9:41] Tell me about that specifically and what it's doing. [9:44] There are two parts of the memory that are interesting. [9:47] One is on each account, there's an agent that runs forever [9:50] because it follows the account throughout the entire life cycle. [9:54] It both builds context on the account and then learns, picks up [9:57] these other concepts that can get rolled up. [9:59] That's the first part of it. [10:00] The second part is even while doing cross-account work, [10:03] you will learn about these cross-account concepts. [10:05] How forecasting works, or how other processes within an org work. [10:10] That's a second class of memory that's less relevant [10:13] maybe on the individual account, [10:14] but it's more relevant within the context of this Watchtower product. [10:18] We use both. [10:20] On every level, you can think about almost this two-by-two where it's like [10:24] some things you learn, you have memory on an account [10:26] and across accounts, you have memory on a user level and an org level. [10:29] We learn all of those things in different ways and then use them [10:34] to enhance all different parts of the product. [10:38] I want to talk to you a little bit more about that, [10:39] but I want to talk to Todd first. [10:41] Todd, we're talking about sandboxing, and how are you using it? [10:45] This is a really cool use case for sandboxing. [10:48] As I shared in the description of what we're doing, [10:50] we're actually snapshotting people's source code. [10:53] In general, that should be sandboxed. [10:54] As you can imagine, if people are interacting with an agent [10:57] and you get access to someone's source code, [10:59] you can potentially get secrets. [11:02] There's just a lot of interesting things. [11:04] Then you also have to be very thoughtful about what tools are exposed [11:07] that's interacting with that source code as well, [11:10] what data is sent to it? [11:11] Then what actions happening. [11:13] At the end, often the result of an interaction [11:16] is essentially generating a pull request, pushing it to the actual repository. [11:23] I think that was the piece where we'd spend some time rolling our own, [11:29] but you're just covering every single edge case, [11:31] and doing all that work, [11:32] it's not core to our value prop. It just needs to work. [11:35] It needs to be safe and secure. [11:37] It needs to make sure, frankly, everyone from small businesses [11:40] to large enterprises will trust it. [11:42] I think we're able to get there pretty quickly with Managed Agents. [11:46] We also have a diversity of sessions. [11:49] We have overnight batch systems which are looking for issues at night. [11:54] If you're trying to look at large-scale funnel changes or other things, [11:57] you don't want to do that in real time. [11:58] You want to do that in a cron process. [12:01] Then we also have sessions where if someone's asking a question, [12:04] we may want to quickly look at the source code [12:06] to understand the source of truth [12:08] to then answer the question more accurately. [12:10] All those are happening in Managed Agents right now. [12:12] Cool. [12:13] Essentially, we have three different examples of agents, [12:16] Managed Agents with sandboxing, Managed Agents with outcomes, [12:19] Managed Agents with memory. [12:20] How do you think about the decision to roll your own harness [12:24] versus use a managed solution like Managed Agents? [12:26] I think all of it comes back to whether that's part of the core competency [12:30] of the feature that you're building. [12:31] For example, when we're building new features like briefs, [12:35] that's something where the harness itself is not a core competency of that feature. [12:40] The thing that really matters is, [12:41] are we delivering the end user the value that they need? [12:44] Whereas for other types of things, for example, if we were thinking [12:47] about building a fully voice-controlled Jarvis system, [12:50] there the harness itself might be the core competency of what we build, [12:54] and it might allow us to deliver completely new capabilities. [12:57] It's a question of basically control versus cost and time. [13:04] It really comes back to if you don't need to roll your own infrastructure for it [13:08] because it's not the thing that defines [13:10] the quality of what you're building, choose to buy. [13:15] In the case where it is, build it yourself. [13:17] Todd, you mentioned a similar point about when you're in this regime [13:21] of rapid experimentation and exploration, [13:23] it's like speed is where it really matters. [13:24] In the early days of any product, [13:26] you're trying to validate product-market fit. [13:27] You're trying to validate certain core features that make your product special. [13:31] All of us are solving some hard problem [13:33] and trying to solve it in a 10X better way than traditional solutions. [13:37] You don't need to roll your own infrastructure to solve those problems. [13:40] None of us in this roundtable, we're not infrastructure folks. [13:45] There may be one day when the problem we're solving necessitates building [13:48] our own infrastructure because we need a little more fine-grained control, [13:51] maybe cost in a certain area, we can't get to where we want it [13:55] from an operating perspective. [13:57] But in the early days, you want to innovate. [13:58] You want to get features in front of customers fast. [14:01] You want to be able to change really, really quickly. [14:03] For that, this is a perfect solution to make sure [14:06] that you're focusing on the right things for your team. [14:08] It's not for everyone, [14:10] but I think if you're in the startup phase, absolutely, [14:13] it's the right way of thinking about and prioritizing your time. [14:15] You mentioned for you it was like 2 weeks from experimentation with this [14:19] to shipping something. [14:21] Look, we had a solution, obviously running. [14:23] That was our own homegrown solution for a while, [14:25] but we started running into issues with it. [14:27] We all know shipping something is easy. [14:31] Once you start seeing a set of issues, we started to understand, [14:33] we have to rebuild this whole thing because it's not where we want it. [14:38] Then we baked it off against, let's just use Managed Agents. [14:42] It was so fast to get up and running, and it covered a lot of the edge cases [14:45] that we were hoping to cover ourselves. [14:48] Just made a lot of sense. [14:49] Within a few days, we had working infrastructure. [14:52] Within a few weeks, we had the whole product rolled over to it. [14:55] -That's pretty compelling. -How about for you? [14:57] What was the time frame for you ramping up on Managed Agents [15:01] for the meeting assistant in particular? [15:03] It allows us to build a first version of the feature in a day [15:05] and then ramp up with 100 to 1,000X scale-up in the user base [15:10] over the course of a few weeks within, [15:13] again, a couple more days of iteration on that part of the system. [15:16] The primary thing that required work [15:17] was figuring out exactly what a user wants out of the feature, [15:20] and that required a bunch of cycles of build it, dogfood it, [15:25] test it with other people, and then iterate that. [15:27] But even that could be done over the course of 2 or 3 weeks. [15:30] During the iteration phase, what surfaces did you find you're modifying a lot? [15:33] Was it the prompts, the skills, tools? [15:35] All those. [15:37] Then the orchestration of the loop really is, for example, [15:41] when are you doing search, then retrieval of different things, [15:46] then ranking of the relevant parts of it, then what do you surface back to the user, [15:50] then how do you orchestrate that across different sources? [15:54] That also process of what's the most effective way to gather the information [15:58] that you need to drive that outcome [16:00] is in the example of the thing that we would iterate on. [16:02] Makes sense. How about you with Watchtower? [16:04] What was the time to ramping that up and getting it deployed? [16:09] It was like 2 weeks from coming up [16:10] with the thing to shipping it to customers. [16:13] Like dogfood somewhere, maybe like a week in or something. [16:16] -Similar kind of- -Yeah, superfast. [16:20] Within that regime of rapid iteration, what are the levers that you feel [16:24] like you're tuning very, very frequently? [16:27] It's obviously things like skills, prompts, tools. [16:30] What are the configurable surfaces that you found to be most useful? [16:33] It's all of those without having to worry about handling things [16:36] like compaction ourselves, [16:37] all of the other aspects of building up the harness. [16:41] Because the question that actually comes up is, [16:44] depending on the meeting you're entering, [16:46] what's the most important information you present to the user? [16:50] If we don't have to worry about the rest of it, we can literally just iterate [16:54] on that question, which goes back to the outcome. [16:56] Does the outcomes feature allow us to guarantee that for this conversation, [17:00] we've assessed the most important information and structured it that way, [17:03] and then rapidly iterate towards it? [17:07] I think I remember reading something about Claude Code in the early days [17:10] where in the very early days, Claude Code didn't have evals. [17:13] -You can iterate your way to- -Vibes-based. [17:16] You can iterate your way to getting a thing to be sticky, vibes-based. [17:19] Then the moment it's sticky, then you have to have evals. [17:21] You'll be really sad as a person if you don't have evals [17:24] in a systematic way to iterate towards it. [17:26] Can you figure… [17:27] If the thing is not sticky, no amount of evaluation [17:30] is going to make the thing sticky. [17:32] You got to figure out how to make it sticky first. [17:35] It's a good clarification. [17:36] In this regime, where your very rapid iteration [17:40] using a managed harness is quite nice [17:42] because it's giving you managed infrastructure. [17:43] It's things like prompt caching are just solved. [17:45] -Exactly. -Although it's very important for cost. [17:47] -For sure. -Tell me about how this plays into memory. [17:50] Memory is one that I've actually thought about a lot, [17:52] and I've worked on quite a bit. [17:53] It can be quite tricky to get right. [17:55] For Watchtower, you're using memory in Managed Agents. [17:58] Tell me about how you're using it, and how you've configured it. [18:01] What are the configurable surfaces from memory that you think are important? [18:04] Maybe walk me through that a little bit. [18:06] There are parts of memory that are very core, [18:09] and we need a lot of customization over. [18:11] That's the memory on every account. [18:16] There are a lot of levers we need to pull there to make that really stable. [18:20] It lasts forever. It's pretty detailed on every account. [18:23] It can get corrupted over time. [18:24] There are a lot of things that we do that we manage ourselves, [18:27] because we have to index it in certain ways to make it efficiently queryable. [18:31] We do a lot of work on that to make sure that that's really good. [18:36] But then there's all this other memory on what your different fields mean, [18:40] and how your org works, and user preferences [18:43] and all these sorts of things that are less core. [18:47] It's like they need to work, and they need to be good, [18:50] but they don't need to work any differently for us [18:53] than they need to work for other people. [18:55] For us, the managed memory just unblocks a lot of that. [19:00] Similarly, part of it is just being able to iterate [19:03] and get this to be a valuable product for people. [19:05] But even longer term, I think it's like for us, [19:08] a large part of how we view our job is to make tokens useful for salespeople. [19:15] In our view, a lot of that just comes from the actual product, [19:18] and the product experience [19:20] and the surface that you build around it and so on. [19:22] It also lets us both get it off the ground. [19:24] We built this thing in 2 weeks, [19:26] and so it lets us get it off the ground pretty quickly and make it good. [19:29] But then also, longer term, we haven't rebuilt it [19:32] or felt the need to go off of Managed Agents. [19:34] It's been a few months because we're just focusing. [19:38] There's so much on top of product surface that we can build that's valuable. [19:42] That, I think, just creates a lot more value for customers. [19:45] That's how we think about it. [19:47] Just to clarify slightly. [19:50] Every agent has memory, and that's what you internalize. [19:54] These agents are built, they're not in Claude Managed Agents. [19:56] Correct. [19:57] They have a memory architecture that you own that's completely independent. [20:01] Then the managed agent, this Watchtower product, [20:04] basically can access that memory [20:06] that you've built across this fleet of agents. [20:09] Say I'm one account executive, I have N agents, one for each account. [20:13] The Watchtower can access the memory across that whole fleet [20:16] and then do what specifically? [20:19] If I ask a question, if I go to Watchtower and ask a question, [20:21] "What are the five accounts I need to work on today?" [20:23] That's a function of a couple of different things. [20:25] A, there's like, obviously, it's a function of the account. [20:27] If you own 100 accounts, [20:29] then it's some function of those accounts' memory, [20:31] but it's also a function of a lot of preferences that you have [20:34] and a lot of knowledge about how your organization works. [20:39] Meaning how you prioritize, what accounts you go after, and so on. [20:43] There's the account-level memory is one of the inputs, [20:47] and this managed agent will actually write code [20:49] to be able to query those and can even do fan-out, [20:52] and all these different things on top. [20:54] But then it's also a function. [20:55] The other half of this is a function of you and your company [20:59] and how you work and so on. [21:00] That's all in this managed memory. [21:02] That makes a lot of sense to me now. [21:03] The Watchtower product is like an account executive-facing, [21:07] and that's the interface that they might use [21:09] to access all these different individual account agents [21:11] with their own memory. [21:13] Exactly. You can do work. [21:14] Obviously, there are parts of the product where you can just say, [21:15] "I'm going deep on this one account, [21:17] and I want to just do work on that one account." [21:18] But there are a lot of other things that require going across accounts [21:21] or prioritizing them and so on. [21:23] Both for individual users as well as it's been a big thing for us to expand beyond [21:29] ICs to leaders as well. [21:30] Where a sales leader, if they manage five different AEs, [21:33] now manages 500 accounts, [21:35] and they're responsible for those and works up the chain. [21:38] This product has been useful for building useful stuff for them. [21:43] Then Todd, with sandboxing, I wonder how you think about that? [21:47] It sounds like basically the analytics product has access [21:50] to not only the application [21:51] and all the telemetry coming off the application, [21:53] but also the actual source. [21:55] Correct. [21:56] The customer is actually giving you that access, which is pretty interesting. [22:00] You're taking that code, and you're sandboxing it, [22:01] and then you're interrogating it jointly with telemetry off the application [22:06] to do what then? [22:08] Tell me about that. How do you make- [22:09] There's a variety of use cases that we're using it for. [22:12] I think one of the key things around Managed Agents, [22:14] which you heard just earlier, [22:15] was the speed and time to market was pretty compelling. [22:19] I think under 2 weeks, we actually had a solution leveraging Managed Agents, [22:23] where we were working on ourselves. [22:25] We had a few teams working on it, [22:27] but the speed in which we can just check that box [22:30] for this infrastructure is handled, this is done, it works. [22:34] Of course, there's some iteration and some tweaking, [22:36] and some tuning that we had to do to get it within our operational controls. [22:41] But generally speaking, it was a positive surprise. [22:44] But from there, there's just a number of use cases [22:47] that can spin up. [22:49] For example, we are integrated into a source code repository. [22:52] Every time there is a change to the source code, [22:55] pull request submitted, [22:56] we're firing off a managed agent session to review the code, [22:59] understand what's changed. [23:00] We're actually applying a lightweight UX review [23:03] to make sure we're preventing, [23:04] honestly, bad UX from being committed in the first place. [23:08] But then once something's merged, we're looking at, what happened now? [23:11] This is merged. [23:12] We need to go potentially update analytics. [23:14] Maybe we need to create instrumentation for them. [23:18] We'll fire off our own pull request on top of the ones they just submitted [23:22] so they can have accurate instrumentation. [23:24] At night, we have processes which are looking at what happened the previous day. [23:29] Did a funnel drop in conversion rate overnight [23:33] or even across the 30-day period, [23:35] do we see a slight degradation week over week? [23:38] In which case, we're going to look at what changed 4 weeks earlier, [23:42] and how do we start suggesting a fix for it. [23:45] We're also assessing things like agentic conversations. [23:48] We're looking at how are if humans are interacting with agents, [23:51] how is this conversation going? [23:52] Are there higher error rates than usual? [23:55] We may suggest a prompt fix based on what we're seeing. [23:58] I think the goal is, how do we ultimately get to a self-healing piece of software? [24:04] Where, as people go fast, as people check in more and more changes, [24:07] we're seeing what actual users are doing beyond just our eval sets, [24:11] you have to have those as well, [24:12] but what actual users are doing or how it's actually operating, [24:17] how do we understand and then suggest fixes [24:19] and allow you to continue going as fast as you want to go? [24:22] You actually made an interesting point about evals. [24:24] I'm curious to hear how do you think about evaluations. [24:26] You referenced this earlier. [24:28] Maybe in the very early stages, like vibe-based and ensuring [24:31] you have product-market fit can be quite important. [24:33] I've seen this myself. [24:34] You can have a very carefully curated eval set [24:37] that is not on distribution for your users. [24:39] You will hill climb on it, and then it doesn't matter. [24:41] Your users don't care. [24:42] How do you think about this, evals in general? [24:45] Phase zero is vibes-based. [24:47] Once you have a thing that feels sufficiently good that you like it, [24:50] then that's when you're trying to hill climb to make it really great [24:53] in the whole distribution of cases. [24:55] It's really easy to start internally and gather feedback internally [24:59] on a lot of different data points for what's good and bad. [25:02] That runs the risk, though, of overfitting to your own users at your own company. [25:06] From there, it's just a matter of gathering data points [25:09] and feedback from people externally. [25:12] You got a thumbs up, thumbs down on the outcome of the feature. [25:15] With enough users, you're going to get enough examples [25:17] that curate a pretty decent eval set pretty quickly. [25:21] That allows you to think about building evals [25:23] for just different types of scenarios. [25:24] There's going to be a cohort of evals for regressions. [25:27] There's going to be a cohort of evals for hard, challenging scenarios [25:30] and different kinds of partitions of your data. [25:34] There's going to be cohorts of evals depending on the type of task [25:38] being assigned to the agent. [25:40] Then there's going to be the rubrics that you assign to each of these things. [25:44] As much as you can sync up your rubrics for your offline evals and outcomes [25:47] is also ideal for these kinds of things. [25:49] That's overall how I think about evals. [25:52] Do you have any particular thoughts on evals? [25:54] Or I actually have a different question I want to ask. [25:56] There are two things for evals. [25:58] One is we struggled for a while to come up with good evals because of the thing. [26:03] We would dogfood. Obviously, we're vibe-based. [26:05] Our sales team dogfoods the product. That was good. [26:06] But it's just not representative, like what you said, of actual customers. [26:11] One thing that we did that has been helpful [26:14] is actually looking at real customer queries [26:18] and talking to the user, that person, and understanding [26:21] why did you run this, and what were you looking for, [26:24] and what did you want, and then building evals around those. [26:28] We can do it, obviously, within that customer's data and so on. [26:32] That's been really helpful to avoid the abstraction risk. [26:37] Because what we were finding was our evals were just too abstracted [26:39] and not representative. [26:40] That's one thing. [26:41] I don't think we found a good way to eval given memory, [26:45] a system that's very stateful with this memory that is live. [26:49] Obviously, you can do trajectory evals [26:51] on how well does it interact with the memory. [26:52] But doing these offline result evals given this external state has been… [26:59] I don't think we've fully figured that out. [27:01] Same for us on MCPs with tool calls to third-party services [27:04] with mutating state over time. [27:07] If these evals are going to rely on Slack MCP to produce information, [27:11] it's really hard to get a good systematic suite of it [27:14] that doesn't just drift over time [27:16] because of things that are outside of our control. [27:17] Unless you mock the whole thing, which is just a lot. [27:20] Mocking a customer's Slack I'm not getting in that territory. [27:26] That's not too tractable. [27:28] How do you think about model migrations? [27:30] Opus drops, what do you do? [27:33] How do you think about updating your prompts [27:37] if you think that's necessary? [27:38] How do you think about testing new models? [27:40] Are there things that you think that could be better communicated, [27:42] maybe from us in terms of how to use models? [27:45] We have prompting best practices. [27:47] I think it's mostly that. [27:48] If you have a good eval suite, then you do the model migration. [27:51] Hopefully, the way that you've done your prompting [27:53] has not done historically too much hyper-optimization [27:57] to a specific model family. [27:59] What I've found, though, is a lot of new models at different points [28:01] have different gotcha scenarios [28:03] where they do bad things in different ways. [28:05] The whole five series of models tends to produce writing [28:10] in a way that has actually a lot more AI telltale signs [28:13] than previous series of models. [28:14] More em dashes. [28:16] More of all of that, the sentence structures [28:20] that are just like unreadable sentence structures. [28:25] If there are things where there are known failure modes [28:28] of the new category of models that you're trying to hedge against [28:31] versus trying to optimize your prompts [28:33] to get the last 1% out of the model family. [28:35] I don't care that much about the latter thing [28:38] because that'll keep improving with each generation. [28:40] But if a new model introduces new failure modes [28:43] that are specific to it that you have to avoid, [28:46] that's probably the most valuable to us. [28:48] A few of you were mentioning this point about cost and attribution [28:51] as being one of the major challenges and pain points broadly [28:54] when building agents. [28:56] Tell me about that. [28:58] I think nobody working on agents right now [29:03] is not worried about cost to some degree. [29:07] If you're fully not worried about cost, you're probably not going to build a thing [29:09] that scales long-term in terms of a sustainable business structure. [29:13] I think one of the questions that often comes up for us internally [29:16] of how long do we rely on Claude Managed Agents [29:18] is when does the cost trade-off versus value trade-off of control [29:23] for doing things in-house mean that we should do things in-house? [29:26] I think on the surface, the markup of Managed Agents is pretty low [29:32] in terms of the amount of CPU overhead [29:33] that you're paying for running the infrastructure. [29:36] For example, you can't run batch mode that easily today with it. [29:40] If we're running things 24 hours in advance, we could probably save [29:43] 50%-75% on cost and make that possible. [29:48] Today, for us, it still comes back to that question [29:50] of if we have a lot more value, [29:51] we know we can create for our users in the short to medium term. [29:53] Cost optimization is something where it's be mature, [29:56] be mindful, and spend well, [29:58] and then come back to actually optimizing all of the details over time. [30:04] Hopefully by that point, you will have built features for us to do [30:06] all those things directly within Managed Agents. [30:10] I think both you, and you had mentioned previously to me [30:13] that it's also a point about attribution. [30:15] Where specifically in agent harness are you accumulating cost? [30:18] Is that something that you guys think about? [30:20] You're looking at this combination of cost and flexibility and control. [30:24] You're ceding control. [30:26] Managed Agents, we're using infrastructure like this. [30:29] Over time, of course, your roadmap will give us more control in certain key areas. [30:34] But right now, it's obviously, I think, the best solution for us to go forward. [30:38] Getting really fine-grained telemetry of what calls cost what. [30:44] When you have something, we have large batch processes [30:46] that happen in the evening, which process lots of data. [30:50] It gets expensive, or it can get expensive. [30:53] We can do some really inexpensive things right now [30:55] in terms of engineering time to just cut this, [30:57] do it less frequently, send less data. [31:00] We can do things like that just because we don't have to over-optimize. [31:03] But when you get to a certain level of scale and maturity, [31:06] you're going to want to have a little more control over that. [31:08] Hopefully, we'll have that ability when we get to that point. [31:12] We'll see. [31:13] What do you think? What can we do better, generally? [31:15] Anthropic, it could be Claude broadly, or also Managed Agents. [31:19] Maybe there are a couple of things. [31:20] I think one is cost, and then two is for us, [31:23] ways to improve the user experience. [31:25] On the cost side, I think it makes total sense for us today [31:29] on parts of the product that need truly frontier intelligence. [31:34] If we're running Opus or whatever, then it's just a really good model, [31:39] and I think it's relatively well priced. [31:41] But there are other parts of the product that don't need as much intelligence, [31:44] or we just can't do that because we run over such a high end. [31:49] For example, even this Watchtower thing, [31:51] the agent that coordinates the search or the main agent [31:55] needs frontier intelligence. [31:56] But if you're fanning out across 500 different accounts, [32:00] you can't do the same thing there. [32:02] That is a place that we've had to cost-optimize. [32:05] We'll use much cheaper models and so on to do that. [32:09] That's one side. [32:10] I think the inherent structure of the product makes it hard, [32:13] of Claude Managed Agents makes it hard to get… [32:17] When you have to do the cost optimization, it's very constraining [32:21] because it's all or nothing in some sense. [32:23] That's one side. There aren't that many levers for cost. [32:26] Yeah, I see. That's good. [32:28] Better surfacing of levers for cost. [32:30] We have things like effort, we have multi-agent, [32:32] we have obviously many models. [32:34] Some flex or batch mode would help. [32:36] That's a 2X, 50% right there. [32:40] Things like that would help a lot on one side. [32:43] I do think there's a class of things, [32:45] and I don't know how important these are yet, [32:49] but there are other things we want to do to improve the user experience for people. [32:53] For example, in latency-sensitive parts of the product, [32:56] we want to pre-warm sandboxes in different clever ways to reduce cold start latency. [33:02] Those are things also just the nature of the managed agent product. [33:06] The way we think it works really well now is when you need frontier intelligence, [33:10] you're relatively price insensitive, and the task is hard, [33:15] so it runs for minutes or something, [33:16] and then you don't really care about seconds here or there. [33:19] For those things, it works very well and totally makes sense. [33:22] It just lets us get those out really quickly. [33:24] But for anything else, there are not that many levers to improve UX [33:29] or make it cheaper, things like that. [33:32] Cool. [33:33] Anything on your side? [33:36] I think that's the bulk of it. [33:38] It comes back to if optimizing cost or optimizing control [33:43] allows you to build a thing that you otherwise can't, [33:45] then it makes a ton of sense. [33:46] If you can't build a certain feature without reducing the cost by 90%, [33:50] then you got to figure out a way how to do that. [33:53] If we're building our Jarvis voice-controlled chief of staff over time, [33:57] odds we're going to probably have to roll [33:59] our own harness for that thing seem quite high. [34:01] But that doesn't mean that we roll it for everything. [34:04] The same way that you have parts of your harness, [34:06] and I'm sure you have parts of your system that you build entirely yourself, [34:10] and you have parts of your system that you rely on other things. [34:12] I like your three criteria. [34:14] I think our fourth criteria is when you're validating viability, [34:19] where, again, the cost of an experiment is in the time it takes to learn. [34:27] That's the only other case in which I think it really shines [34:30] in addition to the ones that Mihir mentioned. [34:33] Thank you all for being here. Really enjoyed the conversation. [34:35] -Likewise. -Thank you.