Anthropic Equips Managed Claude Agents With Self-Grading and Vaults

Build a Research Agent with Claude Managed Agents (Part 2 of 4) — Anthropic — AI Engineer

AI Engineer

Long-running autonomous research requires automated outcome rubrics that force agents to evaluate and retry their own work until deliverables meet formal specifications.

AI-assisted summary of Build a Research Agent with Claude Managed Agents (Part 2 of 4) — Anthropic — original source. Check the original for full context.

Source published:

Key ideas

  • Anthropic's Claude Managed Agents introduces "Outcomes," a self-supervision mechanism combining task definitions, evaluation rubrics, and automated grader subagents that trigger retries until criteria are satisfied. (17:41)
  • To secure API credentials, the platform uses "Vaults" that expose only placeholder tokens to model sandboxes, swapping in live secrets strictly at the egress proxy when requests match designated domain whitelists. (23:53)
  • Cross-session state and organizational intelligence are preserved through shared Memory Stores, supplemented by an offline "dreaming" engine that consolidates transcripts and notes into structured knowledge. (29:14)
  • Private enterprise data can be integrated using three deployment architectures: public MCP endpoints backed by Vaults, reverse-connection MCP Tunnels that bypass inbound firewall rules, or self-hosted sandboxes inside private VPCs. (33:15)

Detailed notes

## Background Published on YouTube by AI Engineer on October 12, 2026, this video documents part two of Anthropic's four-part Claude Managed Agents workshop recorded at the AI Engineer World's Fair in San Francisco. Presented by Anthropic technical staff members Priyanka Phatak and Gabe Cemaj, the session guides developers through evolving a single-prompt console agent into an end-to-end 'research desk' application equipped with custom skills, subagents, persistent memory, and self-supervising outcome rubrics.

## TL;DR Following a quick demonstration of creating a hosted agent from natural language in Anthropic's web console, Gabe Cemaj leads a hands-on architecture build of a multi-session financial research desk. He demonstrates how to stream real-time events to a chat interface, provision an SEC EDGAR filing skill, isolate child subagents, and wire up a shared memory store that persists context across runs. The build culminates in a self-supervised Nvidia research run where a dedicated grader subagent inspects deliverables against a predefined rubric and triggers retries until quality standards are met. Throughout the session, the speakers field audience questions on Vault credential egress swapping, differences between the Agent SDK and managed infrastructure, loop guardrails, session steering, and private enterprise data connectivity via MCP tunnels.

## Atmosphere & style The session has the candid, pragmatic energy of a live technical workshop where engineers speak peer-to-peer. When venue Wi-Fi constraints leave most attendees unable to clone dependencies, Gabe and Priyanka smoothly adapt the format, turning potential friction into an open, high-bandwidth architectural discussion. The dynamic between the speakers is collaborative and grounded, balancing transparent admissions of active roadmap gaps with deep, reassuring explanations of sandbox caching, credential virtualization, and multi-agent coordination.

## Bulletin - [17:41](https://www.youtube.com/watch?v=FGftGOzfohc&t=1061s) Anthropic's Claude Managed Agents introduces "Outcomes," a self-supervision mechanism combining task definitions, evaluation rubrics, and automated grader subagents that trigger retries until criteria are satisfied. - [23:53](https://www.youtube.com/watch?v=FGftGOzfohc&t=1433s) To secure API credentials, the platform uses "Vaults" that expose only placeholder tokens to model sandboxes, swapping in live secrets strictly at the egress proxy when requests match designated domain whitelists. - [29:14](https://www.youtube.com/watch?v=FGftGOzfohc&t=1754s) Cross-session state and organizational intelligence are preserved through shared Memory Stores, supplemented by an offline "dreaming" engine that consolidates transcripts and notes into structured knowledge. - [33:15](https://www.youtube.com/watch?v=FGftGOzfohc&t=1995s) Private enterprise data can be integrated using three deployment architectures: public MCP endpoints backed by Vaults, reverse-connection MCP Tunnels that bypass inbound firewall rules, or self-hosted sandboxes inside private VPCs.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=FGftGOzfohc&t=0s) Session storage was re-architected to decouple event ingestion, persistence, and session mechanics by virtualizing interfaces across service boundaries and the event layer. - [01:04](https://www.youtube.com/watch?v=FGftGOzfohc&t=64s) Claude Managed Agents abstracts runtime infrastructure, enabling cloud-hosted agent deployment, sandbox environment provisioning, and event streaming via prompt-driven console workflows or the /v1/agents API. - [10:16](https://www.youtube.com/watch?v=FGftGOzfohc&t=616s) Execution compute supports multiple topologies, including Anthropic-managed cloud hosting, AWS Bedrock using enterprise billing credits, or self-hosted sandboxes on providers like Modal, Daytona, and Vercel. - [11:21](https://www.youtube.com/watch?v=FGftGOzfohc&t=681s) Client-agent interactions rely on a continuous, bidirectional event stream using Server-Sent Events (SSE) rather than standard synchronous request-response REST patterns. - [13:41](https://www.youtube.com/watch?v=FGftGOzfohc&t=821s) When spawning subagents, child sessions inherit system prompts and tool schemas but do not inherit the parent session's conversation transcript, preventing context pollution and token inflation. - [17:41](https://www.youtube.com/watch?v=FGftGOzfohc&t=1061s) Outcomes provide autonomous quality control by having a dedicated grader subagent evaluate finished work against an explicit rubric before marking tasks complete. - [22:09](https://www.youtube.com/watch?v=FGftGOzfohc&t=1329s) Skills function as live, versioned artifacts; updating a skill definition immediately propagates improved instructions and tool behaviors to newly created sessions without redeploying the core harness. - [23:53](https://www.youtube.com/watch?v=FGftGOzfohc&t=1433s) Vaults mitigate secret exfiltration by restricting live credential injection to the network egress proxy level, ensuring placeholder tokens are only substituted when calling whitelisted domains. - [26:08](https://www.youtube.com/watch?v=FGftGOzfohc&t=1568s) The Claude Agent SDK operates as a standalone headless binary intended for developer-hosted compute, while Managed Agents provides a multi-tenant, fully hosted cloud execution service with sandboxes and event streaming. - [27:15](https://www.youtube.com/watch?v=FGftGOzfohc&t=1635s) While infinite-loop prevention in custom harnesses currently requires developer-side termination logic, native platform features for hard financial caps and per-session token budgets are in development. - [28:04](https://www.youtube.com/watch?v=FGftGOzfohc&t=1684s) Active sessions support live operator steering by accepting user message events queued into the live stream, whereas session forking and history rewinding remain roadmap items. - [29:14](https://www.youtube.com/watch?v=FGftGOzfohc&t=1754s) Memory stores support concurrent layering, permitting an agent session to draw simultaneously from private, user-specific memories and global, organization-wide knowledge bases without vendor lock-in. - [31:54](https://www.youtube.com/watch?v=FGftGOzfohc&t=1914s) Outcome rubrics currently grade execution at the macro-session boundary rather than independently evaluating intermediate child subagent operations. - [33:15](https://www.youtube.com/watch?v=FGftGOzfohc&t=1995s) MCP Tunnels allow enterprises to connect internal data repositories and RAG pipelines to Claude Managed Agents over secure reverse tunnels without opening inbound public firewall ports. - [34:27](https://www.youtube.com/watch?v=FGftGOzfohc&t=2067s) Sandbox container environments resolve and cache package dependencies during environment creation, avoiding package installation overhead across consecutive session reinstantiations.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=FGftGOzfohc&t=0s) Evolving Session Storage: Decoupling and Virtualization

The session opens with an audience question about the architectural evolution of Claude Managed Agents: specifically, what changed in the session storage design between the first and second iterations. Priyanka Phatak directs the question to Gabe Cemaj to explain how the underlying data model shifted as the platform matured.

Gabe explains that the core motivation for redesigning the storage system was modularity and decoupling. In earlier iterations, the storage layer was too tightly bound to session mechanics, making it difficult to introduce new event types or support alternative backends without breaking runtime assumptions. Gabe Cemaj summarizes their architectural philosophy: “So we basically apply the same principle of building and virtualizing interfaces all the way down, not only across service boundaries but also at the event layer.” By virtualizing these boundaries, the team ensured that storage mechanisms could handle arbitrarily complex event streams going forward.

- **Component Isolation**: The storage redesign partitioned event ingestion, persistence, and session mechanics into independent abstractions rather than a monolithic session loop. - **Extensible Event Layer**: New event types can be registered and stored without requiring modifications to the core session orchestrator.

[01:04](https://www.youtube.com/watch?v=FGftGOzfohc&t=64s) The No-Code Console Quickstart: Creating a Hosted Agent in Seconds

Acknowledging that audience members are battling spotty venue Wi-Fi and may need to follow along on mobile hotspots or review the slides later, Priyanka transitions to a quick live demonstration in the web console. She sets out to prove how minimal the barrier to entry is by building a fully functional cloud-hosted agent from a single natural language instruction: *"build me an agent that tells me jokes every time given a topic."*

Upon submitting the prompt, Claude immediately synthesizes a full agent specification in the background. It generates an appropriate agent name, writes a descriptive overview, authors a specialized system prompt, selects a model, and provisions access to the standard agent toolset. Priyanka shows that developers can override any of these defaults, such as swapping the underlying foundation model, before clicking "create agent"—which hits the platform's `/v1/agents` API endpoint. Next, she configures an environment template for the sandbox. While she leaves access unrestricted for this demonstration, she emphasizes that production setups should enforce strict network access controls, whitelisting only authorized egress domains or specific Model Context Protocol (MCP) servers. After defining the environment and dependencies, she spins up a session and tests it with *"Tell me jokes about cats."* Claude displays its internal thinking block and immediately streams back a cat joke.

Priyanka Phatak underscores the operational simplicity of the managed model: “All you have to do is start a session with the agent, send events, get back events. Everything else is managed by us.” Rather than forcing developers to stand up sandbox fleets, write execution harnesses, manage session scaling, or maintain distributed event logs, the platform abstracts the entire runtime infrastructure so developers can focus strictly on product logic. With the console demo complete, she hands the stage to Gabe for the hands-on engineering track.

[03:47](https://www.youtube.com/watch?v=FGftGOzfohc&t=227s) Introducing the Hands-On Build: The Research Desk Architecture

Gabe Cemaj takes the microphone, inviting attendees squeezed into the back of the room to move forward into empty seats. He displays a QR code linking to the workshop GitHub repository and instructs the audience to clone the project, change into the `research-desk` directory, and install dependencies using either `npm install` or `bun i`.

Gabe then maps out the technical journey of the workshop. The goal is to evolve a basic single-agent chat endpoint into a sophisticated, multi-session financial research desk. The build begins with a custom chat interface running directly over the session event stream. From there, the agent is upgraded into a multi-agent system capable of delegating specialized work to subagents. They will equip the agent with an SEC EDGAR skill so it can query the U.S. Securities and Exchange Commission's filing systems, locate individual company tickers, pull down regulatory filings, and extract specific forms. Gabe notes that this setup powers long-running asynchronous agentic workflows, culminating in a map-reduce orchestration pattern where a primary agent delegates sub-tasks across multiple parallel sessions. Finally, he outlines the use of the Deployments API to schedule recurring background runs that keep financial analysis artifacts continuously refreshed.

```mermaid flowchart TD accTitle: Research Desk Workshop Progression accDescr: Progression from a bare chat interface to a multi-agent scheduled research system. subgraph UI["Interface Layer"] A["Chat UI on Event Stream"] end subgraph Core["Agent Capabilities"] B["SEC EDGAR Skill"] C["Subagent Delegation"] end subgraph Scaling["Orchestration & Automation"] D["Map-Reduce Multi-Session Harness"] E["Deployments API for Scheduled Runs"] end UI --> Core Core --> Scaling ```

- **Interactive Event Stream**: Interfacing directly with real-time session events rather than using blocking request-response patterns. - **SEC EDGAR Integration**: Automated retrieval and structural analysis of regulatory filings (such as 10-K and 10-Q forms) for public equities. - **Map-Reduce Agent Harness**: Orchestrating multiple agent sessions concurrently to process large data corpora before reducing findings into a unified report. - **Recurring Deployments**: Running autonomous analysis jobs on a cron-like cadence to produce self-updating research desks.

[06:10](https://www.youtube.com/watch?v=FGftGOzfohc&t=370s) Project Setup: Claude Code, Environment Variables, and the Six-Act Workshop Skill

Gabe transitions to his development environment, selecting Claude Code as his terminal agent while remarking with a laugh: “I don’t think anybody’s writing code by hand, so I’m not going to do that either.” He explains that the repository includes an automated interactive tutor packaged as a Claude Code skill. By invoking `workshop start`, attendees can execute a guided, six-act tutorial that manages project state, verifies code, and advances using `workshop next`. Gabe humorously apologizes that there will be no intermission between the six acts before enabling Claude Code's auto mode and instructing it to hold his hand through the setup steps.

Running `bun run dev` spins up the local Next.js research desk server, presenting a setup checklist in the browser. The automated skill initiates Act 1 by duplicating `.env.example` into `.env.local`. Gabe warns the audience that this configuration requires two mandatory credentials: 1. **Anthropic API Key**: Grants access to Claude Managed Agents endpoints. 2. **SEC EDGAR User-Agent Identifier**: Formatted as a user's name and contact email address (e.g., `Sample Company AdminContact@<sample company domain>.com`), which the SEC requires on all automated API calls to monitor traffic and enforce rate limits.

``` # Research Desk Environment Configuration (.env.local) ANTHROPIC_API_KEY=your_anthropic_api_key_here EDGAR_IDENTITY="Your Name your.email@example.com" ```

A quick show of hands in the room reveals that venue Wi-Fi failures have left nearly the entire audience unable to clone or run dependencies, with only a single attendee successfully keeping pace. Gabe and Priyanka pivot smoothly, urging participants to treat the session as an open architectural discussion, ask deep technical questions, and rerun the guided skill at home using their own network connections.

[10:16](https://www.youtube.com/watch?v=FGftGOzfohc&t=616s) Q&A: Infrastructure, Cloud Hosting, and Self-Hosted Sandboxes

As Gabe returns to the setup dashboard to click the button that provisions the agent and sandbox environment, an audience member asks where the managed infrastructure physically executes and whether it runs on Amazon Web Services (AWS).

Gabe details the deployment topology supported by Claude Managed Agents. Anthropic provides a fully managed first-party (1P) hosting environment that runs directly on Anthropic-managed infrastructure, abstracting all server operations. For enterprise customers who must maintain compute within existing cloud billing agreements or utilize enterprise discount programs, the service is also accessible through AWS Bedrock using existing AWS credits.

Furthermore, Gabe highlights that execution environments do not have to be hosted by Anthropic at all. Gabe Cemaj notes: “What we also offer and maybe we can talk about this after this demo is self-hosted sandboxes. So if you want to bring your own sandbox through different providers like Vercel or Modal or Daytona or a bunch of others, you can do that.” By supporting self-hosted sandbox integrations across providers like Modal, Daytona, Vercel, Google Cloud Platform (GCP), and AWS, engineering teams can keep sensitive execution environments and proprietary runtime packages entirely within their own virtual private clouds while continuing to rely on Claude Managed Agents for agent reasoning, session lifecycle management, and event routing.

[11:21](https://www.youtube.com/watch?v=FGftGOzfohc&t=681s) Setting Up the Event Stream and Chat Interface

Gabe Cemaj begins by configuring the initial environment for the research agent. For this specific demonstration, the agent is granted network access to external resources, specifically the U.S. Securities and Exchange Commission (SEC) website, along with predefined Python packages installed in its environment so it can query SEC endpoints. At this starting stage, the agent is completely bare-bones: it possesses standard basic tools and a system prompt, but lacks domain-specific skills, custom orchestration, or sub-agent coordination. Gabe prompts Claude to advance the build and wire up the application's conversational interface.

He explains that communication with a Claude Managed Agents session centers around an event stream. Rather than relying on simple request-response REST patterns, the application listens to an incoming stream for all lifecycle and output events emitted by the agent, while a dedicated ingress endpoint allows the client to push user events back up into the session. Gabe points out: “So really everything that happens in the communication for these more chat-like applications happens on that stream.” Claude writes the code to capture enter keypresses in the chat input, package the text into a `user_message` event targeting the active session, and pipe the resulting Server-Sent Events (SSE) back from the backend API to the browser for real-time frontend rendering.

Testing the freshly wired integration under the application’s "Desk" tab, Gabe sends a quick greeting and confirms the agent replies over the stream. While functional, he emphasizes that this is merely table stakes. If an engineer only wants a standard chatbot, they could stop here, but the true capability of managed agents lies in sustained, autonomous execution. As Gabe emphasizes, “The whole point of Claude Managed Agents is that you can do long-running asynchronous tasks—not just for minutes, but hours, days, weeks, and even months.”

```mermaid flowchart LR accTitle: Client-Agent Event Stream Architecture accDescr: Flow diagram showing bidirectional event communication between browser UI, backend server, and managed agent session.

subgraph Client ["Frontend"] UI["Chat UI"] end

subgraph AppServer ["Application Backend"] Endpoint["Ingress API"] SSEProxy["SSE Relay"] end

subgraph Platform ["Managed Agents Session"] Stream["Event Stream"] end

UI -->|"user_message event"| Endpoint Endpoint -->|"Push event"| Stream Stream -->|"SSE stream"| SSEProxy SSEProxy -->|"Render response"| UI ```

---

[13:41](https://www.youtube.com/watch?v=FGftGOzfohc&t=821s) Upgrading the Agent: Skills, Sub-agents, and Shared Memory

Having established basic connectivity, Gabe moves to augment the agent so it can climb what he calls the "ladder of intelligence." By entering `workshop next`, Claude updates the infrastructure-as-code provisioning definitions to equip the agent with specialized domain skills, custom tooling, and hierarchical sub-agent delegation. The agent receives an SEC EDGAR skill detailing how to use the pre-installed Python libraries and how to identify itself using the SEC EDGAR user-agent header requirements. Claude also defines a custom tool named `dispatch_analysis`, which will later let a primary "Head of Research" agent coordinate and spin up multiple parallel analysis sessions.

When Claude updates the primary agent definition, it produces a new version configured with dedicated sub-agents and, crucially, a recursive self-reference so the primary agent can clone its own prompt and tool persona if unexpected workloads arise. Gabe addresses a frequent point of developer confusion regarding sub-agent spawning: “It doesn't copy the transcript of that session into the sub agent. It just copies over the tool description and the schemas and the prompt into that sub agent.” This isolation prevents prompt context pollution and token bloat while giving the child agent a clean slate.

Gabe runs into a brief "cooking show" hiccup when trying to click the "Staff the desk" button—a pre-existing skill from an earlier run causes a provisioning conflict. He quickly deletes the stale skill and re-provisions the environment. In addition to sub-agent definitions and skills, the provisioning creates a long-running shared memory store. This persistent layer allows isolated sessions to write and read accumulated learnings about various financial tickers and market instruments, passing institutional context forward over months or years of automated runs.

- **SEC EDGAR Skill**: Encapsulates EDGAR querying rules, rate limits, identity headers, and Python library usage. - **`dispatch_analysis` Custom Tool**: Enables programmatic multi-session dispatch from the top-level agent. - **Sub-Agent Isolation**: Child sessions inherit system instructions and tool schemas without copying the parent session's conversational history. - **Cross-Session Memory Store**: Provides persistent, shared knowledge retention across independent agent executions.

---

[17:41](https://www.youtube.com/watch?v=FGftGOzfohc&t=1061s) Outcomes and Self-Supervision: The Nvidia Research Run

With sub-agents and memory established, Gabe transitions away from conversational interaction to launch a long-running, asynchronous task governed by "outcomes." In the Claude Managed Agents architecture, an outcome acts as a self-supervision primitive combining a task objective, an explicit evaluation rubric, and an automated grading loop. Gabe summarizes the mechanism: “In an outcome, you tell Claude, 'This is your task. This is your rubric to evaluate your task.' It's going to go off and do that work and then when it thinks it's completed, it's going to ask a sub agent to look at the rubric and the work that Claude did and evaluate: did it do it correctly and did it do it to completion?” If the work falls short of the criteria, the agent automatically retries until it passes or hits a defined maximum iteration ceiling.

Claude implements the underlying `user_defined_outcome` event, after which Gabe navigates to the application's scorecard view and triggers an analysis of Nvidia. Because a full SEC filing audit takes around ten minutes, Gabe switches over to review an identical session he ran the previous day. The prompt instructed Claude to analyze Nvidia's most recent quarterly SEC filings using valid EDGAR credentials, inspect shared memory first to avoid redundant research, extract key metrics into a standardized JSON scorecard saved to a designated local path, and log analytical notes back into the memory store.

Tracing the run through the console's observability interface, Gabe points out individual visual threads representing sub-agent lifecycles:

1. The root agent verified the shared memory store and discovered no prior notes for Nvidia. 2. It dispatched two specialized sub-agents in parallel: a Financial Expert Analysis extractor to pull filing numbers and a Risk Evaluator to dissect qualitative risk factors. 3. Upon sub-agent completion, the root agent synthesized the gathered data, wrote the formatted JSON scorecard deliverable to disk, and updated shared memory. 4. The outcome self-supervision loop automatically engaged: a dedicated grader sub-agent evaluated the final artifacts and execution log against the rubric. 5. The grader verified that all metrics were well-defined and that memory updates occurred, formally marking the outcome as satisfied.

---

[21:55](https://www.youtube.com/watch?v=FGftGOzfohc&t=1315s) Scaling Up: From Single Tickers to Multi-Agent Harnesses

After demonstrating that the completed scorecard extracts successfully from the session sandbox to the local environment (or into an external database in a production deployment), Gabe poses the next architectural hurdle. Analyzing a single company like Nvidia via a manual UI trigger works well for demonstration purposes, but institutional research workflows demand processing ten, fifty, or a hundred tickers concurrently. He pivots to the next phase of the workshop: leveraging the conversational interface to instruct the Head of Research agent to build and manage a custom multi-agent execution harness capable of orchestrating ticker evaluations at scale.

[22:09](https://www.youtube.com/watch?v=FGftGOzfohc&t=1329s) Updating Skills as Live Artifacts and Multi-Session Orchestration

Gabe Cemaj outlines how the custom orchestration harness operates inside the server. When communicating with the primary agent, that agent can autonomously decide to spawn parallel worker sessions—for instance, spinning up ten distinct sessions to execute isolated work in parallel before aggregating the results back into a final set of scores. Before demonstrating the orchestration layer, however, Gabe catches himself skipping an essential prerequisite: updating the underlying skill.

Skills in Claude Managed Agents are not static files frozen at deployment time; they operate as dynamic, versioned resources that can be continuously refined. In this case, Gabe demonstrates modifying the SEC skill to inspect a new SEC filing form. As soon as a revised skill is published, any subsequent agent session referencing that skill automatically inherits the updated instructions and schema without requiring a rebuild of the entire agent harness. Gabe emphasizes that developers can continuously improve skills by inspecting historical session performance, noting that Anthropic automates part of this process through a capability called "dreaming" demonstrated later in the session.

As Gabe transitions toward the orchestration setup, Priyanka Phatak steps in to handle audience logistics and kick off a live Q&A segment. Addressing attendees who were waiting on API promotional credits, she clarifies that although the signup form warns of a one- to two-day turnaround, credits typically provision in about five minutes.

- “I wanted to show you that these skills are live artifacts that you can update, so you can improve the skill.” — Gabe Cemaj - Once updated and published, any newly instantiated agent session instantly references the revised skill artifact. - Skills can be evolved over time based on edge cases encountered in earlier runs or consolidated automatically via offline "dreaming" passes.

---

[23:53](https://www.youtube.com/watch?v=FGftGOzfohc&t=1433s) Q&A: Vaults and Egress Credential Swapping

An attendee asks whether accessing SEC EDGAR filings requires credentials and how credential injection or brokering is securely handled by the system. Gabe clarifies that while the public SEC API technically only requires a name and contact email in the user-agent header—which they simply inject into the system prompt for the quick demo—production deployments should never expose raw credentials to the model. Instead, managed agents rely on a dedicated security abstraction called **Vaults**.

In a Vault, developers configure environment variables and secrets using placeholder tokens rather than raw API keys or passwords. While Claude is executing tools and writing code inside its execution sandbox, it only ever observes and manipulates the placeholder token. The sensitive swapping happens strictly at the network egress boundary, and only when the request destination matches an explicit domain whitelist assigned to that specific credential. If an agent attempts to exfiltrate an SEC credential to an unauthorized domain such as `anthropic.com`, the egress proxy refuses to substitute the placeholder, neutralizing credential theft or leakage.

Priyanka and Gabe walk through the console management interface for Vaults. Beyond static environment variables, Vaults natively handle OAuth token flows for Model Context Protocol (MCP) servers and third-party SaaS integrations like Notion or Google Workspace CLIs installed directly inside the sandbox environment. Each Vault yields an alphanumeric ID that can be passed directly into session creation calls to instantly grant safe, proxied access.

```mermaid flowchart LR accTitle: Vault Egress Swapping Mechanism accDescr: Diagram showing how Claude works with placeholder credentials inside the sandbox, while egress proxies swap placeholders for real secrets only when calling whitelisted external domains.

subgraph Sandbox["Managed Agent Sandbox"] Claude["Claude Agent"] Env["Environment Variable<br/>(Placeholder Token)"] Claude -->|"reads"| Env end

subgraph SecurityBoundary["Anthropic Network Egress Proxy"] Proxy["Vault Egress Interceptor"] Whitelist{"Is destination<br/>domain allowed?"} Proxy --> Whitelist end

subgraph External["External APIs"] SEC["Whitelisted API<br/>(e.g., sec.gov)"] Untrusted["Unauthorized Host<br/>(e.g., untrusted.com)"] end

Env -->|"HTTP Request with Placeholder"| Proxy Whitelist -->|"Yes: Swap for Real Secret"| SEC Whitelist -->|"No: Send Placeholder / Block"| Untrusted ```

- “Instead of putting the real secret in, what we put is a placeholder, and then when Claude is doing work and it wants to use that environment variable, it just injects the placeholder value; we at egress swap it out for the real value only on the host domains that you specified are allowed for that secret.” — Gabe Cemaj - **Domain Whitelisting**: If a secret is scoped exclusively to SEC endpoints, any outbound call targeting another domain receives only the inert placeholder string. - **Console & API Provisioning**: Vaults can be created programmatically or via the web console, returning an ID that can be attached to any session. - **Protocol Support**: Vaults support static API keys, environment credentials, and automated OAuth lifecycles for external MCP tool servers.

---

[26:08](https://www.youtube.com/watch?v=FGftGOzfohc&t=1568s) Q&A: Claude Agent SDK vs. Managed Agents

An audience member asks for architectural clarification regarding the boundary between the Claude Agent SDK and Claude Managed Agents, wondering whether the managed service is built as a direct wrapper on top of the SDK. Gabe explains that the two serve distinct operational paradigms despite sharing common architectural patterns.

The Agent SDK was originally packaged as a headless, developer-facing harness—effectively an unbundled, standalone binary of Claude Code that developers can download, containerize, and execute inside their own local or self-hosted compute environments. Managed Agents, by contrast, represent a fully hosted cloud infrastructure running on Anthropic's managed servers. While the managed agent harness draws heavily on design lessons, prompt harnesses, and execution primitives proven in Claude Code and the SDK, the team built a generalized, cloud-native orchestration harness capable of running arbitrary, non-coding research workflows, keeping the two codebases closely coordinated rather than in a strict parent-child dependency.

- “Agent SDK is basically like Claude Code in the form of a binary that you can take and run wherever you want; this is a fully hosted solution running in our cloud and we have our own harness that's heavily inspired by Claude Code's harness and we frequently stay in touch for like any enhancements across the board.” — Gabe Cemaj - **Agent SDK**: A headless, standalone distribution of the Claude Code harness intended to run anywhere developers choose to host it. - **Managed Agents**: A multi-tenant, fully hosted cloud execution service featuring managed sandboxes, persistent session streaming, and out-of-the-box infrastructure.

---

[27:15](https://www.youtube.com/watch?v=FGftGOzfohc&t=1635s) Q&A: Loop Guardrails and Token Budgets

A participant raises concerns about runaway multi-agent orchestration, asking what guardrails exist to prevent subagents from triggering infinite communication loops and rapidly burning through massive amounts of context tokens.

Gabe acknowledges that in custom agent harnesses, infinite-loop prevention and multi-agent coordination logic are currently left largely in developer hands. Because developers control the server-side orchestration harness, they are free to implement custom terminating conditions, call out to alternative LLM providers, or enforce internal cycle detection. However, Gabe shares that Anthropic is actively developing native, fine-grained budgeting primitives. The roadmap includes controls where developers can set hard monetary caps or strict session-level token thresholds, causing the runtime to automatically halt execution once an agent crosses its assigned resource ceiling.

- “We are exploring the idea of what it means to do like budget tighter budget controls that you can say use no more than this amount of money per session, use no more than this amount of tokens per session, but that's not something we offer today.” — Gabe Cemaj - Developers building custom orchestration harnesses are currently responsible for circuit breakers and recursion limits. - Hard currency ceilings and per-session token limits are in active research and planned for future platform updates.

---

[28:04](https://www.youtube.com/watch?v=FGftGOzfohc&t=1684s) Q&A: Live Steering vs. Session Forking

Referring to the web console transcript viewer, an attendee asks whether developers can step in when an agent starts drifting down an erroneous path—either to correct history, rewind state, or branch execution with updated instructions.

Gabe demonstrates that live steering is supported out of the box through session events. By selecting "Send Event" in the console or dispatching an API call, a human operator can inject a new user message event directly into the active session stream. The runtime automatically queues the message, allowing developers or end users to deliver course corrections or stack multiple instructions while the agent runs. However, rolling back state or editing prior history is not currently supported. Priyanka adds that true "session forking"—the ability to snapshot an agent's sandbox and conversation history at a specific turn, branch off in a new direction, and explore alternative paths without mutating the original session—is a heavily requested feature from enterprise customers and is slated on Anthropic's active product roadmap.

- “Session forking is essentially what you're asking for; we've had a few customers who have asked for it, it's on our roadmap, so hopefully we'll have it at some point.” — Priyanka Phatak - **Active Steering**: Operators can post real-time steering events into an running agent session, which queues and processes them sequentially without terminating execution. - **Session Forking**: Rewind-and-branch capabilities are not yet supported in the public API, but remain an active item on the platform roadmap.

---

[29:14](https://www.youtube.com/watch?v=FGftGOzfohc&t=1754s) Q&A: Shared Memory Stores, Dreaming, and Cloud Flexibility

A participant asks whether managed agents support a shared, collective memory layer across hundreds of different users working on financial research within the same organizational deployment. Gabe confirms that shared memory is already implemented and active in the workshop demo through the platform's **Memory Store** construct.

During the Nvidia research workflow, multiple autonomous agents concurrently read and write facts, intermediate findings, and edge-case learnings into a shared memory store. To prevent memory stores from accumulating fragmented, contradictory, or redundant context, Anthropic implements an offline consolidation process called "dreaming." During dreaming passes, background routines synthesize raw session logs and agent notes into a unified, clean knowledge base that improves subsequent agent runs across the entire organization.

When the attendee asks about infrastructure lock-in—specifically whether teams bound to AWS Bedrock or proprietary self-hosted sandboxes can decouple their data from Anthropic's cloud—Gabe and Priyanka emphasize that the architecture is fully open and decoupled. The memory store acts as a standard API resource: teams can extract memories, pipe them into external databases or vector stores, or wire in third-party memory systems via custom MCP tools. Furthermore, Priyanka explains that agents support layered memory hierarchy: a single session can attach both an individual user-level memory store and a broader, organization-wide memory store simultaneously, enabling personalized context alongside shared institutional intelligence.

- “None of this is locked in; you can take your data and go anywhere else if you wanted to, everything is an API call.” — Priyanka Phatak - “All of this is API based, so you can extract the memories and put them in different systems if you want to.” — Gabe Cemaj - **Multi-Store Attachment**: Sessions can concurrently attach multiple memory stores to isolate private user context while drawing from global organizational knowledge. - **Dreaming**: An automated background synthesis mechanism that processes disparate session records and edge-case outcomes to consolidate knowledge stores over time. - **Vendor Agnosticism**: Memory primitives, custom tools, and sandbox workflows can interface with external clouds and storage systems using Model Context Protocol (MCP) bridges.

---

[31:29](https://www.youtube.com/watch?v=FGftGOzfohc&t=1889s) Q&A: Grading Subagentic Workflows

An attendee observes that in the workshop architecture, the rubric grader evaluates outputs at the primary agent level, and asks how developers should grade and manage memory across nested subagentic tasks.

Gabe begins breaking down the two operational dimensions of this architecture, explaining that subagents are intentionally designed as context-augmentation primitives. Rather than forcing the root agent to ingest overwhelming volumes of raw SEC filings or tool responses, subagents take on focused delegation scopes to insulate the primary context window and preserve model reasoning capacity.

- “Subagents in the way that we've built them are meant to augment your context and leverage the...” — Gabe Cemaj

[31:54](https://www.youtube.com/watch?v=FGftGOzfohc&t=1914s) Session-Level Outcomes and the "Dreaming" Memory Engine

Addressing how outcome verification interacts with hierarchical subagents, Gabe Cemaj clarifies that evaluation rubrics operate strictly at the top-level session scope rather than on individual subordinate agents. The developer specifies the target outcome for the overarching session, and the primary agent coordinates the necessary subagents to accomplish the work, delegating tasks and reviewing intermediate progress until it determines the objective is met. Gabe notes that there is currently no native mechanism to assign discrete, isolated sub-outcome definitions to individual subagents.

An audience member asks a follow-up about the platform's "dreaming" feature, wondering how the system learns when subagents drift off course or perform unintended actions. Gabe explains that dreaming is currently scoped to memory stores. The system periodically inspects the transcripts and accumulated memories generated across prior sessions, consolidating and refining them so subsequent agent runs know precisely which contextual knowledge to pull. While today this background consolidation is restricted to memory updates, Anthropic plans to extend continuous learning workflows to refine prompts, agent definitions, and skills over time.

As Gabe Cemaj explains regarding the execution model: “The outcome definition happens at like an overall arching session level so you define the outcome and you say this is what I want you to accomplish.” Later, outlining the scope of autonomous memory consolidation, he adds: “Basically the way dreaming works is it will analyze the transcripts and the memories created across multiple sessions so that it can update its memories so that future agents know what to pull.”

- Outcomes evaluate the macro-session rather than grading individual subagent steps independently. - The dreaming engine runs asynchronously across session transcripts to synthesize durable memories for future sessions. - Long-term roadmap goals for dreaming include autonomous self-tuning of system prompts, agent tools, and skill definitions.

[33:15](https://www.youtube.com/watch?v=FGftGOzfohc&t=1995s) Ingesting Private Enterprise Data: MCP Tunnels and Self-Hosted Sandboxes

A workshop attendee raises a common enterprise security question: if a company has internal proprietary data housed in an internal retrieval-augmented generation (RAG) pipeline, custom database, or command-line interface, what is the safest pathway to expose those resources to managed agents without leaking them to the public internet? Gabe outlines three distinct integration architectures depending on the organization's security posture and firewall restrictions.

If an organization can expose an internal service via an API or Model Context Protocol (MCP) server, they can register the MCP server with the platform and protect credentials using an authentication vault to mediate outbound requests. For teams strictly prohibited from exposing internal services to the public internet, Anthropic provides "MCP tunnels," an egress-free networking bridge that punches a secure tunnel from the customer's private corporate network directly into Anthropic's managed agent environment. Finally, for organizations with rigid compliance boundaries that cannot use managed compute at all, Claude Managed Agents supports self-hosted sandboxes, allowing teams to bring their own compute instances running inside their private virtual private cloud (VPC).

```mermaid flowchart TD accTitle: Enterprise Data Ingestion Options for Claude Managed Agents accDescr: Three architectural patterns for connecting internal data stores to Claude Managed Agents including Direct MCP, MCP Tunnels, and Self-Hosted Sandboxes.

subgraph AnthropicCloud["Anthropic Managed Environment"] Agent["Claude Managed Agent"] Vault["Authentication Vault"] end

subgraph EnterpriseNet["Enterprise VPC / Private Network"] DataStore["Internal Data & RAG Stores"] TunnelClient["MCP Tunnel Client"] PrivateSandbox["Self-Hosted Sandbox"] PublicMCP["Public/Exposed MCP Server"] end

Agent -->|"1. Secure API Call with Tokens"| Vault Vault --> PublicMCP PublicMCP --> DataStore

Agent <-->|"2. Secure Bidirectional Stream"| TunnelClient TunnelClient --> DataStore

Agent -->|"3. Orchestrates Tasks"| PrivateSandbox PrivateSandbox -->|"Local Network Peering"| DataStore ```

Highlighting the secure networking middle ground, Gabe Cemaj states: “If you don't want to expose your MCP to the internet, we have a feature called MCP tunnels which lets you create a tunnel from your network into ours so that you don't expose it to the internet and we would do it that way.”

- **Public MCP with Vault:** Standard remote MCP server where runtime secrets and API tokens are swapped securely at outbound boundaries via managed vaults. - **MCP Tunnels:** Reverse-tunnel architecture connecting on-premise or private cloud MCP tooling directly to Anthropic without opening inbound public firewall ports. - **Self-Hosted Sandboxes:** Fully managed by the enterprise, running directly within corporate network perimeters to preserve direct internal peering with enterprise services.

[34:27](https://www.youtube.com/watch?v=FGftGOzfohc&t=2067s) Environment Dependencies, Caching, and Sandbox Runtimes

The discussion transitions to container environments and package dependency management inside sandbox runtimes. An attendee asks whether environment dependencies are fetched dynamically at runtime during agent execution or pre-baked during sandbox provisioning. Gabe explains that environment dependencies are resolved upfront during environment creation time. The resulting execution environment is cached, enabling rapid re-instantiation across consecutive agent runs without incurring the latency penalty of pulling and reinstalling packages repeatedly.

When asked whether the sandbox can consume proprietary or internal dependencies that do not exist on public registries, Gabe clarifies that the automated provisioning hooks currently expect packages accessible via public registries such as npm, Cargo, or pip. However, developers needing internal libraries can upload private wheels, tarballs, or raw source files directly into the session workspace, or clone proprietary GitHub repositories directly inside the sandbox during initialization. As another attendee spots a "Deployments" tab in the console UI and asks for details, Gabe pauses the audience questions to steer the room back to the live coding demonstration.

Explaining how container setup latency is minimized, Gabe Cemaj notes: “It's at creation time, and then we'll cache it and reuse that environment across reinstantiations of the sandbox.”

- Sandbox dependencies are resolved at environment creation time rather than on every individual agent action loop. - Caching environments across sandbox reinstantiations avoids redundant package installations during multi-turn executions. - Standard public registries supported include npm, PyPI (pip), and Rust crates (Cargo). - Internal or private code can be loaded by uploading binary artifacts or files straight into the active session file tree or checking out source repositories directly.

## Notable quotes > [11:21](https://www.youtube.com/watch?v=FGftGOzfohc&t=681s) The whole point of Claude Managed Agents is that you can do long-running asynchronous tasks—not just for minutes, but hours, days, weeks, and even months. — Gabe Cemaj

> [01:04](https://www.youtube.com/watch?v=FGftGOzfohc&t=64s) All you have to do is start a session with the agent, send events, get back events. Everything else is managed by us. — Priyanka Phatak

## Mentioned Claude Managed Agents, Priyanka Phatak, Gabe Cemaj, Anthropic, Claude Code, Claude Agent SDK, Model Context Protocol (MCP), SEC EDGAR, Nvidia, Amazon Web Services (AWS), AWS Bedrock, Modal, Daytona, Vercel, Google Cloud Platform (GCP), Next.js, Bun, Python, Notion, Google Workspace, PyPI, Cargo, npm

## Video description From a one-line console agent to a research desk with skills, subagents, memory and self-grading outcomes.

NEXT UP: AI ENGINEER CODE SUMMIT Join us in SF for the second annual AIE CODE Summit, Nov 10–12, our last flagship summit of 2026. Applications live now: https://aie.codes

Priyanka Phatak finishes a no-code agent in the console, then Gabe Cemaj leads the hands-on build: a 'research desk' app with a chat UI on the session's event stream. He adds an SEC EDGAR skill, subagents, a custom tool and a shared memory store, then runs an outcome-graded analysis of Nvidia, where a grader checks the work against a rubric and Claude retries until it passes. Q&A covers vaults, the Agent SDK vs managed agents, guardrails, memory sharing and MCP tunnels.

This is part 2 of 4 of the Claude Managed Agents workshop from Anthropic.

In this video: • Building a chat UI on the session event stream • Outcomes: a task, a rubric and a grader that makes Claude retry • Vaults that swap placeholder secrets at egress • Shared memory stores and self-hosted sandboxes

SPEAKERS Priyanka Phatak, Member of Technical Staff, Anthropic (leads the Claude Managed Agents engineering team)

Gabe Cemaj, Member of Technical Staff, Anthropic GitHub: https://github.com/gcemaj

LINKS Claude Managed Agents docs: https://platform.claude.com/docs/en/managed-agents/overview Scaling Managed Agents: Decoupling the brain from the hands: https://www.anthropic.com/engineering/managed-agents

CHAPTERS 0:00 Q&A: how session storage changed 1:04 Console quick start demo 3:47 The workshop repo 4:41 What we're building 6:10 Workshop skill and project setup 10:16 Q&A: where managed agents run 11:52 Chat interface over the event stream 13:41 Skills, subagents and custom tools 17:41 Outcomes and self-supervision 19:05 Nvidia research session walkthrough 22:40 Updating skills as live artifacts 23:53 Q&A: vaults and credential injection 26:08 Q&A: Agent SDK vs managed agents 27:15 Q&A: guardrails for agent loops 28:04 Q&A: steering and session forking 29:14 Q&A: shared memory and lock-in 31:29 Q&A: grading subagents 33:16 Q&A: private data and MCP tunnels

Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI.

AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/

#Claude #AIAgents #AIEngineer

[Source](https://www.youtube.com/watch?v=FGftGOzfohc)

Reading summary

Anthropic engineers Gabe Cemaj and Priyanka Phatak present the architecture of Claude Managed Agents, demonstrating how event-driven streams, sandboxed execution, dynamic skills, and self-supervising outcomes enable persistent, multi-agent research workflows. Event Streams and Heterogeneous Sandbox Hosting (0:00) Claude Managed Agents decouples session mechanics, event ingestion, and storage by virtualizing boundaries across the event layer. Instead of traditional request-response REST patterns, the platform relies on bidirectional event streams: applications push user messages into a session ingress endpoint and ingest Server-Sent Events (SSE) in real time to monitor agent lifecycles, thoughts, and outputs for asynchronous tasks that can span days or weeks. Dynamic Skills, Context Isolation, and Memory (13:41) The platform structures agent capabilities through modular, live-updated skills and hierarchical subagent delegation. Skills function as dynamic artifacts; updating an integration—such as an SEC EDGAR query schema—instantly propagates to newly instantiated sessions without requiring a full harness rebuild. When spawning child subagents, the system passes tool descriptions and prompts but intentionally omits prior conversation transcripts to prevent context pollution and token bloat. To bridge disparate sessions, agents write to and read from persistent memory stores. Cemaj explains that Anthropic mitigates context fragmentation through an asynchronous consolidation engine termed "dreaming." In background passes, dreaming analyzes multi-session logs and intermediate notes to synthesize clean, unified knowledge bases across individual and organization-wide memory layers. Self-Supervised Outcomes and Runtime Steering Controls (17:41) Long-running tasks can be orchestrated using "outcomes," an architectural primitive combining an explicit task objective, an evaluation rubric, and an automated grading loop. When a primary agent believes a task is complete, a grader subagent evaluates the final deliverables against the rubric, initiating automated retries if criteria remain unsatisfied. Cemaj clarifies that outcome definitions currently operate exclusively at the overarching session level rather than grading subagents independently. For runtime interaction, human operators can perform live steering by injecting queued event messages directly into active sessions. However, Phatak and Cemaj acknowledge that state rewinding and "session forking"—branching a session's sandbox and history from an earlier turn—are not yet supported. Similarly, while native per-session token and monetary budget caps are on the development roadmap, infinite-loop guardrails and recursion limits currently depend on custom developer harnesses. Egress Vault Security and Private Data Tunnels (23:53) To prevent credential exfiltration during autonomous execution, Managed Agents utilizes an abstraction called Vaults. Agents manipulate only placeholder tokens within their execution sandbox; Anthropic's network proxy swaps in real API secrets at egress, and only when the request destination matches an explicit domain whitelist. Vaults also manage OAuth lifecycles for third-party SaaS integrations and Model Context Protocol (MCP) servers. For internal enterprise data, organizations can choose between public MCP servers routed through Vaults, egress-free MCP tunnels that bridge private corporate networks without open inbound firewall ports, or self-hosted sandboxes (such as Modal, Daytona, or AWS VPCs). Sandbox container dependencies are resolved and cached upfront at environment creation time, eliminating repeated installation latency across consecutive session reinstantiations. Claude Managed Agents abstracts runtime infrastructure, persistent memory, and egress security for autonomous research, but fine-grained subagent grading, session forking, and automated monetary budget ceilings remain active roadmap items that currently require developer-managed orchestration.

Source quotes

[11:21] Gabe Cemaj: The whole point of Claude Managed Agents is that you can do long-running asynchronous tasks—not just for minutes, but hours, days, weeks, and even months. [1:04] Priyanka Phatak: All you have to do is start a session with the agent, send events, get back events. Everything else is managed by us.

Original transcript

[0:01] [music] [0:12] Thank you for the presentation. Out of [0:14] curiosity for the design of the session [0:17] in um first iteration and second [0:20] iteration what has changed? [0:22] >> Good question. Gabe, do you want to take [0:24] that question on what changed within uh [0:26] the storage of the session between the [0:28] first and the second iteration? [0:31] >> Um, [0:33] sure. The the the main thing that [0:35] changed was that we wanted to abstract [0:37] away as many things and comp [0:39] compartmentalize them into different [0:40] components um to make it so that we [0:43] could extend the storage mechanisms for [0:45] future events that we wanted to store uh [0:48] without tying them you know so tightly [0:50] coupled to the mechanics of the session [0:52] work. Um, so we basically apply the same [0:55] principle of building and virtualizing [0:58] interfaces all the way down uh not only [1:00] across service boundaries but also at [1:02] the event layer. [1:04] >> All right, thank you. Um, so it sounds [1:06] like maybe everyone might not be able to [1:08] follow along, but hopefully um we can [1:10] find a way to share out these slides and [1:12] you can try it out at home or if you [1:14] have your phone, maybe you can try to [1:16] pair it with your hotspot and see if you [1:18] can follow along. Um, all right. So let [1:20] let's try this one more time. And we're [1:22] going to say, build me [1:25] an agent that tells me jokes every time [1:31] given a topic. [1:33] And then you just hit enter. [1:39] And you can see Claude is coming up with [1:42] a system prompt. It came up with a name, [1:45] a description, chose a model. You can [1:47] like go change this model if you want [1:49] to. and um give the agent access to the [1:52] agent tool set. And then you'll hit uh [1:55] create agent. [1:57] That should basically call the v1 agents [2:00] endpoint to create the agent. Okay, that [2:01] that got created. Next, we're going to [2:03] configure the environment [2:06] uh which is the template for the [2:07] sandbox. [2:12] Uh we're just going to create a new one [2:19] and then we'll just give it unrestricted [2:21] access for this one. But you can [2:22] obviously when you're building a [2:24] production agent you you want to tighten [2:26] the access control which domains it can [2:28] uh reach out to or specific MCP servers. [2:31] So we'll do unrestricted. [2:35] And then here you can see you can like [2:36] choose packages you want to install um [2:39] etc. And then next we'll just go and [2:41] start the session. [2:44] All right. Now we're going to go test [2:46] run our session. [2:48] Uh right. Tell me jokes about cats. [2:54] And you'll see the model [2:57] started thinking [3:01] and it came up with a cat joke. [3:06] Yeah. That that was like a that that's [3:08] how easy it is for you to go build your [3:10] agent. Now this uh agent is up and [3:12] running in the cloud. All you have to do [3:13] is start a session with the agent, send [3:16] events, get back events. Everything else [3:18] is managed by us. So we're managing all [3:20] the infrastructure, the hosting, the [3:22] harness, the sandbox management, the [3:24] scaling, the storage of your event log. [3:27] Um and then you can focus on building [3:28] your product around this. Um thank you. [3:31] I'll now hand it over to Gabe to do the [3:34] more hands-on bit of the workshop and [3:36] hopefully hopefully we have better luck [3:38] with Wi-Fi. Thank you. [3:42] [applause] [3:47] Can you hear me? Great. I see there's a [3:49] bunch of people that are kind of [3:50] squished in the back. There's room here [3:52] and here if you want to come in uh so we [3:54] can fit everybody in. Um, before we [3:57] start, here's a repo that we're going to [4:00] be using. Um, so if you want to take a [4:03] picture of the QR code, get clone that [4:06] repo. Um, cd into the research desk [4:10] directory and then run uh npm install or [4:13] bonai. Um, I'll let you do that and then [4:17] I'll walk you through through what we're [4:19] going to be building today uh together. [4:21] Um, [4:23] still see some phones out. I know you're [4:26] there's Wi-Fi issues. Hopefully, you can [4:29] you can get uh you can get to the repo. [4:31] Um Okay. Oh. Oh, I see you. Okay. [4:37] Cool. [4:39] All right. So, what are we what are we [4:42] going to be building today? We're I'm [4:43] going to show you how you can use cloud [4:45] manage agents to build a variety of [4:46] different applications. We're going to [4:48] start with uh building a custom [4:50] chat-like interface so that you can talk [4:52] to an agent. We can say hello. Um, then [4:55] we're going to augment that agent into a [4:57] multi- aent uh ecosystem so that it can [5:00] delegate tasks uh to sub agents. We're [5:03] going to give it skills uh specifically [5:05] a skill to be able to call um the SEC [5:08] website and get SEC filings for [5:10] individual tickers uh and analyze [5:12] specific forms. Um then we're going to [5:15] be able to use the these new enhanced [5:17] agents to run more asynchronous um [5:21] agentic tasks that can take a long time [5:23] to run. Um and then we're going to bring [5:26] it all together by enabling the sort of [5:28] map reduce interface where we can talk [5:31] to an agent and that agent can then [5:32] delegate to many agents uh and do sort [5:35] of like a map produce thing. um showing [5:38] you how you can build a custom harness [5:40] on top of CMA to orchestrate you know [5:43] your own work with custom tools uh and [5:45] manage multiseession um kind of [5:48] interactions. Uh and then we'll finish [5:50] up by I'll show you uh our deployments [5:52] API that allows you to set up recurring [5:55] work on a schedule so that you can run [5:57] uh sessions uh on a on a given cadence [6:00] to run kind of longer term analysis that [6:02] keeps updating itself. Sound good? [6:07] quiet. Okay, great. Um, all right. So, [6:10] let's switch over to um a coding [6:13] interface. I'm going to be using cloud [6:15] code for this. You're free to use [6:17] whatever coding agent you prefer. Uh, I [6:19] don't think anybody's writing code by [6:21] hand, so I'm not going to do that [6:23] either. Um, there is a skill that we've [6:26] loaded up in your repository that's [6:28] going to walk you through the workshop [6:30] step by step. Um, so if you want to [6:32] rerun the workshop after I'm done, uh, [6:35] later on when you have better Wi-Fi and [6:37] better internet access, uh, you're able [6:39] to do so, um, at home. Um, so the first [6:42] thing you're going to want to do is I [6:44] assume you've checked out the [6:45] repository. You've cded into the [6:47] research desk repo. You've run buni. [6:50] We're going to spin up our uh, server [6:53] with uh, bun rundev. I'm going to make [6:54] this bigger. I'm assuming nobody can [6:56] see. Okay. Is that better? [6:59] >> Yeah. And then if you have questions [7:01] while Gabe Gabe is presenting, feel free [7:03] to raise your hand and I can run to you [7:05] with the mic. [7:05] >> We we have helpers. So if you run into [7:07] issues, raise your hand. People will [7:09] come um help you help you out. Um cool. [7:13] So if you've gone this far, you'll [7:16] basically be able to load this website. [7:18] Uh and the first thing you're going to [7:19] want to do is go to the setup page. Um [7:23] everybody with me so far? [7:25] >> Yeah. Okay. Um now we're gonna [7:30] Yeah. [7:32] Um, okay. Once uh once uh he takes a [7:35] picture, we're we're going to start up [7:37] uh our workshop skill. So in cloud code, [7:40] you can type in uh workshop start. Uh [7:44] and it's going to start running through [7:46] the workshop for you. We're going to do [7:47] it in six acts. Uh I apologize there's [7:50] no intermission in between. Um I hope [7:52] that's okay with y'all. Um but we're [7:55] going to go through it. Uh Claude will [7:58] uh sure [8:01] we're going to go into auto mode. Um [8:03] Claude Claude is going to keep track of [8:04] each of the acts so that you can [8:06] continue on. You can keep typing [8:08] workshop next so that it you know it's [8:10] going to keep uh progressing through the [8:13] through the thing. Uh I'm going to ask [8:15] Claude to do it for me but handhold [8:17] handhold me and teach me what it's [8:19] doing. Um and so the first thing it's [8:22] going to basically make sure that our [8:23] prerequisites are set. Um, it's going to [8:26] copy the m.example file into amp.local. [8:31] Um, oh no, stop. We're not there yet. [8:35] Okay, it's a little eager. Okay, it's [8:37] going to copy that file. That file um [8:40] has a few values for you to fill in. [8:41] You're going to want to paste in your [8:42] API key. Um, you're going to want to [8:45] paste in the SEC wants to know who you [8:47] are when you're calling their API. So, [8:49] you're going to have to put in your name [8:50] and email. Um, this is just so that they [8:53] know who's calling them. Um, and then [8:56] that's basically it. Once, um, once [8:59] you've done all that, um, it's going to, [9:03] well, we're going to move on to the to [9:04] the next thing. So, everybody with me so [9:07] far, you've put in your API key into [9:09] your.local, [9:11] you put in your Edgar identifier. [9:14] >> Show of hands, [9:14] >> be a quick show of hands. Or how many of [9:16] you are able to follow along who might [9:18] have Wi-Fi or [9:20] >> Okay, we got one one person at least. [9:21] So, [laughter] [9:25] MVP, pardon. [9:29] Uh oh. Oh, for the repo. This one. [9:36] >> Uh Rohan, you want to hand the mic? [9:39] >> Yeah, thank you. [9:39] >> Yeah, the slide with the credential. [9:41] That's it. The setup. [9:42] >> This one. [9:42] >> Thank you. [9:42] >> Okay. Sure. And then and then given [9:44] given people don't have Wi-Fi plea [9:46] please like stop us and ask questions so [9:48] we can at least make this more [9:49] interactive and help you understand [9:51] concepts so that you can build with your [9:53] own time after the workshop. [9:58] >> Okay. Um I'm yeah I have a [10:05] >> yeah this is the app spinning up uh when [10:07] you clone and cd into research desk and [10:10] run bun run rundev. This is a website [10:13] you will you will get. [10:15] >> Can I ask a question? [10:16] >> Yeah, I don't see you but [10:18] >> Oh, you're here. [laughter] [10:19] Uh I'm wondering like you know so when [10:22] you say cloud handle and manage the [10:24] agents where is this running? Is this [10:26] running on AWS? [10:27] >> Yeah, that's a good question. Um so we [10:30] we we're going to run it in our own [10:31] infrastructure. Um today we we have like [10:34] a onep offering uh that we run on but we [10:38] also have cloud platform on AWS if you [10:41] want to run on AWS using your AWS [10:43] credits. What we also offer and maybe we [10:45] can talk about this after this demo is [10:47] self-hosted sandboxes. So if you want to [10:49] bring your own sandbox through uh [10:51] different providers like Burcell or [10:53] Modal or Daytona or a bunch of others, [10:56] you can do that. Uh even you know [10:59] through Google cloud and and AWS if you [11:01] want to run your sandbox in that uh [11:02] environment, you can. Um okay, I'm going [11:06] to continue with the demo. U the the [11:08] first thing you the next thing you want [11:09] to do is create push this button which [11:11] is going to create um your agent. So [11:14] it's going to create two things for us. [11:17] um an environment which Priyanka talked [11:19] about is a template of what the sandbox [11:21] is going to look like. Um for this demo, [11:24] we're going to give it access to a few [11:26] websites that it can reach out to, [11:28] mainly the SEC. We're going to add uh [11:30] some Python packages so that it can [11:32] reach out to to the SEC with you know [11:35] some predefined uh Python packages. Um [11:38] it's also going to create uh an agent. [11:41] So it's going to create this very basic [11:43] agent. Right now it has no tools. Um, it [11:46] has uh just a prompt. Uh, well, it has [11:49] the basic tools, but there's no skills. [11:51] There's no sub agent coordination, [11:52] nothing like that. Um, I'm going to tell [11:55] Claude to [11:58] um continue and implement the interface [12:02] with which we can talk to this agent in [12:04] our application. Uh, like Bianca [12:06] mentioned, the main uh way you [12:08] communicate with a session is through an [12:10] event stream. So you listen onto that [12:12] stream for all the events that the agent [12:14] is pushing back down to you and you can [12:16] use that uh an endpoint to push events [12:18] back up. So really everything that [12:20] happens in the communication for these [12:22] more chat-like applications happens on [12:25] that stream. So we can see here what [12:27] Claude is doing is it's uh implementing [12:31] the section so that when I press enter [12:33] in my chat box, it's going to send a [12:35] user message event with the content that [12:38] I'm sending to that session. And then [12:40] it's going to route the SSE stream back [12:43] onto my browser from my back end. So [12:45] it's going to listen onto um the events [12:48] that are coming in the stream connection [12:50] to our API and then reream them directly [12:53] back onto the front end so that I can [12:55] render them in a nice way. Um and once [12:58] Cloud has done that, we should be able [13:00] to go to this desk tab and just say hi [13:04] to our agent. Um, and if everything [13:07] worked [13:10] hopefully great. Um, so we can see here [13:13] now we're we're talking to our agent. If [13:15] you wanted to build a chat-like [13:16] application, this is where you would [13:18] stop. Um, which is great, but we want to [13:21] do more. The whole point of cloud [13:23] managed agents is that you can do [13:24] longunning asynchronous tasks. Um, not [13:27] just for minutes, but hours, days, [13:29] weeks, and even months. Um, so let's see [13:33] what we we have to do next. Right. The [13:35] next thing that we're gonna want to do [13:37] is um Oh, Claude is very eager today. [13:42] Um, okay. The next thing is we we want [13:45] to augment our agent, right? We want to [13:47] give our agent a skill so it can call [13:48] the SEC. We want to use multi- aent [13:51] coordination so that um our session can [13:53] delegate work to sub agents that are [13:55] more specialized and capable of doing [13:57] some tasks. Um, and we want to basically [14:00] move up the ladder of intelligence. Um [14:04] so let's do that um by saying workshop [14:07] next. Um now what claude is going to do [14:10] is it's going to implement the [14:11] provisioning scheme to upgrade our agent [14:14] so that it uses uh more sub aents or any [14:18] sub aents um is we're going to give it a [14:22] skill a skill that lets it uh um talk to [14:26] the SEC. We've described how to use the [14:28] Python package. We've described that it [14:30] has to use the Edgar identity to call. [14:32] Um, we've we've described all of that [14:35] and we're going to give it a custom tool [14:37] uh for later in the workshop. Um, that [14:40] custom tool is what we're going to be [14:41] able to use to tell our head of research [14:44] agent to spin off multiple sessions that [14:46] we will coordinate uh for a sort of [14:48] custom harness. So, this is what Claude [14:50] is doing here. Um, it's updating its [14:53] progress, creating this custom tool [14:54] called dispatch analysis. We're going to [14:56] come back to this in a second. um we [14:59] give it a schema. Um it's going to [15:02] update the the description of the main [15:05] agent so that it creates a new version [15:07] of that agent. Uh this new version will [15:10] have um oh pager duty. Um [15:15] uh this new version will have uh a few [15:17] different sub aents uh that it can [15:19] delegate work to as well as a reference [15:21] to itself. So if the main agent ever [15:24] wants to spin off a copy of itself with [15:26] its same prompt, it can do that. Um this [15:28] is important because uh sometimes you [15:31] don't necessarily know what sort of [15:33] tasks your session is going to want to [15:35] do, but you have a general idea that you [15:38] want to have sub agents that you know [15:40] spin up and do work. Um so you can [15:42] define a self reference to you know [15:44] yourself that clock can then spin up a [15:46] sub agent for. Um important this [15:49] question has come up in the past. It [15:51] doesn't copy the transcript of that [15:52] session into the sub agent. It just [15:54] copies over the tool description and the [15:57] schemas uh and the prompt uh into into [16:00] that sub agent. [16:02] Um cool. So now if we go into here, we [16:09] should see that there's a new agent. Oh, [16:12] I think I need to provision them. So [16:13] there should be a button in our setup uh [16:16] that says staff the desk. When I do [16:18] that, it's actually going to run Oh. [16:21] Uh uh okay. I ran this before. Let me [16:24] delete the skills so that I can recreate [16:26] it for you. Um I was trying to do the [16:29] magic uh cook show thing, but that's [16:32] fine. Um okay. So we'll delete the [16:36] skill. We'll ask cloud to reprovision [16:37] it. [16:39] Um it's going to create all those [16:40] resources and then we'll be able to see [16:42] here all the sub aents, the skill that [16:44] we created, um a memory store. So for [16:47] these sessions, we we we also support um [16:51] longunning memory store that can be [16:52] shared across sessions. Uh and we're [16:54] going to use that so that different [16:56] sessions can learn about different [16:57] financial instruments and they can share [16:59] that knowledge uh with each other. And [17:01] at the end of this workshop, I'll show [17:02] you how you can use that memory [17:04] longunning memory to keep improving your [17:06] agents long term over many many uh [17:08] sessions, days, months, years. [17:12] Um cool. So we should see here that the [17:15] agent now in here [17:19] um [17:21] has the skill that we described. It has [17:23] sub agents uh with their own prompts and [17:26] tools and permissions. Uh it has a [17:28] prompt and it has a model. Um [17:32] and I think we're we're ready to use [17:34] this new dedicated sub agent to do our [17:36] first asynchronous longunning task. So [17:40] let's uh tell Cloud to move on to the [17:42] next thing. The next thing that we want [17:44] to do is um we want to tell Claude not [17:47] in a chat-like application but a more [17:49] asynchronous thing. Hey, research this [17:52] financial instrument and give me a [17:53] report in a format that I define and I [17:56] care about. And we're going to use [17:58] something called outcomes. Outcomes is [18:00] how we've created self-s supervision. So [18:02] in an outcome, you tell Claude, "This is [18:05] your task. This is your rubric to [18:07] evaluate your task. It's going to go off [18:09] and do that work and then when it thinks [18:11] it's completed, it's going to ask a sub [18:13] agent to look at the rubric and the work [18:16] that Claude did uh and evaluate did it [18:18] do it correctly and did it do it to [18:20] completion. Uh if it does then great and [18:23] if it doesn't it will try again. Um so [18:26] here what Claude is implementing for us [18:28] is basically this userdefined outcome [18:31] event. So that anytime we start a [18:33] session in our UI that I'll show you for [18:36] a specific ticker, it's going to send [18:38] this userdefined outcome with a specific [18:41] rubric uh max iterations um and a [18:44] description of the task at hand. And so [18:47] then Cloud will go run that task for us. [18:50] Um and so what does that look like in [18:52] the UI? If you look to this scorecard, [18:56] um I can tell Claude, hey, I I want to [18:57] analyze Nvidia. Oh, [19:01] please. [19:06] Okay, great. So, it's going to kick off [19:07] a session to do the analysis on Nvidia. [19:10] Um, this is going to take a while. I'm [19:12] going to do the cook show thing and show [19:14] you a previous session um of of what [19:16] happened. Uh, it's going to take about [19:17] 10ish minutes. Um, but here's something [19:20] I ran yesterday. Uh, the first thing [19:23] we're telling Claude is here is an [19:25] outcome. I want you to analyze Nvidia's [19:27] most recent uh SEC quarterly filings. Um [19:31] you're going to use Edgar identities. [19:33] This is where your Edgar identities are. [19:36] Um you have memory. So check if you've [19:38] already done analysis previously and [19:40] make sure that you use that instead of [19:42] refiling and researching things that you [19:44] already know. Um I want you to find the [19:47] most uh recent form that I care about [19:51] and run the analysis over that. Um and [19:53] then I want two deliverables. I want you [19:55] to write a note in in my memory um for [19:59] that specific company and I want you to [20:01] write a scorecard in a well- definfined [20:03] structured output and I want you to save [20:05] that in a specific file location. Um so [20:09] then Claude went off to work started [20:12] calling tools. It didn't find any notes [20:14] in its memory store. Um it uh looked in [20:19] a few different places and then it [20:20] decided okay I know what to do. I'm [20:22] going to leverage my sub agents to do [20:24] the bulk of this work. Um I'm going to [20:27] spin up a financial um expert analysis [20:31] agent uh or extractor and then a risk [20:34] evaluator analysis. And you can see all [20:36] of that in the uh console here for full [20:38] observability. Each of these lines here [20:41] represents a thread and a thread [20:43] represents the work done by a sub agent. [20:45] And so each of the sub aents went off to [20:47] work and then when they were done they [20:49] let the main agent know hey this is what [20:51] you asked of me uh and then the main [20:53] agent woke up uh got that information [20:55] from the sub agents and then uh filed [20:58] everything it needed to do and created [21:00] the deliverable. It created the JSON [21:02] file that you know we specified uh and [21:05] it made sure to update the memory and [21:07] then the cell supervision started. So it [21:10] kicked off a sub agent to verify that [21:12] the outcome that we defined is being [21:14] met. Um it looked at the rubric. It [21:17] analyzed all the work that was done uh [21:20] in the session and analyze the artifacts [21:22] that it created. Uh and it decided in [21:25] the end that this was satisfied that [21:26] this you know scorecard was delivered uh [21:29] that the numbers are well defined uh [21:31] that the memory update you know was [21:33] correct um and everything looked good. [21:36] Um I don't I think this is still running [21:38] but once that completes uh you would see [21:41] a table here of each of those tickers uh [21:44] because we can extract the report the [21:46] scorecard from the session and download [21:48] it. You put in a database for this demo [21:50] we put it locally. Um but I think you [21:53] get the idea. [21:55] Um okay this is great but what if you [21:57] want to do a hundred or or 10 tickers at [22:00] a time? Wouldn't it be great if I could [22:02] chat with an agent to build a custom [22:04] harness that could then delegate work uh [22:07] for each of the tickers into a different [22:09] session and then when it's all done [22:11] aggregate the results and give me uh a [22:13] bunch of scores. Um so that's that's [22:16] what we're going to do next. Um we're [22:18] going to leverage that tool that custom [22:20] tool that we created to build our custom [22:22] orchestration harness in our server. um [22:25] where when we talk to our main agent at [22:27] some point that main agent is going to [22:29] decide hey I want to spin up uh 10 [22:32] different sessions to do work um we'll [22:35] get that oh I skipped a step sorry um [22:39] for we're going to update the skill uh [22:41] first u so I I wanted to show you that [22:44] these skills are are live artifacts that [22:47] you can update um so you can improve the [22:49] skill uh in this case uh we're adding [22:52] the ability to look at a new form Uh and [22:54] then when you do so any new sessions [22:57] that the reference at skill will get [22:59] that new information. Um so this is [23:02] something that you can continuously [23:03] update. You can look at previous [23:04] sessions to see how they performed. [23:06] We're going to see that at the end of [23:08] this uh demo that that happens with with [23:11] something we call dreaming. Um but yeah [23:14] this this is what is happening. Um okay [23:17] I'm going to move on to the [23:18] orchestration bit. Um, so now that yeah, [23:23] that basically had skills published. Um, [23:25] I think if I go here, we should see [23:28] >> any questions so far. Um, I know this is [23:30] a lot to take in. Uh, Travis, there's a [23:33] question over here. [23:42] >> Um, and I I know someone came up to me [23:44] asking about like how long it takes to [23:46] get the credits. It should be only five [23:47] minutes. Um, I know the form says one to [23:50] two days, but you should see it within [23:52] five minutes. [23:53] >> Um, I was curious about the the data [23:55] access uh from the SEC that the the [23:58] research agent is grabbing. Does that [24:00] require some sort of credentials? And [24:02] are you going to walk through the part [24:03] of how the how that credential injection [24:06] or or brokering works? [24:08] >> Yeah. So, for for the SEC, it just [24:10] requires your email and your name. Um, [24:12] for this demo, we just inject that in [24:14] the prompt. But in a real production [24:16] application, we would use something [24:17] called vaults. Uh the way vaults work is [24:19] you define a secret uh that you want to [24:22] you inject in an environment variable. [24:24] And instead of putting the real secret [24:26] in, what we put is is a placeholder. Uh [24:29] and then when Claude is doing work and [24:31] it wants to use that environment [24:33] variable, um it just injects the [24:36] placeholder value. We at egress swap it [24:38] out for the real value. only on the host [24:41] domains that you specified are allowed [24:44] for that secret. So if you say uh my [24:47] super secret MVAR can only be used for [24:49] the SEC and then cloud tries to use it [24:51] for entropic.com we wouldn't inject we [24:54] wouldn't do the swap but when it tries [24:56] to call the sec.com it would do the swap [24:58] for that [24:59] >> is essentially like an API where you can [25:02] use it on console where you just go [25:03] create a vault give it a name you'll get [25:05] an ID and then you essentially include [25:07] that ID in your session as a way for us [25:09] to authenticate with your MCPS your CLI [25:12] you for example you could do like a [25:13] Google Workspace CLI that you install on [25:15] your um sandbox and then you can give us [25:18] like your API key as your environment [25:19] credential and again we'll we'll kind of [25:21] follow the same method that [25:22] >> uh Gabe talked about. So that that [25:24] allows us to [25:25] >> essentially [25:26] >> bots also work with ooth credentials [25:28] forps and they also work [25:30] >> Gabe I don't know if you want to just [25:31] show it on on console like the quickly [25:34] like the vault creation. [25:36] >> Yeah. Um so you you can create a vault [25:38] here and then in a vault you would [25:40] bundle credentials. Um so you can have [25:42] MCP O um maybe you do notion you know we [25:46] have a few predefined things for OOTH [25:48] things um so whenever you reference a [25:53] session you inject that vault ID and it [25:55] will bundle all the credentials into [25:57] your session and whenever it tries to [25:59] use any of the domains that you [26:01] specified in your credentials it will [26:03] swap for the real um credential instead [26:05] of the placeholder. [26:08] >> Yeah. Um, quick question. What part of [26:10] the capabilities in the cloud agents SDK [26:14] and what's about it? [26:16] >> Can you what what do you what do you [26:18] mean? [26:18] >> You have the agent SDK, right? I [26:20] presumably this is built on top of the [26:22] agent SDK. I want to know what part of [26:24] the capabilities are in the agent SDK [26:26] layer and what's outside of it. [26:27] >> So the the agent SDK is packaged for uh [26:31] like more more it's a headless cloud [26:33] code version. So it's actually we've [26:35] built a generic version of that. We've [26:37] took taken a lot of inspiration about [26:39] the benefits and lessons from the agent [26:41] SDK, but we've rebuilt uh the harness to [26:44] be a more generic not just coding [26:46] specific thing. [26:47] >> So it's sort of a fork. [26:49] >> Uh yeah. Yeah. It's basically like its [26:51] own harness. Agent SDK is basically like [26:54] a cloud code in the form of a binary [26:56] that you can take and run wherever you [26:58] want. This is a fully hosted solution [27:00] running in our cloud and we have our own [27:02] harness that's heavily inspired by cloud [27:05] codes harness and we frequently stay [27:07] stay in touch for like any enhancements [27:09] across the board. [27:10] >> Uh I I saw another hand in the in the [27:12] back over there. [27:13] >> We have a question here. Yeah, [27:15] >> couple more. [27:15] >> Thank you. Uh I guess the question as [27:17] you're orchestrating between multiple [27:18] agents um is there some guard rails that [27:21] those agents would not go into loops and [27:23] like consume a lot of tokens um as [27:26] they're generating or working together? [27:28] >> Um some of that we leave up to you to [27:32] orchestrate and figure out. I mean we're [27:34] if you're building these sort of custom [27:35] harnesses, there's nothing stopping you [27:37] from using uh like other providers if [27:41] you wanted to do a sort of multi- uh [27:43] multi-provider system. Um, we are [27:47] exploring the idea of what it means to [27:48] do like budget tighter budget controls [27:50] that you can say use no more than this [27:52] amount of money per session, use no more [27:54] than this amount of tokens per session. [27:56] Um, but that's not something we offer [27:57] today. [28:00] >> Um, yeah, go ahead. [28:01] >> Couple more hands. [28:03] >> Question. Uh, earlier you were showing [28:05] us in console kind of a transcript of [28:07] the manager and then the sub aents and [28:10] you could kind of go through and see [28:11] what they were doing. if you were to [28:13] come across maybe uh a mistake or like [28:16] it was going down the wrong path or you [28:18] wanted to change history. I mean [28:20] obviously that's what's happened. Is [28:21] there some way to correct it or to kind [28:24] of go back to that point in time and [28:25] kind of start over try it with some [28:26] additional guidance? Is there any [28:28] anything like that in here? [28:30] >> You can steer it. So I could I could uh [28:33] I think here if I say send event I could [28:35] send a new user uh message event and [28:38] steer the agent. Uh it'll queue it up. [28:40] it's, you know, you can send that 10 [28:42] messages at a time and it's fine. We [28:44] don't currently have a way for like [28:46] rewind and roll forward. Um, but that's [28:49] definitely something that uh if if [28:51] people are interested, we we would look [28:52] into [28:52] >> Yeah, session forking is essentially [28:54] what you're asking for. We we've had a [28:56] couple a few customers who have asked [28:58] for it. That's it's it's on our road [29:00] map. So hopefully we'll have it at some [29:01] point. [29:02] >> Thank you. [29:03] >> Uh yeah, I see a couple hands here if [29:06] you want to hand. Yeah, thank you Rohan. [29:08] Uh so [29:11] >> where are you? Okay. Okay. Yeah. Go [29:13] ahead. [laughter] [29:14] >> Assuming all of this is running in the [29:16] cloud like like you know u you guys have [29:19] any road map to get collective knowledge [29:22] across these agents like let's say in [29:24] this one particular application you have [29:25] 10 users. [29:26] >> Would would you mind speaking up? It's a [29:27] little hard to hear. Oh, so so if you [29:32] assuming all of this is running in the [29:33] cloud and this application is being [29:35] distributed across tens and hundreds of [29:37] users and all of them are using it for [29:39] the purposes of like you know doing this [29:40] financial research. Do you have I know [29:43] you guys have session logs. Is there a [29:45] collective [29:46] >> memory that you guys are looking into or [29:50] even have a road map of? [29:52] >> We have it today. We actually used it in [29:54] this demo. So we use the a memory store [29:57] um that allows you to share knowledge [29:59] across sessions. Uh and you can see here [30:01] that what's happening is um the agents [30:04] are each populating individual [30:06] information about what they've done so [30:09] far. Um and then I'll show you at the [30:11] end how you can consolidate that [30:13] knowledge over time using dreaming so [30:15] that you can continuously improve the [30:17] memory stores as different agents run [30:19] into different edge cases and [30:20] collectively learn from each other. So [30:22] just to follow up on this so you're [30:25] storing this in anthropics [30:26] infrastructure. Say if you want to [30:28] delineate that and and have things [30:30] running in AWS bedrock and I have my own [30:33] like uh data clusters or something for [30:36] sandboxing. Then you mentioned about [30:38] self-hosting. How much of a lockin is [30:41] this piece versus like how flexible is [30:44] it? So all of this API based so you you [30:47] can extract the memories and put them in [30:50] different systems if you want to. We [30:52] don't currently support out of the box [30:54] other memory providers in the session [30:57] but you would just build an MCP or a [30:59] custom tool to do so. It's the way I [31:01] would imagine doing it. And [31:02] >> and also to add to the initial question [31:04] about like having user level org level [31:06] memory we do allow for attaching [31:08] multiple memory stores. So you can [31:10] imagine having a user level memory store [31:12] and then having some sort of orwide [31:14] memory store and then you bring both to [31:16] each session. So you can start [31:17] collecting user level as well as or [31:19] level memories and then like Gabe said [31:22] none of this is locked in. You can like [31:23] take your data and go anywhere else if [31:25] you wanted to. Everything is an API [31:27] call. [31:29] >> Yeah. [31:29] >> Um I had a quick question as you walk [31:31] through the agent workflow. I noticed [31:33] that the grader is at the main agent [31:36] level, right? like so how do you grade [31:39] these sub aentic tasks that are [31:41] happening and then how do you manage [31:43] memory across those as well [31:44] >> right [31:45] >> that's a good question so uh there's two [31:47] dimensions to this sub aents in the way [31:49] that we've built them are meant to uh [31:52] augment your context and leverage the [31:54] capabilities of your of your session by [31:57] collaborating between them uh the [32:00] outcome definition happens at like an [32:02] overall arching uh session level so you [32:05] define the outcome and you say this is [32:07] what I want you to accomplish. The main [32:08] agent then uses you know uses sub agents [32:11] to accomplish that work. It collaborates [32:13] between them and it eventually says uh [32:15] I'm done. Um there is no currently [32:18] support for like a sub outcome [32:20] definition. [32:22] >> Quick followup like so as you have this [32:25] dreaming feature right like so how do [32:27] you learn like if the sub agents have [32:29] some sort of you know uh you know offc [32:31] course sort of actions then how do you [32:33] sort of learn from that? Yeah, uh today [32:35] the dreaming is constrained to uh memory [32:37] stores. So I I'll show you that uh uh in [32:41] a second, but basically the way dreaming [32:43] works is it will analyze the transcripts [32:45] and the memories created across multiple [32:47] sessions uh so that it can update its [32:50] memories so that future agents know what [32:52] to pull. Um but you can imagine that we [32:54] can continue to expand that for skills [32:56] and for prompts and for agent [32:58] definitions so that you can you have [33:00] this continuous learning uh [33:02] functionality. But right now we just [33:03] have it for for memory stores. [33:07] >> See a hand here. And then I'll I'll go [33:09] back to [33:10] >> Hello. [33:11] >> Yeah. [33:12] >> Here. [33:12] >> Where where is here? [33:13] >> Here. [33:14] >> Oh, hi. [33:15] >> Thank you. Uh if you have your let's say [33:18] company data in either I don't know [33:21] traditional store or rag or whatever and [33:23] you want to expose it to this type of [33:25] agents through a custom CLI or MCP. what [33:29] would be the safe path to put it in your [33:32] environment? [33:32] >> Yeah, that's a great question. There's [33:34] two two ways to do this. If you are [33:36] using if you can access that uh via CLI [33:39] with an API or via uh an MCP, uh you [33:42] would just create an MCP, give it uh [33:45] give it the authentication vault uh and [33:48] it would just be able to call your MCP [33:50] directly. If you don't want to expose [33:52] your MCP to the internet, we have a [33:53] feature called MCP tunnels which lets [33:56] you uh create a tunnel from your um [34:00] network into ours so that you don't [34:02] expose it to the internet and we would [34:04] do it that way. If if that's uh too big [34:07] a barrier to climb, we do support [34:09] self-hosted sandboxes so that you can [34:10] bring your own sandbox which has which [34:13] is peered with your network and has [34:14] access to your applications and services [34:17] and the agent would just call the [34:19] sandbox that you manage and host um and [34:22] be be able to access your your resources [34:25] that way. [34:27] >> Yeah. [34:28] >> Um I had a question on dependencies for [34:30] the environments. So I saw something you [34:32] could load in dependencies. Um is that [34:36] at runtime or right before um it's [34:39] created? It [34:40] >> it's at creation time. Uh and then we'll [34:42] cache it and reuse uh that environment [34:44] across like re reinstantiations of the [34:48] sandbox. [34:49] >> So can you specify like internal [34:51] dependencies or do you have to pick mpm [34:54] package? [34:54] >> Right. right now uh they are all they [34:57] all have to be accessible via npm cargo [35:00] uh pip whatever but you can upload files [35:03] uh directly into um the session if or [35:07] GitHub repos if you want to install them [35:09] that way [35:10] >> okay and one more thing uh deployments I [35:12] see a tab for that [35:13] >> yeah yeah I'm going to get to it I'm [35:15] going to continue to the with the demo [35:17] and then we'll we'll do more questions [35:18] in a