Anthropic Limits Claude Multi-Tenant Data Isolation to Workspaces

Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic — AI Engineer

AI Engineer

Enterprise developers must provision separate workspaces per customer via continuous-integration pipelines to guarantee data segregation, as Claude Managed Agents enforces security boundaries only at the workspace level rather than inside shared sessions.

AI-assisted summary of Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic — original source. Check the original for full context.

Source published:

Key ideas

  • Anthropic manages durable agent definitions declaratively as YAML files in source control, deploying them via an internal infrastructure-as-code CLI wrapper called ant. (0:13)
  • Internal evaluation harnesses at Anthropic spin up over 10,000 parallel cloud agent sessions to validate prompt changes and pre-release model weights against curated benchmark datasets. (2:45)
  • Claude Managed Agents enforces strict execution pausing during custom tool resolution; dispatching new user messages during this blocked phase triggers an HTTP 400 error unless handled via a coordinator-subagent architecture. (1:10)
  • Specialized compute, including GPU acceleration and expanded memory limits, requires connecting external self-hosted sandboxes such as Modal, E2B, or Daytona rather than relying on default cloud infrastructure. (12:10)

Detailed notes

## Background Published on YouTube by AI Engineer on October 12, 2026, this video documents the final segment of Anthropic's four-part Claude Managed Agents workshop held at the AI Engineer World's Fair in San Francisco. The session features Anthropic technical staff members Priyanka Phatak and Gabe Cemaj fielding audience Q&A on managing agent configs via CI/CD, tenant isolation, custom compute, evals, and handling asynchronous tool calls.

## TL;DR In this closing Q&A, Gabe Cemaj and Priyanka Phatak outline architectural patterns and operational best practices for deploying Claude Managed Agents at scale. They explain how teams can manage configurations declaratively via the `ant` CLI in CI/CD, enforce customer tenant boundaries across workspaces or session metadata, and connect self-hosted sandboxes (like Modal or Daytona) for GPU-heavy tasks. They also share Anthropic's internal eval methodologies, explore automated pull request reviews, and detail a coordinator subagent pattern that prevents HTTP 400 blocking errors when handling long-running custom tools.

## Atmosphere & style The session carries the candid, highly pragmatic energy of a technical conference deep dive led by the engineers who actually built the platform. Gabe Cemaj and Priyanka Phatak trade answers seamlessly, speaking in a direct, unvarnished style that treats edge cases, API quirks, and platform boundaries with absolute transparency. It feels like an open office-hours discussion where complex enterprise architecture problems are swiftly distilled into actionable design patterns.

## Bulletin - [00:13](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=13s) Anthropic manages durable agent definitions declaratively as YAML files in source control, deploying them via an internal infrastructure-as-code CLI wrapper called ant. - [02:45](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=165s) Internal evaluation harnesses at Anthropic spin up over 10,000 parallel cloud agent sessions to validate prompt changes and pre-release model weights against curated benchmark datasets. - [01:10](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=70s) Claude Managed Agents enforces strict execution pausing during custom tool resolution; dispatching new user messages during this blocked phase triggers an HTTP 400 error unless handled via a coordinator-subagent architecture. - [12:10](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=730s) Specialized compute, including GPU acceleration and expanded memory limits, requires connecting external self-hosted sandboxes such as Modal, E2B, or Daytona rather than relying on default cloud infrastructure.

## Key takeaways - [00:13](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=13s) Agent configurations can be version-controlled in YAML files and synced across environments in CI/CD pipelines using the ant CLI. - [01:10](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=70s) Dispatching messages to an agent awaiting custom tool output produces an HTTP 400 error because the execution loop pauses to guarantee message ordering. - [02:45](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=165s) Anthropic tests pre-release models by spinning up over 10,000 parallel cloud agent sessions across both generic and coding benchmarks. - [03:42](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=222s) Poorly framed agent evaluation benchmarks often yield deceptive 100% pass rates, making real-world session logs the preferred foundation for test datasets. - [04:42](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=282s) Platform workspaces serve as the primary security boundary; all API keys inside a given workspace share access to its agents, memory banks, and MCP tools. - [06:29](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=389s) Anthropic automates pull request reviews by running multi-session agent swarms to catch regressions and post line-by-line diff commentary. - [07:29](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=449s) Live network tunneling into hosted sandboxes is unsupported, so Claude conducts QA autonomously via in-sandbox browser automation and returns visual artifacts. - [08:48](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=528s) Claude Managed Agents operates purely as a managed cloud service with no on-premises deployment tier. - [10:00](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=600s) Multi-tenant data isolation can be achieved either by scoping separate workspaces via CI/CD pipelines or by using a shared workspace segmented by session metadata. - [12:10](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=730s) Heavy compute workloads requiring GPUs or custom memory configurations must be offloaded to self-hosted sandbox providers such as Modal, E2B, or Daytona. - [13:21](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=801s) Multiplayer Slack integrations work best when threads are mapped to individual sessions with distinct user identity tags, or by dedicating single agents to specific channels. - [13:34](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=814s) To keep Slack channels clean, agents should be given an explicit tool to post messages rather than continuously streaming raw output. - [14:34](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=874s) Managed agent sessions have no artificial six-hour timeout and can sustain long autonomous runs using outcome rubrics and custom milestone status tools. - [16:30](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=990s) The coordinator subagent pattern decouples responsive chat agents from blocking custom tool executions by delegating long-running tool calls to subagents.

## Detailed walkthrough ### [00:13](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=13s) CI/CD for Agent Configs and Custom Tool Blocking

An audience member asks Gabe Cemaj and Priyanka Phatak how teams should manage CI/CD pipelines for their agent configurations. Gabe explains that Anthropic provides a dedicated tool called the ant CLI (`ant`), which acts as an API wrapper designed for infrastructure-as-code workflows. Developers can define durable, long-term agent resources as YAML files within their own git repositories, mirroring the declarative style of tools like Terraform. Whenever changes are merged, automated CI/CD jobs execute the ant CLI using those YAML files to synchronize and update the resources on the platform. Gabe Cemaj confirms that Anthropic uses this exact workflow internally: “Yeah, that's how if if we need to build durable long-term resources, we just we can check them in.”

Another attendee follows up regarding message delivery timing, specifically asking whether an end user or application can still push new messages while the model is actively waiting on the output of a custom tool call. Gabe clarifies that the execution lifecycle explicitly halts during that phase to guarantee message ordering: “Not when it's waiting for a custom tool result. We basically pause execution uh at that time.” While the system normally permits callers to queue incoming messages while Claude is busy running other tasks, attempting to dispatch a message while an unresolved custom tool request is pending results in an immediate HTTP 400 Bad Request error.

- **Agent configuration deployment:** Managed declaratively using YAML manifests and the ant CLI tool inside CI/CD workflows. - **Custom tool blocking behavior:** The execution loop strictly pauses awaiting tool output; queuing regular messages during this specific window is rejected with an HTTP 400.

---

[02:06](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=126s) Workshop Repositories and Anthropic's Eval Practices

Addressing an audience question about companion slides and additional hands-on code examples found across Anthropic's workshop repositories, Gabe Cemaj and Priyanka Phatak clarify that many of those materials originated from Anthropic’s Code with Claude developer conference held roughly a month earlier in London. Gabe notes that those recorded conference sessions are accessible on YouTube and that the public GitHub READMEs are designed to be largely self-explanatory and closely align with the current session.

The conversation then shifts to evaluation harnesses and how Anthropic maintains quality for Claude Managed Agents (CMA). Gabe details how Anthropic approaches internal testing, emphasizing that the managed platform team tests upcoming model weights and capabilities well ahead of public availability. Because the platform runs in the cloud, Anthropic can effortlessly spin up 10,000 parallel agent sessions across curated benchmark datasets. This automated harness evaluates model behaviors, tests prompt adjustments, and prunes obsolete operational assumptions across generic tasks, not just coding workloads.

```mermaid flowchart LR accTitle: Anthropic Managed Agents Evaluation Loop accDescr: Diagram illustrating how internal datasets feed parallel cloud sessions to optimize prompts and assumptions before model release. subgraph Input ["Data & Models"] A["Pre-release Model Weights"] B["Curated Benchmark Datasets"] end subgraph Execution ["Parallel Cloud Harness"] C["Spin up 10,000+ Parallel Agent Sessions"] end subgraph Evaluation ["Refinement Loop"] D["Score Session Outputs"] E["Tune Prompts & Harness Assumptions"] end

A --> C B --> C C --> D D --> E E -->|"Iterate Harness"| C ```

When asked what concrete eval strategies external developers should adopt for their own custom agents, Gabe emphasizes data quality over complex scaffolding, warning: “I think Claude code is pretty good at building an eval for you, but garbage in, garbage out.” He recounts that early internal attempts sometimes scored deceptive 100% pass rates simply because the evaluation scenarios were poorly defined or overly simplistic. Priyanka adds a pragmatic recommendation for teams starting from scratch: inspect existing production session logs, score the actual historical performance of the agent, and construct an empirical benchmark dataset from those real-world interactions.

- **Scale of internal testing:** Anthropic routinely executes 10,000+ concurrent cloud sessions to test model updates and prompt alterations across diverse non-coding and coding domains. - **Eval harness pitfall:** Ill-defined benchmarks frequently produce false 100% success metrics without reflecting true operational capability. - **Practical dataset generation:** Builders can curate representative eval suites by pulling historical session transcripts and scoring real historical outputs.

---

[04:42](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=282s) Multi-Tenant Resource Boundaries and Workspace Permissions

A participant who uses Claude extensively asks about permission boundaries across large organizations. Specifically, they want to know whether assets like agents, memory stores, and Model Context Protocol (MCP) channels can be shared selectively across internal departments or if entire stacks must be exposed at once. Priyanka explains that resources are anchored to the platform workspace level: “They're bound to your workspace. So, any API key within your workspace can have access to the resources that are in it.”

Priyanka explains that organizational control begins with the Claude platform administrator, who provisions top-level organizations and partitions them into specific workspaces. If teams want to segregate access to specific resources, they should map distinct teams or trust domains to separate workspaces. When the attendee asks about partial sharing—such as sharing access to a specific memory bank with one team while keeping an MCP connection restricted—Priyanka and Gabe explain that fine-grained access control lists (ACLs) do not natively exist inside CMA. Instead, application developers must manage granular permissions within their own integration layer by controlling which API keys and workspace credentials their internal systems invoke.

| Organizational Layer | Administrative Control | CMA Resource Access & Isolation | |:--- |:--- |:--- | | **Organization (Org)** | Platform Administrator | Top-level corporate tenancy; groups all workspaces and global billing. | | **Workspace** | Designated Admin / Team Leads | Primary security boundary; all API keys inside a workspace share its agents, memory stores, and MCP channels. | | **Integration Layer (Custom Code)** | Application Developer | Enforces fine-grained permissions, sub-resource slicing, and user-level access checks before calling CMA. |

---

[06:29](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=389s) Reviewing Agent-Generated Code and Automated QA Sandboxes

Highlighting the sheer volume of code generated by autonomous agents, an attendee asks how teams can overcome the review bottleneck before merging changes, wondering if Anthropic deploys code to ephemeral environments for testing. Gabe reveals that the original concept for this workshop was centered specifically on building automated code review agents: “Internally, we have these sorts of agents do a lot of the uplift for us, and we've seen that that enables us to reduce the amount of time we need to spend on a per PR basis and reduce the number of bugs and keep the quality high.” He explains that whenever a pull request is opened, an integration triggers multiple CMA sessions with specialized subagents that analyze the diff, detect subtle bugs, and post line-by-line review comments. Gabe Cemaj admits that this remains an industry-wide challenge: “This is partially an open question of now we've removed the boundary of typing and exploded the number of PRs we can make per day, hour, second.”

Addressing how to ensure the reviewed code actually functions at runtime, Gabe and Priyanka explain that the agent's environment itself serves as an automated QA testbed. Claude can start the local development server inside its hosted sandbox, interact with the service, run test suites, and execute self-validating checks. When the audience asks if engineers can open long-running tunnels into these sandboxes to manually test user interfaces, Gabe clarifies that live user network tunneling is not supported today. Instead, Claude functions as the QA tester directly: it can spin up browser automation inside the sandbox, interact with the UI, validate the visual and functional behavior, and return summary artifacts—such as screen recordings or video logs—back to human engineers for final verification.

- **PR review automation:** Anthropic kicks off automated multi-session agent swarms on incoming PRs to handle first-pass code reviews and surface regressions. - **In-sandbox QA execution:** Claude launches applications directly within its compute sandbox to test and validate runtime behavior rather than relying solely on static analysis. - **Tunneling limitations:** Direct developer tunneling into active CMA sandboxes is currently unsupported; testing is driven by Claude operating browsers and returning output artifacts such as videos.

---

[08:48](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=528s) Enterprise Tenant Isolation Models

An enterprise attendee asks how to enforce strict data segregation across hundreds of distinct corporate customers, noting they had heard CMA does not yet support on-premises deployments. Gabe confirms that on-premise deployments are unsupported, but emphasizes that cloud isolation can be modeled flexibly depending on the enterprise's architectural and regulatory needs. Priyanka explains that developers have multiple isolation tiers at their disposal: “You can define an agent per customer, and then that way the data will be segregated, or you can use a workspace per customer if you want like true isolation you could do an org per customer.”

Priyanka points out that builders must track the mapping between internal tenant identifiers and CMA assets on their own backend. Teams can isolate data logically by scoping an individual agent to each customer, isolate by API access boundaries by provisioning a workspace per tenant, or isolate entirely by assigning distinct customer organizations for enterprise accounts that demand complete operational and administrative separation. As the attendee begins formulating a follow-up scenario regarding multi-customer deployments, the session transitions deeper into isolation architectures.

- **On-premise support:** Claude Managed Agents is strictly a managed cloud service; on-premises hosting is not offered. - **Tenancy options:** Segregation can be structured via distinct agent instances, dedicated platform workspaces, or isolated top-level organizations. - **Mapping ownership:** The application backend is responsible for tracking tenant-to-agent and tenant-to-workspace routing tables.

[10:00](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=600s) Multi-Tenant Architecture and Data Segregation

An audience member asks how to build an agent definition once and deploy it across multiple clients when serving two or more customers. Gabe Cemaj and Priyanka Phatak lay out two primary architectural patterns depending on regulatory and security constraints. The first approach relies on configuration-as-code: developers define the agent as a template (comprising system prompt, model name, and tool definitions) in a YAML file in their code repository. Continuous integration (CI) tooling then synchronizes this single configuration across multiple workspaces, keeping them identical. The alternative approach hosts all customers within a single workspace, using a single agent definition while maintaining an application-level mapping of which customer owns which execution session.

```mermaid flowchart TD accTitle: Multi-Tenant Agent Deployment Models accDescr: Comparison between isolating customers by workspace via CI/CD sync versus sharing a workspace using session metadata.

subgraph Isolated["Workspace-per-Customer Pattern"] Repo["Agent Config (YAML in CI)"] Repo -->|"Sync"| WS1["Workspace A (Customer 1)"] Repo -->|"Sync"| WS2["Workspace B (Customer 2)"] end

subgraph Shared["Single-Workspace Pattern"] SingleAgent["Single Agent Definition"] SingleAgent --> S1["Session (Metadata: Cust_ID A)"] SingleAgent --> S2["Session (Metadata: Cust_ID B)"] end ```

When the attendee presses further on how strict data segregation requirements can be satisfied if customer data is lumped into one workspace, Gabe clarifies that legal and compliance boundaries dictate the deployment model. If customers have strict contractual or legal guarantees forbidding shared resources, separate workspaces must be provisioned because resource allocations and data boundaries are enforced at the workspace boundary.

“If you care about data segregation, then you need to put them in individual workspaces, and the resources are allocated per workspace,” Gabe Cemaj explains.

- **CI/CD Synchronization:** Tools provided by Anthropic allow developers to push updates from a centralized YAML configuration to all targeted workspaces in automated deployment pipelines. - **Session Metadata:** For shared workspaces, each session supports custom metadata fields (such as `customer_id`) to map sessions back to specific client tenants.

---

[12:10](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=730s) Custom Compute, Higher Memory, and GPUs via Self-Hosted Sandboxes

An audience member asks whether Claude Managed Agents allows users to modify the underlying sandbox compute environment—specifically requesting beefier memory limits or GPU acceleration for compute-heavy workloads. Gabe explains that Anthropic’s managed sandbox infrastructure currently provides a uniform, standardized environment. To run jobs requiring specialized hardware, developers must decouple the agent loop from default infrastructure and rely on self-hosted sandboxes.

“Today, if you want to do custom resource allocations for your sandbox, you would use self-hosted sandboxes,” Gabe Cemaj explains.

Priyanka notes that Anthropic maintains first-party integrations, documentation, and cookbooks illustrating how to wire up external sandbox providers such as Modal, E2B, and Daytona. By offloading execution to an external provider like Modal, teams can provision custom GPU slices and expanded memory envelopes while retaining Claude Managed Agents' orchestration features.

---

[13:21](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=801s) Architecting Multiplayer Slack Agents

A participant asks how to use Claude Managed Agents to build a multiplayer conversational experience in chat channels, referencing team collaboration tools like Claude Tag. Gabe explains that the system naturally supports multiplayer interaction by mapping an individual chat thread to a single agent session. When multiple users participate in the thread, incoming messages are injected into the session as discrete user messages labeled with the respective author's identity (such as noting whether a prompt came from Gabe or Priyanka).

“You don't necessarily want to just mirror whatever the agent is responding,” Gabe Cemaj cautions, noting that rather than streaming raw internal chatter back to the channel, developers should equip the agent with an explicit communication tool (like a Slack posting action) so Claude actively decides when and what to post.

Priyanka adds that for experiences modeled after channel bots, the cleanest pattern is dedicating one agent per Slack channel. This topology guarantees that each agent maintains isolated memory scoped strictly to the conversation history and context of that specific room.

---

[14:34](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=874s) Running Multi-Hour Sessions and Tracking Checkpoints

An attendee asks about building long-running agents that operate autonomously for hours at a time, such as six-hour execution spans, and how to track progress without losing visibility. Gabe points back to the outcome-driven rubric architecture demonstrated earlier in the workshop as the premier pattern for sustained autonomous execution. Instead of hardcoding steps, developers supply clear exit criteria, allowing Claude to iterate continuously and evaluate its own artifacts until the target standard is satisfied.

“The outcome system that I demoed is the best way to do really long sessions because you define an exit criteria and Claude will just keep going until it meets that criteria,” Gabe Cemaj states.

Priyanka points out that there are no built-in artificial timeouts preventing a session from running longer than six hours; it will continue executing until completion. To monitor progress and observe checkpoints during long runs, developers can listen to the session's underlying Events API. Gabe also recommends a practical software design pattern: provisioning a custom tool such as `update_status_for_human`, which Claude can invoke periodically whenever it achieves significant milestones to keep external operators informed.

---

[16:30](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=990s) The Coordinator Subagent Pattern for Custom Tool Waits

The workshop closes with a question addressing edge-case concurrency: if an agent triggers a custom tool that pauses for human-in-the-loop input, attempting to post another user message to that session produces an HTTP 400 error. The attendee asks how to keep an agent responsive to incoming user messages while still awaiting an asynchronous external tool result. Priyanka and Gabe suggest adopting a coordinator subagent architecture.

```mermaid flowchart TD accTitle: Coordinator Subagent Pattern for Custom Tools accDescr: A main coordinator agent receives user messages while delegating long-running custom tools to a subagent thread.

subgraph Session["Unified Core Session"] User["User / Client Chat"] -->|"Sends message"| Main["Main Coordinator Agent (Idle / Chatty)"] Main -->|"Delegates task"| Sub["Subagent B"] Sub -->|"Emits custom tool call"| HITL["Custom Tool / Human-in-the-Loop"] HITL -.->|"Tool result"| Sub end ```

Under this pattern, developers define a primary coordinator agent alongside worker subagents. The primary agent remains unencumbered and available to receive inbound user chat, while an assigned subagent executes the custom tool. Because the Managed Agents API automatically routes events and tool results based on internal IDs, the main thread remains idle and ready to chat even while worker threads are blocked waiting for external tool returns.

“So if you're trying to do something where you want like the main agent to be always chatty, then I think the coordinator pattern works pretty well for that,” Gabe Cemaj explains.

With the session time expired, Gabe and Priyanka thank the audience, offer to continue discussions outside the hall, and conclude the workshop to applause.

## Notable quotes > [02:06](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=126s) I think Claude code is pretty good at building an eval for you, but garbage in, garbage out. — Gabe Cemaj

> [08:48](https://www.youtube.com/watch?v=QSeaZ6qHyXs&t=528s) You can define an agent per customer, and then that way the data will be segregated, or you can use a workspace per customer if you want like true isolation you could do an org per customer. — Priyanka Phatak

## Mentioned Claude, Claude Managed Agents, Anthropic, Gabe Cemaj, Priyanka Phatak, ant CLI, Terraform, Code with Claude, Claude Code, Model Context Protocol, Modal, E2B, Daytona, Slack, Claude Tag, AI Engineer World's Fair 2026

## Video description How to version agent configs, isolate customers, add GPUs and keep a long-running agent responsive.

NEXT UP: AI ENGINEER CODE SUMMIT Join us in SF for the second annual AIE CODE Summit, Nov 10–12, our last flagship summit of 2026. Applications live now: https://aie.codes

The workshop closes with audience Q&A for Priyanka Phatak and Gabe Cemaj: syncing agent configs in CI/CD with the ant CLI, what happens when you message an agent waiting on a custom tool, how Anthropic runs evals, sharing resources across a workspace, code-review agents, tenant isolation, GPUs via self-hosted sandboxes, modeling a multiplayer Slack agent, and a coordinator pattern that keeps the main agent free while a subagent handles custom tools.

This is part 4 of 4 of the Claude Managed Agents workshop from Anthropic.

In this video: • Keeping agent definitions in YAML and syncing them in CI/CD • Isolating tenants by workspace, org or session metadata • Self-hosted sandboxes for GPUs and extra memory • A coordinator pattern for custom tools

SPEAKERS Priyanka Phatak, Member of Technical Staff, Anthropic (leads the Claude Managed Agents engineering team)

Gabe Cemaj, Member of Technical Staff, Anthropic GitHub: https://github.com/gcemaj

LINKS Claude Managed Agents docs: https://platform.claude.com/docs/en/managed-agents/overview Scaling Managed Agents: Decoupling the brain from the hands: https://www.anthropic.com/engineering/managed-agents

CHAPTERS 0:00 Q&A: CI/CD for agent configs 1:10 Q&A: messages during custom tool waits 2:06 Q&A: other workshop repos 2:45 Q&A: Anthropic's eval practices 4:42 Q&A: sharing resources across an org 6:29 Q&A: reviewing agent-written code 8:48 Q&A: tenant isolation 9:54 Q&A: one agent, many customers 12:10 Q&A: custom compute and GPUs 13:20 Q&A: a multiplayer Slack agent 14:34 Q&A: very long-running sessions 16:30 Q&A: coordinator pattern for custom tools 18:30 Closing

Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI.

AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/

#Claude #AIAgents #AIEngineer

[Source](https://www.youtube.com/watch?v=QSeaZ6qHyXs)

Reading summary

Anthropic engineers Gabe Cemaj and Priyanka Phatak outline architectural patterns for deploying Claude Managed Agents (CMA) in production, focusing on CI/CD configuration management, multi-tenant boundaries, and concurrency workarounds. They emphasize declarative infrastructure, empirical evaluation datasets, and delegated subagent topologies to balance isolation requirements with conversational responsiveness. Declarative CI/CD and Rigorous Cloud Evals (0:13) Anthropic manages durable agent resources declaratively using YAML manifests and the dedicated ant CLI (`ant`) within automated CI/CD pipelines, mirroring an infrastructure-as-code pattern like Terraform. Gabe Cemaj confirms Anthropic uses this workflow internally to check in and synchronize agent resources across environments. To maintain platform quality, Anthropic runs automated harnesses capable of spinning up 10,000 parallel cloud sessions across curated benchmark datasets to evaluate pre-release model weights and prompt adjustments beyond coding tasks. Multi-Tenant Boundaries and Workspace Isolation (4:42) Because Claude Managed Agents is strictly cloud-hosted and does not offer fine-grained access control lists (ACLs) within a single workspace, resources like memory stores and Model Context Protocol (MCP) connections are accessible by any API key in that workspace. For multi-tenant architectures, developers must either share an agent across clients using application-level session metadata (such as `customer_id`) or enforce hard compliance boundaries by provisioning separate workspaces per tenant. Cemaj and Phatak note that teams can use YAML templates in CI pipelines to synchronize identical agent configurations across isolated customer workspaces. Automated Code Review and Sandboxed QA Testing (6:29) To handle high volumes of agent-generated pull requests, Anthropic triggers automated CMA multi-session subagents that analyze diffs, detect bugs, and write PR review comments. Rather than relying solely on static reviews or manual network tunneling into active instances—which is currently unsupported—Claude executes applications directly inside its hosted sandbox. It can spin up browser automation to interact with user interfaces and return artifacts, such as screen recordings or video logs, for human engineers to verify. Sustained Execution and the Coordinator Concurrency Pattern (12:10) When tasks exceed default compute or memory boundaries, developers must decouple the orchestration loop from Anthropic's uniform sandboxes and link external providers like Modal, E2B, or Daytona for specialized hardware or GPUs. For long-running operations lasting several hours, Cemaj advises using outcome-driven exit criteria and custom status-reporting tools so agents iterate against self-evaluations while dispatching updates through the Events API. Addressing an execution block where waiting on a custom tool returns an HTTP 400 error on new user inputs, Cemaj and Phatak suggest a coordinator subagent pattern: a primary agent remains idle to converse with users while delegating blocking tool executions to worker subagents. While Claude Managed Agents simplifies stateful autonomous workflows, enterprise adopters must implement external application logic to handle fine-grained permissions, custom compute hardware, and non-blocking human-in-the-loop interactions.

Source quotes

[2:06] Gabe Cemaj: I think Claude code is pretty good at building an eval for you, but garbage in, garbage out. [8:48] Priyanka Phatak: You can define an agent per customer, and then that way the data will be segregated, or you can use a workspace per customer if you want like true isolation you could do an org per customer.

Original transcript

[0:01] [music] [0:13] >> All right, we have a question here. [0:21] >> Hey. [0:23] How do you see ICD the [0:25] agent convex? Um [0:27] >> Ah, that's a great question. Uh so, we [0:29] have something called Ancho CLI, which [0:31] is a CLI that basically wraps our API to [0:34] make calls [0:35] um and allows you to do sort of [0:37] Terraform-like things where you pass in [0:39] YAML. So, you can put your uh long-term [0:44] uh resources in YAML in your code base [0:46] and then build um CICD to basically [0:50] update the Ancho with Ancho CLI using [0:53] those YAMLs [0:54] uh every time you make a change to your [0:56] to your resource. [0:57] Does that make sense? [0:58] >> Got it. Uh so, I assume [1:00] you guys use the same internally too. [1:02] >> Yeah, that's how if if we need to build [1:04] durable long-term resources, we just we [1:06] can check them in. [1:07] >> All right, thanks. [1:08] >> Thanks. [1:09] >> Um so, I think you mentioned you can [1:12] send messages whenever you want. [1:14] Um [1:15] can you still send messages if the [1:16] model's waiting for a a custom tool [1:18] result? [1:19] >> Uh not when it's waiting for a custom [1:20] tool result. We basically pause [1:22] execution uh at that time. [1:24] Um but if the model is doing other work, [1:27] uh then we do allow you to queue up [1:29] messages. [1:30] Um but when it's doing a custom tool [1:31] result, we're expecting you to send that [1:34] um so, because of ordering purposes, we [1:36] we block the calls. [1:38] >> Well, what happens if you do send the [1:39] message? [1:40] >> You'll get a 400. [1:41] >> Okay, that makes sense. [1:43] >> Uh there was an ask to see the QR code [1:45] slide one more time. [1:47] >> The uh [1:48] >> The uh the credits one, yeah. [1:54] >> Any any other questions? If you're if [1:56] you're shy, we [1:57] >> Yeah, we'll hang out after for any any [2:00] questions you don't feel [2:07] >> Yeah, I'm looking through a repo. [2:08] There's a lot of interesting workshop [2:10] talks. Is there anything that's missing [2:12] from that like slides that would help us [2:14] go through other [2:16] >> Yeah, yeah. [2:16] >> sessions like that you can post or is it [2:18] pretty much self-explanatory through the [2:20] the readme files you just like have you [2:22] know cuz you had like a skill that did [2:24] workshop. [2:25] >> Yeah, yeah. The other workshops came [2:26] from our Code with Cloud conference. [2:29] Most of which you can find online [2:31] for the version we did in London about a [2:33] month ago. [2:34] >> Yeah, all all the all those talks are on [2:36] YouTube if you're interested in the [2:37] other repos that are in there, but this [2:40] one should give you like a pretty good [2:41] overview and it's pretty similar to what [2:42] we've done at other conferences. [2:45] >> Yeah. [2:46] >> Yeah, so just going back to evals, can [2:48] you talk a little bit about you know how [2:50] you guys are maintaining quality, best [2:51] practices around the the eval harness? [2:55] >> Uh yeah, we run a lot of evals [2:57] especially as new models come up as new [2:59] capabilities come up. We have access to [3:01] the models before you all do obviously. [3:04] So one of the things that we as a [3:06] managed offering want to make sure we do [3:08] is that we update our assumptions about [3:11] the harness as early as possible so that [3:13] we can remove things that are no longer [3:15] needed and we do all of that in a eval [3:17] base. So we have [3:19] internal tooling that basically looks [3:21] like different data sets that we can [3:22] then run our different sessions. It's [3:25] because it's all in the cloud it's easy [3:27] to say like spin up 10,000 sessions to [3:29] do this all in parallel [3:32] and then we evaluate those results and [3:34] tune prompts and tune assumptions, [3:36] remove things, add things to make sure [3:39] that for every model release we're doing [3:41] the best we can and we're evaluating [3:44] all sorts of things for generic purposes [3:46] not just coding. [3:48] >> So so how does that extend into the [3:50] agents that we would build? Is there a [3:51] recommended path for [3:53] for evals or is there kind of just bring [3:55] your own solution? [3:56] >> Uh [3:56] I think the the best advice we can give [3:58] you is to get high quality data [4:01] to evaluate over. I think Claude Claude [4:04] code is pretty good at building an eval [4:05] for you, but garbage in, garbage out. [4:08] We've seen that some of our initial [4:10] ideas for how to evaluate things were [4:13] sort of ill-defined and you get 100%, [4:16] but what does that mean in performance? [4:18] So, iterating on your data and making [4:20] sure that you're setting up the problem [4:21] correctly. [4:22] Um I know it's like not out of the box [4:24] solution, but it generally the advice [4:26] that we've seen works. [4:27] >> And you can probably look at like your [4:29] historical sessions and like what [4:31] results you got and then [4:33] sort of try to score it and see how the [4:34] model's doing. [4:36] Um that would be a good way of like [4:37] building your data set looking at your [4:39] historical sessions. [4:42] >> I have a question over here. [4:43] >> Hey, thanks for sharing. Uh I have a [4:45] question about how to share those agent [4:48] and all the resources across all the the [4:50] entire organization. The reason I'm [4:52] asking is because we use Claude as code [4:54] quite a lot. We're building shared [4:56] skills. We're building shared [4:57] connectors. But for the everything you [5:00] shared today, for example, agent, memory [5:02] store, or MCB channel, can we actually [5:05] share them or can we partially share [5:07] them across the organization to [5:09] different departments? So, what is [5:11] boundary? What is the privilege about [5:13] those resources? [5:15] Are those resources bonded to my account [5:17] or how can I control this? So, what's [5:19] the what's the way to do that? [5:21] >> They're bound to your workspace. So, any [5:23] API key within your workspace can have [5:25] access to the resources that [5:27] >> So, can my can my teammates edit or [5:30] modify what I'm building or is Yes, [5:33] completely controlled, right? [5:34] >> So, I think your Claude platform admin, [5:37] so you'll have an org for your [5:39] organization. And then your admin will [5:42] be the one setting up the workspaces and [5:44] deciding who gets access to which [5:46] workspace. So, I think [5:47] >> you would basically design your [5:48] workspaces depending on your access [5:50] control. [5:51] >> Correct. [5:51] >> You'd probably want to do like [5:53] wh- whatever group of people you want [5:55] access to resources X, put them in one. [5:58] >> So, you can share whatever you want, [5:59] even partially of the, you know, you [6:01] build an agent that can do something, [6:02] but I only want to share part of that. [6:04] For example, I share the memory to [6:05] someone that share the MCP channel to [6:07] the others. Is that possible? Or I need [6:09] to share the entire tech stack of what [6:11] I'm building? [6:12] >> We don't have fine-grained controls of [6:14] like this to share this or this not, but [6:16] you as a as a builder and integration [6:18] would be able to control that at pretty [6:20] granularly so that your end users can [6:22] have access to partial sharing. [6:25] >> Cool. Sweet. Thanks very much. [6:27] >> Thank you. [6:27] >> Yeah. [6:29] >> Yeah, so [6:30] basically we can generate a lot of code [6:32] with a lot of agents, and then we're [6:33] getting to the bottleneck of [6:36] uh reviewing that code, making sure you [6:37] can actually merge it. Um [6:40] how do you guys think about that in [6:41] terms of are you deploying like I see [6:44] you deploy every 30 days, which uh might [6:47] be a different definition of deployment, [6:48] but is it deploying something you can [6:50] actually test to make sure it works [6:52] well, or how are you reviewing [6:55] uh this many lines of code from all the [6:56] agents? [6:57] >> So, the the original workshop that we [7:00] wanted to do was actually answering that [7:02] question. How do you How do you build [7:04] agents to review code? Um what we found [7:06] is you can build an integration with CMA [7:09] so that on PR, it kicks off [7:13] one, two, three, many sessions with many [7:15] sub-agents to evaluate what has changed [7:18] and find bugs. Internally, we we have uh [7:21] these sorts of agents do a lot of the [7:22] uplift for us, and we've seen that that [7:24] enables us to uh reduce the amount of [7:27] time we need to spend on a per PR basis [7:30] and reduce the number of bugs and keep [7:32] the quality uh high. Um but this is [7:36] partially an open question of now we've [7:38] removed the boundary of uh typing and [7:41] exploded the number of PRs we can make [7:44] per day, hour, second. [7:46] Um how do we keep up? And we're still [7:48] exploring that, but [7:49] >> Yeah. [7:50] >> CMA is part of the answer, I think. [7:52] >> Yeah, and you can iterate in a loop, you [7:54] know, to like review, add comments, [7:56] review until it's done, but then at some [7:58] point you still need to run the thing to [8:00] make sure it works well locally. So, I [8:02] think that's where the deployments can [8:04] come in, where you can [8:05] um [8:06] you know, auto deploy something that you [8:07] can test with a custom deploy agent, [8:09] like you said. [8:11] Um it just depends, like if you have [8:12] different environments for each branch, [8:14] but [8:15] >> You you can run your application in the [8:18] sandbox as well and have Claude [8:19] self-validate the changes that it made. [8:22] Um and make sure that [8:24] like [8:25] it acts as a QA agent as well. [8:28] >> Are these like long-running sessions [8:29] that you can go in and actually play [8:31] around on the UI? [8:33] >> We don't have uh kind of tunneling [8:35] today, [8:36] >> So, you would have Claude go in there, [8:38] play around with the browser. [8:40] >> That's right. And then [8:41] >> and validate it. And then maybe even [8:42] give you back a video, maybe. [8:43] >> Yep, exactly. [8:45] >> Cool. [8:46] Thanks. [8:48] >> Any other questions? [8:49] >> Hello. [8:50] >> Hi. [8:50] >> Uh for an enterprise company, if at all [8:53] we want to maintain true tenant [8:55] isolation that we want to support, I [8:58] heard that you are not supporting yet [8:59] the enterprise uh company uh use cases, [9:02] probably. [9:03] >> We're not supporting on-prem. [9:05] >> On-prem. [9:05] >> Yeah. [9:05] >> But what if I want to have uh let's say [9:09] I have 100 customers that as an [9:11] enterprise company, can I have separate [9:14] uh Claude-managed agent instances [9:16] >> Yeah. [9:16] >> per [9:17] company of ours that we want to support? [9:20] Is that a viable option that you [9:21] support? Is it only one instance that we [9:23] get, or can we get one instance per [9:27] company that we support as an [9:28] >> It's however you choose to define it. [9:30] You can define an agent per customer, [9:33] and then that way the data will be [9:35] segregated, or you can use a workspace [9:37] per customer if you want like true [9:39] isolation you could do an org per [9:41] customer. [9:43] So it depends on how you want to set up. [9:45] Your agents and you would probably want [9:47] to maintain that mapping of like which [9:49] customer maps to which agent. [9:51] >> Okay. [9:54] I think I have a follow up question. So [9:57] just to given the scenario that [10:00] we have two customers and we want to [10:02] build the agent once but the agent going [10:05] to be shared by the two customers. [10:08] How to achieve that? [10:11] >> There's many ways to do it. You could do [10:13] it through automation. So you could have [10:15] your agent definition live in Yamo in [10:16] your code and then build tooling to [10:19] update it to both of your workspaces and [10:21] basically synchronize them or you can [10:23] model everything in a single [10:24] organization and then do the the [10:27] modeling that way. [10:28] Uh [10:28] >> You could create like a single agent [10:30] across all your customers and then you [10:32] manage [10:33] the mapping of like which customer is [10:35] using which session. [10:37] >> Yeah, then in in which workspace I can [10:40] accurate this agent? [10:42] >> Yeah, if if you do it per workspace then [10:44] you need we give you tooling to [10:46] synchronize a single definition in your [10:49] code to multiple places so that whenever [10:51] you update it in your code through CI [10:53] you can update all of the definitions at [10:55] once. [10:56] >> So you can either make a copy of your [10:58] agent which is essentially a template of [11:00] system prompt model name tools and you [11:03] can like make a copy of across every [11:05] workspace. Let's say you go down like [11:07] the one workspace per customer or you [11:09] can just put all your customers in a [11:11] single workspace and then you maintain [11:12] the mapping of [11:14] which either you do again have like [11:16] copies of your agent or you use a single [11:18] agent and then have [11:20] the mapping of like which session maps [11:22] to which customer. We have like a [11:23] metadata field on the session where you [11:25] can put in like let's say customer ID so [11:27] you know which session what map to which [11:30] customer. [11:31] >> Okay, that you know like a putting all [11:33] the customer data in one workspace, how [11:35] do we satisfy the data segre- [11:39] segregation [11:41] >> Yeah. [11:41] >> requirement, you know. [11:42] >> Yeah, if you if you care about data [11:43] segregation, then you need to put them [11:45] in individual workspaces, and the [11:46] resources are allocated per workspace. [11:48] But, there's basically tooling to enable [11:51] you to do each of each of those things. [11:54] >> And when you say data segregation, like [11:56] is is that like a legal requirement, or [11:57] is that something just you want to [11:59] >> Yes, so [11:59] >> something legal, like customer doesn't [12:01] want to share data with other customers, [12:04] you know, they want to dedicate [12:06] >> Yeah, you would do it per workspace. [12:08] >> Okay. Okay. Thank you. [12:10] >> I saw a question here. [12:12] You have to speak up, or [12:14] >> Um [12:16] Is there a way to modify [12:19] the compute environment, meaning [12:22] uh in some cases, I may need to have [12:24] higher memory [12:26] for my agent to do the work, or I need a [12:28] GPU for my agent to actually do the [12:30] work. [12:31] Is it supported in this, or [12:33] >> To to the Today, if you want to do [12:36] custom resource allocations for your [12:37] sandbox, you would use self-hosted [12:39] sandboxes. You would use something like [12:41] Modal that has GPU offerings. And we [12:43] have cookbooks and kind of [12:45] first-party integration, so that you can [12:48] hook up your Modal instances to run your [12:51] agent loop inside of Modal instead of [12:53] our provided sandboxes, which today are [12:56] all the same. So, if you care about [12:58] different scale, you would use a [12:59] different provider for that. [13:00] >> And do you have like a list of [13:04] specifications of what the environment [13:06] these agents have access to? [13:07] >> Yeah, you you can find that in our docs, [13:08] and you can find cookbooks to integrate [13:10] with third-party providers as well in [13:13] our docs. [13:13] >> Yeah, we have a bunch of cookbooks on [13:15] like how to work with Modal or EB2, [13:17] Daytona, whatever provider that you want [13:19] to use for sandboxes. Yeah. [13:21] Yeah. Hi. [13:22] >> Hi. [13:23] >> Thanks again. So, out of curiosity, how [13:26] to build a sim- [13:28] version of Cloud Tag using CMA in a [13:31] different chat channel, especially for [13:33] the multiplayer mode. [13:36] >> Uh [13:36] this definitely something you could do [13:39] today where you would on a per thread [13:42] level map it to a session and then [13:44] anytime a user messages that session, [13:48] you inject that as a user.message [13:51] letting the agent know, hey, this is [13:52] coming from Gabe or if it's coming from [13:54] Priyanka. [13:55] And so everybody can message that agent [13:58] independently. You can have it share [14:00] memory across sessions so that it learns [14:02] from each other. You can give it tools [14:04] to reply back using like the Slack post [14:08] tool or something like that so that the [14:10] agent can decide when it wants to post. [14:12] You don't necessarily want to just [14:14] mirror whatever the agent is responding. [14:16] You want it to be explicit. Uh but you [14:19] could definitely model this using CMA [14:20] and [14:21] >> And if you want it to be like Cloud Tag, [14:22] you'd probably do like one agent per [14:24] Slack channel and then every agent has [14:26] its separate memory and only has access [14:29] to messages within [14:32] that Slack channel. [14:34] >> Yes. [14:34] >> Hey, thank you for the presentation. [14:36] Very useful. [14:38] Quick question in terms of um [14:40] looping and doing long runs um [14:44] in this Cloud Console, how do you see [14:47] that? Um [14:48] is this something you have like some [14:50] specific module that I I'm not finding [14:53] or [14:53] um are you you're planning to develop [14:56] that um more in the future? Um thank [14:58] you. [15:00] >> Do you mean how do we provide the [15:02] observability for really long sessions [15:04] or [15:06] >> No, no, no. Um so more focus on um [15:09] using this platform to build those uh [15:12] long-running agents like hours and hours [15:15] like last 6 hours for example. [15:17] Um is this something that is already [15:19] embedded in what you have built in the [15:22] agents or [15:23] I'm more curious to know if you are [15:25] developing something specifically for [15:27] that or how you see that trend. [15:29] >> The the outcome system that I demoed is [15:32] the best way to do really long sessions [15:35] because you define an exit criteria and [15:38] Claude will just keep going until it [15:39] meets that criteria. And it could be a [15:41] really long task like [15:43] you know, [15:44] rebuild blank [15:46] and it will just mercilessly iterate [15:49] until it matches the the rubric. [15:51] >> There there there's nothing special you [15:53] would need to do to get the session [15:54] running longer than six hours. It should [15:57] it should just keep [15:59] keep going. [15:59] >> One quick follow-up. So, if I run at the [16:02] agent for hours and hours and I want to [16:04] understand like checkpoints status, the [16:06] best way to that will be through the [16:08] session tab. [16:10] >> Uh yeah. [16:11] >> be the session event stream. Yeah, [16:12] there's the events API on under the [16:14] session. [16:15] >> Okay. [16:15] >> You could have a custom tool called like [16:17] update status for human or something [16:20] like that and Claude would on intervals [16:23] whenever it feels like it's made [16:24] progress call that tool and give you a [16:27] status update. [16:28] >> Thank you. Thank you very much. [16:30] >> All right, any other final questions? [16:32] Yes. [16:33] >> Um following up on that previous [16:35] question about using this for Claude tag [16:38] or just similar functionality. I've been [16:40] uh analyzing a whole bunch of uh agent [16:43] frameworks. [16:45] So, if you get a message while the um [16:48] agent is waiting for a custom tool [16:50] result for example, human in the loop, [16:52] um you said it would 400. Do you have uh [16:55] any suggestions on how you would deal [16:56] with that? [16:58] >> Um [16:59] >> Can you do custom tools on a sub agent? [17:03] I forgot. [17:03] >> You you could do custom tools on a sub [17:04] >> Cuz then cuz you could basically have [17:06] like a sub agent that's handling your [17:08] custom tool and then that way your main [17:09] kind of coordinator agent's always free [17:11] for [17:12] receiving. [17:13] >> In the API, can you directly return a [17:17] custom tool result for the sub agents [17:19] tool. [17:20] >> Well, the sub agent would emit the [17:21] custom tool use and then you would [17:24] populate it back to that. [17:25] >> you you define like your main [17:27] coordinator agent and then you can say [17:29] agent A can talk to agent B and C and [17:31] then agent B is the one let's say [17:33] processing your custom tools and your [17:35] agent B definition is where you'll [17:36] define your custom tool and then that's [17:39] the session thread that you would listen [17:40] to get the [17:42] >> Is that a different session thread or [17:44] the same? [17:44] >> No, it's it's part of like the same core [17:46] session. [17:46] >> a different thread technically, but you [17:48] only interact with the main thread and [17:50] we correctly delegate it cuz we know [17:52] where it needs to go based on IDs. [17:54] Um [17:54] >> Okay. [17:55] >> And then the main agent will pass it [17:57] along to the [17:58] sub agent. [17:59] >> So the session is still idle even though [18:01] the [18:02] >> If if if a sub agent is doing work, it's [18:05] not idle. It's [18:06] >> But like if the main coordinator agent [18:08] is idle [18:09] the sub agents could still be [18:11] >> still doing work. Yeah, you you can see [18:12] that here. [18:13] >> Okay, that's pretty impressive. [18:14] >> This uh gap here, the main session is is [18:17] idle. The main the main thread is idle [18:19] while the sub agents were doing work. [18:21] >> And they could be waiting for tool [18:22] results. [18:22] >> Exactly. Yeah. So if you're trying to do [18:24] something where you want like the main [18:25] agent to be always chatty, then I think [18:27] the coordinator pattern works pretty [18:29] well for that. [18:30] >> That's awesome. [18:30] >> Yeah. [18:31] >> I think we're we're at time, but we'll [18:33] we'll be just outside if you want to [18:35] come talk to us. Uh thank you for having [18:37] us. [18:37] >> All right. Th- Thank you so much. [18:40] >> [applause] [18:54] [music]