Microsoft Cuts Debugging Risks With Ephemeral SSH Across 12,000 Clusters

微软通过临时SSH降低12,000个集群的调试风险 微软将短期SSH证书与AI智能体框架相结合，消除了其全球集群中的长期生产环境访问权限，可在37秒内查明根本原因。 微软的Cosmic平台管理着跨越数十万个节点的12,000多个Kubernetes集群，需要一种无需长期高特权访问权限的现代化生产环境调试方法。为应对安全与运维挑战，微软开发了集群调试工具（CDT），该工具利用短期SSH证书、即时（JIT）SSH服务器以及具备完全可审计性的严格RBAC授权来强制执行基于任务的操作。在CDT之上，智能体到智能体（A2A）AI架构协调各专用子智能体，在严格执行安全防护规范的同时，可在数秒内完成只读的根本原因分析。该系统实现了高诊断准确率，并成为值班工程师集成式智能体驱动Shell环境的基础。 Substrate 为 Microsoft 365 核心服务提供统一底层架构；为解决容量规划痛点，微软在 AKS 上构建了多租户平台 Cosmic，目前管理着 1.2 万个 Kubernetes 集群和近 600 万个计算核心。 (0:04) 为消除常态化高特权访问带来的安全隐患，微软开发了集群调试工具 CDT，采用严格的即时（JIT）授权模型，跨 Windows 与 Linux 节点提供按需且完全可审计的交互式调试。 (3:15) CDT 将授权置于关键执行路径上，通过签发嵌入特定命令的短时 SSH 证书，并借助集群内自定义控制器直接与容器运行时交互，实现对无内置 shell 的 distroless 容器的安全排错。 (4:42) 在 Cosmic 运营门户之上，微软引入了 Agent-to-Agent（A2A）多智能体架构，通过只读工具和严格的 RBAC 权限继承，以平均 37 秒的响应时间和 85% 至 90% 的准确率自动化完成根因分析。 (8:30)

Microsoft engineers Shridivya Sharma and Luke Kelly present an operational model for securing hyperscale production debugging within Microsoft Cosmic by eliminating standing privileged access. They demonstrate how ephemeral execution tooling paired with a multi-agent diagnostic framework enables automated, audited root-cause analysis across thousands of Kubernetes clusters. Cosmic and the Challenges of Hyperscale Fleet Management (0:04) Microsoft Substrate originated as a foundational platform providing unified deployment, infrastructure, and compliance controls for core services such as Microsoft Teams, Exchange, OneDrive, and SharePoint. To eliminate rigid capacity planning tied to physical hardware, Microsoft introduced Cosmic (Container On Substrate Managed Intelligent Clusters), extending Substrate into a managed multi-tenant platform built on Azure Kubernetes Service (AKS). Cosmic represents the second-largest consumer of AKS within Microsoft, supporting hundreds of thousands of nodes, tens of millions of pods, and nearly six million compute cores spanning 12,000 clusters across commercial and sovereign cloud regions. The environment uniquely hosts both Windows and Linux workloads side-by-side. Operating across this heterogeneity made traditional troubleshooting difficult, particularly because manual debugging historically relied on high-privilege standing access, creating significant audit gaps and heavy operational burdens on platform teams. Architecting Zero Standing Access with the Cluster Debugging Tool (4:42) To remediate the security risks of standing privileges, Microsoft implemented the Cluster Debugging Tool (CDT), which enforces an operating-system-agnostic, Just-In-Time (JIT) access model. When an engineer executes a CLI command targeting a pod or node, the request enters an authorization service gatekeeper that evaluates user identity, eligibility, and command safety. Safe, read-only commands proceed automatically, whereas higher-risk commands require explicit human approvals from leads or incident managers. Once approved, the system generates a short-lived SSH certificate valid for minutes, directly embedding the authorized command into the certificate's native SSH force command fields. The user CLI connects to an ephemeral JIT SSH server restricted to corporate networks, while an in-cluster custom controller schedules a temporary environment to execute the payload. By integrating directly with the host container runtime, CDT can inspect distroless containers that lack shells or local debugging binaries, while streaming comprehensive audit logs detailing user identity, execution duration, and command outputs. Automating Root Cause Analysis via Multi-Agent Coordination (8:30) While CDT isolated execution, triage still depended on human operators to interpret telemetry, query runbooks, and select troubleshooting commands. To automate this phase, Cosmic added an AI orchestration layer to the Cosmic Operational Portal (COP) using an Agent-to-Agent (A2A) pattern. Rather than relying on a monolithic model, a central orchestrator parses user prompts and delegates tasks dynamically to specialized sub-agents. Under this design, specialized diagnostic sub-agents trigger read-only CDT commands, while dedicated knowledge-base agents cross-reference incident histories, platform action logs, and internal runbooks. Operating in parallel, these agents iteratively construct, test, and refine operational hypotheses based on live execution feedback, narrowing down underlying failures without human manual intervention. Operational Guardrails and Performance Benchmarks (11:18) To prevent accidental mutations or insecure privilege escalation, the AI diagnostic system enforces hard architectural boundaries. The underlying Model Context Protocol (MCP) tools restrict models to read-only commands (`get`, `list`, and `describe`), containerized agents run in restricted sandboxes, and authentication uses On-Behalf-Of (OBO) token exchange to ensure agents inherit only the specific RBAC permissions of the invoking engineer. Evaluations across 19 operational scenarios—encompassing 16 diagnostic profiles such as pending pod states, networking issues, runtime degradation, and image pull failures—demonstrate an 85% to 90% root cause accuracy rate and 70% to 80% factual grounding, with an average resolution speed of 37 seconds. To curb model drift and cost, investigations are strictly capped at 10 tool cycles and must consult historical runbooks at least once. If conclusive evidence is absent, a built-in hallucination guard requires the agent to state that no evidence was found, preventing speculative diagnoses. Microsoft plans to integrate CDT's compliance and sandboxed execution boundaries into a unified, agent-first shell environment. This environment aims to pre-package diagnostics, internal wikis, and documentation into auditable execution containers, equipping autonomous agents and on-call engineers with a single interface for secure incident resolution.

微软工程师 Shridivya Sharma 与 Luke Kelly 阐述了 Cosmic 平台如何通过零常备特权的集群调试工具（CDT）与多智能体协作架构，在保障安全与合规的前提下重塑超大规模 Kubernetes 生产环境的故障排查与根因分析。 超大规模架构演进与调试困境 (0:04) 微软 Substrate 作为支撑 Microsoft 365 核心服务（如 Exchange、OneDrive 和 SharePoint）的基础平台，早期因各业务团队独立管理基础设施而面临环境碎片化与合规难度大的问题，且容量规划受制于物理机静态配置。为此，平台团队构建了基于 Azure Kubernetes Service（AKS）的多租户平台 Cosmic（Container On Substrate Managed Intelligent Clusters），统一接管部署、伸缩、路由与合规职责。据 Shridivya Sharma 介绍，Cosmic 目前是微软内部 AKS 的第二大客户，运行在遍布全球公有云与主权云区域的 12,000 个 Kubernetes 集群上，管理着近 600 万个计算核心、数十万个节点与数千万个 Pod，并同时承载 Windows 与 Linux 工作负载。 然而，传统的生产环境调试高度依赖高特权的长效访问权限（standing access）。随着集群扩展至上万规模，长效权限不仅带来了审计盲区和严重的安全风险，还导致业务团队在日常排障中过度依赖核心平台团队。加之早期工具针对 Linux 和 Windows 割裂开发，运维体验缺乏一致性，建立统一且安全的调试机制成为迫切需求。 零常备特权集群调试工具架构 (4:42) 为消除常备权限隐患，微软设计了集群调试工具（CDT）。Luke Kelly 指出，CDT 采用即时授权（Just-In-Time, JIT）模型，在提供交互式调试的同时实现了零常备访问、范围受控的授权与全流程审计追溯，且完全兼容 Windows 和 Linux 节点。工程师在排查故障时只需通过单一 CLI 指定目标与待执行命令，无需手动编写 YAML 清单或直连 SSH，底层复杂性由工具全权抽象。 该工具在架构上严格贯彻“无授权即无执行”的原则：所有请求必须先经由授权服务判定，安全的只读命令自动放行，高风险操作则必须获得团队主管或突发事件负责人的明确审批。审批通过后，系统签发仅在数分钟内有效的短期 SSH 证书，并将执行命令与元数据固化在强制选项中，下游验证时无需反向查询授权中心。随后，CLI 调起专用的 JIT SSH 服务建立受限网络连接，集群内的自定义控制器调度临时容器环境直接与宿主机容器运行时交互，不仅能避免产生交互式 Shell，还支持直接调试缺乏 Shell 和调试二进制文件的无发行版（distroless）容器，所有操作细节与输出均实时归档至端到端审计日志。 多智能体协同诊断与安全护栏 (8:30) 尽管 CDT 提供了标准化的执行杠杆，但初步分诊依然依赖工程师编写指令和解读文档的经验。为此，微软在统一运维门户 COP（Cosmic Operational Portal）之上引入了智能体协同（Agent-to-Agent, A2A）架构。中央调度智能体通过专用协议协调专门的诊断智能体与知识库智能体：前者通过 CDT 执行只读排障指令，后者检索历史事故记录、操作日志与内部运行手册，多智能体并行协作形成假说、验证数据并逐步收敛诊断结论。 针对安全风险与模型幻觉，系统设立了多层严密防护。工具层基于 MCP 协议仅开放 get、list 与 describe 等只读接口，权限校验通过 OBO（On-Behalf-Of）令牌交换机制严格继承发起用户的 RBAC 权限，且系统不具备自动修复权限，仅能生成建议命令。在针对 16 类诊断在内的 19 种场景评测中，该系统实现了 85% 至 90% 的根因定位准确率、70% 至 80% 的事实依据支撑度，平均响应耗时 37 秒。系统还限制最多进行 10 次工具调用循环，并强制要求至少检索一次知识库；若缺乏有效证据，幻觉防御机制会强制智能体明确汇报未找到证据，杜绝捏造推论。 智能体优先的集成运维环境展望 (13:49) Luke Kelly 阐述了将 CDT 机制融入智能体时代的演进目标。微软计划将合规性、审计追踪和 CDT 执行能力统一封装至面向工程师的“智能体优先”（Agent-First）Shell 容器环境中。该环境将预置所有内部排障工具、维基与技术文档，使内嵌的 AI 智能体全面感知可用工具与上下文，从而为值班工程师提供一站式、高度合规且受严格审计的超大规模运维支持。 微软在 Cosmic 上的实践表明，超大规模 Kubernetes 集群的生产调试可以通过短期证书与即时特权机制消除常备高危权限，而 AI 智能体的引入能够在保留人工最终核准权的前提下将根因分析收敛至数十秒内；如何在未来的智能体优先 Shell 中持续维持严格的只读与审计边界，是该体系规模化演进的核心考量。

## Background 该视频由技术分享频道 @Scale 于 2026 年 6 月 26 日发布在 YouTube 平台上。视频记录了来自微软的 Shridivya Sharma 与 Luke Kelly 的技术演讲，系统介绍了微软如何在超大规模 Kubernetes 集群舰队中，通过结合即时提权（JIT）、任务级执行与 SSH 机制实现安全可审计的生产调试，并利用 Semantic Kernel 与 AI 智能体技术协助一线值班工程师高效解决线上故障。 ## TL;DR 演讲详细拆解了支撑 Microsoft 365 与 Teams 的超大规模容器平台 Cosmic（涵盖 1.2 万余个 K8s 集群与数百万计算核心）在生产调试领域的演进历程。为根除传统常驻高权限带来的安全风险，微软打造了集群调试工具（CDT），依托短效 SSH 证书中嵌入的强制命令约束与集群控制器，实现了面向极简无发行版容器的受控隔离调试与全链路审计。在此基础上，团队引入由 Semantic Kernel 驱动的 Agent-to-Agent 协同排障架构，结合工具层、鉴权层、提示词层与沙盒环境构筑的四层默认只读安全防线，达成了高达 85% 至 90% 的根因分析准确率。这一实践展示了在零常驻权限的前提下，融合自动化与严格护栏的大规模云原生智能运维蓝图。 ## Atmosphere & style 整场演讲呈现出典型顶级云计算基础设施团队的硬核工程气质，语调沉稳自信、逻辑严丝合缝，充满了基于真实生产痛点演进出的架构智慧。两位讲者分工明确且叙述流畅，从庞大集群治理的现实挑战平滑切入到底层 SSH 证书约束与 K8s 控制器的实现细节，层次极其分明。观看过程中既能深切体会到面对数千万 Pod 时对“零信任”与审计合规近乎偏执的严谨，又能感受到拥抱 AI 智能体时那种兼顾前沿创新与深度防御工程克制的专业魅力。 ## Bulletin - [02:34](https://www.youtube.com/watch?v=UAs6-gNzFks&t=154s) 微软 Cosmic 平台在 12,000 多个 Kubernetes 集群、近 600 万计算核心规模下，彻底淘汰传统常驻高权限，转向基于短效 SSH 证书的零常驻权限集群调试工具（CDT）。 - [04:42](https://www.youtube.com/watch?v=UAs6-gNzFks&t=282s) CDT 创新性地将批准的调试命令直接嵌入短效 SSH 证书的 ForceCommand 选项中，节点在本地校验签名即可执行，杜绝了交互式 Shell 和越权操作风险。 - [08:30](https://www.youtube.com/watch?v=UAs6-gNzFks&t=510s) 微软基于 Semantic Kernel 构建了 Agent-to-Agent 自动排障架构，结合工具层、鉴权层、提示词层及沙盒运行时的四层纵深防御，确保 AI 全链路默认只读。 - [12:15](https://www.youtube.com/watch?v=UAs6-gNzFks&t=735s) 生产基准评测显示，AI 诊断智能体在 19 类典型故障场景中的根因分析准确率达 85% 至 90%，平均排障响应时间仅为 37 秒。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=UAs6-gNzFks&t=4s) Cosmic 作为微软 Substrate 的容器化演进平台，托管着数千万 Pod 与近 600 万核心，统一接管部署、扩缩容与合规，成为微软内部仅次于第一大客户的 AKS 第二大租户。 - [02:34](https://www.youtube.com/watch?v=UAs6-gNzFks&t=154s) 超大规模集群的生产排错过去依赖高危的常态化高权限访问，不仅审计困难且工具碎片化，催生了统一且免常驻权限的 CDT 方案。 - [04:42](https://www.youtube.com/watch?v=UAs6-gNzFks&t=282s) CDT 实行严格的单行 CLI 交互与逐条授权机制，遵循“无授权即无执行”原则，将只读命令与高风险变更分级审批。 - [04:42](https://www.youtube.com/watch?v=UAs6-gNzFks&t=282s) 系统利用短效 SSH 证书与 ForceCommand 关键选项，将目标元数据与单次执行命令在签发阶段深度固化，任务结束立即失效。 - [04:42](https://www.youtube.com/watch?v=UAs6-gNzFks&t=282s) 集群内瞬态控制器能够直接穿透与底层容器运行时通信，即便是缺少诊断工具和 Shell 的 Distroless 无发行版容器也能顺畅调试。 - [04:42](https://www.youtube.com/watch?v=UAs6-gNzFks&t=282s) 整个调试流水线内嵌端到端审计日志，精确记录操作人员、目标、耗时、执行命令及标准输出/错误，保障全链路可追溯。 - [08:30](https://www.youtube.com/watch?v=UAs6-gNzFks&t=510s) 针对排障手册繁杂与人力认知瓶颈，微软在 COP 运维门户中引入 Semantic Kernel，构建中央编排器与专业子 Agent 协同的自动化体系。 - [10:36](https://www.youtube.com/watch?v=UAs6-gNzFks&t=636s) AI 排障架构在工具层（只允许 get/list/describe）、鉴权层（On-Behalf-Of Token）、提示词层与运行时沙盒建立四层防御，物理与逻辑层面彻底杜绝写操作。 - [12:15](https://www.youtube.com/watch?v=UAs6-gNzFks&t=735s) 在覆盖 Pending Pod、性能抖动等 19 类故障的评测中，AI 智能体达成 85%-90% 的根因定位准确率，并将平均响应时间压低至 37 秒左右。 - [12:15](https://www.youtube.com/watch?v=UAs6-gNzFks&t=735s) 系统设置了步数熔断（最多 10 次工具调用）、强制知识库引用、标准 RCA 格式以及无证据时强制认同等严密护栏，有效杜绝大模型幻觉。 - [13:49](https://www.youtube.com/watch?v=UAs6-gNzFks&t=829s) 微软未来的 CDT 演进目标是将审计追踪与工具链打包进专属排障 Shell，让具备完备知识库与调试技能的 AI 智能体成为值班工程师的一站式助手。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=UAs6-gNzFks&t=4s) 缘起与演进：从 Substrate 基础设施到 Cosmic 容器平台 微软高级软件工程师 Shridivya（Divya）Sharma 介绍了她与同事 Luke Kelly 在微软超大规模（Hyperscale）生产环境下的调试体系重塑经验，尤其是如何在 Microsoft Cosmic 平台上保障调试安全。在深入了解 Cosmic 之前，必须先理解微软的 Substrate。Substrate 是支撑整个 Microsoft 365（包括 Exchange、OneDrive、SharePoint 等服务）的底层基石平台，为其在全球范围内可靠运行提供共享基础设施、部署模型和运营控制。在早期阶段，各个业务团队各自管理专属的基础设施，这种方式虽然赋予了团队高度的灵活性，但随之而来的是碎片化的环境、五花八门的部署模式，以及参差不齐的安全与合规水位。 尽管 Substrate 后来通过推行标准化部署模型解决了上述许多痛点，但“容量管理”（Capacity）仍然是一大瓶颈。过去容量通常与具体的物理机器紧密绑定，需要极为繁琐的事前容量规划，一旦业务需求发生剧烈波动，系统很难动态扩缩容。这促使团队构建一个全新的共享平台——不仅能够动态按需分配资源，还能提供一个统一的标准环境，让各类工作负载即插即用、稳定运行。 Cosmic 正是在这一背景下诞生的。Cosmic 代表“基于 Substrate 托管的智能集群容器平台”（Container on Substrate Managed Intelligent Clusters）。它基于 Azure Kubernetes Service (AKS) 构建，将 Substrate 拓展为一个容器化的多租户平台。在 Cosmic 架构下，部署、自动扩缩容、流量路由和安全合规全部由 Cosmic 核心团队统一兜底负责，彻底解放了上游业务团队。这种由平台全权接管的设计不仅统一了运行规范，从架构根源上固化了合规边界，也极大提高了整体基础设施的利用率与资源隔离性。 ### [02:34](https://www.youtube.com/watch?v=UAs6-gNzFks&t=154s) 超大规模治理挑战与集群调试工具（CDT）的诞生 Cosmic 当前的体量极为庞大，其运行规模已达到惊人的超大规模：在全球范围横跨 12,000 多个 Kubernetes 集群，承载着 Microsoft Teams 以及众多核心 M365 服务的运行。在任意时刻，Cosmic 都同时托管着数千万个 Pod 和数十万个节点，底层拥有近 600 万个计算核心（Cores），并广泛分布在微软 Azure 的每一个公有云与主权云（Sovereign Clouds）区域。无论 Windows 还是 Linux 负载，都能在 Cosmic 的统一环境中平稳运行，这也使 Cosmic 成为微软内部仅次于第一大客户的 AKS 第二大租户。 然而，在如此庞大的超大规模集群上进行生产调试，曾给运维安全带来巨大冲击。过去，生产环境的排错高度依赖常态化的“高权限访问”（Standing High-Privilege Access）——工程师为了帮业务伙伴排查问题，往往可以直接连入生产集群。随着上云业务与集群数量井喷式爆发，这种传统模式难以为继：它不仅留下了难以审计的安全隐患与常驻特权，还让业务方在遇到极其基础的故障时严重依赖底层平台团队。团队随后尝试过针对 Linux 或 Windows 开发的多种分散工具，但修修补补的做法导致了工具链割裂与操作体验不一致。经过数轮架构迭代，团队最终打造了核心统一方案——集群调试工具（Cluster Debugging Tool，简称 CDT）。CDT 彻底颠覆了旧模式，采用即时授权（Just-in-Time, JIT）机制：访问权限仅针对特定排错任务临时授予，任务结束立即自动销毁，整个执行过程处于严格隔离的环境中，并天然抹平了操作系统底层的差异。 - **运行规模指标**：12,000+ 个集群、数千万个 Pod、数十万个 Node、近 600 万计算核心。 - **云与系统覆盖**：覆盖所有 Azure 公有云和主权云区域，同时原生支持 Windows 与 Linux。 - **业务定位**：承载 Teams 及核心 M365 资产，系微软内部 Azure Kubernetes Service (AKS) 第二大使用方。 - **核心痛点转变**：从高危且难审计的常驻高权限（Standing Access）转向免常驻权限、任务级绑定的 JIT 隔离排错。 ### [04:42](https://www.youtube.com/watch?v=UAs6-gNzFks&t=282s) 零常驻权限下的安全调试：CDT 架构与五步执行闭环 微软工程师 Luke Kelly 详细拆解了 CDT 的系统架构。面对成千上万跨操作系统、跨云环境的节点，要在不泄露权限的前提下排查那些只在超大规模并发下才会暴露的诡异 Bug，CDT 提供了“交互式排错、零常驻访问、严格受限授权与端到端完整审计跟踪”的核心机制。整个排错链路从工程师遇到线上故障（例如 Pod 崩溃或异常）发起调查开始，完全由 CLI 驱动，将底层极其严密的五层安全拦截对用户完全透明化。 ```mermaid flowchart TD accTitle: CDT 安全调试流程 accDescr: 工程师通过 CLI 触发调试，经由认证服务、短效 SSH 证书、JIT 临时服务并在集群控制器中运行容器执行 subgraph UserInteraction["用户操作层"] A["工程师执行 CLI 命令<br/>(指定 Target 与 Debug 命令)"] end subgraph SecurityControl["安全与授权层"] B["授权服务 (Authorization Service)<br/>核验身份/白名单/审批流"] C["签发短效 SSH 证书<br/>(嵌入 ForceCommand 关键约束)"] end subgraph ClusterExecution["集群执行层"] D["创建临时 JIT SSH Server<br/>(企业内网限制/无交互 Shell)"] E["K8s 自定义控制器<br/>调度临时环境并直连容器运行时"] end subgraph AuditLayer["全面审计"] F["全流程端到端审计日志<br/>(人员/目标/命令/标准输出/时长)"] end A --> B B --> C C --> D D --> E B -.-> F C -.-> F D -.-> F E -.-> F ``` 1. **CLI 极简交互体验**：工程师不需要手写任何复杂的 K8s YAML 配置，也不用手动配置底层 SSH。他们只需在命令行执行一条单行指令，指定调试目标（具体的 Pod 或 Node，不论系统是 Linux 还是 Windows）以及想要执行的命令。CLI 层封装了所有复杂性，工程师的使用感受就像是在本地直接运行命令一样自然。 2. **强制授权判定（关键路径）**：命令发送至后端的授权服务。工程师没有任何常驻权限，每一条调试命令都需要独立授权。系统会严格核对“此人是谁”、“具备何种资格”，以及“该命令是否在允许列表”。所有命令都实行集中化策略管理，只读命令（如抓取进程日志）能走低门槛审批，而高风险操作则必须经过 Team Lead 甚至事故指挥官（Incident Manager）的显式审批。Luke 特别强调，授权绝不是外挂在系统侧面的补丁，“没有授权，就没有执行”（"No authorization, no execution."）。 3. **签发内嵌约束的短效 SSH 证书**：授权通过后，系统不会直接抛给用户一个开放式的 Shell 会话，而是颁发一个寿命仅以“分钟”计的短效 SSH 证书。团队利用了 SSH 原生的强制命令机制（`ForceCommand`），将批准的具体命令直接嵌入到证书的“关键选项”（Critical Options）中，同时绑定目标、用户身份、命名空间等全部元数据。这意味着下游组件根本不需要回调授权服务，只需在本地通过签名校验合法性即可解析出所有强制约束。 4. **受限的 JIT SSH 连接**：拿到证书后，CLI 触发创建一个专用的即时（JIT）SSH 服务端。该连接受到严苛的网络策略限制，仅允许来自微软企业内网的访问。默认情况下完全禁用交互式 Shell，命令执行完成后连接立刻掐断。 5. **集群内控制器与无发行版容器（Distroless）穿透执行**：在集群内部，自定义 Kubernetes Controller 监听调试请求并快速拉起瞬态执行环境。该环境接收证书、验证签名并提取内嵌指令直接执行，完全忽略证书之外的任何用户额外输入。针对 Pod 目标，该临时容器直接穿透并与节点底层容器运行时（Container Runtime）通信，即便目标是一个连 Shell 或基础诊断工具都没有的无发行版极简容器（Distroless Container），也能顺畅排查。最后，进程的标准输出（stdout）、标准错误（stderr）和退出码均原汁原味回传给客户端，同时“全流程内嵌审计”（"Every component in the pipeline emits audit trails"）会完整记录谁在何时、对何目标、执行了多久以及命令输出了什么。 ### [08:30](https://www.youtube.com/watch?v=UAs6-gNzFks&t=510s) 迈向智能化：基于 COP 平台与 Agent-to-Agent 架构的诊断编排 尽管 CDT 解决了安全连入与合规执行的问题，工程师依然需要面对繁重的认知负荷：面对突发警报，究竟该拉取哪个 Pod 日志？何时需要执行节点排空（Drain Node）？面对成百上千篇故障排查手册（Troubleshooting Guides, TSG），到底该参考哪一篇？如果所有决策都需要依赖工程师个人的领域专业知识，排障效率依然会受限于人力瓶颈。 为了简化这一全流程，Divya 阐述了将 AI 编排层引入排障体系的全新演进。在 Cosmic 体系内，团队原本就为业务方打造了一个名为 COP（Cosmic Operational Portal，Cosmic 运维门户）的统一控制台。该门户为所有合作团队提供了单一全景视图，涵盖服务健康度、Pod 状态、节点运行状况、集群部署与灰度发布进度等指标，其初衷是让业务团队无需凡事找底层平台求助，即可自主完成初筛（First-level Triage）。 如今，团队在 COP 界面之上搭建了由“语义内核”（Semantic Kernel）驱动的“Agent 到 Agent”（Agent-to-Agent）智能协同架构： - **中央 AI 编排器（Central AI Orchestrator）**：作为智能中枢，它负责解析工程师的自然语言诉求与告警上下文，并自主规划解决路径，决定调用哪些子模块。 - **专业子 Agent 集群**：编排器不再事事亲力亲为，而是通过规范的 Agent-to-Agent 协议动态指派任务给各个专业代理。这其中包括基于 CDT 工具链的安全诊断 Agent、沉淀了海量历史事故复盘与处置预案的知识库 Agent，以及负责执行底层平台治理动作的执行 Agent。 - **假设检验闭环**：各个专业 Agent 在接收到编排指令后，会基于实时指标与历史基线动态构建排查假设，并自动联动 CDT 等工具在受控沙箱中安全获取推断证据，进而大幅加速线上故障的根因定位（RCA）。 ### [10:36](https://www.youtube.com/watch?v=UAs6-gNzFks&t=636s) 从手动排障转向 AI 编排：多层深度防御的只读安全架构 在传统的故障排查流程中，值班工程师需要手动提出假设、运行命令并反复求证，而借助 AI Agent，这一流程演进为完全由系统编排驱动的端到端自动化工作流。智能体能够并行发起多条任务：一边分析容器日志与排查系统健康状况，一边实时检索相关的排障指引文档；多个专职智能体各司其职，最终将分散采集到的事实汇聚成一份结构化报告交还给工程师。 为了让 AI 智能体进入超大规模生产集群而不引入安全风险，系统在设计之初就坚持了“全链路默认只读”的原则。研发团队在工具层、鉴权层、提示词层和运行容器层构建了四层纵深防御，从物理和逻辑上彻底杜绝了智能体对集群状态进行任何非授权写操作的可能： ```mermaid flowchart TD accTitle: AI Agent 多层只读安全架构 accDescr: 展示工具层、鉴权层、提示词层及运行时沙盒的四层只读安全设计 subgraph SecurityLayers ["四层只读安全防御体系"] direction TB TL["工具层 (MCP Tools)<br/>仅开放 get / list / describe，无任何写 API"] AL["鉴权层 (Auth Layer)<br/>On-Behalf-Of Token 继承用户自身 RBAC，严禁提权"] PL["提示词层 (Prompt Layer)<br/>系统级指令严密约束仅允许只读行为"] RL["运行时环境 (Runtime Sandbox)<br/>沙盒容器运行，物理限制宿主机节点访问"] end TL --> AL --> PL --> RL ``` - **工具层（Tool Layer）**：所有暴露给智能体的 MCP（Model Context Protocol）工具均仅支持 `get`、`list` 和 `describe` 等只读操作，系统完全没有注册任何包含变更或写操作的 API。 - **鉴权层（Auth Layer）**：采用代理凭据交换（On-Behalf-Of Token Exchange）机制，智能体严格继承发起请求的当班工程师自身的 RBAC 权限，绝对不会被授予超过用户权限范围之外的访问能力。 - **提示词层（Prompt Layer）**：系统指令（System Instructions）层面做了强制约束，明确规约智能体的一切决策与推理都仅限于只读行为。 - **运行时环境（Runtime Layer）**：智能体执行环境被封闭在沙盒容器中运行，对底层宿主机节点的网络和资源访问被施加了极为严格的限制。即使工程师在对话中主动命令智能体去执行修改动作，各防御层也会协同拦截，从设计上确保了写操作无法发生。 --- ### [12:15](https://www.youtube.com/watch?v=UAs6-gNzFks&t=735s) 生产评估表现与杜绝幻觉的五大护栏机制 为了验证智能体在真实运维中的有效性，系统在涵盖诊断（Diagnostics）与修复（Remediation）的 19 类典型故障场景中进行了全面评测。其中诊断测试细分为 16 种故障类型，包括 Pod 处于挂起状态（Pending Pod State）、性能抖动、运行时异常、网络互通故障以及容器镜像拉取退避（ImagePullBackOff）等核心痛点。当前系统对“修复”环节保持了克制，仅限于自动化生成排障建议命令，并不直接代行执行。 评测数据显示，智能体在真实复杂场景下的根因分析准确率（Root Cause Accuracy）达到了 85% 至 90%，事实依附度（Factual Grounding）在 70% 至 80% 之间，平均排障响应时间被压缩在 37 秒左右，极大缩短了故障定界耗时。 为确保输出结论的可解释性并消除大模型的幻觉风险，系统构建了严密的排障护栏（Guardrails）： - **证据溯源要求**：智能体提出的每一个分析结论和断言，都必须明确附带具体的执行工具调用与查询参数，彻底杜绝无依据的推断。 - **执行步数熔断**：单次排查被严格限制在最多 10 次工具调用周期（Tool Cycles）之内，防止模型陷入失控的死循环逻辑；在实际运行中，智能体通常会远早于上限完成收敛。 - **强制检索知识库**：每次排障过程中，智能体必须至少查阅一次官方知识库与运维指南，确保诊断方案牢牢依托团队已有的排障标准。 - **标准 RCA 格式输出**：排查结果统一以标准化 RCA 报告形式返回，包含总体摘要、根本原因判定、附带来源引用的发现数据表，使工程师能一眼看清诊断推导的来龙去脉与可信赖度。 - **防幻觉安全兜底**：系统内置幻觉防护机制，一旦智能体检索并分析后未发现有力证据，必须明确返回“我无法找到任何证据（I could not find any evidence）”，严禁无中生有编造推论。 --- ### [13:49](https://www.youtube.com/watch?v=UAs6-gNzFks&t=829s) CDT 的未来演进：打造智能时代合规审计的运维工作台 微软计划将 CDT（集群排障工具）沉淀已久的成熟审计追踪（Audit Trail）与核心工具链全面带入“智能体时代”（Agent Age）。未来的规划是将系统所积累的全部运维工具、团队 Wiki 以及架构文档预先打包进一个专用的排障 Shell 中，并让 AI 智能体直接驻留在这一交互环境内部。 在这个架构下，智能体既精通各类调试工具的使用技巧，又能即时查阅海量的企业内部文档。它将为一线值班工程师提供一个真正的一站式排障工作台（One-Stop Shop）：工程师只需连接至合规且全程受审的沙盒容器，就能在 AI 智能体的即时辅助下高质高效地完成复杂的生产集群故障排查。 整个方案的核心宗旨在于：无需为任何工程师开放横跨海量 Kubernetes 集群的全局常驻权限（Standing Access），就能实现更安全、更智能的大规模生产环境调试。在保证权限即用即审、动作全程可追溯的前提下，深度融合 AI 自动化调查与严格的安全护栏，最大化赋能云原生基础设施的可靠性运维。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=UAs6-gNzFks&t=4s) 基于 Substrate 托管的智能集群容器平台 — Shridivya Sharma > [04:42](https://www.youtube.com/watch?v=UAs6-gNzFks&t=282s) 交互式排错、零常驻访问、严格受限授权与端到端完整审计跟踪 — Luke Kelly > [04:42](https://www.youtube.com/watch?v=UAs6-gNzFks&t=282s) No authorization, no execution. — Luke Kelly > [04:42](https://www.youtube.com/watch?v=UAs6-gNzFks&t=282s) 没有授权，就没有执行 — Luke Kelly > [04:42](https://www.youtube.com/watch?v=UAs6-gNzFks&t=282s) Every component in the pipeline emits audit trails — Luke Kelly > [04:42](https://www.youtube.com/watch?v=UAs6-gNzFks&t=282s) 全流程内嵌审计 — Luke Kelly > [10:36](https://www.youtube.com/watch?v=UAs6-gNzFks&t=636s) 全链路默认只读 — Shridivya Sharma > [12:15](https://www.youtube.com/watch?v=UAs6-gNzFks&t=735s) I could not find any evidence — Shridivya Sharma > [12:15](https://www.youtube.com/watch?v=UAs6-gNzFks&t=735s) 我无法找到任何证据 — Shridivya Sharma > [13:49](https://www.youtube.com/watch?v=UAs6-gNzFks&t=829s) 无需为任何工程师开放横跨海量 Kubernetes 集群的全局常驻权限（Standing Access），就能实现更安全、更智能的大规模生产环境调试。 ## Mentioned Microsoft, Shridivya Sharma, Luke Kelly, Microsoft Substrate, Microsoft 365, Microsoft Teams, Exchange, OneDrive, SharePoint, Microsoft Azure, Azure Kubernetes Service (AKS), Cosmic (Container on Substrate Managed Intelligent Clusters), CDT (Cluster Debugging Tool), COP (Cosmic Operational Portal), Semantic Kernel, Model Context Protocol (MCP), Kubernetes, SSH, ForceCommand, Distroless Container ## Video description This presentation will go over how Microsoft uses SSH to debug across our global fleet of K8s clusters at hyperscale. We'll cover how we integrate that with JIT elevations and Task-Based Execution in a secure and auditable way. We will alsogo over how we use these building blocks to engage AI and the Semantic Kernel to empower our on-call engineers to resolve issues quickly. Learn more about the @Scale conferences here: atscaleconference.com [Source](https://www.youtube.com/watch?v=UAs6-gNzFks)

[0:04] Shridivya Sharma: 基于 Substrate 托管的智能集群容器平台 [4:42] Luke Kelly: 交互式排错、零常驻访问、严格受限授权与端到端完整审计跟踪 [4:42] Luke Kelly: No authorization, no execution. [4:42] Luke Kelly: 没有授权，就没有执行 [4:42] Luke Kelly: Every component in the pipeline emits audit trails [4:42] Luke Kelly: 全流程内嵌审计 [10:36] Shridivya Sharma: 全链路默认只读 [12:15] Shridivya Sharma: I could not find any evidence [12:15] Shridivya Sharma: 我无法找到任何证据 [13:49] 无需为任何工程师开放横跨海量 Kubernetes 集群的全局常驻权限（Standing Access），就能实现更安全、更智能的大规模生产环境调试。

[4:42] Luke Kelly: we didn't bolt authorization onto the side. It's the critical path. No authorization, no execution. [4:42] Luke Kelly: zero standing access, scoped authorization, and a full audit trail. [4:42] Luke Kelly: minutes, not hours. [4:42] Luke Kelly: If it happens, we know about it. [11:18] Shridivya Sharma: I could not find any evidence

[0:04] Hi everyone. I'm Divya and I'm a senior [0:07] software engineer working on the [0:08] Microsoft Substrate. [0:10] Today, Luke and I am going to talk about [0:13] how we have rethought and secured the [0:15] production debugging at hyperscale, [0:17] specifically within Microsoft Cosmic. [0:20] I will walk through how the platform [0:22] shifts changes the way we debug the live [0:24] systems and how we are using the AI [0:27] agents to make the debugging, the root [0:29] cause analysis safer, faster and more [0:32] scalable. [0:34] Before I talk about what Microsoft [0:36] Cosmic is, I should talk about what [0:38] Substrate is. [0:40] Substrate is the foundational platform [0:41] layer beneath M365. [0:44] It provides the shared infrastructures, [0:46] the deployment models and the [0:48] operational controls that services like [0:50] Exchange, OneDrive, SharePoint depend on [0:54] to run reliably at global scale. [0:58] In early days, individual services [1:00] managed much of their infrastructure. [1:03] That was fine because that gave the team [1:04] flexibility, but it also led to the [1:07] fragmented environments with different [1:09] deployment models and varying levels of [1:12] security and compliance. [1:14] Substrate addressed many of these [1:16] challenges by introducing standardized [1:18] deployment models. [1:20] But one thing that we could not solve [1:22] fully was capacity. [1:24] Capacity was still largely tied to the [1:26] machine level. That required significant [1:29] upfront planning, making it difficult to [1:32] adjust dynamically as demand changed. [1:36] This created the need for a shared [1:38] foundation that could not only allocate [1:41] the [1:42] the capacity dynamically, but it also [1:45] provides a unified platform where the [1:47] workloads could just come on board and [1:50] run consistently. [1:52] Cosmic was born out of that need. [1:55] Cosmic stands for container on substrate [1:58] managed intelligent clusters. [2:01] It extends substrate with a [2:03] container-based multi-tenant platform [2:06] built on Azure Kubernetes services where [2:09] the deployment, scaling, routing, and [2:12] compliance, everything is handled by the [2:14] Cosmic team rather than these individual [2:16] services. [2:18] This shift standardized how services [2:21] run. [2:22] It improved the isolation. It enforced [2:24] the compliance boundaries by design. And [2:27] it also enabled far more efficient use [2:30] of [2:31] the infrastructure. [2:34] Cosmic today runs across 12,000 clusters [2:38] hosting services like Microsoft Teams [2:40] and other M365 services. [2:43] At any given point of time, we are [2:45] hosting tens of millions of pod and [2:48] hundreds of thousands of nodes. [2:50] We are backed by nearly 6 million cores [2:53] of compute and we are spread across [2:55] every Azure regions, be it public clouds [2:58] and sovereign clouds. [3:00] We support both Windows and Linux [3:02] workloads running in single unified [3:05] environment in Cosmic. [3:07] Cosmic is the second largest customer of [3:09] Azure Kubernetes services at Microsoft. [3:15] Earlier, the production debugging [3:17] dependent on high privilege accesses [3:19] where engineers could just directly [3:21] access the production clusters to help [3:24] our partners troubleshoot their issues. [3:26] As Cosmic scaled and more clusters and [3:30] workloads started onboarding on the [3:32] Cosmic clusters, this approach stopped [3:34] being practical. It created standing [3:37] accesses that was hard to audit. It [3:39] created security risk. And it also made [3:42] partners dependent on us to troubleshoot [3:45] very basic issues. [3:47] Over the time, we developed many [3:50] different tools to address these [3:51] challenges. [3:53] Some really worked for Linux workloads. [3:55] Some really worked for Windows. [3:58] Although both of them tried to solve the [4:00] parts of these problems, this approach [4:02] led to fragmented tooling and [4:04] inconsistent operational experiences. [4:08] Through several iterations, this led to [4:10] the development of a core unified [4:12] solution, the cluster debugging tool, [4:14] which we also call as CDT. [4:17] CDT addresses this in just-in-time [4:20] model. [4:21] The access is granted only for a [4:23] specific debugging task, and it is [4:25] automatically revoked once the task is [4:27] completed. The task is executed in an [4:31] isolated environment, and this tool is [4:33] completely OS agnostic. [4:36] In the next slide, Luke will walk us [4:38] through how this works in practice. [4:41] Over to you, Luke. [4:42] >> Thanks, Divya. Hey everyone. My name is [4:44] Luke, and I'm a C engineer at Microsoft. [4:46] Today, I'm going to be talking about the [4:48] cluster debugging tool, or CDT. We run [4:51] Kubernetes at a massive scale. We have [4:53] hundreds of thousands of nodes, multiple [4:55] operating systems, different clouds and [4:57] architectures. This makes [4:59] troubleshooting debugging issues really [5:00] hard, especially for those pesky ones [5:02] that only show up at scale. [5:04] We built a framework that gives [5:06] engineers interactive debugging with [5:08] zero standing access, scoped [5:10] authorization, and a full audit trail. [5:12] I'm going to walk you through how that [5:13] works step by step now. [5:16] Everything starts with an engineer. [5:18] There's an incident or an investigation [5:20] into a failing pod and the need to run a [5:21] command against a specific target. [5:24] Whether that be a pod or a node, [5:25] regardless of its operating system, they [5:27] don't need to write YAML, they don't [5:29] need SSH, they just run a single CLI [5:31] command. [5:32] Specifically, the target and the debug [5:34] command they want to execute. The CLI is [5:36] the user experience layer. It hides all [5:38] the complexity you're about to see in [5:40] the next five steps to the engineer. It [5:42] feels like they're running their command [5:43] locally. [5:45] The first back-end hop is to our [5:47] authorization service. [5:49] As I said earlier, the engineer has no [5:50] standing access. Every single debug [5:53] command is individually authorized. [5:55] The authorization checks for simple [5:56] things like who is this person, what are [5:58] their eligibilities, and is the command [6:00] they're requesting even allowed. [6:02] Commands are essentially managed, and [6:04] each one is defined its own approval [6:05] requirements. [6:07] Some are safe for read-only commands [6:08] like checking a process logs. Others can [6:10] be much more risky and require explicit [6:13] approval from a team lead or even an [6:15] incident manager. [6:16] Key design design decision here is that [6:18] we didn't bolt authorization onto the [6:20] side. It's the critical path. [6:22] No authorization, no execution. [6:26] Once the command is approved, the system [6:27] doesn't just hand back a session. [6:29] It issues a short-lived SSH certificate [6:32] where the authorized command is embedded [6:33] directly into the SSH critical options. [6:37] This is using SSH's native force command [6:39] along with some specifics for the [6:42] target, the user, the namespace, and [6:44] everything else all the other metadata [6:46] that's required. [6:47] The certificate is the key. [6:49] This means downstream components don't [6:50] need a call back to the auth service. [6:52] They just validate the certificate [6:53] signature, read the embedded [6:55] constraints. [6:56] Certificates are only valid for a short [6:58] window. We're talking about minutes, not [7:00] hours. They're scoped to an execution [7:02] target and a user. [7:06] With a certificate in hand, the CLI [7:08] establishes an encrypted SSH connection [7:10] to the target. [7:12] First by prompting the creation of a JIT [7:14] SSH server. [7:15] This is purpose-built for our specific [7:18] constraints. [7:19] The connection is locked down at [7:20] multiple layers, at the network level, [7:22] only allowing access from corporate [7:24] networks, [7:25] to many other different constraints. By [7:27] default, there is no interactive shell. [7:29] The user executes the command, and the [7:31] termination SSH sub connection is [7:33] terminated. [7:36] Inside the cluster, a custom controller [7:38] watches for debug requests and schedules [7:40] these ephemeral environments. [7:43] They accept the certificate, validate [7:44] the signature, extract the constraints, [7:46] and execute the command. [7:48] User input beyond the certificate is [7:50] completely ignored. When the command [7:52] finishes, the session closes. [7:54] For pod targets, the container interacts [7:57] directly with the container runtime on [7:58] the node. This lets us debug even [8:00] distroless containers that have no shell [8:02] or tooling of their own. [8:04] Standard out and standard error, exit [8:06] codes are all preserved. It feels like [8:08] you ran the command yourself. And [8:10] everything is logged. Every component in [8:11] the pipeline emits audit trails. Who ran [8:15] it, against what target, for how long, [8:16] and what the output was. The audit trail [8:18] isn't an afterthought, it's bolted on. [8:20] It's baked into every component from [8:22] start to finish. Every step in the flow [8:24] you've seen today produces the audit [8:25] record. [8:27] If it happens, we know about it. [8:29] Back to you, Divya. [8:30] >> Thank you, Luke. [8:31] So, yeah. So, today we already have [8:33] tooling like CDD that allows engineers [8:36] to execute operational commands and [8:38] interact with the platform directly. [8:41] Alongside that, we also have other [8:43] diagnostic methods such as pulling the [8:46] Kubernetes pod logs, draining a node, [8:48] and following the troubleshooting guides [8:50] to triage the incidents and determine [8:53] the right course of actions when [8:55] something goes wrong. [8:57] Now, the challenge here is that it still [8:59] depends on the user's domain knowledge. [9:01] Like which command to run, when to run [9:03] it, and how to interpret the output of [9:05] that command, and which troubleshooting [9:08] guide that one has to follow when the [9:10] incident happens. [9:11] Now, what we're trying to do here is to [9:13] simplify that experience. Instead of [9:16] users driving the workflow step by step, [9:19] we introduce an AI-driven orchestration [9:22] layer on the top of these tools. [9:24] At Cosmic, we have a unified operational [9:27] portal called COP, which gives our [9:29] partners a single consolidated view into [9:32] the health of their services, pods, [9:35] nodes, [9:36] cluster deployment, and rollout status. [9:39] The intent behind it was very simple. [9:42] We wanted the partners to able to [9:44] investigate issues on their own. [9:46] Because that saves a lot of time on the [9:48] both the sides. [9:50] And it also reduces the dependencies on [9:52] our team for the first level triage. [9:56] This is where the agent-to-agent [9:58] architecture comes in. [10:00] We introduced a central AI orchestrator [10:03] to this interface [10:05] that understands the user request and [10:07] decides how to solve it. [10:10] Instead of doing everything on its own, [10:13] it coordinates with a set of specialized [10:15] agents, including the diagnostic agent [10:18] through CDT, [10:19] a knowledge base of historical incidents [10:23] and platform actions. [10:25] Using the agent-to-agent protocol, the [10:28] orchestrator dynamically delegates the [10:31] task to these agents. [10:33] Each agent then forms a hypothesis, [10:36] runs a tool to validate it, [10:39] and refines based on what it finds. [10:42] For example, it can analyze the logs, [10:46] it can look into the system health, [10:49] and it retrieves all the relevant [10:51] guidance. Everything is happening in [10:53] parallel. [10:55] Each agent focuses on the specific [10:57] capability. [10:59] It retrieves all the information, brings [11:02] it back into a single response to the [11:05] user. [11:06] So, overall, we move from a manual [11:09] user-driven model to AI orchestrated [11:11] one, where the system composes and [11:14] executes the right workflow end-to-end. [11:18] It is important for us to talk about the [11:20] security, the evaluation, and the [11:22] guardrails that we have set up on these [11:25] AI agent. [11:27] On the security side, all the layers are [11:29] read-only by design. [11:31] At the tool layer, the MCP tools that we [11:34] expose only allow get, list, and [11:37] describe operations. There are no write [11:39] APIs. [11:40] At the auth layer, we use on behalf of [11:43] tokens exchange, which means the agent [11:46] is going to inherit the user's R back [11:48] and is never elevated beyond what the [11:51] user can already do. [11:53] At the prompt layer, the system [11:55] instructions explicitly enforce [11:57] read-only behavior. [11:58] And at the runtime layer, the agent [12:00] containers runs in a sandbox environment [12:03] where the node access is limited. [12:06] So, even if the user asks the agent to [12:09] take an action, it cannot by design at [12:12] multiple layers. [12:15] Regarding the correctness, we evaluated [12:18] the system across 19 scenario types [12:21] covering both diagnostics and [12:23] remediation. [12:24] The diagnostics had 16 different types [12:27] including the pending pod state, [12:29] performance, runtime, networking, and [12:32] image back off issues. [12:34] Remediation today is limited to only [12:36] command generation. [12:38] Root cause accuracy that we found is [12:41] around 85 to 90%. [12:43] The factual grounding is around 70 to [12:46] 80% and the average response time of [12:49] these agents is around 37 seconds. [12:53] Regarding the guardrails, [12:55] every claim that the agent make has to [12:58] cite a specific tool and query [13:00] so that there is no unverified [13:02] assertions. [13:04] We kept each investigation at 10 tool [13:06] cycles to prevent the runaway loops, [13:10] though it always converge much earlier. [13:13] Every investigation must consult the [13:15] knowledge base at least once [13:18] so that the agent is grounded. [13:21] The output is always comes back in a [13:23] standardized RCA format with a summary, [13:26] a root cause, finding tables with the [13:29] source attributions. So, users knows [13:32] what to trust and where it came from. [13:35] And last but not the least, we build in [13:37] a hallucination guard. [13:39] That means if the agent doesn't find any [13:42] evidence for a claim, [13:44] it always comes back and says, "I could [13:46] not find any evidence." Rather than [13:48] fabricating one. [13:49] >> Thanks, Divya. All right, let's talk [13:51] about what's next for CDT. [13:54] Specifically, what we're trying to do is [13:55] take the awesome audit trail and tooling [13:57] that we built in the CDT tool and bring [13:59] it forward into the agent age. We're [14:02] trying to pre-package all of our tools, [14:04] all of our wikis and documentation into [14:06] a shell for the user. And then bring it [14:08] agents inside of that who have access to [14:10] all of the tools, or understand how to [14:12] use them, and all of the documentation [14:14] that exists. This provides a one-stop [14:16] shop for all of our on-call engineers to [14:19] connect to a compliance and audited [14:22] container, and still do everything they [14:24] need to do with agents to help them. [14:27] Back to you, Divya. [14:29] >> To wrap things up, we wanted to make the [14:31] production debugging safer and smarter [14:34] without giving everyone standing access [14:37] to every other cluster. [14:39] We also added AI agents to help [14:42] investigate issues automatically. [14:44] Made sure that there are proper [14:46] guardrails in place.