Philo Deploys MCP AI Agents to Automate Cloud Triage and Drift Scans

AI-Driven Platform Engineering With Philo — Ai4

Ai4

Binding language models to structured MCP tool interfaces eliminates platform engineering toil by enabling self-service triage, automated Prometheus pruning, and drift reconciliation.

AI-assisted summary of AI-Driven Platform Engineering With Philo — original source. Check the original for full context.

Source published: 2026-08-20

Detailed notes

## Background Published by Ai4 on YouTube on August 20, 2026, this presentation features Nitish Mane, an Infrastructure Engineer at Philo. According to the page description, the session explores how platform engineering teams leverage the Model Context Protocol (MCP) and frameworks like FastMCP across cloud-native tools to eliminate operational toil, speed up incident response, and enable self-service infrastructure.

## TL;DR Nitish Mane demonstrates how Philo applies AI agents powered by the Model Context Protocol (MCP) to transform cloud infrastructure operations and empower service developers. The talk walks through concrete MCP integrations across the stack, including Kubernetes pod and network debugging, Prometheus cardinality optimization via Grafana, autonomous Terraform drift detection, and automated on-call handoffs and postmortems with PagerDuty. Mane concludes by showing how to build custom, secure internal MCP servers using FastMCP with built-in RBAC, pagination, and PII redaction guardrails.

## Atmosphere & style The presentation carries a pragmatic, relatable conference vibe grounded in the shared operational headaches of platform engineering, DevOps, and on-call rotations. Mane speaks in an approachable, engineer-to-engineer style, validating common frustrations like repetitive Slack triage and 1:00 AM shift handoffs with humor and real-world war stories. The overall arc feels empowering and reassuring, demystifying AI agents from speculative hype into deterministic, structured tools that meaningfully reduce burnout.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=4s) The primary goal of AI-driven platform engineering at Philo is to eliminate repetitive operational triage questions and empower service developers to debug their own workloads self-sufficiently. - [01:55](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=115s) The Model Context Protocol (MCP) replaces stochastic LLM guesswork and raw API hallucinations with structured, deterministic tool definitions bound to explicit platform actions. - [03:04](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=184s) Kubernetes MCP servers transform cluster debugging into self-service workflows by autonomously diagnosing unschedulable pending pods, networking ingress route failures, and config rollout triggers. - [05:23](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=323s) Integrating Grafana, Loki, and Tempo via official MCP servers enables natural-language telemetry querying, bypassing the need to hand-craft intricate PromQL or LogQL queries during incident triage. - [06:44](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=404s) Philo resolves Prometheus Out Of Memory (OOM) crashes by using Grafana MCP to inventory active dashboard metrics, identify unused scrape targets, and generate relabeling configs to drop unneeded time series. - [09:05](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=545s) Embedding an `agent.md` context file within Terraform module directories provides AI models with explicit repository conventions, approved providers, and architectural layouts. - [09:05](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=545s) Autonomous daily drift detection agents leverage Terraform MCP to evaluate plans against cloud state, generating targeted `terraform import` blocks for unmanaged resources and mitigating change anxiety. - [10:54](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=654s) Argo CD MCP servers surface synchronization errors such as immutable storage class edits, missing container tags, and multi-cloud configuration discrepancies across AWS, GCP, and OCI. - [12:41](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=761s) PagerDuty MCP acts as the primary operational entry point, automating overnight shift handoffs by aggregating Slack discussions and alert history into structured Notion documentation. - [13:43](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=823s) Incident postmortems are streamlined by using PagerDuty MCP to automatically reconstruct participant timelines, acknowledgement metrics, and diagnostic tool links. - [15:11](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=911s) GitLab MCP servers provide structured, AI-native schemas that outperform raw CLI text outputs when feeding failed CI/CD pipeline job logs and complex merge requests into LLMs for root-cause analysis. - [16:29](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=989s) The n8n MCP server enables automated pre-scaling and load testing workflows ahead of high-traffic broadcast surges, such as the *Yellowstone* finale, while extending automation to non-technical teams. - [17:42](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=1062s) Custom MCP servers built with FastMCP allow safe querying of sensitive internal systems like Amazon RDS by enforcing PII redaction, strict RBAC, and response pagination before data reaches the model context. - [17:42](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=1062s) Any standard OpenAPI microservice specification can be automatically converted into an MCP server without manually hand-crafting protocol boilerplate.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=4s) Introduction to Philo and the Platform Engineering Reality

Nitish Mane introduces himself as a cloud infrastructure engineer working on the backend at Philo. When mingling in the hallways before the session, he noticed that many attendees were unfamiliar with what Philo actually does, so he begins by setting the stage. Philo is a direct-to-consumer (B2C) cable TV streaming platform delivering live entertainment in partnership with major television network providers such as Paramount and HBO. Subscribers sign up for the consumer product to stream television channels, and Mane's primary charter on the infrastructure engineering team is to maintain and guarantee that the core video streaming pipelines remain consistently operational and resilient.

Addressing the audience, Mane gauges the room by asking how many attendees work as DevOps engineers, platform engineers, or Site Reliability Engineers (SREs). He touches on a universal frustration shared across platform teams: the constant, repetitive interruptions. Every time a platform engineer gets pinged with an operational question, the immediate internal reaction is, "Why am I getting pinged for this?" The goal of the session is exploring how teams can transition away from answering mundane triage questions and instead empower service developers to be fully self-sufficient in debugging their own workloads, thereby unlocking 10x engineering productivity across the organization.

[01:55](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=115s) Defining the Model Context Protocol (MCP) in Platform Automation

Before diving into specific tooling, Mane establishes a working definition of the Model Context Protocol (MCP). While the concept has been a dominant theme throughout the conference, he clarifies its concrete architectural role: MCP provides a structured, deterministic tool-calling interface through which large language models (LLMs) or agentic workflow harnesses execute external actions against underlying systems.

```mermaid flowchart LR accTitle: Architecture of AI Agent with MCP Integrations subgraph Core ["Agentic Control Plane"] A["LLM / Agent Harness"] end subgraph Protocols ["Model Context Protocol (MCP)"] B["Kubernetes MCP"] C["Grafana / Loki / Tempo MCP"] D["Terraform MCP"] E["Argo CD MCP"] F["PagerDuty MCP"] end subgraph Infra ["Target Infrastructure"] G["K8s Clusters & Pods"] H["Metrics, Logs & Traces"] I["Cloud IaC State"] J["GitOps Deployments"] K["On-Call & Incidents"] end

A --> B & C & D & E & F B --> G C --> H D --> I E --> J F --> K ```

Mane emphasizes that MCP replaces stochastic guesswork with explicit tool definitions. Rather than asking an LLM to hallucinate shell commands or raw API interactions, the model is bound strictly to pre-declared tool calls exposed by each MCP server. This constrained environment ensures that autonomous workflows only invoke valid, authorized operations against cloud-native infrastructure.

[03:04](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=184s) Streamlining Kubernetes Troubleshooting with MCP

Mane highlights Kubernetes as the industry's default workload platform. While there is not yet an official Cloud Native Computing Foundation (CNCF) MCP server for Kubernetes, several robust open-source implementations provide full CRUD operations against Kubernetes clusters. Giving an LLM harness access to a Kubernetes MCP server transforms everyday cluster debugging from a platform-team bottleneck into an automated self-service experience.

Developers deploying microservices frequently reach out to platform teams when pods become stuck in a `Pending` state. An LLM armed with the Kubernetes MCP server can immediately inspect the cluster state to pinpoint whether the pod is unschedulable due to CPU or memory resource exhaustion, node limits being hit, or missing node taints and tolerations. Similarly, Mane points out that microservice networking across ingress controllers is notoriously tedious to debug. When multi-endpoint ingress routes fail to forward traffic due to incorrect path definitions, blocked internal ports, or firewall misconfigurations, an agent can trace the routing path directly through the Kubernetes API. The protocol also simplifies day-two operational chores, such as automatically rolling out deployments when underlying application secrets change.

- Common root causes resolved via Kubernetes MCP include exhausted node pod capacity limits, unsatisfied pod anti-affinity rules, and missing tolerations. - Routing failures spanning multi-endpoint ingress controllers, service definitions, and security groups can be diagnosed without manual `kubectl` context switching. - Enables autonomous trigger rollouts of pods and stateful workloads immediately following configuration or secret updates.

[05:23](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=323s) Observability Workflows: Grafana, Prometheus, Loki, and Tempo

Observability is a cornerstone of Philo's operational stack, where open-source Grafana forms the centerpiece alongside Prometheus, Loki for logs, and Tempo for distributed tracing. Mane highlights that Grafana has released official MCP server implementations supporting Loki and Tempo. Integrating these telemetry backends into an AI context allows engineers to query complex operational metrics, log streams, and traces using natural language rather than crafting intricate PromQL or LogQL queries by hand.

The Grafana MCP server drastically reduces the barrier to building dynamic dashboards and analyzing metric anomalies during production alerts. By pulling rich time-series context and distributed spans straight into the agent's context window, engineers can cross-examine active alarms against metric spikes across microservices, isolating root causes far faster than manual dashboard hopping permits.

[06:44](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=404s) Production Triage: Broken Dashboard Plugins and Prometheus OOMs

Mane shares two distinct, practical war stories demonstrating how Philo applies Grafana MCP in day-to-day operations. The first involved an upgrade to Grafana 12. As Grafana updates, its sprawling ecosystem of community and third-party plugins automatically updates alongside it. Following the upgrade, a critical internal business user plugin containing custom JavaScript abruptly stopped rendering because underlying backend API endpoints had changed. Tracing the breakdown through the Grafana MCP server and an LLM enabled Mane's team to rapidly diagnose the altered endpoints and identify the exact JavaScript changes needed to restore the form plugin.

The second issue centers on the common dilemma of Prometheus Out Of Memory (OOM) crashes. When engineering teams initially stand up Prometheus, they routinely leave scrape targets at default configurations, ingesting every metric emitted by client libraries under the assumption that someone might need them later. As cardinality explodes, the Prometheus instance repeatedly crashes due to memory limits:

> "If you have seen this issue before, you know that this issue can be solved using dropping unused metrics, right? So how do you find out which metrics are not used?"

Using the Grafana MCP server, an AI agent can scan every dashboard active in Grafana, compile an inventory of every metric actually referenced in dashboard panels, and cross-reference that list against the active Prometheus scrape dictionary. The agent then generates Prometheus metric-relabeling overrides to systematically drop all unreferenced metrics, curbing memory pressure while leaving an easy path to re-enable them if future requirements demand it. Mane adds that this approach similarly catches broken dashboard variables that produce "no data" states when upstream metric names change across software version migrations.

[09:05](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=545s) Terraform MCP and Autonomous Drift Detection

Philo relies on Terraform for all of its Infrastructure as Code (IaC). To optimize how AI models generate and maintain Terraform configurations, Mane recommends structuring IaC repositories with dedicated markdown guidance files, such as an `agent.md`, placed directly within every module directory. This practice gives the agent explicit context regarding directory layouts, module relationships, approved providers, and standard conventions, removing the guesswork over which resource or data blocks to construct.

```mermaid flowchart TD accTitle: Autonomous Terraform Drift Detection Workflow subgraph Detection ["Scheduled Scan"] S["Daily Scheduled Job"] --> T["Terraform MCP Server"] T --> P["Generate `terraform plan`"] end subgraph AgenticAnalysis ["AI Drift Agent"] P --> A["Analyze Discrepancies\n& Cloud Account State"] A --> G["Categorize Drift"] end subgraph Remediation ["Actionable Output"] G --> R1["Suggest `terraform import`\n(Unmanaged Resources)"] G --> R2["Suggest Code Reverts\n(Manual Console Changes)"] G --> R3["Prepare PR / Plan Review"] end ```

A major challenge in mature organizations is configuration drift. Startups rarely begin their lifecycle with disciplined IaC practices; they construct their core products first, only standardizing on Terraform and Kubernetes later once multiple team members already possess direct access to cloud consoles. Consequently, manual tweaks accumulate, creating chronic drift. When engineers follow proper IaC processes by submitting merge requests, those requests often stall indefinitely because teams fear that applying changes might inadvertently sever a network port or break a dependent service. To eliminate this anxiety, Philo implemented an autonomous drift detection agent:

- Runs on a daily automated cadence via the Terraform MCP server to generate plans against live cloud infrastructure. - Analyzes plan outputs against actual cloud account configurations to distinguish between unmanaged resources and unintended manual modifications. - Recommends concrete actions, such as generating precise `terraform import` blocks for newly adopted assets or alerting teams to safely reconcile obsolete resources.

[10:54](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=654s) GitOps Delivery with Argo CD MCP

For continuous deployment and GitOps reconciliation, Philo relies on Argo CD to continuously synchronize live cluster state with git repositories. Mane explains that deployments frequently end up stuck in an unhealthy or degraded status for subtle reasons. Common culprits include attempts to alter immutable storage class fields on persistent volumes, references to container image tags that have been pruned or do not exist, missing Custom Resource Definitions (CRDs), and pods trapped in continuous `CrashLoopBackOff` cycles.

An Argo CD MCP server surfaces these synchronization failures directly to the LLM agent. The agent can quickly surface when development-specific container tag overrides were mistakenly pushed to production environments where those tags are absent. Furthermore, when deploying across complex multi-cloud and multi-region topologies—such as workloads distributed across Oracle Cloud Infrastructure (OCI), AWS, and Google Cloud Platform (GCP)—the Argo CD MCP integration allows engineers to filter, synchronize, and inspect applications across heterogeneous cloud environments simultaneously.

[12:41](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=761s) Incident Response and Shift Handoff with PagerDuty MCP

Mane highlights the PagerDuty MCP server as arguably the most critical and frequently used integration in the platform toolkit, describing it as the primary entry point for operational AI agents. PagerDuty sits at the center of Philo's on-call lifecycle, aggregating alerts across disparate systems, notifying the appropriate engineers, and tracking incident resolution and schedules.

Because Philo organizes on-call coverage across specialized teams—including frontend, backend, billing, and platform engineering—handoff friction during shift rotations is an ongoing challenge. An engineer woken up at 1:00 AM to handle an incident typically struggles to deliver a comprehensive, structured briefing when their shift concludes and the next engineer rotates on call. By integrating PagerDuty via MCP, Philo began designing an automated on-call handoff agent capable of tracking overnight incidents, summarizing remediation steps taken, and ensuring that critical context seamlessly transfers across shifts without human fatigue causing operational blind spots.

[13:43](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=823s) Automating On-Call Handoffs and Postmortems with MCP

Shifting on-call responsibilities between engineers often introduces operational friction and lost context. Nitish Mane describes how the platform team at Philo found manual handoffs increasingly tedious because engineers had to spend extensive time explaining ongoing issues and active system states to the next person coming on rotation. To streamline this transition, they built an automated handoff agent. Operating as a scheduled workflow during shift transitions, the agent pulls recent communication logs from Slack, aggregates the state of active operations, and compiles an organized Notion document so the incoming engineer knows precisely where to take over.

```mermaid flowchart TD accTitle: Automated On-Call and Incident Workflow subgraph Inputs["Telemetry & Comms"] A["Slack Discussions"] B["PagerDuty Alerts"] end subgraph AgentLayer["Agentic Processing"] C["Handoff & Incident Agent"] D["PagerDuty MCP Server"] end subgraph Outputs["Generated Documentation"] E["Shift Handoff (Notion)"] F["Timeline & Postmortem"] end A --> C C --> E B --> D D --> F ```

The team applied a similar agentic approach to incident postmortems—an essential engineering requirement that is universally regarded as "another tedious task which we all hate to do, but we have to do it." When an outage occurs, engineers must reconstruct who was present on the incident call, when alerts were acknowledged, links to external diagnostic tools, and potential revenue impact. Using a PagerDuty MCP server, the incident timeline and alert metadata are automatically synthesized into a comprehensive retrospective document. Beyond postmortems, MCP serves as a conversational on-call assistant capable of dynamically pulling organizational context—such as identifying the exact engineering point of contact for the billing team during a service disruption.

- **Automated Handoffs:** Slack channel activity is parsed and formatted directly into a Notion handoff page. - **Incident Postmortems:** PagerDuty MCP structures timeline events, responders, acknowledgement timestamps, and external metric links. - **Conversational Directory:** Agents route domain-specific questions to the correct on-call teams during live triaging.

---

[15:11](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=911s) GitLab MCP Server for CI/CD Diagnostics and PR Reviews

While many organizations operate on GitHub Actions, Philo's infrastructure ecosystem runs heavily on GitLab. Using a GitLab MCP server allows developers and platform engineers to create and update pull/merge requests, inspect failed CI/CD pipeline job logs, and feed runtime diagnostic output straight into an LLM for root-cause analysis. This capability substantially reduces review fatigue on massive, complex merge requests that might otherwise consume an entire sprint's worth of engineering time.

Mane addresses a common architectural question: why deploy an MCP server when command-line tools like the official `glab` CLI already exist? The critical distinction lies in formatting and intent. While CLI tools output raw terminal streams engineered for human inspection, an MCP server delivers structured, AI-native schemas optimized for language model tokenization, tool-calling, and reasoning.

```mermaid flowchart LR accTitle: GitLab MCP Diagnostic Pipeline subgraph Pipeline["GitLab CI/CD"] A["Job Failure Logs"] end subgraph Bridge["Protocol Layer"] B["GitLab MCP Server\n(Structured Schemas)"] end subgraph Assistant["AI Diagnostic Layer"] C["LLM / Claude Code"] D["Root Cause & MR Fix"] end A --> B B --> C C --> D ```

- **Pipeline Failure Remediation:** Job logs from failing CI runs are ingested directly into the LLM context to pinpoint errors. - **MR Comprehension:** AI-assisted descriptions and clear inline comments make massive code changes digestible for reviewers. - **CLI vs. MCP:** Unlike human-oriented CLI tools like `glab`, MCP servers provide deterministic, AI-native responses designed for model consumption.

---

[16:29](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=989s) Event-Driven Scaling and Workflows with the n8n MCP Server

Connecting LLMs to the workflow automation platform n8n provides a natural-language orchestration layer across technical and non-technical business units. As a streaming service, Philo experiences massive, predictable traffic surges during marquee broadcast events—such as season finales of major shows like *Yellowstone*—or high-concurrency retail milestones like Thanksgiving sales. Platform engineers utilize an n8n MCP server to rapidly generate and trigger automated workflows that pre-scale cluster resources and run load tests in advance of peak viewership.

This workflow interface also extends AI capabilities beyond core infrastructure teams. Cross-functional departments, including data analytics, product, and marketing, can utilize the n8n MCP server to automatically compose and execute business-specific automation flows without requiring dedicated platform engineering interventions.

- **High-Concurrency Scaling:** Automates cluster pre-warming and load testing for major streaming events like the *Yellowstone* finale. - **Cross-Functional Automation:** Marketing and data teams leverage n8n MCP integrations to generate operational workflows on demand.

---

[17:42](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=1062s) Building Custom, Secure MCP Servers with FastMCP and OpenAPI

When out-of-the-box MCP servers do not cover internal tooling or sensitive environments—such as direct access to production databases—engineering teams must implement custom servers. FastMCP, a high-level Python framework, allows platform engineers to expose internal services securely. Mane presents a pattern for querying production Amazon RDS databases where the custom MCP server acts as an enforcement boundary, automatically filtering out personally identifiable information (PII) before records reach the model's context window.

```mermaid flowchart TD accTitle: FastMCP Production Guardrails Flow subgraph Client["AI Agent Layer"] A["Developer AI Agent"] end subgraph CustomServer["FastMCP Server Layer"] B["FastMCP Gateway"] C["RBAC Authorization"] D["PII Redaction & Pagination"] end subgraph Storage["Internal Infrastructure"] E["Production RDS DB"] end A -->|"SQL Query Request"| B B --> C C --> E E -->|"Raw DB Records"| D D -->|"Sanitized, Truncated Payload"| A ```

Furthermore, any existing internal microservice with a standard OpenAPI specification can be automatically converted into an MCP server, removing the need to hand-craft protocol boilerplate. Mane emphasizes two non-negotiable operational guardrails when building and exposing custom MCP servers:

| Guardrail | Purpose & Implementation | |:--- |:--- | | **Role-Based Access Control (RBAC)** | Restricts sensitive actions (e.g., direct database access, infrastructure provisioning) strictly to authorized platform engineers. | | **Response Pagination & Chunking** | Prevents large query outputs or log dumps from exhausting the LLM's context window and driving up inference costs. |

Mane closes the talk by reassuring engineers about the expanding role of AI in platform operations, noting, "don't be afraid of the agents, they are here to help us."

## Notable quotes > [00:04](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=4s) Why am I getting pinged for this? — Nitish Mane

> [06:44](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=404s) If you have seen this issue before, you know that this issue can be solved using dropping unused metrics, right? So how do you find out which metrics are not used? — Nitish Mane

> [13:43](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=823s) another tedious task which we all hate to do, but we have to do it. — Nitish Mane

> [17:42](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=1062s) don't be afraid of the agents, they are here to help us. — Nitish Mane

## Mentioned Nitish Mane, Philo, Ai4, Paramount, HBO, Model Context Protocol (MCP), Kubernetes, Cloud Native Computing Foundation (CNCF), Grafana, Prometheus, Loki, Tempo, Terraform, Argo CD, PagerDuty, Slack, Notion, GitLab, GitHub Actions, Claude Code, n8n, Yellowstone, FastMCP, Python, OpenAPI, Amazon RDS, AWS, Google Cloud Platform (GCP), Oracle Cloud Infrastructure (OCI)

## Video description Nitish Mane - Infrastructure Engineer, Philo

Platform Engineering teams are increasingly using AI to reduce operational toil, speed up incident response, and enable self-service infrastructure. This talk explores using AI as a natural-language interface to cloud-native systems through the Model Context Protocol (MCP). It shows how platform engineers apply AI to workflows like infrastructure authoring, debugging, monitoring, and alert analysis. The session demonstrates practical MCP integrations, including Terraform, Kubernetes, Grafana, AWS, Alertmanager, and GitLab MCP servers for tasks such as IaC generation, drift detection, resource inspection, and CI/CD decision-making. Finally, it introduces FastMCP, a framework for safely building custom MCP servers that expose internal tools to AI.

[Source](https://www.youtube.com/watch?v=zxQRuDs-gkU)

中文简报

Philo 部署 MCP AI Agent 自动化云故障排查与漂移扫描将语言模型与结构化 MCP 工具接口相绑定，通过实现自助排障分诊、自动化 Prometheus 裁剪与漂移对齐，消除了平台工程的繁重琐事。 Nitish Mane 展示了 Philo 如何运用基于模型上下文协议（MCP）的 AI Agent 来变革云基础设施运维，并赋能业务开发者。演讲详细介绍了跨技术栈的具体 MCP 集成，包括 Kubernetes Pod 与网络调试、通过 Grafana 进行 Prometheus 基数优化、Terraform 自主漂移检测，以及结合 PagerDuty 的自动化值班交接和复盘总结。Mane 在最后展示了如何使用 FastMCP 构建安全、定制化的内部 MCP 服务器，并内置 RBAC、分页和 PII 脱敏等安全护栏。 Philo 开展 AI 驱动平台工程的首要目标，是消除重复性的运维排障分诊咨询，并赋能业务开发者自主调试自身的工作负载。 (0:04) 模型上下文协议（MCP）通过将明确的平台操作与结构化、确定性的工具定义相绑定，取代了大语言模型的随机猜测以及调用原始 API 时产生的幻觉。 (1:55) Kubernetes MCP 服务器通过自主诊断无法调度的 Pending 状态 Pod、网络 Ingress 路由故障以及配置发布触发因素，将集群调试转变为自助式工作流。 (3:04) 通过官方 MCP 服务器集成 Grafana、Loki 和 Tempo，可实现自然语言遥测查询，在故障排查初审过程中无需手动编写复杂的 PromQL 或 LogQL 查询。 (5:23) Philo 利用 Grafana MCP 清点活跃仪表盘指标、识别未使用的抓取目标，并生成重新标记（relabeling）配置以丢弃不需要的时间序列，从而解决了 Prometheus 内存溢出（OOM）崩溃问题。 (6:44)

Reading summary

Nitish Mane argues that platform engineering teams can overcome constant operational interruptions by adopting the Model Context Protocol (MCP) to give AI agents structured, deterministic access to infrastructure tools. This architecture replaces manual triage with developer self-service across cluster debugging, observability optimization, configuration drift, and incident management. Deterministic Tool Calling for Cluster and CI/CD Diagnostics (1:55) Nitish Mane defines the Model Context Protocol (MCP) as a structured tool-calling interface that prevents large language models from hallucinating shell commands by binding them strictly to pre-declared system operations. In Kubernetes environments, open-source MCP implementations allow agents to inspect live cluster state directly, identifying why pods stall in a Pending state—such as node capacity limits, unsatisfied pod anti-affinity rules, or missing tolerations—and tracing broken multi-endpoint ingress routes without manual kubectl context switching. Mane contrasts this protocol-driven approach with traditional developer command-line interfaces such as GitLab's glab CLI. While CLI utilities emit raw text designed for human reading, GitLab MCP servers return structured schemas optimized for model tokenization and reasoning. This enables agents to ingest failed CI/CD job logs directly to diagnose build breakages and summarize sprawling merge requests to alleviate review fatigue. Telemetry Analysis and Prometheus Memory Optimization (5:23) Philo's observability architecture combines Grafana, Prometheus, Loki, and Tempo, where official Grafana MCP implementations allow engineers to query logs and traces using natural language instead of handcrafting complex PromQL or LogQL queries. Mane illustrates this capability through two production triage scenarios. In one instance, an upgrade to Grafana 12 broke an internal business plugin due to modified backend API endpoints; an agent using Grafana MCP identified the altered endpoints and pinpointed the exact JavaScript changes needed to restore the form. In another scenario, Mane addresses recurring Prometheus Out Of Memory (OOM) crashes triggered by high cardinality from uncurated scrape targets. An AI agent queries the Grafana MCP server to compile an inventory of every metric actively used across production dashboards, compares it against the active Prometheus scrape dictionary, and generates metric-relabeling configurations to drop unreferenced metrics automatically, curtailing memory pressure without permanently deleting collection paths. Continuous Drift Detection and Multi-Cloud GitOps Reconciliation (9:05) To improve how models interact with Infrastructure as Code (IaC), Mane recommends placing explicit guidance files (such as an agent.md) inside Terraform module directories to outline approved providers, folder hierarchies, and block structures. Philo utilizes an autonomous drift detection agent that runs daily via a Terraform MCP server. By evaluating generated plan outputs against live cloud configurations, the agent distinguishes between unmanaged assets and unintended manual console edits, generating terraform import commands or safe code reconciliations to unblock stalled merge requests. For continuous delivery, Philo links Argo CD to an MCP server to diagnose synchronization stalls across multi-cloud deployments spanning Amazon Web Services (AWS), Google Cloud Platform (GCP), and Oracle Cloud Infrastructure (OCI). The agent detects common reconciliation failures, such as modifications to immutable storage class fields, missing Custom Resource Definitions (CRDs), and invalid development image tags mistakenly pushed into production manifests. Incident Response Automation and Secure Custom Gateways (13:43) Mane identifies the PagerDuty MCP server as Philo's primary operational entry point. To minimize context loss and fatigue across shift rotations, Philo developed an automated handoff agent that analyzes Slack discussions and active alerts to generate organized transition notes in Notion. For outage postmortems, the agent extracts responder timelines, alert acknowledgement timestamps, and external diagnostic links directly from PagerDuty to draft retrospective documentation. When out-of-the-box integrations are insufficient, Mane explains that teams can build custom MCP servers using the FastMCP Python framework or by converting existing OpenAPI specifications. When exposing sensitive internal systems like Amazon RDS production databases, Mane mandates strict guardrails: role-based access control (RBAC) to restrict provisioning actions, automated redaction of personally identifiable information (PII), and response pagination to keep database payloads from exhausting model context windows. Philo also pairs an n8n MCP server to orchestrate cluster pre-warming ahead of major viewership surges, such as the Yellowstone season finale. Mane concludes that platform engineers should view autonomous agents as operational collaborators rather than replacements, emphasizing that sustainable adoption depends on pairing model access with explicit architectural boundaries like RBAC, data sanitization, and deterministic schemas.

精炼阅读摘要

Philo云基础设施工程师Nitish Mane指出，平台工程团队长期受困于重复琐碎的日常排障打扰，提出通过模型上下文协议（MCP）构建AI智能体控制平面，将大语言模型与Kubernetes、Grafana、Terraform及PagerDuty等云原生工具深度集成，以实现开发者自主排障并释放团队工程生产力。确定性架构：MCP取代随机性猜测 (1:55) Mane阐明，MCP的架构核心在于为大语言模型或智能体运行环境提供结构化、确定性的工具调用接口。以往直接让模型猜测Shell命令或直接操作API极易引发幻觉，而MCP通过显式暴露预声明的工具，将智能体严格限制在受控且合规的操作集合内，确保与云原生基础设施交互的确定性与执行安全性。针对为何不沿用现有CLI工具（如GitLab的glab）的疑问，Mane解释称传统命令行输出主要针对人类阅读，数据格式粗糙且易造成Token冗余；而MCP服务器提供的是针对模型Token化、工具调用与逻辑推理优化的AI原生结构化模式，能更高效地完成报错分析与合并请求审查。集群排障与可观测性智能诊断 (3:04) 在Kubernetes集群中，微服务开发者频繁面临Pod处于Pending挂起状态或Ingress路由转发失败的问题。集成Kubernetes MCP后，智能体能够直接调取API状态，自主定位是CPU与内存耗尽、节点容量超限，还是污点容忍配置缺失及路径定义错误，并能在配置或密钥更新后自动触发工作负载滚动更新。在可观测性领域，基于Grafana官方支持Loki和Tempo的MCP服务，工程师可用自然语言直接检索时序指标、分布式链路与日志流，替代繁琐的手写PromQL或LogQL。Philo曾通过该方案成功修复Grafana 12升级中因底层API变更而失效的自定义JavaScript看板插件；针对Prometheus常见的内存溢出（OOM）崩溃，智能体通过扫描所有活动看板引用的指标，与Prometheus抓取字典比对，自动生成指标重标记规则以过滤无用指标，从根源上缓解基数爆炸压力。 IaC漂移治理与跨云GitOps交付 (9:05) 针对初创团队在未规范IaC之前因控制台人工干预而积累的配置漂移难题，Philo在Terraform每个模块目录中配置名为agent.md的指导文件，为智能体提供明确的目录结构、模块关联与合规Provider上下文。团队部署了基于Terraform MCP的自动化检测智能体，每日定时执行terraform plan，对比云账户实际状态以区分未受管资源与意外修改，并主动生成精准的terraform import配置或代码回滚建议，消除团队对变更引发服务中断的顾虑。在持续交付层面，Philo利用Argo CD持续同步集群状态与Git仓库。Argo CD MCP服务器可实时捕获持久卷不可变存储类变更、镜像标签缺失、CRD缺失及CrashLoopBackOff等异常状态，并在Oracle云（OCI）、AWS和GCP等多云混合部署中跨环境过滤与排查应用，防止开发环境的标签覆盖误推至生产集群。自动化值班交接与事件驱动伸缩 (12:41) PagerDuty是Philo平台运维AI智能体的核心接入点。针对跨前端、后端、计费等多团队轮值时，夜间值班工程师因疲劳无法高效交接的痛点，Philo构建了自动化交接智能体。该智能体在换班周期内自动拉取Slack沟通记录与系统运行状态，汇总为结构化的Notion文档。在事故复盘场景中，智能体结合PagerDuty告警元数据，自动提取事故响应人员、确认时间戳及监控诊断链接，消除繁琐的人工复盘阻力，并能动态路由故障排查责任人。此外，借助自动化平台n8n的MCP集成，基础设施团队能够在大规模流媒体转播（如《黄石》季终集）或感恩节促销等高并发流量来临前，通过自然语言触发自动预热集群与压测工作流；该接口同时支持数据、产品与营销团队在无需平台团队介入的情况下自主构建业务流。自定义服务集成与生产安全防护 (17:42) 针对自研内部工具以及直接访问生产数据库等敏感场景，Mane展示了利用基于Python的高阶框架FastMCP搭建自定义服务器的模式。对于具备标准OpenAPI规范的内部微服务，平台可快速将其转换为MCP服务器，避免繁重的协议开发工作。在安全防护机制上，Mane强调了两个必须落实的生产护栏：其一是基于角色的访问控制（RBAC），严格限制数据库查询和资源配置等高危权限仅对获授权的平台工程师开放，并在数据进入模型上下文前对生产RDS数据库记录进行自动个人身份信息（PII）脱敏；其二是响应分页与切片截断机制，防止大规模查询日志或全量数据直接撑爆大语言模型的上下文窗口，进而控制不必要的推理成本。 Nitish Mane总结强调，工程师无需担忧自主智能体取代人力，其本质是辅助平台团队减负的效能杠杆。然而根据Philo的实践，该体系的有效运转高度依赖规范的上下文元数据（如模块内的指导文档）以及严格的访问控制与数据脱敏护栏；如何在扩大智能体自主操作范围的同时确保多云环境的受控与安全，仍是推进AI平台工程的核心前提。

Detailed notes — zh

## Background 本内容由频道 Ai4 于 2026 年 8 月 20 日发布在 YouTube 平台上，演讲者为流媒体公司 Philo 的基础设施后端工程师 Nitish Mane。视频主题围绕平台工程团队如何利用大模型与模型上下文协议（Model Context Protocol，简称 MCP）降低运维琐事、加速故障排查并实现基础设施自助化。 ## TL;DR 演讲系统介绍了 Philo 团队如何借助 MCP 协议连接 Kubernetes、Grafana、Terraform、Argo CD、PagerDuty、GitLab 及 n8n 等底层基础设施，构建覆盖 IaC 漂移检测、CI/CD 根因分析、流媒体流量高峰编排与自动化值班交接的 AI Agent 体系。同时，针对安全合规痛点，团队利用 FastMCP 框架自建工具服务，实现了敏感 PII 数据脱敏、RBAC 权限控制与上下文窗口保护，为业务研发提供了受控且高效的自助运维能力。 ## Atmosphere & style 整场分享呈现出务实、严谨且极具一线实战气息的技术沉淀感。讲者从基础设施工程师面对频繁打扰与技术债的真实痛点切入，逻辑层层递进，既有对具体技术踩坑（如版本升级、OOM 治理）的坦诚剖析，也有对架构演进的清晰构想。演讲情绪克制而自信，传递出对拥抱 AI Agent 变革、重塑平台工程生产力的开放与坚定信念。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=4s) Philo 平台工程团队通过引入 AI 与 MCP 机制赋能业务研发自助排障，旨在消除运维中的“打扰疲劳”并大幅提升人效。 - [01:55](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=115s) MCP 为大语言模型提供了标准化的工具调用边界与控制机制，消除了盲目操作与凭空猜测的风险，确保基础设施操作的安全可控。 - [01:55](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=115s) 借助 Kubernetes MCP，业务研发无需平台工程师介入即可用自然语言排查 Pod Pending、资源限制与网络路由等问题，并自动化执行滚动重启。 - [05:23](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=323s) Grafana MCP 将可观测性数据（日志、链路与指标）注入模型上下文，极大简化了仪表盘构建以及组件升级过程中的 API 兼容性排障。 - [05:23](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=323s) 利用 Grafana MCP 扫描大盘实际引用的指标，能够自动生成 PromQL 丢弃规则以清理无用指标，有效治理 Prometheus OOM 内存溢出并降低资源成本。 - [09:05](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=545s) 在 IaC 目录中配置 agent.md 说明文件，Terraform MCP 能精准指导 Agent 生成规范代码，并通过自主定时检测 Plan 解决历史遗留的配置漂移难题。 - [10:54](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=654s) Argo CD MCP 支持大模型直接提取应用同步事件与日志，快速诊断不可变字段变更、镜像 Tag 缺失及 CRD 缺失导致的 GitOps 交付异常。 - [12:41](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=761s) 基于 PagerDuty MCP 构建的交接智能体（Handoff Agent）在值班轮换时自动归纳未决告警与排查上下文，有效避免跨团队信息断层。 - [13:43](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=823s) 智能体通过聚合 Slack 聊天与 PagerDuty 事件，可自动化生成 Notion 交接页面与事故复盘（Postmortem）报告，大幅减轻工程师的文档负担。 - [15:11](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=911s) 相比面向人类的传统 CLI 工具，GitLab MCP Server 提供了更适合 AI 消费的结构化接口，可实现流水线秒级归因诊断并优化复杂 PR 的审查体验。 - [16:29](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=989s) 通过 n8n MCP Server，不仅能用自然语言快速编排流媒体大事件前夕的集群弹性扩容与压测，还能赋能非技术团队按需构建业务数据管道。 - [17:42](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=1062s) 使用 FastMCP 构建内部服务时，可在查询生产数据层内置脱敏逻辑，安全过滤 PII 敏感信息，避免终端用户隐私泄露给上层 Agent。 - [17:42](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=1062s) 落地 AI 基础设施必须严格推行 RBAC 角色权限控制与分页截断策略，防止非法越权操作并避免海量原始数据挤爆模型的上下文窗口。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=4s) Philo 的背景与平台工程的提效困境讲者 Nitish 是 Philo 公司的云基础设施后端工程师。Philo 是一家 B2C 有线电视与实时流媒体服务平台，与 HBO、Paramount 等多家大型电视网络提供商合作，为注册用户提供在线电视直播服务。Nitish 在团队中主要负责维护与保障底层视频流管道（video streaming pipeline）的高可用与稳定运行。在日常工作中，平台工程、DevOps 和 SRE 团队常常面临严重的“打扰疲劳”——每当业务团队遇到故障，平台工程师就会收到各种排查请求，陷入“为什么这种事又来找我”的被动响应模式。本次演讲的核心目标，正是探讨如何利用 AI 和 MCP（Model Context Protocol，模型上下文协议）赋能平台工程，让业务研发团队具备自助排障能力（self-sufficient），从而减少日常运维琐事，帮助工程师成为“10x 工程师”并成倍提升工作效率。 ```mermaid flowchart LR accTitle: 平台工程 AI 赋能架构 subgraph AgenticLayer["AI Agent / LLM 编排层"] LLM["LLM / Agent Harness"] end subgraph MCPLayer["MCP 协议工具层 (Tool Calls)"] K8sMCP["Kubernetes MCP"] GrafanaMCP["Grafana / Loki MCP"] TFMCP["Terraform MCP"] ArgoMCP["Argo CD MCP"] PDMCP["PagerDuty MCP"] end subgraph InfraLayer["底层基础设施与平台"] K8s["K8s 集群"] Obs["监控与日志系统"] IaC["云资源配置 (IaC)"] GitOps["GitOps 持续交付"] Alerts["值班告警系统"] end LLM --> MCPLayer K8sMCP --> K8s GrafanaMCP --> Obs TFMCP --> IaC ArgoMCP --> GitOps PDMCP --> Alerts ``` --- ### [01:55](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=115s) 什么是 MCP 与 Kubernetes MCP 落地实战 MCP 本质上是一套标准化的工具调用（tool calls）机制，让 LLM 或 Agent Harness 能够安全、精准地执行外部动作。与让大模型凭空猜测不同，MCP 为 Agent 提供了明确的工具调用边界，Agent 只能在授权范围内执行预定义操作，彻底消除了盲目操作的风险。在 Kubernetes 运维方面，虽然 CNCF 尚未发布官方的 Kubernetes MCP Server，但开源社区已有许多成熟的 CRUD 实现。在实际业务中，研发人员常因资源不足导致 Pod 卡在 Pending 状态，或遇到 CPU 限制、缺少 Toleration、节点达到 Pod 数量上限等问题；通过接入 Kubernetes MCP，研发无需呼叫平台工程师，即可通过自然语言让 Agent 自行调取集群状态排查根因。 - **排查 Ingress 与网络路由**：在大型微服务架构中，常因多 Endpoint 路由配置错误、端口被阻断或防火墙问题导致流量不通，MCP 可快速分析流量链路与配置。 - **配置变更后的滚动重启**：在 Secret 或 ConfigMap 更新后，通过 MCP 自动化触发对应 Deployment 的 Rollout 重启，确保新凭证即时生效。 --- ### [05:23](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=323s) Grafana MCP：从自然语言排障到 Prometheus 内存降本 Philo 深度采用开源的 Grafana 监控生态。Grafana 官方推出的 MCP Server 支持与 Loki（日志）及 Tempo（分布式追踪）联动，让工程师能直接用自然语言查询指标与日志，并将监控上下文直接注入到 LLM 对话中，极大简化了仪表盘构建与告警分析。 Nitish 分享了在升级 Grafana 时的真实踩坑经历：在将 Grafana 升级到更高版本时，某个业务表单插件突然失效。排查后发现该插件内嵌了大量 JavaScript 脚本，因底层 API 端点发生变更而报错；借助 LLM 与 MCP 的上下文检索，团队迅速定位了不兼容的 API 并完成了适配修复。 - **Prometheus OOM 治理与废弃指标剔除**：很多团队在部署 Prometheus 时默认抓取所有指标，导致实例频繁发生 OOM（内存溢出）。利用 Grafana MCP，Agent 可以自动扫描当前所有大盘中实际引用的指标，对比生成 PromQL 丢弃规则（Metric Relabeling/Drop Rules），过滤掉未被任何图表使用的无用指标，显著降低资源消耗。 - **版本迁移中的指标断流诊断**：当组件从低版本升级时（例如指标命名由 `v1` 变更为 `v3`），Grafana 面板常因变量失效而显示“No Data”，LLM 能协助比对指标重命名映射并修复查询表达式。 --- ### [09:05](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=545s) Terraform MCP 与自主架构漂移检测 Philo 的云基础设施完全基于 Terraform 进行代码化管理。借助 Terraform MCP Server，Agent 能精准识别应使用的 Provider、Resource 块及 Data 块，避免大模型生成幻觉代码。为了让 Agent 更好地理解代码库，推荐在 IaC 项目的每个模块目录下放置一个 `agent.md` 说明文件，使 MCP Server 与 Agent 在生成代码前能自主读取当前架构规范与模块依赖。 - **历史技术债导致的配置漂移（Drift）**：初创公司在初期往往优先交付业务，后期才引入 Terraform 和 Kubernetes，期间多人手动操作云控制台，导致云上实际资产与代码严重漂移。 - **长时间挂起的合并请求（MR）**：由于担心 `terraform apply` 会意外删除线上端口或资源，团队常常不敢合并 IaC 变更。 - **自主漂移检测 Agent**：构建定时运行的检测 Agent，自动生成执行计划（Plan），智能分析差异并建议哪些资源应当执行 `terraform import` 纳入纳管，哪些配置需要代码同步，消除人工干预的顾虑。 --- ### [10:54](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=654s) Argo CD MCP：GitOps 交付异常快速诊断在持续部署（CD）环节，Philo 使用 Argo CD 将 Git 仓库中的代码持续同步到生产环境。然而，实际部署中常会出现应用处于“Unhealthy”或“OutOfSync”状态的情况。通过引入 Argo CD MCP Server，LLM 可以直接拉取应用同步状态、事件日志及底层 Kubernetes 资源定义，快速定位应用异常的深层原因： - **不可变字段变更冲突**：例如业务配置尝试修改不可变（Immutable）的 `StorageClass`，导致同步失败。 - **镜像标签（Tag）丢失**：在开发阶段使用了 Dev 覆盖标签，推向生产时镜像未推送或 Tag 错误，导致拉取镜像失败。 - **底层资源缺失与 CrashLoopBackOff**：快速检测集群中是否缺少相应的 CRD（自定义资源定义），或捕获微服务容器的崩溃循环日志。 - **多云多地域同步编排**：针对跨 AWS、GCP、OCI 的混合多云架构，通过 MCP 集中筛选、过滤并协同触发多集群的应用同步。 --- ### [12:41](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=761s) PagerDuty MCP：构建跨团队交接智能体 PagerDuty 作为生产告警与 On-Call 调度的核心中枢，是构建运维 Agent 最关键的入口。在多团队协作场景下（包括前端、后端、计费、平台工程等团队），各团队均有人员轮值 24 小时 On-Call。当凌晨 1 点出现突发线上告警时，当班值班人员处理后往往需要进行跨班次交接。Nitish 团队基于 PagerDuty MCP 构建了“交接智能体”（Handoff Agent），用于在值班轮换（Shift Change）时自动汇总未决告警、处理上下文及跟进状态，避免告警遗漏与信息断层。 ### [13:43](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=823s) 值班交接自动化与事后复盘（Postmortem）生成在平台工程团队中，轮值交接历来是一项极为繁琐且容易遗漏信息的任务。工程师不仅需要编写文档，还要费尽口舌向下一位值班人员逐一解释当前各项任务和系统状态，导致交接过程变得异常冗长耗时。为了解决这一痛点，团队构建了一个专属的“交接 Agent”（Handoff Agent）。该 Agent 作为自动化工作流在交接时刻定时触发，自动从 Slack 频道中拉取近期的值班沟通记录与事件细节，并结构化地汇总生成一份 Notion 页面，让接棒的工程师对全局一目了然，清楚知晓当前各事项的处理进度以及应该从何处接手推进。紧接着是令每位工程师都头疼不已却又不得不做的差事——事后复盘文档（Postmortem）的编写。每当生产环境发生故障，事后都需要耗费巨大精力还原现场：谁当时参与了紧急呼叫（Call）、告警在何时被何人响应认领（Ack）、故障期间是否造成了营收损失（Revenue loss）等，所有细节都必须详实记录进文档。借助 PagerDuty MCP Server，系统能够自动梳理并生成完整的事故事件时间线，智能关联相关外部链接，大幅减轻了值班工程师的案头负担。此外，该方案还提供了对话式值班助手（Conversational On-call Assist）；例如，当值班人员在半夜处理问题却不知该向计费团队（Billing team）的哪位具体负责人求助时，Agent 能够自动将相关人员上下文检索并注入对话中，帮助值班人员快速破局。 ```mermaid flowchart TD accTitle: 值班交接与复盘工作流 subgraph Inputs["数据采集与告警"] A["Slack 交流记录"] B["PagerDuty 告警与呼叫记录"] end subgraph MCP["MCP 协作与处理层"] C["Handoff Agent"] D["PagerDuty MCP Server"] end subgraph Outputs["交付产物"] E["Notion 交接页面"] F["标准化 Postmortem 事故报告"] G["对话式值班问答与人员推荐"] end A --> C --> E B --> D --> F B --> D --> G ``` - **Handoff Agent 落地流程**：在交接周期触发，自动抓取 Slack 频道内关键对话并沉淀到 Notion，消除口头交接容易遗漏信息的问题。 - **Postmortem 自动编排**：抓取包括呼叫参会人、响应时间戳、业务营收影响及外部排查链接在内的完整证据链，全自动化组织故障时间线。 - **动态联系人检索**：值班工程师遇到跨领域疑难杂症时，助手可基于组织架构上下文精准定位计费等垂直业务团队的负责人。 --- ### [15:11](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=911s) GitLab MCP Server：CI/CD 故障排查与代码审查优化在代码托管与持续集成方面，虽然业内不少团队偏好 GitHub Actions，但该团队主要扎根于 GitLab 技术生态。两者在核心概念与管道能力上高度相通。引入 GitLab MCP Server 后，大语言模型（LLM）便具备了直接创建、更新合并请求（MR/PR）以及实时拉取流水线作业日志（Job logs）的能力。当某些 CI/CD 任务发生失败报错时，LLM 可以直接提取失败日志的上下文并进行智能归因诊断，快速定位出构建或测试中断的根本诱因。在实际代码审查（Code Review）中，该工具带来了显著的体验提升。面对那些动辄成百上千行、体量巨大到“足以耗尽整整一个 Sprint 周期”的超大型 PR，工程师往往望而生畏甚至产生抗拒心理。但若在 AI（如 Claude Code 等工具）的协助下生成结构清晰、意图明确的 PR 描述，并附上详实的代码内注释，审查者的心智负担便会大幅减轻。对于部分工程师可能提出的疑问——“既然官方早已有成熟的 CLI 工具 `glab`，为何还要专门引入 MCP Server？”——发言人解释道，`glab` 主要是面向人类终端操作设计的命令行工具，而 MCP Server 则是真正“AI 原生”（AI-native）的交互界面，其输出的数据格式更加规整紧凑，专为大模型的理解与消费进行了结构优化。 - **流水线日志秒级诊断**：CI/CD 任务爆红时，模型直接调取报错片段并给出修复提示，省去人工逐层翻阅冗长日志的成本。 - **审查体验大幅改善**：对于耗时过长的庞大变更，利用 AI 补全上下文与注释，帮助 Reviewer 迅速理清提交意图。 - **AI 原生协议 vs. 传统 CLI**：传统的 `glab` CLI 主要是人机交互工具，而 MCP 协议能以极低的解析摩擦力为 LLM 提供结构化输入。 --- ### [16:29](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=989s) n8n MCP Server：应对流媒体高峰与跨团队工作流编排 n8n 是一款功能强大的开源自动化工作流编排平台。通过搭建 n8n MCP Server 并与 LLM 联动，不仅基础架构工程师能够大幅加速工作流的设计与搭建，还能赋能产品设计、数据分析等其他跨职能业务团队，显著降低其自动化构建门槛。针对 Philo 自身作为流媒体电视公司的业务特质，这一机制展现出了极具说服力的应用场景。每逢平台迎来重大收视热点事件——例如备受瞩目的《黄石》（Yellowstone）季终集开播——或者类似电商平台的“黑五”大促销时，海量并发流量都会对底层系统形成剧烈冲击。为了保障系统平稳度过流量高峰，基础架构团队需要提前执行严苛的负载压力测试，并对集群进行前置扩容。借助 n8n MCP Server，工程师可以通过自然语言指令直接生成扩缩容工作流并由系统自动执行调度，确保整个集群处于最新就绪状态。这种工作流生成能力不仅限于平台团队，营销（Marketing）与数据（Data）团队同样能够灵活调用该能力，按需自动创建自身业务所需的后台数据管道。 - **流媒体大事件备战**：针对热门影视上线或大型促销等高并发流量潮，快速编排压测与集群弹性预扩容方案。 - **全公司多业务线赋能**：将工作流构建能力延伸至数据团队与市场部门，非底层架构人员也能通过对话驱动 n8n 节点生成。 --- ### [17:42](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=1062s) FastMCP 实践与安全治理：敏感数据保护、RBAC 与上下文管理在演讲的最后阶段，发言人深入探讨了平台工程团队如何自主封装敏感能力，并重点介绍了 Python 生态下的快速开发框架 FastMCP。当业务线开发者开始尝试构建自己的 AI Agent 时，往往需要查询生产环境数据库。然而，生产数据库不仅涉及核心机密，还包含大量敏感信息。平台团队的职责就是在确保绝对安全的前提下，受控地向开发者暴露必要的数据接口。利用 FastMCP，团队可以自行编写轻量级 MCP Server，在执行底层 RDS 数据库查询的同时，于服务层内置脱敏逻辑，自动剔除所有个人身份信息（PII，Personally Identifiable Information），确保上层 Agent 绝对无法触碰到真实的终端用户隐私数据。此外，市面上凡是遵循 OpenAPI 标准的第三方 API 规范，如今都能非常便捷地一键转换或生成为 MCP Server，从而极大地拓宽了内部工具的接入边界。 ```mermaid flowchart LR accTitle: FastMCP 安全查询与脱敏流程 subgraph AgentFlow["上层 Agent 交互"] A["开发者 / AI Agent"] end subgraph SafetyServer["FastMCP Server 自建服务"] B["身份校验与 RBAC 权限判定"] C["SQL 查询执行"] D["PII 数据脱敏与过滤"] E["分页切片截断（防止 Context 溢出）"] end subgraph Storage["生产存储"] F[("生产 RDS 数据库")] end A -->|"自然语言 / 查询请求"| B B -->|"安全授权"| C C <-->|"拉取原始数据"| F C --> D --> E E -->|"脱敏且适量的安全响应"| A ``` 在落地此类 AI 基础设施时，发言人强调了几个绝不可忽视的安全与工程治理考量： - **基于角色的访问控制（RBAC）**：由于涉及的基础架构能力极其敏感，必须建立极其严密的角色权限管理机制，确保只有平台工程师或承担高级别职责的核心人员才拥有特定工具的执行权限。 - **数据分页与上下文窗口保护（Pagination）**：严禁向大模型无节制地倾倒海量查询原始数据，因为这会瞬间吞噬有限的上下文窗口（Context Window），必须在 MCP 服务侧推行严格的分页策略与结果限制。 - **拥抱 AI Agent 时代**：发言人在结语中充满信心地向在场同行喊话："不要害怕 Agent，它们是来帮助我们的（Don't be afraid of the agents, they are here to help us.）"，鼓励大家以开放的心态将 Agent 融入日常平台工程实践中。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=4s) 为什么这种事又来找我 — Nitish Mane > [00:04](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=4s) 10x 工程师 — Nitish Mane > [05:23](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=323s) No Data — Nitish Mane > [10:54](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=654s) Unhealthy — Nitish Mane > [10:54](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=654s) OutOfSync — Nitish Mane > [15:11](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=911s) 足以耗尽整整一个 Sprint 周期 — Nitish Mane > [15:11](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=911s) AI 原生 — Nitish Mane > [17:42](https://www.youtube.com/watch?v=zxQRuDs-gkU&t=1062s) 不要害怕 Agent，它们是来帮助我们的（Don't be afraid of the agents, they are here to help us.） — Nitish Mane ## Mentioned Nitish Mane, Philo, Ai4, Model Context Protocol (MCP), Kubernetes, CNCF, Grafana, Loki, Tempo, Prometheus, Terraform, Argo CD, PagerDuty, Slack, Notion, GitLab, GitHub Actions, Claude Code, n8n, FastMCP, AWS, GCP, OCI, HBO, Paramount, 《黄石》（Yellowstone） ## Video description Nitish Mane - Infrastructure Engineer, Philo Platform Engineering teams are increasingly using AI to reduce operational toil, speed up incident response, and enable self-service infrastructure. This talk explores using AI as a natural-language interface to cloud-native systems through the Model Context Protocol (MCP). It shows how platform engineers apply AI to workflows like infrastructure authoring, debugging, monitoring, and alert analysis. The session demonstrates practical MCP integrations, including Terraform, Kubernetes, Grafana, AWS, Alertmanager, and GitLab MCP servers for tasks such as IaC generation, drift detection, resource inspection, and CI/CD decision-making. Finally, it introduces FastMCP, a framework for safely building custom MCP servers that expose internal tools to AI. [Source](https://www.youtube.com/watch?v=zxQRuDs-gkU)

Source quotes

[0:04] Nitish Mane: 为什么这种事又来找我 [0:04] Nitish Mane: 10x 工程师 [5:23] Nitish Mane: No Data [10:54] Nitish Mane: Unhealthy [10:54] Nitish Mane: OutOfSync [15:11] Nitish Mane: 足以耗尽整整一个 Sprint 周期 [15:11] Nitish Mane: AI 原生 [17:42] Nitish Mane: 不要害怕 Agent，它们是来帮助我们的（Don't be afraid of the agents, they are here to help us.）

Source quotes

[0:04] Nitish Mane: Why am I getting pinged for this? [6:44] Nitish Mane: If you have seen this issue before, you know that this issue can be solved using dropping unused metrics, right? So how do you find out which metrics are not used? [13:43] Nitish Mane: another tedious task which we all hate to do, but we have to do it. [17:42] Nitish Mane: don't be afraid of the agents, they are here to help us.

Original transcript

[0:04] Hello everyone, my name is Natish. Uh I work for a company called Pho. Uh I when I was roaming around [0:10] this halls, a lot of people ask me what Pho does. So maybe I'll start from uh what the Pho is about. [0:17] Pho is a cable TV platform where we uh it's a B2C product where we have a where we do live streaming [0:24] and we are partnered with multiple uh network providers like HBO, Paramount and then uh users [0:32] sign up for our service and uh they watch cable TV and I work as a cloud infrastructure engineer back [0:40] end where my job is to manage and make sure the video streaming pipeline is always up and running. [0:46] Uh so how many of you guys are DevOps platform engineering S sur in this room? [0:54] So this talk is about how to you know increase the efficiency of in our day-to-day [1:00] uh activities. So I know that platform engineering team every time they get pinged they feel like [1:07] you know why why am I getting pinged for this and how can I make sure that our teams are [1:12] self-sufficient and can debug their own things. So uh some of the things which we will talk about [1:18] today is how to uh become a 10x engineer and how to increase the productivity in our [1:25] day-to-day life from the uh platform engineering point of view. Uh so yeah I'll go over some of the [1:32] MCP servers which will make your life easy if you're building uh Asians or uh if you are [1:38] building any uh tools any workflow automations uh uh which will help either for internal use case or [1:46] for external use uh and how to uh effectively uh make sure that our uh day-to-day life is easier. [1:55] So uh I know everyone must be aware of what Kubernetes is and uh what uh it's like a uh [2:03] go-to platform for building any applications and uh there's no so let me take a step back [2:10] and talk about what MCB server is. I I believe everybody here is aware of MCB by [2:15] now since this is a day two for AI conference and there have been a lot of talks about MCB. [2:22] So MCP is nothing but a tools calls where uh your LLM model or your harness which [2:29] you're using for your uh agentic workflow can make a tools call and execute certain actions [2:35] which are external right so whether you're using cloud or any other uh agentic workflow [2:41] you can integrate MCP with it and make that tool call and execute certain actions. So there is no [2:48] uh guessing framework involved in this and you have certain tools calls which your agent is [2:57] uh given access to and it can only execute those tools calls without doing any guess work. Right? [3:04] So the first one I'll talk about is the Kubernetes MCP. There's no official Kubernetes MCP server [3:09] from uh CNCF project but there are many online uh you know uh GitHub repos which you can take [3:17] a look at which does provide the crude operations for the Kubernetes and the one which I have used [3:23] uh is Kubernetes. I have put a link over here. Uh I'll talk about some of the real world use cases [3:31] how it makes life easy right? uh a lot of time I get pinged about why my pod is stuck pending, [3:38] there are not enough resources available in the cluster or my node is exhausted. So having uh uh [3:47] the Kubernetes MCP access uh makes this debugging very easy, right? [3:52] developers who are working on deploying any applications can debug their own things by [3:57] making the tools calls and finding out why there's a CPU limit or if it is missing any tolerations [4:04] or there's uh you know the node has reached the maximum port it can support things like that [4:10] another uh very common issue which I notice is when you're working in a large organization [4:17] you are uh deploying microservices and you are expecting certain ports or traffic uh via ingress [4:25] and the routes are not set up correctly, right? You are you have multiple endpoints and uh [4:32] this this takes a while to debug why particular endpoint is uh not accepting [4:38] traffic or if there's any port which is blocked or if there's any other firewall issues which is [4:44] causing uh the data not flow. So uh solving the ingress issue is another uh very good example [4:52] which uh Kubernetes MCP server can make use of. And then uh [4:57] again there this is very common scenario where sometimes your secrets change and then the [5:04] the pods or the application which is running needs a restart. Right? So there are some external [5:11] there are some applications which solves this problem but having access to a roll out with a [5:17] with a secret change is again another uh good useful tools which I have found. [5:23] Uh moving on let's move on to the observability part. We use graphana because it is open source [5:29] we love graphana. uh having access to graphana MCP server makes things uh much much easier uh because [5:39] uh building dashboard is very it's it is uh the MCP server is very good at building dashboards and [5:46] if you are in the graphana ecosystem having MCP servers with Loki or uh Prometheus [5:54] can gives you access to query your data using a natural language right and it brings the context [6:00] text right into your LLM uh tool and it helps you uh diagnose any metrics issues or [6:08] if you're debugging any alert uh or if you're building any agent. I will talk about agents [6:14] little but little later. Uh but yeah having Grafana MCB access is really useful and then [6:22] it it does have uh Grafana has released official MCPA servers which does support uh Loki as well as [6:29] uh the tempo tracing. So if you are using any of those tools and platform engineering uh [6:35] can make the use of these MCP servers to debug things. [6:44] uh I'll talk about some of the real world use cases which I have personally used. Uh [6:49] so when you use graphana right you have this tons of graphana ecosystem related plugins. [6:56] Every time that graphana upgrades happens plugins also upgrades automatically. [7:02] I was I was working on upgrading graphana to graphfana third uh [7:06] 12 and I realized one of the plug-in which is a business user plug-in had has bunch of [7:14] uh uh uh you know the JS code in it and it just stopped working because the API has changed [7:21] and we wanted to debug that information. So uh it took took us a while to figure that out that [7:28] this is what is happening and I need to use a different uh API because that business form [7:33] API which I was using the endpoints have changed and with the help of LLM it was [7:39] easier to diagnose or come to the conclusion that this is the root cause. [7:44] Another uh uh problem which which I have solved using graphana MCP is when you deploy Prometheus [7:50] right you keep things at default you ingest everything hoping you will use those matrix [7:55] someday but Prometheus is uh keeps you know running into OOM error and if you are uh [8:03] if you have seen this issue before you know that uh this issue can be solved using [8:09] dropping unused metrics right so how do you find out which metrics are not used Then [8:14] with the MCP server, you can find out all the graphs which you have built in Graphana [8:19] and then write an override in Prometheus to drop those metrics and then whoever wants to [8:25] uh make use of those metrics can then again reintroduce them and uh start working on them. [8:33] uh sometimes the metrics names gets changed when we are upgrading things from v_sub_1 to v_sub3 [8:39] and if that metric is being used as a yeah uh variable then your graphana panels [8:48] usually will show no data right this is again another common scenario so debugging this uh [8:54] is uh llm can help you uh do that uh [sighs] we use terapform for all our part. Uh so having [9:05] Terapform MCB server access is again uh useful. Uh it helps it removes the guesswork which provider [9:11] you have to use which uh resource block which data blocks you have to use if you have. So the way I [9:18] way I would set up your ISC directory is at every level at every module right I will put like a [9:24] uh MD file agent MD file so my agent knows how my structure needs to be uh for my codebase and then [9:32] my uh MC when when I'm writing a code my MCP server already knows that it needs to go and read [9:38] this block uh this provider for any information it for any of the code it it is building [9:46] again the the the the use cases are provider aware A and uh this one is again I have [9:54] uh recently worked on auto autonomous drift detection agent so one of the challenge with [10:00] terapform is we don't start using terapform right when you're building a new company [10:05] right you first build a product and then you adopt terapform kubernetes all these tools [10:10] but by that time lot of people have access to uh your cloud accounts And [10:15] there's always going to be a drift, right? So if you and if you are fi following IC standards like [10:20] creating a modge request and if you see a drift those mod request stays open for a long time [10:27] because nobody knows that if you apply them what things will break right opening certain ports or [10:33] so having this if detection agent uh solves an issue where it you can have it run every day [10:42] and you know generate a plan and according to your uh plan it can suggest you what what are [10:48] the things which needs to be imported what are the things it needs to apply [10:54] uh the next uh s MCP server which we we will talk about is Argo CD which is our CD part continuous [11:02] deployment part right uh Argo CD uh application continuously syncs your code into your live uh [11:11] servers and it deploys what is there in the get uh in Argo CD there there are scenarios where [11:20] application will stay unhealthy for XYZ reason uh maybe you are changing the storage class which is [11:26] immutable or maybe uh things like uh you have deployed a image tag which is no longer exist [11:34] right and having access to aroc server can bring that context into your LLM and helps you solve [11:44] those issues effectively uh some of the live examples again debugging unhealthy apps. uh [11:54] why is particular if there's a CRD missing or certain uh uh [12:00] app is expecting some containers or some container clash crash back looping [12:06] uh if there are any dev over tags overrides uh like image override when you are in development [12:12] and when you move to production those tag doesn't exist and then the app is unhealthy. uh [12:19] it can also help you if you're deployed on a multi cloud structure right sometime OCI AWS GCP [12:26] and you want to have sync at the same time uh multi cloud multi-reion then this Argo CD will uh [12:33] MCP server will give you that advantage of syncing things or filtering out applications effectively [12:41] I'll talk about pagerd MCP server this is the I would say the most important or the the most used [12:47] uh MC GCP server uh because uh this is basically is an entry point to agents which you would build [12:56] in this area right uh pad duty uh is a tool which generates alerts and then [13:03] somebody acknowledges resolves the issue and then helps you uh uh manage who is on call uh [13:14] I'll talk about the agent which we have built is handoff agent right What happens is uh [13:21] you have a bunch of teams, right? You have a front- end team, you have a backend team, [13:25] you have a billing team, you have a platform team. Everybody everyone from one of these team [13:29] will be all always on call, right? And when you get an issue uh at uh 1:00 a.m. uh you [13:36] you have the on call person who answers to that issue. And then when the shift changes you want to [13:43] create a document so the next person who on boards knows exactly where things are. So this handoff [13:50] was getting very tedious. We were explaining uh a lot of things to the next person and we decided [13:56] to build an handoff agent uh which is basically a workflow which runs uh during the handoff time and [14:03] it pulls in the data from Slack and it creates like a notion page and the person who comes next [14:10] knows exactly where things are and where it needs where he needs to take off from right [14:16] uh postmortem generation again another tedious task which we all hate. to do but we have to do [14:23] it [laughter] whenever we have we say an issue we have to generate a postmortm right like who was on [14:28] the call and when somebody acknowledged that alert and how many how many you know uh if there's any [14:36] revenue loss you have to put all those details in that document so generating a postmortm using [14:44] uh pagodmc streamlines like the entire timeline and uh if there are any external links [14:51] and uh makes the makes the job of on call uh engineer easier. Again, conversational on call [15:00] assist like if you if you are on call and if you don't know who to who to reach out to, right? [15:05] Uh from the billing team, it can pull those details into the context and uh [15:11] uh let you do that. GitLab MCV server, we are on the GitLab side of the world. A lot of users use [15:17] GitHub actions but again GitLab's side is again the same thing same as GitHub actions. [15:24] Uh lets you create update uh your uh PRs and then uh you can see the job logs if if there [15:31] are any CI/CD job which are which have failed and you can pull in the the data into your LLM and [15:38] let you diagnose the things. uh what I have observed it with the with the help of cloud [15:45] code the things are much cleaner right sometimes we get a PR which is big enough that it would take [15:51] my entire sprints worth of time right so I don't even want to look at it sometimes so but having [15:57] a good description and a well commented code makes things uh better for a reviewer to look at [16:04] and then uh yeah some of you may think like why would you use G uh like uh MCP server when [16:13] there is a GLAB utility already present right which is a CLI which does the same thing but [16:19] uh the the MCP server is AI native and it the response is much simpler for AI uh to uh consume. [16:29] The next one which we will talk about is NAN. NAN is a server where [16:34] uh which allows you to build uh automation workflows. uh if you are building any [16:39] uh uh automations for your other part of your companies like product uh data analytics uh [16:47] having N10 server MCP server with the with the LLM can help you build do build them faster [16:55] uh yeah we have this use case where we want to whenever so we are a streaming company [17:01] every time we we have like a big event coming right like let's say Yellowstone [17:06] finale We want to make sure that our servers are up to date and we we usually run like a load [17:12] testing to make sure everything is up to date or a thanksgiving sale for a e-commerce uh website. So [17:19] uh scaling that cluster using n workflow uh is so create a workflow using net and then uh uh [17:27] trigger that workflow right that's what this use case is about and again other parts of the company [17:34] like a marketing or data team can also make use of n workp server to generate the workloads [17:42] which they need. At the end I will talk about fast MCP. So if your devs are building agents [17:50] and you want to expose your production DB access which is very sensitive but you want to securely [17:57] expose access to your production DB to your devs uh you have to build your own MCP servers right [18:04] and you have to limit how much response it it can generate. So uh I have just put like a small [18:13] uh code here which queries the DB and then it filters out filters out any PII data which [18:21] you may want to do uh so that agent does not know uh about your actual uh user information. So you [18:27] can build your own MCP server and fast MCP which is a Python based framework makes it uh seamless. [18:36] Uh yeah, open API. Yep. Let me talk any tools which are in the market which you [18:42] must be making use of if it has an open API uh implementation can be converted into MCP. So uh [18:50] there is a lot of uh there are some ways which you will find online. I have put some of them here [18:56] which allows you to generate uh MCP server for the open API API standards and again production [19:03] RDS queries. Uh with that being said uh there is lot to unload here like are back [19:08] and you want to make sure that uh because some of these tools which I discussed here are uh very [19:14] sensitive and only few like a platform engineer or engineer with the higher responsibility [19:19] should have access to. So having uh a well uh defined arbback control is very important. [19:26] You also want to take a look at how much response you're sending because uh uh you you don't want to [19:33] send like a huge data to your LLM and finish up your which eats up your context space. So uh [19:39] pagionation is also again equally important. Uh yeah with that being said uh uh don't be afraid [19:47] of the agents they are here to help us. And if you have any more questions you can reach out to me. [19:55] [applause]