Netflix Engineer Vijayanathan Automates Outage Prevention With Predictive AI

Netflix工程师Vijayanathan利用预测性AI实现故障预防自动化 基于过往故障解决方案训练模型，推动系统从被动排查转向实时自愈基础设施。 Prasanna Vijayanathan概述了大规模可观测性的演进，分享了他在LinkedIn和Netflix工作中的见解。他探讨了各组织如何从被动、手动的系统优化，转向利用预测性AI模型分析历史故障模式以识别隐性漏洞。通过向AI输入根本原因分析与解决方案数据，系统能够实现持续自动调优，提出诸如垃圾回收调整等补救建议，并支持自愈基础设施。最终，这一转变使可观测性从充斥嘈杂告警与孤立仪表板的状态，转变为主动的故障预防和自动化根本原因分析。 Netflix 软件工程师 Prasanna Vijayanathan 结合近 15 年行业经验，主导涵盖移动端、电视端和 Web 端的端到端可观测性建设，致力于为全球用户提供始终如一的顺畅体验。 (0:04) 传统优化与“左移”模拟难以覆盖复杂的生产环境极端案例，而 AI 可深度分析历史根因与故障前置模式，在故障发生前动态预测系统瓶颈和延迟峰值。 (20:31) 通过学习历史故障解决闭环，AI 正迈向直接给出 GC 参数微调或内存泄漏修复建议，并与自动化调优联动构建实时自愈基础设施。 (23:07) AI 驱动的现代可观测性实现了核心范式转变，全面摆脱告警疲劳与耗时的人工排查，迈向自动化根因分析与事前预防。 (24:08)

Netflix software engineer Prasanna Vijayanathan contends that enterprise observability must shift from reactive troubleshooting to predictive, AI-driven architectures capable of auto-tuning and pre-empting system failures. Limits of Manual Optimization and Shift-Left Testing (20:31) Historically, infrastructure performance optimization occurred reactively during periodic, quarter- or year-long initiatives prompted by urgent business requirements. Because optimizations were treated as afterthoughts, hidden bottlenecks routinely lingered until live service failures forced engineering teams to respond. While organizations adopted 'shift left' strategies—incorporating extensive pre-production testing and digital twin simulations—these environments proved incapable of anticipating every production edge-case scenario before real-world traffic revealed them. Predictive Outage Prevention from Historical Patterns (20:31) Vijayanathan argues that machine learning models alter this dynamic by turning comprehensive root-cause analysis records into predictive tools. By ingesting past incident histories along with the precise telemetry patterns that preceded outages, AI systems can identify latent vulnerabilities and forecast upcoming latency spikes before a breakdown occurs. Rather than relying solely on static test suites, developers can leverage these models to uncover runtime bottlenecks and dynamically generate scenario-specific test cases. Automated Tuning Toward Self-Healing Infrastructure (23:07) Beyond risk identification, AI systems can ingest the historical sequence connecting a fault pattern, the resulting incident, and the engineering remedy applied to resolve it. Vijayanathan forecasts that continuous training on these resolution cycles will soon allow AI models to directly instruct engineers on precise technical fixes, such as modifying microservice garbage collection (GC) parameters or isolating memory leaks. Connecting predictive models to automated tuning controls creates a continuous feedback loop, moving systems toward self-healing infrastructure that adjusts operating parameters in real time before users experience degradation. The Structural Paradigm Shift in Observability (24:08) The broader evolution of observability centers on replacing fragmented, high-noise operational workflows with unified intelligence. Engineering organizations are shifting away from noisy alert thresholds toward actionable signals, and from slow, manual triaging to automated root cause analysis. This transition replaces isolated monitoring dashboards and retrospective post-mortem write-ups with integrated platforms designed to preempt incidents and preserve a consistent end-user experience across client platforms. Vijayanathan frames this AI-driven evolution as the foundation for guaranteeing flawless application experiences across diverse client devices, though real-time self-healing remains dependent on successfully bridging AI recommendations with automated infrastructure adjustments.

Netflix 软件工程师（L5）Prasanna Vijayanathan 指出，将人工智能（AI）引入端到端可观测性体系，正在使系统运维从被动的事故后抢修与人工排查，转向基于历史数据的预测性预防、自动化根因分析（RCA）以及具备自愈能力的基础设施。 传统系统优化与“左移”测试的局限 (20:31) Vijayanathan 拥有近 15 年的性能、规模化扩展与用户体验从业经验，先后在 LinkedIn 与 Netflix 领导跨移动端、电视端及网页端的端到端可观测性工作，同时担任 IEEE 高级会员并主导内嵌安全设计的生成式 AI 标准委员会工作。基于其实践经验，他指出传统的系统优化与故障预防往往属于事后补救，仅在出现紧迫业务需求时才被提上日程。企业通常需要启动耗时一个季度甚至一年的大型专项工程来提升服务速度与效率，导致潜在瓶颈长期潜伏，直至演变为实际危机才迫使工程师介入处理。 为改变这种被动局面，业界曾尝试通过“左移”（shifting left）策略来提前应对，即在进入生产环境前进行高强度测试，并构建数字孪生（digital twins）等仿真环境。然而 Vijayanathan 强调，在实际工程中，仅仅依靠预生产测试与模拟，几乎不可能穷尽并预测所有仅在真实生产环境中才会触发的极端边缘场景（edge-case scenarios）。 基于历史数据的AI预测性预防机制 (20:31) AI 技术通过将历史事故数据转化为预测模型，从根本上改变了上述被动范式。Vijayanathan 解释称，工程团队可以将根因分析（RCA）中的全面数据投喂给 AI 系统，特别是历次宕机事故发生前的完整前兆模式（lead-up patterns）。通过模式识别，AI 系统能够在隐藏的系统脆弱性演变为线上故障之前将其精准识别。 在这一预测性分析框架下，开发人员无需等到系统崩溃后再做反应，而是可以主动利用 AI 模型动态预测即将出现的延迟峰值（latency spikes），或定位应用程序中的性能瓶颈。此外，AI 模型还能基于历史故障前兆分析，针对性地推荐全新的测试场景用例，从而弥补传统模拟测试无法覆盖未知生产场景的短板。 持续自动调优与自愈基础设施 (23:07) 除了识别潜在风险，AI 还能深入分析历史事故的最终解决方案，将“初始前兆模式—引发的故障—实际采用的修复措施”串联为完整的训练闭环。随着系统在这些解决周期数据上持续训练，其功能将从单纯提供修复建议演进为自动指导系统调优。Vijayanathan 预测，鉴于当前 AI 技术的快速发展速度，模型很快就能直接向工程师提供具体的技术修复建议，例如调整垃圾回收（GC）参数，或指出微服务中的内存泄漏（memory leaks）问题。 当自动化调优机制与 AI 驱动的洞察相连接时，系统便形成了从模式识别、预测分析、测试用例生成到自动调优的持续反馈闭环，进而开启“自愈系统”（self-healing systems）的愿景。在此架构下，基础设施能够实时自动调整自身参数，在用户感知到服务降级之前化解问题，从而实现端到端效率与全球用户体验的最优化。 AI时代可观测性体系的四大演进 (24:08) Vijayanathan 总结认为，现代 AI 驱动的可观测性范式与传统监控方式相比，已经在四个核心维度上实现了演进。首先是从“告警”转向“洞察”，摆脱过去繁杂且容易引发疲劳的告警噪音，代之以具备可执行性的智能信息；其次是从“人工分诊（triaging）”转向“自动化根因分析（Automated RCA）”，消除了缓慢的手动故障排查流程。 同时，监控工具的形态也从“孤立工具”转向“集成系统”，用统一的智能化平台取代了过去碎片化、各自为政的监控仪表盘；而在运维策略上，则从“事后复盘（post-mortems）”转向“主动预防”，不再依赖静态的事故后分析报告，而是借助集成化智能平台主动阻止宕机事故的发生。 Vijayanathan 总结强调，从被动监控向 AI 驱动的预测性预防与自愈架构转型，其核心目标在于兑现他所秉持的工程信念——确保全球各地的每一位用户在每一次使用跨平台应用时，都能获得流畅、可靠且无瑕的体验。同时，从演讲披露的内容来看，虽然基于历史数据训练的自愈闭环架构已经明确，但模型对调整 GC 参数和定位微服务内存泄漏等具体技术修复的直接指导，目前仍被演讲者界定为伴随 AI 快速发展而即将实现的演进方向。

## Background 本视频由技术峰会频道 Ai4 于 2025 年 8 月 29 日发布在 YouTube 平台。视频内容为 Netflix 软件工程师 Prasanna Vijayanathan 带来的主题演讲，主要探讨 Netflix 如何利用数据与 AI 驱动的洞察构建端到端可观测性系统，以支撑其全球化产品的高性能与超大规模扩展。 ## TL;DR 演讲深入剖析了传统可观测性在微服务架构下因数据孤岛与缺乏上下文（如“透过钥匙孔看《怪奇物语》”）而导致的告警疲劳和排障困境，指出盲目将原始数据塞给大语言模型并不可行。Netflix 提出通过知识图谱注入因果语义与拓扑关系，并结合由通用协调 Agent 与垂直领域 Agent 构成的专家混合架构（Agentic AI），实现自然语言精准排障与自动根因分析（Automated RCA）。该体系进一步将事故处置经验反哺于测试用例生成与动态参数优化，推动运维范式从“事后复盘报告”彻底迈向“事前主动预防”与系统自愈。 ## Atmosphere & style 整场演讲呈现出典型硅谷大厂技术专家的沉稳、务实与清晰逻辑，开篇用通俗形象的影视剧比喻精准切中分布式系统排错的真实痛点。主讲人既直言不讳地指出当前业界盲目滥用 LLM 和 RAG 导致幻觉频出的误区，又层层拆解从图建模到多 Agent 协同落地的工程硬核细节。整体节奏紧凑流畅、极具说服力，展现了顶级流媒体巨头将前沿 AI 技术真正融入严苛生产环境的系统性思考与演进底气。 ## Key takeaways - [01:50](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=110s) 端到端可观测性（End-to-End Observability）对于保障全球用户体验至关重要，缺乏全链路上下文就如同透过钥匙孔看《怪奇物语》，无法看清系统运行的全貌与本质。 - [01:50](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=110s) 系统性能与延迟直接影响业务核心指标：亚马逊研究表明延迟增加 100ms 销售额下降 1%，谷歌发现 0.5s 延迟会导致流量锐减 20%，Netflix 自身的自动播放体验也显著影响用户观看时长。 - [06:01](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=361s) 传统可观测性面临严重的日志、指标与独立仪表盘孤岛问题；在微服务架构下，单纯堆砌 CSV 或原始数据给 LLM 会导致严重的幻觉与系统崩溃。 - [08:39](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=519s) 引入知识图谱（Knowledge Graphs）为分布式实体注入因果与行为语义，能够让 AI 模型真正理解用户操作与后端微服务之间的调用链逻辑。 - [12:20](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=740s) 将微调的领域模型、知识图谱与 LLM 协同结合，工程师可以直接使用自然语言进行故障排查，系统能够自动拆解目标并返回具体的因果诊断结论。 - [13:18](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=798s) 针对告警疲劳（Alert Fatigue）问题，通过关联告警生态与基于感知的智能分诊（Awareness-based Triaging），系统能自动定位根因服务并精准路由告警，避免无效干扰前端工程师。 - [16:34](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=994s) 在高敏感线上 On-call 场景中，简单的 RAG 助手容易产生幻觉；采用智能体 AI（Agentic AI）与专家混合（Mixture of Experts）架构能够更有效地协同排查并给出 Top 3 根因排序。 - [20:31](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=1231s) 借助 AI 学习事故前置特征与历史解决方案，系统能够提前预测延迟瓶颈并自动生成测试用例反哺测试环境，实现从被动应对向事前预防的转变。 - [20:31](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=1231s) 通过 AI 对 JVM GC 参数调优及内存泄漏预警等动态调整手段，现代可观测性正演进为具备自愈能力（Self-Healing）的主动防护系统。 - [24:18](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=1458s) Netflix 的新一代可观测性愿景是告别人工分诊、孤立看板和事后复盘报告，全面走向自动化根因分析、系统集成智能与故障事前主动预防。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=4s) 演讲者背景与端到端可观测性的演进 Netflix 软件工程师 Prasanna Vijayanathan 开启了关于“AI 时代产品端到端可观测性（End-to-End Observability）演进”的演讲。Prasanna 拥有近 15 年的软件工程经验，职业生涯始终聚焦于性能、系统扩展性以及用户体验。他坚信全球各地的每一位用户每次打开应用都理应获得出色的体验。目前他在 Netflix 负责领导跨移动端、电视端和网页端等全平台应用的可观测性提升项目；在此之前，他曾在 LinkedIn 负责类似的产品可观测性与性能工程。 除工程实践外，Prasanna 还是 IEEE 高级会员（IEEE Senior Member），在相关委员会主导制定内置安全机制的生成式 AI（GenAI by Design）标准与框架，同时在多家倡导利用科技与 AI 推动社会公益的非营利组织董事会任职。他指出，本次分享是他多年来在数据、机器学习、专家系统（早于现代 AI 概念成型前）以及近年最新 AI 技术实践中的经验提炼与深度总结。 --- ### [01:50](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=110s) 丢失上下文的代价：透过钥匙孔看《怪奇物语》 为了生动说明端到端视角的不可或缺，Prasanna 提出了一个形象的比喻：“想象你在看《怪奇物语》（Stranger Things），但你是透过一个钥匙孔在看。”透过狭小的孔洞，你可能只看到闪烁的圣诞彩灯和骑着自行车欢快离开的孩子，于是得出结论：“哦，这是一部温馨欢快的儿童剧。”然而，这种局部视角完全忽略了背后的整个“颠倒世界”（Upside Down）、夺心魔（Mind Flayer）以及所有人为何在流鼻血。Prasanna 强调：“Context is everything（上下文就是一切）。”如果不具备端到端的全链路视角，观察复杂系统就如同透过钥匙孔看电影，只能捕捉到时空切片，却完全丢失了整体上下文。 缺乏全链路可观测性，工程团队就无法真正理解用户。多项行业权威研究反复证明，用户体验对于消费级科技产品至关重要，是驱动用户留存与参与度（Engagement）的核心基石，而系统性能与可靠性则直接决定了体验的优劣： - **亚马逊（Amazon）研究**：系统延迟每增加 100 毫秒，就会导致销售额直接下滑 1%。 - **谷歌（Google）研究**：搜索与页面响应延迟增加约 0.5 秒，整体用户流量就会锐减约 20%。 - **Netflix 自身研究**：自动播放功能能够显著提升单次会话时长；若禁用该功能，用户的每日观看时长将出现大幅度下降。 - **工程师生产力瓶颈**：如果看不清系统幕后全貌，排查与修复故障就会变成一场噩梦，缺乏洞察工具会让工程团队彻底陷入被动。 --- ### [04:11](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=251s) 告别孤立仪表盘：AI 重塑可观测性四大支柱 在当下激烈的竞争环境中，用户对产品速度和流畅度的要求近乎苛刻，一旦体验不佳便会迅速流向竞品。因此，当故障发生时，团队必须以最快速度获知，甚至恨不得“昨天就已经把它修好了”。然而，传统的监控指标与可观测性系统高度割裂，仅仅堆砌孤立的仪表盘（Isolated Dashboards）已无法满足需求，现代系统亟需能够洞悉全局的智能化洞察。 幸运的是，现代 AI 技术提供了破局的关键。AI 具备三大核心优势：能够以极高速度在大规模数据中精准识别出人类容易遗漏的微弱模式；能够跨越孤立系统将分散的零碎信号串联起来；能够推动团队从“被动响应”转变为“主动预防”。基于此，Prasanna 梳理了构建端到端可观测性系统的四大支柱，并由此展开对第一大支柱——数据层面的深度剖析。 ```mermaid flowchart LR accTitle: 端到端可观测性四大支柱概览 subgraph Pillars["端到端可观测性四大支柱"] direction LR P1["1. 数据层 (Data)"] --> P2["2. 关联与建模"] P2 --> P3["3. 智能洞察与分析"] P3 --> P4["4. 主动治理与行动"] end ``` --- ### [06:01](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=361s) 第一大支柱：数据孤岛与微服务时代的断层危机 数据是系统的“基石与事实真相”（Ground Truth），涵盖产品健康指标与业务核心指标（True North Indicators）。然而在超大规模分布式架构下，现有的数据体系面临着巨大的孤岛挑战。回顾数据系统的演进：最基础的是日志（Logs），散落在代码各处的 error、debug、info 打印被统统倾倒到日志平台，但在海量信息中，“除非迫不得已，否则根本没有人会主动去看日志”；而指标（Metrics）则是被汇总并展示在精美仪表盘上的数字，存储在与日志完全隔离的集中追踪系统中。 这种孤立的数据体系带来了致命的割裂：日志、指标、仪表盘、异常捕获与堆栈跟踪各行其是，彼此缺乏关联。当一个页面由数千个微服务协同渲染时，一旦出现偶发问题且无法自行恢复，排错就会变成灾难。例如，当负责前端 App 的工程师发现 GraphQL 查询突然报错失败时，由于缺少全链路上下文，根本无从查起是后端哪个微服务环节出了差错。此时系统呈现的不是一部连贯的故事，而是一部杂乱无章的“故事选集（Anthology）”，工程师只能祈祷自己能碰巧猜中原因。 --- ### [08:39](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=519s) 超越简单图结构：利用知识图谱（Knowledge Graphs）注入语义上下文 针对数据孤岛问题，许多人盲目跟风，试图将所有 CSV 文件和原始数据库表直接倒进大语言模型（LLM），寄希望于 AI 自动搞定一切。Prasanna 直言这种做法完全行不通：“即使在中小规模下，系统也会瞬间崩溃并产生严重的幻觉（Hallucinations），输出毫无准确性可言，单纯依靠 LLM 绝对不是解法。”为了让模型能够有效消费数据，必须对数据进行结构化转换，而图（Graph）结构是极佳的基础表达形式。 在基础图模型中，实体以节点（Nodes）表示，依赖以边（Edges）表示，并可在其上附加延迟、错误率等属性。例如，用户节点连接到 App 节点，App 节点再连接到微服务 A 节点。然而，仅仅使用普通图结构依然不够：在普通图中，App 到服务 A 与 App 到服务 B 仅仅是两个带有不同属性的普通连线，模型并不知道这两个服务的业务含义与因果机制，缺少核心语义。 真正释放 AI 潜力的是**知识图谱（Knowledge Graphs）**。知识图谱为图中的实体赋予了丰富的上下文与真实交互关系。例如，它不仅记录节点连接，更定义了语义链条：“用户在 App 上滑动浏览 Feed 流页面”，该动作触发 App 向“服务 A 发送请求 1”，进而触发向“服务 B 发送请求 2”。正是这种包含因果与行为语义的上下文关系，才使得 AI 模型能够真正理解分布式系统的运行逻辑，为后续的智能异常归因奠定基础。 ```mermaid flowchart TD accTitle: 普通图与知识图谱在可观测性建模中的对比 subgraph BasicGraph["普通图结构 (仅有节点与连线，缺乏语义)"] direction TB U1["用户 (User)"] --- A1["应用 (App)"] A1 --- S1["服务 A (Service A)"] A1 --- S2["服务 B (Service B)"] end subgraph KG["知识图谱 (具备丰富上下文与行为因果关系)"] direction TB U2["用户 (User)"] -- "动作: 滑动浏览 Feed 流" --> A2["应用 (App)"] A2 -- "触发请求 1 (Request 1)" --> S3["服务 A (Service A)"] A2 -- "并发调用请求 2 (Request 2)" --> S4["服务 B (Service B)"] end ``` ### [12:20](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=740s) 知识图谱与 LLM 协同：实现自然语言排障与上下文推导 在构建端到端可观测性时，将实体、依赖关系和上下文完整沉淀在知识图谱（Knowledge Graph）中，能够让模型更精准地学习特定领域知识并做出准确判断。当专门训练的模型理解了上下文图谱的各组成部分，并与大语言模型（LLM）协同工作时，即可构建出强大的自然语言交互系统。 ```mermaid flowchart LR accTitle: 自然语言排障与知识图谱协同流程 subgraph Input ["用户交互"] Q["自然语言提问<br/>'Feed 页面延迟为何升高？'"] end subgraph Intelligence ["AI 与上下文图谱"] LLM["LLM 目标拆解"] KG["知识图谱上下文匹配<br/>(实体 / 依赖边)"] end subgraph Output ["根因定位"] Ans["精准结论输出<br/>'下游服务发生 Load Shedding'"] end Q --> LLM LLM --> KG KG --> Ans ``` 开发者可以直接用日常语言提问，例如：“为什么我的 Feed 页面延迟恶化了？”LLM 接收到该请求后，会将其拆解为具体的子目标，随后从知识图谱中提取关联的上下文节点与拓扑关系进行分析，最终直接给出具备因果关系的诊断结论，例如指出“这很可能是因为下游服务在同一时间触发了负载丢弃（Load Shedding）”。 ### [13:18](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=798s) 信号检测的困境与突破：告警疲劳与基于感知的智能分诊 信号检测（Detecting Signals）当前面临的核心挑战，根本原因在于底层数据系统的孤岛化。由于各微服务和应用的指标相互割裂，当某个指标出现退化时，工程师根本无法得知它与系统中其他微服务或组件有何关联。十年前业界普遍引入异常检测技术，能够自动捕捉到指标异常并触发告警，但这并没有彻底解决问题。 现实中开发者常常抱怨告警缺乏可操作性（"this alert is not actionable"），认为许多告警只是短暂抖动、放一会儿自己就会恢复，从而产生了严重的“告警疲劳”（Alert Fatigue）。海量嘈杂告警中可能只有一两条真正需要人工介入，这导致值班工程师（On-call）对监控系统丧失信任，甚至抱有“凌晨 2 点被叫醒也是误报、先拖延过去再说”的心态，进而埋下真实严重故障被忽视的隐患。 - **关联告警生态（Connected Alerts）**：AI 不仅持续演进异常检测算法，更赋予告警丰富的拓扑上下文，自动比对关联服务与应用切面的指标表现。 - **基于感知的智能分诊（Awareness-based Triaging）**：当前端应用触发延迟退化告警时，系统能自动识别出故障根源其实位于深层下游服务，从而直接将告警路由给下游服务的 On-call 工程师，避免前端团队被无效告警打扰。 ### [16:34](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=994s) 调试与根因分析：从 RAG 痛点到 Agentic AI 与专家混合架构 在传统调试模式下，数据孤岛导致了各自独立的仪表盘（Standalone Dashboards）。即便团队尝试将大量图表堆积在同一个看板上以便横向对比，排查故障依然如同“在大海里捞针”（finding a needle in a haystack）。特别是在 Netflix 这种超大规模分布式架构下，一个前端体验背后可能串联着成千上万个微服务，工程师排查时往往互相推诿（“问题不在我的应用，肯定在下游服务”），且极少有人会主动去翻阅下游两层之外的日志与指标。 ```mermaid flowchart TD accTitle: Agentic AI 专家混合调试架构 Coord["通用协调 Agent<br/>(Coordinator Agent)"] subgraph SpecializedAgents ["垂直领域专有 Agent"] GQL["GraphQL 专有 Agent<br/>(理解 Schema / 查询模式)"] FE["前端架构 Agent<br/>(感知发布版本 / 代码变更)"] Downstream["下游服务 Agent<br/>(分析微服务日志 / 依赖)"] end Coord --> GQL Coord --> FE Coord --> Downstream GQL --> RCA["自动根因分析 (Automated RCA)<br/>输出 Top 3 潜在根因排序"] FE --> RCA Downstream --> RCA ``` 许多团队尝试利用 LLM、Bot 与检索增强生成（RAG）搭建排障助手，但在高敏感度的线上 On-call 场景中，通识大模型的幻觉以及缺乏微调会导致排障耗时比以往更长。更为行之有效的方案是采用智能体 AI（Agentic AI）构建专家混合（Mixture of Experts）系统： - **垂直领域专用 Agent**：构建专注单一领域的轻量模型。例如专门理解 GraphQL 查询结构与服务的 Agent，或专门掌控前端版本发布记录与变更细节的 Agent。 - **全局协调 Agent（Coordinator Agent）**：负责任务编排、工具调用与跨 Agent 调度，执行复杂的排障任务链。 - **自动根因分析（Automated RCA）**：在发生指标退化时，多 Agent 协同筛查关联事件并进行相关性排序，自动给出 Top 3 最可能的潜在根因。 ### [20:31](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=1231s) 预防与自愈系统：从被动优化到基于 AI 的持续动态调优 传统的系统防护与性能优化往往是一种“事后补救”（afterthought）。通常是由业务需求驱动发起一个为期数个季度的大型专项优化战役，这意味着系统中始终潜伏着未被察觉的瓶颈。前几年业界提倡“测试左移”（Shift Left）与数字孪生（Digital Twin）仿真测试，试图在上线前穷举异常场景，但依然无法完全覆盖真实生产环境中的复杂边界情况。 结合自动根因分析沉淀的历史模式，AI 展现出了强大的前瞻性预防与自愈能力： - **瓶颈预测与测试场景扩充**：AI 善于学习导致事故的前置特征模式与事后规律，能预测服务潜在的延迟瓶颈，并将高危模式自动转化为新的测试用例与测试计划，反哺测试环境。 - **事件闭环与持续参数调优**：将每次事故的最终解决方案（Resolutions）反馈给模型进行持续训练。当系统出现异常苗头时，AI 能直接给出具体调优建议（如自动调整 JVM 的 GC 参数，或定位并预警下游服务的内存泄漏隐患）。 - **系统自愈（Self-Healing Systems）**：通过自动化的参数与资源持续调优，系统从被动告警演进为自主修复，最大化保障终端用户的端到端体验。 ### [24:18](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=1458s) 从事后排查走向主动预防：Netflix 可观测性的未来愿景 演讲在最后对 Netflix 新一代可观测性架构的演进方向进行了高度概括与总结。未来的运维与排障模式正在经历深刻的范式转变：团队正从依赖人工分诊（manual triaging）获取洞察，全面转向**自动化根因分析（Automated Root Cause Analysis）**；同时彻底告别过去零散孤立的仪表盘（dashboards）与工具链，构建起高度集成的智能化系统，能够主动且精准地直接告知工程师核心问题所在。 更加颠覆性的是，正如演讲者所言——“忘掉那些事后复盘报告吧（Forget your postmortem reports）”，Netflix 正在大步迈向**事前的主动预防（Proactive Prevention）**阶段，在异常真正造成破坏前就将其化解。这一整套基于 AI 与数据的端到端可观测性闭环，最终目标是让全球各地的每一个用户在每一次打开 App 时，都能享受到极致流畅且稳定的流媒体体验。演讲者 Prasanna Vijayanathan（Pras）在致谢中结束了本次分享，并欢迎大家在 LinkedIn 上保持交流连接。 ```mermaid flowchart LR accTitle: 可观测性演进路径 subgraph Traditional["传统运维模式"] A["人工分诊排查"] B["孤立的监控看板"] C["事后复盘报告"] end subgraph Modern["智能可观测性架构"] D["自动化根因分析"] E["集成化智能系统"] F["主动预防潜在故障"] end A --> D B --> E C --> F F --> G["全球极致 App 体验"] ``` - **自动化根因定位**：摆脱繁重的人工分析流程，由智能算法直接下钻并锁定根本原因。 - **集成化智能认知**：统一观测数据平台，系统具备自主认知能力，按需推送最关键的上下文信息。 - **故障事前预防**：核心重心从“事后编写复盘（Postmortem）”转变为“事前主动扼杀隐患”。 ## Notable quotes > [01:50](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=110s) Context is everything — Prasanna Vijayanathan > [06:01](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=361s) unless you're forced to, nobody looks at logs — Prasanna Vijayanathan > [08:39](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=519s) even at a small scale, it will quickly fall apart and you will get a ton of hallucinations, highly inaccurate results, and LLMs alone is definitely not the answer. — Prasanna Vijayanathan > [13:18](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=798s) this alert is not actionable — Prasanna Vijayanathan > [16:34](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=994s) finding a needle in a haystack — Prasanna Vijayanathan > [16:34](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=994s) It's not in my application, it's definitely the downstream service. — Prasanna Vijayanathan > [24:18](https://www.youtube.com/watch?v=Ncpz5HFLwGI&t=1458s) Forget your postmortem reports — Prasanna Vijayanathan ## Mentioned Prasanna Vijayanathan, Netflix, Ai4, LinkedIn, IEEE, Stranger Things, Amazon, Google, GraphQL, Knowledge Graphs, LLM, RAG, Agentic AI, Digital Twin ## Video description Prasanna Vijayanathan, Software Engineer, L5, Netflix Abstract Learn how we use data and AI driven insights to enable end-to-end observability for building cutting edge products that scale across the world. [Source](https://www.youtube.com/watch?v=Ncpz5HFLwGI)

[1:50] Prasanna Vijayanathan: Context is everything [6:01] Prasanna Vijayanathan: unless you're forced to, nobody looks at logs [8:39] Prasanna Vijayanathan: even at a small scale, it will quickly fall apart and you will get a ton of hallucinations, highly inaccurate results, and LLMs alone is definitely not the answer. [13:18] Prasanna Vijayanathan: this alert is not actionable [16:34] Prasanna Vijayanathan: finding a needle in a haystack [16:34] Prasanna Vijayanathan: It's not in my application, it's definitely the downstream service. [24:18] Prasanna Vijayanathan: Forget your postmortem reports

[0:04] Prasanna Vijayanathan: everyone, everywhere in the world, deserves a flawless app experience every single time.

[0:04] Hello everyone, my name is Prasan [0:06] Vijinatan. [0:07] Um, and I'm here to talk to you about [0:10] how end to end observability on your [0:14] products uh are going to evolve in the [0:17] age of AI. Before we get started, a [0:21] little bit about me. Um, I'm a software [0:24] engineer been one for almost 15 years [0:26] now. uh and all that time I've focused [0:29] on performance, scale and user [0:31] experience. [0:33] I believe that everyone all over the [0:35] world deserve a good app experience [0:37] every time. I currently lead initiatives [0:40] at Netflix that focus on improving the [0:44] observability into our apps across our [0:47] various platforms, mobile, TV, web, and [0:50] so on. And prior to that I was at [0:53] LinkedIn doing something similar on [0:55] LinkedIn's uh product offerings. [0:58] I'm an IT senior member. I lead [1:01] standards and committees that uh focus [1:04] on improving the frameworks and creating [1:08] standards for building Genai solutions [1:12] with safety embedded by design. Um I [1:15] also work with other nonprofit [1:17] organizations. I'm on the board of a few [1:20] of them that focus on using tech and AI [1:23] for social good. [1:25] So my talk today uh is [1:28] from a summary of all my experiences and [1:31] learnings in this space for many years [1:34] uh using data AI before it was called AI [1:38] machine learning and expert systems. So [1:40] I want to talk through through the [1:42] evolution of all that and how things [1:44] have changed uh in the more recent [1:47] years. [1:50] So, imagine you're watching Stranger [1:52] Things. Now, imagine you're watching [1:55] Stranger Things through a keyhole. [1:58] You spot Christmas lights blinking and [2:00] you see kids riding away on bikes and [2:03] you probably think, "Uh, this is a happy [2:05] kids drama." But what you missed is the [2:08] entire upside down mind flare and why [2:12] everyone's nose is bleeding. [2:16] Context is everything. [2:18] So trying to understand your system [2:20] without an end toend lens is like seeing [2:23] a movie through a keyhole. You just get [2:26] a snapshot in time and space but you [2:28] really don't understand [2:31] any context beyond that. So how far do [2:34] you think you can understand your users [2:36] with that kind of a system? [2:39] There have been numerous studies time [2:41] and again that show why user experience [2:44] is so important uh for any tech product [2:48] consumerf facing uh solution that we [2:51] build. It is supreme to engagement which [2:54] is a fundamental driver for many [2:55] software businesses. [2:58] And then you all know what impacts that [3:01] user experience beyond your business [3:03] value is the reliability and performance [3:06] of your uh software. Again, [3:10] numerous studies that have shown Amazon [3:12] did one that showed every 100 [3:14] milliseconds latency led to a 1% drop in [3:17] sales. Google released a study a while [3:19] ago that said I think around.5 seconds [3:23] uh in delay reduced user traffic by [3:26] about 20%. And even Netflix did a study [3:29] a while ago that was that showed that [3:32] having an autoplay capability leads to [3:34] higher session lengths. disabling it [3:37] would reduce the daily watch time [3:39] significantly. All those studies have [3:41] been pointing to how important user [3:44] experience and reliability and [3:46] performance of your apps are. But beyond [3:48] that, there is also this aspect of [3:50] productivity for the people who actually [3:53] build those solutions. You can't find [3:55] and fix problems when you don't have [3:57] enough understanding of what is going on [4:00] behind the scenes. without the right [4:02] tooling and insights, it would be a [4:04] nightmare, right, for anyone to go and [4:06] debug their system to figure out what's [4:08] going on. [4:11] So with these challenges for just [4:14] observability, [4:15] I want to summarize our users want [4:18] faster, smoother, and better experience [4:20] all the time. If not, we lose to [4:23] competition. [4:24] So when there is an issue, you want to [4:26] know about it as soon as possible. And [4:28] we want it fixed yesterday. [4:32] But when your metrics and your [4:34] observability systems are so siloed, you [4:36] don't have the ability to understand [4:38] them entirely. [4:40] And so what we want really are not [4:42] isolated dashboards anymore, but we want [4:45] modern systems that can build [4:47] intelligent insights into understanding [4:50] your systems better. [4:53] But fortunately, [4:55] we have AI now. [4:57] So what we'll do today is we'll walk [4:59] through the different pillars of [5:00] observability and see how AI has started [5:05] impacting them and can impact how we [5:08] observe our systems and build software [5:10] for just better user experience. [5:13] Let's think about what AI is good at. [5:16] Can identify patterns that humans might [5:19] miss and it can do that at scale and [5:22] very quickly. It can connect desperate [5:25] signals across your system and it can [5:28] enable proactive action. You don't have [5:30] to be reactive. [5:33] So in my experience, there are four [5:36] pillars of an end to-end observability [5:38] solution. [5:40] Uh this shouldn't surprise most of you. [5:44] What we'll do is we'll go through these [5:45] pillars one by one and look at what are [5:48] the current challenges that most large [5:50] scale systems would face and then we'll [5:52] bring it all together and see how AI is [5:55] unlocking values and potentials uh in [5:58] each of these pillars [6:01] starting with data the most [6:04] fundamentally of course so data here is [6:08] the ground truth all the metrics that [6:10] you define for your products health uh [6:13] the true north indicators of your uh [6:16] business value all of that those data [6:18] those those are the ground truth that [6:20] really happen and you collect them [6:22] there's the information that gives you [6:23] the understanding of your system [6:27] but then there are bunch of challenges [6:29] that we face today at the scale at which [6:31] we operate like think about um how our [6:36] data systems have evolved over the years [6:40] um starting with logs and metrics. If [6:45] you know, you know. Logs are just that. [6:48] All the errors, debug, info, prints that [6:51] you add in your code, they all get [6:53] dumped somewhere, you know, sent to [6:55] logging system. But who's really looking [6:57] at them unless you absolutely have to? [7:00] No one is going to go and look at those, [7:01] right? That's how logging systems work [7:03] today. Think about this at the scale of [7:05] thousands of microservices uh working to [7:08] compose one single feed page on your [7:11] application. [7:13] And then comes metrics. Metrics are the [7:17] the cool numbers that make it to your [7:19] dashboards, right? Uh that you collect, [7:22] send it to a central tracking system [7:24] that is completely different from your [7:27] logging systems. And now you have siloed [7:31] data systems. You have logs, you have [7:33] metrics, you have dashboards all in [7:36] their own ways and forms. You have [7:38] things that collect exceptions and stack [7:40] traces. But what is connecting all of [7:42] them? I'm sure you've all run into the [7:45] this problem at scale when you're [7:48] debugging something. This is going to be [7:50] a nightmare if if the problem just [7:53] either goes away on its own or it's [7:54] simple enough for you to quickly fix. [7:57] It's almost never getting fixed. [8:00] And these siloed data systems create [8:04] these inherent gaps. [8:06] Imagine an engineer who works on the [8:09] front end app suddenly needs to figure [8:11] out why GraphQL query is failing because [8:15] something is happening in the back end. [8:17] Where would an engineer go to go figure [8:19] what's happening today? [8:22] This these are the gaps that we uh we [8:25] have today in our systems that [8:28] uh that doesn't really present you an [8:30] end to-end picture anymore. There is no [8:32] story. You probably have an anthology [8:34] and you can only cross your fingers and [8:36] hope it's good. [8:39] So let's talk about what AI can unlock [8:41] here. Remember we said AI is really good [8:44] at understanding patterns at scale. [8:47] So if people have jumped on the cool [8:51] geni LLM bandwagon recently, you [8:54] probably think just throw in all the CSV [8:57] files uh all the tables you have into [8:59] these systems and AI will figure it out. [9:03] But if you really know uh you know [9:05] that's not going to work. [9:07] um [9:09] you want to structure your data. If not, [9:12] even at a not even at a large scale, at [9:16] a medium scale, things are going to hit [9:18] the roof. Um you're going to run into [9:20] hallucinations. The the information that [9:23] you're going to get from any model is [9:25] not going to be accurate enough. So no, [9:27] LLMs are not the answer. [9:30] uh what you want to build is you want to [9:32] make your data uh transformed in a way [9:36] that is consumable for models and what [9:40] really works well for models uh graphs. [9:43] U you have nodes, you have uh entities, [9:47] you have edges and you can add [9:49] attributes to them. [9:51] So think about something like I have a [9:54] user who's using my app that becomes an [9:59] entry in your graph. You have the user [10:02] as an entity or a node. You have an app [10:04] as a node and you have an edge [10:05] connecting them. Right? And now you can [10:08] add more things that app talks to say a [10:11] service A. So that also becomes a node [10:13] and there's a connecting edge [10:16] and you can add more attributes on each [10:18] of these nodes and edges. Say you want [10:21] to capture latencies, you want to [10:22] capture errors or any other information [10:24] that would be useful for you. [10:27] What this does, it presents connections [10:30] in your system for models to consume [10:34] easily. [10:37] But this is not good enough yet. [10:40] Beyond uh just graph- based data [10:43] representations, knowledge graphs have [10:47] have been [10:48] very useful in unlocking that potential [10:51] because it provides context and [10:53] relationships between all our different [10:56] entities in our graph. [10:58] For example, let's go back to our [11:00] previous scenario, right? Let's say [11:02] there is another service B here that [11:05] your app talks to [11:08] to a model. If you to understand what [11:10] this means in a simple graph-based [11:12] system, app to service A, app to service [11:16] B are really no different. It doesn't [11:18] know that these two are different [11:19] things. They're just two different names [11:22] with some different attributes. [11:24] If there's another connection from [11:25] service A to service B, that's all it [11:28] knows. It has no context [11:31] about what app does, what service A [11:33] does, what service B does, or what the [11:35] user does. They're all just nodes and [11:38] edges. [11:39] But with knowledge graphs, [11:42] you create that context. You say here is [11:45] your user who uses [11:48] app your application. Better yet, you [11:51] can add more context. You can say here's [11:54] user scrolls your feed page on this app. [12:00] And then that causes app to send request [12:04] one to service Z. [12:07] Maybe it causes request two to go to [12:10] service B. [12:12] And this is the context and relationship [12:14] that models are getting really smart at [12:16] understanding. So if you have a [12:18] knowledge graph that presents your [12:20] entities and edges along with the [12:22] context, it makes it very easy for [12:26] models to be more accurate and train on [12:28] specific things. [12:31] So knowledge graphs go for context and [12:34] relationships and and with that you [12:38] build your models which understand [12:41] different parts of your contextual graph [12:45] and then use them in tandem with LLMs. [12:49] Now you can have natural language [12:51] systems where your users can go and ask [12:53] hey why is my the latency of my feed [12:56] page regressing? The LLM would be able [12:59] to break that down into concrete goals [13:02] and then it'll pick the right aspects of [13:05] the graph in context and then tell you [13:08] that's probably because SB had a load [13:10] shedding at the same time. [13:13] That's the kind of answers you can get. [13:18] So that takes me to the next pillow [13:21] detecting signals. Our current [13:24] challenges with detection, as you might [13:27] have guessed, are basically stemming [13:29] from how data systems existed. [13:33] Um, with isolated siloed data systems, [13:36] we ended up with isolated metrics and [13:39] alerts. You have a metric that is [13:41] regressing. We have no idea how it is [13:44] connected to some other metric in some [13:46] other microser or some other app. [13:50] We all jumped on the alerting bandwagon [13:53] at some point. Uh, anomaly detection. [13:55] Anyone? Yep, we did that. Um, and now we [14:00] can automatically probably detect [14:02] signals. You can detect regressions and [14:04] say, "Aha, this metric regressed." That [14:07] that was a pretty cool uh power to have [14:11] probably a decade ago. So, how's it [14:14] going today? Developers still [14:16] complaining cuz we get that a lot too. [14:19] this alert is not actionable. I have no [14:21] idea what to do with this and it's [14:24] probably going to go away in a bit. Just [14:25] give it some time. That happens a lot. [14:28] And that leads to what I call alert [14:31] fatigue. Developers get noisy alerts. A [14:35] ton of them and probably one or two of [14:37] them are actionable for them to do [14:39] anything with. And that causes alert [14:42] fatigue. [14:44] And what that does is with that fatigue, [14:48] people don't trust the system anymore. [14:50] Your on calls think, I don't want to [14:52] wake up at 2 a.m. This is probably going [14:54] to be a noisy alert. I don't want to [14:56] wake up to this page. If I do, I'm going [14:58] to [15:00] uh kick the kick the bucket down the [15:03] road. But the problem with that approach [15:06] is uh your systems are going to suffer. [15:09] What if that signal was real? [15:12] You would never know. [15:14] But with AI like I said uh anomaly [15:17] detection obviously uh we'll continue [15:20] evolving ways in which we can detect [15:23] regressions but also give it context. We [15:27] can say here is a regression uh but also [15:31] look at other related services or other [15:34] related aspects of the apps and see if [15:37] those metrics are also regressing. So [15:39] you can create an ecosystem of connected [15:42] alerts. [15:44] And when you have a connected u set of [15:48] alerts, what you can do at this point is [15:52] uh [15:54] the actionability on those uh signals is [15:58] way higher. Now what you get is [16:01] awareness based triaging. [16:04] Say you're on call on your front end [16:05] application gets a page today. That need [16:08] not happen if your system can [16:10] automatically say this regression [16:12] happened but it's probably coming from a [16:15] service that's way downstream. So the [16:16] alert would actually go to the [16:17] downstream service on call not to your [16:20] uh app developer [16:24] and so we can catch signals early [16:27] easily. Then what next? How do we debug [16:31] and resolve issues? [16:34] In a current state of debugging, you [16:36] already understand what problems are [16:38] going to exist. Silo data leading to [16:41] standalone dashboards. You have some [16:43] metrics that you want to track for your [16:45] application, your stack, your service, [16:47] you have it on a dashboard. We kind of [16:50] got better at it by saying, let's put in [16:52] more charts on the same dashboard. So [16:54] when one metric moves, we'll know [16:56] quickly if other related metrics also [16:58] moved or not. But that's still finding a [17:01] needle in a in a haststack. Maybe a few [17:04] more needles, now that's it. But when [17:07] there are unrelated logs and metrics, it [17:10] gets really hard to form connections. [17:13] Especially when your system is working [17:15] at the scale at which large [17:18] organizations do like Netflix, you have [17:20] thousands of microservices again that [17:22] might just work to compose one aspect of [17:25] your app. And when something goes wrong [17:28] there, if the logs and metrics are not [17:30] related, you're going to run into [17:31] problems. problems, especially things [17:33] like this issue is not on my app. It's [17:37] probably coming down from that service. [17:39] No, I don't see that happening with my [17:41] service. It's probably way downstream, [17:43] right? How often do people look at logs [17:46] from two two services downstream? Forget [17:50] the upstream ones. It's their problem, [17:51] right? But we almost never do. [17:56] But then what does AI unlock for us [17:59] here? One of the most popular use cases, [18:02] assistance. You have LLMs, BS, use rag [18:07] and let it guide you to light. [18:10] Now, uh if you've been down that path, [18:12] you know it's not easy. Creating [18:15] accurate results, reliable assistance to [18:18] guide through a debugging process is [18:20] just not that easy at all. Especially [18:23] when you have the criticality of an on [18:25] call system. uh if you have your models [18:29] hallucinating and not fine-tuned enough, [18:32] it's going to take longer than it used [18:34] to. [18:36] So [18:38] what we do is we want to make sure and [18:42] it would be remiss if I don't mention [18:44] the keywords here agentic AI but what we [18:48] really want to do is build specialized [18:50] agents smaller models focused agents [18:53] that are really good at a specific [18:55] domain or doing specific things well and [18:59] then use a more general coordinator [19:01] agent that can control and orchestrate [19:04] using these different agents. [19:06] and tools and then it automatically [19:08] executes on a bunch of tasks. Think [19:11] about it. [19:13] Uh you can have an agent if you have a [19:16] GraphQL based system, you have an agent [19:17] that understands your GraphQL queries [19:20] and services very well. It doesn't uh [19:23] know much outside of it, but it's just [19:25] really good at that. If you have a [19:27] front-end application layer, there's an [19:29] agent that really understands everything [19:31] about it. What versions went out went [19:33] out at what time? what changes went in [19:35] uh in those versions and so on. And when [19:38] you have such specialized agents working [19:41] together and then there's a coordinator [19:44] uh leading to a mixture of experts [19:46] system, we get way better results than [19:49] you do just using LLMs and rags. [19:54] And with such a system that can [19:56] coordinate and work together and I'm [19:58] getting a little futuristic here uh but [20:01] these are things and this is an active [20:03] area of research that I'm working on. So [20:04] if you're interested talk to me right [20:06] after automatic root cause analysis [20:09] where you can automatically figure out [20:11] using these agents when a regression [20:14] happens what could be the potential root [20:16] causes [20:17] at least stack rank and say these are [20:19] the potential things that happened [20:21] around this time and these are the most [20:23] related ones. So this is these are [20:25] probably the top three uh root causes [20:29] potentially. [20:31] Uh that takes me to the next pillar of [20:34] prevention. Prevention typically existed [20:38] as an afterthought [20:40] like what typically happens is [20:43] mostly driven by some business uh need. [20:47] There will be a large scale effort [20:49] saying we need to optimize our [20:50] application and services. we need to [20:52] optimize this whole thing. We need to [20:54] make it faster or more efficient and [20:57] then there would be a quarter year long [21:00] effort to do something better. And [21:02] that's how we optimize. [21:06] What that means is there are always [21:08] hidden bottlenecks in your system that [21:10] we never know about until we really have [21:12] to. [21:15] Something that happened a few years ago [21:16] was shifting left. We say let's test as [21:21] much as possible before production. So [21:24] very little breaks afterwards. Shift [21:26] left. Uh which means you come up with [21:30] scenarios uh blind spots that are part [21:34] of your system and then create testing [21:36] environments, simulations. Uh digital [21:39] twin was a thing too for a while uh to [21:42] start testing beforehand before you go [21:44] to production. But again, it was almost [21:47] impossible to test every scenario until [21:50] you actually know that it exists in the [21:51] real world. [21:53] But what AI unlocks here for us? [21:57] Remember we talked about root cause [21:58] analysis? Well, with enough root cause [22:01] analysis and enough data around the [22:04] patterns that lead to a particular [22:06] incident and then we can always feed in [22:08] the actual root cause. Your AI is really [22:12] good at understanding patterns. patterns [22:14] after something happened, but also [22:16] patterns that led to an incident. And [22:20] over time, that will give you the [22:22] ability to analyze your system for [22:24] bottlenecks predictively. Say predict [22:28] where my bottlenecks are in this app. [22:31] What how can I improve my latency, [22:35] right, on my service, things like that. [22:39] Uh what that also unlocks for us is if [22:42] it's able to identify those patterns, we [22:45] can unlock scenarios. Scenarios that [22:47] lead to certain bottlenecks that could [22:49] lead to certain bottlenecks and we can [22:51] add it to our testing strategy. We say [22:53] these are scenarios that are getting [22:55] flagged. So let's now add it to our [22:57] actual test cases, test plans, whatever. [23:00] And then you start testing for those [23:02] scenarios based on the recommendations [23:03] you get from your AI. [23:07] And then we can also take it a step [23:08] further about these were the resolutions [23:11] ultimately. So feed that information as [23:13] well into your AI and what that can see [23:16] now is here were the patterns here was [23:19] the actual incident and this was the [23:21] resolution for it. Got it? And then you [23:23] keep training on the resolution piece as [23:26] well. You can have an AI eventually that [23:28] can say uh at the pace at which AI is [23:32] evolving we'll be able to say okay for [23:35] this issue you need to here is a [23:38] potential bottleneck go tune your GC [23:40] parameters here is another potential [23:43] issue that's coming up that's probably [23:45] from uh a memory leak in one of your [23:47] services things like that you can have [23:50] this continuous tuning mechanism enabled [23:52] with your AI which can ultimately lead [23:54] to things like self-healing systems [23:57] If tuning is possible, systems can start [24:00] healing. But optimize for the end to end [24:03] efficiency, the end to end experience [24:05] that you're going to deliver for your [24:06] users. [24:08] So that's all possible with [24:09] observability in the age of AI. Where we [24:13] started, we had alerts, noisy alerts [24:16] everywhere. But now we move on to [24:18] insights from manual triaging processes, [24:21] we can move to automated root cause [24:23] analysis. And instead of isolated [24:25] dashboards and tools, we have integrated [24:28] systems that are intelligent enough to [24:30] tell us exactly what we need to know. [24:33] And forget your postmodern reports. We [24:35] are moving towards proactively [24:37] preventing issues from happening. All of [24:40] this leading to a better app experience [24:43] for everyone every time all over the [24:45] world. That's it from me. Uh Pres feel [24:50] free to connect with me on LinkedIn. [24:51] Thank you.