AI Agents In Production Infrastructure: What The Industry Learned The Hard Way with JP Morgan Chase
Ai4
AI-assisted summary of AI Agents In Production Infrastructure: What The Industry Learned The Hard Way with JP Morgan Chase — original source. Check the original for full context.
Source published:
Detailed notes
## Background Published on YouTube by the Ai4 channel on August 19, 2026, this session features Rahul Tanniru, Senior Vice President of Software Engineering at JPMorgan Chase. According to the description, the presentation details the real-world failures, prompt engineering patterns, agentic architectures, and governance guardrails developed when deploying Claude-based autonomous AI agents across enterprise Infrastructure as Code (IaC) and SRE workflows.
## TL;DR Rahul Tanniru examines the operational shift from reactive autocomplete assistants to fully autonomous agentic workflows capable of writing Terraform, triaging incidents, and managing Kubernetes clusters. He breaks down common production failure modes—including context dilution, embedding mismatch across environments, and ambiguous ticketing prompts—and presents proven patterns like root-level AGENTS.md configurations, multi-worker agent loops, and hard retry budgets. Through real-world enterprise case studies like EKS flapping node remediation and multi-cloud migrations, the talk outlines vital guardrails including test-first agent development, mutation testing, and least-privilege access control. Finally, the presentation provides an ROI framework balancing raw token costs against first-pass reliability, urging engineers to become proactive architects rather than passive tool subscribers.
## Atmosphere & style The presentation carries a pragmatic, engineering-first tone that deliberately cuts through AI hype to focus on battle-tested production realities. Tanniru speaks with the candid, measured authority of a seasoned platform leader, treating autonomous agents with the same rigorous scrutiny one would apply to a junior engineer. The narrative builds methodically from messy real-world failures and edge-case pitfalls into actionable architectural solutions, leaving technical viewers with a grounded, highly disciplined blueprint for enterprise AI adoption.
## Key takeaways - [00:04](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=4s) Agentic coding shifts AI from reactive autocomplete within a single file to autonomous, multi-file execution that evaluates logic gates, drafts execution plans, and resolves infrastructure errors. - [01:52](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=112s) A five-stage closed-loop pattern (Plan, Context Assembly, Tool Invocation, Observation, and Replanning) allows agents to detect subtle omissions—like missing DB subnet groups—before executing a terraform apply. - [04:28](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=268s) Standard context retrieval methods often fail in infrastructure: grep misses multi-level variable lookups, vector search blends dev and prod configurations, and direct file injection quickly exhausts context windows. - [07:02](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=422s) Autonomous triage and remediation workflows can compress SRE diagnostic and patching cycles from 1.5 hours down to approximately 15 minutes, as proven in live EKS flapping node incidents. - [08:49](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=529s) Autonomous agents should be treated like new junior hires, governed by strict least-privilege permissions, confined to non-production environments during validation, and backed by end-to-end trace logging. - [10:48](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=648s) Evaluating LLMs purely by token cost is deceptive; higher-cost models with superior first-pass accuracy (such as Claude) reduce expensive multi-turn retry loops and yield lower overall operational spend. - [13:03](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=783s) Standardizing root-level AGENTS.md configuration templates provides a persistent single source of truth for disaster recovery regions, backend state locking, and corporate compliance rules. - [15:29](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=929s) Relying on raw, colloquial issue tracker descriptions introduces catastrophic hallucination risks, such as an agent interpreting 'reiterate the database' as a directive to delete and recreate the database. - [17:35](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=1055s) Production architectures must enforce a strict three-retry budget rule; if an agent fails to resolve an infrastructure bug after three attempts, it will churn randomly and must escalate to a human engineer. - [21:45](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=1305s) Effective automated incident remediation uses a multi-agent orchestrator-worker hierarchy with Model Context Protocol (MCP) tool integration rather than monolithic end-to-end agents. - [30:42](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=1842s) Validating AI-generated infrastructure requires robust test-first frameworks, property-based testing, LLM-as-a-judge inspections, and mutation testing to catch latent configuration and security gaps. - [36:56](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=2216s) Engineering teams must become architects rather than passive tool subscribers, measuring agent ROI through operational stability and business outcomes rather than vanity metrics like PR count or lines of code.
## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=4s) Introduction and Coding Assistants vs. Agentic Coding
Rahul Tanniru, Senior Vice President of Software Engineering at JPMorgan Chase, opens the session by addressing the shift from experimental AI to real-time industrial deployment of agentic architecture workflows. Rather than treating artificial intelligence as a theoretical novelty, organizations are actively deploying agents into mission-critical infrastructure to write Terraform, handle incident triage, and manage cloud resources. The objective of the talk is to transparently examine the failures encountered along the way, how teams overcame them, the prompt engineering patterns that survived production load, and the architectural design of multi-agent systems.
To establish a baseline, Tanniru contrasts conventional coding assistants with true agentic coding across three core dimensions: triggers, scope, and error handling. Traditional coding assistants function primarily as reactive autocomplete engines. If an engineer sets out to define an AWS RDS module in Terraform, a standard assistant simply suggests inline resource blocks within a single active file. In contrast, agentic coding operates autonomously across an entire repository or module structure; it evaluates logic gates, drafts a Terraform execution plan, triggers tools, and can autonomously run `terraform plan` and `terraform apply`.
When handling errors, traditional assistants require constant human intervention to read error logs, diagnose the issue, and manually implement a fix in that isolated file. Agentic systems, however, are capable of ingesting operational runbooks, determining the necessary remediation steps across multiple files, and triggering automated workflows to fix the errors directly.
| Dimension | Traditional Coding Assistant | Agentic Coding | |:--- |:--- |:--- | | **Trigger & Execution** | Reactive autocomplete for resource snippets | Evaluates logic gates, drafts execution plans, and executes tools/commands | | **Operational Scope** | Single isolated file | Whole repository, module hierarchy, and multi-file dependencies | | **Error Handling** | Relies on human to diagnose and apply fixes | Ingests runbooks, diagnoses root causes, and autonomously resolves errors |
---
[01:52](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=112s) Anatomy of Agentic Loops and the Five-Stage Closed-Loop Pattern
The technical anatomy of agentic development begins with a clear developer goal or intent, such as provisioning an EC2 instance or adding a read replica to an existing RDS database instance. The developer entrusts this goal to a Large Language Model (LLM), which auto-generates the corresponding resource blocks. However, enterprise production demands strict adherence to organizational guardrails, such as mandatory resource tagging, encryption enforcement, and security baseline compliance. In a typical development loop, code pushed to GitHub or Bitbucket triggers an automated feedback loop where validation policies check the code against organizational standards, prompting the agent to iteratively refine the implementation until it meets all governance criteria.
```mermaid flowchart TD accTitle: Five-Stage Agentic Infrastructure Closed-Loop accDescr: Flowchart showing the five stages of an agentic infrastructure loop from intent to observation and apply.
subgraph Phase1["1. Planning & Context"] A["Goal / Intent<br/>(e.g., Create RDS Read Replica)"] --> B["Context Assembly<br/>(Read State & Terraform Files)"] end
subgraph Phase2["2. Execution & Verification"] B --> C["Tool Invocation<br/>(Generate Code & Run Plan)"] C --> D{"Observation & Validation<br/>(Analyze Plan Output)"} end
subgraph Phase3["3. Self-Correction & Apply"] D -- "Missing Subnet Group Detected" --> E["Replanning & Auto-Fix<br/>(Inject DB Subnet Group)"] E --> C D -- "Plan Verified Clean" --> F["Terraform Apply<br/>(Execute to Cloud Infrastructure)"] end ```
Tanniru highlights a five-stage closed-loop pattern: 1. **Plan / Intent:** The developer specifies the desired end state (e.g., "create a read replica for an RDS database"). 2. **Context Assembly:** The agent gathers ambient context by scanning repository files and inspecting current Terraform state to determine precisely where modifications belong. 3. **Tool Invocation:** The agent generates the necessary HCL configurations and invokes the Terraform CLI to generate an execution plan. 4. **Observation:** The critical evaluation phase where the agent analyzes the plan output. Tanniru notes a common pitfall in infrastructure: "The plan looks good, nothing to destroy, but apply fails." A sufficiently intelligent agent catches hidden omissions during observation—realizing, for instance, that while the plan reports one resource to add and zero to destroy, the read replica lacks an explicit DB subnet group association that will trigger an API error during actual deployment. 5. **Replanning and Apply:** Recognizing the missing configuration, the agent enters an internal correction loop, injects the missing DB subnet group into the resource definition, re-verifies the plan, and safely proceeds with the apply phase.
---
[04:28](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=268s) Context Retrieval Failure Modes: Grep, Embeddings, and Direct Reads
Tanniru polls the audience on whether anyone has ever asked an AI agent to perform a task and received a 100% accurate, completely hallucination-free result on the first attempt. Acknowledging that hallucinations are an inherent reality of LLMs, he dissects how different context retrieval strategies fail in infrastructure codebases:
- **File Search / Grep Matching:** When tasked with updating security group rules for a resource named `SecurityGroupA`, an agent utilizing simple keyword search or grep behaves like a `Ctrl+F` scan. While it successfully finds direct references to the string, it fails completely when security group IDs or rules are abstracted two or three levels deep through variable lookups, local maps, or module outputs. Failing to resolve these indirect references leads directly to configuration drift and production outages. - **Semantic Search / Vector Embeddings:** Instead of matching literal strings, embedding-based approaches search by conceptual meaning. When an agent attempts to retrieve configuration patterns for an RDS database, semantic similarity matches database definitions across development, staging, and production environments indiscriminately. Because different environments maintain different database engine versions, parameter groups, or sizing tiers, cross-referencing across environments can cause the agent to blend configurations—such as applying a development version tag to a production deployment. - **Direct File Reads:** Providing explicit file paths and structured guidelines yields the highest precision, as the agent reads the exact source code required. However, direct file injection only scales across a small number of files. When applied to massive repositories, it saturates the model's context window, introduces noise, triggers false inferences, and significantly increases API token costs.
---
[07:02](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=422s) Real-World Incident Walkthrough: Autonomous EKS Flapping Node Remediation
To illustrate agentic workflows under real operational pressure, Tanniru walks through an automated incident response scenario involving an Amazon EKS cluster suffering from flapping worker nodes.
```mermaid flowchart LR accTitle: Real-Time EKS Flapping Node Auto-Remediation Flow accDescr: Step-by-step incident response flow executed by an AI agent to resolve flapping Kubernetes nodes.
A["Alert Ingestion<br/>EKS Flapping Nodes"] --> B["Cluster Triage<br/>Execute kubectl diagnostics"] B --> C["Config Inspection<br/>Evaluate Node Group Terraform"] C --> D{"Root Cause Analysis<br/>Spot Eviction vs Lifecycle Hook"} D --> E["Code Remediation<br/>Fix Spot Config & Add Drain Hook"] E --> F["Safe Deployment<br/>Plan, Verify & Apply Terraform"] ```
When the monitoring alert fires, the agent executes an end-to-end remediation sequence: 1. **Diagnosis:** Ingesting the alert, the agent invokes `kubectl` CLI commands to identify specifically which worker nodes are in an unhealthy or flapping state. 2. **Configuration Inspection:** It locates the corresponding node group Terraform files responsible for those nodes to evaluate the underlying provisioning code. 3. **Hypothesis Testing:** The agent formulates two potential root causes: aggressive spot instance interruptions or misconfigured EC2 Auto Scaling lifecycle hooks. It cross-references diagnostic data across running instances and confirms that spot instance market evictions are destabilizing the cluster. 4. **Remediation and Future Prevention:** The agent updates the Terraform configuration to resolve the node group capacity settings. Critically, it acts proactively rather than merely applying a temporary patch: it configures an Auto Scaling lifecycle termination hook combined with cluster drain selection annotations to ensure graceful pod eviction during future termination events. 5. **Execution and Validation:** The agent generates the Terraform plan, validates syntax and structural integrity, executes `terraform apply`, and verifies recovery across monitoring dashboards.
Tanniru emphasizes the dramatic efficiency gains: while an on-call Site Reliability Engineer (SRE) typically spends 1 to 1.5 hours diagnosing, patching, and testing such an incident manually, the autonomous agentic architecture completed the entire triage and remediation loop in roughly 15 minutes.
---
[08:49](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=529s) Principle of Least Privilege and LLM Evaluation Dimensions
Deploying AI agents requires treating them like human personnel: "Think of your agent as a new member who joined the team." Organizations do not grant root credentials or unrestricted access to production databases to a new hire on day one; similarly, autonomous agents must be governed by strict least-privilege access controls. In production architectures, agent execution should be confined to isolated CI/CD pipeline stages and restricted to non-production or development environments while models and toolchains are being validated. Furthermore, robust observability is mandatory. Teams must capture comprehensive trace logs of all agent actions, tool calls, and execution outputs to determine conclusively during an outage whether an error was introduced by a human engineer or an autonomous agent.
Evaluating foundation models for infrastructure agents requires balancing five core metrics: - **Code Correction / Accuracy:** The percentage of syntactically and semantically valid infrastructure code the LLM generates without human intervention. - **Context Retention:** The model's capacity to process expansive input state files and multi-module schemas without degrading or truncating output accuracy. - **Tool Use Reliability:** How consistently and accurately the model structures arguments for Model Context Protocol (MCP) servers, cloud APIs, and local CLI tools. - **Execution Latency:** The turnaround time for reasoning loops and tool calling during time-critical incidents. - **Inference Cost:** The ongoing operational expenditure associated with high-token agentic loops.
Tanniru notes that recent benchmark data demonstrates Anthropic's Claude leads the industry in code correction and technical accuracy for infrastructure workflows, though it represents a more expensive option relative to alternative foundation models.
[10:48](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=648s) Model Selection: Code Reliability vs. Raw Token Cost
When teams evaluate LLMs for infrastructure automation, comparing models purely on API token cost is a deceptive metric. While some models appear cheaper on paper, models like Anthropic's Claude often come out ahead in practice because of first-pass code reliability. If a cheaper model produces incorrect or brittle infrastructure code, engineers find themselves executing three to four retry loops to get a working configuration. Those repeated roundtrips quickly multiply token usage and inflate overall costs. Claude often delivers syntactically correct and reliable infrastructure code on the first attempt, drastically cutting down on retries. Selecting an LLM for agentic workloads requires evaluating how well its reliability profile offsets the raw per-token price tag.
[11:31](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=691s) Tested Prompt Engineering Patterns: Context Injection and Chain of Thought
Real-world deployments highlight critical lessons around structured context injection and downstream dependency handling. In one case, an agent was given a simple prompt: *"Hey, can you please add me an auto scaling group of AWS EC2 instance?"* The model successfully generated the Auto Scaling Group (ASG) resource block, but because it lacked operational guidelines, it defaulted arbitrarily to a minimum of two instances and a maximum of four, tied to a generic CPU threshold. That default might technically work, but it rarely matches actual production workload demands. Providing a tightly structured prompt that details exact replica bounds, specific evaluation time windows, and target CPU/memory thresholds ensures the agent outputs actionable infrastructure code instead of guesswork.
```mermaid flowchart TD accTitle: Structured Context Injection vs Naive Prompting subgraph Naive["Naive Prompting"] N1["Prompt: Add EC2 ASG"] --> N2["Agent Guesses Attributes"] N2 --> N3["Random Bounds: Min 2, Max 4<br>Arbitrary CPU Metric"] end
subgraph Structured["Structured Prompt Injection"] S1["Prompt + Workload Specs"] --> S2["Agent Ingests Context"] S2 --> S3["Deterministic ASG Config<br>Exact Replica & Metric Rules"] end ```
Single-shot Chain of Thought prompts can still introduce subtle infrastructure breakage. When asked to modify existing VPC cluster configurations, an agent successfully updated the target VPC subnet groups and security group references. However, the agent broke all indirect references that relied on variable lookups and implicit downstream dependencies rather than direct VPC IDs. Without explicit dependency mappings, agents routinely miss implicitly coupled resources across Infrastructure as Code (IaC) codebases.
[13:03](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=783s) Grounding Agents with Root-Level `AGENTS.md` Templates
To enforce organization-wide infrastructure guardrails, teams should define a root-level `AGENTS.md` configuration template. This file acts as the single source of truth for the agent whenever it generates or updates resources:
```markdown # Infrastructure Standards Template - AWS: - Primary Region: us-east-1 - DR / Backup Region: us-west-2 - GCP: - Dedicated Data Platform Region: us-central1 - Terraform State Management: - State Backend: Amazon S3 with DynamoDB state locking - Workspace Strategy: Dedicated dev workspaces - Security & Tagging Policies: - Mandatory default encryption across all resources - Strict ban on hardcoded account IDs and regions - Standardized corporate tagging schemas enforced ```
By reading this file upfront, the agent consistently honors designated disaster recovery regions, backend storage configurations, and compliance policies, preventing invalid or unauthorized infrastructure changes.
[14:32](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=872s) The Emerging Tooling Ecosystem for Agentic Infrastructure
Modern infrastructure automation stacks are rapidly converging on specialized tooling across the entire agent lifecycle:
| Layer | Popular Tooling & Platforms | |:--- |:--- | | **IDE Agents** | Cursor, GitHub Copilot, Windsurf | | **CLI & Terminal Agents** | Claude Code | | **Agent Orchestration** | LangGraph, AutoGen | | **Context Management** | Root-level `AGENTS.md` configuration templates | | **IaC Validation & Security** | `terraform plan`, Terratest, tfsec, proprietary internal linters | | **Agent Observability** | LangGraph tracing, AWS CloudTrail integrations, event logs |
[15:29](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=929s) Feature Engineering: Mitigating Hallucinations and the Jira Risk
Reducing hallucinations in production agentic workflows requires feeding the model continuous, deterministic context:
- **Terraform Plan & Output Pairing:** Ingesting `terraform plan` along with `terraform show` output lets the agent compare the desired state against the planned diff, verifying its own work before changes apply. - **CloudTrail & Git History Integration:** Correlating AWS CloudTrail event streams with GitHub commit history provides the agent with chronological context on who changed what, narrowing down unexpected drift. - **Side-by-Side Terratest Integration:** Passing active Terratest test suites directly into the agent's context window ensures code generation is immediately measured against formal acceptance tests. - **Upstream & Downstream Dependency Maps:** Maintaining an explicit service graph lets the agent know exactly which dependent systems need parallel updates whenever an upstream resource changes.
Relying on unstructured Jira descriptions alone is a major anti-pattern. Because engineers write tickets using ambiguous, colloquial phrasing, agents can misinterpret intent. In one real-world incident, a Jira ticket asked an agent to *"reiterate the database evaluations."* The agent interpreted "reiterate" as an instruction to delete and recreate the database, wiping out the resource. Production integrations with issue trackers require strict, structured ticket templates rather than raw natural language.
```mermaid flowchart LR accTitle: Grounded IaC Context Engine subgraph Inputs["Context Sources"] A["Terraform Plan & Show Output"] B["CloudTrail + Git Commits"] C["Terratest Test Suites"] D["Service Dependency Graph"] end
subgraph AgentEngine["Agent Processing"] E["Context Injection & Validation"] end
subgraph Output["Safe IaC Execution"] F["Verified, Grounded Code Changes"] end
A --> E B --> E C --> E D --> E E --> F ```
[17:35](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=1055s) Hyperparameter Tuning and Production Guardrails
Operating agentic cloud systems requires tuning key operational knobs to keep agents reliable and budget-conscious:
- **Context Window Management:** Carefully balance the token ratio between injected context and generated output to avoid degradation. - **The Three-Retry Budget Rule:** Enforce a hard ceiling on automated retries. If an agent fails to resolve an infrastructure bug within three retries, subsequent attempts almost never fix the root issue; the model merely churns and swaps random lines of code. Escalate to an engineer when the retry budget hits three. - **Multi-File Task Chunking:** Avoid dumping 20 to 30 files into a single context window. Massive multi-file payloads trigger context dilution and hallucinations. Break workloads into smaller, logically grouped multi-file chunks.
[18:57](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=1137s) Common Failed Approaches in Cloud Agent Adoption
Teams frequently run into four common architectural failure modes when introducing autonomous agents:
- **Prompting Without Infrastructure Context:** Requesting resources without explicit boundary conditions yields arbitrary configurations (such as auto-scaling groups with default, unvetted sizing parameters). - **Premature `terraform apply` Permissions:** Granting agents autonomous apply permissions too early invites catastrophic outages. Treat every new agent like a new junior engineer: restrict its initial scope to isolated sandbox environments, and require mandatory human-in-the-loop approval before running applies in production. - **Multi-Service Auto-Completion Across Helm Charts:** Agents indexing cross-environment Helm configurations frequently mix up parameters, accidentally copying staging or dev configurations directly into production releases. - **Monolithic End-to-End CI/CD Agents:** Assigning a single agent to manage an entire end-to-end CI/CD pipeline causes context overflow as it progresses through build, test, and release phases. Instead, assign isolated, single-purpose agents to each distinct stage of the pipeline.
[20:38](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=1238s) The Shift in Incident Response: Manual Triage vs. Agentic Workflows
Traditional production incident response is slow and manual. When an alert fires:
1. The on-call engineer gets paged. 2. The engineer logs in, identifies the broken application, and sifts through system logs. 3. If the root cause is unclear, the engineer pages subject-matter experts (SMEs), hoping they pick up quickly. 4. Once the engineer identifies the root cause, they write a patch, test it in a development environment, open a pull request, wait for human approvals, and deploy to production.
Even when everyone responds immediately, this traditional workflow routinely takes two to three hours. Agentic engineering transforms this lifecycle by autonomously ingesting the initial alert page, parsing telemetry logs, and kicking off immediate automated remediation steps.
[21:45](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=1305s) Multi-Agent Architecture for Automated Remediation
Autonomous incident remediation relies on a structured sequence where an agent queries logging pipelines, analyzes application logs, consults historical runbooks for documented fixes, and tests the remediation via a `terraform plan` in the development environment before submitting a draft pull request for human approval. This end-to-end cycle compresses an engineering task that typically takes 30 minutes to 3 hours down to roughly 20 to 25 minutes.
To coordinate these responsibilities, a multi-agent system uses a central orchestrator paired with specialized worker agents. A planner agent evaluates broad high-level goals and breaks them into smaller subtasks, dispatching specific workers for code generation, automated testing, or documentation. Balancing agent granularity is critical: over-indexing on single-purpose agents creates significant maintenance overhead, requiring teams to deliberately combine compatible tasks into unified agent runs.
```mermaid flowchart TD accTitle: Multi-Agent Architecture for Automated Remediation subgraph Planning["Coordination & State"] ORC["Central Orchestrator"] --> PLAN["Planner Agent"] ORC <--> MEM[("Memory Store / Vector DB<br/>(Session & State Info)")] end
subgraph Workers["Worker Agents & Tools"] PLAN --> W_CODE["Code Worker"] PLAN --> W_TEST["Test Worker"] PLAN --> W_DOC["Documentation Worker"] PLAN --> MCP["MCP Worker Node"] MCP <--> TOOLS["External Tools<br/>(GitHub, Bitbucket, Confluence)"] end ```
- **Session State Storage**: Similar to maintaining state files during a Terraform execution, agents persist dynamic session state and context within a vector database memory store. - **Model Context Protocol (MCP)**: Tool integration across enterprise repositories (e.g., Confluence, GitHub, Bitbucket) is routed through dedicated MCP worker agents.
---
[24:20](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=1460s) Developer Productivity and SRE Operations
Platform engineers leverage agentic workflows to compress three to four days of manual engineering effort into a single day without displacing the developer's role. A standard daily workflow illustrates this velocity increase: triaging an incident at 9:00 AM takes 30 minutes (down from 1 to 2 hours), followed immediately by drafting Prometheus alert rules, authoring Terraform modules, generating Terratest coverage suites, and shipping to production. Rather than replacing engineers, "it's basically increasing the productivity of developer, how many tasks he can do."
In site reliability engineering (SRE), subject matter experts define service level agreements (SLAs), service level objectives (SLOs), and error budgets using natural language prompts. The agent ingests these operational requirements and provisions dashboards directly within monitoring platforms like Dynatrace. Runbook automation functions similarly: on-call engineers supply baseline operational procedures, and autonomous agents cross-reference and execute the documented steps during active service disruptions.
---
[26:39](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=1599s) Enterprise Migration Case Studies: EKS, Multi-Cloud Terraform, and Logging
Real-world production rollouts highlight substantial efficiency gains across large-scale migration and provisioning initiatives when standard templates guide agentic execution:
```mermaid flowchart LR accTitle: Agentic Infrastructure Migration Workflow subgraph Prep["1. Standardization"] TPL["Define Golden Templates<br/>(Patterns, Tags, Security Policies)"] end
subgraph Gen["2. Agentic Parallel Run"] EXEC["Parallel Generation / Upgrade<br/>(200+ Clusters / 80+ Modules)"] TEST["Parallel Test Suite Creation<br/>(Terratest / Dev Deployments)"] end
subgraph Validation["3. Promotion & Review"] AUTO["Auto-Validated Workloads<br/>(Dev Verified)"] HUMAN["Human Review / Approval<br/>(Edge Cases & Interconnects)"] MAIN["Merge to Master/Main"] end
TPL --> EXEC EXEC --> TEST TEST --> AUTO TEST --> HUMAN AUTO --> MAIN HUMAN --> MAIN ```
- **Amazon EKS Cluster Upgrades (v1.24 to v1.29)**: - Scope: Migrating 200+ Kubernetes workloads under a strict 8-week AWS deprecation deadline, a process typically requiring 6 to 7 weeks of manual configuration audits. - Outcome: Standardized migration patterns enabled agents to identify configuration mismatches, update cluster manifests, and generate test suites in parallel. 190 of the 200+ workloads were upgraded and validated in development autonomously, leaving only complex edge cases for manual engineering review. - **Multi-Cloud Terraform Modules (AWS & GCP)**: - Scope: Developing 80+ enterprise Terraform modules complying with internal encryption, resource tagging, and security baselines, which normally requires months of engineering. - Outcome: The agent generated multi-cloud resource blocks alongside corresponding Terratest suites simultaneously, cutting total project delivery time down to 2 weeks. Complex cross-cloud network interconnectivity was flagged for human validation prior to merging to the `main` branch. - **Logging Library Upgrades**: - Scope: Refactoring deprecated logging framework versions across distributed architectures. - Outcome: The agent successfully located dependencies and upgraded 38 out of 40 services, with engineers handling the remaining 2 legacy edge cases.
---
[30:42](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=1842s) Production Realities and Infrastructure Testing Strategies
Real-world deployments show a clear split between speed and full autonomy: 55% of respondents observe substantially faster delivery cycles, yet 38% encounter security gaps where agents fail to apply required controls. While agents reliably pass basic unit and integration tests, deeper system health checks still fail to reach 100% pass rates, highlighting the absolute necessity of automated guardrails and mandatory human approval gates.
``` +-----------------------------------------------------------------------+ | AI Code Generation Engine | +-----------------------------------------------------------------------+ | v +-----------------------------------------------------------------------+ | Testing & Validation Harness | | | | [Property-Based Testing] [LLM-as-a-Judge] [Mutation Testing] | | Validate resource props Secondary LLM Inject deliberate | | and config states prior validates logic faults to verify | | to infrastructure rollout against errors agent detection | +-----------------------------------------------------------------------+ | v +-----------------------------------------------------------------------+ | Human-in-the-Loop & Governance Gates | +-----------------------------------------------------------------------+ ```
To validate AI-generated infrastructure before deployment, engineering teams use three core testing methodologies:
- **Property-Based Testing**: Validates granular Terraform resource properties and state configurations against target infrastructure rules before provisioning resources. - **LLM-as-a-Judge**: Employs an independent, secondary LLM model to inspect the primary model's infrastructure manifests. Discrepancies between model outputs flag logic errors for inspection. - **Mutation Testing**: Introduces deliberate faults and misconfigurations into the agent-generated code to verify whether the AI test harness correctly catches the breakage.
[32:49](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=1969s) Test-First Agentic Development and Production Guardrails
When developing agentic systems, testing cannot be an afterthought. Engineering teams must establish their unit tests, integration tests, and outcome-based functional tests before letting an agent write any application logic. By defining the test suite first, the agent can inspect the target specifications, understand what constitutes correct functionality, and generate code specifically designed to satisfy those test cases. This test-first methodology significantly reduces hallucination and constrains the agent's generative output to verifiable functional requirements.
```mermaid flowchart TD accTitle: Test-Driven Agentic Development Workflow subgraph Spec ["1. Specification & Testing"] A["Define Functional Requirements"] --> B["Write Unit & Integration Tests"] end subgraph AgentLoop ["2. Agentic Execution"] B --> C["Agent Reads Test Suite"] C --> D["Generate Implementation Code"] end subgraph Validation ["3. Production Gate"] D --> E{"Tests Pass & Least Privilege Met?"} E -- No --> C E -- Yes --> F["Human-in-the-Loop Review"] F --> G["Deploy to Production"] end ```
Beyond test-driven generation, autonomous agents must operate under strict security boundaries. Teams should enforce least-privilege execution, granting the agent only the bare minimum permissions required to perform its task. Under no circumstances should agents have unrestricted autonomy in live infrastructure; robust human-in-the-loop approval gates are mandatory before any automated action modifies a production environment.
- **Test-Driven Agent Development**: Writing integration and outcome-based test suites upfront gives the agent clear boundaries, drastically cutting down on hallucinations. - **Least-Privilege Execution**: Limit autonomous agents strictly to the minimal IAM and API scopes needed for the assigned task. - **Production Gating**: Require explicit human-in-the-loop approvals whenever an agent touches or mutates production state.
---
[33:44](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=2024s) Data Freshness, Bias, Observability, and Compliance Governance
Prompt injection defense and data integrity are central to safe agent operations. A cautionary example highlights the danger of stale reference data: a company implemented an AI resume-filtering agent for hiring, only to find the hiring manager asking, *"Hey, why are we getting all of our interview resumes only for men? Why are women not being selected at all?"* The agent had been fed outdated historical hiring data where men predominated, leading it to mirror and amplify that past bias in its screening decisions. If an agent references stale or skewed datasets, it inherently produces flawed and biased outcomes. Continuous data freshness and validation are critical.
Full observability into agent behavior is equally necessary. Teams must monitor the end-to-end execution paths, logging every action, tool call, and decision branch the agent takes. From a governance standpoint, evaluating cost and token consumption must go hand-in-hand with measuring code correctness and reliability. Deploying AI-generated code without rigorous guardrails exposes organizations to legal, compliance, and security risks—including open-source licensing violations, introduced vulnerabilities, and demographic bias.
- **Data Freshness and Bias Mitigation**: Stale reference data causes agents to perpetuate historical biases; knowledge bases and reference datasets must be actively updated and audited. - **Full Execution Observability**: Maintain complete telemetry on workflows, tool invocations, and intermediate reasoning steps taken by autonomous agents. - **Legal and Compliance Safeguards**: Scan all AI-generated code for security vulnerabilities, software licensing risks, and compliance policy violations before integration.
---
[35:24](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=2124s) The Economics of AI Coding and ROI Modeling
Quantifying the economic viability of AI coding agents requires a structured token-cost modeling framework. To calculate true return on investment (ROI), teams must track the direct token cost of running an agent through a specific task, multiply that cost across the anticipated operational volume, and measure it against the net reduction in developer hours.
Calculating agent economics involves looking well beyond base API pricing:
| Cost Factor | Operational Impact | Evaluation Metric | |:--- |:--- |:--- | | **Direct Token Spend** | Base API cost per run multiplied by total execution volume | Total task cost vs. developer hours saved | | **Prompt Engineering & Retries** | Developer time spent refining prompts and managing retries to yield structured outputs | Engineering hours spent on prompt iteration | | **Infrastructure Utilization** | Compute, hosting, and orchestration overhead for running agent frameworks | Active runtime vs. idle compute hosting costs | | **Model Drift** | Upstream LLM version updates causing unexpected behavioral shifts or regressions | Regression rate across model version upgrades |
---
[36:56](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=2216s) Becoming Architects, Not Subscribers: Focusing on Business Outcomes
Engineers must evolve into "architects, not subscribers." Rather than passively consuming AI APIs or relying on shallow tooling, technical teams should master context engineering, embrace disciplined agentic workflow designs, and align their efforts directly with concrete business impact.
Organizations should avoid falling into the trap of using legacy or vanity developer productivity metrics for AI adoption. Evaluating engineering performance based on raw commit counts, lines of code written, number of pull requests merged, or token volume consumed is counterproductive. True productivity and system health must be measured solely on functional outcomes, operational stability, and the business value delivered to users.
## Notable quotes > [01:52](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=112s) The plan looks good, nothing to destroy, but apply fails. — Rahul Tanniru
> [08:49](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=529s) Think of your agent as a new member who joined the team. — Rahul Tanniru
> [11:31](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=691s) Hey, can you please add me an auto scaling group of AWS EC2 instance?
> [15:29](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=929s) reiterate the database evaluations.
> [24:20](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=1460s) it's basically increasing the productivity of developer, how many tasks he can do. — Rahul Tanniru
> [33:44](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=2024s) Hey, why are we getting all of our interview resumes only for men? Why are women not being selected at all?
> [36:56](https://www.youtube.com/watch?v=vWdwtvP0lbk&t=2216s) architects, not subscribers. — Rahul Tanniru
## Mentioned Rahul Tanniru, JPMorgan Chase, AWS, Amazon RDS, Amazon EC2, GitHub, Bitbucket, Amazon EKS, EC2 Auto Scaling, Kubernetes, Anthropic, Claude, Cursor, GitHub Copilot, Windsurf, Claude Code, LangGraph, AutoGen, Terraform, Terratest, tfsec, AWS CloudTrail, DynamoDB, Amazon S3, Google Cloud Platform (GCP), Jira, Helm, Model Context Protocol (MCP), Confluence, Dynatrace, Prometheus
## Video description Rahul Tanniru - Senior Vice President of Software Engineering, JP Morgan Chase
AI agents are already writing Terraform, triaging incidents, and modifying Kubernetes configs in production and most teams are learning the hard way what goes wrong. In this talk, I'll share real failures from deploying Claude-based agentic systems across IaC and SRE workflows, including terraform apply disasters, runbook hallucinations, and CI/CD context overflow. I'll walk through the prompt engineering patterns, agentic loop architectures, and production guardrails that actually held up under real infrastructure load. Attendees will leave with a concrete model selection framework, a security and governance checklist for autonomous agents, and three live scenario walkthroughs, EKS auto-remediation and SLO dashboard generation.
[Source](https://www.youtube.com/watch?v=vWdwtvP0lbk)