AWS Deploys Autonomous Agents to Overhaul Enterprise Codebases
@Scale
Amazon transformed internal maintenance on 30,000 applications into AWS Transform while advancing proactive coding agents that bypass single-model limitations through multi-agent review structures.
## Background
Published on YouTube by the @Scale channel on July 16, 2026, this episode features host Francois Richard in conversation with Anoop Deoras, applied science leader for AWS's Agentic AI business unit. The video explores Deoras's career trajectory from speech recognition to agentic AI, detailing how AWS builds autonomous agents, handles code migrations, and navigates context engineering.
## TL;DR
Anoop Deoras traces his professional journey from statistical speech processing to leading applied science for AWS's Agentic AI initiative. He details how Amazon's massive internal Java migration project laid the groundwork for AWS Transform and discusses the development of Kiro, AWS's coding agent. Deoras highlights key technical challenges in modern agentic systems, including context engineering, multi-agent critique setups for code review, agent memory and intentional forgetting, and the ultimate industry shift from reactive to proactive agents.
## Atmosphere & style
The conversation is deeply technical, peer-to-peer, and pragmatic, feeling more like two veteran engineering leaders exchanging hard-earned wisdom than a marketing presentation. Francois Richard and Anoop Deoras share a warm collegiality, drawing nostalgic parallels between early speech recognition work in the 1990s/2000s and today's AI breakthroughs. The mood is grounded yet visionary, seamlessly moving from corporate enterprise strategies to lighthearted personal anecdotes about weekend robotics projects and PhD editor debates.
## Key takeaways
- [00:00](https://www.youtube.com/watch?v=gijVIvSmgoA&t=0s) The frontier of AI is shifting from reactive agents that await user intent to proactive agents that continuously anticipate requirements and execute complex tasks autonomously.
- [17:05](https://www.youtube.com/watch?v=gijVIvSmgoA&t=1025s) Anoop Deoras's career trajectory across speech recognition, Microsoft, Netflix, and AWS centers on language modeling and sequence modeling as the underlying technical substrate for modern AI.
- [06:36](https://www.youtube.com/watch?v=gijVIvSmgoA&t=396s) AWS's Agentic AI strategy operates on two layers: a primitive infrastructure layer (memory, evals, customization) and frontier agent applications (SDLC and enterprise productivity).
- [11:09](https://www.youtube.com/watch?v=gijVIvSmgoA&t=669s) The seed for AWS Transform came from Amazon's internal project to modernize 30,000+ legacy Java applications, proving that AI agents can automate high-friction codebase maintenance.
- [19:55](https://www.youtube.com/watch?v=gijVIvSmgoA&t=1195s) Kiro, AWS's coding agent, handles both brownfield development (adapting to existing repository patterns) and greenfield development (building from scratch under organizational policies).
- [22:44](https://www.youtube.com/watch?v=gijVIvSmgoA&t=1364s) Context engineering is critical over explicit rule-teaching, as finite context budgets require carefully representing enterprise knowledge without overloading model attention mechanisms.
- [27:39](https://www.youtube.com/watch?v=gijVIvSmgoA&t=1659s) To solve the code review bottleneck caused by overconfident LLM generation, AWS uses multi-agent critique setups with isolated critic personas and steering hooks.
- [30:14](https://www.youtube.com/watch?v=gijVIvSmgoA&t=1814s) Long-running research agents require token-efficient hierarchical execution, utilizing tree pruning to evaluate candidate hypotheses without running unguided parallel loops.
- [34:28](https://www.youtube.com/watch?v=gijVIvSmgoA&t=2068s) Production agent trust depends on both offline and online evals alongside persistent memory and continual learning, allowing agents to evolve from junior new hires to Principal Engineer equivalents.
- [38:34](https://www.youtube.com/watch?v=gijVIvSmgoA&t=2314s) Agent memory requires intentional forgetting and isolation between personal user preferences and shared team conventions, combining plain text Bash/grep operations with vector semantic search.
- [44:48](https://www.youtube.com/watch?v=gijVIvSmgoA&t=2688s) Instead of building single giant super-agents, complex workflows across operations and development are best addressed by specialized agents connected via explicit communication bridges.
- [55:31](https://www.youtube.com/watch?v=gijVIvSmgoA&t=3331s) As foundation model architectures evolve beyond transformers, hardware specialization is expected to shift toward broader, more flexible, and energy-efficient architectures.
## Detailed walkthrough
### [00:00](https://www.youtube.com/watch?v=gijVIvSmgoA&t=0s) From Speech Compression to Statistical ASR and Deep Learning
The conversation kicks off with Anoop Deoras introducing what he sees as the true next frontier in AI: moving from today's **reactive agents**—which simply sit and wait for a user to express intent—to **proactive agents** that continuously anticipate requirements and execute complex tasks autonomously.
Tracing his path back to where it all started in India, Anoop shares how his journey into AI originated in speech compression and coding rather than recognition. Working at a telecommunications firm that built its own hardware chips, his job was to design software algorithms tightly coupled with hardware to squeeze audio packets into the smallest possible payload. It was during this period that he became fascinated by the statistical mechanics and quasi-stationarity of speech. When applying to graduate school, he sought to continue researching speech coding but was instead admitted for speech recognition. He accepted the offer with a laugh, reasoning that at least "speech" was the common denominator.
Host Francois Richard relates to this timeline, noting his own early internship in speech recognition back in 1995 and pointing out how 2007 was a major turning point in the field. They recall the evolution of interactive voice response (IVR) systems—transitioning from rigid "press 1, press 2" options to early "call steering" applications (pioneered by AT&T Bell Labs) that allowed users to state their intent naturally.
During his PhD, Anoop tackled **Large Vocabulary Continuous Speech Recognition (LVCSR)**. Back then, models could not handle open-vocabulary tasks seamlessly; every single word required explicit acoustic and language modeling. Any unencountered word resulted in an "out-of-vocabulary" (UNK) token, presenting massive challenges around verbalization and scaling to 500,000-word vocabularies. Furthermore, these classic systems were heavily cascaded—stitching together isolated acoustic models, language models, and downstream NLP modules while hoping the full pipeline would work.
Everything changed with two major breakthroughs:
* **Joint Deep Learning Architectures:** Replacing cascaded modules (and traditional Hidden Markov Models / Gaussian Mixture Models) with end-to-end deep neural networks.
* **Subword/Tokenization:** Breaking text down into tokens/subwords (akin to an expanded alphabet), effectively solving the open-vocabulary problem and eliminating UNK failures.
This shift was a massive relief for researchers. Anoop recalls his PhD advisor—whom he affectionately honors as one of the fathers of statistical speech recognition—telling a persistent Anoop asking when he could graduate: *"Give me a 5% absolute improvement in speech recognition and I will approve your thesis."* Because progress was excruciatingly incremental for years, both speakers admit they contemplated leaving the field entirely until deep learning triggered an explosion in accuracy and real-world efficacy.
---
[06:36](https://www.youtube.com/watch?v=gijVIvSmgoA&t=396s) Speaking @Scale and AWS Agentic AI Strategy
Shifting to the present, Francois asks Anoop about his experience speaking at the @Scale conference and his day-to-day responsibilities leading applied science for AWS's **Agentic AI** business unit. Anoop expresses his gratitude for the conference's deeply technical focus, noting that it was a real honor to present alongside industry peers like Boris and address real-world technical pain points rather than sales pitches.
At AWS, Anoop's organization operates across two primary layers:
1. **Primitive Infrastructure Layer:** Building the "undifferentiated heavy lifting" substrate for autonomous agents. This layer is application- and LLM-agnostic, covering essential primitives such as:
* **Agentic Memory:** Long-term memory, state consolidation, and selective forgetting.
* **Observability & Agentic Evals:** Moving beyond simple LLM benchmark evals to evaluate multi-step, tool-using autonomous workflows.
* **Customization:** Enabling enterprise teams to tailor agent behavior to internal conventions.
2. **Frontier Agent Applications:** Developing autonomous, long-running, long-horizon agents with high agency across two primary domains:
* **Enterprise & Business Productivity:** Automating daily professional workflows across tools like Outlook, Slack, and internal communications.
* **Software Development Life Cycle (SDLC):** Reimagining software engineering end-to-end using generative and agentic AI.
As an applied science leader, Anoop describes his daily routine as a balance of hands-on science experimentation, hypothesis testing, and rigorous science reviews. Following Amazon's core leadership principle of "Working Backwards," his team focuses strictly on solving real developer friction points rather than conducting isolated academic research, ensuring their scientific breakthroughs align directly with broader product strategy.
---
[11:09](https://www.youtube.com/watch?v=gijVIvSmgoA&t=669s) The Amazon Java Migration and the Birth of AWS Transform
The conversation turns to product strategy in action, with Francois bringing up Amazon's massive internal code migrations and asking how agentic AI came to power them. Anoop takes a trip back to 2023—before "agentic AI" was even a coined term—when his team set out to push the limits of LLMs beyond simple conversational chatbots.
To stress-test LLM capabilities, AWS leadership posed a challenge: could AI tackle the complex, high-friction maintenance tasks that human developers dread? While developers want to spend their time building exciting new features, critical hygiene tasks like writing unit tests, maintaining codebases, and updating language versions often fall by the wayside, creating massive technical debt that slows down entire organizations.
AWS picked a massive internal target: migrating Amazon's legacy codebase across **30,000+ Java applications** to modern Java versions. Because the task had well-defined input and output characteristics (an old Java codebase transforming into a functional, modernized equivalent), the team could take a workflow-centric approach with embedded constraints and automated equivalence testing.
Key details and achievements from the migration project include:
* **Scope:** Upgraded over 30,000 internal Amazon applications from legacy Java versions to modern releases.
* **Equivalence Verification:** Beyond merely rewriting syntax, the system validated functional equivalence between the old and modernized applications.
* **Evolution into AWS Transform:** What began as an internal experiment evolved into **AWS Transform**, a standalone AWS service that now addresses far broader modernization challenges:
* **Version-to-Version Upgrades:** Modernizing language versions within the same stack (e.g., legacy Java to current LTS Java).
* **Language-to-Language Translation:** Porting legacy codebases from one language to another.
* **Mainframe & Monolith Modernization:** Deconstructing legacy COBOL applications running on mainframes and rewriting them into microservices-based architectures on hyper-scalers using modern languages like Rust.
Anoop emphasizes that while agents cannot magically solve complex migrations at the single click of a button, taking care of 80% to 90% of the heavy lifting slashes multi-man-month migration projects down to manageable tasks, drastically reducing human developer toil. As the interview segment wraps up, Francois pivots the conversation toward Anoop's prior history before AWS, asking about his work in sequence modeling and recommendation systems during his tenure at Netflix.
[17:05](https://www.youtube.com/watch?v=gijVIvSmgoA&t=1025s) Personalization as a Language Modeling Problem
Anoop Deoras reflects on the underlying thread connecting his career trajectory, observing that despite moving between different companies and domain problems, the core substrate across all his work has consistently been language modeling. He notes that while a layman might view his background as fragmented, it was an intentional progression. After doing his PhD in language models, Microsoft provided his first opportunity to apply these concepts to early prototypes of personal assistants. He then transitioned to Netflix, where his team completely reimagined the recommendation engine by treating movie and show recommendations not through traditional matrix factorization or linear models, but directly as a large language modeling and sequence modeling problem.
The rationale behind Netflix's pivot stemmed from the purity of user event streams. Unlike noisier datasets, a user's viewing history offers a dense, high-signal sequence of behavioral intent. Deoras highlights two fundamental takeaways from his time at Microsoft and Netflix: first, sequence modeling forms the core technical stack for modern AI applications; second, solving personalization is an existential requirement for any user-facing software. If an application fails to track who the user is and what they care about across sessions, it forces the user to start from scratch every single time they log in.
- **Sequence modeling as a substrate:** Reinterpreted recommendation engines by moving away from legacy matrix factorization toward sequence-based language models.
- **High-signal event streams:** Leveraged pristine user event data on Netflix to predict future viewing behavior with minimal ambient noise.
- **Session continuity:** Emphasized that failing to maintain persistent user context forces systems to rebuild context on every single login.
---
[19:55](https://www.youtube.com/watch?v=gijVIvSmgoA&t=1195s) Inside Kiro: AWS's Coding Agent and Feature Scenarios
Transitioning back to AWS's current portfolio, Deoras introduces **Kiro**, AWS's flagship coding agent designed to operate both as an inline developer companion and as an autonomous, long-running agent. The core vision for Kiro is to serve as a deeply trusted co-worker to whom engineers can confidently delegate heavy workloads with "eyes closed," knowing the agent will deliver correct results. Routine maintenance tasks—such as fixing open bugs, writing unit tests, updating documentation, code refactoring, and general repository upkeep—represent the most immediate, low-hanging delegation use cases.
```
┌───────────────────────────────────────────────────────────┐
│ Kiro Deployment Modes │
├──────────────────────────────┬────────────────────────────┤
│ Brownfield Development │ Greenfield Development │
│ • Reads existing legacy code │ • Builds systems from zero │
│ • Identifies project patterns│ • Lacks existing codebase │
│ • Preserves team conventions │ • Must enforce team policy │
└──────────────────────────────┴────────────────────────────┘
```
The real engineering friction, however, occurs when delegating full feature development, which Deoras splits into two distinct operational paradigms:
- **Brownfield Development:** Building new capabilities within massive existing codebases. The agent must gather context, analyze historical patterns, and adhere to established architectural conventions without unnecessarily reinventing wheels.
- **Greenfield Development:** Constructing net-new applications from scratch. Lacking local code to copy from, the agent must draw inspiration from surrounding organizational artifacts and strictly align with broader team policies from day one.
---
[22:44](https://www.youtube.com/watch?v=gijVIvSmgoA&t=1364s) Context Engineering vs. Explicit Rule Teaching
When host Francois Richard asks whether agents need to be explicitly taught company policies or if they absorb them, Deoras cautions against brute-force instruction. Rather than explicit rule-teaching, AWS focuses heavily on **context engineering**. Modern foundation models operate under finite context budgets (typically $200,000$ to $300,000$ tokens), and even models claiming million-token context windows suffer from degraded attention mechanisms across ultra-long prompts. Consequently, the challenge lies in succinctly representing an enterprise's years of accumulated engineering knowledge within a strictly bounded prompt budget.
Francois shares a parallel challenge from Meta, where widespread adoption of internal coding agents created subtle architectural drift. Meta relies on strict, standardized conventions—such as single canonical implementations for UI components like buttons or reaction pickers—so that any engineer can immediately understand any codebase. When coding agents pulled external patterns from public open-source training data and inserted them into internal Meta repositories, everything worked fine initially. However, during later debugging sessions, engineers discovered unexpected patterns that violated internal standards, forcing teams to roll back the changes and explicitly re-guide the agent.
> "The model's capability to pay attention on every token is still limited beyond a point. So, you need to be really careful in what you are putting in that 200, 300,000 token budget."
- **Parametric vs. In-Context Knowledge:** 99% of an LLM's knowledge sits frozen in its parameters from public training data, making pure in-context learning feel fragile when overriding defaults.
- **Steering Hooks:** Implementation of automated verification checks that force the agent to consult senior engineering standards or skill lists prior to committing code.
- **Junior Developer Analogy:** Treating agent supervision like onboarding a human junior developer: requiring automated sign-offs from virtual "principal engineers" before shipping to production.
---
[27:39](https://www.youtube.com/watch?v=gijVIvSmgoA&t=1659s) Addressing the Code Review Crisis with Multi-Agent Critique
As code generation becomes effortless, the central bottleneck in software engineering has shifted dramatically from *writing* code to *reviewing* it. Deoras explains that because LLMs express extreme confidence in their own outputs, a single-pass model will almost never critique its own logic mid-generation. To combat this, AWS is experimenting with multi-agent critique topologies. By initializing a completely separate instance of the same underlying LLM—primed explicitly with a critic persona and detached from the authoring context—the reviewer model can objectively evaluate diffs for correctness and adherence to organizational norms.
However, Deoras emphasizes that simply showing a code diff to a frozen LLM and expecting zero-shot reasoning is insufficient for enterprise environments. Reviewers must feed historical context—showing what successful code reviews and team norms looked like in past pull requests—into the model's review flow. While expanding context windows may allow teams to stick with frozen foundation models operating purely at the context layer, deep parametric adaptation or specialized fine-tuning may ultimately be required to truly master enterprise-grade code review at scale.
- **Persona Isolation:** Utilizing a fresh, un-biased LLM instance specifically primed as a critic to uncover flaws hidden by the generation model's overconfidence.
- **Pattern Encoding:** Injecting explicit historical examples of high-quality pull requests and review comments to set baseline standards.
- **Shift in Human Effort:** Recognizing that human engineering bandwidth is increasingly consumed by verifying AI-generated diffs rather than typing syntax.
---
[30:14](https://www.youtube.com/watch?v=gijVIvSmgoA&t=1814s) Deep Research Agents and Token-Efficient Autonomy
Looking toward the future of agentic autonomy, Deoras shares a personal weekend project to illustrate the power of long-running, proactive agents. As his organizational leadership responsibilities grew at AWS, he found himself lacking time to manually investigate backlogged research hypotheses. To solve this, he used Kudo to construct autonomous "deep research agents." These agents independently crawl external repositories, analyze similar codebases, extract architectural patterns, and run background experiments on his hypotheses over extended periods—delivering consolidated reports that Deoras can review and iterate on during his free time.
To prevent long-running agents from burning through massive token budgets during exploration, Deoras advocates for structured, hierarchical execution rather than naive brute-force loops:
- **Avoid Mindless Loops:** Running 100 parallel unguided experiments is economically impractical and token-inefficient.
- **Incremental Tree Pruning:** Agents should start with a small pool of candidate ideas, quickly evaluate early signals, prune unpromising branches, and spawn targeted sub-tasks only for winning approaches.
- **Safety in Non-Production:** High-autonomy background tasks (like research, analysis, and feature prototyping) carry low risk of breaking production systems while offering massive productivity leverage.
- **Guardrails & Recovery:** Long-running agents require safety mechanisms to detect drift, restore state, or reboot entirely when trajectories go off-course.
<ElicitationsGroup message="Explore related topics from this interview:">
<Elicitation label="Discuss memory mechanisms in autonomous agents" query="How do modern agentic systems handle persistent memory, forgetting, and context consolidation over long time horizons based on Anoop Deoras's strategy?"/>
<Elicitation label="Examine the evolution of AWS Transform and Java migration" query="Tell me about the story of AWS Transform and how Amazon's internal migration of 30,000+ Java applications seeded their agent strategy."/>
</ElicitationsGroup>
[34:28](https://www.youtube.com/watch?v=gijVIvSmgoA&t=2068s) Observability, Evaluation, and Trust in Autonomous Agents
When setting up guardrails for production agents, Anoop Deoras highlights observability and evaluation as non-negotiable fundamentals. Even for non-autonomous tasks running for just a few minutes, strong governance, offline (build-time) evals, and online (production) evals are mandatory. The core challenge of online evals lies in maintaining periodic checks to ensure the agent's actions still respect the user's original intent without bothering the user constantly. For deterministic problems with known input-output spaces, engineers can pre-bake verification directly into the system. Over-relying on human confirmation backfires because users will simply spam "yes" to every prompt without thinking.
Building real trust in autonomous agents comes down to whether they possess continual learning—learning from past mistakes rather than repeating them. Deoras draws an "uncanny" analogy to a fresh college graduate joining a engineering team: initially, the new hire lacks context and tribal knowledge, but as they learn from failures and feedback, they mature into a trusted Principal Engineer who can handle complex operations independently. Memory and continual learning are two sides of the same coin; without them, long-running autonomous systems cannot achieve the level of reliability required for broad adoption.
* **Production Guardrails:** Requires two distinct eval forms—offline (build-time) and online (production runtime).
* **Human Friction:** Excessive human-in-the-loop prompts cause user fatigue, leading people to blindly "click yes yes yes" without checking correctness.
* **The Fresh Grad Analogy:** Junior engineers gain organizational tribal knowledge over time to become trusted Principal Engineers; AI agents must follow this exact evolutionary path through persistent memory.
---
[38:34](https://www.youtube.com/watch?v=gijVIvSmgoA&t=2314s) Memory Consolidation, Forgetting, and Text vs. Vector Search
Managing agent memory presents a tricky challenge when underlying system topologies evolve. As host François Richard notes, a rule or dependency learned one week might change the next week after a system upgrade. Deoras explains that memory consolidation requires agents not only to store knowledge, but also to intentionally *forget* outdated information. Furthermore, agents must differentiate between personal preferences (user-scoped) and team-wide conventions (team-scoped) so niche habits do not corrupt shared workflows.
```
+-------------------------------------------------------------------------+
| MEMORY SEARCH STRATEGIES |
+------------------------------------+------------------------------------+
| Bash & Text Matching (e.g. grep) | Vector / Semantic Search |
+------------------------------------+------------------------------------+
| • Deterministic and transparent | • Operates directly in vector space|
| • Pairs naturally with LLM tool-use| • Finds semantically similar terms |
| • LLM reformulates queries dynamically| • Built as primitive (e.g., Agent Core)|
+------------------------------------+------------------------------------+
```
Addressing how memory is stored, Deoras pushes back against the assumption that using standard Markdown (`.md`) files or raw text files is "primitive." Because modern LLMs excel at executing Bash tools—which natively handle text parsing and standard Unix commands—text-based files offer immense practical value. While `grep` performs exact string matching, an LLM dynamically reformulates search queries to achieve broad coverage. However, raw text matching is not a complete replacement for semantic vector search; AWS implements primitives like "Agent Core Memory" to map information directly into vector space, using both deterministic text execution and semantic search in tandem.
* **Memory Scope:** System memories must be cleanly isolated between individual user preferences and broad team-wide standards.
* **The Power of Bash:** Markdown and plain text remain state-of-the-art storage formats for agents because LLMs use command-line Bash tools effortlessly.
* **Hybrid Search:** Combining LLM-driven query expansion over `grep` alongside vector-based semantic retrieval yields optimal memory performance.
---
[42:14](https://www.youtube.com/watch?v=gijVIvSmgoA&t=2534s) Tool Discovery, Overwhelm, and Agent Personalization
When users set up development environments, a common strategy is pre-loading coding agents with standard utilities like Python runtime environments, PDF readers, or Google Docs connectors to establish predictable behavior. François shares how Meta engineers use their internal coding assistant, "Meta Code" (built on Codex), to supply tools upfront so the model defaults to established patterns rather than inventing arbitrary solutions.
```
[ Tool Selection ]
│
┌────────────────────────────────┴────────────────────────────────┐
▼ ▼
[ Specialized / Niche Tools ] [ Shared / Global Tools ]
• Tailored to personal workflows • Base environment utilities
• Learned per-user preferences • Standardized team scripts
│ │
└────────────────────────────────┬────────────────────────────────┘
▼
[ Personalized Filtering ]
(Weeds out noise from signal)
```
However, dumping too many tools into an agent's context window can cause "tool overwhelm," causing performance to degrade. Furthermore, forcing users to repeatedly re-declare tool preferences every time they switch devices—such as moving from a desktop to a mobile environment—creates unnecessary friction. Deoras highlights that agents must automatically learn user-level personalization: knowing "who François is, who Anoop is" and discovering which specific tools work best for individual engineers on specific tasks, effectively sifting out the noise from the signal.
* **Meta Code Practice:** Engineers pre-install standard toolchains inside environment containers to enforce convention over novel, unproven code execution paths.
* **Tool Overwhelm:** Overloading an agent's context window with dozens of unused capabilities degrades decision-making accuracy.
* **Personalized Tooling:** The ideal agent dynamically pairs specific tools with specific users based on historical efficiency, analogous to "personalized medicine."
---
[44:48](https://www.youtube.com/watch?v=gijVIvSmgoA&t=2688s) Specialized Multi-Agent Networks vs. Unified Super Agents
Engineering organizations currently employ distinct specialized agents across the software delivery lifecycle. François details Meta's internal breakdown:
* **Coding Agent:** Focuses on feature creation and code generation.
* **Ops Agent ("OpsMate"):** Handles system investigation, anomaly detection, and incident remediation.
* **Release Agent:** Manages software deployments and release pipelines.
* **Data Agent:** Formulates queries and interprets telemetry logging formats.
This split leads to a fundamental architectural debate: should organizations maintain fine-tuned, specialized agents to optimize token usage and latency, or consolidate everything into a single monolithic agent?
```
┌────────────────────────────────────────────────────────────────────────┐
│ ARCHITECTURAL APPROACHES │
├───────────────────────────────────┬────────────────────────────────────┤
│ Specialized Multi-Agent Mesh │ Unified "Super Agent" │
├───────────────────────────────────┼────────────────────────────────────┤
│ • Ops, Coding, and Release split │ • Single entry point with full context│
│ • Lower token usage per task │ • Prevents information loss in handoffs│
│ • Requires explicit bridges │ • Harder to train and balance │
└───────────────────────────────────┴────────────────────────────────────┘
```
Deoras uses the structure of human software engineering teams to resolve this tension. When diagnosing an incident, an engineer does not treat operations and codebase modifications as disjointed domains; they trace the system anomaly straight back to the underlying code. Therefore, an Ops Agent must directly communicate with a Coding Agent to convert log analysis into a functional code patch and proactively mitigate future failures. While building a single "giant super agent" is technically difficult today, the sweet spot involves building intentional communication bridges between tightly coupled domains rather than letting agents run in total isolation.
* **Communication Bridges:** Specialized agents must share contextual state; an operational incident report should automatically generate a pull request in the coding agent.
* **Proactive Patching:** Next-generation operational tools must move beyond basic log analysis to generate root-cause code fixes automatically.
---
[48:37](https://www.youtube.com/watch?v=gijVIvSmgoA&t=2917s) The Shift to Proactive Agents and Weekend AI Experiments
Looking toward the frontier of AI research, Deoras argues that today's agentic systems remain fundamentally reactive—they wait for a human to explicitly state an intent before taking action. The next evolutionary leap in the coming months will be *proactivity*, where agents anticipate needs, formulate hypotheses, and execute workflows independently within broad user-defined boundaries. Using the research space as an example, instead of waiting for a scientist to provide a research hypothesis, a proactive AI agent will autonomously scan language modeling literature, propose new scientific hypotheses, and gather supporting empirical evidence on its own.
Closing out the discussion with a lightning-round personal anecdote, Deoras shares how AI tools have transformed his home life with his son. Wanting to give his son opportunities he missed out on as a child, Deoras uses AI coding assistants over the weekend to rapidly build complex, customized applications combining robotics, Lego construction, and digital painting. These tools allow him to go from an idea to a fully functional, personalized project in a single weekend, bridging technical engineering with family bonding.
* **The Next Frontier:** Agents transitioning from reactive execution to proactive hypothesis generation and autonomous exploration.
* **Proactive Research:** Future AI research agents will monitor literature domains to suggest novel scientific features without step-by-step human prompts.
* **Weekend Hacking:** Deoras uses AI tools to build custom robotics and art software applications with his young son over a single weekend.
[51:49](https://www.youtube.com/watch?v=gijVIvSmgoA&t=3109s) Go-To AI Tools and Personal Tech Preferences
Francois Richard asks Anoop Deoras about his personal tech setup and whether he builds hardware projects using specific Lego kits or Arduino pieces with his son. Anoop explains that they stick to an approved kit provided by his son's school, though he admits to blanking on the exact name of the kit at the moment. Shifting to software, Anoop touches on his preferred AI tools, stating that he uses virtually all available tools on the market. He grounds this approach in two reasons: first, he views external tools as fundamentally complementary, each bringing distinct strengths and nuances; second, as someone who builds coding agents for AWS customers, using products like Perplexity provides crucial inspiration by revealing what resonates with end users. Francois chips in to praise Perplexity's computer capabilities as "impressive," a sentiment Anoop enthusiastically echoes.
Francois then pivots to a classic developer debate, asking Anoop to choose between Vi and Emacs for his "first love." Anoop laughs and declares himself "Emacs all the way." He shares that while studying in India, he did not use either editor, but after moving to the U.S. for his PhD, he adopted Emacs because his lab mates were using it. He customized and tailored the editor to fit his personal productivity workflow, cementing a lifelong devotion.
- Anoop and his son utilize a school-approved robotics/building kit for home projects.
- Anoop regularly uses a broad array of AI tools, including Perplexity, to stay informed as a developer of agentic systems.
- When choosing between Vim/Vi and Emacs, Anoop remains loyal to Emacs, having adopted and customized it during his PhD studies in the United States.
---
[53:21](https://www.youtube.com/watch?v=gijVIvSmgoA&t=3201s) Advice for Prospective PhD Students and the Future of AI Research
When asked what advice he would give to a student considering a PhD in today's rapidly changing AI landscape, Anoop jokes that he feels lucky his son is only in sixth grade, sparing him from having to navigate those decisions for another six or seven years. Transporting his mindset back to his own undergraduate days, Anoop notes that if he had to choose a direction now, he might dive into quantum computing, calling it a field where significant excitement and investment are currently concentrated.
Addressing core LLM research, Anoop emphasizes that model architecture moves in cycles—moving from transformers today to potentially new paradigms tomorrow. While research in foundational models remains essential, he cautions aspiring researchers about the crucial role of funding. Because scalable research and stress-testing hypotheses require immense compute resources, securing reliable funding sources is a vital factor for anyone entering the field.
- Anoop highlights quantum computing as a promising frontier for incoming PhD students due to strong momentum and capital investment.
- He emphasizes that modern model architectures evolve in cycles beyond the transformer.
- He advises new researchers to carefully evaluate funding sources, as access to compute is critical for testing hypotheses at scale.
---
[55:01](https://www.youtube.com/watch?v=gijVIvSmgoA&t=3301s) Hardware Convergence, Architectural Shifts, and Brain Efficiency
Francois draws a historical parallel to the early days of speech recognition at companies like Meta, Yahoo, and Facebook, recalling how systems relied on specialized custom chips for audio and signal processing alongside standard CPUs for orchestration. Over time, as CPUs became sufficiently powerful (such as during the Pentium era), those dedicated chips vanished into unified processors. Francois asks if today's split between specialized training GPUs, inference GPUs, and application CPUs will eventually consolidate in a similar fashion.
Anoop finds the comparison compelling, noting that today's hardware and software stacks are heavily indexed on the specific requirements of current foundation model architectures. However, he expects newer, more efficient architectures to emerge, challenging existing hardware assumptions. Pointing out that current approaches prioritize scaling data and compute over efficiency, Anoop contrasts current AI with the human brain, which is far more effective and energy-efficient. As new algorithmic variants develop, he predicts the industry will naturally converge toward versatile hardware capable of supporting diverse architectures rather than being tied to a single paradigm.
- Early speech recognition transitioned from specialized DSP/audio chips to general-purpose CPUs during the Pentium era.
- Current AI deployments remain fragmented across specialized training GPUs, inference GPUs, and host CPUs.
- Anoop anticipates future architectural shifts inspired by efficiency that will push hardware toward broader flexibility across multiple model paradigms.
Francois concludes the conversation by thanking Anoop for his time, generosity, and insights regarding both his professional career in agentic AI and his personal journey. Anoop expresses his gratitude in return, noting that it was a pleasure chatting with Francois to wrap up the interview.
## Notable quotes
> [00:00](https://www.youtube.com/watch?v=gijVIvSmgoA&t=0s) Give me a 5% absolute improvement in speech recognition and I will approve your thesis. — Anoop Deoras's PhD Advisor
> [22:44](https://www.youtube.com/watch?v=gijVIvSmgoA&t=1364s) The model's capability to pay attention on every token is still limited beyond a point. So, you need to be really careful in what you are putting in that 200, 300,000 token budget. — Anoop Deoras
Anoop Deoras, applied science leader for AWS's Agentic AI organization, argues that the next frontier in artificial intelligence requires shifting from reactive chat interfaces to proactive, autonomous agents. Drawing on operational lessons from large-scale code modernization at Amazon, he outlines an agent strategy centered on context engineering, persistent memory consolidation, and specialized multi-agent coordination.
Pragmatic Modernization and SDLC Automation (6:36)
AWS structures its agentic AI strategy across two layers: an agnostic primitive infrastructure layer managing memory, observability, and evaluations, and an application layer focused on enterprise workflows and the software development life cycle. Rather than pursuing purely theoretical research, AWS refined its autonomous tooling by tackling maintenance tasks that developers resist, beginning with an internal migration of more than 30,000 legacy Java applications. This initiative required automated functional equivalence testing and subsequently evolved into AWS Transform, an external service for language-to-language translation and mainframe COBOL refactoring into cloud-native microservices.
This workflow-driven foundation informs AWS's coding agent, Kiro, which operates across two software development environments: brownfield and greenfield. In brownfield codebases, agents must gather historical context and adhere to established architectural patterns without rewriting existing functionality. In greenfield projects, where local reference code is absent, agents rely on surrounding organizational artifacts and strict policy enforcement from inception.
Context Engineering and Multi-Agent Code Review (22:44)
Deoras cautions against brute-force instruction, arguing that foundation models suffer from degraded attention mechanisms across prompts exceeding 200,000 to 300,000 tokens. Because the vast majority of an LLM's knowledge resides frozen in its parameters, agents frequently default to public open-source patterns that violate internal enterprise conventions. To mitigate this drift, AWS emphasizes context engineering and automated steering hooks that verify code against senior engineering standards prior to commit.
With code generation increasingly automated, Deoras identifies code review as the primary industry bottleneck. Single-pass LLMs suffer from overconfidence and rarely critique their own generation mid-stream. AWS addresses this by implementing multi-agent critique topologies, where a distinct LLM instance is isolated with a critic persona and historical pull request context to objectively evaluate diffs for correctness and institutional compliance.
Memory Architecture, Continual Learning, and Evals (34:28)
Establishing trust in autonomous systems requires continual learning and disciplined guardrails, which Deoras compares to onboarding a junior developer who gradually gains tribal knowledge to operate as a principal engineer. AWS implements both offline build-time evaluations and online runtime checks. To prevent confirmation fatigue—where users blindly approve system prompts—online evaluations must verify intent programmatically rather than constantly interrupting human operators.
Memory management requires both recording context and intentionally forgetting obsolete dependencies as system topologies shift. AWS separates memory into individual user preferences and shared team standards. Structurally, Deoras defends plain text and Markdown files as viable formats because LLMs easily manipulate them through command-line Bash tools, while hybridizing these deterministic lookups with semantic vector search (such as Agent Core Memory) for comprehensive retrieval.
Specialized Mesh Topologies and Proactive Autonomy (44:48)
Addressing the tension between monolithic "super agents" and specialized micro-agents, Deoras advocates for connected meshes over isolated silos. Human engineering workflows link operational anomalies directly back to code changes; similarly, an operations agent performing incident analysis should pass context directly to a coding agent to generate an automated root-cause patch. Deoras warns against unguided, parallel exploratory loops, advising instead that long-running agents use hierarchical planning and tree pruning to discard weak hypotheses before consuming excessive tokens.
Looking forward, Deoras anticipates agents evolving from reactive systems that wait for user prompts to proactive agents that monitor literature, formulate independent hypotheses, and conduct background research. On the hardware front, he expects that future algorithmic breakthroughs focused on brain-like energy efficiency will eventually push the industry to consolidate today's fragmented training GPUs, inference chips, and host CPUs into versatile, unified hardware architectures.
While autonomous agents promise significant reductions in software maintenance toil, practical enterprise deployment remains bounded by context degradation, verification costs, and the technical challenge of building reliable continuous learning without introducing human review fatigue.
[0:00] Anoop Deoras's PhD Advisor: Give me a 5% absolute improvement in speech recognition and I will approve your thesis.
[22:44] Anoop Deoras: The model's capability to pay attention on every token is still limited beyond a point. So, you need to be really careful in what you are putting in that 200, 300,000 token budget.
[34:28] Anoop Deoras: click yes yes yes
[38:34] Anoop Deoras: grep
[38:34] Anoop Deoras: Agent Core Memory
[42:14] Francois Richard: Meta Code
[44:48] Francois Richard: OpsMate
[51:49] Anoop Deoras: Emacs all the way
Original transcript
[0:00] The agents that we are building today,
[0:01] regardless of the use case, they are
[0:04] still reactive. What I mean is they are
[0:07] looking for a user to express an intent
[0:10] and then they are achieving the task.
[0:13] The next frontier, in my opinion, is
[0:15] about proactivity. The agents are
[0:17] anticipating what is needed and then
[0:20] they're doing it by themselves.
[0:22] >> [music]
[0:24] >> So, what drew you to speech recognition
[0:26] in these years?
[0:27] >> Um well, in India, after my undergrad,
[0:31] was working on speech coding,
[0:33] uh not not recognition, speech
[0:34] compression.
[0:36] And virtue of that, I got fascinated by
[0:39] the statistical aspects of speech.
[0:42] Um and I was interested to do research.
[0:44] So, I was applying, um but I wasn't very
[0:47] focused on a particular field until
[0:49] speech coding just struck as a as a area
[0:52] that I want to dive deep into. But then
[0:54] I pivoted to do statistical
[0:56] >> Yeah.
[0:57] >> part of speech coding. Um but I did not
[0:59] get an admit on speech coding. I got
[1:02] speech recognition. I was like, "Well,
[1:04] okay. There is at least speech common."
[1:06] >> And what was the context for speech
[1:08] coding? What was the application of it
[1:10] in that at that time?
[1:11] >> Yeah, so the company that I was working
[1:13] for in India is a telecommunication
[1:15] firm. Um and so, just sending the audio
[1:17] packets in the most compressed way was
[1:20] what they were working on. And they were
[1:22] building their own chips. Uh so, having
[1:25] um software stack that is very close to
[1:26] the hardware and building the algorithms
[1:29] that are tuned to your hardware was the
[1:31] essence of the work. So, was primarily
[1:34] doing speech coding, audio coding, image
[1:36] coding as well. But speech was the is a
[1:39] big part of it. Um and I thoroughly
[1:42] enjoyed working on the
[1:43] quasi-stationarity of speech, which
[1:45] really is the basis of how statistical
[1:47] speech recognition works. Now, you know,
[1:50] >> Now it's established. Yeah.
[1:53] I was looking a bit at your background
[1:55] and yeah, in these years, like I've
[1:57] started to work in speech recognition,
[1:59] like my first internship.
[2:00] >> Uh-huh.
[2:01] >> Since 1995.
[2:02] >> Uh.
[2:03] >> Um.
[2:04] And then I
[2:05] when I was reviewing a bit about about
[2:07] your background, I kind of realized it,
[2:08] yeah, you kind of came in 2007
[2:12] in this area, where like, yeah, it's
[2:14] this is basically when the switch
[2:15] happened.
[2:16] >> Yeah.
[2:17] >> When we went from like, what I call like
[2:18] the previous like classic
[2:20] >> Yeah.
[2:20] >> when it was like, you know, a little bit
[2:22] of like single word recognition plus NLP
[2:25] apply after the fact
[2:26] >> Yeah.
[2:27] >> to like the statistical modeling.
[2:29] >> Yeah. Yeah.
[2:30] >> In these years, I remember doing like
[2:32] the first like, you know, we used to
[2:34] call them like call steering apps.
[2:36] >> Mhm.
[2:36] >> Where instead of having a very directed
[2:39] dialogue, where like, okay, you know,
[2:42] say XYZ for this and say this for that.
[2:45] >> Mhm.
[2:45] >> Like which was kind of an evolution of
[2:46] press one and press two.
[2:48] It was like, I'm going to help you.
[2:50] >> Yeah. Yeah.
[2:50] >> Like there's the first few years like
[2:52] around that time.
[2:53] >> I think the AT&T Bell Labs were
[2:56] pioneering because they were into call
[2:57] routing business. So, they wanted to
[2:59] solve this problem. And
[3:01] during my PhD, the challenge was how to
[3:05] scale the speech recognition's
[3:06] vocabulary.
[3:07] Um so, particularly the field that I was
[3:10] working on was called large vocabulary
[3:12] speech recognition systems. Um so, the
[3:15] thing right now everybody talks tokens.
[3:17] >> Yeah.
[3:18] >> And they don't appreciate that tokens
[3:20] have solved the open vocabulary problem.
[3:23] But back then, every word needed to be
[3:25] modeled explicitly.
[3:26] >> Yes.
[3:27] >> And anytime you encounter a word that
[3:29] you have not seen in the training data,
[3:31] you have unk.
[3:32] >> Yeah.
[3:33] >> Um and how do you verbalize an unk? So,
[3:35] all sorts of glaring problems that we
[3:37] used to have. So, the challenge was can
[3:39] I have very large vocabulary systems?
[3:41] And we are talking 500,000
[3:44] um vocabulary systems and all. So, the
[3:46] language modeling of that, the auditory
[3:48] modeling of that were the real
[3:50] challenges. And they were I think at at
[3:52] that time it was still a cascaded
[3:54] system. You build an acoustic model in
[3:56] isolation. You build a language model in
[3:58] isolation.
[3:59] And then, you stitch them together
[4:01] hoping everything just works. For
[4:03] machine translation and spoken language
[4:05] understanding, it was yet another
[4:07] modules getting stitched together.
[4:09] I think with deep learning, what
[4:11] revolutionized was that our advent into
[4:14] joint modeling, like one model that does
[4:16] everything. And I think that has changed
[4:18] the game. The next big revolution that
[4:21] has really solved the problem of speech
[4:23] recognition is the discovery of sub
[4:26] words or tokens. So, when you have
[4:28] tokens, you can form words. It's like
[4:30] working with the alphabets, right? Um
[4:34] and uh yeah, these are some of the
[4:35] things that uh have really changed the
[4:37] landscape of how we do text translation,
[4:41] how do we do speech recognition, and
[4:42] >> Yeah, I used to work with There was a
[4:44] concept called phonemes and trigram and
[4:47] >> Yeah, we we still we still have. It's
[4:49] just that not being modeled
[4:51] >> Yeah, it's not being modeled with
[4:52] Gaussian mixture models anymore.
[4:54] >> It it's all deep learning. Uh there were
[4:56] hidden Markov models, Gaussian mixture
[4:58] models that used to model um how, you
[5:01] know, phonemes are verbalized. Now,
[5:03] nowadays, because it's one giant model,
[5:06] uh deep learning all the way. Um it's uh
[5:09] very fascinating that we have been able
[5:11] to do that. And it shows in the
[5:13] products.
[5:14] >> It shows. Like, you know, one of the
[5:15] reason I got kind of frustrated at that
[5:18] time and it's kind of switch from, you
[5:20] know, speech recognition to kind of like
[5:22] go into more of the search area.
[5:24] >> Uh-huh.
[5:25] >> I felt like we're not making any more
[5:27] progress. It was like, "Oh, speech
[5:29] recognition is always like buggy. Like,
[5:31] it never recognize properly." And to a
[5:33] point like you kind of like, "Okay,
[5:35] let's let's
[5:36] move on into something else."
[5:37] >> Yeah. Yeah. Yeah. No, no, precisely to
[5:39] your point. I remember my adviser
[5:41] um he's he's no more.
[5:44] Um but he's the father of statistical
[5:45] speech recognition.
[5:47] And [clears throat] he told me, I used
[5:49] to pester him, I used to ask him, "When
[5:50] can I graduate? When can I graduate?" He
[5:52] says He said, "Give me 5% absolute
[5:55] improvement in speech [laughter]
[5:57] >> 5%
[5:58] >> I will approve your thesis." Uh so, yes,
[6:01] it was very incremental for a very long
[6:03] time.
[6:03] >> long time and then suddenly it exploded.
[6:06] >> in my head, too, I had numerous times
[6:08] the thought of just making a transition
[6:11] over to a different field where more
[6:14] gains can be had. Um but deep learning,
[6:17] I think, um what is what we really
[6:20] needed to make that step change in
[6:23] accuracies and and efficacies of the
[6:25] system.
[6:26] >> Awesome. So, so now you're transitioning
[6:29] to this podcast. You're transitioning
[6:31] from the ad scale conference that we had
[6:33] like just downstairs from from here.
[6:36] >> Um how did the talk go? Like
[6:39] >> I enjoyed the talk very much. So, over
[6:41] the past few years, I have watched
[6:43] numerous presentations and talks
[6:46] that happened at scale. Um so, it was a
[6:49] real pleasure and an honor, I would say,
[6:51] to be here talking about some of the
[6:53] pain points that uh we are going through
[6:55] and some of the solutions that allow us
[6:57] to close these pain points. So,
[6:59] connecting with the audience, uh meeting
[7:01] the other speakers. I heard Boris was
[7:03] >> Yeah. Yeah.
[7:04] >> in the panel. Uh such a fascina-
[7:05] fascinating series of talks and
[7:07] presentations. So, I thoroughly enjoyed
[7:09] it. I continue to be enjoying the event
[7:11] here and looking forward to chatting
[7:13] with you as well.
[7:13] >> Yeah, I am totally like happy and
[7:16] satisfied cuz you know, I do push for
[7:19] that conference a lot and I'm happy that
[7:22] we are able to remain it like in a very
[7:24] technical so that we're talking about
[7:26] problem. We're not trying to sell stuff.
[7:28] So, this is like, you know, this is
[7:30] amazing like hearing people talk about
[7:32] their real issues and so on.
[7:34] >> Yes.
[7:34] >> Like you about real issues.
[7:35] >> Yes.
[7:36] >> Tell me more about what is your
[7:38] day-to-day job at at Amazon at AWS. I
[7:42] know you're working on something called
[7:44] Cairo. If you can explain it a bit for
[7:46] the audience what it is.
[7:47] >> Yes. Yes. Okay. So, I
[7:50] lead applied science for AWS.
[7:55] And the business unit that I'm part of
[7:57] is called Agentic AI. So, the the focus
[8:00] of this organization is to do two
[8:02] things. One is to lay down the core
[8:05] infrastructure or build set of
[8:07] primitives for agents.
[8:10] So, if you
[8:11] want to build your own Agentic
[8:13] experience, we want our customers to
[8:15] focus on what values to them. What is
[8:18] the value add for them? And so, the
[8:20] undifferentiated heavy lifting in the
[8:22] Agentic space,
[8:24] understanding what that is, and then
[8:25] building those core set of primitives.
[8:28] So, very concretely, things such as
[8:30] memory for agents or building
[8:32] observability or Agentic eval, which is
[8:35] very different from your LLM evals,
[8:38] customization of agents. And so, things
[8:40] that are LLM and application agnostic
[8:44] forms this substrate, primitive layer.
[8:47] So, we do that. And then on top of this
[8:49] substrate, we are also building frontier
[8:52] agents.
[8:53] Our definition of frontier agent is the
[8:55] agents that are autonomous, long
[8:58] running, long horizon.
[9:01] We are We want these agents to have more
[9:04] and more agency.
[9:06] And so, in terms of use cases, there are
[9:08] two large use cases that we are
[9:10] currently focusing on.
[9:12] One is improve the business
[9:13] productivity. So, your day-to-day
[9:17] activities such as connecting to your
[9:19] Outlook or Slack, and just improving the
[9:21] productivity of a professional. That's
[9:23] one area of focus. The other is software
[9:28] development life cycle. And that is
[9:30] where I and my teams come in. And what
[9:33] we are doing is we are looking at the
[9:34] entire SDLC, software development life
[9:37] cycle, and identifying areas where we
[9:40] can, for the right reason, transform and
[9:43] reimagine
[9:46] the components of SDLC through either
[9:48] generative AI or agentic AI. And it's a
[9:50] pretty big
[9:52] life cycle, and there are pockets and
[9:54] areas in the life cycle that are
[9:57] you know, they have rich spectrum in
[9:59] themselves.
[10:00] So, what I do, going back to your
[10:02] question on what I do on a daily basis,
[10:05] so I, because I have a background in
[10:09] doing LLM science, if you will,
[10:12] I do a lot of experimentation. And when
[10:15] I say I, I'm really talking on behalf of
[10:17] my team.
[10:18] We
[10:19] are a team of innovators. We come up
[10:21] with a lot of hypotheses, and experiment
[10:24] with those hypotheses. And we are true
[10:27] to the tenant of working backwards from
[10:29] customer. So, we don't want to do in
[10:31] vacuum research. We want to really work
[10:35] on the problems that will have
[10:36] meaningful impact for the customers. So,
[10:38] understanding what are the customers'
[10:39] pain point, what are some of the use
[10:41] cases that they truly want to solve, and
[10:44] then work backwards from that and create
[10:46] very concrete science hypotheses. And
[10:49] you work on those. So, on a day-to-day
[10:51] basis, it's really science
[10:52] experimentation, science reviews, and
[10:55] then aligning with the product
[10:56] strategies.
[10:57] How does this map in the overall
[10:59] business
[11:00] vision, product product vision. So,
[11:03] that's what we do.
[11:05] And
[11:06] we have a lot of fun in doing what we
[11:08] do.
[11:09] >> Talking about like product strategy and
[11:11] aligning with what you have, like like
[11:13] in terms of offer.
[11:15] I heard that you guys in in in Amazon or
[11:17] AWS, you guys had to do like a lot of
[11:19] Java migration of some kind.
[11:22] >> Mhm.
[11:22] >> Can you explain a little bit like, you
[11:24] know, what was the migration that was,
[11:28] and then what sort of agent you use
[11:30] along the way? Like,
[11:32] you know, to give you a reference point,
[11:33] like at Meta, like Java is not something
[11:35] we use a lot.
[11:36] >> Uh-huh.
[11:37] >> Like it is rare. There's a couple of
[11:38] service and they're suffering.
[11:40] But I'm I'm really wondering about like
[11:42] you know, what happened now at at big
[11:44] scale like AWS.
[11:45] >> Yeah, yeah, yeah. Um so
[11:48] like back in 2023, when the whole
[11:52] agentic AI space was still in its
[11:54] nascent, um
[11:56] we wanted to stress test the
[11:58] capabilities of systems built on large
[12:01] language model. Um we knew LLMs are only
[12:03] going to become stronger and and more
[12:06] capable. But at that time, I don't think
[12:09] agentic AI as a term was coined.
[12:11] >> Yeah.
[12:13] >> Um
[12:13] >> It was more like early application type
[12:15] of thing.
[12:16] >> Yes, yes. And so
[12:18] when we saw the
[12:20] capabilities of the large language model
[12:22] to do
[12:24] you know, to build out chatbots and all,
[12:26] we
[12:27] gave a challenge to ourselves. We said,
[12:29] can we do bigger than what we are doing
[12:31] now? Are there complex tasks that your
[12:34] typical developer wouldn't want to do
[12:36] because
[12:37] they are just not you know, excited
[12:39] about doing that. Like maintenance of
[12:41] code is something that your developer
[12:42] doesn't want to do.
[12:44] Um writing unit tests is something that
[12:46] they would rather not do but focus on
[12:48] shiny feature development.
[12:49] >> Yeah.
[12:49] >> Um but these are the things if not done
[12:52] well, is going is holding the
[12:54] organization back.
[12:55] >> it slows you down.
[12:56] >> So our leadership gave us a challenge.
[12:58] They said,
[13:00] looking at the abilities of the LLM, can
[13:03] we build a system that can allow us to
[13:06] go from one version of Java to another
[13:08] and as a way of and by virtue of that,
[13:11] you are helping organizations to
[13:13] transform, modernize. Um at that time,
[13:16] the approach to solve this problem was
[13:19] um
[13:20] take workflow centric approach. What I
[13:23] mean is that
[13:25] um right now we are talking about a lot
[13:27] of open-ended scenarios, but back then
[13:29] because we were still new and early in
[13:31] this in this race in this game, um, we
[13:33] wanted to tackle the problems which had
[13:35] very concrete input and output
[13:37] characteristics. So, in this particular
[13:40] case, we know there is um, there is a
[13:43] code base that is in the older version
[13:45] of Java and then the the transform state
[13:49] is going to be the same code base but
[13:51] upgraded. So, in that regard, the input
[13:54] and output is kind of known. So, you can
[13:56] have some assumptions baked into your
[14:00] agent. Um, and I'm surprised that we did
[14:03] not actually call it agent agent. We
[14:05] still refer to it as capabilities of
[14:07] LLM. Um, and so to to test that, um, we
[14:13] look no further than our own backyard
[14:15] where we have so many organizations
[14:16] within Amazon who are still working with
[14:18] the older versions of Java. Going back
[14:20] to your point on Meta may not be using
[14:22] Java, I think it's less about a
[14:25] particular language. It's more about
[14:26] proving that you can do X to Y
[14:29] migration. And at Amazon, it just
[14:32] happened that we have a lot of code that
[14:35] is written in Java and we had a lot of
[14:37] opportunities to migrate or modernize.
[14:39] So, we picked that use case. It could
[14:41] have been any other use case. It just
[14:42] happened to be the most compelling use
[14:44] case for us back then. Um, and we did
[14:47] it. Um, like as you you know, know some
[14:51] of the public stats that we have shared,
[14:53] 30,000 applications
[14:55] >> Yeah, 2,000 and that was probably one of
[14:57] these big Java upgrade where you needed
[14:59] a lot of code change to to
[15:01] >> Yes. Yes, and it's it's not just, you
[15:05] know, going from X to Y and saying the
[15:07] job is done. You have to also show that
[15:09] it's equivalent.
[15:10] >> Yeah.
[15:11] >> Um, and that has been a very fascinating
[15:14] exercise um, because that led us to um,
[15:19] build very scalable services. So, today
[15:21] there is a service called AWS Transform
[15:23] Um, and I would think it's also part of
[15:26] this big giant SDLC,
[15:28] but AWS Transform is a world in itself
[15:32] because if you open that box of AWS
[15:34] Transform, you're talking
[15:36] within language version upgrades, which
[15:38] I just talked about. You're talking from
[15:41] one language to another.
[15:42] >> language to another.
[15:43] >> And you are talking
[15:45] system migration, system modernization.
[15:46] So, let's say there is
[15:48] there is an app written on monolith
[15:51] mainframes
[15:52] on COBOL
[15:54] you want to modernize get on to hyper
[15:56] scalers and with microservices and let's
[15:59] say you want to use Rust as the
[16:01] programming language.
[16:02] >> Yeah.
[16:02] >> Today, if you were to do this, it's many
[16:05] man month effort
[16:07] and we have this huge opportunity to
[16:10] apply agents.
[16:12] And now you cannot really hope that with
[16:14] a click of a button everything will just
[16:16] magically be done, but then there are
[16:18] portions where you can have guarantees
[16:21] that agent will be will take you 80 to
[16:24] 90% of the way. So, the human
[16:27] involvement
[16:28] significantly reduces. So, that 2023
[16:31] exercise that we did opened the doors
[16:33] for imagining much more, you know,
[16:36] broader set of problems and that has led
[16:39] to the building of this of this service
[16:41] AWS Transform. And
[16:43] yeah, we have been tackling some of the
[16:45] very hard problems in this in this
[16:47] world.
[16:48] >> And I want to go back a little bit more
[16:50] before we move more on the technical
[16:51] side. Your past at Netflix.
[16:54] >> Yeah.
[16:54] >> I was again like, you know, you you're
[16:56] solving a problem that I I totally
[16:58] appreciate. Like, you know, I want I
[16:59] want you to describe a bit what was your
[17:01] work back then because like what I
[17:03] believe you're you were doing is part of
[17:05] like figuring out what people were
[17:07] watching and then from that point of
[17:09] view generate kind of like the next type
[17:11] of suggestion which in my case, let's
[17:14] say I open Netflix like once or twice a
[17:16] week. I don't have the time to follow
[17:18] everything. So,
[17:20] I really rely on the suggestion to make
[17:22] something happen. So, so tell us more
[17:24] about about that type of work you were
[17:26] doing.
[17:26] >> Yeah, interestingly, when I look back, I
[17:29] mean it was an intentional choice also.
[17:31] But, if a layman were to look at my
[17:35] journey, they would find that the
[17:37] substrate or the commonality across all
[17:40] the things that I have worked on is
[17:42] language model. And I'll tell what I
[17:44] mean by that. So, of course my PhD was
[17:47] in language model and I was very
[17:48] interested in seeing the real world
[17:50] applications of how language model can
[17:52] be applied. Microsoft gave me that
[17:54] chance to work on the early prototypes
[17:56] of personal assistant.
[17:58] At Netflix, we reimagined doing
[18:00] personalization as a language modeling
[18:02] problem.
[18:04] And the status quo at that time was to
[18:06] build linear models of the user
[18:10] >> collection.
[18:11] >> basic linear models, matrix
[18:12] factorization and all.
[18:14] But, then we had the
[18:16] like we wanted to challenge the status
[18:18] quo. So, can we think of this as a large
[18:20] language modeling problem?
[18:22] Because the data was pure. What I mean
[18:24] by that is that is the user's event
[18:27] stream,
[18:28] which has very little noise. It's very
[18:30] rich in signal. So, you can do a lot of
[18:32] meaningful things.
[18:34] What I learned working on both the
[18:36] personal assistant at Microsoft and
[18:39] personalization problem at Netflix is
[18:41] that two things. One is the technology
[18:44] substrate is really the sequence
[18:46] modeling stack. And the second, you
[18:49] really need to get the personalization
[18:50] problem right. If you are in the
[18:52] business of building
[18:54] an application for a user, if that
[18:57] application doesn't understand who the
[18:58] user is,
[19:00] it's starting over every time. And this
[19:02] is exactly what I talked about in the
[19:04] morning presentation as well.
[19:06] Getting understanding user's intent, Um,
[19:10] both explicit and implicit, is a very
[19:13] big important problem that needs to be
[19:15] solved. Otherwise, every time you open a
[19:17] session,
[19:18] um, what is it you have to mention you
[19:19] have to start from scratch. So, um,
[19:22] for me, it's a it's a journey that I'm
[19:25] just continuing. The journey of
[19:27] personalization, the the journey of
[19:29] building large sequence models. And
[19:31] here, be it AWS transform or be it Kiro,
[19:34] I'm sorry I didn't answer your earlier
[19:37] question on what Kiro is and I'll get to
[19:39] that. It's it's really about building
[19:41] um, agentic experiences for developer,
[19:44] for the team, for the organization. And
[19:47] uh, personalization sits at the core of
[19:48] it. And the technological stack is
[19:50] really the sequence models, the large
[19:52] language, uh, models. Um, so that's
[19:55] that. Um, and going back to your earlier
[19:58] question on what Kiro is, Kiro is our
[20:00] coding agent.
[20:00] >> That's a coding agent. Okay, that's
[20:02] good.
[20:02] >> is our coding agent. And, um, we want,
[20:05] um,
[20:06] coding agent to be alongside the uh,
[20:10] developer. Um, at the same time, um,
[20:13] because it has shades of capabilities,
[20:15] um, we also want it to take on tasks
[20:18] that are long-running and autonomous.
[20:20] So, we are we are making Kiro
[20:23] trustworthy companion for the developer
[20:26] so that they can delegate the task, um,
[20:29] and have their eyes closed and have the
[20:30] trust in the agent and hope that it does
[20:33] the right thing.
[20:33] >> Give me give me an example of like a
[20:35] task that a developer will will actually
[20:37] delegate.
[20:39] >> There are tasks that are obvious tasks
[20:41] such as bug fixing.
[20:42] >> Okay.
[20:43] >> Uh, like if you have a long list of
[20:45] things that you want to tackle, um,
[20:47] you can point your Kiro agent to some of
[20:50] these open issues. You point the agent
[20:52] to,
[20:53] um, your repository and say these are
[20:55] some of the issues, go figure. That's
[20:57] one. Um, the other is, um, upkeep,
[21:00] maintenance, um, refactoring of the
[21:03] code, uh, writing documentation, unit
[21:05] tests. These are the obvious use cases.
[21:08] The challenging ones are feature
[21:09] development
[21:11] and there too you have two types of
[21:14] scenarios. One is the brown field
[21:16] scenario and the other is the green
[21:17] field. Brown field means there exists a
[21:20] legacy of features that have been built.
[21:23] There exists a code base. Now you are
[21:25] doing the incre
[21:26] for the lack of a better term.
[21:27] >> Yeah, yeah, you need to follow the
[21:28] pattern. It's already in the code, the
[21:30] convention, the data flows needs to like
[21:32] you know you not reinventing
[21:34] >> reinventing. Yes. So that's brown field.
[21:37] So the agent needs to be really good at
[21:39] understanding the context and then build
[21:42] the next set of feature. Feature
[21:43] building is a very hard problem
[21:45] because it requires a lot of ingenuity,
[21:48] a lot of you know deep analysis and all.
[21:51] So there you have the true organic need
[21:54] for the agent to go off and gather
[21:57] context and understand the system.
[21:59] And so you want to give agency to the to
[22:01] the agent so that it does its homework
[22:04] before it hypothesizes what to do and
[22:06] how to do it. For green field
[22:08] application
[22:10] it's challenging for two reasons. One is
[22:13] because first of all it's green field so
[22:15] you need to build system from scratch.
[22:17] Second, you still need to adhere to your
[22:19] team's policies
[22:22] >> like policy patterns
[22:23] >> patterns, right? So
[22:26] so making a sense of things that have
[22:28] been done in the past for different
[22:30] different purposes, drawing inspiration
[22:32] from that and then building this net new
[22:34] application. So Kiro is being designed
[22:39] and this is how
[22:39] >> Do you kind of absorb this policy or you
[22:42] have to kind of teach it?
[22:44] >> You don't want to explicitly teach it.
[22:46] You want your agent to learn. So you
[22:48] want to be really good at context
[22:49] engineering
[22:50] and everything is is subsumed in context
[22:52] engineering. Personalization is
[22:54] subsumed, memory is subsumed,
[22:55] understanding what are the different
[22:57] artifacts that you need to draw
[22:59] inspiration from because see at the end
[23:00] of the day your workhorse is a finite
[23:03] context large language model.
[23:04] >> Yeah. So, you need to be really
[23:07] prudent about what information to feed.
[23:10] Even if you know, companies advertise
[23:12] that I have a million token context and
[23:14] all, the
[23:15] the model's capability to pay attention
[23:18] on every token is still limited beyond a
[23:20] point. So, you need to be really careful
[23:21] in what you are putting in that 200,
[23:23] 300,000 token budget.
[23:26] And so, context engineering plays a very
[23:28] big role. And especially when you have
[23:30] teams and organizations that over the
[23:32] years have built many applications, how
[23:35] do you succinctly represent that?
[23:38] I think getting that right is going to
[23:41] unlock the next set of
[23:42] >> is one of the challenges that that we
[23:44] see inside inside Meta. We use a lot of
[23:46] like, you know, coding agents and things
[23:48] like that. And you're probably aware
[23:50] like, you know, React, like for example,
[23:53] like their their their their JavaScript
[23:55] framework comes from here.
[23:56] But we have a very specific way. Like we
[23:58] have a lot of conventions. So, that the
[24:01] convention are strong enough so that any
[24:04] person can actually figure out what the
[24:06] code is doing in any given point in
[24:08] time. We have, you know, one way to
[24:10] implement button, one way to implement
[24:11] reaction and type of things.
[24:13] And what I've noticed right now with
[24:16] people doing a lot more code gen with
[24:17] from agent and things like that that
[24:20] the agent will take a pattern that they
[24:23] will get from open source or from
[24:25] outside and will start applying in our
[24:27] code. Yes. And this goes well up to the
[24:30] point where you do have to debug it.
[24:33] That becomes a problem because you're
[24:34] like, wait, what? Like I was expecting
[24:36] these interaction to go to that file or
[24:38] I was expecting this.
[24:39] And then you can realize, oh, there was
[24:41] like a recent change that was made by,
[24:44] you know, with agent assistance and then
[24:46] now we have to roll it back and kind of
[24:48] like really guide the agent. It's like,
[24:50] okay, you're going to redo this change
[24:51] following these convention. That's quite
[24:53] a challenge. And
[24:56] I don't know, do you guys have like some
[24:57] insights so far about how do you guide
[25:00] the agents like more automatically?
[25:03] >> Yeah, I think this is a challenging
[25:04] problem. I must acknowledge because your
[25:06] large language models are trained on
[25:08] public data and the primary source of
[25:12] information for LLMs is the data on
[25:14] which they are trained on. The
[25:16] in-context learning is a useful tool,
[25:19] but it often feels limiting.
[25:22] >> Yeah.
[25:22] >> Because 99% of the information is in the
[25:25] parametric knowledge of the LLMs. So you
[25:27] do
[25:29] need to build
[25:31] tools that can course correct or shape
[25:33] the behavior if you will.
[25:35] And I talked some of that in the earlier
[25:37] presentation.
[25:38] Like the you can the definition of
[25:41] intent can be very broad. One definition
[25:43] of intent is I have certain way of
[25:46] writing my code. There are there are
[25:48] certain norms that my organization
[25:50] adheres to. I want my agent to respect
[25:53] that.
[25:54] So so having those
[25:56] preferences
[25:58] into into agents memory and then making
[26:01] sure that the agent is respecting that
[26:04] as it is making progress.
[26:07] There isn't
[26:09] um
[26:09] a very
[26:10] sophisticated way of doing this today.
[26:13] There are tools and hacks that people
[26:15] people have.
[26:18] We we resort to couple techniques. Like
[26:20] for example, if you have hooks,
[26:22] steering hooks, then anytime you are
[26:24] writing a piece of code, you can ask
[26:27] your agent
[26:28] >> Yeah, go and verify.
[26:29] >> With that list of skills or list of like
[26:32] commands.
[26:33] >> Yes, exactly. It's like you have let's
[26:35] say a junior developer joining your
[26:37] organization. The instruction that I
[26:39] will give them is anytime you are
[26:41] shipping a code in production, just
[26:43] verify with a principal engineer.
[26:45] I think if that type of philosophy we
[26:47] want to reflect that into agents because
[26:50] at least for the next foreseeable
[26:52] future, we will continue to doubt the
[26:55] intentions that the agent has because
[26:57] they exude a lot of confidence.
[26:59] >> Yeah.
[26:59] >> And they and they mask these some of
[27:01] these inefficiencies. So, having these
[27:04] steering hooks periodically into your
[27:06] agentic workflow allows the agent to
[27:08] keep itself honest. And it's it's the
[27:11] organic course correction, if you will.
[27:14] >> We talk a lot about like agents are
[27:16] producing code. That's one.
[27:18] Um, what's your take about agents doing
[27:22] like kind of like the back-end portion,
[27:23] which is like the review portion?
[27:26] Anything that you have observed that,
[27:28] you know, patterns that the industry is
[27:30] going on and to like, "Hey, how we can
[27:32] how can we do better review of the code
[27:34] that's produced by agents and how do we
[27:36] reduce the risk?"
[27:39] >> Yeah.
[27:40] Um, there's a lot of code getting
[27:42] written
[27:42] >> Yes.
[27:43] >> [laughter]
[27:43] >> Yes.
[27:44] >> And so, the burden is um, on the review
[27:47] rather than writing the code. Um, and
[27:50] it's a it's a it's a real problem. Um,
[27:53] and there are um,
[27:55] like I think going back to your earlier
[27:57] uh, point that you were making where you
[27:59] use variety of LLMs to tackle different
[28:03] tasks because they're complementary in
[28:05] their in in their strengths. I think
[28:07] this is some of the strategies that we
[28:09] are experimenting with where um, it
[28:12] doesn't even have to be a different LLM,
[28:13] by the way. It's just a different
[28:15] personality of the same
[28:16] >> Yeah. Different context. Completely
[28:18] separate.
[28:19] >> You you prime the model to be its own
[28:21] critic. Now, in its own flow, it's not
[28:24] going to critic its output. It's going
[28:26] to be super confident. But us um,
[28:29] um,
[28:29] a completely new instantiation of the
[28:31] same LLM, but in the mode of being a
[28:34] critic, it can tell um, whether or not
[28:37] the the code that has been written is
[28:41] correct for the task, etc. So, some of
[28:43] these approaches, like multi-agentic
[28:45] approach, if you will, um, allows you to
[28:47] build these bells and whistles,
[28:50] um, but solving the code review problem
[28:52] at scale, I think the fundamental thing
[28:54] that is needed for large enterprises, if
[28:57] not for everybody, is baking in the
[29:00] patterns. Um,
[29:02] like there is I as a as a reviewer, as a
[29:05] principal engineer of the of the
[29:06] company, I have certain expectations
[29:08] that I want to see in the code. Baking
[29:11] baking that into your agentic, um,
[29:14] workflow is very very critical. And, um,
[29:17] I don't know to what extent context, uh,
[29:21] in-context learning is going to, um, be,
[29:24] um, be helpful because beyond a point,
[29:27] um, going deep into the parametric
[29:29] knowledge of the of the model is going
[29:30] to be needed is my personal take. Yeah.
[29:33] But, the model's context budget is also
[29:35] increasing. Uh, so maybe there is hope
[29:37] that we can continue to work with the
[29:39] frozen LLM and then just operate at the
[29:41] context layer. But, um,
[29:44] just hoping that showing a piece of diff
[29:47] and code, uh, and the model will just
[29:50] reason, uh, I think that's too limiting.
[29:52] You have to also tell model what has
[29:54] worked in the past. Yeah. Um, and I
[29:56] think that's again going back to context
[29:57] engineering. Context engineering takes
[29:59] variety of forms and and and shape, uh,
[30:02] but in the context of reviewing, you
[30:04] need to tell what a good review means.
[30:07] Um, what are the norms of the teams? So,
[30:10] uh, encode that, um, is is is necessary.
[30:14] >> Switching gears a little bit now, moving
[30:16] from like protection, uh, I want to talk
[30:18] a bit about agent autonomy.
[30:20] >> Uh-huh.
[30:21] >> Like I I want you to paint us a picture
[30:24] about where do you see kind of like the
[30:27] agentic technology going and how much
[30:30] should we expect from the autonomy and
[30:32] like like, you know, if you have like
[30:34] some killer example?
[30:36] >> I have examples, um, from my own
[30:38] personal use. Yeah. Yeah.
[30:40] >> Yeah. Um,
[30:41] I have, um, I have come to believe that
[30:44] um,
[30:45] you know, autonomous agents are a real
[30:47] thing and they
[30:50] are going to be crucial to solve tasks
[30:53] that otherwise would have been
[30:55] impossible if not
[30:58] hard or rather hard if not impossible
[30:59] the other way around.
[31:01] Um, so I'll give my personal example.
[31:05] I um,
[31:06] I have tinkered with some of the ideas
[31:09] in the past, some research ideas. Now,
[31:12] with the size of my organization
[31:14] growing, I don't get as much time to to
[31:16] work on some of these hypotheses.
[31:19] Um,
[31:19] I wish I if only I had a research
[31:21] assistant on my side
[31:23] >> Oh, yeah, yeah.
[31:23] >> who did not demand a lot of pay, then I
[31:26] could just ask them to go explore.
[31:29] And so there were these hypotheses that
[31:31] I have been nurturing for quite some
[31:33] time but made very little progress. So
[31:35] over the weekend during my free time,
[31:38] I built a using Kudo, I built these
[31:42] deep research agents
[31:44] >> Hm.
[31:44] >> um, who could go and explore
[31:47] repositories or code bases
[31:50] that are of similar type and are trying
[31:52] to solve problems of the similar nature
[31:54] and then draw inspiration and then
[31:56] tinker with the hypothesis that I have.
[31:58] Doing that personally would have taken
[32:00] me
[32:01] >> Yeah.
[32:01] >> many months.
[32:02] >> Yes.
[32:02] >> Now, these are the things where you are
[32:05] less worried about the
[32:08] the potential
[32:10] breakage of the production, if you will.
[32:12] These are the things that can safely run
[32:14] in the back end and are producing
[32:17] responses that you can validate, you can
[32:20] review and then spin off another task
[32:22] that goes on and does things. These are
[32:25] truly long running because there's a lot
[32:26] of work involved.
[32:27] >> Yeah, there's probably a lot of data and
[32:29] a lot of repos to collect and
[32:31] >> and artifacts to to analyze.
[32:33] So I have seen success stories
[32:36] of applying autonomous agents for these
[32:39] type of use cases. For coding too, like
[32:43] if I'm building my team for example,
[32:45] when they experiment with the next new
[32:47] feature for for Kiro, they have a lot of
[32:50] ideas that they want to
[32:53] want to
[32:54] experiment with. Now, you can be
[32:59] token efficient
[33:00] or you can choose not to. So, if you
[33:03] choose not to, then you know a mindless
[33:05] for loop over 100 experiments and then
[33:08] one of them wins and then you use that,
[33:11] but that's not the practical way to go
[33:13] about it.
[33:14] A more token efficient way is to build
[33:17] incrementally. So, you start with couple
[33:19] ideas, then you weed off weed out the
[33:21] ideas that are not promising and then
[33:23] focus on and then spawn off few more sub
[33:25] ideas. So, giving this
[33:28] plan to the agent so that it builds out
[33:32] its trajectory and letting it do its
[33:34] thing,
[33:36] I think has led to a lot of unlocking of
[33:38] the productivity for feature development
[33:42] in software engineering and otherwise.
[33:44] So, I feel very very excited about the
[33:46] possibility of having these agents run
[33:48] for hours and doing meaningful work. Of
[33:51] course, trust is a very big thing that
[33:54] we all need to be very mindful of. But,
[33:57] there are ways to
[33:59] to put bells and whistles, if you will,
[34:01] to make sure that the agent is, you
[34:03] know, you you need to have ways to get
[34:05] the agent back onto the track
[34:06] >> Yes. Yes.
[34:08] >> Or restart it almost like
[34:10] >> Or yeah. Yeah, so
[34:11] >> How do you like yeah yeah
[34:13] I totally agree with you. On personal
[34:15] project, I tend to have like less
[34:17] guardrail and I think at this point in
[34:19] time we all have like our Mac mini or
[34:21] Mac studio at home that runs stuff in
[34:23] the background in the house. We all we
[34:25] all do this.
[34:26] Um
[34:28] but in in a more production context,
[34:30] what sort of guardrail do you think are
[34:33] kind of like necessary by default for
[34:35] agents like that you see like in your
[34:38] work today that people are using and
[34:40] then and then like
[34:41] definitely need to have?
[34:43] >> Yeah, I feel observability and
[34:45] evaluation plays such a massive role. If
[34:48] you want to have your agent go run off
[34:50] or even if they were not autonomous
[34:52] agent, even if they are running for a
[34:54] few minutes.
[34:55] >> Yeah.
[34:55] >> I would still want a very strong
[34:58] observability and evaluation tenant for
[35:01] building the agent. So governance is is
[35:03] a is a very big
[35:04] aspect of agent building.
[35:07] So eval eval takes two forms. One is the
[35:10] offline eval or the build time eval and
[35:12] the other is the online eval. I think
[35:14] what your question is mostly how do you
[35:17] check that the agent is doing the right
[35:18] thing when deployed in production. So I
[35:21] focus more on the online evals. And
[35:24] for online evals, I think
[35:27] what it comes down to is um
[35:30] having periodic checks on the agent
[35:34] and making sure that the original intent
[35:36] that was specified is still being
[35:38] respected. It's a hard problem to solve.
[35:41] But for things that are
[35:43] deterministic and what I mean mean by
[35:45] that is that when you know the rough
[35:48] input and the output space of the
[35:50] problem you can pre-bake a lot of
[35:54] verification into into the system and
[35:58] then periodically check whether the
[35:59] agent is still on the on the right
[36:01] track. And these are programmatic ways
[36:03] of
[36:05] making sure your agent is on the track
[36:07] because if you involve the user
[36:11] very regularly, then it's defeating the
[36:13] purpose of building autonomous agent.
[36:14] >> People will click yes yes yes yes yes.
[36:17] Absolutely click.
[36:19] >> And then there is also repeating
[36:20] behavior. Like if the agent has made a
[36:22] mistake in the past and by human
[36:26] intervention I am collecting its
[36:28] behavior. I'm telling you should not you
[36:30] you should not have used this API
[36:32] version versus you should have used
[36:33] that.
[36:34] The agent should internalize that.
[36:36] >> Yes.
[36:36] >> I think that's the core of it.
[36:39] If you are
[36:40] like when we say do I have trust in the
[36:43] autonomous agent, what we really are
[36:46] trying to ask is is it learning from its
[36:50] failures? Because if I have a junior the
[36:52] analogy is uncanny by the way. When I
[36:54] have a fresh grad joining the team, they
[36:58] have no context on what works and what
[37:00] doesn't or they have very little context
[37:02] because they are new to the industry.
[37:04] And over time they gather that context
[37:06] or tribal knowledge as you as you
[37:09] you know, as we call it.
[37:11] And then they graduate and then they
[37:12] become principal engineer one day and
[37:14] they have all the knowledge
[37:15] >> Yeah, they have all the knowledge.
[37:16] >> of the organization and people go to
[37:18] them to seek that to seek that
[37:21] knowledge. And so you trust your
[37:22] principal engineer. You know that they
[37:24] would be doing the right thing. You want
[37:26] your agent to become a principal
[37:27] engineer. I think that's the intent. So
[37:29] then you can close your eyes and your
[37:30] agent is running for many hours. For
[37:32] that to happen, they need to learn as
[37:35] they go. So continual learning of the
[37:38] agent. If you don't solve that problem,
[37:41] I don't think we will be solving the
[37:42] trust problem of agents. So memory and
[37:46] continual learning are like two sides of
[37:47] the same coin. And yeah, we need to
[37:50] build that.
[37:51] >> Yeah, I I I I often feel like you know,
[37:53] when we correct agent and things like
[37:55] that, it's kind of like re-adding almost
[37:58] like special instruction back in the
[38:00] context or rules that to be persisted in
[38:03] memory so that subsequent session
[38:07] the place where I find difficult like is
[38:09] also
[38:11] the system topology for us will change
[38:14] all the time and a rule that might apply
[38:17] the previous week
[38:18] >> Yep.
[38:18] >> this week will be slightly changed. And
[38:22] often we need to re-inject a human to
[38:24] kind of like reason about, "Yeah, that
[38:27] particular dependency or system was
[38:29] upgraded. Now you need to do X, Y, Z
[38:31] again." That discovery is difficult.
[38:34] >> 100% like the consolidation of the
[38:36] memories
[38:37] uh is a is a very neat problem in itself
[38:39] and it's an important problem because
[38:41] what makes sense today may not make
[38:43] sense tomorrow.
[38:44] Um I learned that API um from going from
[38:49] X to Y, I need a particular API
[38:52] dependency. It may not hold true
[38:54] tomorrow because the world has changed.
[38:56] So your agent needs to forget
[38:58] >> Yeah.
[38:58] >> some of the things. And so it's part of
[39:00] the consolidation of the memories and
[39:02] keeping and upkeeping the memory as the
[39:05] agent goes. By the way, um
[39:07] when we say agent, the agent is within
[39:09] the scope of a user um and also within
[39:12] the scope of a team. So consolidating
[39:14] memories um
[39:16] for the user, for the team, all of that
[39:19] needs to be done well because some of
[39:22] the things hold true at the team level,
[39:24] some of the things are too niche and
[39:26] which hold true only at the personal
[39:27] level.
[39:28] >> Yeah.
[39:28] >> So making that differentiation is
[39:29] important and um forgetting the memories
[39:33] that don't make sense anymore um uh is
[39:37] is part of the problem. Um so when you
[39:38] do that and have confidence that the
[39:41] agent is um
[39:43] you know, able to do this memory
[39:45] handling well, then you are making the
[39:49] agents become more and more trustworthy.
[39:51] So, yeah.
[39:52] >> How do you guys manage like, you know,
[39:55] memory these days tends to be managed
[39:57] with a suite of markdown files that you
[40:00] have left and right?
[40:01] >> Yeah.
[40:02] >> Um have you evolved to something else a
[40:04] little bit like more like decent storage
[40:07] and structure so that we can do the
[40:09] consolidation and things like that? Or
[40:11] is still we're still at the MD file
[40:13] level?
[40:14] >> Uh if um we think of working with and
[40:17] I'll give the full answer to this one,
[40:19] but I want to say
[40:20] the first impression when people hear
[40:23] that we are still working with MD files
[40:25] or text file is that the system is
[40:27] primitive. I want to challenge that
[40:29] assumption because over the years what
[40:33] we have seen happen is that the LLMs
[40:35] have become extremely capable in working
[40:39] with bash tools.
[40:41] >> Yes.
[40:41] >> And bash tools are extremely good at
[40:43] working with markdown files, text files.
[40:46] >> Yes.
[40:46] >> Uh as opposed to
[40:48] working with say semantic embeddings,
[40:51] your bash is not going to be able to
[40:52] make sense of that. So sometimes being
[40:55] very naive about it and being very
[40:57] simplistic about it actually pays a lot
[41:00] of dividends.
[41:01] So that's the main motivation as to why
[41:03] you may still want to stick to MDs and
[41:06] text. However, there is a need for you
[41:08] to go beyond text matching. Uh brittle
[41:11] text matching. So grep is
[41:14] While grep is doing brittle text
[41:15] matching, the input query is still being
[41:18] generated by the LLM. So the LLM can be
[41:20] creative and reformulate the query. So
[41:23] you still have the possibility of um
[41:26] doing semantic search if you will
[41:28] because your LLM is expanding the
[41:30] coverage of the query. Although grep is
[41:32] still doing a brittle text match. As
[41:34] opposed to embedding base semantic
[41:36] search, there you are not doing a
[41:38] brittle text match. You are searching in
[41:40] the vector space. So you are looking for
[41:42] things that are semantically similar.
[41:45] Um that can be achieved via query
[41:47] reformulation and doing
[41:48] >> Yeah. Yeah. Yeah. Yeah. Yeah. Yeah.
[41:49] >> Um but if you have a direct way of doing
[41:51] semantic search, you should have that in
[41:54] addition. So I would not say either or.
[41:56] I think it's the and proposition.
[41:59] And we do use semantic semantic search
[42:02] like agent core memory as a primitive
[42:05] that we have built.
[42:06] It it does map things into the semantic
[42:09] space so that you have a broader
[42:11] horizon. The search space is larger.
[42:13] >> Yeah. I I'm
[42:15] I'm not sure this if it's a common
[42:16] practice, but something that I do when I
[42:19] use like you know, internally our coding
[42:21] agents called Meta code that we use code
[42:23] X and we code code code code and
[42:26] every time I set up a new environment, I
[42:28] will part of inside the coding agent, I
[42:31] will install a bunch of utility kind of
[42:33] like standard okay, install Python,
[42:35] install PDF reader, install Google
[42:38] Docs connector and all of this.
[42:41] And maybe it's an illusion that I have,
[42:44] but
[42:45] the fact that I give instruction, that I
[42:48] have these tool available
[42:50] next when I'm trying to tackle the real
[42:53] problem, it's going to understand I
[42:54] already have these tools and it's going
[42:56] to default already to these instead of
[42:58] like reinventing like a complete other
[43:00] way. So, it's I'm kind of giving
[43:03] the agent like a set of convention and a
[43:05] set of tool that I want
[43:07] the agent to prioritize and then from
[43:09] there like okay, work and build from
[43:11] that.
[43:11] >> Yes. Yes. Yes. Sometimes you also run
[43:14] into
[43:15] um
[43:16] um like agent getting overwhelmed
[43:18] >> Yeah.
[43:18] >> by a lot of tools. So, see when I um
[43:21] tell my agent that these are the tools
[43:23] that I want it to prioritize over the
[43:25] others, I'm just sometimes I'm just
[43:28] shoving a lot of tools
[43:29] >> Yeah.
[43:29] >> at it hoping that it will just figure
[43:31] out. Sometimes it doesn't.
[43:33] >> Yeah.
[43:33] >> Uh so, that's also part of the problem.
[43:35] But, the very fact that you have to tell
[43:38] the agent at every new invocation that
[43:40] say you're going from
[43:42] um your desktop to let's say mobile. I'm
[43:44] making things up at this point. Um and
[43:46] in mobile it's like start over. You have
[43:49] to retell the agent that hey, you your
[43:52] preference is to use this tool versus
[43:54] that or you go to archive for paper
[43:56] reading and all of that. It's like
[43:58] re-educating the agents. The agent
[43:59] should just know who Francois is, who
[44:01] Anoop is. So, that's one problem. The
[44:04] other is understanding what tools work
[44:06] for you personally.
[44:08] >> Yeah.
[44:08] >> So, I may
[44:11] benefit from using tool A, you may not.
[44:14] While tool A is really capable at
[44:16] solving a particular problem, but there
[44:18] may be other tool that is personalized
[44:21] for me, catered to It's like
[44:22] personalized drugs.
[44:24] That type of a thing.
[44:25] And I think so knowing
[44:27] if the agent can understand um which
[44:30] tools work for this user for what tasks,
[44:35] that's getting the agent become more and
[44:37] more personalized even at the tool
[44:38] discovery level. So, then you can be
[44:40] mindless about just shoving the tools at
[44:42] its disposal because you know your agent
[44:44] is just going to weed out noise from the
[44:46] signal.
[44:48] >> What's your take on
[44:50] agent personalization? Because I I will
[44:52] explain a little bit.
[44:54] Like
[44:55] I usually work with a coding agent. Like
[44:57] we have like that's help you produce
[44:59] features, produce code, and things like
[45:01] that.
[45:02] From operating system, we work with a
[45:05] dev agent.
[45:06] Internally, we call it Opsmate, and it
[45:08] helps you with investigation and
[45:10] figuring out if some
[45:12] anomaly in the system and helps you
[45:14] remediate.
[45:15] There's some teams that have like which
[45:17] would I call like a
[45:19] um release agent that will help with the
[45:23] software release. And we also have
[45:25] another like, you know, we have a bunch
[45:27] of analytics tools that you can analyze
[45:29] data. Like we have like a data agent
[45:31] that helps you understand, oh, this data
[45:33] was logged this way. How should I build
[45:35] a query for that and all of that?
[45:37] And it seems to be that we're, you know,
[45:39] there's
[45:40] uh one way of thinking is that we need
[45:42] to specialize these agents because they
[45:43] become more efficient and they use less
[45:45] token.
[45:46] And there's the complete other side of
[45:48] the story where, like, oh, no, we're
[45:50] going to use like one agent that's going
[45:53] to do it all. So, what's your take on,
[45:55] you know, which approach will actually
[45:57] win?
[45:58] >> So, I usually try to
[46:01] look at the parallels with how software
[46:04] engineering organization is set up. Um
[46:07] do you like often times when I'm talking
[46:10] about anomalies in my system and then
[46:12] going back to the code that result into
[46:14] anomaly, I'm usually not talking to
[46:16] different teams, different person. I'm
[46:18] talking to one individual. I'm saying,
[46:20] "Hey, this broke and so we need to go to
[46:22] the anomaly and make this configuration
[46:23] change and all." Um so which means that
[46:27] you need to have one agent that just
[46:30] knows it all.
[46:31] Um and so it's like connective tissue
[46:33] that you need to build. However, um
[46:37] at this point because
[46:39] um each system is complex in its own
[46:41] right,
[46:42] getting that agent do
[46:45] um do that task well
[46:47] uh is is not a solved problem yet. Um so
[46:51] it's a balance between how much deep do
[46:53] you want to go building that agent
[46:54] versus, you know, do you want to focus
[46:56] your attention in connecting these
[46:58] agents together? Um I would I would not
[47:01] go one extreme or the other. I think
[47:03] it's a sweet spot somewhere. So very
[47:05] concretely, uh because you talked about
[47:07] the ops agent, I think it makes sense
[47:09] for the ops agent to speak with the
[47:11] coding agent.
[47:12] >> Yeah.
[47:13] >> Um because doing operations
[47:16] um in isolation can only go so far
[47:18] because you're limited to only logs and
[47:20] machine generated data.
[47:21] >> how it's built.
[47:22] >> Yes, you need to understand what led to
[47:24] this anomaly and to go to take the next
[47:27] step, you also need to suggest what can
[47:30] avoid this anomaly to incident to happen
[47:33] in the future. So providing with a patch
[47:36] that will fix the issue and also being
[47:38] proactive about it, like anticipating
[47:40] what other things can potentially break.
[47:43] So if the two systems are not
[47:46] two sides of the same coin, then you are
[47:49] looking into errors getting cascaded or
[47:51] the the communication flow between the
[47:53] two agents is not efficient. So while we
[47:55] are building these agents in isolation,
[47:57] uh more or less, but um an intentional
[48:00] approach needs to be needs to be taken
[48:02] to have these two agents speak to each
[48:04] other much more closely. You can still
[48:07] have two separate intents two separate
[48:08] agents built in isolation, and then you
[48:11] have a communication bridge between
[48:12] them, or you can think of building one
[48:15] giant super agent.
[48:16] >> Yeah, one giant super agent.
[48:17] >> yeah, which is highly personalized,
[48:18] which understands. So, I would not at at
[48:21] least at this point, I would not say one
[48:23] agent rule rules them all, but there are
[48:26] organic uh pockets where it makes sense
[48:30] to club
[48:31] um two or three agents together and
[48:33] build one
[48:35] one super agent.
[48:37] >> Now, if you wear your you know, frontier
[48:40] researcher
[48:41] uh hat,
[48:42] >> Yeah.
[48:43] >> what's for you like the the the
[48:47] next evolution for the agent? Like,
[48:49] what's what is the thing that you guys
[48:52] are thinking or the hypothesis that you
[48:54] guys are working on that hey, you know,
[48:57] the wider market down the road maybe
[48:59] like 6 months or 4 months from now, like
[49:02] we'll start seeing. What's the frontier
[49:04] right now in terms of evolution of
[49:05] agents?
[49:07] >> I have a certain take on this one. Um I
[49:09] feel the agents that we are building
[49:11] today, regardless of the use case, they
[49:14] are still reactive. What I mean is they
[49:17] are looking for a user to express an
[49:19] intent, and then they are achieving the
[49:22] task. The next frontier in my opinion is
[49:25] about proactivity. Uh the agents are
[49:29] anticipating uh what is needed, and then
[49:31] they're doing it by themselves. Now,
[49:33] autonomous agents are proactive because
[49:36] they are making decisions as they go,
[49:38] but that is still in the in the confines
[49:40] of
[49:41] >> of an area
[49:42] or yeah.
[49:42] >> problem that the user has specified. Um
[49:45] be it be it a bit more open-ended. Um
[49:48] like I talked about AI scientists at the
[49:50] beginning of our chat. Um if there is an
[49:53] hypothesis, my agent can go do
[49:55] exploration as a research scientist and
[49:57] then come back with some evidence. But I
[50:00] want it to come up with hypothesis
[50:01] itself.
[50:03] Of course, I will still define some
[50:05] constraints for it. I will say I'm
[50:07] interested in the language modeling
[50:09] literature. I'm just making things up.
[50:10] So go do research in this area. What is
[50:12] the next net new science that can be
[50:14] built, next new feature that can be
[50:16] built. I think that is where
[50:18] I feel the next frontier is.
[50:21] And it's the continuum of your
[50:23] autonomous agent really, but you are
[50:25] increasing the scope of your autonomous
[50:27] agent.
[50:28] >> Great.
[50:29] To wrap up, I'm going to kind of like
[50:32] improvise a couple of like quick
[50:34] lightning question. Some of them will be
[50:36] related to kind of like your work, some
[50:38] of them will be more like personal use,
[50:40] and then we can take it from there.
[50:42] Um what's, you know, work life or home
[50:46] life? What's your favorite AI tool
[50:49] today?
[50:50] >> My
[50:52] um
[50:52] So I enjoy I have one kid at home. Um
[50:55] and my wife wants me, but I don't listen
[50:59] to her. I'm reliving my life through
[51:01] him.
[51:02] So there are a lot of things that I wish
[51:04] I had done when I was a kid. I am doing
[51:07] that through him.
[51:09] And what I mean is there are like common
[51:11] set of activities that we do. Um and I
[51:14] would have never imagined building
[51:16] complex robotic applications with my
[51:18] son. He's he's into building. He likes
[51:20] to build either be it Legos or do
[51:23] robotics. And he's also an artist and he
[51:25] paints. So I was like, what is
[51:28] what app can be built at the
[51:30] intersection of his
[51:32] his interest? I would I can I can't
[51:35] imagine doing that without these AI
[51:38] tools. And now over a weekend I was able
[51:40] to build something very compelling with
[51:43] him. So that gave me a chance to teach
[51:45] him, that gave me a chance to feel good
[51:47] about what correctly father son did
[51:49] together.
[51:49] >> Are you building on special Lego kit or
[51:51] Arduino with Arduino pieces or
[51:54] >> So there is an approved kit that his
[51:56] school has
[51:57] has has been using. So we use that.
[52:00] I'm blanking on the name of the of the
[52:02] kit at this point. So we use that
[52:04] mainly. In terms of my go-to tool,
[52:07] I
[52:08] use all the tools and I have
[52:11] a few reasons for that as well. One is I
[52:13] truly believe that
[52:16] the tools out there are complementary.
[52:18] They have their own strengths and
[52:20] nuances.
[52:23] And the second is because I'm in the
[52:24] business of building coding agents for
[52:26] my own customers.
[52:28] I want to draw inspiration from what
[52:30] works great for for the customers and
[52:31] the users. So that lets leads me to use
[52:36] Perplexity and and products
[52:39] >> computer's impressive.
[52:40] >> Yes, it's very impressive.
[52:41] >> It's impressive. It's a
[52:43] It's a kind of like an amazing concept
[52:46] when you think about it. It's like
[52:46] >> That is true.
[52:47] >> Yes.
[52:48] >> Yes, yes.
[52:49] >> Okay, so we cover AI tool. I'm going to
[52:52] use one of my standard question. VI or
[52:55] Emacs? What was the first love?
[52:57] >> [laughter]
[52:57] >> Emacs all the way. Like when I
[53:01] back in India, I was not using either of
[53:03] Emacs or Vim. When I came into the US
[53:05] for my PhD, my lab mates were using
[53:09] Emacs
[53:09] >> Yeah, yeah, yeah. That's an interesting
[53:11] case, yeah.
[53:11] >> And then I just tailored Emacs for my
[53:14] own pro productivity and so I love Emacs
[53:17] to death. So no Vim for me. Emacs all
[53:20] the way.
[53:21] >> [snorts]
[53:21] >> If if a kid approach you today about
[53:26] doing a PhD,
[53:27] >> Uh-huh.
[53:28] >> what would be your recommendation
[53:30] understanding the landscape of AI today?
[53:34] >> Like on the topic?
[53:35] >> Yeah, on the topic, school to go and
[53:39] >> Ah, the fun. I am so lucky that my son
[53:42] is still in the sixth grade, so I don't
[53:44] have to [laughter] worry about these
[53:45] things for another 6-7 years.
[53:48] Um, but let's say I transport myself
[53:51] back.
[53:52] >> Yeah.
[53:52] >> Um, to um, my undergrad. Um,
[53:56] and I have to now make a call on what
[53:58] should I do next?
[53:59] I feel,
[54:00] um,
[54:01] quantum
[54:02] >> Yeah, quantum is the next.
[54:03] >> Is the is the next. Like, I might want
[54:05] to sink my teeth into quantum. Um,
[54:08] that's that's a place where uh, people
[54:10] are, um, they feel very excited about
[54:13] and I see a lot of investment pouring
[54:14] in. So, I might advise my younger self
[54:18] to look into quantum. Um, at the same
[54:21] time, I feel there are cycles,
[54:23] um, when it comes to modern
[54:25] architecture. Like, today there is
[54:26] transformer, tomorrow there might be
[54:28] another. There's always a need to
[54:30] continue to do research in the core LLM
[54:34] uh, areas. But, um, funding is a is a
[54:37] very big, uh, factor into to what extent
[54:40] you can carry out, um, you know,
[54:43] scalable research. So, um,
[54:45] I would not hold anybody against doing
[54:48] research in this area. I would just
[54:50] caution them that they should look out
[54:52] for the sources of funding because
[54:53] that's a really critical piece in
[54:56] scaling their research and, um, and, um,
[54:59] stress testing their hypothesis.
[55:01] >> All right, last question, which is more
[55:03] about like industry trend.
[55:06] Um,
[55:07] like I said earlier, I've been working
[55:08] in speech recognition for a long time
[55:10] before, like, you know, my time at at
[55:13] Facebook, Meta, Yahoo, and others.
[55:15] Um,
[55:16] in the early days of speech recognition,
[55:19] we used to have, you know, custom chip
[55:22] for audio processing, custom chip for
[55:24] signal processing, and then then like,
[55:26] you know, the regular normal CPU for
[55:29] doing kind of like the orchestration of
[55:31] everything and moving forward.
[55:33] But, there was a time where then, I
[55:35] think it's in the Pentium days type of
[55:37] thing where the CPU became powerful
[55:39] enough
[55:40] that all these customs chip disappear
[55:42] and then we really focus on that and
[55:44] then that was kind of like you know
[55:46] there was a lot of progress made from
[55:47] them.
[55:48] I can kind of see the same parallel
[55:50] happening today like you know we have
[55:52] like specialized GPU for training,
[55:54] specialized GPU for
[55:57] inference and then we have like you know
[56:00] the CPU for running the apps and and the
[56:01] products.
[56:03] From your point of view, do you think
[56:04] that this could eventually merge like it
[56:06] did back then?
[56:08] >> That's a very interesting thought.
[56:11] I mean I think right now
[56:14] that like the the hardware and the
[56:16] software stack is very much indexing on
[56:20] a certain architecture
[56:23] of the algorithms of the foundation
[56:24] models.
[56:25] And I truly believe that there will be
[56:28] newer architectures
[56:30] coming coming soon
[56:32] and there will be variants too because
[56:34] right now we are really not bothered or
[56:36] thinking about efficiencies. We are just
[56:39] you know going all in on data and
[56:41] computer and all.
[56:43] And I don't know to what extent we
[56:45] should draw inspiration from how brain
[56:46] works and all but our brain is
[56:49] significantly more
[56:50] >> effective
[56:51] >> effective and and efficient
[56:52] >> a lot of stuff along the [laughter] way.
[56:54] >> Sure.
[56:56] >> But it is indeed effective. Yes.
[56:57] >> Yeah
[56:58] and so I'm pretty sure there will be
[57:00] advances made in the architecture
[57:03] which would then
[57:05] I guess challenge some of the hardware
[57:07] assumptions. So the community and the
[57:10] industry will naturally converge towards
[57:12] building hardware that works for variety
[57:15] of different architectures.
[57:18] So I guess naturally we will get into
[57:20] that state where we don't have to you
[57:22] know have a particular hardware for a
[57:25] particular architecture. I think we will
[57:27] have more
[57:28] like more scope um, in things that we
[57:30] can try with.
[57:32] >> Great. Well, thank you.
[57:34] Thank you so much for the great
[57:35] conversation and for being generous with
[57:37] your opinion, ideas, and sharing about
[57:40] your personal life. Thank you.
[57:41] >> Thank you so much, Francois. It was a
[57:43] pleasure chatting with you.
[57:44] >> Great.
[57:48] >> [music]
