Ai4
Enterprise AI projects fail to reach production because traditional connection-based Zero Trust breaks under ephemeral multi-agent swarms, requiring cryptographic security embedded directly into data payloads.
AI-assisted summary of Filling the POC-to-Production Gap With Operant Networks — original source. Check the original for full context.
Source published:
Detailed notes
## Background Published on YouTube by the Ai4 conference channel on August 25, 2025, this presentation features Keith Rose, CEO of Operant Networks. According to the session abstract, the talk explores why 85% of enterprise AI projects fail after the POC stage due to missing operational requirements and ROI measurement, introducing a dedicated pilot trust layer to bridge the gap to production.
## TL;DR Keith Rose analyzes why the vast majority of enterprise AI initiatives fail to reach production, arguing that the bottleneck stems from infrastructure, security, and governance challenges rather than flawed model outputs. Through an RFP automation case study, he introduces a 'crawl, walk, run' deployment framework that inserts a secure pilot phase to validate operational constraints and non-human identity. Rose details four core architectural criteria for pilot environments, emphasizing a strict separation between data and control planes. Finally, he outlines how ephemeral multi-agent swarms outgrow traditional connection-based Zero Trust, advocating instead for data-centric security powered by Named Data Networking (NDN) to embed cryptographic trust directly into network packets.
## Atmosphere & style The presentation carries a pragmatic, authoritative tone that speaks directly to the boardroom anxieties of enterprise technology leaders confronting high POC failure rates. Rose balances conversational candor—lightheartedly referencing 'grumpy' compliance gatekeepers—with sober, disciplined systems engineering and historical networking context. The talk feels like a reassuring, no-nonsense executive briefing that cuts through agentic AI hype to anchor complex deployments in battle-tested infrastructure realities.
## Key takeaways - [00:04](https://www.youtube.com/watch?v=TLxd61MVdlU&t=4s) Roughly 85% of enterprise AI proofs of concept (POCs) fail to reach production due to operational hurdles and trust gaps rather than model failure. - [00:04](https://www.youtube.com/watch?v=TLxd61MVdlU&t=4s) Traditional POCs yield skewed, overly optimistic ROI estimates because they test model outputs in isolation from enterprise compliance and security constraints. - [01:26](https://www.youtube.com/watch?v=TLxd61MVdlU&t=86s) Successful deployment requires a three-phase progression: Crawl (POC answering 'what'), Walk (Pilot answering 'how'), and Run (Production realizing full ROI). - [01:26](https://www.youtube.com/watch?v=TLxd61MVdlU&t=86s) High-value agentic workflows, such as accelerating RFP turnaround from weeks to days, regularly stall when cybersecurity and compliance teams intervene over data access and legacy integration. - [04:05](https://www.youtube.com/watch?v=TLxd61MVdlU&t=245s) An enterprise AI pilot layer requires a friction-free internal deployment model and dedicated non-human identity authentication, since standard Zero Trust is built for humans. - [04:05](https://www.youtube.com/watch?v=TLxd61MVdlU&t=245s) A secure AI control plane must enforce dynamic inter-agent access rules while remaining entirely invisible and immutable to the AI agents operating on the data plane. - [04:05](https://www.youtube.com/watch?v=TLxd61MVdlU&t=245s) Early agent connectivity standards like Model Context Protocol (MCP) enable rapid prototyping but currently lack native, dynamic enterprise-grade security controls. - [07:00](https://www.youtube.com/watch?v=TLxd61MVdlU&t=420s) Ephemeral, multi-agent swarms that dynamically summon peer agents render static perimeter firewalls and point-to-point connection security obsolete. - [07:00](https://www.youtube.com/watch?v=TLxd61MVdlU&t=420s) Named Data Networking (NDN) shifts trust from network connections to the data itself by cryptographically signing, encrypting, and binding access policies directly to every data packet. - [07:00](https://www.youtube.com/watch?v=TLxd61MVdlU&t=420s) Data-centric NDN architecture is already proven in mission-critical infrastructure, securing roughly 2% to 3% of US power generation including nuclear facilities.
## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=TLxd61MVdlU&t=4s) The AI POC Failure Metric and the Trust Gap
Keith Rose, CEO of Operant Networks, introduces the central dilemma confronting enterprise artificial intelligence initiatives: the well-documented statistic that 85% of AI proofs of concept (POCs) never reach production. While understandable given that the industry is still learning and experimenting in real time, Rose highlights how agonizing this reality is for AI leadership at Fortune 500 enterprises. Having to stand in front of a board of directors or a CEO to report that nearly nine out of ten AI initiatives fail to operationalize creates a "pretty uncomfortable situation."
To understand why this failure rate persists and how to fix it, Operant Networks deconstructed the problem into three fundamental pillars: 1. **Incomplete Business Data & Inaccurate ROI**: POCs evaluate model output in isolation rather than operational reality, leading to skewed return-on-investment calculations. 2. **Borrowing Proven Frameworks**: Examining established methodologies from cybersecurity and network architecture to address deployment challenges. 3. **Underlying Technological Shifts**: Identifying the emerging paradigms in AI interaction that make existing security approaches obsolete.
``` CRAWL (POC) WALK (Pilot) RUN (Production) ┌────────────────────────┐ ┌────────────────────────┐ ┌────────────────────────┐ │ Answers: WHAT │ │ Answers: HOW │ │ Full ROI Loop │ │ Model output & task │─▶│ Security, compliance, │─▶│ Scaled, trusted agent │ │ optimization │ │ & legacy integrations │ │ execution in enterprise│ └────────────────────────┘ └────────────────────────┘ └────────────────────────┘ ```
[01:26](https://www.youtube.com/watch?v=TLxd61MVdlU&t=86s) The RFP Scenario: Shifting from Model Problems to Trust Problems
To illustrate why promising POCs stall, Rose outlines a common enterprise workflow: an engineering or construction firm responding to complex bids and RFP proposals. Gathering cross-departmental subject-matter expertise for an RFP normally consumes three to four weeks and roughly 50 to 100 labor hours. When automated with multi-agent AI systems, that turnaround time can shrink to three days. For an enterprise handling dozens of RFPs every month, a standard POC yields astronomical projected economics.
However, the transition to production derails as soon as the project touches real-world enterprise constraints—such as bidding on a power plant in a heavily regulated industry. At this point, the "somewhat grumpy teams of cybersecurity and compliance" intervene with "Whoa, whoa, not so fast." These teams raise critical infrastructure and governance questions: how data flows across the network, who has authorization to view specific inputs, regulatory exposure, and how to extract data trapped in legacy systems. Consequently, what started as a model optimization exercise transforms into an infrastructure and trust bottleneck that traditional POCs completely fail to account for in their ROI models.
To bridge this gap, enterprises must adopt a structured "crawl, walk, run" deployment strategy: - **Crawl (POC Phase)**: Answers the *what*—demonstrating what AI capabilities can achieve for a specific functional task. - **Walk (Pilot Phase)**: Answers the *how*—validating how agentic systems operate within the existing enterprise environment across security boundaries, compliance requirements, and legacy integrations. - **Run (Production Phase)**: Combines the answers to *what* and *how* to produce an accurate ROI calculation that justifies enterprise rollout.
[04:05](https://www.youtube.com/watch?v=TLxd61MVdlU&t=245s) The Four Criteria for an Enterprise Pilot Trust Layer
Rose explains that solving the "how" requires a dedicated operational pilot environment—a secure sandbox or pilot trust layer designed specifically for testing AI agents against production-grade enterprise constraints. Instead of focusing merely on product features, Rose outlines four mandatory architectural criteria necessary to establish trust and safe experimentation inside enterprise networks:
``` +--------------------------------------------+ | SECURE CONTROL PLANE | | (Authentication, dynamic trust rules, RBAC)| +--------------------------------------------+ ▲ Invisible to | (Cannot be modified AI Agents | or bypassed by AI) | +--------------------------------------------+ | DATA PLANE | | [AI Agent A] ──▶ [AI Agent B] ──▶ [Legacy] │ | (Model context, prompts, system data) | +--------------------------------------------+ ```
1. **Frictionless On-Premises/Internal Deployment**: The sandbox must spin up within existing internal infrastructure without demanding massive platform upgrades, network overhauls, or complex reconfigurations. 2. **Non-Human Identity & Authentication**: Zero Trust frameworks were built around human identity verification (such as multi-factor authentication). AI agents are neither human nor traditional static software packages, requiring a new mechanism to verify that an autonomous agent is truly who it claims to be. 3. **Dynamic Inter-Agent Access Controls**: In multi-agent environments, explicit policies must govern what agents are permitted to say to one another as well as what data they can pass back to the end user. 4. **Strict Plane Separation**: A complete boundary must exist between the data plane (where information flows) and the control plane (which enforces trust, identity, and security rules). The control plane must remain entirely invisible to the AI so agents cannot alter security configurations or bypass constraints.
Rose references the Model Context Protocol (MCP) as a prime example of where these gaps appear. While MCP enables rapid prototyping between clients and servers, it currently lacks native production-grade security; changing trust rules often requires tearing down connections entirely rather than managing permissions dynamically.
[07:00](https://www.youtube.com/watch?v=TLxd61MVdlU&t=420s) The Evolution of Trust: Moving to Named Data Networking (NDN)
Rose places the current AI security challenge into a broader historical context, mapping out three distinct eras in enterprise trust architecture:
``` Era 1: Perimeter Defense Era 2: Connection Security Era 3: Data-Centric Security ┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐ │ [ Secure Firewall ] │ │ [ Zero Trust VPN ] │ │ [ Signed Packet ] │ │ ┌──────────────┐ │ │ Client ──▶ Host │ │ ┌──────────────────┐ │ │ │ IP & Data │ │ ───▶ │ 1-to-1 encrypted │ ───▶ │ │Data + Trust Rules│ │ │ └──────────────┘ │ │ static connections │ │ └──────────────────┘ │ └──────────────────────┘ └──────────────────────┘ └──────────────────────┘ (Broken by Remote Work) (Broken by Agentic AI) (Self-Securing Data) ```
- **Perimeter Defense**: Historically, enterprise data was placed behind a protected physical and digital perimeter ("a wall around that"). - **Connection-Based Zero Trust**: Remote work accelerated by COVID shattered the network perimeter, forcing security teams to transition to point-to-point encrypted tunnels and Zero Trust connection policies. - **Data-Centric Trust (The AI Shift)**: Autonomous agentic systems render point-to-point connection models obsolete. When an AI agent autonomously summons four specialized peer agents to spin up an ad-hoc communication network for twenty minutes and then disbands, security teams lose control over the sheer volume and dynamic churn of connections.
To prevent security breakdowns in ephemeral agent swarms, trust must be embedded directly into the data itself. Operant Networks utilizes **Named Data Networking (NDN)**, an architectural paradigm developed in academia where every individual data packet is cryptographically signed, encrypted, and governed by embedded trust rules detailing who can access or modify it. Rose points out that this technology has already been hardened in high-consequence operational environments, securing approximately 2% to 3% of the total power generated in the United States, including nuclear power facilities.
Rose concludes that saving the 85% of AI POCs stuck in the "POC valley of death" requires three coordinated steps: adopting a data-centric security model where trust is bound to the payload, deploying a secure sandbox layer to test operational readiness in enterprise environments, and establishing an accurate ROI assessment that gives leadership the confidence to move to production.
## Notable quotes > Most AI projects never make it past the POC — not because the models fail, but because teams lack the tools to measure ROI and meet operational requirements. — Keith Rose
> [00:04](https://www.youtube.com/watch?v=TLxd61MVdlU&t=4s) pretty uncomfortable situation — Keith Rose
> [01:26](https://www.youtube.com/watch?v=TLxd61MVdlU&t=86s) somewhat grumpy teams of cybersecurity and compliance — Keith Rose
> [01:26](https://www.youtube.com/watch?v=TLxd61MVdlU&t=86s) Whoa, whoa, not so fast. — Keith Rose
> [01:26](https://www.youtube.com/watch?v=TLxd61MVdlU&t=86s) crawl, walk, run — Keith Rose
> [07:00](https://www.youtube.com/watch?v=TLxd61MVdlU&t=420s) a wall around that — Keith Rose
> [07:00](https://www.youtube.com/watch?v=TLxd61MVdlU&t=420s) POC valley of death — Keith Rose
> If you're responsible for turning AI demos into deployed systems, this session will give you a clear roadmap for success. — Keith Rose
## Mentioned Keith Rose, Operant Networks, Ai4, Fortune 500, Model Context Protocol (MCP), Zero Trust, Named Data Networking (NDN)
## Video description Keith Rose, CEO, Operant Networks
Abstracts
Most AI projects never make it past the POC — not because the models fail, but because teams lack the tools to measure ROI and meet operational requirements. This talk will explore why real-world deployment is so difficult, and what’s missing from most efforts to move from POC to production. I’ll describe the missing “pilot” layer and how to create the right environment to test AI with real data, enforce guardrails, and evaluate ROI more accurately. If you're responsible for turning AI demos into deployed systems, this session will give you a clear roadmap for success.
[Source](https://www.youtube.com/watch?v=TLxd61MVdlU)