Pfizer, Cummins Bind AI to Deterministic Tools to Tame Context Rot
Ai4
Long-running enterprise AI systems require replacing unconstrained generative reasoning with deterministic tool execution and relational audit logging to prevent context rot, spiraling token costs, and compliance failures.
AI-assisted summary of Memory, State, and Knowledge Management: Powering Long-Running AI Systems — original source. Check the original for full context.
Source published:
Detailed notes
## Background Published by the Ai4 channel on YouTube on August 22, 2026, this panel session explores how organizations build memory, state, and knowledge management layers to transition from stateless copilots to persistent, auditable, and long-running enterprise AI systems. Moderated by Alain Biem alongside leaders from Pfizer, Atos, UTMB Health, and Cummins, the discussion examines the practical deployment of vector databases, knowledge graphs, feature stores, and hybrid context fabrics while addressing critical trade-offs in cost, latency, and scalability.
## TL;DR Industry leaders from healthcare, pharmaceuticals, consulting, and heavy manufacturing discuss the shift from single-turn chat prompts to multi-agent architectures capable of sustaining complex workflows over extended horizons. The panel defines the operational triad of active state, historical memory retrieval, and synthesized causal knowledge, highlighting real-world implementations such as dynamic patient risk scoring, deterministic knowledge graph assembly for commercial pharma, and industrial context fabrics. To combat context rot, crippling latency, and escalating inference costs, the experts advocate for offloading deterministic logic into standardized tool calls, implementing upstream model routing, and utilizing relational databases for audit-ready compliance. Ultimately, they emphasize that sustainable long-running systems rely on rigorous observability, specialty-specific memory lifespans, and human-in-the-loop oversight.
## Atmosphere & style The panel maintains a pragmatic, collegial, and grounded tone, trading generic artificial intelligence hype for hard-won engineering lessons and authentic operational war stories. Alain Biem moderates with structured curiosity, allowing speakers to bounce seamlessly between deep technical debates on tokenomics and accessible real-world metaphors. Listening to the discussion feels like sitting in an executive engineering debrief, characterized by an overarching ethos of caution, architectural discipline, and a deep appreciation for human-centered oversight in high-stakes environments.
## Key takeaways - [00:04](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=4s) Enterprise AI must evolve beyond single-turn copilots and document summaries into long-running, multi-step systems capable of managing state, memory, and compliance over extended workflows. - [02:00](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=120s) In regulated commercial environments like pharmaceutical marketing, language models are architected as deterministic selectors over pre-approved content rather than unconstrained generators. - [07:14](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=434s) Long-running industrial AI separates runtime execution context (state) and episodic recall (memory) from causal lineage and contextual experience (knowledge). - [13:06](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=786s) Managing agentic time-series memory and complex state machines over months creates major engineering challenges in state deconstruction, temporal auditing, and data sovereignty. - [15:15](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=915s) Healthcare systems use dynamic multi-specialty AI triage to recalculate patient clinical risk in real time, serving as an assistive safety net while preserving human-in-the-loop decision-making. - [17:42](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=1062s) Knowledge graphs paired with deterministic retrieval pipelines maintain coherence and regulatory compliance by linking real-time commercial triggers with structured, approved claims. - [22:56](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=1376s) Industrial reliability relies on a centralized context fabric that integrates low-level real-time sensor telematics with decades of historical warranty and failure data. - [25:27](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=1527s) Full-stack observability using OpenTelemetry and evaluation frameworks is critical to measure operational ROI against hidden infrastructure costs and iterative token spend. - [30:40](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=1840s) High-stakes enterprise automation, such as purchase order exception triaging, requires decoupling LLM conversational summarization from deterministic tool execution to eliminate hallucinations and prompt injection risks. - [33:35](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=2015s) Unlimited memory introduces context rot and latency penalties; memory retention must be tailored to domain-specific lifespans or offloaded to traditional machine learning baselines. - [37:07](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=2227s) To control tokenomics and latency, architectures offload deterministic reasoning steps to standardized tool calls and deploy upstream lightweight models for intelligent routing. - [42:14](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=2534s) Regulated healthcare deployments bifurcate agent memory between strictly minimized patient-facing interactions and relational database audit trails for internal clinical recommendations.
## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=4s) Welcome and Introductions: Moving Beyond Single-Task Copilots
Alain Biem opens the session by welcoming attendees and introducing the overarching theme: long-running AI systems. Recounting his own trajectory—from an early career as a research scientist at IBM Research to serving as Chief AI Officer at S&P Global and New York Life Insurance Company, teaching machine learning and analytics at Columbia University, and co-founding AI startups—Biem sets an experienced, practical tone for the discussion. He reflects on the early days of generative enterprise AI, which centered primarily on prompt-and-response interfaces like copilots or virtual assistants performing discrete, isolated chores such as drafting an email or summarizing a document.
Biem argues that single-interaction paradigms fall far short of real enterprise needs. Modern organizations operate through intricate business processes, persistent cross-team collaborations, complex workflows, and stringent governance and compliance frameworks. To deliver true business value, enterprise AI must evolve beyond single-turn tasks to execute end-to-end workflows that can run continuously over days or even weeks. Biem then introduces a cross-industry panel of leaders to discuss how their organizations manage memory, state, and knowledge across extended time horizons.
[02:00](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=120s) Panelist Introductions: Real-World AI Across Pharma, Consulting, Healthcare, and Heavy Industry
The panel begins with brief introductions highlighting how different sectors approach AI operations. Johannes Plambeck, Senior Managing Data Scientist and AI Engineer at Pfizer, explains that his North American commercial group supports sales representatives and marketing operations during healthcare provider engagements, including phone and field force calls with doctors. In Pfizer’s heavily regulated domain, the AI system does not generate freeform sales pitches; rather, it was intentionally architected from inception as an intelligent selector to curate and serve pre-approved marketing materials.
```mermaid flowchart TD accTitle: Cross-Industry Panel Enterprise Contexts subgraph Panel["Panelist Enterprise Ecosystems"] direction TB subgraph P1["Pfizer (Commercial & Life Sciences)"] J["Johannes Plambeck"] --> J1["HCP Calls & Rep Engagements"] J1 --> J2["LLM as Selector for Pre-Approved Content"] end subgraph P2["Atos (Global Systems Integration)"] B["Brian Ray"] --> B1["560-person NA Data & AI Org"] B1 --> B2["Enterprise Operations & Project Delivery"] end subgraph P3["UTMB Health (Academic Medicine)"] M["Mike King"] --> M1["$4B Operation & 1,200 Hospital Beds"] M1 --> M2["Safety Net: Surfacing Red-Flag Clinical Signals"] end subgraph P4["Cummins Inc. (Global Power Solutions)"] C["Chitra Bhagat"] --> C1["Telematics & Business Systems"] C1 --> C2["Context Fabric Across Years-Long Lifecycles"] end end ```
Brian Ray, Head of Data & AI North America at the French global systems integrator Atos, borrows a working microphone and shares that his 560-person team delivers both project-based implementations and continuous operations across diverse client verticals. Mike King, Deputy CIO at the University of Texas Medical Branch (UTMB Health), outlines the scope of their $4 billion operation encompassing four campuses and approximately 1,200 hospital beds. UTMB operates uniquely complex service lines, including managing 75% of healthcare for the Texas Department of Criminal Justice inmate population via extensive telemedicine and a specialized prison hospital in Galveston, as well as providing medical care for cruise lines and researchers stationed at the South Pole. For UTMB, clinical AI acts primarily as an automated diagnostic safety net—scouring massive data volumes to flag missed lung nodules, cardiovascular risks, or early warning indicators so clinicians can intervene immediately.
Chitra Bhagat, Senior Director of Analytics and AI leading the Business Solution Design group at Cummins Inc., introduces her company after noting with good humor that an informal poll she conducted on the exhibit floor revealed 60% of attendees had never heard of Cummins. She clarifies that Cummins is an Indiana-based power and technology giant operating in 180 countries: its signature green generator sets sit behind skyscrapers and data centers, while its engines power 50% of heavy semi-trucks on the highway alongside massive mining and infrastructure equipment. Cummins’ data landscape spans real-time telematics tracking asset health in the field alongside core business systems (HR, finance, CRM, and CAD/engineering platforms). Bhagat explains that her focus is designing a unified "context fabric" that securely resides within the enterprise data layer rather than being locked inside third-party vendor models.
- **Pfizer Commercial AI**: Operates strictly within compliance bounds, using language models as targeted selectors over pre-cleared regulatory content for field force physician detailing. - **UTMB Health Operations**: Covers distinct care environments—from academic hospitals to telemedicine for Texas correctional facilities and extreme-environment support in the South Pole. - **Cummins Footprint**: Powers critical infrastructure, global logistics, and hyperscale data centers, combining IoT telematics with disparate transactional enterprise data.
[07:14](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=434s) Defining State, Memory, and Knowledge in Industrial AI
Responding to Biem's question on what long-running systems mean in an industrial setting, Bhagat demystifies the technical jargon by breaking down the conceptual triad of *state*, *memory*, and *knowledge* into plain business terms:
```mermaid flowchart LR accTitle: State Memory and Knowledge Distinction subgraph State["1. Current State"] S1["Execution Context"] S2["Identity & Authorization"] S3["Constraints & Active Topic"] end subgraph Memory["2. Memory Retrieval"] M1["Episodic & Procedural"] M2["Short-Term vs. Long-Term"] M3["Past Decisions & Supplier Logs"] end subgraph Knowledge["3. Synthesized Knowledge"] K1["Facts + Cumulative Experience"] K2["Causal Lineage: 'Why did AI act?'"] K3["Decade-Long Contract Context"] end State --> Memory --> Knowledge ```
- **State** represents what is happening in the exact moment—the active topic, the runtime execution environment, and the specific user impersonation and authorization credentials. For instance, Cummins recently deployed an agentic scheduling engine for a production plant where human planners historically struggled to finalize factory line schedules by 8:00 a.m. due to shifting overnight demand, machine constraints, and prior dependencies. The agentic system tracks the active operational state to resolve these multi-variable scheduling bottlenecks automatically. - **Memory** is the retrieval mechanism for past occurrences. Bhagat uses the analogy of casually asking Alain what his very first job was—an answer he can recall instantly despite not consciously thinking about it. In manufacturing, memory allows the system to determine whether a particular supplier delivered parts that met quality thresholds across specific historical quarters. Architecting this involves assembling infrastructure for episodic, procedural, short-term, and long-term storage. - **Knowledge** elevates raw facts and memory by infusing them with contextual experience. To illustrate, Bhagat compares factual data to knowing the composer and vocalist of a song, whereas knowledge is remembering where you were when you first heard it and whether you felt happy or sad. In enterprise systems, providing AI with "experience" requires logging the full causal lineage—recording not just what actions the agent took in the past, but *why* it took them, so it can trace and reuse that reasoning later.
Because corporate transactions and vendor relationships in heavy manufacturing span years of negotiations, warranties, and multi-stage contracts rather than isolated, transient chat sessions, Bhagat underscores that AI architectures must remain inherently persistent and long-running.
[11:30](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=690s) Client Expectations vs. Reality: Persistence, Context, and Industry Semantics
Alain turns to Brian Ray to ask how Atos addresses client demands when legacy systems and varied enterprise contexts collide. Ray points out an important disconnect between industry jargon and what end users actually request. Clients rarely walk in asking for a "long-running AI system"; instead, they ask for very intuitive, human-centered behavior: *"I want a system that remembers where I left off."*
Ray shares a personal anecdote about his mother in her eighties, who marvelled that ChatGPT retained her conversational history and remembered specific personal preferences, like her affinity for apple pie. In the corporate arena, clients expect that same seamless persistence coupled with an intrinsic grasp of their industry-specific semantics and operational boundaries. Ray emphasizes that fundamental state-tracking capabilities—retaining session histories, isolating user contexts, and tracking AI observations—are largely solved problems from an engineering standpoint. The true engineering bottleneck that enterprises are currently confronting involves coordinating autonomous agent harnesses and orchestrating multi-step workflows over extended durations without losing behavioral coherence.
[13:06](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=786s) Time Series, Complex State Machines, and the Rise of Data Sovereignty
As autonomous agentic workflows mature, managing long-term memory becomes significantly more complicated when time is introduced as a dimension. Brian Ray explains that basic state tracking is manageable, but debugging or auditing a complex state machine that spans months presents major engineering hurdles. He presents a relatable scenario: if someone's mother relied on an AI assistant to manage her shopping list and order groceries, and an error occurred six months in the past that is only discovered today, the system must be capable of traveling back in time through the state history. Layering semantic data models over temporal time-series streams makes querying, decomposing state, and preserving contextual continuity extraordinarily difficult.
This challenge is magnified across diverse industries—such as healthcare, financial services, and manufacturing—where every single enterprise use case demands a bespoke memory architecture. Furthermore, the industry is witnessing an intense focus on "sovereignty"—specifically data sovereignty and the sovereign control of agentic systems. Stricter regulatory landscapes, governance mandates, and responsible AI frameworks mean organizations cannot rely on a single, one-size-fits-all tool. Meeting these demands requires hundreds of engineers configured across specialized arrangements, though Ray emphasizes that this evolution mirrors past technological shifts: "We've been through engineering troubles like this through the web, through other things. It's just reinventing it now for agents."
- **State Deconstruction**: Querying historical semantic layers combined with time-series data requires decomposing state on top of complex relationships rather than treating memory as a static snapshot. - **Enterprise Diversity**: Atos deploys cross-functional teams segmented across dozens of technology stacks to handle heterogeneous memory models tailored to specific vertical regulations. - **Agentic Sovereignty**: Heightened compliance demands are forcing enterprises to establish strict boundaries and auditability over how autonomous agents retain, process, and act on proprietary state.
---
[15:15](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=915s) Patient Access and Dynamic Clinical Risk Scoring at UTMB Health
Addressing how long-running AI operates within a dual healthcare and academic setting, Mike King details the transformation of patient access and care optimization at the University of Texas Medical Branch (UTMB Health). In large academic medical centers, specialty clinics receive thousands of inbound patient referrals every month. Traditionally, these scheduling workflows ran on a basic first-in, first-out (FIFO) queue that ignored clinical nuance and urgency. Because patient conditions vary drastically, UTMB Health deployed multi-specialty AI agents that continuously monitor both inpatient records and ambulatory visit data to evaluate patient necessity dynamically.
```mermaid flowchart TD accTitle: UTMB Health Dynamic Patient Risk Architecture subgraph DataSources["Data Ingestion"] A["Inpatient Encounters"] B["Ambulatory Visits"] C["New Lab Results"] end
subgraph RiskEngine["AI Agent Processing"] D["Continuous Risk Assessment"] E["Dynamic Score Recalculation"] end
subgraph ClinicalAction["Human-in-the-Loop Delivery"] F["Elevated Priority Queue"] G["Provider Scheduling Decision"] end
A --> D B --> D C --> D D --> E E --> F F --> G ```
Whenever a patient undergoes an ambulatory visit or has a new lab processed, the AI recalculates an aggregate clinical risk score for that specific day. If routine lab work suddenly reveals an acute risk, the system immediately escalates the patient to the scheduling team with an urgent status. Crucially, the AI functions strictly as an assistive layer rather than an autonomous decision-maker. As King highlights, clinical judgment must remain grounded in human oversight: "Medicine is not black and white. We definitely have to have some gray areas where AI can lead us there, but ultimately we want a physician or care provider to make that final decision."
- **Volume & Velocity**: Managing thousands of monthly specialty referrals by replacing static FIFO queues with dynamic, risk-adjusted triage. - **Cross-Setting Monitoring**: Agents synthesize inpatient history and outpatient lab changes in real time to calculate up-to-date risk indices. - **Human-in-the-Loop Protocol**: AI produces risk evaluations and recommendations, while clinical staff retain final scheduling and treatment authority.
---
[17:42](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=1062s) Curating Deterministic Content with Knowledge Graphs at Pfizer
Transitioning into system architecture, session chair Alain Biem outlines the structural prerequisites of long-running systems—planners, stateful short- and long-term memory stores, vector databases, and knowledge graphs. Johannes Plambeck breaks down how Pfizer leverages knowledge graphs within its commercial marketing operations to optimize NBRx (new-to-brand prescriptions) and overall TRx (total prescriptions). Historically, pharmaceutical marketing relied on field representatives delivering physical brochures or product samples to physicians to influence prescribing habits. Pfizer has augmented this model by using AI to correlate prescription volume directly with specific conversational language and curated content.
```mermaid flowchart LR accTitle: Pfizer Knowledge Graph Content Curation Pipeline subgraph Drivers["Business Triggers & Segmentation"] A["KPI Engine<br/>(e.g., Market Share Drops)"] B["Doctor Segment Clusters<br/>(High vs. Low Prescribers)"] end
subgraph KG["Pfizer Knowledge Graph"] C["Approved Medical Claims"] D["Productive Questions"] E["Closing Statements"] end
subgraph Synthesis["Deterministic Output"] F["Deterministic Retrieval System"] G["LLM Summarization & Packaging"] H["Contextual Sales Asset"] end
A --> F B --> F C --> F D --> F E --> F F --> G G --> H ```
At the center of Pfizer's architecture is a curated knowledge graph that clusters verified information tailored to distinct physician segments, such as high-prescribing versus low-prescribing doctors. Rather than allowing large language models to generate synthetic marketing prose from scratch—which introduces severe compliance risks—the system operates deterministically. It pulls pre-approved clauses, productive questions, and structured closing statements directly from the graph to address precise business drivers. The state within this ecosystem fluctuates when dynamic KPIs fire (such as a drop in local market share for a specific drug), while the underlying medical content remains relatively static subject to formal review cycles. The knowledge graph serves as the durable artifact that maintains coherence across business context, approved literature, and rep solicitations.
- **Metric Optimization**: Directly tying commercial messaging to NBRx (new prescription write) and TRx (total prescription) performance indicators. - **Deterministic Assembly vs. Generation**: Restricting the LLM's role to summarizing and formatting verified clauses, closing statements, and targeted questions retrieved from the knowledge graph. - **Dynamic Triggers with Static Content**: Coupling real-time commercial KPIs with strictly regulated, peer-reviewed promotional assets.
---
[21:40](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=1300s) Synthesizing Clinical Notes and Physician Workflows
When addressing how to bridge structured metrics with unstructured clinical documentation, Mike King points to the daily reality of clinical practice. In standard healthcare interactions, physicians often spend upwards of 90% of an appointment staring into their screens, attempting to manually synthesize complex electronic health record (EHR) data. This information includes past clinical notes, family histories, lab trends, notes from external specialists, and diagnostic imaging summaries.
To alleviate this cognitive overload without disrupting clinical workflows, UTMB Health designs its AI systems to mirror the exact cognitive steps of a practicing physician. The models anchor first on the primary clinical note—capturing documented symptoms, diagnoses, and baseline treatment plans—before expanding outward to detect patterns, historical commonalities, and diagnostic gaps across the broader unstructured medical record.
- **Cognitive Mirroring**: Engineering AI retrieval pipelines that follow the natural diagnostic workflow of medical practitioners. - **Unstructured Fusion**: Ingesting narrative clinical notes, external physician evaluations, and imaging synopses alongside discrete lab values. - **Provider Time Recovery**: Streamlining information synthesis to reduce screen-time burden during patient consultations.
---
[22:56](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=1376s) Industrial Reliability and the Context Fabric at Cummins
Chitra Bhagat explores how Cummins maintains mission-critical reliability across heavy industrial systems where there is zero tolerance for operational error. Drawing on a 10- to 12-year analytics journey, Bhagat stresses that enterprise-grade AI cannot simply sit on top of ad-hoc data feeds. Long before the recent wave of generative AI, Cummins established robust machine learning models to predict mechanical failures across engines and components, feeding real-time maintenance alerts to fleet managers via mobile applications to direct vehicles into service bays before catastrophic downtime occurs.
To make raw telematics meaningful, Cummins established an architectural interpretation layer—termed the "context fabric" or "context layer." This layer joins raw sensor telemetry with historical warranty databases, past claim histories, and component failure rates to derive actionable engineering context. Bhagat describes this modular architecture through the metaphor of an artist's palette: having mastered underlying feature functionalities, the engineering team can select specific capabilities from the palette to construct reliable, standardized design patterns that feed directly into systems of record and customer-facing experience layers.
- **10+ Year ML Foundation**: Leveraging long-standing predictive maintenance models for heavy-duty engines and components. - **Context Fabric**: An interpretation layer that marries raw, low-level telemetry with historical warranty records, parts failure catalogs, and customer claim histories. - **Standardized Design Patterns**: Treating core data capabilities as a modular "palette of colors" to rapidly assemble auditable, reliable architectures across industrial use cases.
[25:27](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=1527s) Tool Sets, Observability, and Telemetry in Multi-Agent Stacks
Brian Ray examines the evolving landscape of tooling required to build and govern multi-agent architectures. While open-source frameworks like LangChain remain prominent, foundation model providers and major hyperscalers are rapidly rolling out native agent-management capabilities. Brian emphasizes that there is no single "silver bullet"; rather, organizations are deploying heavily layered architectures. When gauging the room, he notes that technical audiences must pay special attention to evaluation and observability platforms. He highlights tools such as Portkey within their sovereign agentic factory, which capture token usage across diverse runtimes (from Microsoft stacks to sovereign enterprise environments) and track operational KPIs in real time.
```mermaid flowchart LR accTitle: Real-Time Agent Telemetry and KPI Evaluation Architecture subgraph Agents["Agent Runtimes"] MS["Microsoft Stack"] SOV["Sovereign Stack"] end subgraph Telemetry["Observability Layer"] OTEL["OpenTelemetry Engine<br/>(Logs, Spans, Traces)"] end subgraph Valuation["Value Analysis (Portkey)"] KPI["Operational Value<br/>(Hours Saved, Disasters Avoided)"] COST["Total Token Cost + Hidden Services"] ROI["Real-Time KPI / ROI Score"] end MS --> OTEL SOV --> OTEL OTEL --> KPI OTEL --> COST KPI --> ROI COST --> ROI ```
Brian explains that observability must be treated as a core component of memory and state rather than an afterthought. Using OpenTelemetry standards to harvest traces, logs, and spans directly from agents, systems can continuously balance business outcomes—measured in concrete metrics like widgets produced, labor hours saved, or operational disasters averted—against total token usage and supporting infrastructure overheads. He points out a counterintuitive finding from industry studies (such as Ernst & Young): solving a problem with a cheaper, simpler foundation model can end up costing as much as running an expensive flagship model because the simpler model requires deeper, multi-step iterations to converge on an accurate output.
- Observability spans and traces stream operational signals directly into evaluation engines to assess cost-to-value performance in real time. - Total cost calculations include model token usage, surrounding infrastructure services, and environment-specific enterprise discounting. - Simpler models often trade lower per-token pricing for higher iterative volume and multi-hop reasoning steps, leveling total operational costs with frontier models.
---
[29:00](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=1740s) Shared Context, Multi-Agent Governance, and the Return of Data Science
Turning to the core operational hurdles of agentic systems, Brian highlights that understanding multi-agent workflows retrospectively and maintaining consistent shared context across an enterprise are extraordinarily difficult. A primary challenge arises when two employees in different departments ask the exact same business question but receive disparate answers due to divergent agent context states. Furthermore, because AI tooling evolves at a relentless pace, organizations that previously downsized their technical teams are now reversing course.
Brian observes that enterprises are actively bringing back data scientists and machine learning engineers because measuring baseline value, auditing evaluations, and distilling raw operational logs into concise context require deep statistical and data-engineering rigor. Just as in data warehousing paradigms a decade ago, teams must build sophisticated summarization and data minimization pipelines so agents can ingest operational histories without context bloat or signal degradation.
---
[30:40](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=1840s) Deterministic Execution in Supply Chain: The Smart Purchase Order System
Chitra Bhagat details how Cummins approaches agentic deployments across enterprise procurement, where thousands of procurement managers spend a significant portion of their workday handling purchase order exceptions—such as sudden shifts in demand or supplier component obsolescence across an inventory of millions of active parts. Cummins launched an agentic planning signal solution dubbed the Smart Purchase Order System to automate this complex triaging process.
```mermaid flowchart TD accTitle: Deterministic Agentic Flow for PO Exception Triaging subgraph Inputs["Data Ingestion"] PO["PO Exceptions & Dynamic Demand"] SUP["Supplier Email Chains & History"] end subgraph AgentCore["Smart PO Agent System"] LLM_SUM["LLM Summarization & Trace Analysis"] TOOL_SEL["Tool Execution Routing<br/>(Deterministic Engine)"] end subgraph Actions["Downstream Outcomes"] TRIAGE["Deterministic Triage Resolution"] REUSE["Reusable Enterprise Agent Framework"] end PO --> LLM_SUM SUP --> LLM_SUM LLM_SUM --> TOOL_SEL TOOL_SEL --> TRIAGE TOOL_SEL --> REUSE ```
To eliminate hallucinations and safeguard against prompt injection in such a high-stakes domain, Cummins made a deliberate architectural choice: strictly decoupling LLM generation from execution. The LLM evaluates supplier communication histories, interprets why a supplier responded in a certain way, and determines which specific tool to invoke, but it is not permitted to generate arbitrary unconstrained operational actions. This design sacrifices slight generative flexibility in exchange for fully deterministic business logic and yields a reusable framework where new prompts and tools can be dropped into the same core engine.
- Replaces manual purchase order exception triaging across millions of parts subject to dynamic demand shifts and component obsolescence. - Architectural separation of tool execution from text generation neutralizes prompt injection vectors and ensures deterministic outputs. - Retains LLM strengths for conversational email summarization and audit trace reconstruction.
---
[33:35](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=2015s) Context Rot, Specialty Lifespans, and Traditional ML Cost Baselines
Mike King explains that within a combined healthcare, education, and research enterprise like UTMB Health, unlimited data retention is not always an advantage. In clinical environments, AI memory can quickly turn into a dangerous liability through "context rot." Clinical relevance varies drastically by domain: a nephrologist may only care about lab results from the preceding six months, whereas a cardiologist requires a decade-long longitudinal view. Consequently, memory layers and agent governance policies must be customized around specialty-specific data lifespans.
Mike also challenges the default assumption that every problem demands real-time foundation model inference. For non-urgent predictive workflows, traditional machine learning models that take two weeks to train or compile are often dramatically more cost-effective. Enterprises can pair these low-cost statistical models with targeted agents that inject insights into clinical workflows only at the precise moment of care, maintaining high clinical efficacy without ballooning token costs.
---
[34:50](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=2090s) Tokenomics Dashboards and Knowledge Graph Latency Bottlenecks
Addressing how organizations maintain control over runaway inference expenses, Mike King outlines UTMB Health's two-pronged tokenomics framework. For internal multi-agent builds, a centralized monitoring dashboard tracks per-agent token usage against an upfront ROI hurdle rate established before project approval. For employee-facing workflows, they invested in a startup offering a fixed-subscription workspace that provides access to 12 distinct LLMs within a secured perimeter, absorbing token volatility while allowing staff to select the model best suited for their specific task.
Johannes Plambeck takes up the conversation by emphasizing that context rot carries a severe indirect penalty: crippling latency. He shares an early production workflow at Pfizer where a medical representative needed to pair productive physician questions with approved scientific content and closing statements. Injecting the necessary knowledge graph into the context window consumed 10,000 to 50,000 characters, introducing a 15-second latency delay that proved completely unacceptable for interactive field operations, highlighting knowledge graph compression and retrieval optimization as immediate technical imperatives.
[37:07](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=2227s) Stripping Determinism into Tool Calls and the Tokenomics of Human Evaluation
To mitigate context rot and rein in token consumption, organizations must carefully audit what an LLM is actually attempting to reason through. The core strategy is to strip as much deterministic logic out of the model's workload as possible, replacing unstructured model reasoning with deterministic tool calls—such as standardized Model Context Protocol (MCP) tools. By offloading rule-based execution and data fetching to external tools, systems drastically cut down on trace size and prevent context degradation.
However, token economics extends well beyond runtime inference; offline evaluation and human-in-the-loop workflows represent a massive, often overlooked cost center. Even after trimming determinism, remaining context windows can be substantial. When that context must be reconstructed, formatted, and spun up so a human evaluator can review it, token usage surges. This stage is non-negotiable for critical tasks like tuning guardrails, engineering prompt modifications, and measuring true positive and true negative rates across distinct failure modes. Production-grade cost management therefore operates across two fronts: shrinking live production costs—often by fine-tuning targeted Small Language Models (SLMs)—and managing the heavy offline evaluation footprint required to make complex context legible to human reviewers.
- **Deterministic Offloading**: Replace multi-step LLM reasoning with dedicated MCP tool calls to keep trace sizes small and prevent context rot. - **Evaluation Token Burden**: Preparing large, convoluted contexts for human inspection during failure mode analysis and guardrail tuning significantly inflates offline token spend. - **Model Downsizing**: Fine-tuning foundation models into specialized SLMs serves as a primary lever to reduce real-time inference costs.
```mermaid flowchart TD accTitle: Context Optimization and Dual Tokenomics Architecture subgraph Optimization["Deterministic Logic Offloading"] A["User Input / Complex Task"] --> B{"LLM Reasoning"} B -->|"Strip Deterministic Steps"| C["Deterministic MCP Tools"] C -->|"Structured Output"| D["Compressed Context Engine"] end
subgraph Tokenomics["Dual-Front Token Economics"] D --> E["Production Inference\n(Fine-tuned SLMs)"] D --> F["Offline Human Evaluation"] F --> G["Context Formatter\n(Human Comprehension)"] G --> H["Guardrailing, Prompts &\nFailure Mode Metric Analysis"] end ```
[39:10](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=2350s) Intelligent Routing, Observability, and Pragmatic Agent Architectures
The conversation around "tokenomics" and optimization frequently overlooks clever architectural routing. Placing a lightweight, primitive model upstream as an intelligent router in front of a powerful foundation or foundry model can substantially lower overall operational costs across multi-agent systems. While fine-tuning remains a lively topic among engineers, the industry's center of gravity is shifting away from raw model functionality and toward the surrounding application and orchestration layers.
A central requirement for scalable architectures is real-time observability. Being able to detect latency spikes instantly and apply dynamic fixes is the gold standard, though practical implementation requires a realistic balance. There is a wide spectrum between optimistic startup founders who promise fully autonomous, self-healing code that writes itself and pragmatic enterprise engineers who maintain deep operational control. The reality lands firmly in the middle: developers will not need to hand-code every single line, but they must thoroughly understand system mechanics to intervene effectively. Ultimately, cost management begins with rigorous, top-down use case qualification to verify that a business problem genuinely warrants building an advanced AI solution in the first place.
- **Upstream Model Routing**: Deploying smaller, specialized models to triage and route queries before invoking expensive foundation models yields dramatic cost efficiencies. - **Shift to the Application Layer**: Competitive advantage and stability reside in the orchestration, integration, and observation tooling built around models. - **Foundational Realism**: Autonomous self-healing systems remain an aspirational ideal; reliable enterprise operations depend on real-time monitoring and sound engineering fundamentals.
[41:33](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=2493s) Philosophical AI Design: Care and Clarification over Raw Memory
While technical discussions often center on deterministic metrics and token optimization, designing long-running systems also demands a human-centered approach. When considering what makes human colleagues memorable and effective, the answer is rarely raw, exhaustive data retention; it is their ability to be collaborative, pleasant, and thoughtful.
Applying that same philosophy to artificial intelligence reframes how memory systems should be engineered. An agent that hoards endless context history is often less effective than an agent designed to exhibit care—one that knows when to pause, recognize ambiguity, and ask a clarifying question before proceeding. Designing for conversational empathy and active clarification ultimately simplifies token economics by ensuring the system only processes what is truly relevant to solving the user's immediate problem.
> "Am I going to remember an agent that has tons and tons of memory, or am I going to remember an agent that shows care? Can we build something that can pause and ask us clarifying questions?"
[42:14](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=2534s) Regulatory Compliance in Healthcare: Bifurcated Memory and Relational Audit Trails
Operating long-running AI systems within heavily regulated domains like healthcare and pharmaceuticals introduces stringent compliance constraints across state and federal jurisdictions, including Texas transparency mandates and New York regulatory frameworks. To maintain strict compliance, healthcare systems must bifurcate agent architecture into two distinct memory paradigms based on the intended audience.
For patient-facing interactions, agents practice strict data minimization, retaining only the bare minimum operational information needed to complete a task, such as appointment scheduling details or verified communication preferences. For internal enterprise knowledge, systems retain extensive historical context, but auditability presents a major challenge. To ensure every decision is defensible and reproducible months later, AI recommendations must be written back to traditional relational databases rather than relying solely on vector stores. This creates an unalterable, structured history trail that regulatory bodies can easily audit without forcing engineers to reconstruct transient semantic states.
- **Bifurcated Agent Scope**: - **Patient-Facing Agents**: Adhere to minimal data capture, storing only critical scheduling attributes and channel preferences. - **Internal Knowledge Agents**: Capture broad organizational context to support staff workflows while enforcing strict compliance controls. - **Relational Persistence for Auditing**: Logging model recommendations directly into structured RDBMS tables guarantees a transparent, queryable historical trail, preventing audit-critical context from becoming lost or obscured inside vector database representations.
## Notable quotes > [11:30](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=690s) I want a system that remembers where I left off. — Brian Ray
> [13:06](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=786s) We've been through engineering troubles like this through the web, through other things. It's just reinventing it now for agents. — Brian Ray
> [15:15](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=915s) Medicine is not black and white. We definitely have to have some gray areas where AI can lead us there, but ultimately we want a physician or care provider to make that final decision. — Mike King
> [41:33](https://www.youtube.com/watch?v=-yOc9IDeSgM&t=2493s) Am I going to remember an agent that has tons and tons of memory, or am I going to remember an agent that shows care? Can we build something that can pause and ask us clarifying questions?
## Mentioned Alain Biem, Chitra Bhagat, Brian Ray, Mike King, Johannes Plambeck, Ai4, Pfizer, Atos, UTMB Health, Cummins Inc., IBM Research, S&P Global Market Intelligence, New York Life Insurance Company, Columbia University, Texas Department of Criminal Justice, ChatGPT, LangChain, Portkey, OpenTelemetry, Microsoft, Ernst & Young, Smart Purchase Order System, Model Context Protocol (MCP)
## Video description Alain Biem — Co-founder; Former Chief AI Officer, New York Life, S&P Global Market Intelligence, Columbia University Chitra Bhagat — Sr. Director, Analytics & AI - Business Solution Design, Cummins Inc Brian Ray — Head of Data & AI NA, Atos Mike King — Deputy CIO, UTMB Health Johannes Plambeck — Senior Managing Data Scientist, AI Engineer, Pfizer Inc.
Stateless models are powerful but limited. True enterprise AI requires systems that can remember, adapt, and reason across time. This panel will explore how organizations are building memory and state management layers to make AI contextual, auditable, and persistent. Panelists will dive into vector databases, knowledge graphs, feature stores, metadata systems, and hybrid approaches that underpin everything from customer-facing chatbots to predictive analytics engines. Expect discussion of trade-offs in scalability, cost, and latency, as well as how leaders are weaving together structured enterprise data with unstructured knowledge to support long-horizon use cases. Attendees will walk away with a clear view of the state-of-the-art tools and strategies for turning raw data into durable knowledge that drives competitive advantage.
[Source](https://www.youtube.com/watch?v=-yOc9IDeSgM)