PNC AI Head Sees On-Premise GPU Compute Halving Three-Year API Costs

Scaling GenAI: From Pilot to Production-Ready Infrastructure With PNC — Ai4

Ai4

Shifting enterprise generative AI from cloud APIs to private GPU clusters can reduce infrastructure expenses by up to 50% over three years while cutting latency and hallucination risks.

AI-assisted summary of Scaling GenAI: From Pilot to Production-Ready Infrastructure With PNC — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 24, 2025, this presentation features Matthew Berezo, Head of AI at PNC Bank, examining the practical realities of scaling generative AI from experimental pilots to production-grade enterprise infrastructure. According to the description, the session explores GPU capacity planning, architectural trade-offs, cost controls, model deployment patterns, and monitoring frameworks for regulated environments.

## TL;DR Matthew Berezo outlines a structured enterprise framework for transitioning GenAI into core business operations, balancing the trade-offs between general-purpose LLMs and domain-specific SLMs across latency, efficacy, and unit economics. He presents a four-tier deployment taxonomy to inform build-versus-buy decisions and details the cost dynamics between API consumption and on-premises GPU clusters. The discussion also establishes robust observability practices mapped to the seven NIST risk pillars and explores multi-agent architectures using emerging standards like MCP and A2A protocols. Ultimately, Berezo advocates for an incremental, hybrid infrastructure approach anchored by clear business metrics and proactive governance.

## Atmosphere & style The presentation carries a pragmatic, authoritative, and analytical tone grounded in the real-world operational challenges of enterprise banking. Berezo delivers a measured, fluff-free masterclass that strips away industry hype, balancing high-stakes technical rigor with approachable, dry humor regarding hardware costs and prompt engineering pitfalls. Listening to the talk feels like sitting in an executive architectural review, characterized by disciplined frameworks, clear economic trade-offs, and actionable governance strategies.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=4s) Transitioning GenAI from experimental pilots to enterprise production requires addressing five interconnected pillars: scalability, infrastructure readiness, observability, agentic systems, and an execution roadmap. - [02:21](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=141s) While frontier Large Language Models (LLMs) provide broad out-of-the-box generalization, scaling them exposes major liabilities in response latency, lack of proprietary context, and escalating token-based API costs. - [02:21](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=141s) Small Language Models (SLMs) offer a compelling alternative for domain-specific enterprise tasks, delivering lower latency, reduced hardware demands, and efficient fine-tuning on internal proprietary data. - [06:13](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=373s) Evaluating GenAI use cases across four grounding patterns—zero grounding, in-prompt context, high-volume RAG, and agentic workflows—creates a clear framework for build versus buy decisions. - [06:13](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=373s) Enterprises should buy commoditized Q&A tools and zero-grounding applications from third-party vendors, while focusing internal development on proprietary RAG architectures and custom data integrations. - [09:11](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=551s) GenAI infrastructure presents a pay-to-play trade-off between the rapid time-to-market and unpredictable OpEx of commercial APIs versus the high CapEx and predictable unit costs of on-premises hardware. - [09:11](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=551s) Unlike traditional machine learning where models were trained on GPUs and frozen to run on cheap CPUs, generative models strictly mandate dedicated GPU compute for production inference. - [09:11](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=551s) On-premises enterprise GPU infrastructure can deliver an estimated 30% to 50% total cost savings over purely API-driven consumption across a three-year operating horizon. - [12:50](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=770s) Enterprise GenAI observability must align with NIST risk dimensions—such as prompt injection, data leakage, and bias—using automated real-time telemetry to intercept unsafe outputs before they reach users. - [12:50](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=770s) Because generative models are non-deterministic, hallucinations remain the most difficult operational risk to manage, requiring a decoupling of LLM-as-a-judge scoring from traditional statistical baseline metrics. - [16:06](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=966s) Multi-turn agentic architectures require a centralized orchestration brain to maintain state and memory, as well as strict constraints and asynchronous calls to prevent latency bottlenecks and infinite tool-calling loops. - [16:06](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=966s) Interoperability across growing multi-agent ecosystems depends on open standards like Model Context Protocol (MCP) for tool context abstraction and Agent-to-Agent (A2A) protocols for inter-agent communication. - [18:43](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=1123s) Successful enterprise GenAI scaling demands an incremental implementation model anchored to tangible, measurable business metrics rather than attempting broad organizational transformations all at once. - [18:43](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=1123s) Sustainable production GenAI requires a hybrid hosting strategy, proactive GPU capacity planning, actionable governance to curb agent sprawl, and a human-in-the-loop culture of adoption.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=4s) Introduction and Enterprise GenAI Scaling Agenda

Matthew Berezo, Head of AI at PNC Bank, opens the session by addressing the core challenge facing modern enterprises: how to transition generative AI from experimental pilots into robust, production-grade infrastructure. Navigating this shift requires making high-stakes architectural and organizational decisions. To structure the discussion, Berezo outlines a five-part agenda designed to cover the major decision points every enterprise confronts on its GenAI roadmap.

The agenda begins with foundational scalability questions, specifically comparing cloud-hosted APIs against on-premises deployments, as well as evaluating large language models (LLMs) against small language models (SLMs). This transitions into infrastructure readiness—an examination of GPU capacity, procurement costs, and hardware availability across both R&D and production environments. From there, the presentation turns to observability and monitoring, focusing on risk mitigation in heavily regulated sectors like banking. The talk then explores agentic systems, highlighting emerging standards like agent-to-agent protocols and the Model Context Protocol (MCP), before concluding with a practical core principle roadmap for taking GenAI projects from initial conception to real-world deployment.

- **Five Core Agenda Topics:** - **Scalability:** Cloud API vs. on-prem hosting and LLM vs. SLM trade-offs. - **Infrastructure Readiness:** GPU sizing, cost modeling, and cluster availability for dev and production. - **Observability:** Risk assessment, compliance, and runtime monitoring in regulated enterprise environments. - **Agentic Systems:** Multi-agent architectures, agent-to-agent communication, and Model Context Protocol (MCP). - **Execution Roadmap:** Core principles for end-to-end GenAI lifecycle injection.

---

[02:21](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=141s) LLMs vs. SLMs: Latency, Efficacy, and Unit Economics

When evaluating models, Berezo adopts a "mutually exclusive, collectively exhaustive" (MECE) mindset, acknowledging that while the inherently generalizable nature of language models makes strict categorization challenging, the framework brings clarity to trade-offs. Large language models provide broad generalizability and baseline accuracy out of the box—the core value proposition that fueled the explosive rise of ChatGPT. However, deploying frontier LLMs at scale introduces three critical liabilities: latency, domain efficacy, and cost. Because LLMs represent massive compute footprints, inference latency increases substantially. On efficacy, commercial models from providers like Meta, OpenAI, or Perplexity are trained on broad public datasets that lack proprietary enterprise context. If a user asks a base LLM to define a niche PNC internal acronym, it lacks the context to provide an accurate answer, leading to hallucinations unless grounded.

Cost dynamics present another major bottleneck. Providers frequently market expanding context windows—"congratulations, your context window got bigger... everyone gets a context window"—allowing users to pass entire books into a prompt. However, API unit economics scale with token and character volumes for both input and output. Even if an enterprise runs a model with a massive context window on private infrastructure, it avoids API billing only to absorb the capital and operational expense of heavy GPU clusters.

```mermaid flowchart TD accTitle: Model Selection Trade-Off Framework subgraph Selection["Architectural Evaluation"] LLM["Large Language Models (LLMs)"] SLM["Small Language Models (SLMs)"] end

subgraph LLM_Factors["LLM Trade-Offs"] L1["High Generalizability"] L2["Higher Inference Latency"] L3["Token-Based Unit Cost Scaling"] L4["Hallucination on Proprietary Terms"] end

subgraph SLM_Factors["SLM Advantages"] S1["Lower Compute Footprint & Latency"] S2["Efficient Fine-Tuning on Custom Data"] S3["Lower Hosting & Hardware Costs"] S4["High Domain-Specific Efficacy"] end

LLM --> LLM_Factors SLM --> SLM_Factors ```

Small language models (SLMs) offer an alternative path for targeted enterprise workloads. With their smaller compute requirements, SLMs deliver lower latency and significantly lower hosting costs. Furthermore, their compact parameter footprint allows organizations to fine-tune them directly on internal proprietary corpora, achieving superior domain efficacy without relying entirely on massive general-purpose models. Enterprise architects must continuously balance latency, cost, and efficacy to determine the minimal viable model required for a task rather than defaulting to the largest model available.

- **LLM Trade-Offs:** - **Latency:** Massive parameter counts require larger compute instances, increasing time-to-first-token and overall response times. - **Efficacy & Hallucinations:** Training sets lack internal company data; base models fail on internal nomenclature without external grounding. - **Unit Economics:** Extended context windows scale API invocation costs and require outsized GPU memory footprints when hosted on-prem. - **SLM Advantages:** - **Efficiency:** Lighter compute profiles reduce response latency and hardware overhead. - **Customization:** Practical to fine-tune on internal documentation for domain-specific accuracy. - **Cost-Effective Scaling:** Eliminates third-party API volume charges and reduces infrastructure strain.

---

[06:13](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=373s) Build vs. Buy Framework and Four Deployment Patterns

To evaluate build-versus-buy decisions, Berezo introduces a four-tier classification model based on how an application handles data and grounding. This taxonomy provides technical and non-technical stakeholders with a shared mental model to assess risk, implementation effort, and vendor viability.

```mermaid flowchart LR accTitle: Four GenAI Deployment Patterns for Build vs Buy subgraph Patterns["Deployment Taxonomy"] P1["Pattern 1: Zero Grounding<br/>(General Knowledge / Base LLM)"] P2["Pattern 2: In-Prompt Context<br/>(Small Data / No Vector DB)"] P3["Pattern 3: Enterprise RAG<br/>(High-Volume Vector Repositories)"] P4["Pattern 4: Agentic Systems<br/>(Composite Multi-Step Logic)"] end

subgraph Strategy["Decision Path"] Buy["Buy (Vendor Solutions)"] Build["Build (Internal Engineering)"] end

P1 --> Buy P2 --> Buy P3 --> Build P4 --> Build ```

The first pattern encompasses tasks requiring zero grounding data, where the base pre-trained knowledge of an LLM suffices (such as asking the model to explain interest rates in layman's terms). The second pattern involves solutions requiring light contextual data that fits directly into the prompt without warranting a vector database, such as providing a job description and two resumes to evaluate candidate fit. The third pattern covers high-volume data retrieval, where massive internal knowledge repositories must be indexed into vector databases for semantic search and retrieval-augmented generation (RAG). The fourth pattern combines the previous patterns into complex, multi-step agentic workflows.

Using this taxonomy, organizations can apply a straightforward "smell test" to guide build-versus-buy investments. Building an internal, general-purpose Q&A chatbot from scratch offers low enterprise value because commercial vendors have already solved that pattern effectively. Conversely, when workflows require searching deep proprietary repositories that third-party vendors cannot ingest or support due to scale or privacy constraints, building a custom internal RAG architecture becomes essential. This decision naturally feeds into the hosting strategy: determining whether solutions should leverage managed cloud APIs or dedicated on-premises infrastructure.

- **The Four GenAI Deployment Patterns:** - **Pattern 1 (No Grounding Data):** Direct prompts relying purely on parametric model knowledge (e.g., standard educational or explanatory queries). - **Pattern 2 (In-Prompt Data):** Lightweight context passed directly within the prompt payload (e.g., comparing candidate resumes against a job description). - **Pattern 3 (High-Volume RAG):** Large-scale unstructured data indexed into vector databases for dynamic retrieval and contextual grounding. - **Pattern 4 (Agentic & Composite):** Dynamic orchestrations combining zero-shot prompts, in-memory context, and RAG across autonomous steps. - **Strategic Rule of Thumb:** Outsource commoditized Q&A and prompt-wrapper applications to vendors; prioritize in-house development for proprietary RAG pipelines and custom data integrations.

[09:11](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=551s) The "Pay-to-Play" Dilemma and GPU Infrastructure Strategy

Generative AI is fundamentally a "pay-to-play game"—organizations are either paying directly to consume commercial APIs or paying substantial capital to build and manage on-premises infrastructure. Leveraging API-based solutions offers an immediate speed-to-market advantage and rapid early wins, but it exposes the enterprise to volatile, consumption-based operational expenses that can lead to major surprise bills down the road. Conversely, an on-premises footprint provides precise cost predictability and compute consistency, though it demands navigating extreme infrastructure complexity and paying hefty upfront capital expenditures.

```mermaid flowchart TD accTitle: Trade-offs between API and On-Premises GenAI Infrastructure

subgraph API["API-Based Strategy"] A1["Fast Time to Market"] --> A2["Quick Wins"] A2 --> A3["Unpredictable Variable Costs"] end

subgraph OnPrem["On-Premises Infrastructure"] B1["High Upfront CapEx"] --> B2["Complex Deployment"] B2 --> B3["Cost & Compute Control"] end ```

The hardware backbone of GenAI relies entirely on specialized GPUs equipped with massive virtual memory and clustered compute nodes. Reflecting on his initial hands-on deep learning project in October 2015, Matthew Berezo quipped that he became keenly aware of Nvidia back then and "wished that I knew then what I know now." Today, heavy workloads standardly require clusters of Nvidia A100 or H100 GPUs, though model quantization and compression techniques offer a practical path to shrink footprints and reduce hardware requirements in production.

Building an adequate infrastructure strategy must address both research and development (R&D) and production environments. In development, data scientists and engineers often end up in shared environments "fighting with one another and competing for compute," which grinds rapid prototyping and experimentation to a halt. In production, GenAI breaks traditional machine learning paradigms: historically, teams trained deep learning models on GPUs and then froze them to run inference cheaply on standard CPUs. Generative models, by contrast, strictly require dedicated GPU compute for production inference. While an enterprise server with eight Nvidia H100 GPUs can cost around $800,000 out of the gate, industry analyses suggest that on-prem deployments can yield 30% to 50% cost savings over API usage over a three-year horizon. As a result, most mature enterprises adopt a hybrid model that balances on-prem capacity with API endpoints.

- **Infrastructure & Cost Benchmarks**: - Enterprise 8x Nvidia H100 servers run approximately $800,000 upfront. - On-premises infrastructure can deliver an estimated 30% to 50% total cost savings compared to pure API usage across a 3-year timeline. - Model quantization enables smaller deployment footprints by compressing model parameters to run on fewer GPUs.

---

[12:50](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=770s) GenAI Observability: Measuring the Seven NIST Risk Pillars

Observability in enterprise GenAI must be grounded in the National Institute of Standards and Technology (NIST) risk framework. To safely scale, every core risk must be made fully observable, measurable, and actionable through real-time telemetry, statistical methods, and automated evaluation frameworks:

```mermaid flowchart LR accTitle: Seven Key NIST GenAI Risks subgraph NIST["NIST Risk Dimensions"] direction TB R1["Prompt Injection"] R2["Hallucinations"] R3["Data Leakage & Privacy"] R4["Harmful Content"] R5["Inadvertent Bias"] R6["IP & Copyright"] R7["Data Quality"] end

NIST --> OBS["Observability Layer"] OBS --> ACT["Real-Time Actions & Mitigation"] ```

- **Prompt Injection**: Demonstrated by the 2022 Reddit jailbreak that forced ChatGPT into the "DAN" (*Do Anything Now*) persona to bypass safety boundaries and output harmful, non-compliant responses. While vendors have patched many loopholes, continuous monitoring remains necessary. - **Hallucinations**: Described as the "long pole in the tent" because GenAI is non-deterministic rather than supervised machine learning; best evaluated by decoupling LLM-as-a-judge scoring from traditional statistical baseline metrics. - **Data Leakage and Privacy**: Tracking whether end users submit sensitive information, whether the model outputs sensitive enterprise data, or whether foundational assets inadvertently train on protected inputs. - **Harmful and Non-compliant Content**: Overlapping with injection risks, requiring strict guardrails against improper generative outputs. - **Inadvertent Bias**: Addressing systemic skews where biased input datasets inherently produce biased outputs. - **IP and Copyright Risk**: Verifying that responses do not incorporate proprietary third-party intellectual property or copyrighted materials without authorization. - **Data Quality**: Ensuring grounding sources in retrieval systems remain authoritative, current, and comprehensive.

Embedding these evaluations into production enables automated, real-time intervention—such as catching and mitigating non-compliant or hallucinated queries before a problematic answer reaches the user.

---

[16:06](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=966s) Agentic Frameworks: Orchestration, State Management, and Open Protocols

Unlike simple, single-turn conversational question-and-answering pipelines that merely return a canned response, agentic systems are endowed with goals, initiative, and the autonomy to accomplish tasks across five, ten, or twenty distinct execution turns. This operational pattern mandates a central orchestration "brain" that maintains continuous state: tracking the full conversational context, intermediate logical deductions, executed tool calls, and final outputs returned to the user.

```mermaid flowchart TD accTitle: Agentic Architecture and Standard Protocols

subgraph AgenticCore["Agentic Core"] Brain["Orchestration Brain / State Manager<br/>(History, Conclusions, Memory)"] Agent["AI Agent"] Brain <--> Agent end

subgraph ContextTools["Context & Execution Layer"] MCP["Model Context Protocol (MCP)"] Tools["Tools & Data<br/>(Databases, Code, APIs, Models)"] Agent --> MCP --> Tools end

subgraph MultiAgent["Multi-Agent Layer"] A2A["Agent-to-Agent (A2A) Protocol"] OtherAgent["External AI Agent"] Agent <--> A2A <--> OtherAgent end ```

Agents must be equipped with external tools—such as databases, web APIs, custom code interpreters, and auxiliary machine learning models—to act upon their working memory. However, giving autonomy to agents introduces severe execution hang-ups and looping traps. For instance, an unconstrained agent instructed to solve a problem at all costs might ping an internal database 100 times in an infinite loop when it fails to find an answer. Preventing these latency bottlenecks requires engineering solutions such as asynchronous calling, strict prompt boundaries, and well-defined operational "personalities" that constrain the agent to narrow responsibilities.

As an organization builds multiple specialized agents, connecting them via ad-hoc, custom-built integrations quickly becomes unmaintainable. Interoperability at scale relies on emerging open standards: - **A2A (Agent-to-Agent)**: A proposed common communication language enabling distinct agentic systems to execute handshakes, negotiate tasks, and collaborate without point-to-point custom code. - **MCP (Model Context Protocol)**: A standardized interface that abstracts how agents access context and external tooling, funneling data cleanly through context providers rather than hardwiring disparate data sources directly to the model.

[18:43](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=1123s) Core Principles for Scaling GenAI and Agentic Systems

Wrapping up the discussion on enterprise GenAI architecture, Matthew Berezo synthesizes the journey from pilot to production into a foundational principle framework. First and foremost is anchoring any initiative in tangible business value. Organizations must ground their first generative or agentic solution in a well-defined business need with clear, measurable metrics for success. Rather than attempting to "eat the elephant as a whole," teams should take an incremental approach—tackling the problem piece by piece to secure quick wins, gather essential feedback, and learn continuously along the way.

When deciding between building custom infrastructure or buying existing capabilities, enterprises must be deliberate and recognize that most production landscapes will ultimately be hybrid. Managed API-driven solutions provide a rapid time to market but introduce ongoing variable operational costs. Conversely, on-premises deployments tend to be far more cost-effective over the long run, though they come with substantial upfront complexity and friction during setup. For on-premises environments, planning and scaling GPU capacity in lockstep with business growth is critical to avoid infrastructure bottlenecks.

```mermaid flowchart TD accTitle: Core Framework for GenAI Scaling subgraph Value ["1. Business Value"] V1["Define Clear Business Need"] --> V2["Measure Success & Iterate Piece by Piece"] end subgraph Strategy ["2. Architecture & Infrastructure"] S1["Build vs. Buy Evaluation"] --> S2["Hybrid Model: API (Speed) & On-Prem (Cost/Scale)"] S2 --> S3["Proactive GPU Capacity Planning"] end subgraph Operations ["3. Governance & Culture"] O1["Actionable Governance Framework"] --> O2["Prevent Agent Sprawl & Drift"] O3["Culture of Adoption"] --> O4["Human-in-the-Loop Iteration"] end ```

Long-term sustainability relies heavily on establishing a governance framework that is "clear, communicable and actionable." Proactive governance ensures that agentic workflows and automated models do not become unmanageable sprawl over time. Finally, scaling AI across the enterprise requires fostering a strong culture of adoption among both technical and non-technical stakeholders. Keeping humans in the loop to iterate side-by-side with agentic systems ensures that teams extract the maximum value from their AI infrastructure.

- **Derive Measurable Business Value:** Start small on high-impact use cases rather than attempting full-scale transformations all at once. - **Architect Hybrid Infrastructure:** Balance the rapid deployment speed of external APIs with the long-term cost efficiencies and capacity controls of on-premises GPU compute. - **Implement Actionable Governance:** Enforce structured oversight from day one to maintain agent maintainability and compliance. - **Foster Human-in-the-Loop Adoption:** Bridge technical and non-technical users to iteratively refine agentic workflows across the organization.

## Notable quotes > [02:21](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=141s) mutually exclusive, collectively exhaustive — Matthew Berezo

> [02:21](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=141s) congratulations, your context window got bigger... everyone gets a context window — Matthew Berezo

> [06:13](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=373s) smell test — Matthew Berezo

> [09:11](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=551s) pay-to-play game — Matthew Berezo

> [09:11](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=551s) wished that I knew then what I know now. — Matthew Berezo

> [09:11](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=551s) fighting with one another and competing for compute — Matthew Berezo

> [12:50](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=770s) DAN — Matthew Berezo

> [12:50](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=770s) Do Anything Now

> [12:50](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=770s) long pole in the tent — Matthew Berezo

> [18:43](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=1123s) eat the elephant as a whole — Matthew Berezo

> [18:43](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=1123s) clear, communicable and actionable — Matthew Berezo

## Mentioned Matthew Berezo, PNC Bank, ChatGPT, Meta, OpenAI, Perplexity, Nvidia, Nvidia A100, Nvidia H100, NIST (National Institute of Standards and Technology), Reddit, Model Context Protocol (MCP), Agent-to-Agent (A2A)

## Video description Matthew Berezo, Head of AI, PNC

Abstracts

What happens when your GenAI pilot becomes a core business function overnight?

As organizations rush to productionize generative AI (GenAI), they're discovering that traditional AI infrastructure patterns don't necessarily scale under the weight of large models, real-time latency demands, and responsible AI constraints.

In this session, we'll explore the realities of scaling GenAI in enterprise environments, including GPU planning, architectural trade-offs, and overall strategy.

Attendees will leave with: - A clear mental model for GenAI infrastructure stacks and scalability levers. - Practical insights into cost control, model deployment, and monitoring strategies.

Why attend? Whether you're architecting your first GenAI application or evolving your current stack, this talk delivers actionable guidance grounded in real-world enterprise experience - helping you avoid common pitfalls and design for scale from day one.

[Source](https://www.youtube.com/watch?v=DOIIw4JOqc0)

中文简报

PNC AI负责人称本地GPU算力可将三年API成本减半 将企业生成式AI从云端API迁移至私有GPU集群，可在三年内将基础设施开支降低多达50%，同时降低延迟和幻觉风险。 Matthew Berezo概述了一个将GenAI融入核心业务运营的结构化企业框架，权衡通用LLM与领域特定SLM在延迟、成效及单位经济效益方面的取舍。他提出了一个四层部署分类法以指导自研与采购决策，并详细阐述了API调用消耗与本地部署GPU集群之间的成本动态。讨论还确立了映射至NIST七大风险支柱的健全可观测性实践，并探讨了利用MCP和A2A协议等新兴标准的多智能体架构。最终，Berezo倡导以明确业务指标和前瞻性治理为依托，采用渐进式的混合基础设施路径。 PNC AI 负责人 Matthew Berezo 提出将 GenAI 从试点推向企业级生产需聚焦五大支柱：可扩展性、基础设施就绪度、可观测性、Agent 系统及落地路线图。 (0:04) 前沿 LLM 虽具备通用性，但存在延迟高、缺乏专有语境及 Token 成本递增等缺陷；小语言模型（SLM）更适用于垂直业务，推理延迟更低且更利于基于内部数据微调。 (2:21) 基于四种落地模式评估“自研还是采购”：企业应直接采购商品化的问答工具与零上下文应用，并集中自研力量构建专有 RAG 架构与定制数据集成。 (6:13) 商业 API 具备快速上市优势但 OpEx 波动大，而本地 GPU 基础设施虽前期 CapEx 高昂，但在三年周期内预计可比纯 API 消耗节省 30% 至 50% 的总成本。 (9:11) 规模化扩展企业级 Agent 需依赖中央编排大脑进行状态管理，并借助 A2A 协议与模型上下文协议（MCP）等开放标准防范死循环并实现跨 Agent 互操作。 (16:06)

Reading summary

Matthew Berezo, Head of AI at PNC Bank, argues that scaling generative AI from pilots to enterprise production requires navigating critical trade-offs between model efficiency, hardware economics, strict governance, and agentic orchestration. To avoid unsustainable API expenses and operational sprawl, enterprises must pair targeted model selection with hybrid infrastructure and standardized evaluation frameworks. Model Sizing and Build-Versus-Buy Framework (2:21) When selecting architectures, Berezo contrasts frontier large language models (LLMs) with small language models (SLMs). While LLMs provide broad generalizability out of the box, they introduce high inference latency, scaling token costs via expanding context windows, and hallucinations on proprietary enterprise terminology. Conversely, SLMs feature smaller compute profiles and can be fine-tuned directly on internal company data, delivering higher domain efficacy at lower hosting expenses. Organizations should identify the minimum viable model required for a task rather than defaulting to the largest base model. To structure investment decisions, Berezo introduces a four-tier deployment taxonomy based on grounding requirements. Pattern 1 relies on zero grounding data and base model parametric knowledge, while Pattern 2 passes lightweight context directly into prompt payloads without external retrieval. Pattern 3 utilizes high-volume enterprise retrieval-augmented generation (RAG) backed by vector databases, and Pattern 4 orchestrates multi-step agentic workflows. Under this heuristic, Berezo advises enterprises to buy commercial vendor solutions for commoditized Q&A and prompt-based tasks, while building proprietary internal pipelines for deep RAG architectures and sensitive data workflows. Infrastructure Strategy and On-Premises GPU Economics (9:11) Berezo frames generative AI as a "pay-to-play" operational decision between variable API consumption costs and capital-intensive on-premises infrastructure. Relying purely on cloud APIs accelerates early deployment but exposes enterprises to unpredictable, volume-driven operational expenditures. Private hosting provides compute consistency and long-term cost stability, but it introduces significant setup friction and requires dedicated GPU capacity across both development and production environments. Unlike traditional machine learning, which frequently trained on GPUs and deployed inference onto commodity CPUs, generative AI requires dedicated GPU compute in production. Hardware procurement presents substantial capital hurdles; an enterprise server equipped with eight Nvidia H100 GPUs costs roughly $800,000 upfront. However, industry analyses cited by Berezo project that on-premises deployments can yield 30% to 50% total cost savings over commercial APIs over a three-year horizon. To manage physical constraints and avoid resource contention among development teams, organizations can deploy model quantization to compress parameters onto smaller GPU footprints, while adopting a hybrid architecture that pairs base API endpoints with internal infrastructure. Risk Observability Aligned With NIST Standards (12:50) Deploying GenAI in regulated enterprise environments requires continuous, telemetry-driven observability mapped against the National Institute of Standards and Technology (NIST) risk framework. Berezo identifies seven distinct risk pillars requiring real-time measurement: prompt injection attacks, hallucinations, data leakage and privacy breaches, harmful or non-compliant content, inadvertent bias, intellectual property and copyright violations, and poor grounding data quality. Because generative systems are non-deterministic, Berezo characterizes hallucinations as the primary scaling obstacle ("the long pole in the tent"), advocating for evaluation pipelines that decouple LLM-as-a-judge scoring from traditional statistical baseline metrics. Real-time telemetry must monitor whether users submit confidential information, whether models output protected intellectual property, and whether grounding corpora remain authoritative. Automated runtime guardrails can intercept problematic or non-compliant generation before responses reach end users. Agentic Orchestration and Open Interoperability Protocols (16:06) Moving beyond single-turn question-answering pipelines, agentic systems operate autonomously over extended multi-turn interactions. This paradigm requires an orchestration layer that maintains continuous state—tracking working memory, logical deductions, tool calls, and final outputs. To prevent operational bottlenecks, such as unbounded agents pinging internal databases in infinite execution loops, engineering teams must establish strict prompt boundaries, asynchronous calling, and narrowly defined operational roles. As enterprises deploy specialized multi-agent architectures, bespoke point-to-point connections become unmaintainable. Berezo points to emerging open standards to solve interoperability: the Agent-to-Agent (A2A) protocol for standardizing handshakes and task negotiation between disparate agents, and the Model Context Protocol (MCP) for abstracting how models access external tools, APIs, and enterprise databases. Berezo concludes that sustainable enterprise scaling depends on grounding initiatives in measurable business value rather than broad technological overhauls. Organizations must balance the speed of APIs with the long-term economics of on-premises GPU capacity, backed by actionable governance frameworks and human-in-the-loop oversight to prevent agent sprawl as systems mature.

精炼阅读摘要

PNC银行人工智能负责人Matthew Berezo指出，企业将生成式AI从实验试点推进至生产级基础设施，必须在模型选型、架构自建与外购、算力部署及风险监管等维度做出关键权衡。他主张采取以明确业务价值为锚点、兼顾云端API与本地GPU集群的混合架构路线，并依托标准化协议与可观测体系安全扩展智能体系统。 模型选型权衡：大语言模型与小语言模型 (2:21) Berezo采用相互独立、完全穷尽（MECE）的框架评估模型选型，指出以ChatGPT为代表的前沿大语言模型（LLM）具备出色的通用泛化能力与开箱即用的基准精度，但规模化落地会带来高推理延迟、领域效能不足和单位调用成本飙升三大制约。基础LLM缺乏企业私有上下文，在面对特定业务术语时极易引发幻觉；服务商虽不断扩大上下文窗口，但API计费随输入与输出的Token数量线性增长，即便在私有基础设施上运行超大窗口模型，也会消耗庞大的GPU集群算力与运维资本。 相比之下，小型语言模型（SLM）因参数体量精简，能显著降低计算开销与响应延迟，并大幅压缩托管和硬件成本。更重要的是，企业能够以较低门槛基于内部私有语料对SLM进行针对性微调，从而在垂直业务场景中取得超越通用模型的准确率。架构师应秉持最小可行模型原则，按任务需求匹配适量算力，避免盲目默认调用参数规模最大的模型。 自建与外购：四种落地模式与选型路径 (6:13) 为厘清企业级生成式AI的自研与采购边界，Berezo提出了依据数据处理深度与上下文增强模式划分的四层分类框架。第一种模式为“零提示增强”，完全依托模型预训练参数回答通用常识；第二种模式为“提示词内上下文”，将轻量数据直接嵌入提示词中传递；第三种模式为“企业级RAG”，需将海量非结构化知识库进行向量化索引，以支持语义检索；第四种模式则是涉及多步骤自主推理的复合“智能体系统”。 这一分类为企业的投资决策提供了明确基准。对于通用问答、简历对比等缺乏私有数据沉淀的标准场景，市场上已有成熟的商业化产品，企业自建的边际价值极低；而一旦应用涉及检索企业深度私有知识库，受限于第三方供应商无法合规摄入或承载大规模私密数据，企业必须集中工程力量自主构建底层RAG架构与私有数据通道。 算力经济学：API与本地GPU集群博弈 (9:11) 生成式AI本质上是一场“付费入场”的长期博弈，企业必须在商业API的高昂调用费与自建本地算力的资本支出之间寻找平衡。调用托管API能够帮助业务快速抢占市场并收获早期成效，但基于调用量的可变运营支出波动极大，存在后期账单失控的风险；本地基础设施虽需承担昂贵的初始采购资本和复杂的部署流程，却具备极高的算力掌控力与成本可预测性。 在底层硬件方面，当前的生产级生成任务高度依赖以Nvidia A100或H100为核心的高显存GPU集群。Berezo援引行业数据指出，配备8张Nvidia H100的单台企业级服务器购置成本约为80万美元，但若拉长到3年的全生命周期来看，本地基础设施相比纯API调用可实现约30%至50%的整体成本节约。此外，生产环境不同于传统机器学习在CPU上低成本推理的模式，企业必须通过模型量化与参数压缩技术缩小模型体积，并为研发测试与生产环境建立隔离配额，防止资源争夺导致业务停滞。 风险治理：对齐NIST框架的实时可观测体系 (12:50) 在银行业等高度合规场景中，生产级规模化部署的前提是将潜在风险转化为可观测、可度量的具体工程指标。Berezo主张将运行时可观测体系对齐美国国家标准与技术研究院（NIST）的七大风险支柱：提示词注入、模型幻觉、数据泄露与隐私侵犯、有害及违规内容、算法偏见、知识产权与版权风险，以及底层数据质量。 针对生成式系统非确定性所带来的模型幻觉痛点，架构方案应将“LLM作为裁判”（LLM-as-a-judge）的自动化打分与传统统计基线指标相解耦。生产系统必须建立端到端的遥测管道，实时捕捉并拦截提示词越狱（如绕过安全限制的DAN攻击）与非合规生成内容，确保在有害回答或受保护敏感信息触达终端用户之前完成自动化阻断与缓解。 智能体编排：状态管理与开放互联协议 (16:06) 不同于仅返回预设答案的单轮对话问答，具备自主目标的智能体（Agent）系统能够通过数十轮执行来拆解并完成复杂长链任务。这一模式要求架构中配备核心编排“大脑”进行持续的状态管理，全流程跟踪对话上下文、中间逻辑推导、工具调用凭证与最终交付物。为防止自主智能体在未检索到答案时无休止地向内部数据库发起请求并陷入死循环，工程上必须引入异步调用、严格的提示词边界及单一职责的执行角色限制。 随着企业内部垂直智能体数量激增，点对点的点状定制代码将迅速导致系统失去可维护性。Berezo强调推进两项开放协议标准：一是智能体间协议（A2A），用于建立不同智能体跨系统握手、任务协商与协同分工的通用通信语言；二是模型上下文协议（MCP），作为统一的上下文接入规范，将数据库、代码解释器与外部API的调用抽象化，彻底消除模型与底层多源数据的硬编码绑定。 Berezo总结指出，企业推进生成式AI不能试图一次性解决所有问题，而应以明确的业务痛点与可衡量指标为起点，采取渐进式迭代路线。尽管本地GPU集群在三年周期内呈现出显著的成本效益，但多数企业最终将采用API敏捷性与本地可控性并存的混合架构。长期的可扩展性最终取决于能否建立清晰可执行的治理框架，并在智能体编排全流程中保持“人在回路”的协作机制，从而防范智能体失控蔓延。

Detailed notes — zh

## Background 本内容由 Ai4 频道于 2025 年 8 月 24 日在 YouTube 平台发布，主讲人为 PNC 银行人工智能负责人 Matthew Berezo。视频围绕企业级生成式 AI 如何从原型试点跨越至生产就绪架构展开，深入剖析了 GPU 算力规划、架构权衡、成本控制与负责任 AI 治理等核心议题。 ## TL;DR 演讲系统梳理了企业规模化落地 GenAI 的关键路径与五大核心议题。Berezo 深入对比了大语言模型（LLM）与小语言模型（SLM）在延迟、有效性与成本上的博弈，并提出了基于数据接地程度的四种部署模式与自研/采购决策树。同时，针对生产环境 GPU 算力规划、NIST 七大可观测性风险治理，以及结合 MCP 与 A2A 协议的现代智能体（Agentic）架构演进，给出了扎根业务价值与人机协同的落地框架。 ## Atmosphere & style 整场演讲呈现出典型金融级工程专家的严谨与务实底色，叙事条理极强且紧贴企业真实痛点。主讲人语调沉稳自信、逻辑层层递进，既有对技术概念的清晰拆解，也不乏对行业算力军备竞赛的幽默洞察。整体给人的观感像是一场毫无浮夸营销、干货密度极高且直击生产现实的企业级 AI 架构实战指南。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=4s) 企业级 GenAI 落地需要统筹考虑可扩展性、算力基础设施就绪度、系统可观测性与合规风险、智能体架构演进以及原则路线图五大核心议题。 - [02:21](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=141s) 大语言模型（LLM）虽具备高通用性，但在企业级落地中面临高延迟、缺乏专有语料导致幻觉以及长上下文/API 计费高昂三大痛点。 - [02:21](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=141s) 小语言模型（SLM）具备响应延迟低、易于使用私有语料微调且本地算力开销小等反转优势，构建企业 MVP 不必盲目追求最大最复杂的模型。 - [06:13](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=373s) 依据数据接地程度，GenAI 可划分为零数据接地、轻量即时数据、大规模 RAG 和复合智能体四种模式，通用场景倾向采购成熟 SaaS，核心私有数据链路宜坚决自研。 - [09:11](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=551s) API 模式胜在产品上市时间快但调用成本不可控；本地私有化部署前期硬件投入巨大，但 3 年 TCO 测算可比商业 API 节省 30% 至 50%，企业主流形态多为混合架构。 - [10:54](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=654s) 生成式 AI 颠覆了传统深度学习“仅训练依赖 GPU、推理可在 CPU 运行”的范式，其生产环境推理仍高度依赖高性能 GPU 持续支撑。 - [12:50](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=770s) 企业必须基于 NIST 框架建立对提示词注入、幻觉、数据泄露、有害内容、偏见、知识产权风险和数据质量七大维度的全生命周期可观测与实时拦截治理。 - [16:06](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=966s) 智能体系统（Agentic Systems）需要具备编排大脑与状态管理能力，并需引入异步调用或约束机制防止模型陷入死循环。 - [16:06](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=966s) A2A（Agent-to-Agent）协议与 MCP（Model Context Protocol）标准化协议为企业跨智能体协作及标准化工具/上下文接入提供了通用接口。 - [18:43](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=1123s) GenAI 落地必须以业务价值为导向从小处切入快跑迭代，同步建立清晰治理框架以防 Agent 蔓延失控，并依托人机协同（Human-in-the-loop）释放长期红利。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=4s) 演讲开场与企业级 GenAI 落地五大议题概览 PNC 银行人工智能负责人 Matt Berezo 向与会者致意，并直奔主题：如何在企业级层面将生成式人工智能（GenAI）从前期的概念验证（Pilot）阶段扩展至生产就绪（Production-Ready）的基础设施。对于任何踏上 GenAI 旅程的企业而言，这都不是一个简单的技术尝鲜，而是面临着极其沉重的决策负担与架构权衡。为此，Berezo 梳理了一份清晰明了的五步议程，涵盖了企业在规模化扩展中必须直面的核心命题： 议程的第一部分聚焦于“可扩展性（Scalability）”，也就是当前业界讨论最激烈的大命题——到底应该选择云端 API 模式，还是自建本地机房（On-prem）部署？同时，这也牵涉到大语言模型（LLM）与小语言模型（SLM）之间的多维度权衡。第二部分顺理成章地切入“基础设施就绪度（Infrastructure Readiness）”，重点剖析 GPU 算力容量、成本开销与资源可用性；尽管本地部署具备巨大的潜在收益，但企业必须回答一系列实质性问题：算力底座现存多少、未来需要多少、采购成本几何，以及如何保障研发与生产环境之间的 GPU 调度顺畅。第三部分转向高度监管行业尤为敏感的“可观测性与监控（Observability）”，强调 GenAI 不仅是令人惊艳的软件技术，更关乎在全流程中识别、监控和缓解模型风险。第四部分则是当下火爆的“智能体系统（Agentic Systems）”，特别是随着智能体间协议（Agent-to-Agent Protocol）和模型上下文协议（Model Context Protocol / MCP）等标准的演进，智能体架构正迎来爆发。最后，演讲将汇总出一套核心原则路线图，帮助企业建立一套从构想到落地的结构化框架。 - **核心演讲者**：Matt Berezo，PNC 银行人工智能负责人（Head of AI at PNC Bank）。 - **五大议题脉络**： 1. 可扩展性与大/小模型架构选型（Scalability: Cloud vs. On-prem, LLM vs. SLM） 2. 算力基础设施就绪度（Infrastructure Readiness: GPU capacity, cost, availability） 3. 系统可观测性与合规风险缓解（Observability, risk mitigation） 4. 智能体架构演进（Agentic systems, MCP, Agent-to-Agent protocols） 5. 从构想到生产的原则性路线图（Core principle roadmap） --- ### [02:21](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=141s) 扩展性权衡：大语言模型（LLM）与小语言模型（SLM）的较量 在深入技术细节前，Berezo 指出他倾向于采用“完全独立、穷尽无遗漏”（MECE，Mutually Exclusive, Collectively Exhaustive）的思维框架来进行技术评估。不过他也坦言，在 GenAI 领域做到绝对 MECE 极其困难，因为大语言模型天生具备极强的“通用性（Generalizability）”，不同概念间难免存在交叉重叠。从大模型与小模型的选型来看，以 ChatGPT、Meta Llama 系列或 Perplexity 为代表的 LLM 固然能力卓越、通用性强且基准准确率高，但这背后隐藏着企业级落地不可忽视的三大痛点：延迟（Latency）、有效性（Efficacy）和成本（Cost）。 ```mermaid flowchart TD accTitle: LLM与SLM的多维对比架构 accDescr: 对比大语言模型和小语言模型在延迟、有效性及成本上的权衡 subgraph LLM["大语言模型 (LLM)"] L1["大计算实例 / 延迟偏高"] L2["预训练语料黑盒 / 内部专有名词易幻觉"] L3["长上下文 Token 计费或巨额机房基建"] end subgraph SLM["小语言模型 (SLM)"] S1["轻量级计算 / 极低延迟响应"] S2["针对专有语料高效微调 / 业务有效性高"] S3["私有化部署算力开销小 / 经济效益优"] end LLM --- vs{{"架构与场景权衡"}} --- SLM ``` 在延迟方面，LLM 庞大的参数体量决定了它是一个极其消耗算力的庞然大物，计算负载越重，实时交互中观测到的延迟就越高。在模型有效性方面，市面上的公有大模型基本都训练在外界无法获知的公开海量语料上，这导致它们在缺乏企业上下文时极易出现幻觉。Berezo 举例道：“如果我在 PNC 内部开会，有人随口报出一个冷门的行内缩写词，我如果直接丢给 ChatGPT 去查，它很可能无法给出正确的业务含义，因为模型根本没有我们的语料背景。”而在成本维度上，厂商们常常以“祝贺你，上下文窗口又翻倍了，每个人都能塞进更长的 Prompt”为卖点，虽然长上下文窗口让用户能从以前塞一本书扩展到现在塞两本书，但一旦采用云端 API，企业每多发一个 Token 或字符都在按量付费；即使是在本地部署千万级 Token 上下文的模型，也必须为此承担极其昂贵的基础设施硬件开支。 相比之下，小语言模型（SLM）展现出了独特的反转优势。由于计算实例小，SLM 的响应延迟更低；企业可以非常经济高效地利用内部专有语料对其进行针对性微调（Fine-tuning），在特定业务场景下获得远超通用大模型的精准度和有效性。同时，SLM 通常不依赖外部收费 API，对本地计算资源的占用也相对温和。Berezo 强调，构建满意的最小可行产品（MVP）并不意味着“任何时候都要上最大、最复杂的模型”，在延迟、成本与有效性之间寻找最佳平衡点，保持架构的敏捷性，才是企业工程化的正道。 - **LLM 的三大落地阻碍**： - **延迟（Latency）**：高参数量带来高推理耗时。 - **有效性与幻觉（Efficacy & Hallucinations）**：缺乏专有上下文支撑，处理企业内部缩写或专有术语时表现欠佳。 - **单位经济学成本（Unit Economics & Cost）**：长上下文窗口虽好，但 API 模式下按 Token 计费激增；本地部署则面临极高算力基建摊销。 - **SLM 的战略价值**：计算体量轻便、延迟低、微调成本可控、场景化精准度高，尤其适合作为智能体子模块灵活嵌入。 --- ### [06:13](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=373s) 自研还是采购？基于数据接地程度的四种部署模式 在权衡自研（Build）还是外采（Buy）时，核心判断标准归结为一点：“该方案需要注入什么样的数据？”基于数据接地的程度，Berezo 梳理了四种 GenAI 部署分类模式。这套模式不仅易于技术人员理解，也让非技术人员能够清晰厘清各个方案对应的风险画像与研发路径。 ```mermaid flowchart LR accTitle: GenAI 方案的四类数据依赖分类 accDescr: 展示从无上下文到复杂智能体架构的四种落地分类 D1["模式 1: 无需专有数据<br/>(基础通识问答)"] D2["模式 2: 少量即时数据<br/>(Prompt上下文比对)"] D3["模式 3: 海量专有数据<br/>(向量数据库与RAG)"] D4["模式 4: 复合复杂系统<br/>(智能体/Agentic)"] D1 --> D2 --> D3 --> D4 ``` 第一类是最简单的**无专有数据依赖方案**，模型无需任何额外上下文即可给出令人满意的回答。例如在银行业务中提问“请用通俗语言解释什么是利率”，通用大模型凭借现存知识库即可完美解答。第二类是**轻量级数据依赖方案**，需要一定上下文辅助，但数据量远未达到需要引入向量数据库的程度。典型场景如“将一份招聘要求与两份候选人简历输入给大模型，让其评估哪位候选人更匹配”，直接利用 Prompt 注入少量上下文即可完成。第三类是**海量专有数据依赖方案**，面对庞大的企业内部知识库，必须对文档进行向量化切块并存入向量数据库，以便在用户提问时进行高效的相似度检索并召回精准上下文，即典型的检索增强生成（RAG）。第四类则是前述三者的任意组合与复杂化延伸，即高度自动化的**智能体架构（Agentic）**。 通过这四种模式，团队可以快速进行自研或采购的“压力测试（Pressure Test）”： - **采购场景**：如果需求仅仅是一个通用型 Q&A 问答机器人（模式 1），企业完全没有必要自建并承担繁琐的后续维护，直接购买成熟的成熟供应商产品性价比最高。 - **自研场景**：如果需求涉及检索企业内部高敏感的核心专有资产（模式 3），而外部供应商要么不支持向量库集成，要么无法承载行内海量知识库的并发与规模，企业就需要坚决自建定制化的 RAG 系统。 最后，这一切又与托管策略紧密相连：究竟是选择将系统架设在本地数据中心（On-prem），还是直接调用公有云厂商的托管 API，构成了企业落地架构设计最底层的分水岭。 - **四类部署模式（Deployment Patterns）**： 1. **零数据接地**：通用通识推理（如基础金融术语科普）。 2. **轻量即时数据**：单次 Prompt 注入短文档（如简历与职位描述横向比对）。 3. **大规模数据 RAG**：向量化入库、语义搜索与上下文精准拼接。 4. **复合智能体（Agentic）**：动态组合多数据源与自动化行动链条。 - **决策经验法则**：通用能力买成熟 SaaS，核心私有数据与特定检索链路自研定制。 ### [09:11](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=551s) 算力基建抉择：API 与本地部署的成本博弈及 GPU 需求 生成式 AI 从本质上来看是一场“花钱入局的游戏”（"pay-to-play game"）——企业要么花钱调用商业 API，要么花钱自建基础设施摆脱对 API 的依赖，两者各有优劣。采用 API 模式最大的优势在于上市时间快，能够迅速拿到业务成果（Quick win）；但长期来看，不可控的调用量会导致变动成本剧增，最终让企业面临在研发或部署阶段完全未曾预料到的高额账单。相比之下，本地化部署（On-Premises）能提供更强的成本控制力和算力一致性，但其前期硬件投入巨大且集群搭建极其复杂。 随着芯片股的持续飙升，硬件算力已成为推动生成式 AI 落地的核心催化剂。回顾自身的经历，演讲者提到自己在 2015 年 10 月初次接触 GPU 并进行深度学习建模时，就已经深知英伟达（NVIDIA）的存在，甚至调侃道：“真希望我当时就知道现在会发生什么，那样我就能抓住机会了。”现代大语言模型（LLM）不仅体积庞大，而且对 GPU 显存有着严苛的要求，无论是研发还是生产阶段，都需要依赖边缘节点或 GPU 节点集群来承载高并发负载。 - **主流硬件选择**：在企业级硬件选型中，NVIDIA A100 与 H100 GPU 往往是满足大模型吞吐与低延迟标准的主力设备。 - **模型量化（Quantization）路径**：通过模型量化技术对权重进行有效压缩，使模型体积缩小，是降低生产环境 GPU 显存需求与硬件开销的可行路径。 --- ### [10:54](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=654s) 研发与生产环境的 GPU 架构：从传统深度学习到大模型的范式转移 构建稳健的算力基础设施不仅关乎生产环境，对研发（R&D）阶段同样生死攸关。在许多企业的开发场景中，常会出现多名工程师处于同一环境中互相争夺算力资源的现象，这会导致快速原型设计与模型迭代陷入停滞。因此，研发端必须配备充足的 GPU 资源以保障敏捷试错。 ```mermaid flowchart TD accTitle: 传统深度学习与生成式 AI 算力架构对比 subgraph DL["传统深度学习架构 (Deep Learning)"] DL_Train["训练阶段 (Training)<br/>重度依赖 GPU 集群"] --> DL_Freeze["模型冻结与优化<br/>(Freeze & Export)"] DL_Freeze --> DL_Infer["推理阶段 (Inference)<br/>可在常规 CPU / 无 GPU 环境运行"] end subgraph GenAI["生成式 AI 架构 (Generative AI)"] GAI_Dev["研发与实验阶段 (R&D)<br/>高并发开发争夺 GPU 资源"] --> GAI_Prod["生产部署阶段 (Production)<br/>大流量高频调用持续锁定 GPU"] end ``` 更核心的范式转变体现在生产端。在传统的深度学习架构下，企业通常只需要在模型训练阶段调用 GPU；一旦模型训练完成并被“冻结”（Frozen），推理阶段往往可以脱离 GPU，直接在常规 CPU 甚至无显卡环境中低成本运行。然而生成式 AI 完全打破了这一规律——其生产推理必须持续依赖高性能 GPU 支撑。一旦业务取得成功，高使用率与海量请求将接踵而至，企业必须提前精确规划算力容量。 - **前期采购与长期 TCO 对比**：以一台搭载 8 张 H100 的服务器为例，落地采购成本预估高达 80 万美元（$800k）；但在 3 年的时间跨度内，部分分析师测算本地私有化部署相比直接调用商业 API 可实现 30% 到 50% 的成本节省。 - **企业主流部署形态**：目前市场上成熟企业的实际策略并非非黑即白，而是广泛采用本地自建与公有云 API 相融合的混合架构（Hybrid Approach）。 --- ### [12:50](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=770s) 大模型可观测性与 NIST 七大核心风险治理 在企业级部署中，系统必须建立全面的可观测性（Observability）。基于美国国家标准与技术研究院（NIST）的框架，生成式 AI 的七大核心风险必须做到“可观测、可衡量、可采取行动”（"observable, measurable, and actionable"）： 1. **提示词注入（Prompt Injection）**：以 2022 年 Reddit 社区发起的“DAN”（Do Anything Now，立即去做任何事）越狱倡议为例，用户诱导 ChatGPT 突破安全合规限制并输出违规甚至有害内容。尽管各大模型厂商持续进行安全加固，该风险仍需持续监控。 2. **幻觉（Hallucinations）**：作为生成式 AI 面临的“核心难点”（"the long pole in the tent"），大模型由于缺乏确定性且非传统的监督学习模型，必然会出现低质或事实错误。治理上通常将“大模型作为裁判”（LLM-as-a-judge）的评估机制与传统统计学度量方法解耦并存。 3. **数据泄露与隐私（Data Leakage and Privacy）**：需全面审查用户输入是否携带敏感信息、系统输出是否包含敏感数据，以及底层 RAG 知识检索是否依赖敏感源并做好风险阻断。 4. **有害与违规内容（Harmful and Non-compliant Content）**：与提示词注入紧密相关，防止系统在交互中输出违反合规底线的非期望内容。 5. **内生与偶发偏见（Bias）**：由于预训练数据在筛选设计上不可避免带有倾向性，存在“输入偏见直接引发输出偏见”的因果链，必须双向检测出入端的数据分布与公平性。 6. **知识产权与版权风险（IP and Copyright Risk）**：系统是否未经授权使用了归属于第三方的受保护知识产权。 7. **数据质量（Data Quality）**：对于需要数据基底（Data Grounding）的场景，确保用于检索增强的数据具备权威性、时效性与完整性。 自 2022 年以来，业内围绕上述七大维度构建了大量的度量体系。现代企业系统应将这些评估指标深度嵌入运行时，不仅用于观察长期趋势和用户行为变化，更要在检测到恶意或违规请求时进行毫秒级实时拦截。 --- ### [16:06](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=966s) 智能体架构演进：从状态编排、控制回路到 A2A 与 MCP 标准 相比单轮问答（Single-turn Q&A）只做简单的“响应”，AI 智能体（Agent）具备目标导向和自主采取行动的特性，能够“达成任务”（Accomplish）。复杂的智能体系统往往需要经历 5 轮、10 轮乃至 20 轮交互才能推导出最终解决方案，这就对底层架构提出了全新的工程要求： ```mermaid flowchart LR accTitle: 智能体编排系统与生态协议交互 subgraph AgentSystem["智能体编排核心 (Agentic Orchestration)"] Brain["编排大脑 (Brain)<br/>状态管理与工作内存"] State["上下文状态 (State)<br/>对话历史 / 中间结论 / 工具返回值"] Brain <--> State end subgraph Protocols["标准化协议与扩展 (Protocols & Tooling)"] MCP["MCP 协议<br/>(Model Context Protocol)"] Tools["工具库与数据源<br/>数据库 / Web 服务 / 代码 / ML 模型"] A2A["A2A 协议<br/>(Agent-to-Agent 通信)"] OtherAgent["外部多智能体集群"] MCP --> Tools A2A <--> OtherAgent end AgentSystem --> MCP AgentSystem --> A2A ``` - **编排大脑与状态管理**：智能体系统必须内嵌一个保持上下文状态的编排大脑，负责追踪完整的会话历史、中间推导结论、已调用的工具记录以及向终端用户的反馈。 - **工具赋能与死循环控制**：智能体需要连接数据库、Web API、代码执行沙箱及其它机器学习模型。为防止智能体陷入死循环或在特定环节耗费过多时间（例如因“必须查出答案”的指令而向数据库盲目轮询 100 次），工程上需引入异步调用机制、设定精准的约束性 Prompt，或通过赋予明确的职责与角色设定将其限制在预定业务边界内。 - **多智能体互联协议（A2A & MCP）**：随着企业内部智能体数量的增加，孤立的系统需要实现跨智能体协作。**A2A**（Agent-to-Agent）协议提供了通用的握手与交互标准，避免了传统模式下高昂的定制开发成本；而 **MCP**（Model Context Protocol）则为智能体连接上下文与工具建立了标准化接口，使数据无需硬编码直连，而是通过标准上下文提供者平滑接入。 ### [18:43](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=1123s) 生成式 AI 与智能体落地的核心原则框架与实施总结 在演讲的总结部分，演讲者提炼了一套用于指导生成式 AI（GenAI）与智能体系统（Agentic Solutions）落地的核心原则框架。首要原则是**聚焦业务价值的实现**。企业在启动第一个 Agentic 或 GenAI 解决方案时，必须将其牢固扎根于清晰明确的业务需求之上，并设定可量化的成功指标。企业完全可以从小处着手，正如俗话所说，“一口吃不成胖子”（Eating the elephant as a whole is not the point），必须一块一块地拆解推进，这样才能在快速取得阶段性胜利的同时持续积累经验、复盘学习。 在技术路径的选择上，企业需要审慎权衡**自建与采购（Build vs. Buy）**。在实际落地中，绝大多数企业最终会采用混合架构（Hybrid Solutions）。直接采用第三方 API 方案能大幅缩短产品上市时间（Time to Market），但调用成本波动较大；而本地化部署（On-prem）虽然前期搭建与调试难度更高，但从长期来看更具成本效益。对于选择本地部署的企业，必须及早规划并锁定 GPU 算力容量，使其与业务的增长曲线紧密对齐。 ```mermaid flowchart TD accTitle: GenAI 落地核心原则框架 subgraph Value["1. 业务价值导向"] V1["锚定明确业务需求"] V2["设定可量化成功指标"] V3["从小处切入与小步快跑"] V1 --> V2 --> V3 end subgraph Architecture["2. 架构与算力决策"] A1["审慎权衡 Build vs Buy"] A2["API 方案: 敏捷上线 vs 成本波动"] A3["On-prem 方案: 建设难度高 vs 长期成本优"] A4["及早规划与匹配 GPU 算力"] A1 --> A2 & A3 A3 --> A4 end subgraph Governance["3. 治理与组织协同"] G1["建立清晰可执行的治理框架"] G2["防范长期 Agent 失控蔓延"] G3["打造全员采纳文化"] G4["坚持 Human-in-the-loop 协同迭代"] G1 --> G2 G3 --> G4 end Value --> Architecture --> Governance ``` 最后，演讲者强调了**系统治理与组织文化**的重要性。企业必须建立清晰、易于沟通且可落地的治理框架，这对于长远的前瞻性规划至关重要，能有效防止 Agent 随着业务扩展而变得失控与难以维护。与此同时，必须在组织内部自上而下建立拥抱 AI 的采纳文化（Culture of Adoption），覆盖从技术人员到非技术人员的各个层级。在系统构建与运行过程中，始终要保持“人机协同”（Human-in-the-loop），通过人类与系统的持续共同迭代，最大限度地释放 Agentic 系统的潜力与价值。 - **业务价值锚定**：优先解决有明确痛点、指标可衡量的场景，以“切片化”路径逐步验证与扩张。 - **混合架构权衡**：API 方案胜在交付速度，本地化方案胜在长期持有成本；本地部署务必前置规划 GPU 算力池。 - **治理体系先行**：制定可执行的治理标准，避免未来多 Agent 并行时出现管理混乱与安全盲区。 - **全员协同与人机在环**：无论技术背景深浅，全员参与协同迭代是榨取 Agentic 架构最大红利的关键所在。 ## Notable quotes > [02:21](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=141s) 如果我在 PNC 内部开会，有人随口报出一个冷门的行内缩写词，我如果直接丢给 ChatGPT 去查，它很可能无法给出正确的业务含义，因为模型根本没有我们的语料背景。 — Matt Berezo > [02:21](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=141s) 祝贺你，上下文窗口又翻倍了，每个人都能塞进更长的 Prompt — Matt Berezo > [09:11](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=551s) pay-to-play game — Matt Berezo > [09:11](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=551s) 真希望我当时就知道现在会发生什么，那样我就能抓住机会了。 — Matt Berezo > [12:50](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=770s) observable, measurable, and actionable — Matt Berezo > [12:50](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=770s) the long pole in the tent — Matt Berezo > [18:43](https://www.youtube.com/watch?v=DOIIw4JOqc0&t=1123s) Eating the elephant as a whole is not the point — Matt Berezo ## Mentioned Matt Berezo, PNC Bank, Ai4, ChatGPT, Meta Llama, Perplexity, NVIDIA, NVIDIA A100, NVIDIA H100, NIST, DAN (Do Anything Now), Reddit, MCP (Model Context Protocol), A2A Protocol (Agent-to-Agent) ## Video description Matthew Berezo, Head of AI, PNC Abstracts What happens when your GenAI pilot becomes a core business function overnight? As organizations rush to productionize generative AI (GenAI), they're discovering that traditional AI infrastructure patterns don't necessarily scale under the weight of large models, real-time latency demands, and responsible AI constraints. In this session, we'll explore the realities of scaling GenAI in enterprise environments, including GPU planning, architectural trade-offs, and overall strategy. Attendees will leave with: - A clear mental model for GenAI infrastructure stacks and scalability levers. - Practical insights into cost control, model deployment, and monitoring strategies. Why attend? Whether you're architecting your first GenAI application or evolving your current stack, this talk delivers actionable guidance grounded in real-world enterprise experience - helping you avoid common pitfalls and design for scale from day one. [Source](https://www.youtube.com/watch?v=DOIIw4JOqc0)

Source quotes

[2:21] Matt Berezo: 如果我在 PNC 内部开会，有人随口报出一个冷门的行内缩写词，我如果直接丢给 ChatGPT 去查，它很可能无法给出正确的业务含义，因为模型根本没有我们的语料背景。 [2:21] Matt Berezo: 祝贺你，上下文窗口又翻倍了，每个人都能塞进更长的 Prompt [9:11] Matt Berezo: pay-to-play game [9:11] Matt Berezo: 真希望我当时就知道现在会发生什么，那样我就能抓住机会了。 [12:50] Matt Berezo: observable, measurable, and actionable [12:50] Matt Berezo: the long pole in the tent [18:43] Matt Berezo: Eating the elephant as a whole is not the point

Source quotes

[2:21] Matthew Berezo: mutually exclusive, collectively exhaustive [2:21] Matthew Berezo: congratulations, your context window got bigger... everyone gets a context window [6:13] Matthew Berezo: smell test [9:11] Matthew Berezo: pay-to-play game [9:11] Matthew Berezo: wished that I knew then what I know now. [9:11] Matthew Berezo: fighting with one another and competing for compute [12:50] Matthew Berezo: DAN [12:50] Do Anything Now [12:50] Matthew Berezo: long pole in the tent [18:43] Matthew Berezo: eat the elephant as a whole [18:43] Matthew Berezo: clear, communicable and actionable

Original transcript

[0:04] Good afternoon everyone. Hope that [0:05] you're all doing well. Thanks for [0:07] joining me today. Um my name is Matt [0:09] Bizo and I lead artificial intelligence [0:11] for PNC Bank. And as just mentioned [0:14] today we're going to be discussing [0:16] scaling Genai from pilot to production [0:18] at an enterprise level. [0:21] A pretty simple agenda today, just five [0:24] uh distinct topics, all of which have a [0:26] tremendous amount of weight and decision [0:27] points for any enterprise as they embark [0:30] on their Genai journey. And we're going [0:32] to start with scalability. So talking [0:34] about the big question that everyone [0:36] seems to be noodling on, which is do I [0:38] do this in the cloud using APIs or do I [0:40] do this on prem? We'll also discuss a [0:43] little bit about the trade-offs between [0:44] large language models and small language [0:46] models. That'll fit nicely into our next [0:48] topic which is infrastructure readiness. [0:51] Um this is all about GPUs capacity cost [0:54] and availability. There's some real [0:56] upside to be realized by doing things on [0:59] prem. But realizing that upside is [1:01] coupled with the fact that enterprises [1:03] have to make some serious considerations [1:05] and answer material questions on how do [1:07] I get there successfully? And that [1:09] really starts with capacity. How much do [1:11] I have and how much do I need? And if I [1:14] go get what I need, how much is that [1:16] going to cost me? And if I decide to go [1:18] do that, how do I make sure that these [1:20] GPUs are readily available in R&D and [1:23] production so that we can have a [1:25] successful Genai deployment of multiple [1:27] solutions. [1:29] Next, we'll talk about observability. I [1:31] think that everyone is really hung up on [1:32] Genai as technology focused and and it [1:35] is. It's really slick and and um [1:37] exciting software. Uh but it's not just [1:40] tech. It's also the ability to [1:42] specifically observe and monitor Genai [1:45] systems. Specifically, assessing, [1:48] monitoring, and mitigating risk where it [1:50] may arise in a given Genai solution. And [1:52] this is especially important in highly [1:54] regulated industries. We'll follow up [1:56] with a little bit more of a fun topic [1:58] which is which is Agentic systems, [1:59] right? The Genai market is going crazy [2:01] over over agentic uh deployments, [2:04] especially with the introduction of [2:06] agentto agent protocol along with model [2:07] context protocol. And then we'll wrap [2:09] this up with a simple core principle [2:11] road map of how we bring this all [2:13] together and establish a framework for a [2:14] successful injection of Gen AI from [2:17] conception to real life. [2:21] So scaling decisions. Um before we start [2:24] I'd like to typically work in a mutually [2:27] exclusive collectively exhaustive [2:29] mindset especially when it comes to AI [2:31] and Gen AI. Uh admittedly that's really [2:33] hard because genai and large language [2:35] models by nature are extremely [2:37] generalizable. So you may notice some [2:39] overlap in different things that I say [2:41] but nonetheless the the effort is there. [2:43] Uh and this is our me framework so to [2:45] speak uh in terms of what we think about [2:47] in terms of scaling and it starts with [2:49] large language models versus small [2:51] language models. Large language models [2:53] are incredibly powerful. I think we all [2:55] know that right? They're extreme [2:56] extremely generalizable and they're [2:58] generally really accurate. That's the [3:01] whole value prop of why everyone's [3:02] excited about Gen AI and why chat GGBT [3:04] took off. Chat GBT took off, right? Um, [3:08] but there are some downfalls to be [3:10] mindful of with large language models [3:12] and it really comes down to latency, [3:14] efficacy, and cost. Uh, let's start with [3:16] latency. Large language models by the [3:19] nature of their name are are large. [3:20] They're big. So, it's a big compute [3:22] instance, right? It's a big thing that [3:24] does compute. And as we all know, the [3:26] larger that things get with compute, the [3:28] more the higher of latency you may [3:29] observe. So there's a latency factor [3:31] with large language models. The second [3:33] is efficacy. Um generally all of the [3:37] models that come to market whether it's [3:38] from meta or open AI or perplexity are [3:41] generally trained on data sets and [3:43] corpuses of data that most of us if not [3:46] all of us in this room are not privy to. [3:48] So that means that hallucinations come [3:50] into play or come come into be a factor. [3:52] Right? So, for example, if I'm in a [3:54] meeting at PNC and someone rattles off [3:56] an acronym that I'm not actually aware [3:59] of, if I outsource the definition of [4:01] that acronym to CHBT, it's likely not [4:03] going to do a very good job of giving me [4:05] a definition of what that actually means [4:06] because it lacks context. So, the [4:09] question is, is there an avenue aside [4:11] from rag, aside from uh using some data [4:13] to ground the model to get a more [4:16] generalizable model with higher [4:17] efficacy? And the last is cost. Um, [4:20] what's the biggest bragging point? Every [4:22] time a new language model comes to [4:24] market, it is congratulations, your [4:27] context window got bigger. Your context [4:29] window got even bigger. Everyone gets a [4:31] context window, right? Which means that [4:33] before I could fit a book perhaps in the [4:35] prompt of the language model and [4:37] retrieve a response, but now with an [4:39] extended context window, I can fit two [4:41] books, which is great and it's [4:43] wonderful. I I I really love that. But [4:45] the thing is, when you're calling out to [4:47] an API, you're paying per token or by [4:49] character. when you call out and you're [4:51] paying for what you bring back in in [4:53] some fashion. So those unit economics [4:55] are really important, right? The more [4:57] that the bigger the context window gets, [4:59] if it's API based, the more that I'm [5:01] actually paying to use the model. It's [5:03] also true for on-prem infrastructure. So [5:05] say that you want to use a large [5:06] language model with 10 million token [5:08] context window, right? No, you're not [5:11] going to be paying for API costs, but [5:13] you are going to be paying for the [5:14] infrastructure to make sure that that's [5:15] a successful project using that type of [5:17] model. [5:19] Enter small language models. So small [5:21] language models by the nature of their [5:23] name are smaller. They're smaller [5:24] compute. So there's potential upside [5:26] with them. There could be lower latency. [5:29] Uh additionally, since they're smaller [5:31] in nature, small language models can be [5:33] more efficiently fine-tuned on your [5:35] specific corpus, which means that you [5:36] could have models or solutions that have [5:38] higher efficacy. And this all ties back [5:40] to cost. Generally, small language [5:43] models are not API based. And given that [5:45] they're smaller, require less compute, [5:47] the infrastructure isn't as heavy to [5:48] actually use them in production or a [5:50] development environment. So it really [5:52] comes down to balancing those three [5:53] buckets of latency, cost, and efficacy [5:55] and understanding what do I need to get [5:57] the job done for a minimal viable [5:59] product that's that that is [6:00] satisfactory. Do I really need the big [6:03] most sophisticated thing all of the time [6:05] or can I be more nimble in certain [6:07] situations? And this certainly flushes [6:08] itself out in aentic frameworks which [6:10] we'll discuss in a few slides. [6:13] The next scaling decision is building [6:14] versus buying. Um, and this for us [6:17] really comes down to what is the data. [6:20] Uh, again with the me in the spirit of [6:22] having a a me framework, we've [6:24] identified four different categories of [6:26] deployment patterns for Genai that are [6:28] admittedly probably imperfect given the [6:30] generalizable nature of Genai solutions, [6:32] but that's okay. uh and the spirit of of [6:35] the different solution patterns is that [6:37] it's a really easy way for a highly [6:40] technical person or a not highly [6:42] technical person map based on the [6:44] solution's intent what are the risks [6:46] affiliated with it and should I go build [6:48] it or should I go buy it so how does [6:50] that work first it works with defining [6:52] the different categories of different [6:53] geni solutions the first being the most [6:55] simple which is there's no data that [6:58] needs to be grounded into the model to [7:00] retrieve an answer that is satisfactory [7:02] Um, I work in finance, so an example of [7:04] this would be something like, explain [7:06] interest rates to me in layman's terms. [7:09] Presumably, a large language model, take [7:11] your pick, has been trained on enough [7:13] data that we don't need to provide any [7:14] more context to get a response that's [7:16] going to answer our question [7:17] successfully. [7:19] The second deployment pattern would be [7:21] solutions that require some data, but [7:23] not enough to merit a vector database. [7:25] Another simple example, suppose that you [7:27] have a job posting and two candidate [7:29] resumes and you say, "Okay, LLM, based [7:32] on this job posting and the two [7:33] candidates, who do you think's the [7:34] better fit in your opinion?" So, there's [7:36] some data to inform the response, but [7:38] not enough where we need to vectorize it [7:40] and put it into a vector database like [7:41] we would with a large repository. [7:44] Third is when we need a large volume of [7:46] data to inform our answer. So, this is [7:48] where vector databases come in, right? [7:50] We vectorize the data of the repository [7:52] or multiple repositories in a different [7:54] vector database. It allows for efficient [7:56] surf search retrieval based on the query [7:58] and informs the answer based on the [8:00] context that is retrieved. So in other [8:02] words, this is retrieval augmented [8:03] generation. And the final deployment [8:05] pattern is any combination of the first [8:08] three that I mentioned to any degree of [8:09] complexity meaning agentic. So how does [8:12] this actually inform a build versus buy [8:14] decision? So let's just do two pressure [8:15] tests that are really simple. Are we [8:18] really actually going to build an [8:20] internal solution that's a generalizable [8:22] chatbot for typical Q&A? The answer is [8:26] probably not. Right? There's a lot of [8:28] vendors out there that seem to have [8:30] taken care of this quite well. So [8:31] building that and then maintaining it [8:34] and so on is is pretty low value prop. [8:36] Right? Now on the flip side of that, if [8:39] we have internal proprietary data and we [8:41] need to enable employees to search those [8:43] repositories and have more efficient [8:45] retrieval of answers, [8:48] perhaps a vendor doesn't support a [8:50] vector database or they can't they can't [8:52] support the volume of which are is in [8:54] our repositories. So this might be a [8:56] custom rag solution, right? So this is [8:58] just a simple framework of how we kind [9:00] of do a first smell test of is this a [9:02] buy situation or a build situation. [9:05] And the last is hosting strategy which [9:07] goes back to are we doing this on prem [9:09] or are we going to use a cloud-based API [9:11] and really it comes down to Genai from [9:14] my perspective anyway is a pay-to-play [9:16] game. You are either paying to use an [9:18] API or you are paying not to use an API [9:22] and there's benefits to both. Uh an API [9:25] based solution obviously is a quicker [9:26] time to market so you can get a quick [9:28] win faster. Longterm there's variable [9:30] costs that might stick you with a bill [9:32] that that you really weren't privy to. [9:35] uh during development or deployment, [9:37] right? On prem you have more cost [9:39] control and probably more control over [9:41] compute consistency. However, standing [9:43] up the infrastructure is complex and [9:45] also it's a pretty big cost up front [9:47] which we'll talk about here in the next [9:49] slide which leads us to infrastructure [9:51] requirements. [9:53] Um so as we all know with the chip [9:55] stocks uh soaring uh they are now the [9:58] catalyst of all things Gen AI. um [10:00] they're incredibly important to make uh [10:03] conceptual solutions come to life. Um I [10:06] I had my first experience with the GPU I [10:08] think in October of 2015. I was doing [10:11] some deep learning model. So I I was [10:13] very aware uh back then that Nvidia [10:16] exists and I wished that I knew then [10:18] what I know now, but but I digress. So [10:21] um it's large language models are big [10:24] and they they require GPUs, right? [10:26] That's what's in the headlines. um they [10:28] require a lot of virtual memory and they [10:30] require even edge edge nodes or clusters [10:32] of nodes of GPUs so you can handle the [10:34] genai loads that come in during [10:36] development and production. From a [10:38] hardware perspective, usually A100's and [10:41] H100s do the trick and quantizing the [10:44] quantizing the models or making them [10:45] smaller is also a viable path to to [10:48] production which would require less GPUs [10:50] because the models are essentially [10:52] complex uh compressed. [10:54] Um, one thing that I wanted to touch on [10:56] is that building a robust infrastructure [10:59] strategy is critical not only in [11:00] production but also in development. So [11:02] let's talk about R&D. I think many of us [11:05] here are probably familiar with the [11:07] phenomenon where a bunch of developers [11:09] are in the same environment and they're [11:11] fighting with one another and competing [11:12] for compute. So fast prototyping and [11:15] iteration kind of slows. This is going [11:17] to be the same thing for R&D and Gen AI. [11:19] I guess that goes unsaid, right? So we [11:21] need to have a robust Genai [11:23] infrastructure in terms of GPUs so that [11:25] we can have that rapid prototyping and [11:27] see success faster but it's also [11:30] important to have the same type of [11:31] infrastructure in production which may [11:33] seem obvious but historically it is [11:35] really not. Uh deep learning models [11:37] which is kind of a foundational part of [11:39] uh generative AI in the past typically [11:42] you would train a deep learning model on [11:43] a GPU but you didn't necessarily need a [11:46] GPU for production. In fact, you could [11:48] actually take a GPU or sorry, a GPU [11:50] trained deep learning model, freeze it [11:52] down and run it without a GPU. That was [11:54] great. Um, that's not true for [11:57] generative AI models. Uh, so you are [11:59] going to need to use a GPU for [12:01] production. And if you do it [12:02] successfully, then high volume and high [12:04] usability is bound to follow you. So [12:06] really planning that out and [12:08] understanding your demand if you go down [12:09] the on-rem route is critical for [12:11] long-term sustainability. [12:14] Um, in terms of cost, uh, obviously [12:17] we've heard that GPUs are really [12:18] expensive. For example, eight H100 [12:21] servers are estimated to cost 800 grand [12:23] at the door, right, which seems steep, [12:25] but some analysts estimate that there [12:27] could be 30 to 50% cost savings by going [12:30] the on-prem route versus API route in a [12:32] three-year time horizon. So really, it's [12:34] understanding the use case, when you [12:36] need your time to market, what your ROI [12:38] is going to be, and the cost that you're [12:40] going to incur. And typically what I'm [12:41] seeing in the market is that enterprises [12:43] are really blending onrem versus API [12:46] based solutions. [12:50] The next is observability. Uh what you [12:52] see here are the seven key geni risks [12:54] that are grounded in the national [12:55] institute of standards and technology. [12:58] Um my stance is that all of these risks [13:03] should be observable, they should be [13:05] measurable and they should be [13:06] actionable. So let's go through them [13:08] really quickly. The first is prompt [13:09] injection. Um, don't know if any of you [13:12] are Redditors out there. I'm not a [13:13] Redditor, but I was aware that in 2022 [13:16] there was a Reddit initiative to uh [13:18] trick Chhat GBT to act in a way that it [13:21] wasn't originally intended to act. Uh, [13:23] in fact, more specifically, these [13:25] Redditors successfully got chatbt to act [13:27] as DAN, which stood for do anything now. [13:30] Um, and they got chatbt factually to do [13:33] things that were definitely [13:35] non-compliant. Uh, return harmful [13:37] content and so on. Now, since 2022, it [13:40] seems as though a lot of the model [13:42] vendors have mitigated this type of [13:43] risk, but nonetheless, if it happened in [13:45] the past, then it's something that we [13:46] need to continue to measure um in the [13:48] future. So, that's one. The second is [13:50] hallucinations. This is the long pole in [13:52] the tent for geni. Genai is not [13:54] deterministic. It is not a supervised [13:56] machine learning model. So, it's bound [13:58] to have answers that are poor or [14:00] incorrect. [14:02] There are ways to measure [14:03] hallucinations. Ideally, you measure [14:05] hallucinations by decoupling a Genai [14:08] judge hallucination measurement from a [14:10] regular statistical hallucination [14:12] method. But we'll talk about that in a [14:14] second. [14:16] Data leakage and privacy. Obviously, [14:17] it's important to understand is the user [14:19] querying the system with sensitive [14:21] information. Is the system returning [14:23] sensitive information or is the system [14:25] inherently relying upon sensitive [14:26] information? And if so, for the for the [14:29] third part, how do we mitigate any risks [14:31] involved with that harmful and [14:33] non-compliant content? Uh, this lends [14:35] itself pretty closely to prompt [14:37] injection. So, I'm going to bypass [14:38] talking more about that. Uh, bias even [14:41] inadvertently. I think I saw a LinkedIn [14:43] post by one of our AI for conference [14:45] members just last night talking about [14:47] how some of these models are inherently [14:49] biased um because they leverage data um [14:53] that that was selected by by design. um [14:57] inherently biased inputs really can [14:59] render inherently biased output. So [15:01] import it's important to measure and [15:03] understand is there a biased input [15:05] coming in or is there a biased output [15:06] going out IP and copyright risk is the [15:09] system using materials that we shouldn't [15:11] be using or materially belong to [15:13] somebody else and last is data quality [15:16] is the data authoritative up-to-date and [15:19] complete in nature to the best of our [15:20] knowledge for data for geni solutions [15:23] that require data grounding since 2022 [15:26] there's been a tremendous amount of work [15:27] around being able to measure across all [15:29] of these seven risks [15:31] Now a lot of these measurements are LLM [15:33] by a jud LLM as a jud judge measurements [15:35] which is fine. What the market is [15:37] telling telling us is that these are [15:39] measurable and so they should be [15:41] embedded into the system and once we [15:43] measure them we should be able to [15:44] analyze trends how user behaviors [15:47] change. We should report on it and we [15:49] should take action inside the system in [15:51] which for example if a harmful harmfully [15:54] compliant or non-compliant query comes [15:56] into the system can we measure it in [15:58] real time and mitigate the system from [16:00] generating a response that might be um [16:02] suboptimal. [16:06] Uh next the more fun part is agentic [16:09] frameworks. Uh so agents are a really [16:10] complex deployment pattern right? Um [16:13] they they are tasked with goals and take [16:16] initiative. So they're unlike single [16:17] turn Q&A. They do more than just [16:20] respond. They end up accomplishing. And [16:22] this starts with orch orchestration [16:24] upfront. Um again, this is not a simple [16:27] one- turd Q&A. Agentic systems can take [16:29] five, 10, 20 different turns before they [16:32] get to their final solution. [16:34] In order for that to work, you need a [16:36] component in the orchestration that [16:37] keeps track of the state of the agent. [16:40] Right? This includes conversation [16:41] history, intermediate conclusions, tools [16:44] that were used and what was returned to [16:46] the end user. So this is effectively the [16:48] brain an orchestration brain needs to be [16:50] embedded in the agentic system. The [16:52] brains hold memory and the the agents [16:55] need compute or sorry tools to act on [16:57] that on that memory. This includes [16:59] equipping agents with databases, web [17:01] services, code, um maybe other machine [17:04] learning models, etc. And what's more [17:07] important aside from just equipping [17:09] these agents with the tools is being [17:11] able to control them. Agentic frameworks [17:13] have the tendency to hang up and spend [17:14] too much time in specific places. Uh so [17:17] to solution against that you have to [17:18] assess where is the agent getting hung [17:20] up based on where it's getting hung up. [17:22] There's ways to mitigate unnecessary [17:25] time spent in certain parts of the [17:26] pipeline. That could be really technical [17:28] like implementing asyn asynchronous [17:30] calling, but it can be also be more um [17:33] specific and easy such as having more [17:35] detailed prompts or assigning [17:37] personalities or duties to an agent that [17:38] are more detailed to box them into what [17:40] they're they actually should be doing. [17:42] The last thing that you would want [17:44] hypothetically is an agent pinging a [17:45] database 100 times because it can't find [17:47] the answer because it was told to work [17:49] until it figured out the answer. So [17:51] these are a couple of solutions uh to [17:53] mitigate that time complexity. [17:56] um AI agents at scale. So once you build [18:00] your first agent agentic system, you're [18:02] bound to build build your next agentic [18:03] system. And what's bound to happen when [18:05] you have enough agentic systems is [18:07] you're going to realize I wish that [18:08] agentic system was able to speak to the [18:10] other agentic system. And this is where [18:12] Atoa comes to play which is a proposed [18:15] common language for agents to [18:17] collaborate. This is important because [18:19] typically creating that agent to a agent [18:21] collaboration would be customuilt. [18:24] Instead, ADA enables a clean handshake [18:26] for a common protocol for agents to work [18:28] with one another. MCP is also important [18:31] because it provides a standard interface [18:32] to for agents to access context and [18:35] tools. So instead of wiring the data [18:37] sources of the tools directly to the [18:38] agent, we can funnel the data through [18:41] context providers so that the agents can [18:43] easily retrieve them and utilize them [18:44] for their for their capabilities. [18:48] In conclusion, we want to draw out the [18:50] core uh principle framework and the [18:52] first is deriving business value. It's [18:54] important to ground your first agentic [18:56] solution or genai solution in something [18:58] that's a clear business need and has [18:59] measurable success. It's okay to start [19:02] small. Uh eating the elephant as a whole [19:04] is not the point. Piece by piece is [19:06] important so that you can win and learn. [19:08] It's important to be deliberate versus [19:10] build versus buy. And likely you're [19:12] going to see hybrid solutions. API [19:14] solutions will have a faster time to [19:16] market but vary in cost where on-prem [19:18] solutions will be uh more cost effective [19:20] in the long term but harder to probably [19:22] stand up. If you're onrem it's important [19:25] to establish uh the GPU capacity and [19:27] align it with growth quickly. And for [19:30] for governance it's important to [19:31] establish a governance framework that is [19:34] clear, communicable and actionable. This [19:36] will lead to excellent planning so that [19:38] you don't have agents that get [19:39] unmanageable in the long term. [19:42] Last is establishing a culture of [19:44] adoption and this is for varying levels [19:47] of people throughout the organization [19:49] whether they're technical or [19:50] non-technical. Uh it's important that [19:52] humans are in the loop. They're [19:53] iterating together and as you're [19:54] building solutions to get the most out [19:56] of your agentic system. And with that I [19:58] see that my time is up. So I will stop [20:00] there. Thank you for your time. I hope [20:02] you all have a wonderful day. Thanks.