Capital One Engineer Replaces RAG With Curated Knowledge Graphs
Ai4
Scaling context windows and vector retrieval fail proprietary enterprise systems, making human-curated domain knowledge graphs necessary to prevent AI hallucinations.
## Background
Published on YouTube by the Ai4 channel on August 16, 2026, this talk features Ravi Teja Prabhala Venkata, Senior Manager of Software Engineering at Capital One, discussing context management challenges in enterprise AI.
## TL;DR
Ravi Teja Prabhala Venkata argues that enterprise AI failures stem from missing organizational context rather than model capacity, making traditional vector RAG and fine-tuning insufficient for proprietary systems and decades of legacy architecture. To resolve this, he introduces the concept of a 'Context Layer'—an independent, human-curated Domain Knowledge Graph that explicitly maps enterprise entities and the architectural rationales behind them. By pairing deterministic subgraph injection with low-level controls to minimize model non-determinism, organizations can eliminate hallucinations and enforce consistent domain ontologies across autonomous agents without relying on continuous model retraining.
## Atmosphere & style
The presentation blends pragmatic enterprise engineering critique with lighthearted, dry humor, using playful analogies about strawberry-counting benchmarks and industry buzzwords to warm up the room. It feels like an authoritative, insider architectural briefing that shifts smoothly from common corporate development headaches to precise graph-based system design and an engaged audience Q&A.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=4s) Frontier models with million-token context windows still fail basic reasoning tasks, highlighting a trust deficit that prevents enterprise adoption without external guardrails.
- [00:04](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=4s) Automated AI code reviews and test generation catch surface-level syntax errors but cannot evaluate broader organizational context or architectural intent.
- [03:03](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=183s) Decades of unindexed legacy systems and proprietary internal standards make it impossible for general internet-trained models to generate architecturally sound enterprise code.
- [03:03](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=183s) Traditional vector RAG fails on internal enterprise acronyms and jargon because mathematical embedding models lack semantic comprehension of private organizational language.
- [03:03](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=183s) Fine-tuning models on internal data introduces source amnesia and creates an unsustainable operational burden of continuous compute, retraining, and redeployment.
- [08:06](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=486s) The Context Layer proposes replacing probabilistic retrieval with a curated Domain Knowledge Graph that maps both enterprise entities (nodes) and architectural reasoning (edges).
- [08:06](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=486s) Decoupling context storage from model infrastructure allows organizations to update business rules and institutional knowledge instantly without retraining or redeploying LLMs.
- [08:06](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=486s) An orchestrator leverages intent classification to extract relevant subgraphs from the knowledge graph and inject authoritative context directly into prompt windows.
- [12:26](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=746s) Non-determinism caused by GPU floating-point variance and sampling randomness poses severe operational risks when deploying generative models into enterprise workflows.
- [12:26](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=746s) Combining technical sampling controls (zero temperature, hardware locking, quantization) with bounded subgraph injection drastically suppresses hallucinations.
- [15:14](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=914s) Enterprise AI advances will come from targeted context injection rather than waiting on model parameter scaling or expanding raw context window sizes.
- [15:14](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=914s) The Context Layer functions as an enterprise-wide ontological filter, forcing multi-agent systems to adopt consistent definitions and constraints across different business domains.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=4s) The Context Window Challenge and the Trust Deficit in Frontier Models
Opening with a humorous observation on industry naming conventions, Ravi Teja Prabhala Venkata jokes about whoever rebranded bots as "agents" so people would stop criticizing them, proposing the term "context layer" to address a fundamental, unnamed bottleneck in enterprise AI. While modern frontier models boast context windows of up to 1 million tokens, that capacity remains insufficient and poorly utilized for enterprise-scale needs. To illustrate the unreliability of current models, an icebreaker quiz highlights how even advanced models fail basic tasks—such as counting the number of letters in the word "strawberry." As an aside, he jokes that once AI reaches Skynet levels, it will probably scout and destroy every strawberry on the planet before coming after humanity.
Beyond humorous trivia, hallucination and unpredictability represent severe structural flaws. When foundational models trained on the entire public internet make elementary mistakes on generic prompts, trusting them with mission-critical enterprise workloads becomes impossible. While techniques like AI-driven test generation and automated code reviews can catch surface-level bugs, they fail to resolve the deeper enterprise dilemma: models lack the organizational context required to make sound decisions.
- Frontier context windows (up to 1 million tokens) still cannot absorb the full scope of enterprise codebases and institutional histories.
- General internet pre-training does not prevent models from hallucinating on basic domain-agnostic queries.
- Automated AI code reviews catch isolated syntax errors but cannot verify broader architectural intent.
---
[03:03](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=183s) Enterprise Realities vs. Isolated AI: The Limits of Traditional RAG and Fine-Tuning
AI tools such as Claude Code can execute isolated, bounded feature tasks with technical precision, but they frequently produce code that is architecturally flawed because they cannot see the larger organizational picture. Enterprise environments possess massive proprietary datasets, strict compliance mandates, internal frameworks, and intellectual property that have never appeared on the public internet. Furthermore, organizations maintain legacy and mainframe systems running for over 30 years whose architectural decisions and infrastructure nuances are entirely unknown to foundational models.
To supply this missing context, organizations traditionally turn to three approaches, each carrying severe limitations:
1. **Traditional Retrieval-Augmented Generation (RAG):** Standard RAG relies purely on mathematical vector similarity embeddings. When confronted with proprietary enterprise terminology or acronyms, vector search fails because the underlying embedding models have no semantic understanding of private internal language, placing documents in irrelevant vector clusters and returning shallow retrievals.
2. **Fine-Tuning:** While fine-tuning adjusts weights, it suffers from LLM "knowledge amnesia" or source amnesia—the inability of the model to track the provenance of its generated tokens. Furthermore, enterprise knowledge changes daily or weekly, turning fine-tuning (via LoRA, QLoRA, or full parameter updates) into an unsustainable operational treadmill of continuous compute, retraining, evaluation, and redeployment whenever foundational models upgrade.
3. **Autonomous Agents and Cloud Skills:** Decomposing problems into sub-agents and bounded tools works well for isolated steps, but agents still operate under restricted context windows. If the overarching enterprise architecture cannot fit into the active window, the entire workflow becomes a gamble.
```mermaid
flowchart TD
accTitle: Limitations of Traditional Context Injection Methods
subgraph S1["Traditional RAG"]
R1["Vector Embeddings"] --> R2["Mathematical Similarity"]
R2 --> R3["Fails on Proprietary Jargon & Shallow Context"]
end
[08:06](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=486s) The Context Layer: Designing a Curated Domain Knowledge Graph
To overcome the pitfalls of static mathematical models like baseline RAG, GraphRAG, HippoRAG, or PathRAG, an alternative architecture is proposed: a dedicated, human-curated Domain Knowledge Graph acting as the enterprise "Context Layer." Rather than leaving information retrieval to chance via mathematical vector distances, this layer explicitly curates enterprise entities (nodes) and the architectural reasoning and relationships behind them (edges).
This curated graph functions as an authoritative single source of truth that is completely decoupled from the LLM infrastructure. When organizational data or business rules change, teams update the graph directly without needing to retrain, fine-tune, or redeploy model checkpoints. Enterprise retrieval strategies can be configured precisely—whether running shallow first-level lookups or executing deep depth-first traversals—depending on domain requirements.
When a user submits a query, an orchestrator classifies user intent and triggers the Context Layer API. This API queries the domain knowledge graph, extracts the relevant subgraph of facts, relationships, and historical rationales, and injects this context directly into the prompt before passing it to the underlying LLM.
```mermaid
flowchart LR
accTitle: Context Layer Knowledge Graph Architecture
subgraph Ingestion["User & Agent Interface"]
UQ["User Query"] --> AG["AI Agent"]
AG --> IC["Intent Classification"]
end
subgraph Synthesis["Execution"]
SG --> Prompt["Injected Context Prompt"]
Prompt --> LLM["Target LLM Engine"]
LLM --> Out["Architecturally Accurate Result"]
end
```
- **Explicit "Whys":** Edges in the knowledge graph capture legacy rationales and architectural decisions dating back decades across legacy repositories.
- **Infrastructure Independence:** Decouples knowledge management from model lifecycle; LLM hosting remains stateless while context storage is managed independently.
- **Customizable Graph Traversal:** Allows enterprises to define exact retrieval logic (e.g., depth-first search vs. neighboring node queries) based on business logic.
---
[12:26](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=746s) Taming Non-Determinism and Hallucinations at Enterprise Scale
Achieving determinism in LLMs is an essential yet difficult engineering challenge, often complicated by low-level floating-point non-determinism across parallelized GPU hardware. Because LLMs inherently produce varying responses for identical inputs, deploying them directly into deterministic enterprise workflows introduces substantial operational risk.
While absolute 100% determinism remains practically impossible in generative architectures, randomness can be systematically minimized through several technical controls:
- Locking physical compute hardware configurations to reduce floating-point variance across diverse GPU architectures.
- Setting sampling temperature to zero and adjusting top-p / top-k parameters.
- Implementing integer quantization to constrain probabilistic token distributions.
When reduced-randomness models are paired with deterministic, curated subgraph retrieval from the Context Layer, hallucination rates drop dramatically. By feeding the LLM an explicit boundary of authoritative truth, developers prevent the model from drifting into speculative answers, effectively boxing its reasoning capabilities strictly within enterprise-approved data.
---
[15:14](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=914s) Why Context Injection Trumps Model Scaling & Audience Q&A on Ontologies
"The next leap in enterprise AI is not a better model," as model scaling laws inevitably hit economic and physical constraints. He points out the absurdity of frontier labs purchasing rare books to train models only to burn them to monopolize knowledge, stressing that context windows cannot scale infinitely. In enterprise environments, models do not need access to the entire public internet; they primarily require the reasoning capability to synthesize strictly bounded, injected proprietary context without wasting millions of tokens on frontier model inference.
During the Q&A session, an audience member asks how the Context Layer manages domain ontologies, noting that different agentic systems often interpret identical business terminology inconsistently.
The speaker explains that the Context Layer acts as an ontological filter:
- When a user query references an internal concept, the Context Layer retrieves the explicit enterprise definition, relationships, and constraints from the curated graph.
- This definition is injected directly into the prompt alongside system instructions compelling the LLM to ignore external, generic interpretations.
- By forcing the model to adhere strictly to the injected context, every agent across the enterprise evaluates the term uniformly, ensuring reliable and consistent downstream decision-making.
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=4s) When foundational models trained on the entire public internet make elementary mistakes on generic prompts, trusting them with mission-critical enterprise workloads becomes impossible. — Ravi Teja Prabhala Venkata
> [00:04](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=4s) Frontier context windows (up to 1 million tokens) still cannot absorb the full scope of enterprise codebases and institutional histories. — Ravi Teja Prabhala Venkata
> [00:04](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=4s) Automated AI code reviews catch isolated syntax errors but cannot verify broader architectural intent. — Ravi Teja Prabhala Venkata
> [03:03](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=183s) Enterprise environments possess massive proprietary datasets, strict compliance mandates, internal frameworks, and intellectual property that have never appeared on the public internet. — Ravi Teja Prabhala Venkata
> [08:06](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=486s) When organizational data or business rules change, teams update the graph directly without needing to retrain, fine-tune, or redeploy model checkpoints. — Ravi Teja Prabhala Venkata
> [08:06](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=486s) Edges in the knowledge graph capture legacy rationales and architectural decisions dating back decades across legacy repositories. — Ravi Teja Prabhala Venkata
> [12:26](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=746s) Achieving determinism in LLMs is an essential yet difficult engineering challenge, often complicated by low-level floating-point non-determinism across parallelized GPU hardware. — Ravi Teja Prabhala Venkata
> [12:26](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=746s) By feeding the LLM an explicit boundary of authoritative truth, developers prevent the model from drifting into speculative answers, effectively boxing its reasoning capabilities strictly within enterprise-approved data. — Ravi Teja Prabhala Venkata
> [15:14](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=914s) The next leap in enterprise AI is not a better model — Ravi Teja Prabhala Venkata
> [15:14](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=914s) In enterprise environments, models do not need access to the entire public internet; they primarily require the reasoning capability to synthesize strictly bounded, injected proprietary context without wasting millions of tokens on frontier model inference. — Ravi Teja Prabhala Venkata
## Mentioned
Ravi Teja Prabhala Venkata, Capital One, Ai4, Claude Code, Skynet, GraphRAG, HippoRAG, PathRAG, LoRA, QLoRA
## Video description
Ravi Teja Prabhala Venkata – Senior Manager, Software Engineering, Capital One
Capital One 工程师用精选知识图谱取代 RAG
扩大上下文窗口与向量检索在企业专有系统中失效，需借助人工构建的领域知识图谱来防止 AI 幻觉。
Ravi Teja Prabhala Venkata 认为，企业 AI 的失败根源在于组织上下文的缺失，而非模型能力不足，这导致传统的向量 RAG 和微调无法应对专有系统及数十年的遗留架构。为此，他提出了“上下文层”（Context Layer）概念——这是一个独立的人工精选领域知识图谱，能够明确映射企业实体及其背后的架构设计原理。通过将确定性子图注入与底层控制相结合以最大程度减少模型的不确定性，企业无需依赖持续的模型重训，即可消除幻觉并在各个自主智能体之间推行一致的领域本体。
即使具备百万 token 上下文窗口的前沿大模型仍会出现基础事实与推理幻觉，单纯依靠自动化代码审查工具无法弥补模型对企业架构意图和历史背景的理解缺失。 (0:04)
传统向量 RAG 难以理解企业私有缩写与内部术语，而通过微调注入知识不仅会导致模型产生“源知识遗忘”，还会带来持续重训和重新部署的高昂成本。 (3:03)
企业 AI 需要建立独立的“上下文层”，通过人工策划的领域知识图谱将实体作为节点、架构决策原因作为边，实现企业知识与底层无状态 LLM 架构的彻底解耦。 (8:06)
通过将零温度等低随机性推断配置与精准的子图检索注入相结合，企业无需无节制扩展上下文窗口，即可大幅遏制模型幻觉并统一跨 Agent 的业务本体定义。 (12:26)
Reading summary
Ravi Teja Prabhala Venkata argues that enterprise AI adoption is fundamentally bottlenecked by a lack of organizational context rather than model intelligence, rendering expanding context windows, fine-tuning, and standard vector search insufficient. To resolve this trust deficit, he proposes an independent, human-curated "context layer" knowledge graph that explicitly decouples institutional rationales from underlying models and injects them directly into prompts.
The Enterprise Context Deficit in Frontier Models (0:04)
While frontier models now feature context windows reaching up to 1 million tokens, Prabhala Venkata asserts that raw token capacity fails to accommodate the full scope of enterprise environments. Enterprise operations rely on massive proprietary datasets, strict compliance mandates, and legacy systems that have operated for over 30 years without ever appearing in public internet training corpora. Because general pre-training leaves models prone to elementary errors and hallucinations even on domain-agnostic queries, foundational models cannot be trusted with mission-critical workloads on their own. Automated code reviews and AI test generation can detect isolated syntax issues, but they lack the organizational visibility required to evaluate overarching architectural intent.
Structural Breakdowns in Traditional Context Strategies (3:03)
Enterprises traditionally turn to retrieval-augmented generation (RAG), fine-tuning, or multi-agent frameworks to supply missing organizational data, but Prabhala Venkata identifies severe technical constraints in each approach. Standard RAG relies purely on mathematical vector similarity embeddings; when encountering proprietary internal terminology or acronyms, vector search misclassifies documents into irrelevant clusters, producing shallow and inaccurate retrievals. Fine-tuning attempts to bake knowledge into model weights but suffers from token source amnesia and creates an unsustainable operational treadmill of continuous compute, retraining, and redeployment whenever baseline models upgrade or business rules change. Meanwhile, decomposing tasks across autonomous agents still leaves workflows constrained by active context windows, turning multi-step execution into a gamble when broad system architecture cannot fit into memory.
Architecting the Context Layer via Knowledge Graphs (8:06)
To replace static vector retrieval, the presentation outlines a dedicated, human-curated Domain Knowledge Graph functioning as an enterprise context layer. In this architecture, business entities serve as nodes, while edges explicitly capture the historical rationales, relationships, and architectural "whys" spanning decades of institutional decisions. Decoupling knowledge storage from model infrastructure allows organizations to treat foundational LLMs as stateless compute engines, updating business rules directly within the graph without retraining or redeploying model checkpoints. During execution, an orchestrator classifies user query intent and calls the Context Layer API, which extracts a bounded subgraph of verified facts and structural reasoning to inject directly into the prompt before LLM synthesis.
Reducing Non-Determinism and Standardizing Enterprise Ontologies (12:26)
Deploying generative models into enterprise workflows is further complicated by low-level floating-point non-determinism across parallelized GPU hardware, leading to varying outputs for identical inputs. While 100% determinism remains unattainable in generative architectures, Prabhala Venkata notes that variance can be minimized by locking compute hardware configurations, setting sampling temperature to zero, tuning top-p and top-k parameters, and applying integer quantization. When reduced-randomness models are paired with deterministic subgraph injection, hallucination rates drop because model reasoning is strictly boxed within enterprise-approved boundaries. Furthermore, the context layer serves as an ontological filter across disparate agents: by retrieving explicit proprietary definitions and injecting them alongside instructions to disregard generic meanings, all enterprise agents evaluate internal terminology uniformly.
Prabhala Venkata concludes that the next leap in enterprise AI will come from precise, bounded context injection rather than perpetual model scaling or expanding token windows, which face physical and economic limits. However, the framework relies on organizations actively maintaining curated knowledge graphs to capture proprietary rationales that automated vector embedding models consistently miss.
## Background
该视频由频道 Ai4 于 2026 年 8 月 16 日发布在 YouTube 平台。内容为 Capital One 软件工程高级经理 Ravi Teja Prabhala Venkata 的主题演讲，探讨企业级 AI 为何需要构建专属的“上下文层”以解决大模型落地难题。
## TL;DR
演讲指出，前沿大模型在单点任务上技术正确，但面对庞大且包含数十年遗留系统的企业架构时往往彻底脱节，传统 RAG、微调和智能体机制均无法有效弥补语义与历史决策的断层。演讲者提出企业应构建人工精选与维护的“领域知识图谱”作为权威真实源，通过上下文层 API 按意图提取精准子图注入 Prompt，从而解耦大模型底座并压制幻觉。企业 AI 的下一次跃升不在于追求无休止扩展参数的基座模型，而在于掌握专有上下文的主导权与确定性推理约束。
## Atmosphere & style
整场演讲风格幽默风趣却直击痛点，开场借由“草莓有几个 r”的经典段子调侃大模型信任危机，迅速活跃现场气氛。随着讨论深入，讲者以资深企业架构师的务实视角，对纯向量数学 RAG 和盲目追求模型 Scaling Law 的行业狂热进行了清醒而辛辣的解构。整体节奏干货密集、论证严密，散发着一线工程实践者破除 AI 概念泡沫、重归严谨软件工程的理性魅力。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=4s) 大模型在简单常识问题上频发低级幻觉，加之 100 万 token 窗口在企业实际落地中极难生效，导致企业无法直接信任未经控制的通用模型。
- [02:31](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=151s) 通用大模型生成的输出往往在代码或语法技术上正确，但由于缺乏内部历史决策与合规背景，在企业整体架构层面上往往是完全错误的。
- [04:52](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=292s) 传统 RAG 本质是向量数学距离计算，无法真正理解企业独有的专有名词与私有流程，深层语义检索极易偏航。
- [04:52](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=292s) 模型微调（Fine-Tuning）存在固有的知识源遗忘问题，且企业知识库的动态更新会导致极高且不可持续的重复训练与维护成本。
- [08:06](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=486s) 企业破局的核心路径是构建自研且人工深度管理的领域知识图谱（Domain Knowledge Graph），作为不可动摇的权威真实源。
- [08:06](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=486s) 知识图谱方案使企业能够将业务知识与底层大语言模型彻底解耦，知识更新时无需重复微调或重新部署模型。
- [11:00](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=660s) 上下文层通过标准 API 抽象层运行：由 AI Agent 识别意图并触发接口，定向抽取结构化子图（Subgraph）精准注入 Prompt。
- [12:55](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=775s) 大模型天然具备非确定性，但可通过锁定硬件、零温度（Temperature）、整数量化以及严格注入受限上下文，最大程度压制随机性与幻觉。
- [15:14](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=914s) 企业级 AI 的未来飞跃不在于更大的基座模型，而在于掌握上下文注入的主导权，利用推理力强的较小模型实现高性价比落地。
- [16:26](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=986s) 通过在 Prompt 中注入专有术语的本体（Ontology）定义并强制模型屏蔽外部公共记忆，可确保多智能体系统在专有业务概念上保持确定性一致。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=4s) 命名“上下文层”与大模型的信任危机
演讲开场，来自 Capital One 的软件工程高级经理 Ravi Teja Prabhala Venkata 带着轻松幽默的语气切入正题。他调侃道，不知是谁灵机一动，把原本饱受诟病的“机器人（Bot）”改叫成了“智能体（Agent）”，从此大家便不再批评它了；既然如此，他也想借此机会为业界尚未准确命名的关键领域树立一个新标签——“上下文层（Context Layer）”。尽管这个名字听起来很宽泛，但它所指向的正是当前所有前沿大模型乃至通用模型最致命的痛点：上下文窗口与可用性问题。市面上最顶尖的模型宣称能支持 100 万 token 的上下文，但真正落到企业级应用场景时，往往极难发挥实际价值。
为了打破冷场，Ravi 现场展示了一个在网络上广为流传的经典破局测试：让大模型数出“strawberry（草莓）”单词里有几个字母“r”。现场听众会心一笑，因为多数模型在这个简单问题上都“惨败收场（failed miserably）”。Ravi 打趣道，未来如果 AI 真的进化到了天网（Skynet）级别，在毁灭人类之前，恐怕得先把全地球的草莓搜出来统统消灭掉。玩笑背后是一个严肃的工程事实：大模型极易产生幻觉，而且连这类完全不涉及深奥领域专业知识的通用问题都会犯低级常识错误，导致用户根本无法轻易信任它们。“当一个用整个互联网数据训练出来的模型依然会犯下如此根本性的错误时，你是绝不可能真正信任它的。”
- 业界前沿模型即使提供高达 100 万 token 的上下文窗口，在复杂企业现实面前依旧难以有效发挥作用。
- 大模型在非专业领域（如计数单词字母）的频发幻觉，凸显出基础模型内在的不稳定性。
- 尽管业内常尝试“用 AI 检查 AI 生成的代码”或引入自动化评审来修复 Bug，但底层依然存在难以逾越的上下文脱节问题。
---
### [02:31](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=151s) 企业级 AI 的困境：技术正确但架构脱节
进入企业落地讨论后，Ravi 提出了一个深刻的工程反差：当工程师将 AI 工具（例如代码生成工具或各类 Assistant）用于单一、孤立的任务时，AI 给出的输出在语法或逻辑技术层面上可能是完全正确的，但在整体架构层面上却可能是彻底错误的（"technically correct but may not be architecturally correct"）。这是因为现代企业的代码库和业务体量极其庞大，根本无法整体塞入有限的上下文窗口中。模型只看得到孤立的代码片段，无法洞察系统全貌。
更深层的原因在于，公开训练的大模型只见过互联网上的通用信息，从未见过任何一家企业的核心专有数据。大模型对企业的合规要求（Compliance Requirements）、专有技术框架和内部知识产权一无所知；它更无法理解企业系统为何要设计成某种特定的形态。在金融等传统企业中，许多核心系统甚至包括平稳运行了 30 多年的大型机系统（Mainframe Systems）。这些系统背后的历史设计决策、特定基础设施限制，完全是大模型的盲区。如果缺乏企业上下文的精准注入，任何大语言模型都无法在企业内部发挥出真正的业务价值。
- 孤立任务中的代码补全无法替代全局架构的一致性与合规性审查。
- 企业内部拥有大量运行数十年的遗留系统（如 30 年以上的大型机架构），其设计意图与历史权衡从未公开于互联网。
- 前沿模型 100 万 token 的物理上限，注定了将所有企业资产一次性倒进 Prompt 是一种不切实际的幻想。
---
### [04:52](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=292s) 现有方案的局限：传统 RAG、微调与智能体
在探讨替代方案时，Ravi 逐一分析了业界现行的三种主流路径，并直接指出了它们各自的死穴。首先是许多人奉为救命稻草的检索增强生成（RAG）。Ravi 直言不讳地警告大家不要盲目相信传统 RAG，因为“传统的 RAG 不过是一个数学方程式（as good as a mathematical equation）”。传统 RAG 依靠向量相似度计算，一旦遇到企业独有的专有名词和私有业务流程，就会彻底失效；它缺乏对企业私有语言的语义理解，往往会将特定的专有流程错误地映射到向量数据库的无关角落，导致检索结果极其浅层，只能召回一些物理聚类相近的文档，根本做不到深层语义关联。
第二个路径是模型微调（Fine-Tuning）。虽然微调可以作为辅助手段，但成本高昂且无法解决大模型固有的“知识健忘症（Knowledge Amnesia / Source Amnesia）”——模型根本记不住自己生成的 token 究竟来自哪一份源数据。更致命的是，企业内部的知识库是动态变化的，每天或每周都会更新。如果依靠微调，企业就必须建立极其沉重的持续微调基础设施，反复调整 LoRA、QLoRA 等算法参数；一旦底层基座大模型迭代更新，所有的微调与部署流程就必须从头再来一遍，形成一个永无止境的维护泥潭。至于第三种路径——Agent 与 Skills，虽然智能体在处理有边界的孤立任务时表现出色，但当面对超出上下文容量的庞大企业系统时，多智能体之间的协同同样是在“撞大运”，无法给出确定性的架构保障。
| 解决路径 | 核心机制 | 在企业级落地中的致命局限 |
|:--- |:--- |:--- |
| **传统 RAG** | 向量嵌入与数学距离计算 | 缺乏专有语言语义理解，专有术语检索极易偏航，检索深度浅层。 |
| **模型微调 (Fine-Tuning)** | 基于企业语料调整模型权重 | 存在“知识源遗忘”，知识库更新需重复训练，算力维护成本极高。 |
| **Agent / Skills** | 目标拆解与子智能体编排 | 擅长受限边界任务，但依然受制于上下文窗口，缺乏系统级全局认知。 |
---
### [08:06](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=486s) 核心假说：构建企业自研的领域知识图谱
面对上述瓶颈，Ravi 提出了他的核心工程假说：企业不应再把希望寄托于冰冷的向量数学公式，而应当构建由人工深度管理与维护的“领域知识图谱（Domain Knowledge Graph）”。即便当前学术界涌现出了 Graph RAG、HippoRAG、PathRAG 等前沿概念，但只要它们底层依然依赖静态的数学映射模型，就终究会倒退回无法真正理解语义关联的原点。真正的解决之道，在于企业拥有完全受控的数据资产，不把命运交给随机的数学方程。
在这种自建图谱的体系下，知识图谱构成了企业不可动摇的“权威真实源（Authoritative Truth）”。当业务逻辑、代码设计或系统架构发生变动时，企业工程师可以直接在图谱中更新节点与关系，完全不需要重新训练或重新部署底层的大语言模型。企业对大模型基础设施的强依赖由此被彻底打破，唯一需要维护的仅仅是图谱数据库及其查询引擎。此外，企业还能根据自身的业务规范定制上下文审计策略——究竟是仅检索一阶子节点，还是采用深度优先搜索（DFS）遍历更多层级，完全由内部业务标准精准掌控。
- **不依赖随机数学映射**：摆脱纯向量检索的失真，转为由业务人员与架构师主动编排、维护的高保真结构化知识。
- **完全解耦大模型**：知识库变更无需重新训练或微调 LLM，消除了昂贵且频繁的模型发布周期。
- **图谱关联承载系统内涵**：不仅记录代码仓节点，更将历时 20 年的遗留系统演进动因（"the whys"）显式编码为图谱的关系边（Edges）。
---
### [11:00](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=660s) 上下文层的架构流程与意图编排
Ravi 随后拆解了“上下文层”的端到端交互架构。整个系统并非让大模型直接面对原始的数据库或外部用户输入，而是在两者之间构建起标准化的 API 抽象层。系统运行时，上层的 AI Agent 接收用户的复杂自然语言提问，首先执行严格的“意图分类（Intent Classification）”。精准识别意图后，Agent 会触发上下文层（Context Layer）暴露的专用 API。
```mermaid
flowchart TD
accTitle: 上下文层架构与请求流转图
subgraph Client["接入与编排层"]
U["用户查询 (User Query)"] --> AG["AI Agent"]
AG --> IC{"意图分类 (Intent Classification)"}
end
subgraph ContextLayer["上下文操作层"]
IC -->|"触发匹配接口"| API["Context Layer API"]
API -->|"定向检索子图"| KG[("领域知识图谱数据库\n(Domain Knowledge Graph)")]
KG -->|"返回精准子图 (Subgraph)"| API
end
subgraph LLMProcess["生成与交付层"]
API -->|"注入精准上下文 Prompt"| LLM["底座大语言模型 (LLM)"]
LLM -->|"生成确定性架构结果"| RES["客户端 / 用户交付"]
end
```
上下文层 API 接收到底层调用后，会深入企业的知识图谱数据库中定向提取高度相关的局部子图（Subgraph），并将这一富含历史设计决策与规范约束的上下文精准返回给 Agent。Agent 此时才将子图注入至 Prompt 中，交付给底层的大模型进行最终的推理与总结。通过这种方式，即便面对有 20 年历史的陈旧代码库，图谱中记录的设计初衷与节点关联也能丝毫不差地传递给大模型，使输出结果既具备技术严谨性，又完全契合企业架构要求。
- 意图分类（Intent Classification）是决定调用哪一个上下文 API 的关键前置步骤。
- 上下文层对外屏蔽了图数据库的复杂性，通过结构化子图（Subgraph）为 Prompt 提供极致精简且高密度的信息输入。
- 数据治理（Data Curation）必须做到严谨无瑕，图关系必须清晰承载历史架构决策的因果逻辑，才能确保推理输出的可靠性。
---
### [12:55](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=775s) 驯服不确定性：消除大模型幻觉的工程路径
谈到大语言模型的确定性（Determinism）时，Ravi 半开玩笑地称这个话题在技术圈简直像是“大逆不道（blasphemy）”的异端邪说。他现场询问听众是否有人了解实现大模型确定性的挑战。从底层的浮点数计算（Floating Point Operations）误差开始，大模型的非确定性就注定了针对同一个 Prompt 几乎不可能百分之百得到完全相同的输出。然而，企业级系统许多场景都要求严格的确定性动作，任由非确定性泛滥等同于向各种生产故障敞开大门。
为了解决这一难题，工程师必须采用一系列组合拳尽可能压制随机性：锁定物理运行硬件、将生成温度（Temperature）归零、精细化调整超参数，甚至采用整数量化（Integer Quantization）等工程手段。尽管 100% 的绝对确定性在物理上无法达到，但在工程上足以大幅降低随机扰动。当“极致压缩随机性的大模型”遇上“人工高度精选与核验的领域知识图谱”时，模型产生幻觉的生存空间被压缩殆尽。由于每次回答都被牢牢限制在精准注入的事实子图之内，系统整体实现了高度受限（Bounded）与受控的高可靠运行。
- 底层硬件执行浮点运算的微小差异是造成大模型天然不具备绝对确定性的物理根源。
- 降低非确定性的工程手段包括固定底层计算硬件、设置零温度、精细调节推理参数以及采用整数量化（Integer Quantization）。
- 高度受限的注入上下文与低随机性模型推理相结合，能够从根本上切断幻觉产生的滋生路径。
---
### [15:14](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=914s) 企业 AI 的下一次飞跃：超越模型缩放法则
在演讲的升华部分，Ravi 斩钉截铁地指出：“企业级 AI 的下一次飞跃，绝对不会是一个更好的基座模型（The next leap in enterprise AI is not a better model）。”虽然模型供应商依然会不断推出新模型，业界甚至出现了某些机构为了垄断知识，疯狂抢购古董罕见书籍、拿它们训练模型后再将原书销毁的极端传闻。但模型参数的 Scaling Law（缩放法则）必然会快速触及物理与边际成本的收益天花板，无限扩大的上下文窗口与无所不知的超级模型只是虚妄的幻想。
对于绝大多数企业级系统而言，业务场景压根不需要模型通晓整个互联网的所有细枝末节，真正需要的是**大模型在特定严格上下文约束下的逻辑理解与推理回应能力**。企业应当尽力摆脱对巨型前沿模型的盲目依赖，不再沉迷于为了填补信息差而大把挥霍 token、养成极其昂贵的资源浪费习惯。只要企业能够真正掌握上下文注入的主导权与控制力，哪怕使用更小、更经济的模型，也能够平稳、安全地将 AI 深度集成进每一个核心生产系统中。
- 试图依靠无限制增加模型参数和预训练语料来解决企业专业问题的思路已逼近天花板。
- 企业 AI 应用的核心在于“模型推理力”与“专有上下文注入”的结合，而非让模型充当通用百科全书。
- 摆脱对昂贵前沿巨型模型的依赖，既能大幅缩减企业 token 账单，又能建立更加可控的软件工程护城河。
---
### [16:26](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=986s) 现场问答：本体定义与专有业务术语的确定性约束
在演讲后的 Q&A 环节，一位现场听众提出了一个在多智能体系统（Agentic Systems）中极具代表性的难题：在实际业务中，不同智能体对于同一个业务专有概念或术语（Terminology）的理解可能存在严重分歧，从而导致决策不一致；上下文层是否包含类似“本体（Ontology）”的定义机制，以统一澄清特定业务术语的确切内涵？
Ravi 给予了高度肯定，并解释了上下文注入如何从根本上强制约束模型。他指出，在使用自建上下文层进行查询注入时，系统可以在 Prompt 中构建严格的指令围栏：要求大模型暂时“遗忘”其预训练习得的所有外部背景知识，仅在本次注入的上下文边界（Bounded Context Injection）之内作答。如果系统将该专有术语的权威内部定义直接作为子图上下文的一部分注入 Prompt，模型就会自动且安全地忽略该词在公共互联网语境中的通用含义，从而 100% 严格依照注入的专有业务定义执行逻辑判定。每一次针对系统的查询，其专属业务上下文与专有词汇本体定义都会作为 Prompt 的有机组成部分被强制灌注进模型中，最终实现跨 Agent 决策的高度一致性与确定性。
- 通过 Prompt 强约束指令，可强制大语言模型屏蔽外部训练记忆，严格基于单次注入的知识边界作答。
- 企业专有业务术语与定义直接通过上下文层注入，使得通用模型能够安全抹去公共语义偏差。
- 业务词汇本体（Ontology）通过知识图谱子图化为结构化数据注入，彻底解决了多智能体协作中术语理解割裂的顽疾。
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=4s) 当一个用整个互联网数据训练出来的模型依然会犯下如此根本性的错误时，你是绝不可能真正信任它的。 — Ravi Teja Prabhala Venkata
> [02:31](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=151s) technically correct but may not be architecturally correct — Ravi Teja Prabhala Venkata
> [04:52](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=292s) 传统的 RAG 不过是一个数学方程式（as good as a mathematical equation） — Ravi Teja Prabhala Venkata
> [15:14](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=914s) 企业级 AI 的下一次飞跃，绝对不会是一个更好的基座模型（The next leap in enterprise AI is not a better model） — Ravi Teja Prabhala Venkata
> [08:06](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=486s) 企业不应再把希望寄托于冰冷的向量数学公式，而应当构建由人工深度管理与维护的领域知识图谱。 — Ravi Teja Prabhala Venkata
> [12:55](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=775s) 谈到大语言模型的确定性时，这个话题在技术圈简直像是“大逆不道（blasphemy）”的异端邪说。 — Ravi Teja Prabhala Venkata
> [15:14](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=914s) 只要企业能够真正掌握上下文注入的主导权与控制力，哪怕使用更小、更经济的模型，也能够平稳、安全地将 AI 深度集成进每一个核心生产系统中。 — Ravi Teja Prabhala Venkata
> [16:26](https://www.youtube.com/watch?v=ldJkWYN2g1A&t=986s) 通过 Prompt 强约束指令，可强制大语言模型屏蔽外部训练记忆，严格基于单次注入的知识边界作答。 — Ravi Teja Prabhala Venkata
## Mentioned
Ravi Teja Prabhala Venkata, Capital One, Skynet, Graph RAG, HippoRAG, PathRAG, LoRA, QLoRA
## Video description
Ravi Teja Prabhala Venkata – Senior Manager, Software Engineering, Capital One
[Source](https://www.youtube.com/watch?v=ldJkWYN2g1A)
Source quotes
[0:04] Ravi Teja Prabhala Venkata: 当一个用整个互联网数据训练出来的模型依然会犯下如此根本性的错误时，你是绝不可能真正信任它的。
[2:31] Ravi Teja Prabhala Venkata: technically correct but may not be architecturally correct
[4:52] Ravi Teja Prabhala Venkata: 传统的 RAG 不过是一个数学方程式（as good as a mathematical equation）
[15:14] Ravi Teja Prabhala Venkata: 企业级 AI 的下一次飞跃，绝对不会是一个更好的基座模型（The next leap in enterprise AI is not a better model）
[8:06] Ravi Teja Prabhala Venkata: 企业不应再把希望寄托于冰冷的向量数学公式，而应当构建由人工深度管理与维护的领域知识图谱。
[12:55] Ravi Teja Prabhala Venkata: 谈到大语言模型的确定性时，这个话题在技术圈简直像是“大逆不道（blasphemy）”的异端邪说。
[15:14] Ravi Teja Prabhala Venkata: 只要企业能够真正掌握上下文注入的主导权与控制力，哪怕使用更小、更经济的模型，也能够平稳、安全地将 AI 深度集成进每一个核心生产系统中。
[16:26] Ravi Teja Prabhala Venkata: 通过 Prompt 强约束指令，可强制大语言模型屏蔽外部训练记忆，严格基于单次注入的知识边界作答。
Source quotes
[0:04] Ravi Teja Prabhala Venkata: When foundational models trained on the entire public internet make elementary mistakes on generic prompts, trusting them with mission-critical enterprise workloads becomes impossible.
[0:04] Ravi Teja Prabhala Venkata: Frontier context windows (up to 1 million tokens) still cannot absorb the full scope of enterprise codebases and institutional histories.
[0:04] Ravi Teja Prabhala Venkata: Automated AI code reviews catch isolated syntax errors but cannot verify broader architectural intent.
[3:03] Ravi Teja Prabhala Venkata: Enterprise environments possess massive proprietary datasets, strict compliance mandates, internal frameworks, and intellectual property that have never appeared on the public internet.
[8:06] Ravi Teja Prabhala Venkata: When organizational data or business rules change, teams update the graph directly without needing to retrain, fine-tune, or redeploy model checkpoints.
[8:06] Ravi Teja Prabhala Venkata: Edges in the knowledge graph capture legacy rationales and architectural decisions dating back decades across legacy repositories.
[12:26] Ravi Teja Prabhala Venkata: Achieving determinism in LLMs is an essential yet difficult engineering challenge, often complicated by low-level floating-point non-determinism across parallelized GPU hardware.
[12:26] Ravi Teja Prabhala Venkata: By feeding the LLM an explicit boundary of authoritative truth, developers prevent the model from drifting into speculative answers, effectively boxing its reasoning capabilities strictly within enterprise-approved data.
[15:14] Ravi Teja Prabhala Venkata: The next leap in enterprise AI is not a better model
[15:14] Ravi Teja Prabhala Venkata: In enterprise environments, models do not need access to the entire public internet; they primarily require the reasoning capability to synthesize strictly bounded, injected proprietary context without wasting millions of tokens on frontier model inference.
Original transcript
[0:04] My topic here today is the context layer. Um I
don't know who uh came up with the name agent
[0:10] uh for instead of naming it a bot, someone
did use a good name called agent. So people
[0:16] don't criticize it anymore. So I want to be the
guy to name something here called the context
[0:21] layer. It's so generic but people don't
have a right name for it so far, you know.
[0:26] Um anyways there is a big challenge with the
current set of frontier models or you know
[0:33] any model in general and that is the context
size window right we don't really have a lot
[0:39] of context size the best we can probably find in
the market is like 1 million tokens and there's
[0:45] hardly any use that you can you know find for it
especially when you're at an enterprise so let
[0:50] me quickly before I dive into that I want to have
a very quick icebreaker So there you go. Who has
[1:00] uh tested this? Has anyone ever tested that?
How did it uh work out for you? Good. Like it
[1:10] it actually failed miserably. Many models fail
miserably for this. I'm sure when uh AI reaches
[1:15] Skynet level, it'll go, you know, scout all the
strawberries across the planet and then, you know,
[1:21] they'll probably destroy everything, every single
strawberry on the planet. before they come after
[1:25] us. And this one too. Uh again, this is something
I stole from the internet. Long story short,
[1:34] uh the challenge with the current models is that
you can't trust them. You can't trust them easily.
[1:40] And if you look at this one, this is uh actually
this is true like it does a lot of you know uh it
[1:46] hallucinates a lot. So there's always a challenge
with getting the right response, right? So,
[1:51] and this is not exactly uh a domain related query,
it's pretty generic. Uh even the strawberry stuff
[1:57] too, it's pretty generic. So, when you have a
model that's trained on the internet of data and
[2:02] if it is still making some fundamental mistakes,
there's no way that you can actually trust it,
[2:08] right? There is definitely a problem. I'm pretty
sure the new models are solving these things.
[2:13] uh uh we did in in a previous session we did
talk about you know using AI uh to catch a
[2:19] lot of these AI generated code and you know
bugs and then you can reinforce them with test
[2:23] uh tests or you know reviews from AI itself
but then uh there is a fundamental challenge
[2:31] uh especially at an enterprise scale because you
don't have the context you want to inject the
[2:38] context enterprises have a lot of context and that
is what uh the major problem that we are going to
[2:43] deal with and I do want to present a hypothesis
so people they'll have some take-home assignment
[2:49] maybe they you know you can think about how to
deal with that uh there's some active research
[2:53] going on in this area but uh uh let's technically
start talking about that when you use AI for your
[3:03] isolated tasks you can get everything technically
correct but it may not be architecturally correct
[3:08] because when you use AI you may want to use it on
isolated tasks a feature work for example you can
[3:14] use cloud code or anything in general it may or
may not know everything about your code repository
[3:19] right it can be much larger than it can fit in the
context window so context is definitely a problem
[3:24] there so you may not be able to inject everything
to get the right output from the model um and
[3:32] again AI models are trained on generic information
anything that's available on the internet it's all
[3:39] trained on that but at at at your enterprises is
you know you have a lot of proprietary data and AI
[3:45] models have never seen any of that like those are
strictly proprietary so they may not be aware of
[3:52] your compliance requirements they may not be aware
of your frameworks your own internal IP anything
[3:57] in those lines so when you want to use AI for your
own internal activities you want to look at how to
[4:04] inject that context so you would get the maximum
benefit you know from an LLM or you know any AI
[4:10] component in general um they don't know why your
systems are built in specific ways maybe they've
[4:16] been around for 30 plus years right there's
some mainframe systems out there they are still
[4:22] operational and uh LLM may or may not understand
why there are things designed in a certain way
[4:29] there could be something on your infrastructure
side that the LLM may not be aware of so when
[4:33] you want to use that you want to make sure that
you inject as much context as possible given the
[4:40] constraints that we have with the models right
now. Even if you go with the frontier models,
[4:44] you may at best get 1 million tokens, right?
So this is uh my hypothesis to make sure
[4:52] uh we can start looking in that direction. But
before we dive into that, I want to make sure
[4:59] uh I explore the available options, right? Uh
the very first approach people would look at is,
[5:07] you know, obviously rag. Rag is my friend. I'm
going to call rag and say hey I need more context
[5:12] but rag is traditional rag it's not good if anyone
says so please don't believe them uh rag is as
[5:19] good as a mathematical equation right and if you
have proprietary terms proprietary knowledge rag
[5:25] will fail miserably because rag is never trained
on those terms and it is a mathematical equation
[5:31] so if you have a very accurate name for a
specific process it may place it somewhere
[5:36] else in your vector database And when you try to
query it, it'll never retrieve that because it
[5:41] doesn't know the semantic understanding. It will
not have a semantic knowledge of your your own
[5:48] proprietary language. So rag is not good enough.
But uh again even if you look at it like the
[5:53] retrievalss are shallow. It doesn't know it only
finds closely grouped documents together when you
[5:59] want to retrieve something from rag databases.
So what else can we do? Uh we can probably look
[6:05] at fine-tuning, right? Finetuning should solve
everything or does it? Fine-tuning is expensive.
[6:13] I think I have that somewhere with it. I don't
know. Anyway, so fine-tuning is not exactly
[6:19] uh a great solution. I mean, you should absolutely
fine tune it if you can, but that'll only be
[6:23] supplemental. There is a fundamental problem with
LLMs, which is uh LLMs have knowledge amnesia. So,
[6:30] they don't know the source of the the tokens that
they're producing from, right? So the source an
[6:37] initia is always baked into it. Obviously there
are some workarounds on that but fine-tuning is
[6:43] a rigorous process for a number of reasons like
your knowledge corpus changes on a daily basis
[6:49] or weekly basis whatever it is. So which
means fine-tuning will become a recurring
[6:53] activity. So you will have to have the necessary
infrastructure to fine-tune it. You may have to
[6:57] optimize Laura Qura any algorithm that you want
to use to fine-tune your models. And then you know
[7:02] obviously the models themselves evolve. So you
may have to fine-tune the new set of models again.
[7:08] Uh you definitely want to make sure that uh every
time there is some update on your end, you have to
[7:14] deploy them and this is a never- ending process.
Let's look at the other option. Agents and skills.
[7:21] Agents and skills are fantastic. They are really
good when it comes to isolated or bounded tasks,
[7:26] but they have a challenge, right? Which is again
context. So if you want to invoke cloud agents
[7:33] or if you build your own agent framework if
you invoke your own cloud skills to perform
[7:38] something a bunch of sub aents to deal with a
problem they're good but uh again they deal with
[7:44] limited context there. If your enterprise system
can never fit into any of these context windows,
[7:52] then everything is basically a gamble there,
right? They can do well in isolated tasks. They
[7:58] can do good job. They may not be perfect. So,
let's look at what we can do. Let's try to, you
[8:06] know, make sure we inject this domain data, domain
knowledge into everything that we know, right?
[8:13] um they they they're very complex but you need to
figure out a way to enable AI query that domain
[8:24] knowledge somehow and we do have some tools at our
disposal and this is my proposal for it right the
[8:32] best thing that you can do right now is instead of
relying on mathematical equation which is rag uh
[8:38] is to use a domain knowledge graph uh you can come
back and say Hey uh I heard about domain sorry
[8:44] graph rag I heard about hippograph I hippo rag I
heard about path rag you know it can be anything
[8:51] honestly but if it is a static mathematical model
you're back to square one it may be better than
[8:57] traditional rag but you still have the underlying
issue of not finding the semantic you know
[9:03] relevance when you want to store your documents
there so it's always uh a challenge there is
[9:09] some active research going on and this field But
my hypothesis here is to build your own domain
[9:16] knowledge graph where you don't leave things at
the mercy of a mathematical equation. You actually
[9:22] curate the data. You build your own data source.
You always maintain that if something changes in
[9:29] your knowledge, if something changes in your data,
it's totally at your control that you can go back
[9:34] and update it. And the AI or the LLM model doesn't
have to know anything about it. You don't have to
[9:39] retrain and redeploy anything. You don't have any
dependency on the LLM infrastructure. The only
[9:45] dependency you have is to run the infrastructure
that can support querying this database,
[9:51] right? And then context auditing. Uh they again uh
when you build this database, you need to define
[9:59] how to accurately query the database to retrieval
is the most important part, right? So uh let's say
[10:05] you have this know following my hypothesis let's
say you built your database now you need to define
[10:11] how to accurately retrieve the context do you want
to just look at the very first level of sub nodes
[10:18] or if you want to go you know like depth first
so it's totally up to how you want to query you
[10:25] know based on your own enterprise requirements or
based on your own business domain requirements. Um
[10:30] it is authoritative truth mostly because you
are curating it. You're not leaving anything
[10:35] to a random system. You are literally curating
everything that goes into this database. And then
[10:41] it is independent. Like I said, every time you
want to bump more information into the system,
[10:46] you would do so with your own systems
without relying on a generic system,
[10:52] a generic platform. So this is exactly what I
talked about. So you let's say you deploy an agent
[11:00] that takes or handles user query and then AI agent
kicks in and it understands the query and then
[11:07] it orchestrates through the context layer API. So
this is the challenge right. So again uh what API
[11:15] needs to be invoked is itself you know in itself
is a challenge because you need to make sure you
[11:20] identify the intent intent classification must
be done properly and then based on the intent
[11:24] classification mapping you would be able to
invoke the right API. But in this case for
[11:28] simplicity sake if I were to take this API this
will internally query this database and then it
[11:34] should be able to retrieve the relevant subgraph
and then pass it back to the AI agent which may
[11:41] use any LLM beneath the shell and be able to
produce the results that the customer or you
[11:46] know the client expects. Um again you can start uh
processing every single detail about your system.
[11:57] So if you have a very age-old repo uh like maybe
20 years old repo and you may want to document
[12:05] that you may want to start creating nodes and
you you may want to add the reason or the wise
[12:12] for the edges. So you may want to make sure that
data curation is absolutely you know flawless in
[12:17] these cases to get the optimal results. Go back.
Yeah. Uh why is it better than rank? Um again I
[12:27] I don't want to read through that but like I said
rag is something uh that doesn't truly understand
[12:35] everything right so especially at an enterprise
scale it'll never truly understand everything.
[12:40] So you may want to use uh the content layer
which is generic in this you know slide but
[12:46] the idea is to have semantic data curation at
your disposal that can be retrieved. Uh this is
[12:55] uh probably blasphemy because determinism in
LLAMs is a very hot topic but also you know
[13:03] a problematic topic. Uh who here knows the
challenges of bringing determinism in LLMs?
[13:10] Anyone? Great. So there are so many concerns like
uh the floating point uh operations right u they
[13:18] they themselves are the very first problem to deal
with and uh long story short you will most likely
[13:25] never get the same response for the same query for
any LLM. You may be able to control the randomness
[13:32] from an LLM but you won't truly ever get that.
So there are some options that you can pursue.
[13:39] You can lock in the physical hardware. You can
actually set temperature. You can actually change
[13:44] some parameters. You may even uh you know uh look
for a quantization, an integer quantization. Uh
[13:52] you can definitely reduce randomness. And
then for something that needs deterministic
[13:58] actions, you definitely want to reduce the
non-deterministic nature of an LLM. Otherwise,
[14:03] you are basically exposing your system to a number
of problems. So you definitely want to do this.
[14:08] uh this is not uh 100% deterministic. It can never
be achieved but you definitely can achieve it to
[14:15] some extent and when you do that when you deploy
this along with your own curated database which is
[14:22] you know basically a graph database there uh you
should be able to have greater accuracy when you
[14:27] retrieve data based on the user queries. Yeah,
hallucination uh something that I want to talk
[14:33] about too. So AI can hallucinate a lot and you
definitely have to babysit it through a bunch of
[14:40] uh issues if you have never encountered them
before and that's not fun for anyone right
[14:45] um and the examples that I shared earlier uh
at the beginning of my session they are kind
[14:49] of some some of those things they hallucinate
a lot but when you actually make your LLM know
[14:55] less deter less non-deterministic and you also
have extremely curated well curated data at your
[15:02] disposal for retrieval you're basically
eliminating any chance of you know having
[15:07] hallucinations in your responses. So you can
make your system truly bounded to a very large
[15:13] extent. The next leap in enter AI is not a better
model. Obviously better models will happen. Uh you
[15:20] know there are organizations that are buying rare
books training them and then burning those books
[15:26] uh to make sure you know the knowledge is
exclusively maintained. But you can only scale up
[15:31] to a certain extent. You cannot have infinite you
know context window. You cannot have significantly
[15:39] greater models better models. Uh so the scaling
up can only go to a level. It can hit the ceiling
[15:46] very quickly which means we have to do much better
in injecting the context. It's always the context.
[15:53] The AI doesn't care. Actually for many systems
you don't even need the internet of knowledge.
[15:57] you need the LLM's capability to respond given
specific context. So you try to minimize your
[16:04] dependency on frontier models where you you'd
be burning lots and lots of tokens and you know
[16:10] inculcating a lot of you know expensive habits
there. But uh if you have better control on the
[16:16] injection of the context you can truly integrate
AI into any system. Thank you. So um you talked
[16:26] about knowledge graph and graph engineering is one
of the hottest topics these days. Um one of the
[16:32] things however I feel uh in agentic system that's
important is the business domain uh concepts and
[16:37] the terminology and a lot of the agents for the
same business term can have inconsistent outcomes
[16:43] or understanding. Is there something related
to ontology or something that your context
[16:48] layer can build to clarify that this business term
kind of relates to this meaning and so that agents
[16:53] are able to make a consistent decisioning based on
those terms? Yeah, that's a great question and I'm
[16:58] pretty sure some of you already know the answer
there. So when you actually tap into rag or you
[17:04] know any kind of rag or any other queryable system
you can enforce the LLM to forget about everything
[17:11] and respond only within that domain or only within
that bounded context injection. So even even
[17:16] though it knows that term you know on a generic
uh uh side it'll still restrict it and it'll only
[17:23] respond based on the injected context. So if your
system can inject the proprietary term and the
[17:29] definition of the term, the LLM safely ignores
everything else and only uses the injection.
[17:37] Yes. Yes. So every query that that is made
to the system here you would get the context
[17:42] injected as part of the prompt and the
prompt is then processed by the LL app.
