Panasonic Engineer Builds Self-Scoring RAG to End AI Hallucinations

Building Self-Evaluating OpenAI Agents for Profiles & Documents with Panasonic Avionics Corporation — Ai4

Ai4

Separating query parsing from extraction and using an independent scoring model guarantees deterministic answer reproducibility while preventing enterprise language models from validating their own errors.

AI-assisted summary of Building Self-Evaluating OpenAI Agents for Profiles & Documents with Panasonic Avionics Corporation — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 16, 2026, this presentation features Vijay Panwar, Staff Engineer at Panasonic Avionics Corporation, discussing the development of self-evaluating OpenAI agents for enterprise documents and profiles. The talk outlines production-grade design patterns—including chunking, embeddings, retrieval-augmented generation (RAG), and evaluation controls—to eliminate hallucinations and establish auditable accuracy in document-centric AI workflows.

## TL;DR Vijay Panwar presents the Doc QA Agent, a patent-pending system engineered to provide deterministic, verifiable answers from complex enterprise files such as legal contracts, medical reports, and candidate résumés. Powered by a FastAPI backend on AWS and vector databases like Chroma and OpenSearch, the architecture uses GPT-4o-mini solely for query normalization while relying on the CAT LLM to achieve 96% accuracy with independent judge scoring. The solution prioritizes inspectable citations down to paragraph-level references, strict multi-tenant data isolation, and explicit 'insufficient evidence' fallbacks when documents lack requested facts. Originating from Panwar's personal experience deciphering accident and legal paperwork, the tool targets enterprise applications across HR, legal compliance, and research literature analysis.

## Atmosphere & style The presentation carries a pragmatic, engineer-driven tone focused squarely on verifiable truth rather than generative AI hype. Panwar's narrative feels authentic and grounded, pivoting from a vulnerable personal anecdote about navigating chaotic accident paperwork to confident architectural deep dives. The live demo and subsequent Q&A create an engaged, collegial atmosphere where the speaker candidly discusses model boundaries, error handling, and roadmap features like multimodal extraction.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=4s) Vijay Panwar introduces the Doc QA Agent to solve the issue of inconsistent, non-deterministic answers and hallucinations in standard AI chatbots when querying enterprise documents. - [01:50](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=110s) The system is engineered for deterministic reproducibility, ensuring that asking the same query against an unchanged document produces identical results across 100 runs. - [01:50](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=110s) An independent evaluation layer scores generated answers directly against source text without retrieval bias, pairing with the CAT LLM to deliver a 96% accuracy rate. - [04:55](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=295s) OpenAI GPT-4o-mini is strictly isolated to query parsing and normalization, preventing confidential enterprise document content from being sent to external public model endpoints. - [04:55](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=295s) The backend architecture is built with FastAPI on Python 3.13 and AWS cloud infrastructure, utilizing page-aware chunking, token cost controls, and automated PII redaction prior to S3 storage. - [07:36](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=456s) Enterprise security and compliance are maintained through OAuth/RBAC, 60-minute session tokens, tenant data isolation, and comprehensive query tracing via AWS CloudTrail and X-Ray. - [07:36](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=456s) The agent operates on four core design pillars: Grounded (confidence is scored, not asserted), Inspectable, Self-Aware (explicitly reporting 'insufficient evidence' when data is missing), and Trustworthy. - [10:20](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=620s) Interactive citations pinpoint exact page numbers, sections, and paragraphs so human reviewers can perform immediate side-by-side verification against the original files. - [10:20](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=620s) Key enterprise use cases include legal contract analysis, HR résumé screening against job descriptions, financial compliance audits, and dense scientific literature extraction. - [13:09](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=789s) The platform ingests unstructured documents directly without manual data massaging and can connect natively to existing governed enterprise repositories like OneDrive and SharePoint.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=4s) Introduction and Speaker Background

Vijay Panwar, Staff Engineer (Member of Technical Staff IV) at Panasonic Avionics Corporation, introduces his presentation on building a "Doc QA Agent" — a self-evaluating OpenAI agent designed for enterprise profiles and documents. Organizations routinely deal with complex, high-stakes paperwork such as résumés, legal contracts, medical files, or 100-page reports where crucial information is buried. When users query conventional AI chatbots with such documents, they often encounter inconsistent answers: "First time is give you A, second time is give you B, third time give you A and B combined." The Doc QA Agent addresses this lack of consistency and verifiable ground truth.

Panwar shares his technical background, which includes over 12 years of software engineering experience and three years dedicated to artificial intelligence. He is a Senior Member of IEEE and a Full Member of Sigma Xi, actively reviewing research papers and contributing to AI model developments. He has authored around 12 research papers focusing primarily on AI-driven query optimization and leveraging web APIs. In addition to being an AWS Certified Cloud Practitioner and AWS Certified AI Practitioner, he was a winner at the MIT Global AI Hackathon (2025), where his team developed a model to evaluate system accuracy.

- **IEEE & Sigma Xi**: Senior Member / Full Member contributing to AI research and peer reviews. - **Publications**: ~12 research papers on AI query optimization and web API integration. - **Certifications**: AWS Certified Cloud Practitioner, AWS Certified AI Practitioner. - **Hackathon Achievement**: Winner of the MIT Global AI Hackathon 2025 for accuracy evaluation models.

---

[01:50](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=110s) The Problem with Overconfident Chat Models and Document Q&A

Modern chat models respond with high confidence, but Panwar emphasizes: "The chat models are confident, but that's not the same as correct." In domain-specific documents across legal, financial, and recruitment sectors, inconsistent responses create operational risk. Panwar's system (currently with a patent pending) guarantees deterministic reproducibility: asking the same question 100 times against an unchanged document will produce the exact same response every single time.

The backend is built with FastAPI running on Python 3.13 and hosted in the AWS cloud. Future feature iterations incorporate automated redaction of sensitive and personally identifiable information (PII) before documents are persisted into Amazon S3 storage.

Retrieval-Augmented Generation (RAG) forms the backbone of the agent. Instead of passively waiting for an unguided generation, RAG dynamically captures and extracts relevant chunks from uploaded documents in real time. To ensure reliability, the system incorporates an independent evaluation mechanism: a secondary model call that evaluates the generated answer against the source text without having seen the intermediate retrieval steps, scoring the output and guarding against the tendency of models to simply agree with themselves. Running alongside the pipeline is the CAT model (LLM), which achieves a 96% accuracy rate.

```mermaid flowchart TD accTitle: System Architecture and Processing Pipeline subgraph Client ["Client Layer"] UI["Web Browser Interface"] end

subgraph Backend ["FastAPI Backend (Python 3.13)"] API["API Gateway / ECS Fargate"] NLP["OpenAI GPT-4o-mini (Query Parsing Only)"] end

subgraph StorageAI ["Data & AI Infrastructure"] S3["AWS S3 (Document Store with PII Redaction)"] VEC["Chroma / OpenSearch (Vector Index)"] SQL["SQLite Database (Metadata Mapping)"] CAT["CAT LLM (96% Accuracy Engine & Scoring)"] end

UI -->|"User Query / File Upload"| API API -->|"Sanitized Parsing"| NLP API -->|"Store Raw / Redacted Docs"| S3 API -->|"Document Chunks & Embeddings"| VEC API -->|"Index Metadata & Tracking"| SQL VEC -->|"Context Retrieval"| CAT CAT -->|"Scored & Grounded Answer"| API API -->|"Response with Citations & Scores"| UI ```

---

[04:55](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=295s) System Architecture and AWS Production Hardening

The end-to-end architecture is structured into clear functional layers:

1. **Browser / Frontend**: Provides the user interface to upload documents, issue queries, view citations, and inspect confidence scores. 2. **Backend**: Powered by FastAPI running Python 3.13 to handle business logic, request routing, and system orchestration. 3. **Data and AI Storage**: - **Chroma / OpenSearch**: Vector index database storing chunked page embeddings for fast similarity retrieval. - **SQLite**: Local metadata store tracking document mapping and cloud storage locations (such as Amazon S3). - **OpenAI GPT-4o-mini**: A constrained NLP agent used solely to parse and normalize messy or colloquial user queries into structured English prompts. No confidential document data is sent to external public model endpoints.

For enterprise production hardening, the pipeline implements page-aware chunking with overlapping boundaries and a three-tier engineering retrieval strategy that decouples evaluation from extraction. Cost controls are embedded to avoid token runaway. The cloud deployment road map integrates Amazon S3 for storage, Amazon OpenSearch for scalable vector search, Amazon ECS Fargate for compute, and AWS API Gateway for endpoint management.

---

[07:36](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=456s) Enterprise Security, Multi-Tenancy, and Core Pillars

Enterprise adoption requires strict data security and compliance safeguards:

- **Authentication & RBAC**: Secure login with OAuth and Role-Based Access Control (RBAC) ensuring users only view authorized documents. - **Short-Lived Session Tokens**: Default token lifespan of 60 minutes, configurable per enterprise customer requirements. - **Encryption & Tenant Isolation**: Full data encryption in transit and at rest. Strict per-tenant data isolation ensures multi-tenant deployments never suffer cross-tenant data leakage or document collisions when identical filenames are uploaded. - **Auditability & Observability**: Every query is logged and traceable through AWS X-Ray, supported by comprehensive audit trails via AWS CloudTrail.

The execution workflow is straightforward: documents are uploaded, chunked, and embedded into the vector index regardless of size (tested from single-page résumés up to 3,000-page books). When a user submits a query, the system retrieves relevant chunks, generates a constrained response, and passes the output to an independent judge model for scoring.

Panwar frames the system around four core design pillars: 1. **Grounded**: Output confidence is "scored, not asserted." 2. **Inspectable**: Users can directly inspect extracted citations and validate responses against original source text. 3. **Self-Aware ("Knows what it knows and what it doesn't know")**: When queried on information absent from the uploaded text, the agent explicitly returns `"insufficient evidence"` instead of hallucinating answers. 4. **Trustworthy**: Builds verifiable trust for high-stakes enterprise workflows.

---

[10:20](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=620s) Live Demo, Use Cases, and Personal Genesis

Demonstrating the live application interface, Panwar highlights the interactive citation feature. When a grounded question is answered, clicking on the source citation opens a granular reference showing the exact page number, section, and paragraph from which the data was pulled. If a query falls outside the uploaded scope, the agent responds with a clear "insufficient evidence" notice.

Panwar shares the personal event that inspired this project: following an accident the prior year, he was overwhelmed by stacks of complex police reports, legal paperwork, and conflicting statements from different attorneys. Unsure of how to navigate the documents, he built this system to analyze the files and determine liability objectively: "I'm checking what is my lawyer is saying, what is the other team lawyer is saying, what is going on, who is the fault party."

Key enterprise application areas include: - **Legal Teams**: Reviewing NDAs, litigation filings, and complex contracts. - **Finance & Compliance**: Inspecting financial disclosures and compliance audits with automated PII redaction prior to cloud storage. - **Recruitment & HR**: Screening large volumes of candidate résumés against target job profiles, surfacing best matches with specific qualification citations. - **Research & Development**: Parsing dense scientific literature to pull out key metrics, novel contributions, specific tables, and column data across multi-page research papers.

---

[13:09](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=789s) Q&A: Judge Scoring, Data Ingestion, and Multimodal Capabilities

During the post-presentation Q&A session, audience members ask about evaluation logic, ingestion constraints, storage integration, and data types:

- **Independent Judge & Scoring Workflow**: An attendee asks whether a separate AI agent judges the output. Panwar clarifies that the OpenAI layer only handles initial query translation/NLP, while the backend CAT LLM performs the extraction and scoring against the vector store. The interface displays confidence scores alongside the specific page citations (e.g., showing matches on page 2, paragraphs 1 and 3) so users can perform an "apple-to-apple comparison" between the model's output and the raw document. - **Unstructured Data Handling**: An attendee asks whether raw documents require pre-processing or structuring before RAG ingestion. Panwar explains that documents are ingested directly without manual data massaging: "RAG is work like a newspaper. It's like it will capture all the missing pieces... available in the data." The vector database handles matching whether the content resides on page 1 or page 20. - **Model Accuracy Comparison**: When asked if other LLMs could replace the CAT model, Panwar notes that CAT delivers ~96% accuracy, whereas alternative models drop to 93–94%, leaving the choice dependent on required precision. - **Multimodal & Inline Diagram Support**: An attendee who builds similar client systems asks if the tool handles inline images or architecture diagrams within PDFs. Panwar explains that the current release supports text-based PDF and DOCX files only, with multimodal image extraction planned on the future development road map. - **Direct Enterprise Storage Integration (SSO/SharePoint/OneDrive)**: An attendee asks whether existing governed document repositories (OneDrive, Google Drive, Egnyte) must be migrated to the agent's storage. Panwar notes that if the Doc QA agent is integrated directly into the company's existing environment with database/storage access permissions, it can pull and vectorize the documents in place without requiring external document migration. - **Handling the 4% Error Margin**: A participant inquires how the remaining 4% error/hallucination gap in the 96% accuracy metric is handled. Panwar reiterates that side-by-side source extraction allows human reviewers to verify claims instantly. Demonstrating on his own résumé, he shows how the system confirms his 12 years of experience and staff engineer title with exact page references, preventing the agent from "making some response by itself to making you happy."

## Notable quotes > [00:04](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=4s) First time is give you A, second time is give you B, third time give you A and B combined. — Vijay Panwar

> [01:50](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=110s) The chat models are confident, but that's not the same as correct. — Vijay Panwar

> [07:36](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=456s) scored, not asserted — Vijay Panwar

> [07:36](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=456s) Knows what it knows and what it doesn't know — Vijay Panwar

> [07:36](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=456s) insufficient evidence — Vijay Panwar

> [10:20](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=620s) I'm checking what is my lawyer is saying, what is the other team lawyer is saying, what is going on, who is the fault party. — Vijay Panwar

> [13:09](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=789s) apple-to-apple comparison — Vijay Panwar

> [13:09](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=789s) RAG is work like a newspaper. It's like it will capture all the missing pieces... available in the data. — Vijay Panwar

> [13:09](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=789s) making some response by itself to making you happy. — Vijay Panwar

## Mentioned Vijay Panwar, Panasonic Avionics Corporation, Ai4, IEEE, Sigma Xi, MIT Global AI Hackathon, OpenAI, GPT-4o-mini, FastAPI, Python, AWS, Amazon S3, Amazon OpenSearch, Amazon ECS Fargate, AWS API Gateway, AWS X-Ray, AWS CloudTrail, Chroma, SQLite, OneDrive, Google Drive, Egnyte, SharePoint

## Video description Vijay Panwar - Member of Technical Staff IV (Staff Engineer), Panasonic Avionics Corporation

The talk will demonstrate how an OpenAI-powered AI agent can read PDFs and enterprise documents and answer user questions with verifiable, source-grounded accuracy. As organizations struggle to extract insights from large volumes of legal, financial, and recruitment documents, this session will address the practical challenges of hallucinations, scalability, and trust in document-centric AI systems. The talk will include production-grade design patterns such as document ingestion pipelines, chunking and embeddings, retrieval-augmented generation (RAG), and response constraints using an application-level OpenAI API key. Attendees will learn how to implement evaluation controls, grounding strategies, and failure handling to ensure reliable, auditable answers in enterprise environments. This session is designed for practitioners who want to move beyond demos and deploy responsible, scalable document Q&A agents that deliver measurable business value.

[Source](https://www.youtube.com/watch?v=gmOXbNJiqpY)

中文简报

松下工程师构建自评分RAG以消除AI幻觉 将查询解析与信息提取分离并采用独立评分模型，在确保确定性答案复现的同时，防止企业级语言模型自我验证错误。 Vijay Panwar展示了正在申请专利的Doc QA Agent系统，该系统旨在从法律合同、医疗报告和求职简历等复杂企业文件中提供确定且可验证的答案。该架构基于AWS上的FastAPI后端以及Chroma和OpenSearch等向量数据库构建，仅使用GPT-4o-mini进行查询规范化，并依托CAT LLM与独立评审模型评分实现了96%的准确率。该解决方案优先提供细化至段落级的可审查引用、严格的多租户数据隔离，并在文档缺少所需事实依据时提供明确的“证据不足”兜底反馈。该工具源于Panwar解读事故及法律文书的个人经历，主要面向人力资源、法律合规以及研究文献分析等企业级应用。 Vijay Panwar介绍了Doc QA Agent，旨在解决标准AI聊天机器人在查询企业文档时出现的回答不一致、非确定性及幻觉问题。 (0:04) 该系统专为确定性复现设计，确保在针对未修改文档提出相同查询时，100次运行均能输出完全一致的结果。 (1:50) 独立评估层可在无检索偏差的情况下直接根据源文本对生成的答案进行打分，搭配CAT LLM实现了96%的准确率。 (1:50) OpenAI的GPT-4o-mini被严格隔离并仅用于查询解析与规范化，防止企业保密文档内容被发送至外部公开模型端点。 (4:55) 后端架构基于Python 3.13下的FastAPI和AWS云基础设施构建，在存入S3之前采用了页面感知分块、Token成本控制以及自动化PII脱敏。 (4:55)

Reading summary

Vijay Panwar of Panasonic Avionics Corporation argues that enterprise document retrieval requires deterministic, self-evaluating architectures to overcome the overconfidence and inconsistency inherent in standard conversational AI models. The Reliability Gap in Conventional Document QA (1:50) Modern large language models respond with high confidence, but Panwar cautions that conversational fluency does not equate to factual accuracy. In document-heavy domains such as legal compliance, medical records, finance, and hiring, conventional chatbots frequently generate inconsistent responses to identical queries over time. This lack of verifiable ground truth introduces significant operational risk when teams must parse critical, complex files. The project originated from Panwar’s personal experience managing conflicting legal filings and police reports following an accident. To eliminate arbitrary variance, the system introduces a patent-pending deterministic framework ensuring that submitting an identical query 100 times against an unchanged document returns the exact same result every time. Decoupled Architecture and Independent Verification (4:55) The technical backend is built on FastAPI and Python 3.13, utilizing Amazon Web Services infrastructure. To prevent data leakage and maintain security, the architecture restricts public model calls: OpenAI’s GPT-4o-mini is deployed solely to sanitize and translate unstructured or colloquial user queries into normalized prompts, ensuring no confidential enterprise document text is transmitted to external public endpoints. Retrieval-Augmented Generation extracts content chunks from an indexed vector database (Chroma or Amazon OpenSearch) alongside SQLite metadata tracking. Retrieval and validation are decoupled: the internal CAT model extracts answers and independently evaluates the output against source chunks without viewing intermediate retrieval steps. Panwar reports that the CAT model achieves 96% accuracy, outperforming alternative models that achieved 93% to 94% in internal benchmarks. Enterprise Governance, Multitenancy, and Storage (7:36) Production deployment is structured around strict data protection protocols. Access is governed by OAuth authentication and role-based access control, backed by short-lived session tokens configured with a default lifespan of 60 minutes. All data is encrypted in transit and at rest, and strict multitenant isolation prevents cross-tenant data exposure even when identical document filenames are uploaded by different clients. Operational transparency is maintained through AWS CloudTrail audits and AWS X-Ray request tracing. For enterprise workflows, the roadmap incorporates automated PII redaction prior to long-term storage in Amazon S3. Panwar notes that organizations using existing governance repositories—such as OneDrive, Google Drive, or SharePoint—can integrate the agent directly via storage permissions to index and vectorize documents in place without requiring bulk file migration. Four Operating Pillars and Inspectable Citations (10:20) The system is anchored by four design principles: grounded confidence that is measured numerically rather than asserted; full inspectability; self-awareness that outputs an explicit 'insufficient evidence' notice rather than guessing when source data is missing; and institutional trustworthiness. The ingestion pipeline supports files ranging from single-page résumés to 3,000-page volumes. Rather than relying on automated scoring alone, the interface provides interactive citations displaying the exact page number, section, and paragraph of each source excerpt. This mechanism allows human reviewers to conduct side-by-side verification across high-volume contract reviews, candidate profile screenings, and multi-page scientific literature. The current implementation remains limited to text-based PDF and DOCX files, with multimodal extraction for inline diagrams and images designated for future releases. Additionally, Panwar acknowledges an existing 4% error gap under the CAT model's 96% accuracy rate, relying on granular, human-inspectable citations to catch remaining discrepancies.

精炼阅读摘要

松下航空电子（Panasonic Avionics Corporation）资深工程师 Vijay Panwar 指出，常规大语言模型在处理高风险企业文档时普遍存在过度自信与输出不稳定的缺陷；为此，他主导开发了具备自评估机制的文档问答智能体（Doc QA Agent），利用检索增强生成（RAG）与独立的评估模型解耦，实现答案的确定性复现与逐段证据溯源。 消除模型过度自信与实现确定性问答 (1:50) Panwar 强调，现代对话模型回答问题时语气笃定，但这并不等同于内容准确。企业在处理简历、法律合同、医疗档案或数百页研究报告时，常规模型常因随机性产生前后矛盾的回复。针对这一风险，该项正在申请专利的系统致力于建立确定性的输出机制，声称针对未修改的文档连续提出相同问题 100 次，系统每一次都会给出完全一致的回答。 该方案以 RAG 技术为核心，在文档上传后实时动态捕获并提取相关切片。为防止大模型陷入自我验证的偏差，系统引入了独立评估机制：在提取上下文后，由二次模型调用将生成的答案与原始文本进行比对打分，且该评分步骤对中间检索过程完全不可见。系统配套运行的 CAT 大语言模型在基准测试中实现了 96% 的准确率，高于候选对比模型 93% 至 94% 的水平。 关注成本与隐私的云端解耦架构设计 (4:55) 系统的底层技术栈基于运行在 Python 3.13 环境下的 FastAPI 构建，并部署于 AWS 云平台。在架构设计上，团队将系统划分为前端交互、后端服务以及数据与 AI 基础设施层。为了避免调用公共端点导致敏感数据泄露，系统仅调用 OpenAI GPT-4o-mini 解析并规范化用户输入的非正式自然语言查询，严禁将企业机密文档内容传递给外部公共大模型端点。 在数据存储与索引方面，系统采用 Chroma 或 Amazon OpenSearch 存储包含页面信息的向量嵌入，配合 SQLite 记录文档映射与元数据。为了控制令牌成本并提高处理效率，系统采用了带有重叠边界的页面感知切片技术，并将文档抽取与质量评估解耦。云端工程化演化路径涵盖了通过 AWS API Gateway 进行接口管理、借助 Amazon ECS Fargate 提供计算弹性，并利用 Amazon S3 实现文档的长期持久化存储。 企业级安全合规体系与四大设计支柱 (7:36) 面向严苛的企业合规要求，系统内建了多项安全防护措施：通过 OAuth 与基于角色的访问控制（RBAC）限制文档查阅权限；默认提供生命周期为 60 分钟的临时会话令牌，并支持企业按需配置；所有数据在传输与静态存储阶段均完成端到端加密，并具备严格的多租户隔离能力，杜绝同名文件冲突及跨租户数据泄露风险；所有调用链路由 AWS X-Ray 和 AWS CloudTrail 全程追踪审计。 Panwar 将系统的工程哲学归纳为四大设计支柱：其一是可依附（Grounded），系统置信度由客观模型打分生成而非模型主观断言；其二是可审查（Inspectable），用户可直接查验引文并溯源原文；其三是自知边界（Self-Aware），面对文档未涵盖的提问，系统主动返回“证据不足”的提示以抑制幻觉；其四是可信赖（Trustworthy），为高风险的企业级决策流程提供具备确定性的事实保障。 多行业落地场景与当前技术能力边界 (10:20) Panwar 透露该项目的起因源自其在交通事故后面对堆积如山的警局报告与律师冲突陈述的个人经历。在企业实践中，该方案已被规划应用于法律合同审查、金融合规审计（集成在文档存入 S3 前的自动 PII 脱敏功能）、海量求职简历精准筛选，以及从复杂科研论文中自动化提取关键指标与表格数据。系统经验证可支持从单页简历到 3000 页书籍的免手动预处理直接向量化，且若授予对应环境权限，可就地挂接企业既有的 OneDrive 或 SharePoint 等受管存储。 但在实际落地层面，系统仍面临明确的技术边界：当前公开发布版本仅支持纯文本格式的 PDF 与 DOCX 文件，针对内嵌架构图和复杂图像的解析仍处于多模态规划阶段。此外，针对 CAT 模型 96% 准确率所遗留的 4% 潜在错误或幻觉空间，系统并非完全自动化闭环，而是通过在界面提供精准到页码与段落的交互式并排引文，依托人工复核来最终消除模型取悦用户的虚假生成。 该智能体通过将问答检索与独立评判解耦，为高风险领域的文档交互确立了确定性与可追溯性的工程标杆；然而，该系统目前仍受制于纯文本输入限制，多模态图表理解尚待完善，且最终可信度依然部分依赖人工对剩余 4% 误差敞口的交互式核验。

Detailed notes — zh

## Background 本内容由 Ai4 于 2026 年 8 月 16 日在 YouTube 平台发布。视频主要记录了 Panasonic Avionics Corporation 的主任工程师 Vijay Panwar 的演讲，内容聚焦于如何构建具备自我评估能力的 OpenAI 智能体，以在法律、金融和招聘等企业级文档场景下实现具备高准确度且可溯源的确定性问答。 ## TL;DR Vijay Panwar 详细展示了针对复杂长文档的 Doc QA Agent 架构设计，直击大模型回答漂移与幻觉的痛点。系统采用确定性 RAG 机制与准确率达 96% 的 CAT 语言模型，并结合独立评估机制对输出进行打分与引用校验。在工程落地层面，系统通过 FastAPI、Chroma 向量数据库以及 AWS 基础设施实现了成本优化、PII 脱敏、RBAC 权限控制与多租户安全隔离，广泛适用于法律诉讼案卷比对、财务报表分析与简历智能筛选等场景。 ## Atmosphere & style 整场演讲呈现出严谨、务实且极具说服力的工程技术分享氛围。Vijay Panwar 展现了资深工程师沉稳自信的专业素养，不仅结合自身诉讼经历与简历比对等真实案例让技术细节生动落地，还在问答环节坦诚回应了多模态边界与准确率把控等硬核技术问题，给观众带来一种直击企业痛点、开箱即用的技术信任感与踏实感。 ## Key takeaways - [01:50](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=110s) 商业对话大模型存在“过度自信”但输出不稳定的问题，Doc QA Agent 旨在通过确定性机制解决文档问答中的幻觉与漂移。 - [01:50](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=110s) 系统采用 RAG 架构配合准确率达 96% 的 CAT 语言模型，确保同一问题执行 100 次均能获得完全一致的事实锚定回答。 - [01:50](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=110s) 引入独立的第二阶段评估机制，将主答案生成与打分校验解耦，杜绝依靠单一模型自圆其说的风险。 - [04:47](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=287s) 架构采用 FastAPI 与 Chroma 向量数据库，OpenAI gpt-4o-mini 仅负责解析用户输入意图，敏感数据不泄露给外部商业大模型。 - [06:14](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=374s) 在企业安全上整合 OAuth、RBAC 与 60 分钟短效令牌，并在入云前对 PII 敏感信息进行自动脱敏。 - [06:14](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=374s) 系统支持严格的租户级（Tenant-level）物理与逻辑隔离，并全面接入 AWS X-Ray 与审计日志以实现全链路溯源。 - [08:53](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=533s) 整套方案立足于事实锚定、可人工检验、边界清晰与开箱即用四大支柱，遇到超出文档范围的问题直接返回证据不足。 - [08:53](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=533s) 应用场景覆盖法律卷宗分析、脱敏财务合规、HR 简历自动匹配以及科研论文重点信息快速定位。 - [14:37](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=877s) RAG 向量检索机制支持直接摄入非结构化 PDF/DOCX 文档，无需人工规整或复杂的前期数据清洗。 - [17:23](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=1043s) 前端设计支持双向证据溯源，通过将答案与原始文档页码及段落高亮比对，弥补模型残余的不确定性并建立人工信任。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=4s) 个人背景与 Doc QA Agent 的诞生背景 演讲者 Vijay Panwar 介绍了自己构建的这套名为“Doc QA Agent”的系统，旨在解决长文档和复杂资料中信息检索与问答的一致性难题。在日常工作中，无论是个人简历、法律合同、医疗报告，还是长达数百页的专业文档，用户往往难以快速定位关键信息。如果直接将这些文档丢给普通的 AI 大模型，常常会遇到回答漂移的问题——“第一次提问给出结果 A，第二次变成结果 B，第三次则是 A 和 B 的混合体”。为了彻底消除这种不确定性，Vijay 开发了具备自我评估能力的文档智能体系统。 Vijay 拥有 12 年以上的软件工程经验，并在近 3 年全职深耕 AI 领域，不仅发表了十余篇关于 AI 驱动查询优化和 Web API 调用的研究论文，还持有 AWS 云及 AWS AI Practitioner 认证。特别是在 2025 年的 MIT Global Learning Hackathon 中，他带领团队斩获冠军，其核心项目正是评估系统生成准确性的模型架构。 - **工作经验**：12 年以上软件工程背景，3 年 AI 研发经验。 - **学术与研究**：发表约 12 篇研究论文，重点研究 AI 查询优化及 Web API 解决方案。 - **荣誉成果**：MIT Global Learning Hackathon 2025 冠军得主。 --- ### [01:50](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=110s) 聊天模型的自信陷阱与确定性 RAG 机制 商业对话模型通常显得“过于自信”——无论用户输入什么问题，它们总能迅速给出流畅的答复，但在法律、金融和招聘等严肃场景下，“自信并不等于正确（confident is not that's not the same as correct）”。一旦涉及关键事实，模型的不稳定性与幻觉就会带来巨大的合规风险。为此，Vijay 设计的系统强调确定性输出：针对同一份文档提出相同的问题，系统无论执行 100 次都会给出完全一致的回答。 该系统采用了检索增强生成（RAG）架构，能够在用户提问时实时提取文档数据并施加严格的响应约束。在底层机制上，Doc QA Agent 引入了独立的第二阶段评估模型调用，该调用完全剥离了最初的检索步骤，专门用于对主模型的输出进行独立打分与校验；同时后台协同运行着准确率高达 96% 的 CAT 语言模型，确保整个问答链路具备极高的置信度与事实锚定能力。 ```mermaid flowchart TD accTitle: Doc QA Agent 确定性问答与双重校验流程 subgraph Ingestion["1. 文档摄取与索引"] A["企业文档 (PDF / DOCX)"] --> B["分块与敏感信息脱敏 (Redaction)"] B --> C["向量数据库 (Chroma / OpenSearch)"] B --> D["文档元数据 (SQLite)"] end subgraph QueryFlow["2. 查询与检索增强生成 (RAG)"] E["用户提问"] --> F["OpenAI Agent 4o-mini (NLP 语义解析)"] F --> G["向量检索与上下文匹配"] G --> H["CAT LLM 生成确定性回答"] end subgraph Evaluation["3. 独立评估与引用校验"] H --> I["独立评审打分 (Independent Judge)"] I --> J["输出带段落引用的确切答案"] end ``` --- ### [04:47](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=287s) 系统整体架构与组件拆解 系统的技术栈遵循高性能与轻量化原则，前端采用直观的 Web 交互界面，实时呈现模型的处理进度、置信度以及具体行为。后端基于 Python 3.13 与 FastAPI 框架构建，兼顾高并发与快速响应。 在数据与 AI 层，系统选用了 Chroma 向量数据库来支撑 RAG 的向量检索需求，并搭配 SQLite 数据库记录文档的上传路径与元数据映射，便于与 AWS S3 等云端对象存储无缝对接。在自然语言理解环节，系统集成了 OpenAI 的 gpt-4o-mini 模型，但其职责仅限于将用户的原始提问转化为规范的自然语言查询指令，任何内部敏感文档与实际业务数据均不会上传或泄露给外部商业大模型。 - **后端架构**：FastAPI + Python 3.13。 - **向量检索与存储**：Chroma 向量索引库，配合 SQLite 进行文档元数据追踪。 - **NLP 语义处理**：OpenAI 4o-mini 处理用户输入意图，数据与大模型提供商物理隔离。 --- ### [06:14](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=374s) 生产环境加固与企业级安全合规设计 面向企业级生产环境，系统在 AWS 云架构上进行了全方位的安全加固。为了控制高频调用产生的高昂 Token 费用，整套流水线围绕“成本分析与优化”展开设计，同时实现了三层工程级别的检索过滤机制，将评估与检索质量层完全解耦，筑牢安全防线。 在安全性与多租户隔离方面，系统整合了 OAuth 与基于角色的访问控制（RBAC），默认签发 60 分钟有效期的短期会话令牌（可按需申请调整）。所有落盘至 S3 存储桶与 OpenSearch 向量数据库的数据均进行强加密，并在进入云端前对个人身份信息（PII）等敏感数据进行自动脱敏。系统采用严格的租户级数据隔离机制，即便不同租户上传了重名文档，数据也绝不会发生交叉混淆；同时集成 AWS X-Ray 与审计日志（Audit Trail），确保每一笔文档查询均可全程追踪和回溯。 - **身份验证与会话**：OAuth + RBAC 权限控制，默认 60 分钟短期 Session Token。 - **多租户隔离**：Tenant-level 物理/逻辑隔离，杜绝跨企业数据泄露与交叉干扰。 - **追踪与审计**：全面接入 AWS X-Ray 进行链路追踪，并保留完整的 AWS Audit Trail 审计日志。 --- ### [08:53](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=533s) 端到端查询流程、四大设计支柱与行业落地场景 系统运作流程对非技术人员极为友好：用户上传文档后，系统自动完成页面切分、文本嵌入并存入向量索引；用户提问时，系统结合检索上下文生成受到严格约束的回答，并由独立的评审机制对结果进行自我打分。整套架构立足于四大支柱：**事实锚定（Grounded，基于评分而非主观臆测）**、**可检验性（Inspectable，支持人工逐行复核）**、**边界清晰（Knows what it knows and doesn't know）**以及**开箱即用（Out of the box）**。在演练演示中，若用户提出超出文档范围的问题，系统会果断返回“证据不足（insufficient evidence）”，杜绝无中生有的幻觉。 在落地应用场景中，该系统展现了巨大的业务价值： 1. **法律合规**：Vijay 分享了亲身经历的一场交通事故诉讼，面对警局报告、多方律师函件等海量卷宗，系统帮助他快速比对双方律师论点与责任判定； 2. **财务与内控**：企业可在脱敏 PII 后放心上传财务报告并精准查询指标； 3. **招聘与 HR**：HR 无需逐字阅读海量简历，只需输入任职资格要求，系统便能自动匹配候选人简历并列出吻合项； 4. **科研与技术评审**：研究人员能够迅速定位论文中的核心创新点、实验数据、具体页码及栏目。 --- ### [13:09](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=789s) 现场问答：独立评分机制与 LLM 的具体职责 现场观众对“独立评审（Independent Judge）打分”的具体技术实现表达了浓厚兴趣，询问是否由另一个独立的 AI Agent 进行裁决。Vijay 澄清道，系统的底层核心是由 CAT 大模型来驱动 RAG 增强与评估打分。当文档摄入并完成向量化后，CAT 模型会在生成答案的同时给出一个置信度分数。 针对“外部 AI 是否深度参与核心数据处理”的疑问，Vijay 明确表示，外部 OpenAI 模型仅仅充当用户输入的自然语言解析器（NLP 解析工具），用来规范化表达不清晰的用户提问；真正的上下文比对、向量提取、文档比对与评分全部在本地受控的管道中完成，用户可直接将提取出的上下文与原始文档进行“同屏对比（apple-to-apple comparison）”。 --- ### [14:37](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=877s) 现场问答：非结构化数据预处理、多模态支持与企业存储集成 现场参会者探讨了文档预处理、多模态解析以及与企业现有网盘集成的实践问题： - **非结构化数据直接摄入**：面对非结构化文档是否需要繁琐清洗的问题，Vijay 解释道，RAG 机制“就像阅读报纸一样”，即便相关联的信息分别位于第 1 页和第 20 页，向量索引也能直接定位并重组碎片信息，因此无需对文档进行人工规整或额外数据处理（Data massaging），可直接上传解析。 - **多模态与内嵌图表支持**：有同行询问系统是否支持解析夹在文本中间的架构图或流程图等内嵌图片，Vijay 表示当前版本聚焦于纯文本 PDF 与 DOCX 文档，多模态图像理解计划在下一年度的版本更新中推出。 - **企业现有存储集成（OneDrive / Google Drive / Egnyte）**：针对企业已有的权限受控网盘，Vijay 说明若将系统部署在企业自身应用内并打通存储权限，系统可直接拉取受权文档并在本地建立向量索引，无需破坏企业既有的数据治理规则。 --- ### [17:23](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=1043s) 现场问答：CAT 模型的 96% 准确率与双重比对机制 在问答环节尾声，针对观众关于“96% 准确率以外的 4% 潜在风险如何把控”的提问，Vijay 详细拆解了 CAT 模型的表现特性与人机协作兜底方案。模型保留的 4% 边际主要用于测试异常边界与检测幻觉。 为了彻底消除不确定性，系统在前端界面设计了“证据双向溯源”面板：系统不仅呈现最终得分，还会在右侧明确标出答案所抽取的原始页码（如“第 2 页第 1 段与第 3 段”）。Vijay 以现场演示中上传自己的简历为例，系统不仅给出了“Vijay 具备 12 年经验、当前为 Staff Engineer”的结论，还直接高亮了原始文档的出处。这种“模型智能提取 + 原始文档证据高亮”的双向比对设计，使得企业审查人员能够对最终结果建立百分之百的信任。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=4s) 第一次提问给出结果 A，第二次变成结果 B，第三次则是 A 和 B 的混合体 — Vijay Panwar > [01:50](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=110s) confident is not that's not the same as correct — Vijay Panwar > [01:50](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=110s) 针对同一份文档提出相同的问题，系统无论执行 100 次都会给出完全一致的回答 — Vijay Panwar > [08:53](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=533s) 事实锚定（Grounded，基于评分而非主观臆测） — Vijay Panwar > [08:53](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=533s) 可检验性（Inspectable，支持人工逐行复核） — Vijay Panwar > [08:53](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=533s) 边界清晰（Knows what it knows and doesn't know） — Vijay Panwar > [08:53](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=533s) 开箱即用（Out of the box） — Vijay Panwar > [08:53](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=533s) 证据不足（insufficient evidence） — Vijay Panwar > [13:09](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=789s) 同屏对比（apple-to-apple comparison） — Vijay Panwar > [14:37](https://www.youtube.com/watch?v=gmOXbNJiqpY&t=877s) 就像阅读报纸一样 — Vijay Panwar ## Mentioned Vijay Panwar, Panasonic Avionics Corporation, Doc QA Agent, CAT LLM, OpenAI, gpt-4o-mini, FastAPI, Python 3.13, Chroma, SQLite, AWS, AWS S3, AWS X-Ray, AWS Audit Trail, OpenSearch, MIT Global Learning Hackathon, OneDrive, Google Drive, Egnyte ## Video description Vijay Panwar - Member of Technical Staff IV (Staff Engineer), Panasonic Avionics Corporation The talk will demonstrate how an OpenAI-powered AI agent can read PDFs and enterprise documents and answer user questions with verifiable, source-grounded accuracy. As organizations struggle to extract insights from large volumes of legal, financial, and recruitment documents, this session will address the practical challenges of hallucinations, scalability, and trust in document-centric AI systems. The talk will include production-grade design patterns such as document ingestion pipelines, chunking and embeddings, retrieval-augmented generation (RAG), and response constraints using an application-level OpenAI API key. Attendees will learn how to implement evaluation controls, grounding strategies, and failure handling to ensure reliable, auditable answers in enterprise environments. This session is designed for practitioners who want to move beyond demos and deploy responsible, scalable document Q&A agents that deliver measurable business value. [Source](https://www.youtube.com/watch?v=gmOXbNJiqpY)

Source quotes

[0:04] Vijay Panwar: 第一次提问给出结果 A，第二次变成结果 B，第三次则是 A 和 B 的混合体 [1:50] Vijay Panwar: confident is not that's not the same as correct [1:50] Vijay Panwar: 针对同一份文档提出相同的问题，系统无论执行 100 次都会给出完全一致的回答 [8:53] Vijay Panwar: 事实锚定（Grounded，基于评分而非主观臆测） [8:53] Vijay Panwar: 可检验性（Inspectable，支持人工逐行复核） [8:53] Vijay Panwar: 边界清晰（Knows what it knows and doesn't know） [8:53] Vijay Panwar: 开箱即用（Out of the box） [8:53] Vijay Panwar: 证据不足（insufficient evidence） [13:09] Vijay Panwar: 同屏对比（apple-to-apple comparison） [14:37] Vijay Panwar: 就像阅读报纸一样

Source quotes

[0:04] Vijay Panwar: First time is give you A, second time is give you B, third time give you A and B combined. [1:50] Vijay Panwar: The chat models are confident, but that's not the same as correct. [7:36] Vijay Panwar: scored, not asserted [7:36] Vijay Panwar: Knows what it knows and what it doesn't know [7:36] Vijay Panwar: insufficient evidence [10:20] Vijay Panwar: I'm checking what is my lawyer is saying, what is the other team lawyer is saying, what is going on, who is the fault party. [13:09] Vijay Panwar: apple-to-apple comparison [13:09] Vijay Panwar: RAG is work like a newspaper. It's like it will capture all the missing pieces... available in the data. [13:09] Vijay Panwar: making some response by itself to making you happy.

Original transcript

[0:04] All right, good afternoon to everyone and thank  you so much for coming and joining me. So today [0:10] I will be going over the system that I built  which is utilize and solve lots of problem [0:18] for each and everyone. So I'm calling it doc  QA agent which is a self evaluating open AI [0:26] agents for profile and documents. Profile of  documents means your resume, your legal forms, [0:33] medical documents or any 100 pages of documents  where you don't know where the information lies [0:39] and whenever you send those documents to any AI,  they always give you different answer. First time [0:45] is give you it's a second time is give you b  third time give you a and b combined. So that's [0:51] what I'm trying to solve here. So this is me a  good-looking picture of me actually. So I have [0:59] 12 plus years of software engineering experience.  Uh for last three years I'm working in AI and I'm [1:06] member of senior writer believe and the full  member of Sigma 11 where I will be contributing [1:11] in the AI models and how the research going on and  reviewing the research papers as well. I launched [1:18] almost 12 research paper mostly focus on AIdriven  query optimization. how leveraging the web API to [1:26] resolve the sol problems and certified AWS cloud  and certified AWS AI practitioner as well. Uh but [1:36] apart from that one of my biggest achievement  was last year I'm the winner of the MIT [1:41] uh global learning hackathon 2025 where we build  the model to evaluate the accuracy of our system. [1:50] Okay, let's jump into the actual things that  we are here for. So chat models, they are very [1:56] confident. They are like you will ask anything,  you will send them anything, you will get the [2:01] answer immediately. So but what is the biggest  problem we are facing there? Specifically in [2:09] legal finance recruitment document, we get the  information, we ask the questions, we always get [2:16] different answer always. So the image you're  looking at on the right side is actually the [2:23] system that is up and running on my system. Uh  patent is in pending for filing. But in this [2:29] system you can ask the question the chat model are  confident that not that's not the same as correct. [2:35] So the model I build you ask the question 100  times you will always get the same response each [2:43] and every time as long as the document is same. If  you change the document you will get the different [2:47] response for sure. But always you will get the  same response from that. It's built on to the fast [2:54] API that is the python uh the deployment model and  everything is cloud AWS cloud and there is future [3:01] development is coming where I am retracted the all  the confidential information before that document [3:08] is going to be stored into the S3 buckets. So what  is rag? This is the most important thing about it. [3:16] It's a retrieval argumented generation. So it's  like capturing the data real time extracting the [3:25] data what do we need to do how do we need to do  or what kind of information do we need and then [3:30] it answer it. So it's not like we docu we upload  the document and then we are like waiting for the [3:38] response what we going to come out of it. So uh  so with this rag model on this system it will read [3:47] the document what you actually uploaded what is  actually there and then it generate the response [3:54] for you. Uh the model grading its own answer  trends to agree with itself. So like you like [4:03] how how this model is processed. The docu agent  adds a second independent model call that never [4:11] see the retrieval step. It's grade the first model  answer. So it's like you generated like upload [4:17] the document it generated the response for you and  then on that basis when you upload the second time [4:23] it will recompare the same model and see what is  changed there. If not then it will be retraed with [4:29] the same thing there. So it's like training itself  and behind that rag model there is LLM is also [4:36] running which is called CAT models. If you search  about it that's the LLM model which give you at [4:40] least 96% accuracy in the results. So which is  also running in parallel with all the processing [4:47] the agent is doing how things are working and  everything. This is the whole system architecture. [4:55] Okay. So the first one is the browser um is  like the homepage which will giving you all the [5:03] information what is coming what is the model is  doing how the model is doing and what is supposed [5:08] to behave second is the back end so back end is  built in fast API python which is called uh python [5:17] version 3.13 in the latest one and the data and  AI so the chroma which is the vector index which [5:24] is required for the rag the seculite that is the  cuculite database for storing the documentation. [5:30] So we can able to map where the document is  uploaded if you need to retrieve the document [5:35] from the S3 or any of the cloud architecture we  are using for it. And last that is the open AI [5:41] agent 40 uh mini which is embedded in the system  which will help to only read your questions. So it [5:51] will be able to process the NLP. Nothing is shared  with the OpenAI. Nothing is shared with the cloud [5:57] or any of the AI model is in the market. It's just  like converting what you asking in proper English [6:04] language. So the model is able to understand what  you are asking and how it can respond back to you. [6:14] Uh this slide is all about production hardening  and the AWS path. So retrieval quality pages [6:22] uh like page aware checking and overlap  like how the production system will behave [6:28] for us. So three engineer level retrieval  is implemented there which is independent [6:35] from evaluations plus quality. So it's like a  safety layer for us. The whole model is built [6:42] on the cost analysis basis. So your cost  will not go very high when it is running [6:47] and like consuming all the tokens for you. Uh  open search for the vector database because [6:54] I'm using rag here and the API gateway  and the compute of the AWS AI. So it will [7:02] uh make all those metrics how things will need to  be done why like all these things are analyze and [7:10] not able to rewrite or AWS rewrite everything for  us. So if you see into the black section the road [7:16] map which is giving you like the actual whole  picture like how S3 works how that injection is [7:23] happening how's the open search is working for  us and how the ECS target plus API gateway will [7:29] talk to each other and provide you the whole  information that's what you're looking for. [7:36] The system is built for confidential documents  as well. So the system is secure login with o [7:44] role based access control. So not like everyone  is going to do and they are stuck there. Sort [7:50] live session tokens. So one token is valued  for 60 minutes but as per the system goes to [7:55] individual person that can be increased by  the request. Document security everything is [8:01] encrypted when it is going to the AWS here. Uh  per tenant data isolation. So if there is a 10 [8:08] people or 10 different companies are utilizing  my system, they will having their own source. [8:14] Nothing will be shared in between us. So it's  like if I'm using it and someone else is using it, [8:19] their data is 100% secure. No leakage of the  data. No uh crossing cross matching the data. [8:26] If I'm uploading something and same file name  uploaded by someone else, it will get confused [8:31] in between. Access and audit every query logged  and traceable. I'm utilizing AWS X-ray here for [8:38] that a pre-documented access role. So we will  have the access level and the last is full [8:45] audit trail for the review. So that is AWS audit  trail is will be playing the biggest picture here [8:53] how they all talk to each other very simple very  straightforward for nontechnical peoples upload [9:00] the document embedded pages you can upload any as  number of pages I run this model against thousand [9:06] numbers of time with where the pages are 200  where the full book I uploaded with 3,000 pages [9:12] I also uploaded single file of rum as well and  I get The very good response store in vector [9:19] index. Everything is stored in the vector index  here. How the query flow work? So first part you [9:25] uploaded system process it and we are good to go.  How the user use it. You will ask the question it [9:34] will retra plus generated and constraint with the  response what you are asking and you will see the [9:39] same response all the time. And the last one is  independent judge score it. So you like you are [9:45] selfc scoring your own response. The four pillar  of this whole system is the trust mechanism that [9:52] falls out of the design. The grounded which is  you can see stored sorry scored not asserted. [9:58] So you are scoring not like assuming what is  happening there. Inspectable you can inspect [10:03] what you asking what is in the document and you  can validate it by yourself and knows what it [10:09] knows it what doesn't know. That means when  you upload the document he didn't know what [10:13] is coming but after that he learned it and he  stored the he stored and he he know what he [10:20] know how this things will be work this is the  live demo um so if you see on the left side is [10:27] the demo script I uploaded the document I asked  the grounded question very soft citation stamps [10:35] uh it's if you can click on the citations on  the right side of the image it can able to show [10:40] you like from where which paragraph which section  which page it is capturing that information and [10:46] you can ask something if you try to ask something  out of scope agent will reply insufficient [10:52] evidence. So it's not like what is document and  you are asking something else it will not going [10:58] to give you response because that thing is not  existing in this model and that information is [11:03] not available in the document itself and anyone  who has trust a document not just git. So whoever [11:11] need it like legal teams legal teams have NDA  signed by each and everyone. Uh I just have a [11:20] uh like last year I had some accident and there  is lot of legal teams involved and uh the police [11:27] report and everything happened. So I'm like  going through every day with the reviews of [11:32] document what is happening and I have no idea what  I'm doing. So I came up with the idea and that's [11:37] why I built this system. So I'm uploading all my  documents here. I'm checking what is my lawyer is [11:42] saying, what is the other team lawyer is saying,  what is going on, who is the fault party kind of [11:47] thing. So it will skim everything for me. Second  one is finance and compliance. You can upload your [11:53] finance document here. No critical information  will be shared with anyone. All the information [11:59] will be retracted like PI compliance before going  to cloud and you can able to ask your question [12:04] directly. Very helpful for recruiting and HR. So  they don't need to skim through all the résumés. [12:11] You can provide the basic information. These are  the profiles I'm looking all the rums you upload. [12:16] It will process and give you these résumés are  matching with the profile and you can proceed with [12:21] further stuffs and everything. And research and  developments there is a huge amount of research [12:27] papers we do review. I do review all the times and  I can see like what are the key values I required [12:33] what are the things they did something new. So I  can retract from the documents. It can give me the [12:39] specific column which page number, which column  I'm looking for, what are the actual wording and [12:44] how I can capture all those information and then  it will be processed into the system and then I [12:49] can provide my feedback to the system like how  things will go from there. Uh last but not least, [12:57] uh thank you so much for listening me. Uh this  is my LinkedIn profile and I will be here for [13:02] next five more minutes. If you have any questions  or anything, please let me know. So about your [13:09] independent charge score. So do you how did you  implement that? So do you have another AI agent [13:15] to charge independently or no? Uh there is a  LLM model running in the behind. So which is [13:23] called cat model which is running the whole  argumentation rag. So once you upload the [13:28] document the system will uh the rag will extract  the data and goes into the vector database and [13:34] then LLM model runs and that will provide you  the score when the response cames how much [13:39] is the self code comes to you so is still the  same AI doing the job or another AI no AI no AI [13:51] so it's a yeah so AI will play only the role  when you asking the question. So you asking [13:58] the question in Gibbre or miss something. So  it will convert it into the NLP and then it [14:04] will goes to the vector in the rack will process  your request capture the data llm run it against [14:10] the document and give you the matching score like  how much is the confident is there and then you [14:15] can also cross check the same information with  the document what is agent is giving you and [14:20] what you are actually seeing in the document  how much is matching. So you can do appletole [14:24] comparison. Oh okay. So you have to pretty  much there compare each other. All right. Okay. [14:37] So uh these documents are highly unstructured,  right? Yes. And um do you pre-process them in any [14:45] way? Like do you do some kind of pre-processing  to make it more structured so that ll rag can [14:51] actually handle it better? rag can handle it very  better. Rag is rag is work like a newspaper. It's [14:58] like it will capture all the missing pieces  all the pieces available in the in the data [15:04] and then it will go from there. So even if you  have unstructured data something on page one [15:10] something on page 20 it will it know what is need  to be done and how to match it. So that's where [15:15] the vector data plays a very critical role for us.  So you directly just ingest it into just directly [15:20] ingest it. No need to do any kind of massaging  or any kind of things to the data any document. [15:27] So it's because of the cat model is if we replace  it with any other LLM will it give you the right [15:34] uh as good a accuracy as like will if you your  accuracy will drop for sure. CAT model give you [15:40] the 96% accuracy. Other model might give you  the 94 93% accuracy. It's totally up to you [15:47] how much accurate information are you looking  for. Oh. Um, thanks. Yeah, I actually built and [15:52] maintain a very similar system for a client. Um,  so it's cool to see this. My question for you is, [15:57] um, you mentioned PDF and DOCX. Do you handle  multimodal like for example inline images? So [16:02] if I had like an architecture diagram in  the middle of a page with text on either [16:06] side. This is one of the big problems that I've  struggled with in my system. So I'm just curious [16:10] um, do you handle images as well? If so, how do  you address that? Not at this moment. Yeah. the [16:16] current model only built onto the documents PDF  and docs that's the future development uh maybe [16:21] next year yeah I have two questions uh I saw you  have SSO you know added um so we have a document [16:31] storage right so a lot of companies use document  store one drive Google drive you know Ignite and [16:36] so all my documents are there where all the  permissions are enabled only certain people [16:40] can see certain documents if I have to use this  docqa so do have to move those documents to your [16:48] repository or you can read directly from our  storage where we already have where we already [16:54] have governance controlled. If you implied the  same system to your application and give the DB [17:02] access or wherever the document is uploaded,  you don't need to provide those information. [17:08] But then you will you pull the documents and  create a vector based on the documents. Yes, [17:12] based on you have to do that, right? Yes. roommate  documents. Yes. All right. Thank you. Sure. [17:23] You got time for one more  last question. [laughter] [17:28] Uh in one of the slides I saw that you mentioned  there was a judge that evaluates uh the reviewer's [17:33] feedback whether it's positive or not. uh would  like to know exactly what the uh judge uh is it an [17:40] LLM that's validating the output or and how is it  validating because you said there's a 96% accuracy [17:45] but what happens to the 4% that you lose and is  it critical and who validates and confirms that [17:50] there was something missing in the rack response  okay so the cat model is a llm model so you [17:56] provide they provide all the uh APIs and how the  system how the lang large language model works so [18:05] uh CAT is the highest accuracy model 96%. 4% uh  they left it out for testing and hallogenating how [18:13] things are working. So that's why I'm using the  CAT model and the scoring part. So when you upload [18:19] the document this goes into the vector database  and then after when you ask the questions it [18:24] capture the response and based on that response  cat model provide it accuracy and when you read [18:30] that document and when you compare when the the  what's the response comes and what's the document [18:36] have on the right side so here you will see the  score I'm not populating it right now but here you [18:42] will see the score and if you see the page two and  page two it is extracting like one and three. So [18:48] that's where you can compare what's the LLM model  is giving you the scoring and how the response is [18:54] looking and what is the response comes is there.  So you can see I I uploaded my own resume to [19:01] evaluate it to capture the screenshot for here. So  yes, VJ is for staff role as the currently staff [19:07] engineer at 12 years of experience. So you can  compare if is the same information is there or [19:12] AI or agent is throwing you some hallucinating  or by making some response by itself to making [19:18] you happy what you are looking for. Yes. So  now you have like two ways of comparing like [19:26] what you see and what model is giving you. So  you will have more confidence in the model.