ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents

ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents Multiverse Computing has introduced ProvenanceGuard, a post-generation verification layer for Model Context Protocol (MCP) agents that detects cross-source conflation by checking not only whether a claim is factual, but whether it is supported by the specific source cited. Existing evaluators pool evidence and can overlook cross-source conflation, where a claim is true in the context pool but attributed to the wrong tool or document. ProvenanceGuard maintains source identities through a five-step pipeline: decomposition, routing, support verification, attribution matching, and final decision emission. Tested on 361 medical agent claims, it flagged 138 of 139 unsupported claims and achieved a reject/block F1 of 0.802, outperforming MiniCheck, RAGAS Faithfulness, AlignScore, and SummaC-ZS. In a controlled attribution-swap test of 50 cases, ProvenanceGuard detected all 50 misattributions, though distinguishing between highly similar sources achieved 50.3% accuracy. The architecture includes an optional RARR-style repair loop and has been adapted into NVIDIA NVFlow for financial agent grounding verification against SEC excerpts. The Problem of Cross-Source Conflation As tool-using LLM agents leverage the Model Context Protocol to query search engines, structured records, databases, and metadata, standard verification tools such as RAGAS Faithfulness, MiniCheck, AlignScore, and SummaC pool all evidence together. This causes them to miss cross-source conflation, where a statement is accurate within the combined evidence pool but attributed to the wrong source. In sensitive domains such as healthcare or finance, misattributing a patient-specific record to medical literature or confusing account records with company policies can be as problematic as factual inaccuracy. ProvenanceGuard Architecture and Pipeline ProvenanceGuard operates as a post-generation layer over black-box MCP agents without requiring retraining. It reads traces, tool outputs, and source identifiers to maintain provenance. The system executes five sequential steps: breaking answers into discrete claims, routing each claim to its most relevant source, assessing whether the source supports the claim using NLI, comparing that source against the cited or implied source, and generating both per-claim verdicts and an overall answer-level decision. The evaluated local configuration utilizes MiniLM for routing, a DeBERTa NLI verifier for support checking, and strict checks on literal values such as dates, numbers, and identifiers. Empirical Performance and Benchmark Results Evaluated on 281 medical agent traces containing 361 human-annotated claims across 40 held-out answers, ProvenanceGuard detected 138 of 139 unsupported claims, letting only one pass while sending 67 supported claims for review under a conservative threshold. It achieved a reject/block F1 score of 0.802, exceeding MiniCheck (0.783), RAGAS Faithfulness (0.758), AlignScore (0.662), and SummaC-ZS (0.436). Unlike the baselines, ProvenanceGuard emitted explicit claim-to-source IDs, correctly identifying the supporting source roughly 86% of the time in the main test and capturing 100% of errors in a 50-case controlled source-swap test. However, exact source identification dropped to 50.3% when distinguishing between highly similar sources. Repair Mechanisms and Real-World Adoption When answers are blocked, ProvenanceGuard can feed them into a RARR-style repair loop. In full-trace testing, the loop resolved all 173 blocked outputs, utilizing fallback responses in 144 instances to avoid fabricating unverified answers. On reconstructed multi-source traces, repair resolved all 59 initially blocked answers with only two terminal fallbacks, operating with local overhead of roughly half a second per answer. Beyond research benchmarks, the source-aware verification approach has been integrated as an optional grounding verification stage in NVIDIA NVFlow to validate finance agent responses against SEC filings.

Multiverse Computing 推出 ProvenanceGuard：面向 MCP 智能体的源感知事实性验证框架 传统事实性检查工具通常将证据混为一谈，难以察觉多工具环境下的归属错误。Multiverse Computing 针对 Model Context Protocol (MCP) 智能体提出验证系统 ProvenanceGuard，可在不破坏证据来源的前提下逐条校验主张与工具源的匹配度，防范跨源混淆问题。 针对 MCP 智能体将真实信息错误归属至其他工具的“跨源混淆”缺陷，传统汇总型检验工具（如 MiniCheck、AlignScore、RAGAS）无法区分具体工具来源。 ProvenanceGuard 采用生成后验证流程，保留 MCP 轨迹中的源 ID，按顺序执行主张拆解、源路由、支持度评分、归属核对与放行/拦截决策。 在医疗智能体 361 条主张的测试中，系统成功拦截专家判定的 139 个不合规主张中的 138 个，拦截 F1 分数达 0.802，并在 50 次人工篡改来源的受控实验中实现 100% 检出率。 系统提供基于 RARR 风格的修复闭环，被拦截的回答可通过有依据的重写或安全兜底文本解决；该方法已被整合至 NVIDIA NVFlow 的金融智能体接地验证阶段。 跨源混淆与源感知验证 基于 MCP 的智能体可调用数据库、病历、搜索等多种工具合成回答，但也引发了跨源混淆风险——陈述在全部证据池中确实存在，却被归咎于错误的工具输出。传统事实检查器将证据汇集为单一匿名上下文，容易放行此类错误归属。ProvenanceGuard 作为后置验证层，不重新训练黑盒智能体，通过保留 MCP 追踪记录与源 ID，逐个检查主张是否真正被其指称或暗示的特定来源支持。 系统架构与实验表现 ProvenanceGuard 在实验中采用 MiniLM 路由相关源、DeBERTa NLI 模型检验支持度，并严格校验数字、日期等字面值。在针对医疗智能体的 281 条真实轨迹和 361 条专家标注主张的测试中，ProvenanceGuard 的拦截 F1 值为 0.802，高于 MiniCheck (0.783) 和 RAGAS Faithfulness (0.758)，且唯一能输出主张到来源 ID 的绑定关系。在 50 例来源篡改实验中系统全数检出错误，但在大量相似来源的困难测试中，精确匹配来源的正确率为 50.3%。 修复机制与实际落地 对于被拦截的回答，系统衔接 RARR 式修复步骤，选择基于事实重写或输出安全兜底文本，避免制造未验证内容；在本地配置下，回答级别的时间开销约为半秒。该源感知验证方法已被应用至 NVIDIA NVFlow，为其金融智能体提供针对 SEC 摘录的接地验证阶段，此外该成果已在 UC Berkeley 的 Agentic AI Summit 2026 展示。

Tool-using LLM agents no longer read from a single retrieved passage. Through the Model Context Protocol (MCP) , an agent can call a search tool, inspect a structured patient or account record, query a database, and pull metadata, then weave all of it into one answer. That m aakes the usual question of factuality more subtle than it looks. Most of the systems built to check LLM answers, from RAGAS faithfulness to fine-grained checkers like MiniCheck, AlignScore, and SummaC, ask whether a claim is supported by the available evidence once that evidence has been pooled together. In their usual form, they do not tell us which MCP tool output supports each claim, or whether that is the source the answer names. Our latest paper, ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents (read it on Hugging Face , or on arXiv in the meantime), targets that gap. The failure mode we care about is one we call cross-source conflation: a claim that is true somewhere in the evidence, but attributed to the wrong source. A source-blind verifier may pass it, because the fact does exist in the pool. A source-aware verifier should not. The problem: supported somewhere is not the same as supported by the right source Consider a customer support agent that answers, "According to the account record, this plan includes a 30-day refund window." The refund window may be perfectly real, but stated in a policy document, not in the account record the answer points to. Pool the two together and the claim looks supported. Keep them separate and the attribution is wrong, and in a data-sensitive setting a wrong attribution can be as damaging as a wrong fact. The same pattern shows up in a clinical agent, where a patient-specific medication detail taken from a patient-history tool becomes misleading the moment the answer presents it as a finding from the medical literature. A claim can be supported by one MCP source while the answer attributes it to another. Source-blind scoring sees support in the pooled evidence and passes it; ProvenanceGuard separately checks whether the supporting source matches the one the answer states or implies. Source: paper Figure 1. This is why faithfulness scores, useful as they are, are not enough for MCP agents. An answer carries provenance, sometimes explicitly ("according to the account record") and sometimes implicitly. ProvenanceGuard keeps that connection between claim and source available for inspection. What ProvenanceGuard does ProvenanceGuard is a post-generation verification layer that sits on top of a black-box MCP agent. It runs after an agent produces an answer, and never collapses the evidence into one anonymous context. Instead it carries the source identity all the way through the pipeline. It reads the captured MCP trace, including the tool outputs and their source IDs, without retraining the agent. Then it does five things in sequence: it breaks the answer into specific claims, finds the source most relevant to each one, checks whether that source actually supports it, compares the source with the one the answer names or implies, and finally emits both a per-claim source verdict and a global, answer-level allow or block decision. The verification flow. Source identity is preserved through decomposition, routing, support scoring, attribution checking, and repair, rather than being pooled. Blocked answers can go through RARR-style repair and be re-verified. Source: paper Figure 2. A few of the design choices are worth calling out. For the experiments in our paper, we used local models so the captured traces could be processed in a controlled, offline setup: MiniLM helps find the relevant source, a DeBERTa NLI verifier model checks whether that source supports the claim, and a local language model helps break answers into claims. The verifier also checks literal values closely: a number, date, or identifier absent from the source cannot pass merely because the sentence sounds plausible. A calibrated decision step combines these signals. If an answer is blocked, a RARR -style repair step can try a source-grounded revision or a safe fallback, which the verifier then checks again. Those named models are the setup we evaluated, not a requirement of ProvenanceGuard. The same claim, source, and decision steps can be adapted to hosted models where a team prefers cloud services; a new setup would need its own testing and calibration. Our reported results come from the local configuration. Its conservative decision policy suits data-sensitive review, where getting the source right matters more than producing the fastest possible answer. Results We tested ProvenanceGuard on answers from a medical agent that had used patient records, research articles, and other tools. This gave us 281 real traces to study. Medicine is a useful test because a fact from a patient's record and a fact from general research cannot be treated as the same source. The method can also be used in other fields when an agent keeps a record of its tool outputs and source IDs. For the main test, human experts checked 361 claims from 40 answers set aside from the data used to develop the system. The most direct result is this: experts said 139 claims should not pass, and ProvenanceGuard caught 138 of them. It let one through. It also held 67 claims that the experts considered supported, sending them for review or repair. This reflects the cautious setting we tested: it favors a second look at some supported claims over letting unsupported ones through. For claims with an identifiable source, it also picked the right source about 86% of the time in this test. We ran four other support checkers on the same claims. ProvenanceGuard scored highest on the paper's measure of how well a system catches claims that should be blocked while avoiding unnecessary blocks. The other checkers in this comparison did not tell us which tool output supported each claim. ProvenanceGuard records that connection, so a reviewer can see the source checked for each claim and the decision it produced. Verifier Reject/block F1 Emits claim-to-source ID ProvenanceGuard (ours) 0.802 Yes MiniCheck 0.783 No RAGAS Faithfulness 0.758 No AlignScore 0.662 No SummaC-ZS 0.436 No Binary support metrics on the same held-out claim packet. ProvenanceGuard matches or beats the source-blind baselines on blocking while also producing per-claim source verdicts. Source: paper abstract and Table III. Checking claims when sources look similar In a separate, harder test with several similar sources, ProvenanceGuard scored 0.846 F1 for deciding which claims to block, but identified the exact source correctly in 50.3% of claims. Telling similar sources apart remains an important area for improvement. We also ran a controlled test focused on wrong attribution: we changed the named source in 50 cases while leaving the supporting evidence intact. ProvenanceGuard caught all 50 swaps. This shows it can detect a clear source error, while the harder test shows the challenge of choosing among many plausible sources. Repairing blocked answers Blocking is only useful if there is something to do with a blocked answer. Wired to the RARR-style repair loop, the full-trace run resolved all 173 blocked answers, though 144 of them ended in fallback text rather than a substantive rewrite, which is the system choosing to avoid an unverifiable answer rather than manufacture one. On reconstructed multi-source test traces, a fresh repair run resolved all 59 initially blocked answers with only two terminal fallbacks. As an offline gate the overhead is modest, roughly half a second per answer on the reported local configuration, with the NLI and routing calls themselves in the tens of milliseconds. Why this fits Multiverse Computing As agents move from single-passage RAG to multi-tool MCP setups, the question of which source a fact actually came from stops being a footnote and becomes part of what factuality means. ProvenanceGuard makes that source connection visible claim by claim. For Multiverse Computing, that means a way to check existing agents while keeping sensitive traces in a controlled environment when needed. The medical study is one use case; the same approach can be adapted wherever an agent's trace preserves its tools and sources. That adaptation is already visible in NVIDIA NVFlow , which merged an optional grounding-verification stage for its finance agent. It checks completed answers against the SEC excerpts the agent retrieved and saves separate decisions without changing the original rollout or training data. The NVFlow contribution uses ProvenanceGuard's source-aware verification approach; the repair loop discussed above belongs to the broader research system. ProvenanceGuard was also presented as a poster at the Agentic AI Summit 2026 at UC Berkeley . Want the full technical details, including the routing and NLI derivations, the calibration ablations, the multi-source stress slices, and the complete results tables? Read the full paper on Hugging Face , or get in touch with our team to talk about applying source-aware verification to your own agents.