Block deploys 50 AI agents to fight Cash App fraud without domain code

Building Reliable AI Agents in Production With Block Inc — Ai4

Ai4

Deploying reliable AI in high-stakes finance requires stripping governance out of LLMs into deterministic application code and declarative rubrics that risk analysts can update in minutes.

AI-assisted summary of Building Reliable AI Agents in Production With Block Inc — original source. Check the original for full context.

Source published: 2026-08-20

Detailed notes

## Background Published on YouTube by the Ai4 channel on August 20, 2026, this technical presentation features Charishma Bhavani Koneru, a Senior Machine Learning Engineer at Block Inc. The talk explores the engineering patterns, governance frameworks, and continuous evaluation pipelines used to build and operate reliable autonomous AI agents for high-stakes financial operations at Cash App.

## TL;DR Koneru details how Cash App transitioned from monolithic batch pipelines to an event-driven Universal Runtime running over 50 specialized subagents across more than 30 fraud and risk typologies. The platform enables risk experts to deploy new investigation domains in minutes using declarative Markdown rubrics and YAML configurations without writing executable code. To ensure safety and regulatory compliance, Block implements a strict four-stage trust ladder from shadow mode to bounded autopilot, backed by a human override flywheel. The runtime relies on deterministic application code rather than LLM prompts for final governance, evaluated through a three-layer pipeline of deterministic graders, sampled LLM judges, and human trend discovery.

## Atmosphere & style The presentation is pragmatic, authoritative, and deeply rooted in production engineering realities rather than speculative AI hype. Koneru delivers a clear, disciplined walkthrough of high-stakes architecture, balancing systemic design patterns with the rigorous statistical gates required in regulated fintech. Listening to the talk feels like an architectural masterclass from the engineering front lines, emphasizing that long-term reliability comes from hard deterministic guardrails and continuous human feedback loops.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=4s) The primary engineering challenge in high-stakes production AI is no longer raw model capability, but architecting the deterministic governance layer and audit trail surrounding autonomous agents. - [03:04](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=184s) Cash App evolved its fraud architecture from monolithic batch LLM pipelines to an event-driven, multi-phase reasoning runtime coordinating 50+ specialized subagents across 30+ fraud typologies. - [03:04](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=184s) Domain experts deploy new investigation skills in minutes using declarative YAML configurations and Markdown rubrics with zero new lines of executable application code. - [05:49](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=349s) Specialist agents execute in parallel isolation without viewing one another's intermediate reasoning, preventing evidence cross-contamination before synthesis. - [05:49](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=349s) Routing between autonomous resolution, copilot review, and human escalation is executed deterministically in application code based on confidence thresholds rather than prompted within an LLM. - [08:31](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=511s) Modular multi-agent architectures outperform single massive-context prompts by providing regulatory provenance, hot-deployability, and isolated golden-dataset testing. - [12:01](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=721s) AI agents must earn autonomy through a four-stage Trust Ladder (Shadow Mode, Copilot Mode, Sampled Spot-Check, Bounded Autopilot) requiring over 95% human agreement over 30 to 60 days. - [12:01](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=721s) Autopilot operations remain intentionally bounded, leaving 40% to 50% of cases in human hands due to high financial exposure and edge-case ambiguity. - [14:22](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=862s) Human analyst overrides are captured with explicit structured reasoning, creating a flywheel where root-cause analysis continuously refines investigation rubrics. - [14:22](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=862s) Production quality is governed by a three-layer evaluation pipeline: 100% deterministic guards, a sampled LLM-as-a-judge layer that prefers abstaining over incorrect scoring, and human trend discovery. - [14:22](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=862s) Because LLM judges are backward-looking and limited to historical patterns, subject matter experts are required in the loop to identify and catalog novel, emerging fraud vectors. - [18:58](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=1138s) Teams deploying autonomous agents should establish deterministic evaluation graders and instrument rich tracing schemas before investing in complex LLM-as-a-judge pipelines.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=4s) Operating High-Stakes Autonomous AI at Cash App

At Cash App, Block Inc.'s consumer finance platform serving more than 50 million customers, machine learning systems handle critical, high-stakes decisions every day. These systems are far from conventional conversational chatbots; they are autonomous agents tasked with determining whether an account is fraudulent, whether a transaction dispute should be approved, or if user activity requires freezing funds. Because Cash App encompasses a broad product suite—spanning banking, peer-to-peer payments, investing in Bitcoin, and borrowing via cash advances—every surface area introduces complex fraud vectors and stringent regulatory obligations.

The fundamental engineering challenge in modern production AI is no longer model capability. Modern large language models like Claude or ChatGPT can readily perform complex investigations with reasonable competence out of the box, but "reasonable isn't good enough anymore" when a false decision can freeze an innocent user's bank account or violate financial compliance mandates. In regulated environments like fintech, healthcare, and legal services, teams must establish an unassailable audit trail showing regulators precisely which piece of evidence drove which specific decision. Consequently, the primary engineering effort lies in architecting the governance layer surrounding the autonomous agents.

```mermaid flowchart TD accTitle: High-Stakes Decision Surface at Cash App subgraph Products["Cash App Product Surfaces"] P1["P2P Payments"] P2["Banking & Cash Advances"] P3["Bitcoin & Investing"] end

subgraph Risks["Operational & Risk Vectors"] R1["Account Fraud"] R2["Dispute Claims"] R3["Sanctions & Compliance"] end

subgraph Decisions["Autonomous Agent Decisions"] D1["Freeze / Clear Account"] D2["Approve / Deny Claim"] D3["Regulatory Audit Trail"] end

Products --> Risks Risks --> Decisions ```

[03:04](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=184s) Production Scale and the Evolution from 2024 to 2026

Cash App's production deployment runs dozens of distinct investigation skills, orchestrates over 50 specialized subagents in parallel, and defends across more than 30 fraud and risk typologies ranging from scam reimbursements to identity verification and sanctions screening. The most notable operational benchmark is achieving zero lines of code per new domain. When a new fraud pattern emerges, subject matter experts author a Markdown rubric file that goes live in production within minutes, bypassing software deployments and engineering bottlenecks.

This current architecture represents a complete rebuild from the paradigm used in 2024. The legacy 2024 approach relied on scheduled batch pipelines, single monolithic LLM calls per claim, and rule-based aggregator systems where every new fraud domain demanded dedicated codebases and deployment cycles taking days or weeks. Because financial adversaries adapt within hours, the architecture was re-engineered in 2026 into a real-time, event-driven system featuring multi-phase agentic reasoning, extended thinking, and hard deterministic governance code.

- Scale in production: 50+ specialized parallel subagents handling 30+ fraud and risk typologies. - Zero engineering bottlenecks: New domain investigations deployed purely via Markdown rubric definitions. - Transition from 2024 to 2026: Shifted from scheduled batch pipelines and single LLM prompts to an event-driven, multi-phase reasoning runtime.

[05:49](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=349s) Multi-Agent Architecture and Evidence Synthesis

A typical operational case, such as an unauthorized dispute claim, appears straightforward on the surface but contains heavy ambiguity: device fingerprints may match historical records while geographical location does not, or claimants may themselves be adversarial bad actors filing first-party fraud claims. Furthermore, financial operations are constrained by strict regulatory SLA deadlines. When a case is ingested, the system fans out execution across bounded, single-responsibility specialist agents—such as payment behavior analysis, identity and profile verification, and account network linkages.

These specialist agents execute in parallel without viewing one another's intermediate reasoning, preventing cross-contamination of evidence. Each agent outputs a structured signal containing confidence scores and precise citations back to underlying source data. These signals feed directly into an evidence-grounded synthesis layer that compiles the findings before passing them to a deterministic routing engine. The router enforces strict operational paths based on confidence thresholds and regulatory flags:

```mermaid flowchart LR accTitle: Multi-Agent Parallel Investigation and Deterministic Routing subgraph Ingestion["Case Ingestion"] Case["Dispute / Event"] end

subgraph Specialists["Parallel Specialist Agents"] direction TB A1["Payment Behavior"] A2["Identity & Profile"] A3["Network Linkages"] end

subgraph Synthesis["Synthesis & Governance"] Syn["Evidence Synthesis Layer"] Router{"Deterministic Router"} end

subgraph Execution["Execution Paths"] direction TB E1["Autopilot (High Conf.)"] E2["Copilot (Med Conf. / Human Review)"] E3["Escalations (Low Conf. / Reg Flag)"] end

Case --> Specialists Specialists --> Syn Syn --> Router Router --> E1 Router --> E2 Router --> E3 ```

The core tenet of this setup is that routing decisions are executed deterministically in application code rather than prompted within an LLM: "LLM provides intelligence, code is the one which provides the governance."

[08:31](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=511s) The Universal Runtime and Declarative Investigation Rubrics

The cornerstone of the platform is the Universal Runtime. In this system, domain authors (risk investigators and operations specialists) interact exclusively with declarative YAML configurations and Markdown rubrics. The YAML files declare required data sources and workflow stages, while the Markdown files specify the investigation rubric and evaluation instructions. Skill authors write zero executable code.

```mermaid flowchart TD accTitle: Universal Runtime Layer Separation subgraph DomainLayer["Skill Author Layer (Domain Experts)"] YML["Declarative YAML Config\n(Data Sources & Workflows)"] MD["Markdown Investigation Rubric\n(Reasoning Guidelines)"] end

subgraph PlatformLayer["Universal Runtime Platform (ML / Software Engineers)"] Fetch["Pre-fetching & Data Cataloging"] Guards["Deterministic Guardrails"] Render["Structured Report Rendering & Audit Trails"] end

DomainLayer --> PlatformLayer ```

The underlying platform manages data pre-fetching, cataloging, deterministic guards, evaluation pipelines, and structured report rendering for regulatory compliance. This separation prevents domain experts and infrastructure engineers from touching each other's codebases. Relying on specialized agents rather than a single massive context window prompt (even with models supporting a million tokens) is essential for four concrete reasons:

1. **System Stability:** Monolithic prompts require complete reprompting and retraining whenever a single scam pattern evolves. 2. **Quality Verification:** Any query or logic change requires validation against a golden dataset of at least 30 SME-labeled cases. 3. **Regulatory Provenance:** Audits require pointing directly to the exact agent, exact data evidence, and exact reasoning path that produced an adverse action. 4. **Hot-Deployability:** Operational teams can modify an investigation rubric for an emerging scam on a Tuesday afternoon and deploy it live within minutes without engineering releases or downtime.

- Specialists emit over 100 structured signals across four distinct evidence domains. - Every signal carries an explicit value, a confidence metric, and an immutable source citation.

[12:01](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=721s) The Trust Ladder: From Shadow Mode to Autopilot

An AI agent does not begin with autonomous privileges; it must climb a structured trust ladder designed to prove safety rather than raw capability. Progression through each step requires strict quantitative validation:

```mermaid flowchart LR accTitle: Autonomous Trust Ladder S1["1. Shadow Mode\n(100% cases, 0% impact, runs weeks)"] --> S2["2. Copilot Mode\n(Recommendations, human signs off, runs months)"] S2 --> S3["3. Sampled Spot-Check\n(Agent acts, human samples, runs months)"] S3 --> S4["4. Bounded Autopilot\n(Restricted scopes, continuous monitoring)"] ```

Promotion to subsequent stages mandates meeting hard statistical gates: maintaining a greater than 95% agreement rate with human decisions over a sustained 30-to-60-day window, keeping false approval rates below risk-adjusted thresholds, showing zero drift in escalation distributions, passing regulatory reviews, and clearing statistical A/B testing. Even in mature deployment, full automation is intentionally restricted; between 40% and 50% of incoming cases remain handled by human specialists due to their complexity, novelty, or high financial exposure.

[14:22](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=862s) The Human Override Flywheel and Three-Layer Evaluation Pipeline

When an analyst in copilot mode overrides an agent's recommendation, the platform captures the decision alongside the explicit reason for the override. These overrides act as high-value, labeled training data that initiate a root-cause analysis workflow to diagnose missing data signals, prompt ambiguities, or emerging fraud patterns. Updating the corresponding Markdown rubric directly drives down subsequent override rates.

```mermaid flowchart LR accTitle: Human Override and Continuous Learning Flywheel AgentAct["Agent Recommendation"] --> Review["Human Analyst Review"] Review -->|Override Triggered| RCA["Root Cause Analysis\n(Missing signal / prompt gap / new pattern)"] RCA --> UpdateRubric["Update Markdown Rubric"] UpdateRubric --> ProdAgent["Enhanced Production Agent"] ProdAgent -->|Lower Override Rate| AgentAct ```

To evaluate agents rigorously in production, Cash App employs a three-layer pipeline:

```mermaid flowchart TD accTitle: Three-Layer Production Evaluation Engine Trace["Production Decision Trace"] --> Gate1{"Gate 1: Deterministic Graders\n(100% traces, fast, structural checks)"} Gate1 -->|Pass| Gate2{"Gate 2: LLM-as-a-Judge\n(10% sample, semantic checks)"} Gate1 -->|Hard Fail| Drop["Hard Rejection / Fail Trace"] Gate2 -->|Parse Error / Uncertainty| Abstain["Abstain (No score > Wrong score)"] Gate2 -->|Pass| Gate3{"Gate 3: Human Trend Discovery\n(SME review on novel patterns)"} Gate3 -->|Disagreement| GoldenSet["Append to Golden Set & Fraud Trend Log"] ```

- **Layer 1 (Deterministic Guardrails):** Evaluates 100% of traces cheaply and rapidly to intercept structural and format violations; acts as a hard failure gate with no LLM override allowed. - **Layer 2 (LLM-as-a-Judge):** Samples approximately 10% of traces for deeper semantic scrutiny; if parsing errors or uncertainty occur, the judge abstains because "no score is better than the wrong score." - **Layer 3 (Human Trend Discovery):** Solves the limitation that LLM judges are inherently backward-looking and only evaluate against historical data. Subject matter experts analyze novel edge cases, adjudicate disagreements, and feed emerging behaviors into a version-controlled, auditable Live Fraud Trend Log.

[18:58](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=1138s) Reliability Philosophy and Production Takeaways

Operating autonomous systems at financial scale necessitates an integrated, five-layer reliability stack where each level catches faults that bypass the others:

```mermaid flowchart TD accTitle: Five-Layer Reliability Stack L5["5. Business & Risk Metrics"] L4["4. Continuous Evaluation (3-Layer Pipeline)"] L3["3. Observability & Tracing"] L2["2. Deterministic Guardrails"] L1["1. Modular Architecture (Universal Runtime)"]

L1 --> L2 L2 --> L3 L3 --> L4 L4 --> L5 ```

Live monitoring tracks error rates and escalation frequencies against historical baselines, triggering automated investigation and immediate rollbacks if anomalies arise. Reflecting on lessons learned building these frameworks, successful autonomous deployments rely on key architectural practices:

- Establish deterministic evaluation graders first before investing in complex LLM judges. - Instrument rich tracing schemas and evaluation data collection from month one prior to launching production systems. - Capture structured reasoning whenever human analysts disagree with an agent to fuel the retraining loop. - Treat emerging fraud trends as structured observations rather than unverified programmatic directives. - Maintain a strict boundary where domain experts define declarative rubrics while infrastructure teams maintain runtime execution.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=4s) reasonable isn't good enough anymore — Charishma Bhavani Koneru

> [05:49](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=349s) LLM provides intelligence, code is the one which provides the governance. — Charishma Bhavani Koneru

> [14:22](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=862s) no score is better than the wrong score — Charishma Bhavani Koneru

## Mentioned Ai4, Bitcoin, Block Inc, Cash App, Charishma Bhavani Koneru, ChatGPT, Claude, Markdown, YAML

## Video description Charishma Bhavani Koneru - Senior Machine Learning Engineer, Block Inc

Building an AI agent is easier than building one people trust in production. In this talk, I'll share lessons from building and operating production AI copilots and autopilot systems for high-volume financial operations using specialized AI agents, deterministic decision logic, and continuous feedback loops. I'll explain how we earned trust through a copilot-to-autopilot rollout strategy, subject matter expert validation, analyst override feedback, and continuous LLM evaluations (LLM-as-Judge) that monitor quality and detect regressions in production. You'll leave with practical design patterns for building reliable AI agents, establishing robust evaluation pipelines, and safely deploying AI systems in regulated, high-stakes environments.

[Source](https://www.youtube.com/watch?v=Vo6NGRCxPWU)

中文简报

Block部署50个AI智能体打击Cash App欺诈，免写领域代码在高风险金融领域部署可靠的AI，需要将治理逻辑从大模型中剥离，转入确定性应用代码和声明式规则中，使风控分析师能在数分钟内完成更新。 Koneru详细阐述了Cash App如何从单体批处理管道转型为事件驱动的通用运行时（Universal Runtime），在30多种欺诈与风险类型中运行50多个专用子智能体。该平台使风控专家能够在数分钟内利用声明式Markdown规则和YAML配置部署新的调查领域，而无需编写可执行代码。为确保安全与合规，Block实施了从影子模式到有界自动驾驶的严格四阶段信任阶梯，并辅以人工介入飞轮。该运行时依靠确定性应用代码而非大模型提示词进行最终治理，并通过由确定性评分器、抽样大模型评判员以及人工趋势发现组成的三层管道进行评估。在高风险生产环境中部署AI的主要工程挑战已不再是模型原生能力，而是构建围绕自主智能体的确定性治理层与审计追踪。 (0:04) Cash App将其反欺诈架构从单体批处理大模型管道，演进为事件驱动的多阶段推理运行时，协调覆盖30多种欺诈类型的50多个专用子智能体。 (3:04) 领域专家无需编写任何可执行应用代码，即可使用声明式YAML配置和Markdown规则在数分钟内部署新的调查技能。 (3:04) 各专家智能体在相互隔离的状态下并行执行，无法查看彼此的中间推理过程，从而在综合分析前防止证据交叉污染。 (5:49) 自主解决、副驾驶审查与人工升级之间的路由分发，是基于置信度阈值在应用代码中确定性执行的，而非在大模型内部通过提示词实现。 (5:49)

Reading summary

At Block Inc.'s Cash App, deploying autonomous AI agents for high-stakes financial investigations requires prioritizing deterministic governance and auditability over raw model capability. The platform relies on declarative domain rubrics, multi-agent parallel investigations, and a graduated trust model to operate safely across millions of users. Governance Precedes Model Intelligence in High-Stakes Operations (0:04) Cash App serves over 50 million customers across peer-to-peer payments, banking, cash advances, and Bitcoin investing. Systems operating on these surfaces make critical determinations, such as freezing accounts, approving dispute claims, or ensuring regulatory compliance. While modern large language models can perform complex investigations out of the box, standard conversational competence is insufficient in regulated environments where false decisions harm users or breach legal mandates. Universal Runtime and Parallel Specialist Architecture (3:04) To mitigate risk and prevent engineering bottlenecks, Cash App redesigned its system from a 2024 legacy setup—characterized by scheduled batch pipelines and monolithic prompts—into an event-driven runtime orchestrating over 50 specialized parallel subagents across more than 30 fraud typologies. Rather than processing disputes through a single prompt, bounded specialist agents evaluate discrete domains like payment behavior and identity linkages simultaneously without sharing intermediate reasoning, preventing cross-contamination. Declarative Domain Rubrics and Deterministic Routing (8:31) Through the Universal Runtime platform, domain experts deploy new investigation logic using declarative YAML files and Markdown rubrics without writing executable code. This design enables hot-deploying fraud defenses in minutes while platform engineers manage pre-fetching, guards, and compliance rendering. Specialist agents emit over 100 structured signals that enter an evidence synthesis layer, where deterministic application code—not the language model—directs cases to autopilot, copilot review, or escalation paths. The Trust Ladder and the Continuous Feedback Flywheel (12:01) Agents acquire operational autonomy gradually by progressing through a four-stage trust ladder: shadow mode, copilot mode, sampled spot-checking, and bounded autopilot. Promotion requires meeting strict statistical thresholds, including sustaining a greater than 95% human agreement rate over a 30-to-60-day window and clearing drift checks. Even at maturity, human specialists retain 40% to 50% of incoming cases due to complexity or financial risk. A Layered Reliability and Production Evaluation Pipeline (14:22) When human operators override agent recommendations in copilot mode, the system captures their explicit reasoning to drive root-cause analyses and update declarative rubrics. Decision safety is reinforced by a three-layer evaluation engine comprising fast deterministic graders checking 100% of traces, an LLM-as-a-judge sampling 10% of cases that abstains if uncertain, and human expert review to catch novel fraud patterns invisible to backward-looking models. Operating autonomous AI agents safely in high-stakes financial domains depends on establishing deterministic evaluation guards, robust tracing, and strict boundaries separating declarative domain knowledge from core runtime governance.

精炼阅读摘要

Block公司旗下Cash App团队指出，在涉及资金冻结与合规审计的高风险金融场景中，自主AI代理的核心工程挑战已不再是模型推理能力，而是外围治理架构的设计；系统通过“大模型提供智能、确定性代码提供治理”的分工原则，确保所有自主决策具备严格的合规性与可追溯性。架构演进与声明式零代码扩展 (3:04) Cash App服务超过5000万用户，业务涵盖点对点支付、银行借贷和比特币投资等多个高风险领域。团队在2026年对系统进行了彻底重构，放弃了2024年依赖定期批处理流水线、单次单体大模型调用以及规则聚合器的旧架构，转向由事件驱动、支持多阶段代理推理和扩展思考的实时运行系统。面对数小时内即可演变的新型金融欺诈，传统需要耗费数天或数周编写专用代码的部署方式已无法满足防御需求。为此，团队构建了通用运行时（Universal Runtime），将业务逻辑与底层基础设施彻底解耦。业务专家与风险调查员无需编写任何可执行代码，仅通过声明式YAML文件定义数据源与工作流，并通过Markdown文件编写调查准则。通用运行时负责底层数据预取、确定性护栏拦截与合规报告生成。这一机制实现了新业务领域的零代码接入，调查团队在周二下午编写的新欺诈防范规则可在数分钟内热部署至生产环境，消除了软件工程部署的发布瓶颈。隔离式多智能体与确定性路由 (5:49) 面对设备指纹吻合但地理位置异常、或涉案用户发起第一方欺诈等高度模糊的争议案件，系统通过受限且职责单一的专业子代理展开并行调查。生产环境中部署了50多个专业子代理，并行覆盖支付行为分析、身份画像验证以及账户网络关联等30多种欺诈与风险类型。关键设计在于，各专业代理在并行执行时完全隔离，禁止查看彼此的中间推理过程，以此杜绝证据链的交叉污染。各专业代理最终输出涵盖4个证据领域的100多个结构化信号，每个信号均包含具体数值、置信度分数以及不可变的数据源引用。随后，证据综合层汇总调查结果并提交给确定性路由引擎。团队强调，工单分流决策由应用程序代码依据置信度阈值和监管标记硬性执行，而非交由大模型提示词决定：高置信度案件进入自动处理（Autopilot），中置信度案件转入人工复核协同（Copilot），低置信度或触发合规标记的案件则直接升级流转（Escalations）。阶梯式信任晋升与人工否决飞轮 (12:01) AI代理在上线之初不具备任何自主权限，必须沿预设的信任阶梯逐步晋升以验证安全性。阶梯包含四个阶段：覆盖100%案件但无实际操作权限的影子模式（运行数周）、仅提供建议并由人工审批的协同模式（运行数月）、代理执行后由人工抽样复核的抽检模式（运行数月），以及最终在受限范围内的有界自动处理模式。晋升必须跨过严格的统计门槛：在持续30至60天的观察期内，代理与人类决策的一致率必须超过95%，错误批准率低于风险调整阈值，升级分布无漂移，并顺利通过监管审查与统计A/B测试。即便系统已进入成熟运行阶段，由于新型欺诈和高资金风险的存在，系统仍主动保留40%至50%的案件完全由人工专家处理。在协同模式下，当分析师否决代理建议时，系统会强制记录否决原因与决策依据。这些结构化人工反馈被转化为高价值标注数据，触发对缺失数据信号或模糊提示词的根本原因分析，进而通过更新Markdown准则直接降低后续的否决率，形成持续迭代的学习飞轮。三层评估流水线与纵深防御体系 (14:22) 为了在生产环境中严密监控模型表现，系统设计了由硬到软的三层评估流水线。第一层为确定性评分器，以极低成本对100%的生产决策轨迹进行格式与结构检查，此层具备硬性一票否决权，不允许大模型覆盖；第二层为大模型裁判（LLM-as-a-Judge），对约10%的轨迹进行语义层面的抽样深检，遵循“没有分数好过错误分数”的原则，遇到解析异常或不确定性时主动弃权；第三层为人工趋势发现，由业务专家复核疑难边缘案件与判定分歧，以克服大模型裁判仅能依赖历史数据的局限性，并将新型攻击手法沉淀至版本受控的实时欺诈趋势日志中。这一流水线依托于整体五层可靠性堆栈：从最底层的模块化通用运行时、确定性护栏、可观测追踪，延伸至连续评估流水线和最上层的业务风险指标。系统实时对比错误率、升级率与历史基线，一旦探测到异常波动即刻触发自动调查或系统回滚。 Block团队的实践表明，在金融规模下运营自主系统的核心经验在于优先建立确定性评估机制与追踪基准，并通过结构化的人机分歧反馈闭环反哺系统。然而，面对能够数小时内变异的恶意欺诈者，如何平衡全自动化的效率诉求与高风险复杂案件中人工专家的不可替代性，仍是金融科技架构必须长期审视的权衡。

Detailed notes — zh

## Background 本视频由 Ai4 频道于 2026 年 8 月 20 日发布在 YouTube 平台上，内容为 Block 旗下 Cash App 资深机器学习工程师 Charishma Bhavani Koneru 的专题技术演讲。演讲基于处理高频金融运营的真实业务背景，分享了团队如何通过专业化 AI 智能体、确定性决策逻辑、专家反馈闭环及 LLM-as-Judge 持续评测机制，在生产环境中构建值得信赖的 AI Copilot 与 Autopilot 系统。 ## TL;DR 演讲系统性介绍了 Cash App 从初代批处理单模型流水线演进至 2026 年事件驱动通用运行时的架构历程，实现了风控业务专家仅凭编写 Markdown 规则便可在数分钟内“零代码”上线新欺诈对抗能力。系统严格践行“LLM 负责推理思考、代码负责确定性治理”的原则，由多个独立子智能体并行挖掘证据并绑定精准溯源，再经确定性路由分流处置。为确保金融安全与合规，团队设立了从影子模式到完全自主的四阶段信任晋升阶梯，并搭建了由确定性守卫、LLM 裁判与人类趋势发现组成的三层评测防线，有效兼顾了自动化效率与极端风控要求。 ## Atmosphere & style 整场分享展现出极其扎实、严谨且面向高可用工程落地的工业界技术风格，完全剥离了虚浮的行业炒作，直击高风险金融对抗的核心痛点。讲者条理清晰、节奏沉稳，通过详尽的架构图解、鲜明的数据对比与务实的踩坑经验，生动呈现了在大规模资金流动与监管审查重压下的系统设计取舍。听众能够深刻体会到在处理真实业务风险时，工程团队对确定性治理、证据可追溯性以及全链路评测体系的严苛追求与工程敬畏。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=4s) Cash App 构建的是能够自主处理高风险资金操作的生产级自主系统，工程核心挑战在于构建合规可信的治理层而非简单聊天机器人。 - [01:59](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=119s) 在强监管金融场景下，“看似合理”的回答远远不够，系统必须提供完整的数据血统与可追溯审计追踪，清晰证明每项决策的底层证据。 - [03:04](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=184s) 实现了新增业务领域“零代码”上线，业务风控专家仅需编写 Markdown 评判手册，数分钟内即可热部署生效并对抗新黑产套路。 - [04:44](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=284s) 系统从 2024 年基于定时调度的单次 LLM 批处理流水线，演进为 2026 年基于多阶段思维链推理与确定性治理的事件驱动实时架构。 - [05:49](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=349s) 争议调查采用多专业子智能体并行分析、合成证据与确定性代码路由分流，确立了“LLM 负责推理，代码负责治理”的架构原则。 - [08:31](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=511s) 通用运行时（Universal Runtime）将业务 YAML/Markdown 配置与平台底座彻底解耦，平台接管数据预取、确定性安全防护与合规报表生成。 - [10:11](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=611s) 放弃单体大模型调用的核心原因在于单体 Prompt 迭代维护脆弱、黄金评估集验证成本极高，且无法满足严格的证据溯源与零停机热部署需求。 - [12:32](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=752s) 自主权晋升阶梯（影子模式 -> Copilot -> 抽样审查 -> Autopilot）旨在证明系统安全性而非模型能力，需经过连续 30-60 天 >95% 人机一致性等多项硬性指标验证。 - [12:32](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=752s) 人类分析师的推翻（Override）被视为高价值的标注数据资产，通过捕获人工干预原因进行根本原因分析（RCA），持续推动 Agent 技能迭代升级。 - [15:28](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=928s) 构建三层评测流水线：覆盖 100% 流量的确定性守卫（硬失败拦截）、抽样 10% 的 LLM 裁判（语义评测，遵循“宁缺毋滥”弃权原则）与人类趋势发现。 - [15:28](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=928s) LLM 裁判本质上是“向后看”的已知模式打分工具，面对动态演进的新型欺诈必须依赖业务领域专家（SME）裁决并将分歧案例沉淀为黄金评测集样本。 - [18:13](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=1093s) 建立版本化、可审计的动态欺诈趋势日志（Live Fraud Trend Log），作为观察性输入被 Agent 技能平滑消费，避免系统发生剧烈波动。 - [18:13](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=1093s) 系统可靠性建立在涵盖架构、安全守卫、可观测性、评测体系及业务指标的“五层可靠性技术栈”之上，依赖多层协同拦截潜在风险。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=4s) 生产环境中的高风险 AI 决策与 Cash App 的业务版图在 Block 旗下的消费金融平台 Cash App，机器学习团队面临着极其严苛的业务场景。Cash App 拥有超过 5000 万活跃用户，其使命是让资金流动变得亲近、即时且普适。如今的 Cash App 早已不仅是一个简单的 P2P 支付应用，而是涵盖了银行业务、投资（如比特币交易）、借贷以及点对点转账的综合金融服务平台。这些丰富的金融功能在赋能大众的同时，也天然成为了金融犯罪和黑产攻击的目标。在这个庞大的生态系统中，系统每天必须对海量交易做出高风险、强实时的关键决策：这笔交易是否合法？该账户是否真实？是否存在欺诈行为？用户的申诉争议是否应该予以批准？演讲者 Charishma 指出，她的团队构建的不是聊天机器人（chatbots），而是能够自主处理“真金白银”的生产级自主系统（autonomous systems）。本次演讲的核心，正是分享团队如何建立一套稳健的治理层（governance layer），使得这些自主智能体在高度合规的场景下安全落地。 - **业务规模**：服务超 5000 万用户，覆盖 P2P 支付、银行存贷款、比特币投资等全套金融产品矩阵。 - **业务挑战**：每个产品入口都面临多样的欺诈向量（fraud vectors）与严格的监管合规要求（regulatory obligations）。 --- ### [01:59](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=119s) 核心挑战：从“模型能力”到“治理与可解释性”的范式转变当前大语言模型的技术发展让“构建一个 Agent”本身不再困难。无论是 Claude 还是 ChatGPT，给它一个复杂的调查任务，它都能给出一个“看似合理”（reasonable）的分析结果。然而在医疗、金融科技（Fintech）或法律等高风险领域，“合理”远远不够——系统做出的每一个决策都可能导致用户银行账户被冻结、欺诈索赔被驳回，或触发监管审查。在这些强监管场景下，系统不仅需要输出正确的结果，更必须提供完整的可追溯审计追踪（audit trail）。当监管机构进行合规审计时，平台必须能够一清二楚地展示是哪一条具体证据驱动了哪一个最终决定。因此，工程上的真正难点已经不再是 Prompt 工程或构建 Agent 本身，而是围绕模型搭建一套能够让其在规模化运行时完全值得信赖的**治理层**。 > “在金融科技等高风险操作中，你不仅需要得到正确答案，你更需要证明你是如何得出这个答案的……真正的工程挑战不是构建 Agent，而是构建围绕它的治理层。” --- ### [03:04](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=184s) 生产环境落地成果与演讲大纲团队已经在生产环境中稳定运行了一套高度成熟的 Agent 系统。该系统具备数十种独立的调查技能（investigation skills），每种技能都对齐了一套特定的业务操作规范；拥有超过 50 个并行运行的专业子智能体（sub-agents），覆盖了从电信诈骗赔付、制裁名单筛查到身份核验等 30 多种欺诈与风控类型。其中最引人瞩目的突破是**新增业务领域“零代码”（zero lines of code per new domain）**：当黑产出现全新欺诈套路时，业务专家只需编写一份 Markdown 格式的调查规范（rubric），几分钟内即可在生产环境生效，彻底消除了工程部署的瓶颈。本次分享涵盖六大核心板块：系统架构、信号工程（Signals）、治理机制（Governance，即 Agent 如何获得自主行动权）、三层评估体系（Evaluation）、生产监控（Monitoring）以及可直接复用的工程模式（Patterns）。演讲重点聚焦于架构设计与评估体系。 ```mermaid flowchart TD accTitle: 六大核心板块演进路线 subgraph Roadmap["核心技术全景"] direction LR A["1. 架构设计 (Universal Runtime)"] --> B["2. 信号工程 (证据与溯源)"] B --> C["3. 治理体系 (确定性准入)"] C --> D["4. 三层评估体系 (LLM-as-Judge)"] D --> E["5. 生产监控 (实时看板)"] E --> F["6. 设计模式 (落地方案)"] end ``` - **生产规模指标**：50+ 并行专业子智能体、30+ 欺诈与风险类型、数十种调查技能。 - **业务迭代效率**：新欺诈模式上线无需编写代码或发版部署，仅靠业务专家更新 Markdown 文件，数分钟内全量生效。 --- ### [04:44](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=284s) 架构演进：从 2024 批处理流到 2026 事件驱动通用运行时团队在 2024 年最初构建的系统与业界普遍做法相似：基于定时调度的批处理流水线（batch pipelines），每起纠纷案件只触发一次单模型 LLM 调用，再配合规则聚合器。这种架构在初期可行，但很快就无法适应对抗环境。黑产欺诈者的策略可以在几小时内迭代更新，而工程团队的发布周期却长达数天，每次支持新的欺诈类型都需要耗费数周的工程代码开发。为了解决这一痛点，团队推倒重来，构建了 2026 全新架构。新系统转为完全实时的事件驱动架构（event-driven），案件在发生瞬间即被捕获；底层结合多阶段思维链推理（multi-phase agentic reasoning with extended thinking），并在顶层叠加确定性治理逻辑；最核心的飞跃是将业务领域逻辑完全解耦为纯声明式的 Markdown 文件。 | 维度 | 2024 年初代系统 | 2026 年现代架构 | |:--- |:--- |:--- | | **处理模式** | 批处理流水线（按计划调度） | 事件驱动，实时逐案处理 | | **模型调用** | 单次 LLM 调用 + 规则聚合 | 多阶段 Agent 推理 + 扩展思考（Extended Thinking） | | **新业务接入** | 独立代码库、新 Agent 开发、耗时数周发版 | 业务专家编写 Markdown 文件，零代码，数分钟上线 | | **安全控制** | 简单规则拦截 | 确定性代码治理层与动态可信路由 | --- ### [05:49](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=349s) 争议调查的业务复杂度与多智能体分流架构以最典型的争议调查（Dispute Investigation）为例：用户联系客服表示“我没有授权这笔交易”。表面看似简单，实际分析极其复杂。证据往往充满歧义（例如设备指纹完全吻合但地理位置异常），参与主体具有强对抗性（部分为真实受害者，部分是恶意“黑吃黑”虚假索赔），同时受制于严格的金融法规时限（SLA 要求在规定时间内必须给出处理结果）。为了应对这一挑战，系统采用了并行多智能体协同架构。当案件上下文与用户申诉触发系统后，请求被分发给 6 个以上的专业子智能体（Specialist Agents）。每个智能体拥有严格限定的上下文与单一职责（例如支付行为分析、身份画像核验、关联网络拓扑等），彼此独立并行分析且互不可见，最终各自输出带有置信度得分与原始证据引用的结构化信号。这些信号汇聚到证据合成层，随后由**确定性路由器（Deterministic Router）**根据分数与合规规则进行分流处理。 ```mermaid flowchart TD accTitle: 争议调查多智能体分析与分流流程 A["案件输入 (申诉与交易上下文)"] --> B["并行专业智能体集群"] subgraph Specialists["专业分析智能体 (独立并行运行)"] B --> B1["支付行为智能体 (Payment Behavior)"] B --> B2["身份与画像智能体 (Identity & Profile)"] B --> B3["网络关系图谱智能体 (Network Linkages)"] B --> B4["其他专有风险分析智能体..."] end B1 --> C["证据落地合成层 (Evidence-Grounded Synthetic Layer)"] B2 --> C B3 --> C B4 --> C C --> D{"确定性代码路由 (Deterministic Router)"} D -- "高置信度" --> E["自动驾驶 Autopilot (系统直接执行动作)"] D -- "中置信度" --> F["辅助驾驶 Copilot (人工复核介入)"] D -- "低置信度 / 监管标记" --> G["人工上报 Escalation (资深专员处置)"] ``` > “核心洞察在于：路由决策是由确定性代码完成的，而不是依靠 Prompt；LLM 负责推理思考，而代码负责规约治理（LLM reasons, code governs）。” --- ### [08:31](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=511s) 通用运行时（Universal Runtime）：业务配置与平台底座彻底解耦为了实现真正的规模化，团队研发了“通用运行时”（Universal Runtime）。其终态原则是：**业务技能仅由 Markdown 与声明式 YAML 组成，不包含任何可执行代码**。业务专家作为技能作者（Skill Authors），只需定义所需的数据源、当前欺诈类型的工作流步骤，以及一份供 LLM 推理参考的标准作业程序（SOP/Rubric）。中间的通用运行时平台则接管了全部复杂的基础设施能力：自动拉取与预取数据源目录、执行所有确定性防护机制（guardrails）、编排智能体调用，并将最终结果渲染为面向合规监管的标准审计报告。这种设计让业务风控专家与平台工程师完全解耦，各自专注于自身领域。 ```mermaid flowchart TD accTitle: 通用运行时 Universal Runtime 分层架构 subgraph SkillAuthor["业务领域层 (业务风控专家)"] A["声明式配置 (YAML) 定义所需数据源与工作流"] B["判定手册 (Markdown Rubric) 定义推理逻辑与审核准则"] end subgraph PlatformRuntime["通用运行时底座 (平台工程团队)"] C["数据源自动目录化与预拉取 (Pre-fetching & Data Catalog)"] D["确定性守卫与拦截 (Deterministic Guards)"] E["多智能体编排与评估执行 (Orchestration & Evaluation)"] F["合规审计报告渲染 (Structured Audit Trail Report)"] end A --> C B --> E C --> D D --> E E --> F ``` - **业务风控专家的职责**：编写 YAML 数据依赖与 Markdown 评判手册，不编写也不接触任何业务执行代码。 - **平台工程团队的职责**：维护运行时调度底座、数据抓取管道、确定性安全守卫与合规报表生成模块。 --- ### [10:11](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=611s) 架构之辩：为何拒绝“单一大模型调用”？针对业界常有的疑问——“既然现代模型具备百万级 Token 的超大上下文窗口（Million-token Context Window），为何不直接将所有数据倒进单次大模型调用中处理，而要费力拆解为多个专业智能体？”Charishma 给出了四大核心工程原因： 1. **单体系统的迭代维护灾难**：在单体 Prompt 方案中，每当新增一种欺诈套路，都必须对整个庞大的系统进行重写、重新调优 Prompt 甚至重新微调。 2. **严格的黄金评估集（Golden Set）约束**：任何微小的改动都需要经过至少 30 个由业务专家手工标注的测试用例进行全面评估验证。单体系统牵一发而动全身，导致迭代成本极高。 3. **精准的数据血统与证据溯源（Provenance & Citations）**：当监管机构或审计员质疑“为什么驳回这笔争议”时，多智能体架构能够精确锁定具体是哪一个子智能体基于哪一条原始事实推导出的结论。 4. **零停机的热部署能力（Hot-deployable Reasoning）**：当周二下午突发新型黑产攻击时，业务人员仅需修改对应的 Markdown 文件即可在数分钟内热更生效，无需全量系统发版或停机。专业智能体最终产出的不仅仅是一个孤立的风险评分，而是跨越四个证据领域的 100 多个细分结构化信号。每个信号都绑定了具体的数值、置信度分数以及指向底层证据源的精准引用链路，从而构成了一套无可挑剔的合规审计闭环。 - **核心架构铁律**：LLM 赋予系统智能与深度推理能力，但底层代码必须牢牢把控确定性的执行边界与治理防线。 ### [12:32](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=752s) 从 Copilot 到 Autopilot：自主权晋升阶梯与人工干预飞轮构建智能体时必须建立一个关键认知：模型在当下其实已经具备了端到端自主决策的能力，自主权晋升阶梯的核心目的“不是为了证明模型的能力，而是为了证明其安全性”（"This ladder isn't about capability of a model. It's about proving safety"）。为了让 Agent 获得自主行动的治理授权，团队设计了一套严格的分阶段推进阶梯： ```mermaid flowchart LR accTitle: Copilot to Autopilot Promotion Ladder subgraph S1["阶段一：Shadow Mode"] A["影子模式运行 零客户影响 / 运行数周"] end subgraph S2["阶段二：Copilot"] B["提供证据与建议 人工做最终决策 / 运行数月"] end subgraph S3["阶段三：Spot Check"] C["Agent 决策 人工抽样审查 / 运行数月"] end subgraph S4["阶段四：Full Autopilot"] D["受限业务切片全自动 持续线上监控"] end A --> B --> C --> D ``` 晋升绝非按照普通的功能路线图推进，而是一套严密构建信任的治理协议。从一个阶段晋升到下一个阶段，必须满足硬性考核指标： - 在 30 天或 60 天的连续观察期内，与人类决策的一致性（Agreement Rate）需持续保持在 95% 以上； - 误批率（False Approval Rate）严格控制在既定的风险阈值之下； - 升级转人工率（Escalation Rates）无漂移； - 获得合规与监管部门的正式审批签字（Regulatory Sign-off），并取得具有统计学显著性的 A/B 测试结果。每次人类分析师推翻（Override）Agent 的决策，都被视作一份宝贵的“礼物”——因为这意味着团队免费获得了一条附带明确原因的标注训练数据。业务闭环通过捕获人类的覆盖决策与原因，触发根本原因分析（RCA）：排查究竟是缺失了特征信号、提示词存在缺陷，还是出现了全新的欺诈模式。随后工程师针对性地更新具体“技能”（Skill），推动 Agent 持续进化，使覆盖率进一步下降并积累信任。明确自动化的边界同样至关重要。并不是所有业务都应该完全自动化，在 Cash App 的高风险金融运营场景中，仍有 40% 至 50% 的复杂、新颖或极高风险的案例由人工专家处理。 --- ### [15:28](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=928s) 三层评测体系：Guardrails、LLM-as-Judge 与人类趋势发现评测（Evaluation）往往是许多工程团队投入不足、也是吸取惨痛教训最多的领域。为了保障生产环境的高可靠性，系统部署了三层相互协作的评测与门控机制： ```mermaid flowchart TD accTitle: Three-Layer Evaluation Pipeline T["生产环境调用链路 (Trace)"] --> G1{"Gate 1: 确定性守卫 (Deterministic Guardrails)"} G1 -- 结构性失败 / 违规 --> F1["硬失败 (Hard Fail) LLM 无法覆盖"] G1 -- 通过 (100% 流量) --> G2{"Gate 2: LLM 裁判 (LLM-as-Judge)"} G2 -- 语义违规 / 解析异常 --> F2["弃权/拦截 (Abstain) 宁缺毋滥"] G2 -- 抽样 10% / 正常 --> G3{"Gate 3: 人类趋势发现 (Human Trend Discovery)"} G3 -- 发现未知新模式 --> F3["SME 领域专家裁决 扩充黄金评测集 & 趋势日志"] G3 -- 已知模式正常 --> PASS["放行并通过"] ``` - **第一层 / 第一道门：确定性守卫（Deterministic Guardrails）** 覆盖 100% 的调用链路（Traces）。其运行速度极快且成本低廉，能瞬间捕获所有结构性错误与硬性合规违规。若第一道门判定失败，即为最终判定（Hard Fail），后置的 LLM 绝无权限覆盖。 - **第二层 / 第二道门：LLM 作为裁判（LLM-as-Judge）** 对约 10% 的样本进行抽样评测。虽然成本较高，但能够有效识别确定性规则无法捕捉的语义层错误。若裁判模型出现解析错误或判断不确定，系统选择“弃权”（Abstain）——“没有评分远好于给出错误的评分”（"No score is better than the wrong score"），系统在评测层坚决不进行无把握的猜测。 - **第三层 / 第三道门：人类趋势发现（Human Trend Discovery）** 这是最关键却最容易被忽视的防线。LLM 裁判本质上是“向后看”的（Backward-looking），只能根据已有训练数据中的已知模式打分；但金融欺诈、用户纠纷与欺诈手法处于动态演化中。当检测到未知新模式时，案例会被路由至 SME（业务领域专家）进行深度裁决。一旦专家意见与 Agent 产生分歧，该案例会立即沉淀为一条全新的“黄金评测集”（Golden Set）样本，使评测基准能够跟随生产环境的真实活数据持续扩充。 --- ### [18:13](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=1093s) 动态欺诈趋势日志与五层可靠性技术栈为了将新出现的风险模式系统化落地，团队构建了动态的“实时欺诈趋势日志”（Live Fraud Trend Log）。该日志具备严格的版本控制与可审计性，由 SME 反馈、欺诈分析师研判、异常检测算法及外部威胁情报持续供数。业务飞轮的运转逻辑为：一旦有人或系统捕获到新型欺诈手法，立即打标并发布至共享趋势日志中，相关 Agent 技能消费该日志并自适应更新。值得注意的是，这些趋势属于“观察性输入”而非不可变更的死板指令，它们会经过模式匹配再逐步吸收，避免系统出现不稳定的剧烈摆动。在日常运维中，监控贯穿于生产运行的每一天，而非仅停留在发布上线时刻。一旦关键错误率超出既定基准线，系统便会触发深度调查并执行回滚。由此总结出的“五层可靠性技术栈”（Five-Layer Reliability Stack）由下至上涵盖： 1. **基础架构层（Architecture）**：底层的执行框架与链路设计； 2. **安全守卫层（Guardrails）**：确定性校验与硬性约束； 3. **可观测性层（Observability）**：全链路 Trace 监控与指标度量； 4. **评测层（Evaluation）**：结合 LLM-as-Judge 与 Golden Set 的综合打分； 5. **业务指标层（Business Metrics）**：最终的业务转化、欺诈资损与合规指标。没有任何单层能够独立解决所有可靠性问题，每一层都在捕获从其他层漏网的潜在风险。如果在项目第一天能够重新开始，核心经验包括：明确成功的衡量指标且优先使用确定性评估工具而非过早依赖 LLM 裁判；尽早在系统上线首月就收集全量反馈数据；务必沉淀人类推翻 AI 的具体原因。最后，这套支撑高可用自主金融 Agent 框架的落地，离不开机器学习工程师、软件工程师、数据科学以及业务运营团队全员的深度协作。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=4s) 构建的不是聊天机器人（chatbots），而是能够自主处理“真金白银”的生产级自主系统（autonomous systems）。 — Charishma Bhavani Koneru > [01:59](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=119s) 在金融科技等高风险操作中，你不仅需要得到正确答案，你更需要证明你是如何得出这个答案的……真正的工程挑战不是构建 Agent，而是构建围绕它的治理层。 — Charishma Bhavani Koneru > [03:04](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=184s) 新增业务领域“零代码”（zero lines of code per new domain） — Charishma Bhavani Koneru > [05:49](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=349s) 核心洞察在于：路由决策是由确定性代码完成的，而不是依靠 Prompt；LLM 负责推理思考，而代码负责规约治理（LLM reasons, code governs）。 — Charishma Bhavani Koneru > [08:31](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=511s) 业务技能仅由 Markdown 与声明式 YAML 组成，不包含任何可执行代码 — Charishma Bhavani Koneru > [10:11](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=611s) LLM 赋予系统智能与深度推理能力，但底层代码必须牢牢把控确定性的执行边界与治理防线。 — Charishma Bhavani Koneru > [12:32](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=752s) This ladder isn't about capability of a model. It's about proving safety — Charishma Bhavani Koneru > [15:28](https://www.youtube.com/watch?v=Vo6NGRCxPWU&t=928s) No score is better than the wrong score — Charishma Bhavani Koneru ## Mentioned Charishma Bhavani Koneru, Block Inc, Cash App, Ai4, Claude, ChatGPT, Universal Runtime, LLM-as-Judge, Bitcoin ## Video description Charishma Bhavani Koneru - Senior Machine Learning Engineer, Block Inc Building an AI agent is easier than building one people trust in production. In this talk, I'll share lessons from building and operating production AI copilots and autopilot systems for high-volume financial operations using specialized AI agents, deterministic decision logic, and continuous feedback loops. I'll explain how we earned trust through a copilot-to-autopilot rollout strategy, subject matter expert validation, analyst override feedback, and continuous LLM evaluations (LLM-as-Judge) that monitor quality and detect regressions in production. You'll leave with practical design patterns for building reliable AI agents, establishing robust evaluation pipelines, and safely deploying AI systems in regulated, high-stakes environments. [Source](https://www.youtube.com/watch?v=Vo6NGRCxPWU)

Source quotes

[0:04] Charishma Bhavani Koneru: 构建的不是聊天机器人（chatbots），而是能够自主处理“真金白银”的生产级自主系统（autonomous systems）。 [1:59] Charishma Bhavani Koneru: 在金融科技等高风险操作中，你不仅需要得到正确答案，你更需要证明你是如何得出这个答案的……真正的工程挑战不是构建 Agent，而是构建围绕它的治理层。 [3:04] Charishma Bhavani Koneru: 新增业务领域“零代码”（zero lines of code per new domain） [5:49] Charishma Bhavani Koneru: 核心洞察在于：路由决策是由确定性代码完成的，而不是依靠 Prompt；LLM 负责推理思考，而代码负责规约治理（LLM reasons, code governs）。 [8:31] Charishma Bhavani Koneru: 业务技能仅由 Markdown 与声明式 YAML 组成，不包含任何可执行代码 [10:11] Charishma Bhavani Koneru: LLM 赋予系统智能与深度推理能力，但底层代码必须牢牢把控确定性的执行边界与治理防线。 [12:32] Charishma Bhavani Koneru: This ladder isn't about capability of a model. It's about proving safety [15:28] Charishma Bhavani Koneru: No score is better than the wrong score

Source quotes

[0:04] Charishma Bhavani Koneru: reasonable isn't good enough anymore [5:49] Charishma Bhavani Koneru: LLM provides intelligence, code is the one which provides the governance. [14:22] Charishma Bhavani Koneru: no score is better than the wrong score

Original transcript

[0:04] I'm Charishma and I lead risk automation ML team at Cash App. That's blocks consumer [0:10] finance platform serving over 50 million customers. My team builds AI agents that [0:15] make final like financial high-stake decisions in production. They're not just chat bots, but they [0:20] are autonomous systems like deciding things like is this account fraudulent or should we approve [0:26] this dispute? real decisions and real money and every day. So today I'm going to share how [0:33] we built the governance layer that lets us trust these agents to operate autonomously. What worked, [0:39] what broke, and the patterns you can take home and apply to your own systems. Everything I'm [0:44] showing you is live in production. Let me give you a quick sense of where I work. If someone [0:50] is not aware of what Cash App is, it's uh our mission is to make money relatable, [0:55] instant and universal. We are building financial tools that empower everyone everywhere. Tens of [1:01] millions of people are using Cash App daily to manage their financial lives. Let me show you the [1:07] scope of what it actually means in Product Sense. So this is the Cash App entire product suit. It's [1:14] not just a payments app anymore. It's a full financial services platform, banking, investing, [1:20] peer-to-peer networking, and borrowing. Each one of these surfaces attacks financial crime, [1:25] which is why risk operations run at such a large scale. Everyone has fraud vectors. Everyone has [1:32] regulatory obligations. When someone sends a peer-to-peer payment, invests in Bitcoin, [1:39] or takes a cash advance, there are decisions being made about whether the transaction is legitimate, [1:45] whether that account is authentic, whether that activity is compliant. That's the surface our AI [1:50] agents are operating across. Now, what brings us to the core challenge we are solving is [1:59] here's the thing. Building an Agent is not the hard part anymore. Models today are capable [2:04] of end-to-end autonomy. You can give cloud or charge GPT a complex investigation and it will [2:11] be a reasonable job, but reasonable isn't good enough anymore. You're making decisions that [2:18] freeze someone's bank account or denying a fraud claim or flag someone for regulatory services. [2:26] In high stakes operations like healthcare or fintech or legal, you don't just need the right [2:33] answer. You need to prove what you did to get there. Like you need an audit audit rail. You [2:39] need to show a regulator exactly which decision or like which evidence drove which decision. So [2:45] the real engineering challenge isn't building the agent anymore. It's building the governance [2:50] layer around it. the thing that makes it trustworthy enough to operate autonomously [2:55] at scale. Let me show you what that looks like when you get it right. So this is what [3:04] we have built and have been running in production today. Dozens of distinct investigation skills, [3:10] each one encoding a different operational procedure. Over 50 specialized subgents [3:15] that run in parallel and more than 30 different fraud and risk typologies from [3:21] scam reimbursements to sanctions, screening to identity verification. And here is the number [3:28] one I'm most proud of. That is zero lines of code per new domain. When a new fraud pattern emerges, [3:35] our domain experts write a markdown file. That's the investigation rubric and it's live [3:40] in production within minutes. No code changes, no deploys, no engineering bottleneck. We'll dig into [3:47] how that works architecturally in a moment. But this is like a main landscape that we are going [3:51] to dive into. And this is the road map for the next 20 minutes. There are six sections. [3:58] I'll spend the most time on architecture and the evaluation. in the section 4. Those are [4:04] where the novel patterns live. Architecture is what the system looks like today. And signals [4:09] is how we engineer the evidence. Governance is how agents earn the right to act autonomously. [4:16] And evaluation is our three layer pipeline. This is where most teams underinvest. And [4:23] I'll get to a point where how I'll show you like in detail how the evaluation takes place. [4:28] The fourth one is the monitoring what production actually looks like. And the sixth one patterns [4:34] is that's what you can steal and apply tomorrow. First let's take the leap. Okay. So we started in [4:44] 2024 with like how most of the team started with like batch pipelines flows running on schedule and [4:51] each claim got a single LM call a rule-based aggregator system. Every new domain means a [4:57] new code base, new agents, and new deployments. It worked, but it couldn't keep up. Adversaries adapt [5:06] in hours. Our deploy cycle was days, and every new fraud type meant weeks of engineering effort. So, [5:13] we rebuilt from scratch. We have now is event driven. Each case is processed in real time as [5:21] it arrives. a multi-phase agentic reasoning with extended thinking and deterministic governance [5:27] layered on top of it and the key unlock is new domains are just markdown files zero code. The [5:33] rest of the talk is about the system on the right which is the 2026 system and [5:38] specifically the governance layer that makes it safe to let these agents operate autonomously. [5:49] Okay. So let's take an example like a dispute investigation. A customer [5:54] contacts us and says I didn't authorize this transaction. It sounds simple, [5:59] right? But it's not because the evidence is ambiguous. Maybe the device fingerprint [6:06] matches but the location doesn't or the actors are adversarial. Some claims are legitimate. [6:13] Some are fraud by the claimant themselves and there are regulatory deadlines that we need [6:19] to follow. There's a fixed window to respond like each dispute claim that we received has a [6:25] certain SLA from the regulatory obligations and we have dozens of different topologies of fraud [6:31] each with different signals and different decision criteria. So what the system needs to do is gather [6:38] the evidence and from different data sources, reason over that evidence, make a decision, [6:45] execute an action and produce an audit trail all in real time for every single claim or [6:51] a request you receive from your customer. So I'm going to show you how we architected it. [6:59] So this is a multi- aent architecture. A case comes in and we have a click context, a customer [7:06] report or some transaction details and it fans out to parallel specialized agents. Each one has [7:12] a bounded context and a single responsibility like for example payment behavior goes over the [7:18] payment patterns about that particular customer and the next one like the identity and profile [7:23] checks the verification details and the network linkages maps connections between accounts. [7:29] So similarly we have like six different kind of specialist agents and all those agents [7:34] run in parallel and they don't see each other's responses and each produces a structured signal [7:40] with conference course and evidence citations. Those signals flow into an evidence grounded [7:46] synthetic layer and it's not just a summary but it's a structured compilation and then [7:52] the deterministic router takes over. So it's like high confidence cases goes into the autopilot and [7:59] the system acts whereas the medium confidence cases goes to co-pilot and the human reviewer [8:04] would be there on top of it who reviews those claims and the low confidence or the regulatory [8:10] flags go to escalation. The key insight here is the routing is deterministic and it's it's the [8:17] code which is making the decision not the prompts but the LLM is the one which reasons and the code [8:23] governs. Now let me show the architectural innovation here that makes this scalable. [8:31] So this is the slide I'm most excited about and this is what we are going to do. We're doing it [8:36] already in 2026. That is the we call it internally the universal runtime and this is what which makes [8:43] it different from most teams which have already adapted in the last one year of using co-pilots [8:48] or autopilots. The end state principle is a skill is only markdown and declarative [8:55] config. So nothing executable ships from the skill authors and the platform. The second [9:00] layer is what handles everything else at the top like the skill authors only write in AML like the [9:07] data sources what are needed and the workflow which needs to run for that particular typology [9:13] and an instruction manual which is like a rubric for the LLM to reason about whereas in the middle [9:19] the platform owns the entire runtime. So the platform is the one which actually does all the [9:25] pre-fetching the catalog of all the data sources which are defined in the AML and then the markdown [9:31] and then it it is the one which does all the deterministic guards and then finally it renders [9:36] a report in a structured format that is what is displayed for the regulators in order to know the [9:42] audit trail and the result is adding a new fraud typology is a new markdown file and the platform [9:52] platform handles the orchestration, evidence fetching, evaluation and the deployment. Domain [9:59] experts write only the investigation rubric and the engineers are the one who maintain the second [10:04] layer which is the platform and the infrastructure behind it. So they both don't touch each other's [10:11] code and you might be wondering why not just have one big LLM call, right? Like a one big model [10:18] call here. So many people ask this question and also models today have milliondoll context like [10:27] a million token context windows and you could dump everything into one call and let it figure [10:33] out like why bother with having a specialization like four different specialist agents or like 10 [10:40] different specialized agents right so there are main like there are mainly four main reasons for [10:45] this so first thing is it's not possible with a monol analytic approach you will need to retrain [10:52] or reprompt the entire system every time you update with a new scam pattern. Second is every [11:00] single change to SQL requires a golden set of like at least 30 subject matter experts labeled [11:06] cases and a passing evaluation. Third, every signal has provenence and citations. So when [11:14] a regulator asks why did you deny this claim we can point to the exact agent the exact evidence [11:20] and the exact reasoning part one. So the fourth one is the hot deployable reasoning which is like [11:26] whenever a new scam pattern emerges like for example on a Tuesday afternoon we can update [11:32] one markdown file and it's live in minutes no deploy no downtown so the bottom line is [11:40] a single agentic call can handle this we choose specialist because regulators auditors and codes [11:47] need to trace exactly which evidence drove which decision. So that would be very easy to point out [11:53] in that scenario. [snorts] Now let's talk about what the specialist agents actually produce. So [12:01] the signals do not emit a score. They emit like 100 plus signals across four evidence domains. [12:08] And each signal carries a value, a confidence and a citation back to the source evidence from [12:14] which it derived the score and a composite package with a full audit rail. Signals also [12:20] aren't enough right you need boundaries like if the LLM has to reason completely end to end and [12:27] this is the single most important architecture principle like LLM provides intelligence code is [12:32] the one which provides the governance here so how does an agent earn the right to act autonomously [12:42] here's the key reframe I want you to take away the model is capable of end to end autonomy today This [12:49] ladder isn't about capability of a model. It's about proving safety. Step one is the shadow [12:55] mode. And here the agent runs on every case. But its decisions have zero customer impact. We are [13:03] just measuring how the agent is performing. And this particular step one takes weeks in order [13:09] to gain trust. And the step two, the agent presents its evidence and recommendations, [13:15] but a human makes the final call. That's the co-pilot stage and this runs for months. Now [13:22] going to the step three. The agent decides but the human spot checks at sample and we run [13:28] this for months and the final step is the full autopilot bounded segments only not the entire [13:35] decision flows and we continuously monitor the final step and each promotion requires a [13:42] hard criteria like greater than 95% agreement rate with human decisions sustained for like [13:48] a 30-day or a 60-day period and false approval rates below some certain like risk thresholds, [13:54] no drift and escalation rates, regulatory sign off and significant like statistical AB test [14:00] results. All this testing takes place for each and every step and to move forward in the next level. [14:06] So this isn't a feature road map, but it's like a trust building protocol in order to [14:11] build a complete autonomous system. And this is the flywheel that makes it work. [14:22] Every time a human overrides the agent's decision, that's like a gift for us because it's free [14:29] labelled training data with a reason attached. And the agent makes a decision. Human reviews [14:36] it. If they override, we capture the override and the reason that triggers root cause analysis like [14:44] was it a missing signal or was it a prompt gap or is it new pattern and based on that we update the [14:51] skill and that's how the agent gets better. So the override override rate goes down and that's [14:59] how you gain trust. But here's the thing I want to emphasize. Not everything should be automated, [15:05] right? Like even in our scenario about 50% or 40% of cases still stay human. They're like [15:13] the complex ones or the novel ones or like the high stake ones. So we need to know our [15:18] boundary. It's not like we can automate every financial operations risk flow that cash app [15:22] is having at this point. But it's important that we are pushing in that direction. Now, how do you [15:28] know the agent is actually working? Here comes the evaluation part. So, this is where most [15:36] teams typically underinvest and it's where we have learned the most painful lessons and we run three [15:45] layers of evaluation currently. The layer one is deterministic guardrails. So, these run on 100% of [15:53] all traces. They're fast and cheap and they catch all the structural failures instantly. So the lay [16:02] two is LLM as judge. These run on like 10% of the samples. They're expensive but they still catch [16:09] semantic failures which the rules can't catch. And going to the third third layer which is the most [16:16] important one and most people miss that is the human trend discovery which is where the subject [16:23] matter experts feedback loops novel pattern detection golden set expansion and living fraud [16:29] trend log all this comes into the picture and here is why the layer three matters more because LLM [16:37] judges are backwardl looking right they grade against known patterns like you train a judge [16:43] with some certain patterns that are already found in your own training data. But the fraud evolves, [16:49] disputes or human behavior, they constantly change. New scam types emerge. And the feedback [16:56] loop on the right shows how human discovery feeds the trend log and then that's taken to update the [17:04] skills and which improves the production agent. Now let me show you how these layers interact. [17:14] Every trace goes through three gates. There are no shortcuts like the gate one is a deterministic [17:20] grader that's a hard fail final verdict. No override by the LLM. And gate two if the LLM [17:28] judge has like a pass error then we abstain. So it's like no score is better than the wrong [17:33] score. So we never guess. And the gate three is if a novel pattern is detected it goes to human [17:40] review and theme or like a subject matter experts is the one who adjudicates that scenario and if [17:47] they disagree with the agent that case becomes a new golden set entry and the evaluation set grows [17:54] continuously with the production live data. So the bottom line is every theme or a human [18:00] disagreement we have with the agent becomes a golden set case and every new fraud pattern [18:06] enters a shared trend log. The pipeline doesn't just grade it learns and the third evaluation [18:13] layer deserves its own slide right now which is this is the uh this is the flywheel which [18:19] shows how the live fraud trend log is uh going to work. It's version controlled, [18:26] auditable and it's fed continuously by SM feedback fraud analyst anomaly detection [18:33] and external threat. So the circle works like this spots something new flags it publishes [18:40] to the shared log relevant skills consume it and the agent adapts. So the key choice is the [18:46] trends are observations not directives. they go through the matching pattern and it's not [18:51] like you adapt to the trends consistently across a period. Now let's talk about production looks [18:58] uh like today. So this is how we keep monitoring it every day and not just at deploy time. So all [19:06] these figures are illustrative but what I would like to advertise is um error rate above certain [19:12] baseline is where you investigate and you roll back if something is not working as expected. [19:19] Oh, I think we have only one more minute. So, I'm just going to go a little bit faster. So, [19:24] this is how like what broke in production like if there are any data gaps or like trace schema, [19:29] this is how you fix it at at individual layers given that you have like every single [19:35] layer in production has all these systems integrated. So, this instruments inform [19:41] our like reliability philosophy. So, this is a five layer reliability stack. So the [19:47] base layer is the architecture and the above that is the guardrails. Then the observability [19:53] evaluation and the business metrics. No single layer is sufficient. Each one catches failures [19:58] that slip through the others. So let me close this patterns. So this is what we have learned [20:05] and what I would do differently if I started all over this tomorrow. Um you need to know how you [20:11] measure your success and you don't need an LLM to catch them. you need to start with deterministic [20:16] graders and three you wish you had all this data from month one not after you deploy your systems. [20:23] So the fourth is like if you can't capture why humans disagree you can't improve it and that's [20:28] very much important feedback. Um finally six patterns you can take home. These are the six [20:34] design patterns which are really useful if you wanted to build an end toend autonomous systems. [20:44] Um so finally it takes a village. So this is the entire team of ML [20:48] software engineers and data science and operations [20:50] who got this entire framework ready. And thank you everyone. [applause]