What It Takes to Build an Agent Platform for Customers with Docusign and PCMag / Fast Company

Ai4

AI-assisted summary of What It Takes to Build an Agent Platform for Customers with Docusign and PCMag / Fast Company — original source. Check the original for full context.

Source published: 2026-08-23

Detailed notes

## Background Published on YouTube by Ai4 on August 23, 2026, this session features Tabrez Mohammed, VP of AI at Docusign, in conversation with tech journalist Rob Pegoraro from PCMag and Fast Company. The discussion covers the architectural tradeoffs, security safeguards, and platform design patterns necessary to build a production-grade, external-facing AI agent platform for enterprise customers.

## TL;DR Tabrez Mohammed details how Docusign developed its customer-facing Intelligent Agreement Management (IAM) agent platform in just four to six months by combining off-the-shelf components with proprietary agreement workflows. To handle sensitive contracts affordably and reliably, Docusign developed privacy-preserving fine-tuning pipelines that achieve frontier-level performance at 1/20th the cost, orchestrating tasks via LangGraph and optimizing prompts using DSPy. The platform mitigates legal hallucinations and security threats through multi-layered extraction pipelines, granular token-level cost telemetry, user-scoped OAuth permissions, and mandatory human-in-the-loop controls for irreversible contract executions.

## Atmosphere & style The session balances sharp technical pragmatism with a conversational, lighthearted vibe, featuring playful banter around legal tech hallucinations, Black Hat security risks, and engineer sleep deprivation. Pegoraro serves as an engaging, mildly skeptical host referencing industry critiques and real-world vulnerabilities, while Mohammed delivers authoritative, grounded insights focused on telemetry, unit economics, and sensible enterprise architecture over generative AI hype.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=sOGtknSLLpU&t=4s) Docusign transitioned from standard e-signatures to AI-driven agreement intelligence to help enterprises extract actionable insights and track obligations across vast repositories of signed contracts. - [02:01](https://www.youtube.com/watch?v=sOGtknSLLpU&t=121s) AI powers both pre-commit contract risk analysis against thousands of company agreements and post-commit lifecycle tracking, such as proactive obligation audits and 30/90-day renewal tracking. - [04:25](https://www.youtube.com/watch?v=sOGtknSLLpU&t=265s) Customer privacy is preserved through an automated multi-layer de-identification pipeline validated by stratified sampling, ensuring zero direct employee inspection of customer contracts. - [04:25](https://www.youtube.com/watch?v=sOGtknSLLpU&t=265s) In-house fine-tuned legal models achieve or exceed frontier model performance on agreement-specific tasks at 1/20th or less of hyperscaler API costs. - [06:21](https://www.youtube.com/watch?v=sOGtknSLLpU&t=381s) Docusign delivered an enterprise agent platform in 4 to 6 months by assembling existing open-source components first and selectively engineering custom replacements only when measurement revealed bottlenecks. - [08:21](https://www.youtube.com/watch?v=sOGtknSLLpU&t=501s) The platform relies on external graph-based orchestration (LangGraph) rather than LLM-internal orchestration to ensure deterministic behavior, strict execution boundaries, and modular component design. - [08:21](https://www.youtube.com/watch?v=sOGtknSLLpU&t=501s) Dynamic model routing assigns frontier reasoning models to multi-step legal analysis while directing discrete tool calls and extraction tasks to smaller, cost-optimized models. - [08:21](https://www.youtube.com/watch?v=sOGtknSLLpU&t=501s) Rigorous token-level telemetry on every input, output, and tool call safeguards unit economics and prevents AI margin erosion. - [11:09](https://www.youtube.com/watch?v=sOGtknSLLpU&t=669s) Implementing DSPy and automated prompt optimization algorithms (GEPA) cut operational prompt and token costs by more than 50%. - [12:13](https://www.youtube.com/watch?v=sOGtknSLLpU&t=733s) Hallucinations are minimized by replacing single-shot prompts with classical ML pre-extraction, continuous self-verification against source text, and inspectable intermediate artifacts. - [14:37](https://www.youtube.com/watch?v=sOGtknSLLpU&t=877s) Security blast radius is constrained by restricting arbitrary code execution, enforcing user-scoped OAuth tokens to prevent confused deputy privilege escalation, and requiring human-in-the-loop checkpoints for signing. - [16:35](https://www.youtube.com/watch?v=sOGtknSLLpU&t=995s) Enterprise customer behavior is shifting from interactive conversational chats to autonomous, event-driven agent workflows integrated directly into Docusign workflow services.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=sOGtknSLLpU&t=4s) From E-Signatures to Agreement Intelligence

Rob Pegoraro opens the conversation by introducing Tabrez Mohammed, Vice President of AI at Docusign, while jokingly acknowledging AI's notoriously rocky track record in legal tech. Pointing to infamous incidents where lawyers used hallucinating LLMs that cited non-existent case law before unimpressed judges, Pegoraro playfully warns any lawyers in the audience: "please don't do that if there are any lawyers in the room." He asks what convinced Docusign to lean heavily into AI despite these well-known risks.

Tabrez explains that Docusign has operated in the agreement space for more than two decades, building an e-signature business that has served over a billion users. However, once businesses accumulate millions of signed documents, a fundamental challenge emerges: agreements cannot just sit in a digital filing cabinet. Organizations urgently need actionable insights from their post-signing repository. Docusign's AI journey began by pairing classical machine learning with modern large language models (LLMs) to extract key contract terms and make existing features "2x, 5x, 10x better, more accurate." Over the last two years, as agentic platforms matured, Docusign shifted focus toward creating net-new capabilities that were previously impossible to build without an agent framework.

- **Scale of Docusign:** Over 20 years in the agreement space with an e-signature user base exceeding one billion people. - **Initial AI Strategy:** Modernizing legacy feature sets with LLMs to yield dramatic accuracy and performance multipliers before developing net-new agentic capabilities.

---

[02:01](https://www.youtube.com/watch?v=sOGtknSLLpU&t=121s) Pre-Commit Risk Analysis and Post-Signing Lifecycle (IAM)

Addressing what AI actually adds to the end-user signing experience, Tabrez breaks the workflow into two critical stages: pre-commit evaluation and post-commit agreement management. On the commit side, reviewing a contract—whether it is a single page or a hundred pages—requires high confidence. AI tools allow users to actively interrogate the document, deeply analyze specific clauses, and compare the incoming draft against hundreds or thousands of existing agreements across the enterprise. This contextual comparison reveals how a new contract shifts the company's liability profile and surfaces hidden obligations at a scale human review could never match before LLMs.

On the post-signing side, AI continuously monitors operational commitments and renewal windows (such as 30- or 90-day deadlines). It cross-references contract terms against real-world vendor performance to determine whether both parties fulfilled their agreed obligations. These insights feed directly into subsequent contract negotiations, optimizing future agreement terms. Tabrez notes that delivering on this promise—telling users exactly what they signed up for—is the core mission behind Docusign IAM (Intelligent Agreement Management), a platform suite introduced two years ago to manage post-signature workflows across enterprise business processes.

- **Pre-Commit AI Capabilities:** Natural-language document interrogation, clause risk analysis, and comparative benchmarking against thousands of enterprise contracts. - **Post-Commit Lifecycle Tracking:** Proactive obligation monitoring, automated 30/90-day renewal tracking, and vendor performance audits to inform future contract renegotiations.

---

[04:25](https://www.youtube.com/watch?v=sOGtknSLLpU&t=265s) Privacy-Preserving Contract Training and Achieving 1/20th the Cost

When asked what datasets Docusign used to train its AI models, Tabrez emphasizes that while inference quality depends strictly on training data quality, contracts are among the most sensitive proprietary assets any business owns. Docusign maintains strict privacy boundaries: employees do not browse customer contracts. To leverage this repository safely, the team developed a novel, multi-layered de-identification and anonymization pipeline designed to remove personally identifiable information (PII) and corporate identifiers without stripping out the semantic structure required for downstream legal reasoning.

The team continuously validates this data pipeline by running stratified sampling on post-anonymization documents to ensure that re-identification of any party is impossible. Once sanitized, this data feeds into fine-tuning and post-training pipelines. Tabrez reports that Docusign's specialized models achieve or exceed the performance of frontier hyperscaler models (such as GPT-4 or Claude) on agreement-specific tasks while operating at "1/20th or less of cost." This drastic efficiency gain enables Docusign to run deep, multi-contract analytics directly on its platform far more affordably than customers could by building DIY integrations on top of off-the-shelf commercial APIs.

```mermaid flowchart TD accTitle: Docusign Privacy-Preserving Fine-Tuning Pipeline

subgraph DataIngestion ["1. Ingestion & Privacy"] A["Enterprise Contract Repository"] --> B["Multi-Layer De-Identification Engine"] B --> C["Stratified Sampling & Re-Identification Checks"] end

subgraph Optimization ["2. Training & Tuning"] C --> D["Sanitized Contract Dataset"] D --> E["Domain-Specific Fine-Tuning & Post-Training"] end

subgraph Serving ["3. Production Inference"] E --> F["Specialized Legal & Agreement Models"] F --> G["Frontier-Level Performance at <= 1/20th Hyperscaler Cost"] end ```

- **Data Privacy Guarantee:** Zero direct human inspection of customer agreements; automated multi-stage anonymization verified through stratified re-identification testing. - **Cost-to-Performance Ratio:** In-house fine-tuned models match or beat frontier LLMs at 1/20th (or lower) the compute and token cost.

---

[06:21](https://www.youtube.com/watch?v=sOGtknSLLpU&t=381s) Building a Customer-Facing Agent Platform in 4 to 6 Months

Moving from foundational models to interactive customer-facing agents required rapid execution. While Docusign's IAM platform was launched two years prior, work on the dedicated agent platform began at the start of the calendar year and reached production in time for announcement at Docusign Momentum in May—an end-to-end development timeline of roughly four to six months. Pegoraro remarks that shipping a customer-facing enterprise platform that quickly sounds like a startup pace and jokingly asks if the engineering team managed to get any sleep.

Tabrez attributes this rapid velocity to an intentional architectural philosophy: recognizing what constitutes Docusign's true intellectual property. Docusign's IP lies in its proprietary domain capabilities, agreement workflows, and platform APIs—not in inventing custom agent harnesses from scratch. The team adopted existing open-source and off-the-shelf components to build an end-to-end functional prototype within weeks. Only after running the integrated system against live data and spotting specific bottlenecks—such as information loss, accuracy degradation, or excessive token costs—did engineers systematically ablate and replace individual framework layers with custom-engineered components.

- **Time to Market:** 4 to 6 months from initial project inception to an enterprise-ready production launch at Docusign Momentum. - **Pragmatic Prototyping Strategy:** Stitching together existing ecosystem tools within weeks, then selectively replacing components only when measurement exposed accuracy, cost, or latency deficiencies.

---

[08:21](https://www.youtube.com/watch?v=sOGtknSLLpU&t=501s) LangGraph Orchestration, Model Tiering, and Token-Level Instrumentation

Discussing the platform's core architecture, Tabrez highlights the industry debate over whether orchestration should live inside the LLM agent or outside the agent via deterministic state graphs. Docusign firmly selected external orchestration using LangGraph. External control provides strict behavioral guarantees and predictable execution paths that enterprise customers expect when handling legally binding documents. However, to avoid vendor or framework lock-in, Docusign designed every subsystem to be modular and swappable within months if architecture requirements shift.

```mermaid flowchart LR accTitle: External Agent Orchestration and Dynamic Model Routing

subgraph OrchestrationLayer ["External Orchestration (LangGraph)"] UserReq["User Input"] --> StateRouter{"External State Graph"} end

subgraph ModelRouting ["Task-Based Model Routing"] StateRouter -->|"Complex Legal Reasoning"| FrontierModel["Core Frontier / High-Parameter Model"] StateRouter -->|"Specialized Tool Calls & Extraction"| FineTunedModel["Lightweight / Fine-Tuned Model"] end

subgraph Observability ["Telemetry & Unit Economics"] FrontierModel --> TokenMonitor["Token & Cost Instrumentation"] FineTunedModel --> TokenMonitor end ```

On model selection, Docusign employs dynamic model routing: heavy frontier models handle complex multi-step legal reasoning, while lightweight, fine-tuned models handle structured tool calls and metadata extraction. When Pegoraro brings up industry skepticism around GenAI unit economics—referencing a presentation by tech critic Ed Zitron arguing that nobody wants to pay the real operational costs of AI models—Tabrez explains Docusign's rigorous cost-control methodology. To prevent cost overruns from threatening margins or inflating customer bills, Docusign instruments every single input and output token across individual tool calls and entire conversation sessions. Tabrez compares this discipline to traditional web engineering, where fixing performance requires relentless telemetry on metrics like time-to-first-pixel.

- **Orchestration Choice:** External graph-based orchestration via LangGraph to maintain strict predictability, deterministic boundaries, and component modularity. - **Model Routing Strategy:** Large reasoning models for complex high-level analysis; smaller, cost-optimized models for discrete tool execution and extraction tasks. - **Token Telemetry:** Comprehensive tracking of input and output tokens per tool call and chat session to ensure predictable unit economics and prevent margin erosion.

[11:09](https://www.youtube.com/watch?v=sOGtknSLLpU&t=669s) Cost Optimization and Automated Prompt Engineering with DSPy

Measuring model performance, expenditures, and token spend is considered a foundational baseline, but the most significant operational efficiency comes directly from cost optimization strategies. A core rule of thumb is straightforward: never deploy a larger model when a smaller model is sufficient for the task.

To systematically handle optimization, Docusign relies heavily on the framework DSPy. Tabrez Mohammed jokes that anyone unfamiliar with DSPy will likely end up building an inferior, homegrown version of it before eventually discovering the framework and switching over anyway. Integrating DSPy established a structured, repeatable iteration loop for refining and improving prompts. By leveraging automatic prompt optimization techniques like GEPA (Generative Prompt Optimization Algorithms), the team managed to reduce operational costs by over 50% across most implementations by optimizing the individual tool prompts.

- Cost reductions exceeding 50% were achieved through automated prompt optimization and tool-level tuning. - DSPy serves as the primary optimization and prompt iteration harness across agent implementations.

```mermaid flowchart LR accTitle: DSPy Prompt Optimization Workflow subgraph Evaluation ["Prompt Iteration"] A["Initial Tool Prompts"] --> B["DSPy Framework"] B --> C["Automated Prompt Optimization (GEPA)"] end subgraph Deployment ["Model Execution"] C --> D["Right-Sized Small Models"] D --> E["50%+ Token Cost Reduction"] end ```

---

[12:13](https://www.youtube.com/watch?v=sOGtknSLLpU&t=733s) Mitigating Hallucinations in Legal Workflows

When off-the-shelf LLMs are fed legal documents, outputs can seem convincing at first glance; however, when applied across hundreds or thousands of complex documents, standard single-shot approaches rapidly degrade and go off the rails. Rather than treating legal analysis as a single-shot prompt where the model is told to simply "go figure it out and come back with the answer," Docusign leverages a multi-layered pipeline.

The architecture pairs LLMs with classical machine learning models and specialized, constrained extraction systems. These systems pre-compile and vet foundational data beforehand, providing a structured trajectory that keeps the LLM within strict guardrails. During extraction and analysis, models are required to self-verify their outputs against explicit citations from the source text.

Looking forward, Docusign is conducting frontier research into concepts like progressive introspection (similar to LLM Wiki mechanisms). This enables models to produce structured intermediate artifacts over long-running jobs (such as 30-minute deep research tasks across 100-page reports). By exposing these intermediate reasoning steps, customers can inspect the progression of the work step-by-step rather than having to reverse-engineer an unverified final output.

- Single-shot LLM document parsing is replaced with pre-compiled, vetted data from classical ML models and constrained LLM extractors. - Output validation relies on continuous self-verification and strict source citations. - Progressive introspection research allows customers to verify long-running (30-minute) research tasks through inspectable intermediate artifacts.

---

[14:37](https://www.youtube.com/watch?v=sOGtknSLLpU&t=877s) Threat Modeling, Sandboxing, and Human-in-the-Loop Safeguards

Rob Pegoraro brings up the concurrent Black Hat conference in Las Vegas, asking about the agent platform's threat model and the need for red teaming. To move rapidly from zero to one while maintaining a tight security perimeter, Docusign made intentional architectural constraints.

To eliminate critical threat vectors at launch, the agent harness was explicitly restricted from writing or executing arbitrary code (a capability being introduced gradually with tighter controls). Furthermore, the platform strictly enforces user tokens rather than system tokens when authenticating with downstream APIs. Because actions execute purely within the individual user's authorization scope, the blast radius of any unexpected failure or hallucination is naturally limited to what that user is already permitted to do. This prevents privilege escalation vulnerabilities such as confused deputy attacks, ensuring that User A cannot manipulate the agent into exposing data from User B's account.

For high-stakes contract modifications, Docusign enforces strict Human-in-the-Loop (HITL) checkpoints. These prevent prompt injection or autonomous execution errors from triggering irreversible actions, such as an agent automatically signing or executing an agreement without explicit human sign-off.

- Blast radius is constrained by forcing downstream tool calls to run via user-scoped OAuth tokens rather than broad system tokens, preventing confused deputy attacks. - Direct code generation was disabled in early releases to eliminate major execution vulnerabilities. - Human-in-the-Loop (HITL) approval gates are strictly mandated for state-changing contract operations, preventing automated unauthorized document execution.

---

[16:35](https://www.youtube.com/watch?v=sOGtknSLLpU&t=995s) Customer Adoption and the Autonomous Roadmap

Customer adoption has scaled rapidly, with Docusign IAM (Intelligent Agreement Management) and its agent platform standing out as the fastest-growing product launch in the company's history. Enterprise customers who previously spent days or weeks on manual contract review processes are seeing those workflows compressed into minutes or hours within a couple of hours of tuning agent workflows.

Customer behavior is already transitioning from interactive, conversational queries toward automated, event-driven agent runs triggered by Docusign workflow services. Behind this expansion, Docusign's dedicated AI science team is actively conducting frontier research—with upcoming academic papers planned for major machine learning conferences. Key research areas include non-formal verification techniques for LLM outputs (outside traditional mathematical or code verification) and platform infrastructure supporting safe semi-autonomous and fully autonomous agent execution in collaboration with third-party ecosystem partners.

Pegoraro and Mohammed close the session with a lighthearted remark to the audience, inviting questions and joking that they won't make anyone sign a contract before chatting.

- IAM and agent tooling represent the fastest-growing product line in Docusign's history. - Workflows are shifting from interactive conversational chats to autonomous, event-driven pipelines integrated into Docusign workflow services. - Docusign's science team is publishing research on verification methodologies and autonomous multi-agent orchestration.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=sOGtknSLLpU&t=4s) please don't do that if there are any lawyers in the room. — Rob Pegoraro

> [00:04](https://www.youtube.com/watch?v=sOGtknSLLpU&t=4s) 2x, 5x, 10x better, more accurate. — Tabrez Mohammed

> [04:25](https://www.youtube.com/watch?v=sOGtknSLLpU&t=265s) 1/20th or less of cost. — Tabrez Mohammed

> [12:13](https://www.youtube.com/watch?v=sOGtknSLLpU&t=733s) go figure it out and come back with the answer, — Tabrez Mohammed

## Mentioned Ai4, Black Hat, Claude, Docusign, Docusign IAM, Docusign Momentum, DSPy, Ed Zitron, Fast Company, GEPA, GPT-4, LangGraph, PCMag, Rob Pegoraro, Tabrez Mohammed

## Video description Tabrez Mohammed - VP of AI, Docusign Rob Pegoraro - Contributor, PCMag / Fast Company

Most AI agent talks focus on building tools for employees—but what changes when your users are your customers? In this talk, I will explore the unique challenges of building an externally facing agent platform at Docusign, including the strategic decisions and architectural tradeoffs required to balance long-term platform growth with the need to ship customer value quickly. I will share practical patterns for designing a secure, multi-tenant, extensible foundation that enables teams to deliver new agent capabilities rapidly without sacrificing scalability or maintainability. If you're building AI into a SaaS product, you'll leave with a framework for making platform decisions that help you move fast today while setting yourself up for success tomorrow.

[Source](https://www.youtube.com/watch?v=sOGtknSLLpU)

中文简报

Docusign 携手 PCMag / Fast Company：如何打造面向客户的 Agent 平台 Tabrez Mohammed 详细介绍了 Docusign 如何通过将现成组件与专有协议工作流相结合，在短短四到六个月内开发出面向客户的智能协议管理（IAM）Agent 平台。为了以可负担的成本且可靠地处理敏感合同，Docusign 开发了保护隐私的微调流水线，以二十分之一的成本实现了前沿模型级别的性能，并通过 LangGraph 编排任务以及使用 DSPy 优化 Prompt。该平台通过多层提取流水线、细粒度的 Token 级别成本遥测、限定用户范围的 OAuth 权限，以及针对不可逆合同执行的强制性人机协同（human-in-the-loop）控制，来缓解法律幻觉和安全威胁。 Docusign 从标准电子签名转型为 AI 驱动的协议智能，帮助企业在庞大的已签署合同库中提取可操作的洞察并跟踪履约义务。 (0:04) AI 既赋能签署前对照数千份公司协议进行合同风险分析，也支持签署后的生命周期跟踪，例如主动义务审查和 30/90 天续约跟踪。 (2:01) 通过经分层抽样验证的自动化多层去标识化流水线来保护客户隐私，确保员工零直接查阅客户合同。 (4:25) 自主微调的法律模型在协议特定任务上的表现达到或超过了前沿模型水平，而成本仅为超大规模云厂商 API 费用的二十分之一或更低。 (4:25) Docusign 在 4 到 6 个月内交付了企业级 Agent 平台，其方式是首先组装现有开源组件，仅在指标测度揭示出瓶颈时才选择性地研发定制替代方案。 (6:21)

Reading summary

According to Tabrez Mohammed, Vice President of AI at Docusign, transforming static e-signature archives into an actionable enterprise agreement platform requires pairing deterministic external orchestration with domain-specific model tuning, rigorous anonymization, and granular cost telemetry to overcome legal hallucination and security risks. Pre-Commit Analysis and Post-Signing Lifecycle Management (2:01) Docusign's expansion from e-signatures—a segment serving over one billion users—into Intelligent Agreement Management (IAM) targets the operational inertia of signed contracts left in digital repositories. Mohammed divides this workflow into pre-commit evaluation and post-commit execution. Prior to signing, natural language interrogation tools analyze single- or multi-hundred-page drafts, evaluating specific clauses and benchmarking proposed terms against thousands of internal enterprise agreements to expose liability shifts and obligations that human review could miss. Post-signature, the IAM system continuously tracks operational commitments, alerting organizations to 30- and 90-day renewal deadlines and auditing vendor fulfillment against contractual obligations. According to Mohammed, these performance audits feed directly into subsequent renegotiations, turning post-signing repositories into ongoing operational intelligence. Sanitized Training Pipelines and 1/20th Compute Costs (4:25) Because contracts represent confidential enterprise assets, Docusign enforces strict privacy boundaries that bar human employees from browsing customer documents. To train domain-specific models safely, the team engineered an automated, multi-stage de-identification and anonymization pipeline that strips personal and corporate identifiers while preserving the semantic structure necessary for legal reasoning. The pipeline is continuously audited through stratified sampling to verify that re-identification remains impossible. Mohammed reports that Docusign's fine-tuned models match or exceed the performance of frontier commercial LLMs on agreement-specific tasks while operating at 1/20th or less of their compute and token cost. This efficiency makes multi-contract comparative analytics economically viable on Docusign's platform compared to building DIY integrations over off-the-shelf commercial APIs. LangGraph State Routing and Automated Prompt Tuning (8:21) To build an enterprise-ready agent platform within four to six months for Docusign Momentum, the engineering team assembled modular open-source components before selectively replacing bottlenecks. Docusign chose external, deterministic state-graph orchestration via LangGraph over internal LLM-directed planning to maintain the predictable execution paths required for legal agreements. The architecture routes high-level legal reasoning to frontier models while offloading structured tool calls and metadata extraction to smaller, fine-tuned models. To manage inference costs and protect gross margins, Docusign instruments token usage across every tool invocation and user session. Utilizing the DSPy framework and automated prompt optimization algorithms such as GEPA, the team systematically optimized tool-level prompts, which Mohammed claims reduced operational token costs by more than 50% across most implementations. Anti-Hallucination Guardrails and Authorization Boundaries (12:13) To prevent the hallucinations that undermine standard LLMs in legal settings, Docusign avoids single-shot prompting across document repositories. Instead, the platform combines LLMs with classical machine learning and constrained extraction engines that pre-compile foundational data within rigid boundaries, requiring models to self-verify statements against explicit source citations. For extensive analysis tasks across 100-page reports, Docusign is researching progressive introspection to expose step-by-step intermediate reasoning artifacts for customer inspection. Security and threat models were addressed by restricting autonomous agent execution at launch, specifically excluding direct code generation. Downstream tool executions are restricted to user-scoped OAuth tokens rather than broad system tokens, preventing privilege escalation vulnerabilities such as confused deputy attacks by keeping actions within the user's existing permissions. High-stakes contract actions, including final agreement execution, strictly require human-in-the-loop approval gates. Mohammed reports that IAM and the agent platform represent the fastest-growing launch in Docusign's history as enterprise review processes shrink from days to hours, yet transitioning customer behavior from conversational prompts to fully autonomous, event-driven pipelines remains dependent on ongoing research into non-formal verification methodologies and safe ecosystem orchestration.

精炼阅读摘要

Docusign AI副总裁Tabrez Mohammed指出，公司正从单一电子签名服务向智能协议管理（IAM）转型，依托外部确定性编排与专用小模型构建客户智能体平台。该架构在严格保护合同隐私与企业权限的前提下，将履约风险分析与事后追踪成本降低至前沿模型的二十分之一，实现从被动归档到事件驱动型自主工作流的跨越。协议全生命周期风险评估与履约追踪 (2:01) 据Tabrez Mohammed介绍，Docusign依托深耕协议领域二十余年、服务超十亿用户的积累，将智能体重点放在签署前风险审查与签署后履约管理两个关键阶段。在签署承诺前，面对单页至上百页的复杂合同，AI支持用户以自然语言深度询问条款，并将拟签文本与企业内部数千份历史协议进行比对基准分析。这种跨文档对比能够快速定位责任条款的变化与隐藏义务，弥补了传统人工审查在处理海量文档时的效率瓶颈。在签署完成后，智能协议管理（IAM）平台持续监控各项运营承诺与关键续约窗口（例如30天或90天预警），并结合供应商的实际履约表现与合同条款进行交叉核验。这些事后审计数据将直接反馈至后续的合同谈判中，为企业优化商务条款提供数据依据。Docusign强调，让企业确切掌握自身签署的法律义务并形成决策闭环，是其构建IAM产品矩阵的核心目标。隐私隔离脱敏与高性价比模型微调 (4:25) 企业合同属于高度敏感的私有资产，Docusign在模型训练上设立了严格的隐私红线，严禁员工直接查阅客户协议。为安全利用海量协议资产，技术团队研发了多层去标识化与匿名化数据处理流水线，在彻底剥离个人身份信息（PII）与企业标识符的同时，完整保留下游法律推理所需的语义和句法结构。该流水线通过对匿名化处理后的文本进行分层抽样检查，确保在统计学和技术上彻底杜绝反向重新识别的可能。完成清洗的数据被输入到下游领域微调与后训练流程中。Tabrez Mohammed表示，Docusign自主微调的专用法律模型在特定协议任务上的表现已经追平甚至超越了诸如GPT-4或Claude等通用前沿大模型，而运行推理的计算与Token成本仅为前沿模型的二十分之一甚至更低。这使得企业能够在平台内直接以极低成本运行多合同深度分析，避免了自行调用商用公有API构建系统时面临的高昂开支。确定性图编排与敏捷组件替换架构 (8:21) Docusign专用的客户智能体平台从立项到在5月Docusign Momentum大会上正式发布，全流程仅耗时4至6个月。团队认为自身的核心知识产权在于深度的协议领域认知、业务工作流与平台API，而非底层的智能体运行框架。因此，工程师在初期选用现成的开源与商用组件于数周内搭建端到端可用原型，随后在真实生产流量中持续监控信息丢失、准确度下降及Token消耗等瓶颈，再针对性地对特定模块进行消融实验与定制化重构。在系统核心架构上，Docusign放弃了让LLM自行内部决定调度路径的方案，选择采用基于LangGraph的外部图状态编排。这种设计为涉及法律效力的严肃业务提供了确定性执行路径与严格的行为边界，同时确保各子系统保持模块化以便在未来灵活替换。系统实施动态模型路由策略：由高参数前沿模型负责复杂的多步骤法律逻辑推理，而轻量级微调模型则专注执行结构化工具调用与元数据提取，兼顾了推理准确度与响应延迟。细粒度Token遥测与抗幻觉验证管道 (11:09) 面对行业对生成式AI单位经济效益的质疑，Docusign引入了如同传统网页性能调优般的细粒度工程遥测体系，对单次工具调用乃至完整会话中的每一个输入和输出Token进行全面度量，严格控制运营毛利率。在具体优化策略上，团队全面采用DSPy框架及其GEPA自动化调试工具提示词，实现小模型对大模型的高效替代，在多数业务实现中将单个工具的运营Token成本削减了50%以上。为解决LLM在面对数千页复杂法律文本时输出迅速失真的问题，平台坚决摈弃了单次Prompt直接求解的单发模式。系统构建了多层复合架构，先由传统机器学习与受限抽取系统对基础数据进行预编译和审查，为LLM提供受控的上下文轨道；在分析过程中，要求模型强制对照源文本引用执行自我验证。此外，团队正探索类似渐进式自省机制，使耗时达30分钟的长周期深度研究任务能够暴露中间结构化工件，供用户分步审查而非仅提供无法追溯的最终结论。严格权限隔离与人在回路安全防线 (14:37) 在安全威胁建模方面，Docusign采取了收缩攻击面的系统性防范措施。在首发版本中，智能体被严格禁止动态编写或执行任意代码，从根源上杜绝了代码执行层面的安全漏洞。更重要的是，平台与下游API交互时强制使用用户范围的OAuth令牌，彻底弃用特权系统令牌。这意味着智能体的所有操作范围被严格限定在当前用户的既有权限内，既从架构上限制了幻觉可能造成的潜在损害，又彻底防御了恶意提示词注入可能引发的混淆代理特权提权攻击。针对涉及合同变更与签署的高风险场景，系统强制设立人在回路（HITL）审批关卡，禁止智能体在没有人工显式授权的情况下自主签署或生效任何法律文本。随着企业客户将合同审查从以周为单位压缩至数小时以内，客户使用行为已从初期的交互式对话，逐步演进为由工作流服务自动触发的事件驱动型流水线。Docusign的AI科研团队亦在同步推进非形式化验证等前沿研究，为未来安全的半自主与全自主多智能体协作奠定理论基础。根据Docusign披露的数据，IAM及智能体工具已成为其历史上增长最快的产品线。然而，如何在大规模多智能体自治协作与第三方生态集成中，持续维持非形式化输出验证的严密性与法律合规确定性，依然是该平台向完全自动化演进过程中尚待解决的核心课题。

Detailed notes — zh

## Background 本视频由 Ai4 于 2026 年 8 月 23 日发布在 YouTube 平台。内容为 Docusign 人工智能副总裁 Tabrez Mohammed 与《PCMag / Fast Company》撰稿人 Rob Pegoraro 的深度对谈，围绕 Docusign 如何为外部客户打造安全、多租户且可扩展的生产级 AI Agent 平台，分享了在 SaaS 产品中平衡快速交付与长期架构演进的实战经验与决策框架。 ## TL;DR 对话系统回顾了 Docusign 从传统电子签名迈向智能协议管理（IAM）的 AI 演进历程，重点阐述了团队在 4 至 6 个月内基于 LangGraph 与成熟生态极速交付 Agent 平台的工程哲学。通过严密的数据脱敏、专用小模型微调以及 DSPy 自动化提示词优化，Docusign 在特定任务上将推理成本压缩至前沿模型的 1/20，并在工具调用中实现 50% 以上的降本。针对法律合同的严苛要求，平台通过“传统 ML 预提取 + 轨迹护栏 + 中间工件可视化”抑制大模型幻觉，并依托用户级令牌隔离与高危操作人机协同（HITL）构建安全屏障。目前该产品已成为公司增长最快的业务线，正加速由对话交互演进为工作流驱动的自主 Agent 系统。 ## Atmosphere & style 整场对谈务实专业、节奏紧凑，开场主持人借法律界大模型幻觉案例轻松破冰，迅速引申出严肃的工程与商业命题。嘉宾的分享褪去了 AI 领域常见的情绪化炒作，以沉着、坦诚且严谨的架构师视角，直击 Token 成本失控、重复造轮子以及提权漏洞等一线落地痛点。观众在聆听过程中能够深刻感受到高阶技术管理者在面对不确定性技术浪潮时，那种兼顾极致敏捷与稳健防御的清醒底气与实战智慧。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=sOGtknSLLpU&t=4s) Docusign 从传统电子签名业务拓展至智能协议管理（IAM），将大语言模型与传统机器学习结合，深挖海量合同在签署后的管理与商业洞察价值。 - [02:01](https://www.youtube.com/watch?v=sOGtknSLLpU&t=121s) AI 贯穿协议全生命周期：签约前支持跨成百上千份合同的大规模深度比对与风险识别，签约后实现到期履约追踪与动态数据反哺。 - [04:31](https://www.youtube.com/watch?v=sOGtknSLLpU&t=271s) 通过多层去标识化与分层抽样脱敏构建安全数据集，微调出的专用轻量模型在特定任务上匹敌前沿大模型，且推理成本降至前沿模型的二十分之一以下。 - [06:22](https://www.youtube.com/watch?v=sOGtknSLLpU&t=382s) 践行“不重复造轮子”的敏捷工程哲学，早期借助现有开源与商用工具快速搭建 PoC，仅对验证为性能瓶颈的层级进行自研剥离，仅耗时 4 至 6 个月便交付生产级 Agent。 - [08:20](https://www.youtube.com/watch?v=sOGtknSLLpU&t=500s) 基于 LangGraph 采用“外部编排”架构，确保系统具备强确定性与高内聚低耦合的可插拔能力，并按任务复杂度在大小模型间实现动态路由。 - [10:04](https://www.youtube.com/watch?v=sOGtknSLLpU&t=604s) 建立类似传统 Web 性能优化的 Token 级可观测体系，严格审计每次工具调用与会话消耗，防止按量计费下的推理成本失控。 - [11:09](https://www.youtube.com/watch?v=sOGtknSLLpU&t=669s) 遵循“小模型优先”原则，重度依托 DSPy 框架进行程序化提示词优化与工具调优，在绝大多数业务场景中将运行成本削减 50% 以上。 - [12:13](https://www.youtube.com/watch?v=sOGtknSLLpU&t=733s) 拒绝大模型“单次直出”黑盒猜测，通过传统 ML 预编译结构化数据与受限轨迹约束来抑制幻觉，并强制生成溯源引用与中间工件以实现渐进式可验证性。 - [14:37](https://www.youtube.com/watch?v=sOGtknSLLpU&t=877s) 在安全威胁模型上主动收敛攻击面，初版禁用代码生成执行，且与下游 API 交互全流程采用 User Token 而非 System Token，彻底消除越权与混淆代理隐患。 - [14:37](https://www.youtube.com/watch?v=sOGtknSLLpU&t=877s) 对协议签署等不可逆法律与资产操作设立强制的人机协同（HITL）拦截护栏，杜绝 Agent 在无人工确认的情况下擅自代表客户签约。 - [16:35](https://www.youtube.com/watch?v=sOGtknSLLpU&t=995s) 客户使用形态正快速从即时对话交互迈向由工作流服务自动触发的事件驱动型 Agent，将耗时数周的手工业务流程压缩至数分钟级自动化。 - [16:35](https://www.youtube.com/watch?v=sOGtknSLLpU&t=995s) 未来路线图聚焦于非形式化验证场景下的模型可信验证体系构建，并推动系统从半自主向更广泛生态协同的全自主 Agent 演进。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=sOGtknSLLpU&t=4s) 从电子签名到智能协议管理：Docusign 拥抱 AI 的契机对谈伊始，主持人 Rob Pegoraro 幽默地提到了 AI 在法律和合同领域应用的尴尬历史——不少律师曾让 AI 做法律检索，结果模型凭空捏造了不存在的判例，导致法官大为光火；他借此打趣现场若有律师千万别这么做，并顺势抛出问题：Docusign 为何决定全力投入 AI 领域？ Docusign 人工智能副总裁 Tabrez Mohammed 回顾了公司的演进历程：Docusign 深耕协议领域超过 20 年，其核心电子签名（e-signature）业务已积累了超过 10 亿用户。然而，签署只是协议生命周期的开始。企业在积累了数以亿计的合同后，最大的痛点在于如何在签署后提取其中的商业洞察并加以管理。Docusign 首先利用大语言模型（LLM）与传统机器学习方法结合，将现有功能的效果与准确率提升了 2 倍、5 倍乃至 10 倍。而在过去两年中，随着 AI Agent（智能体）技术的成熟，团队开始探索以往无法构建的全新功能，将其作为面向未来的核心投资方向。 ### [02:01](https://www.youtube.com/watch?v=sOGtknSLLpU&t=121s) 签约前后的全生命周期 AI 赋能：从风险比对到履约追踪在谈及 AI 具体接管了哪些业务流程时，Tabrez Mohammed 详细拆解了签约前（Commit）与签约后（Post-signing）的典型场景：在签署阶段，面对几十甚至上百页的长篇合同，AI 不仅能辅助用户快速理解合同内容以建立签署信心，更重要的是能跨协议进行宏观比对。企业用户可以将眼前的新合同与库中已有的成百上千份合同进行对比分析，迅速识别出潜在的风险转移、严苛条款与核心义务，这种大规模且高精度的条款深度分析在 LLM 普及之前几乎无法实现。在签署后阶段，AI 则化身为全天候的合同监控助手。比如系统会实时追踪 30 天或 90 天后的合同到期与续约节点，并结合该供应商过往的实际业务运营与履约记录，评估其是否达成了合同承诺。这些动态洞察会反哺到下一轮续约谈判或新合同生成中，实现商业流程的闭环优化。两年前推出的智能协议管理（IAM, Intelligent Agreement Management）正是基于这一逻辑重塑了合同管理的行业认知。 ### [04:31](https://www.youtube.com/watch?v=sOGtknSLLpU&t=271s) 数据安全脱敏与低成本微调：以 1/20 成本匹敌顶尖模型高质量的推理离不开高质量的数据，但商业合同属于企业最敏感的核心资产，绝不可能允许员工随意翻阅。为了解决数据隐私与模型训练的矛盾，Docusign 构建了一套严密的分层去标识化与匿名化（Anonymization / De-identification）流程： ```mermaid flowchart TD accTitle: Docusign 合同数据脱敏与模型优化流程 subgraph S1["数据脱敏阶段"] A["原始合同库<br/>(海量高敏感商业合同)"] --> B["多层去标识化 / 匿名化处理"] B --> C["分层抽样审查<br/>(验证无法反查身份或 PII)"] end subgraph S2["模型微调与部署"] C --> D["安全无害训练集"] D --> E["后训练与针对性微调<br/>(Post-training / Fine-tuning)"] E --> F["专用轻量模型<br/>(成本仅为前沿大模型的 1/20)"] end ``` 在脱敏过程中，系统会通过多层过滤剔除所有个人身份信息（PII）与企业标识，并通过严格的分层抽样监控，确保数据无法被逆向推导还原。在此基础上构建的安全数据集，会被用于后训练（Post-training）与针对性微调（Fine-tuning）。Tabrez Mohammed 指出，相关技术细节已在官方博客中公开；测试结果表明，该方案不仅在特定任务上达到甚至超越了云巨头前沿大模型（Frontier LLMs）的表现，而且推理成本仅为其二十分之一（1/20th or less），这使得平台能够以极低成本为用户提供远超自建方案的深度分析能力。 ### [06:22](https://www.youtube.com/watch?v=sOGtknSLLpU&t=382s) 4 至 6 个月的极速交付：避免重复造轮子的工程哲学从零起步到在 5 月的 Docusign Momentum 大会上正式发布面向客户的生产级 Agent 平台，团队仅耗时 4 到 6 个月。面对主持人关于团队是否需要牺牲睡眠加班的调侃，Tabrez Mohammed 解释了背后的核心研发策略：**极力借力成熟生态，拒绝过早重复造轮子**。团队非常清醒地认识到，Docusign 的核心知识产权（IP）在于基于数据和业务 API 构建的功能层与业务逻辑，而非底层的 Agent Harness 或基础模型。因此，在项目初期： - 团队在短短几周内利用市面上现成的 Agent 工具链、开源模型和已有 API 快速拼装出端到端的最小可行性系统（PoC）。 - 随后通过实测定位系统中的信息损失、准确率不足或成本过高的环节。 - 仅在被验证为瓶颈的特定层级进行剥离（Ablation）和自研替换，从而以初创团队的速度实现了成熟系统的上线。 ### [08:20](https://www.youtube.com/watch?v=sOGtknSLLpU&t=500s) 架构决策：LangGraph、外部编排与模块化可替换性在技术选型上，团队避免盲目追逐行业热度，而是进行了为期数周的多方案 PoC 评估，最终选择了 LangGraph 作为核心 Harness。但 Docusign 的使用方式具有其特殊性： - **外部编排（Outside Orchestration）**：业内常争论编排应该发生在 Agent 内部还是外部。Docusign 倾向于外部编排，认为这能提供更强的确定性保障，更符合企业客户对稳定性的严苛预期，而 LangGraph 的底层语义非常契合这种控制需求。 - **高内聚低耦合的可替换设计**：所有选型组件均设计为可插拔架构，如果未来 3 个月内出现更优方案，可以随时无缝替换。 - **任务驱动的模型路由分发**：核心推理与规划交由更强大的大型模型，而具体的工具调用（Tool Calls）与子任务则路由给轻量化、专门优化过的低成本模型。 ### [10:04](https://www.youtube.com/watch?v=sOGtknSLLpU&t=604s) 成本核算与 Token 可观测性：杜绝失控的账单灾难针对近期业界对按量计费（Usage-based billing）成本暴涨的焦虑（例如 Ed Zitron 在大会演讲中提出“没人愿意为模型的真实运行成本买单”的论点），主持人询问 Docusign 如何在保证盈利与预算可预测性的同时推进 AI 业务。 Tabrez Mohammed 表示，在大公司内部推进新功能时，绝对要避免发生因推理开销失控而演变成“拖垮业务的灾难性发布”（Business-ending launch），更不能把不可控的成本直接甩给客户。为此，Docusign 建立了极其严苛的 Token 监控体系： - 细致审计每一个 Tool Call 以及每一次完整会话的 Input / Output Token 消耗。 - 遵循传统软件工程的性能监控逻辑：正如优化 Web 性能必须先埋点监控首屏渲染时间（First Time to Pixel）一样，唯有全面量化和定位 Token 瓶颈，才能精准推行降本优化。 ### [11:09](https://www.youtube.com/watch?v=sOGtknSLLpU&t=669s) 成本优化、DSPy 与自动提示词优化在构建面向外部客户的 Agent 平台时，监控模型性能与 Token 消耗支出固然是基本功，但最大的实际收益往往来自于全方位的成本优化。DocuSign 遵循的一条核心原则是：“只要小模型能够胜任，就绝不使用更大的模型”。团队重度依赖并推崇开源框架 DSPy。Tabrez 打趣道，如果你现在还不知道 DSPy 是什么，“你很可能会先自己手写出一个劣质版（crappy version）的 DSPy，随后在了解它之后最终还是选择切换过去”。 DSPy 为团队提供了更严密的迭代优化闭环。通过结合自动提示词优化技术与评估算法（如 GEPA 等机制），团队优化了底层各个 Tool 的独立实现。仅仅依靠这些提示词与工具调用的工程优化，就在绝大多数场景下将运行成本大幅削减了 50% 以上。 - **小模型优先原则**：按需匹配模型尺寸，避免大模型过度调用造成的算力与成本浪费。 - **DSPy 框架驱动**：利用程序化优化代替人工反复试错试调 Prompt，提升工具模块的执行效率与稳定性。 - **显著的降本成效**：通过针对工具调用的自动化提示词调优，多数业务场景实现了超 50% 的成本降幅。 --- ### [12:13](https://www.youtube.com/watch?v=sOGtknSLLpU&t=733s) 缓解大模型幻觉：预编译提取、轨迹约束与渐进式可验证性大模型在处理法律与合同类专业文档时，幻觉是不可忽视的致命隐患。如果只是将大量法律文档直接丢给市面上的开箱即用型 LLM，一两次回答看似合格，但在成千上万份海量文档的高并发调用下，系统必然会“脱轨失控（go off the rails）”。DocuSign 并非完全排斥大模型，而是拒绝让 LLM 去做“单次直出（single-shot）”的黑盒猜测，而是结合传统机器学习与受限的大模型任务进行结构化分工。 ```mermaid flowchart TD accTitle: DocuSign 文档信息提取与可验证分析流程 subgraph S1["第一阶段：数据预处理与约束提取"] A["法律/合同文档输入"] --> B["传统 ML / 受约束 LLM 提取"] B --> C["预编译与已校验的结构化数据"] end subgraph S2["第二阶段：带护栏的 Agent 分析"] C --> D["轨迹数据约束 (Trajectory Guardrails)"] D --> E["LLM 深度分析与自我校验 (Self-Verification)"] end subgraph S3["第三阶段：透明交付与人工复核"] E --> F["生成引用标注 (Citations) 与中间工件"] F --> G["客户渐进式自省与校验 (Progressive Introspection)"] end ``` 系统首先通过严格的系统预提取并编译信息，让后续的 Agent 依赖已经过验证的轨迹数据，确保推理过程始终在护栏之内。在输出分析时，模型会强制生成详细引用（citations）并进行自反思校验。此外，团队正开展前沿探索（类似 LLM Wiki 的机制），使模型在执行长达 30 分钟的复杂研报分析时，能够持续输出中间工件。这让客户可以“渐进式自省（progressively introspect）”并核验 Agent 的工作脉络，而无需面对最终生成的百页报告去费力逆推结论从何而来。 - **拒绝 Single-shot 黑盒直出**：结合传统 ML 算法对关键条款与元数据进行预处理与结构化校验，再交由 LLM 处理。 - **轨迹数据与引用溯源**：在推理链条中嵌入严格约束，要求大模型在提取和分析过程中自我验证并标注来源出处。 - **中间工件可视化**：让长时间运行的 Agent 暴露中间执行步骤与工件，赋予企业用户充分的审计与核验信心。 --- ### [14:37](https://www.youtube.com/watch?v=sOGtknSLLpU&t=877s) 威胁模型与安全护栏：权限收敛、用户级令牌与人机协同（HITL）正值拉斯维加斯同时举办 Black Hat 安全峰会，主持人谈及 Agent 的威胁模型与红队测试。针对安全风险，DocuSign 在推进从 0 到 1 的敏捷落地过程中采取了极具针对性的收敛策略：在一开始直接剥离高危权限，严格限制 Agent Harness 的能力边界，初版完全禁止 Agent 自动编写或执行代码，从而直接消除了一个巨大的潜在威胁攻击面。 ```mermaid flowchart LR accTitle: 认证隔离与权限爆炸半径控制 User["终端用户 (User)"] -->|"交互发起"| Agent["DocuSign AI Agent"] Agent -->|"传递 User Token (非 System Token)"| Downstream["下游业务系统 / API"] Downstream -->|"权限边界"| Scope["严格限制于当前用户的自身权限"] ``` 在系统认证与数据隔离上，Agent 与下游工具交互时严格采用当前用户的 User Token，而非高权限的 System Token。这种设计杜绝了“混淆代理（confused deputy）”或越权提权漏洞，防止攻击者利用 Agent 跨租户窃取其他账户数据，将潜在故障的“爆炸半径（blast radius）”严格限制在用户自身权限之内。对于提示词注入（Prompt Injection）以及合同签署等不可逆的高危变更，平台设置了硬性的人机协同（Human-in-the-Loop, HITL）护栏，严禁 Agent 在没有人工明确确认的情况下擅自代表客户签署法律协议。 - **初始能力严格裁剪**：初版禁止代码编写/执行功能，主动缩窄安全攻击面。 - **用户级令牌隔离**：调用下游服务全流程使用 User Token，消除 Confused Deputy 提权风险并收敛爆炸半径。 - **高危操作强制 HITL**：对协议签署、资产变更等法律与业务敏感操作设立拦截机制，必须由人工介入审批。 --- ### [16:35](https://www.youtube.com/watch?v=sOGtknSLLpU&t=995s) 客户采用情况与未来路线图：从交互式向全自主 Agent 演进谈及商业化与客户反馈，Tabrez 透露，智能协议管理（DocuSign IAM）已成为 DocuSign 历史上增长最快的产品线，而在其全球用户大会（Momentum）上发布的 AI Agent 功能收获了最热烈的反响。在产品上线数天内，就有客户将原本耗时数天甚至数周的手动流程迁移至 Agent 平台，仅需一至两个小时的调优，即可将流程处理时间压缩至数分钟。目前客户正迅速从最初的对话交互式 Agent，转向由 DocuSign 工作流服务自动触发的事件驱动型 Agent。面向未来，DocuSign 配备了专属的科学研究团队进行前沿探索，并计划在顶级学术会议上发表论文。团队接下来的研发重点主要涵盖两大维度：一是构建面向非形式化验证（Non-formal verification）场景下的大模型输出可信验证体系；二是伴随客户信任度的建立，通过自研底层平台能力以及与更广泛的第三方生态伙伴合作，推动系统从半自主（Semi-autonomous）向全自主（Fully autonomous）Agent 演进。 - **产品增长强劲**：IAM 创下公司历史最快增长纪录，客户在数小时调试后便能将原本耗费数周的业务流压缩至数分钟级自动化。 - **工作流深度集成**：客户使用模式正从即时单次交互拓展至由 DocuSign 工作流引擎自动触发。 - **前沿科研与生态拓展**：推进大模型内容验证机制的前沿研究与论文发表，携手第三方生态构建面向复杂场景的自主 Agent 解决方案。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=sOGtknSLLpU&t=4s) 企业在积累了数以亿计的合同后，最大的痛点在于如何在签署后提取其中的商业洞察并加以管理。 — Tabrez Mohammed > [04:31](https://www.youtube.com/watch?v=sOGtknSLLpU&t=271s) 不仅在特定任务上达到甚至超越了云巨头前沿大模型的表现，而且推理成本仅为其二十分之一（1/20th or less）。 — Tabrez Mohammed > [06:22](https://www.youtube.com/watch?v=sOGtknSLLpU&t=382s) 极力借力成熟生态，拒绝过早重复造轮子。 — Tabrez Mohammed > [08:20](https://www.youtube.com/watch?v=sOGtknSLLpU&t=500s) 外部编排能提供更强的确定性保障，更符合企业客户对稳定性的严苛预期。 — Tabrez Mohammed > [10:04](https://www.youtube.com/watch?v=sOGtknSLLpU&t=604s) 绝对要避免发生因推理开销失控而演变成拖垮业务的灾难性发布（Business-ending launch），更不能把不可控的成本直接甩给客户。 — Tabrez Mohammed > [11:09](https://www.youtube.com/watch?v=sOGtknSLLpU&t=669s) 只要小模型能够胜任，就绝不使用更大的模型。 — Tabrez Mohammed > [11:09](https://www.youtube.com/watch?v=sOGtknSLLpU&t=669s) 如果你现在还不知道 DSPy 是什么，你很可能会先自己手写出一个劣质版的 DSPy，随后在了解它之后最终还是选择切换过去。 — Tabrez Mohammed > [12:13](https://www.youtube.com/watch?v=sOGtknSLLpU&t=733s) 如果只是将大量法律文档直接丢给市面上的开箱即用型 LLM，一两次回答看似合格，但在成千上万份海量文档的高并发调用下，系统必然会脱轨失控（go off the rails）。 — Tabrez Mohammed > [14:37](https://www.youtube.com/watch?v=sOGtknSLLpU&t=877s) Agent 与下游工具交互时严格采用当前用户的 User Token，而非高权限的 System Token。 — Tabrez Mohammed > [14:37](https://www.youtube.com/watch?v=sOGtknSLLpU&t=877s) 严禁 Agent 在没有人工明确确认的情况下擅自代表客户签署法律协议。 — Tabrez Mohammed > [16:35](https://www.youtube.com/watch?v=sOGtknSLLpU&t=995s) 智能协议管理（DocuSign IAM）已成为 DocuSign 历史上增长最快的产品线。 — Tabrez Mohammed ## Mentioned Docusign, Tabrez Mohammed, Rob Pegoraro, PCMag, Fast Company, Ai4, Docusign IAM (Intelligent Agreement Management), Docusign Momentum, LangGraph, DSPy, GEPA, Ed Zitron, Black Hat ## Video description Tabrez Mohammed - VP of AI, Docusign Rob Pegoraro - Contributor, PCMag / Fast Company Most AI agent talks focus on building tools for employees—but what changes when your users are your customers? In this talk, I will explore the unique challenges of building an externally facing agent platform at Docusign, including the strategic decisions and architectural tradeoffs required to balance long-term platform growth with the need to ship customer value quickly. I will share practical patterns for designing a secure, multi-tenant, extensible foundation that enables teams to deliver new agent capabilities rapidly without sacrificing scalability or maintainability. If you're building AI into a SaaS product, you'll leave with a framework for making platform decisions that help you move fast today while setting yourself up for success tomorrow. [Source](https://www.youtube.com/watch?v=sOGtknSLLpU)

Source quotes

[0:04] Tabrez Mohammed: 企业在积累了数以亿计的合同后，最大的痛点在于如何在签署后提取其中的商业洞察并加以管理。 [4:31] Tabrez Mohammed: 不仅在特定任务上达到甚至超越了云巨头前沿大模型的表现，而且推理成本仅为其二十分之一（1/20th or less）。 [6:22] Tabrez Mohammed: 极力借力成熟生态，拒绝过早重复造轮子。 [8:20] Tabrez Mohammed: 外部编排能提供更强的确定性保障，更符合企业客户对稳定性的严苛预期。 [10:04] Tabrez Mohammed: 绝对要避免发生因推理开销失控而演变成拖垮业务的灾难性发布（Business-ending launch），更不能把不可控的成本直接甩给客户。 [11:09] Tabrez Mohammed: 只要小模型能够胜任，就绝不使用更大的模型。 [11:09] Tabrez Mohammed: 如果你现在还不知道 DSPy 是什么，你很可能会先自己手写出一个劣质版的 DSPy，随后在了解它之后最终还是选择切换过去。 [12:13] Tabrez Mohammed: 如果只是将大量法律文档直接丢给市面上的开箱即用型 LLM，一两次回答看似合格，但在成千上万份海量文档的高并发调用下，系统必然会脱轨失控（go off the rails）。 [14:37] Tabrez Mohammed: Agent 与下游工具交互时严格采用当前用户的 User Token，而非高权限的 System Token。 [14:37] Tabrez Mohammed: 严禁 Agent 在没有人工明确确认的情况下擅自代表客户签署法律协议。 [16:35] Tabrez Mohammed: 智能协议管理（DocuSign IAM）已成为 DocuSign 历史上增长最快的产品线。

Source quotes

[0:04] Rob Pegoraro: please don't do that if there are any lawyers in the room. [0:04] Tabrez Mohammed: 2x, 5x, 10x better, more accurate. [4:25] Tabrez Mohammed: 1/20th or less of cost. [12:13] Tabrez Mohammed: go figure it out and come back with the answer,

Original transcript

[0:04] With me is Deb Reese Muhammad, VP of AI at DocuSign. And so I have to say that there's [0:10] this the history of AI being applied to legal documents, and it's not great. There's lots of [0:15] lawyers who feel compelled to have the AI do the research, and then it makes up cases that never [0:20] existed, and judges just don't find the humor in that. So please don't do that if there are [0:24] any lawyers in the room. Um so what led you all to decide we are going to put AI to work here? [0:31] Um yeah, I I think it's good to be here. Um so DocuSign has been in the agreement space for over [0:40] 20 years. Um the the focus has primarily been on somebody creating an agreement, the e-signature is [0:48] you know, over a billion users that have used that side of it. Um but the focus once you accrue all [0:55] these agreements is you need to do something with them. Like they're post-signing, there's still [0:59] interesting things, insights that businesses want to glean from them. And we looked in [1:07] these agreements that businesses are trying to manage and figured out like how can we use [1:12] AI, both LLMs and more classical approaches, to extract information from them so customers [1:19] can make sense of these hundreds of millions or billions of agreements that they've accrued. And [1:24] so for for us the first approach was just looking at our existing feature set and making them [1:30] 2x, 5x, 10x better, more accurate, etc. with with the introduction of LLMs. And in the past 2 years, [1:39] with agent platforms getting more popular, we're looking at features that we couldn't build before [1:45] without agent platforms as as where we want to invest going forward. Okay, so tell me what what [1:51] features is AI handling? What what is it cuz I've met yeah, I I look at a DocuSign. I of [1:56] actually do read the text. Hopefully I understand it and sign it and that's the end of it. So, [2:01] what what what is AI doing adding to that? So, for On the creating and the the [2:08] signing it. Yeah, so so when customers go to to sign agreements, um we we have [2:14] I think a pretty obvious experience there. You may have a one-page or hundred-page long [2:19] contract. So, you want to interact with that contract, figure out what's in it, [2:23] um give you confidence that you want to sign it. Some of the features we're looking for. So, [2:27] so those were things that I think people knew could have been done for the past couple of years, [2:32] but looking forward, if you are signing this contract, you're you're in an enterprise [2:37] and you want confidence that this is the right contract for you to sign. Well, now you suddenly [2:41] want to compare that contract with the hundreds or thousands of contracts that you already have. [2:45] You know, how does that shift your risk? What obligations are in it? These you know, [2:50] deeply analyzing the clauses that are in it and and surfacing that information in a way that's [2:55] easier for people to consume. You just couldn't really do that at any sort of scale or accuracy [3:00] uh before LLMs. So, that that's happening at the commit side and then post-signing, [3:05] like you're let's say you're on the the other side of that contract. You want to know, [3:11] is this a contract that I need to keep monitoring? Uh there's some obligations that's coming up, you [3:17] know, in 30 days or 90 days it's going to be uh up for renewal. And so, you want to compare that [3:22] agreement perhaps with uh some of the operations that had happened or actions that had happened [3:28] with uh maybe that particular vendor and say, did that vendor actually fulfill that contract? And if [3:33] they did or they didn't, that can feed into, you know, that all this information could feed into [3:38] your next renewal phase of making sure that, you know, you're you're optimizing what you're signing [3:44] or the contract that you're generating for the next iteration. So, on the one hand, it can tell [3:48] me what exactly I actually signed up for, which That's the dream. That that's certainly the dream. [3:54] Um and I think we're we're getting uh pretty good at that. Uh we are a leader in in this space. We [4:00] have the the you know um the the most contracts and and we've been in this space the longest and [4:06] uh we the the kind of frontier capabilities that were already released in the past 2 years since we [4:11] introduced uh IAM uh in uh intelligent agreement management um you know as as disrupted kind of the [4:19] way that people think about uh contracts after they sign them and and helping manage them in [4:24] in their business processes. And the management part of it. So, what did you train your models on? [4:31] So, uh we we have uh like I mentioned lots and lots of data and your your inference the quality [4:39] of your inference is only as good as the data that you have. I think everybody who's dabbled in ML uh [4:44] knows that. But um these contracts are pretty much the most sensitive documents that a business has. [4:50] Um and so we don't have our employees just looking at other people's contracts uh whenever they feel [4:56] like it. So, we have I I would say a a you know a pretty novel approach when it comes to how to [5:02] anonymize the the data that's in that contracts while not crushing the information so it's useful [5:09] for downstream processes. So, we have you know an anonymization or de-identification process [5:15] uh that we put all the contracts through. It's multi-layered. We monitor to make sure when we're [5:20] taking stratified samples of the anonym- of the Adoc's post anonymization that we're not able to [5:25] re-identify the party or any PII. So, once we have that kind of the set that we feel like it's safe, [5:32] then we feed that into, you know, various kind of post-training or uh fine-tuning techniques to to [5:40] uh improve our models. And and the the results kind of speak for themselves. We put out [5:45] blog posts on this for people who are interested. You can do your searches uh for for them and and [5:49] find links to where we documented these processes, but we're able to achieve or exceed frontier level [5:55] performance compared with, you know, hyperscaler LLMs at 1/20th or less of cost. And the reason [6:03] that's significant is it means that we can do that much more with the contracts on our platform [6:08] than users would be able to do themselves if they were rolling solutions themselves on top of, you [6:15] know, chat GPT or Claude or what have you. And how long has it taken did it take to ship the initial [6:22] IM products? So, IM was first launched 2 years ago, but we we started on the agent [6:30] platform actually at the beginning of this year. So, we went from pretty much nothing [6:37] to a production-ready agent platform that was announced at Momentum in in May. So, [6:45] you know, 4 to 6 months end to end to launch our first agent platform that that's customer-facing, [6:52] which I think is a pretty remarkable feat. I've worked at startups before this, [6:58] and that's normal by startup standards. sleep at some point during that uh There There was one or [7:04] two folks who I I think they had a an upset in their work-life balance, but but most of the folks [7:10] No, it was I think the techniques that we leaned into is um is, you know, our our IP, [7:17] if we think about it for Docusign, our features and capabilities that we can put on top of the [7:22] technology we're building. It's not like the agent frameworks and the agent harnesses themselves. [7:28] So, a strategy for us was to use as much as was already available until we proved to ourselves [7:34] that we needed to reinvent certain wheels. So, like the agent harness, off-the-shelf models, like [7:40] all these things, we really just stitched it all together within a matter of weeks to see just what [7:47] we could get on top of our existing data and our existing APIs. And once we started to see places [7:52] that there was like information loss or accuracy issues or it was too expensive, then we started [7:59] to go and you know, ablate those layers away from our infrastructure and replace them with something [8:03] else. But really from scratch within a couple of weeks, we had a functional system end-to-end and [8:10] and it really came down to making sure we're using as much of, you know, as the agentic ecosystem [8:16] that was already out there and then replacing it whenever we what are the the components you're [8:20] using right now? Your your harness and your model? Yeah, so I feel like there's so much hype around [8:26] agent harnesses and and and and people are kind of just chasing the latest hype wave. So we bounced [8:34] around a little bit for the first couple of weeks of the project, a couple of POCs just to figure [8:38] out which one we wanted to commit to for the next three to six months. So we're currently [8:44] using LangGraph internally, but in kind of a non-traditional way that that fits our use cases [8:50] and the kind of cost optimizations that we that we wanted. The big reason to use LangGraph in this [8:56] you'll see like hundreds of tweets and stuff going back and forth, people debating this is does the [9:01] orchestration happen within the agent or should you have orchestration outside the agent? I don't [9:06] know if this is a permanent way that we view the space, but we prefer orchestration on the outside. [9:12] We we think that gives us better guarantees and customer expectations to do so. And and [9:16] LangGraph has is strongly rooted in orchestration semantics. So that that's the decision that [9:21] we made. We could change it in three months from now. That's another thing that we did is [9:25] every kind of component we picked, we made it so it we could swap it out with something else [9:31] if if the need arose. But but for now our harness um is is LangGraph, and then the model itself, [9:41] um, we we we uh we swap between them based on the task, uh, which I think is a very common practice. [9:48] Use a bigger model for your core reasoning agents, and then as you optimize that your tool calls, you [9:53] start to use different models, uh, that do maybe aren't so big and powerful and not necessary. [9:59] So, you mentioned your cost before. You were saying 1/20th. And, you know, there there there [10:04] is that anxiety over, you know, the switch to usage-based billing. If if any of you saw Ed [10:09] Zitron's talk Tuesday, which may or may not get him invited back here, he's he his thesis is, you [10:15] know, that no one wants to pay what these models actually cost to run. Yeah. How are you making [10:19] sure that your costs are at a level that lets you make money and predictable? Yeah, so that that [10:25] that's always a fun conversation working inside of a big company. It's like, "Hey, all these features [10:29] are awesome, but at the same time you don't want like a business-ending launch with, you know, [10:34] your your expenditure just running away or or even worse like offloading that to to your customers." [10:39] So, uh we we scrutinize pretty much everything that has to do with input and output tokens. We [10:44] we analyze it, um, per tool call. We analyze it, um, you know, for end-to-end for a particular chat [10:50] session, uh, just to to understand like you can't fix a problem that you haven't identified. is key. [10:57] This is no different for anybody who's been doing software engineering for extended period of time. [11:01] Like, you can't fix web performance issues unless you're instrumenting, you know, [11:04] how your website is loading, first time to pixel, and all that stuff. So, we're doing the same thing [11:09] for for analyzing the performance of the models and and the expenditures or the token spend, [11:14] um, that that we're using, but I I think that's obvious that you should measure it. [11:19] We're we're finding the biggest benefit in in cost optimizations, um, because there's just so [11:24] many different things you can do. I mentioned a few of them is don't use a bigger model if a [11:28] smaller one will do. Uh, we're big fans of Deep Spy, uh, for folks who uh who know about it, um, [11:37] you likely are also huge fans about it. If you don't know what Dspy is, [11:41] you will probably end up building a crappy version of Dspy and then learn about it and [11:45] end up switching over to it. But having Dspy was very useful for us because it helps us [11:51] get into a better loop for how we iterate and improve our prompts. And we found just by using, [11:56] you know, kind of basic, uh, automatic prompt optimization techniques, um, stuff like Jeppa, [12:02] where we're able to reduce the the the cost dramatically by like over 50% in most cases, [12:08] uh, by by optimizing the the individual implementations of the tools themselves. [12:13] And since I mentioned the the hallucination problem at the start, uh, how do you make [12:18] sure that in this case your model is not outputting things that that aren't there? Yeah, [12:24] so this this is kind of the perennial problem with with LLMs and, you know, when when somebody takes [12:30] a bunch of legal documents and just throws it in an off-the-shelf LLM and the answers seem okay, [12:35] but once you start doing that hundreds and thousands of times with even more documents, [12:39] the uh, things start to go off the rails. So, um, we we we're not like anti-LLM by any stretch [12:45] of the imagination for solving this problem, but we have a bunch of other products, some that use [12:50] classical ML techniques, some of them that use LLMs but in more constrained use cases, where we [12:56] are effectively extracting and pre-compiling information that we like to rely on first, [13:01] um, because we have a lot of guardrails when it comes to extracting information from documents [13:06] that we don't throw it as like a single-shot problem. It's like, "Hey, [13:10] LLM, here's your prompt. Just go figure it out and come back with the answer." Uh, we we [13:14] we guide the system to to rely on information that has already been vetted by other systems, [13:20] um, and the this sort of trajectory data helps keep the LLM on on guardrails. [13:25] We also heavily use citations and and have the LLM kind of self-verify as it's um [13:34] extracting information and and doing its analysis. Uh so so the the these are pretty pretty key. [13:40] And then going forward when you see techniques like LLM Wiki, the these you know, really anything [13:47] where the LLM can do work, it generate these intermediate artifacts that it can self-introspect [13:53] or even better allow the customer to introspect what an LLM has done. This is a place that we're [14:00] actually doing what I would call some frontier research in is how can you have an LLM do a bunch [14:05] of work? It may have done like 30 minutes of work for a research task for for a particular customer, [14:12] but how can you make sure that the the work that it did is is something that the customer [14:17] can verify as opposed to just looking at the final answer and trying to look in reverse. It's like, [14:21] well, how did it come up with that? So you're really giving the customer a way to see what the [14:25] LLM did progressively so that when the result pops [clears throat] out at the end in 30 minutes, a [14:30] 100-page report or what have you, we the customer has the confidence that we did the right thing. [14:37] And since this week Black Hat is also happening at the other end of the strip, I've been bouncing [14:41] from one conference to the other, which is not great for my sleep, but whatever. What what's the [14:46] threat model look like for something like this? What's so do you do you need red teaming to make [14:50] sure that you know, this this agent is not running a mock and not Yeah, so so we we in the name of [14:57] like shipping fast and going from zero to one as quickly as possible, we did make some compromises, [15:02] which I think were totally reasonable for for trying to ship our first version. So one of the [15:07] things that we did I feel like doesn't get spoken about enough is we we didn't have our harness just [15:14] capable of doing anything everywhere out of the gate. So we didn't let it write code initially. [15:20] This is changing actively right now in development, but that removed a huge threat [15:25] vector for us of of like the agent being really constrained with what it was supposed to do. Uh, [15:30] we also focused heavily where our customers are today, which are interactive use cases. And so, [15:34] the the internally when agents and tools like they're authenticating to downstream systems, [15:39] they're all using user tokens. So, like your blast radius of where something could go off [15:44] the rails is really just constrained to what the user could do themselves within the environment. [15:49] They're not able to you know, exploit like confused deputy or these kinds of, you know, [15:54] unexpected escalation of privileges through system tokens uh, to start, you know, extracting [15:59] information from user A but at user B's account. So, so that that's a, you know, these are things [16:05] that helped us really limit the blast radius, but we're doing kind of the traditional stuff like [16:09] checking to make sure that if somebody with prompt injection attacks and things like that, that you [16:14] don't have an agent start automatically like signing agreements on a customer's behalf. And [16:18] then really anything that you taste a customer's environment, we have a lot of strict guardrails in [16:23] place to require HITL so that these really risky events that can happen in the contracting space, [16:29] uh, an agent can't uh, uh, just start doing them without a customer being involved. [16:35] And what have you learned from customer adoption? Like is this How it's How is feedback [16:41] changed, uh, led you to add, delete features, whatever? Yeah, so it it's really in two [16:47] two spots. So, I am already as of the fastest growing product in DocuSign's history, [16:52] um, and then agents people were very excited about it at momenta. There are other products [16:59] my my team works on, but the agents is probably the one that's receiving the most fanfare. Uh, [17:04] we're seeing customers within days of us launching taking processes that were, um, [17:10] you know, taking them manually maybe days or weeks to to work on and they're able to work on [17:15] an agents on our platform within maybe an hour or two of tuning and optimizing that they're they're [17:22] able to realize those savings. You know, in this this window of minutes to two hours and it's very [17:26] very early days in kind of the adoption cycle. So so we we see it quite positive and people are [17:33] rapidly moving from these interactive agent use cases to now having them triggered off of the [17:39] different workflow services that we have within DocuSign. And so when we when we move past early [17:45] days, what's what's next? What's the road map look like? So we have a dedicated science team. [17:50] That might be a surprise to people, but we have a dedicated science team that is doing Frontier [17:54] research. We're going to to the to the level that we're going to start publishing in a upcoming [17:58] conferences. So keep an eye out for that. A big space for us is as as you alluded to the beginning [18:06] the danger of llms you know, generating content and not really anybody knowing if these things are [18:11] accurate. So so we're also really interested in how can you verification of the outputs of an llm [18:17] when it's not a traditional formal verification process like in the mathematics and coding space [18:22] that you see today and then just going deeper into use cases where customers in their early [18:27] adoption cycle. They are doing interactive stuff, but as they get more comfortable with agents. They [18:32] want to move more to semi-autonomous and fully autonomous agents and so we're actively working [18:38] in things in our platform and partnering with third parties in the in the broader [18:42] ecosystem to make these autonomous agents possible. Okay, so a lot to look forward to. [18:47] If you all have any questions, we'll be around for a little bit. Come talk to us. We won't make [18:50] you sign a contract first. We won't. Yes. All right. Thank you everyone. Thanks everybody. [18:53] [applause]