Etsy Deploys 70 Risk Tests on Seller AI Agents to Accelerate Rollouts

Infrastructure for User Trust With Etsy — Ai4

Ai4

Etsy argues that systematic trust infrastructure accelerates AI deployment rather than slowing it down, because repairing eroded user confidence costs far more time than maintaining proactive safeguards.

AI-assisted summary of Infrastructure for User Trust With Etsy — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 18, 2026, this presentation features Etsy's AI Staff Product Manager Tomer Sabo and Engineering Manager Beth Sebian. According to the page description, the session details why user trust must be treated as a product and how investing in dedicated technical and organizational infrastructure enables teams to move faster and build AI products with confidence.

## TL;DR Tomer Sabo and Beth Sebian outline Etsy's comprehensive framework for safeguarding user trust while scaling agentic AI tools for sellers. They argue that trust is an upside growth asset rather than just downside risk mitigation, detailing a three-layered approach across technical evaluations (identifying 70 failure modes across 12 categories), organizational controls (a 4-phase gated release process scaling from alpha to general rollout), and cultural practices (a responsible AI curriculum developing moral attention). Ultimately, they demonstrate that institutionalizing evidence-based trust guardrails accelerates shipping velocity by preventing catastrophic customer relationship breakdowns.

## Atmosphere & style The talk maintains a pragmatic, thoughtful, and collaborative tone, reflecting an earnest engineering and product partnership. Beginning with a relatable audience check-in about the uncertainty of scaling AI, the speakers alternate seamlessly to deliver a grounded, hype-free masterclass on operational discipline. It feels transparent, instructional, and deeply principled, balancing concrete risk-scoring matrices with nuanced reflections on professional judgment and ethics.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=jekXJWBLI-I&t=4s) User trust in agentic AI can erode subtly over a long tail of edge cases even when the model delivers technically correct recommendations, such as an ad-optimizing agent repeatedly asking a seller to increase spend. - [02:52](https://www.youtube.com/watch?v=jekXJWBLI-I&t=172s) User trust is a growth asset rather than just a downside risk; higher trust unlocks deeper user engagement and allows users to delegate broader and more complex workflows to autonomous agents. - [02:52](https://www.youtube.com/watch?v=jekXJWBLI-I&t=172s) Etsy operationalizes trust across three interconnected layers: a technical infrastructure layer, an organizational release layer, and a cultural curriculum layer. - [05:33](https://www.youtube.com/watch?v=jekXJWBLI-I&t=333s) The broader AI industry often conflates raw model capability with trustworthiness, ignoring key dimensions like honesty, consistency, boundary adherence, and equitable impact. - [05:33](https://www.youtube.com/watch?v=jekXJWBLI-I&t=333s) Evaluating agent trustworthiness requires an exhaustive risk assessment; Etsy mapped 70 distinct failure modes across 12 categories, scoring each by frequency, severity, reach, and financial impact. - [05:33](https://www.youtube.com/watch?v=jekXJWBLI-I&t=333s) Trustworthiness evaluations cannot be outsourced to third-party vendors because outside entities lack the deep context of platform-specific user relationships and proprietary risk profiles. - [11:15](https://www.youtube.com/watch?v=jekXJWBLI-I&t=675s) Etsy manages risk exposure using a four-phase gated release pipeline (Research Alpha, 1,000 users, 25,000 users, and General Release) with explicit exit criteria and risk guardrails at each gate. - [14:08](https://www.youtube.com/watch?v=jekXJWBLI-I&t=848s) Static compliance rules are insufficient for nuanced AI edge cases; Etsy developed a case-based ethics curriculum to build moral attention and decision-making judgment across product and engineering teams. - [14:08](https://www.youtube.com/watch?v=jekXJWBLI-I&t=848s) The cultural curriculum emphasizes recognizing failure modes like treating 'compliance as a ceiling,' designing team ceremonies for dissent, identifying unrepresented stakeholders, and exercising productive tact. - [18:45](https://www.youtube.com/watch?v=jekXJWBLI-I&t=1125s) Investing in comprehensive trust infrastructure acts as a speed enabler rather than an impediment, allowing teams to experiment ambitiously because repairing broken trust is far slower and more costly. - [19:56](https://www.youtube.com/watch?v=jekXJWBLI-I&t=1196s) Deploying customer-facing AI requires replacing intuition and gut checks with empirical validation, using dedicated infrastructure to enforce safeguards with systematic rigor.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=jekXJWBLI-I&t=4s) Introduction and the Nuance of User Trust in Agentic AI

Tomer Sabo, an AI Product Manager at Etsy, kicks off the presentation alongside his colleague Beth Sebian, an Engineering Manager at Etsy. Acknowledging that the audience has trekked all the way up five escalators to the end of the hall late in the day, Tomer opens with a quick temperature check of the room: he asks for a show of hands from anyone who feels confident that their organization's process for deciding when to scale an AI feature is truly good enough for their customers. When not a single hand goes up, he notes with a smile that the session will examine how Etsy approaches this exact problem through intentional trust infrastructure.

Tomer introduces the concept of user trust in agentic AI by walking through a concrete example from Etsy’s seller tooling. Etsy developed an autonomous agent designed to optimize seller advertising budgets and recommend strategic actions to drive sales. In a scenario where a seller is experiencing strong return on ad spend, the mathematically optimal recommendation is straightforward: invest more budget to generate higher revenue. However, if the agent repeatedly prompts the seller to increase their ad spend over and over again, the seller’s perception shifts. Even if the agent is performing its task flawlessly, the seller begins to question the underlying motivation: "Hey, wait a second. Does this agent actually look out for me? Does it care about what I'm doing? Or is it... really just looking out for Etsy and trying to drive more revenue for Etsy at my expense?"

This dynamic illustrates how trust can erode subtly along a long tail of edge cases, even when the model delivers the technically correct outcome. When trust degrades, the user stops taking recommendations or abandons the tool entirely, nullifying the agent's utility.

```mermaid flowchart LR accTitle: The Mechanics of Subtle Trust Erosion subgraph Behavior["Agent Behavior"] A["Agent optimizes ad budget"] --> B["Technically correct action:<br/>'Increase ad budget'"] end subgraph Perception["User Perception & Impact"] B --> C{"Repeated recommendations<br/>over time"} C --> D["Seller suspects conflict of interest<br/>('Etsy prioritizing its own revenue')"] D --> E["Seller ignores future recommendations<br/>and abandons agent"] end ```

---

[02:52](https://www.youtube.com/watch?v=jekXJWBLI-I&t=172s) Reframing Trust: From Downside Risk to Growth Asset

Tomer argues that user trust must be treated as a first-class consideration in AI product development rather than a mere defensive posture. Traditional approaches often treat trust like cybersecurity—a pure downside risk where failure leads to churn or reputational fallout. In reality, trust delivers immense upside: deepening user trust unlocks higher engagement, expands the surface area of potential workflows, and allows the platform to deliver more value. A seller who tentatively trusts an agent might accept an isolated ad suggestion, but a seller with high trust will proactively ask for strategic advice and delegate complex tasks across multiple domains.

To highlight the non-binary spectrum of trust, Tomer references an earlier conference presentation titled *How I Use 150 Agents to Run My Life*. While most people hesitate to hand over credit card numbers, social security details, or bank account authorization to autonomous agents, those power users do so because their trust threshold has expanded. Building and safeguarding user trust is a continuous, incremental endeavor that allows organizations to move users further along that adoption curve. To operationalize this, Etsy structures its trust approach across three core layers:

- **Technical Infrastructure Layer:** Systematic evaluations and guardrails focused on trustworthiness rather than raw capability. - **Organizational Layer:** Gated release frameworks that systematically control risk exposure as features scale. - **Cultural Layer:** A responsible AI curriculum that builds moral attention and decision-making judgment across the engineering and product organization.

---

[05:33](https://www.youtube.com/watch?v=jekXJWBLI-I&t=333s) The Technical Layer: Trustworthiness Evaluations and In-House Risk Modeling

Tomer highlights a quote from Robert C. Martin ("Uncle Bob"), author of *Clean Code* and co-author of the Agile Manifesto, to illustrate how the frontier of software engineering leverages agents. Martin shared that because he cannot read all the code his agents generate, his strategy is "to surround the agents with extreme constraints, all kinds of tests" so the generated output must run a gauntlet of automated validation before reaching production. Because software engineering represents the most mature frontier of agent adoption, product builders across all domains should adopt this rigorous testing mindset.

```mermaid flowchart TD accTitle: Etsy Risk Assessment and Evaluation Process subgraph Identification["1. Risk Identification"] A["Brainstorm with domain experts"] --> B["Group into 12 risk categories"] B --> C["Identify 70 failure modes"] end subgraph Scoring["2. Scoring & Mitigation"] C --> D["Score by frequency, severity,<br/>reach, and financial impact"] D --> E["Assign risk tier<br/>(Low, Medium, High, Catastrophic)"] E --> F["Deploy mitigation strategy<br/>(Banners vs. strict workflow controls)"] end subgraph Evaluation["3. Ongoing Measurement"] F --> G["Build dedicated evaluation per mitigation"] G --> H["Verify true frequency & mitigation efficacy in-house"] end ```

The broader AI industry tends to conflate capability with trustworthiness. Model release headlines consistently celebrate intelligence breakthroughs, such as solving novel math theorems, while ignoring essential trust dimensions: honesty, consistency, boundary adherence, and equitable impact across distinct user cohorts. Evaluating trustworthiness requires an exhaustive, structured risk assessment:

- **Risk Categorization and Mapping:** Etsy convened cross-functional domain experts to brainstorm potential user risks for their seller agent, identifying **12 core risk categories** containing **70 distinct failure modes**. - **Multi-Dimensional Scoring:** Each risk is evaluated on expected frequency, severity of harm, user reach, and potential financial impact, resulting in a categorized severity rating (low, medium, high, catastrophic). - **Proportional Mitigation:** Low-risk items can be mitigated with lightweight interventions such as descriptive banners or hedged terms-and-conditions language. Critical risks—such as cross-seller data leakage—require strict deterministic controls embedded directly into the agent's runtime workflow. - **Dedicated Mitigation Evaluations:** Every mitigation requires its own automated evaluation to verify two points: whether the baseline risk occurs in reality at the predicted rate, and whether the defensive controls are catching it effectively. - **In-House Ownership:** While third-party platforms and consultancies can help build agents or evaluate raw technical capability, Tomer emphasizes that user trust evaluations cannot be outsourced, as outside vendors lack deep context regarding platform-specific user relationships and proprietary risk profiles.

---

[11:15](https://www.youtube.com/watch?v=jekXJWBLI-I&t=675s) The Organizational Layer: Gated Releases and Risk-Mapped Scaling

Beth Sebian takes the stage to explain the organizational machinery that connects risk evaluation to feature rollout. As an engineering manager, her goal is to ensure team resources are directed toward delivering maximum customer impact while maintaining the delicate balance between moving quickly and safeguarding trust. Etsy operationalizes this through a gated release framework that matches risk maturity against progressively larger user cohorts.

Every release gate acts as an explicit checkpoint enforcing two foundational questions: "Does it deliver the value that we expect?" and "Are we mitigating the risk that we've wanted to tackle at that level with that population?"

```mermaid flowchart LR accTitle: Etsy Gated Scaling Pipeline subgraph Gate1["Phase 1"] A["Research Alpha<br/>(Small cohort, rich feedback)"] end subgraph Gate2["Phase 2"] B["1,000 Users<br/>(Initial scaled testing)"] end subgraph Gate3["Phase 3"] C["25,000 Users<br/>(Broad validation)"] end subgraph Gate4["Phase 4"] D["General Release<br/>(Full platform rollout)"] end Gate1 --> Gate2 Gate2 --> Gate3 Gate3 --> Gate4 ```

To advance from one stage to the next, teams must define explicit release artifacts for each threshold: - **Hypothesis:** A clearly articulated theory of user impact. - **Open Questions:** The specific unknowns regarding user value or friction that need answering in the wild. - **Exit Criteria:** The empirical bar of engagement and performance necessary to justify cohort expansion. - **Risk Guardrails:** Specialized thresholds monitoring error rates, failure modes, and negative user signals.

Etsy structured the rollout of its recent AI features through four defined phases: beginning with an intimate research alpha experiment, scaling up to **1,000 users**, expanding to **25,000 users**, and finally transitioning to **general release**.

---

[14:08](https://www.youtube.com/watch?v=jekXJWBLI-I&t=848s) The Cultural Layer: Cultivating Moral Attention and Decision-Making Judgment

Beth explains that technical checks and release gates are insufficient without a cultural foundation. In agentic development, consequential decisions are made daily at the ground level by engineers and product managers writing prompts, setting parameters, and creating evaluation sets. Because edge cases are nuanced and impossible to anticipate fully with rigid rules, Etsy established a dedicated curriculum aimed at developing "moral attention as a team practice."

Relying purely on static rules causes team members to abdicate personal responsibility by delegating critical thinking to a checklist. Instead, Etsy draws inspiration from case-based ethics training used in fields like medicine, law, and the military. These interactive, facilitated group discussions help practitioners build judgment under conditions of uncertainty.

```mermaid flowchart TD accTitle: Core Capabilities of the Trust Curriculum Curriculum["Cultural Foundation:<br/>Moral Attention as a Team Practice"] Curriculum --> S1["1. Identifying Unrepresented Stakeholders<br/>(Exercising imagination to detect secondary impact)"] Curriculum --> S2["2. Failure Mode Recognition<br/>(Moving past 'compliance as a ceiling')"] Curriculum --> S3["3. Designing for Dissent<br/>(Structured premortems and sanctioned pausing)"] Curriculum --> S4["4. Productive Tact<br/>(Offering constructive solutions alongside risk flags)"] ```

The curriculum focuses on developing four core capabilities: - **Identifying Unrepresented Stakeholders:** Cultivating the imagination to consider how agent behaviors impact indirect or marginalized user groups who may not be in the room. - **Failure Mode Recognition:** Recognizing anti-patterns, specifically the trap of treating "compliance as a ceiling." Beth points out that because regulations and laws lag behind generative AI, anchoring purely on legal compliance creates a false sense of security and leaves massive blind spots for harm. - **Designing for Dissent:** Instituting formal team ceremonies such as release premortems that actively encourage and sanction team members to raise concerns and pause rollouts when necessary. - **Productive Tact:** Training team members to avoid being obstructionists who simply shut down conversations by dropping problems on the table. Instead, practitioners are encouraged to act as high-context problem solvers who bring actionable mitigation paths to hard technical calls.

---

[18:45](https://www.youtube.com/watch?v=jekXJWBLI-I&t=1125s) Infrastructure as a Speed Enabler: Shipping Faster by Preserving Trust

Beth concludes by reinforcing that investing in technical, organizational, and cultural trust infrastructure is not an impediment to velocity—it is the primary catalyst for sustainable speed. When an organization builds these guardrails from day one, engineers and product managers can experiment more ambitiously and release features with genuine confidence.

When mistakes or unexpected agent behaviors inevitably emerge in production, the team relies on established rails to catch anomalies early, fortify mitigations, and move forward rapidly without causing catastrophic harm. Discovering vulnerabilities before they damage the customer relationship is essential, as "repairing trust after you've broken it is the slowest" and most expensive process in product development.

[19:56](https://www.youtube.com/watch?v=jekXJWBLI-I&t=1196s) Grounding Safeguards in Evidence: Rigor and Scale for AI Products

Wrapping up the presentation, the speaker returns to the prompt posed to the audience regarding confidence in guardrails: "how confident are you that your safeguards are strong enough?" The core takeaway being elevated is that relying merely on intuition is insufficient in AI product development. Instead, "using evidence to actually check your gut is going to save you," pointing to the vital role empirical validation plays when deploying sensitive customer-facing systems.

All of the technical and organizational tooling presented serves as a concrete mechanism to accomplish that evidence-based verification at scale. Without dedicated infrastructure, teams are left guessing; with it, they can enforce safeguards with a level of rigor that genuinely matches the inherent complexity of modern generative AI products. With that final emphasis on institutionalizing trust through structured measurement, the speaker concludes the session: "I think I'll leave it there. Thanks everyone."

- **The Role of Evidence**: Intuition and gut feelings must be systematically checked against real data and automated evaluation rather than taken on faith. - **Infrastructure as an Enabler**: Specialized trust infrastructure allows teams to maintain safety guarantees at organizational scale without sacrificing product velocity or sophistication.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=jekXJWBLI-I&t=4s) Hey, wait a second. Does this agent actually look out for me? Does it care about what I'm doing? Or is it... really just looking out for Etsy and trying to drive more revenue for Etsy at my expense? — Tomer Sabo

> [05:33](https://www.youtube.com/watch?v=jekXJWBLI-I&t=333s) to surround the agents with extreme constraints, all kinds of tests — Robert C. Martin ("Uncle Bob")

> [11:15](https://www.youtube.com/watch?v=jekXJWBLI-I&t=675s) Does it deliver the value that we expect? — Beth Sebian

> [11:15](https://www.youtube.com/watch?v=jekXJWBLI-I&t=675s) Are we mitigating the risk that we've wanted to tackle at that level with that population? — Beth Sebian

> [14:08](https://www.youtube.com/watch?v=jekXJWBLI-I&t=848s) moral attention as a team practice — Beth Sebian

> [14:08](https://www.youtube.com/watch?v=jekXJWBLI-I&t=848s) compliance as a ceiling — Beth Sebian

> [18:45](https://www.youtube.com/watch?v=jekXJWBLI-I&t=1125s) repairing trust after you've broken it is the slowest — Beth Sebian

> [19:56](https://www.youtube.com/watch?v=jekXJWBLI-I&t=1196s) how confident are you that your safeguards are strong enough? — Beth Sebian

> [19:56](https://www.youtube.com/watch?v=jekXJWBLI-I&t=1196s) using evidence to actually check your gut is going to save you — Beth Sebian

> [19:56](https://www.youtube.com/watch?v=jekXJWBLI-I&t=1196s) I think I'll leave it there. Thanks everyone. — Beth Sebian

## Mentioned Tomer Sabo, Beth Sebian, Etsy, Ai4, Robert C. Martin ("Uncle Bob"), Clean Code, Agile Manifesto, How I Use 150 Agents to Run My Life

## Video description Tomer Sabo - AI Staff Product Manager, Etsy Beth Sebian - Engineering Manager, Etsy

User Trust *is* a product.

Building and keeping user trust, in the context of AI product development, unlocks new types of user engagement and opens up new opportunities to connect with and create value for customers.

User trust is not built and maintained by happenstance. Safeguarding and nurturing it is a systematic process that requires investment in specific technical and organizational infrastructure. We want to share our approach and tactics to making those investments, and how doing so allowed our team to move more quickly, ambitiously, and confidently as we develop AI enabled products for our customers.

[Source](https://www.youtube.com/watch?v=jekXJWBLI-I)

中文简报

Etsy部署70项卖家AI Agent风险测试以加速发布 Etsy认为，系统性的信任基础设施不仅不会拖慢AI部署，反而能加快其进程，因为修复受损的用户信心远比主动维护防护措施耗费更多时间。 Tomer Sabo和Beth Sebian阐述了Etsy在为卖家扩展Agentic AI工具的同时保障用户信任的综合框架。他们认为，信任是一种推动增长的资产，而不仅仅是防范下行风险的手段，并详细介绍了涵盖三个层面的方法：技术评估（识别12个类别中的70种失效模式）、组织控制（从Alpha测试扩展到全面上线的4阶段门禁发布流程）以及文化实践（培养道德关注度的负责任AI课程）。他们最终表明，通过建立基于证据的信任护栏制度，可以避免客户关系发生灾难性破裂，从而加快产品的交付速度。 用户对 Agentic AI 的信任可能在长尾场景中微妙侵蚀，即便模型输出技术上正确（如广告 Agent 反复建议追加预算引发利益冲突怀疑）；Etsy 指出信任并非单纯防范风险的成本，而是驱动用户授权更复杂业务流程的核心增长资产。 (0:04) 针对行业常混淆模型能力与可信度的现状，Etsy 梳理出 12 类共 70 种失败模式，结合发生频率、严重程度、触达范围与财务影响进行量化评估，并强调第三方缺乏业务上下文使信任评估无法对外外包。 (5:33) 在组织落地层面，Etsy 采用四阶段门控发布框架（Research Alpha、1,000 名用户、25,000 名用户及全量发布），各阶段均定义清晰的假设、退出标准和风险护栏以科学控制风险暴露。 (11:15) 由于静态合规清单无法应对复杂的 AI 边缘案例，Etsy 借鉴医疗和法律领域的案例式伦理课程，在团队内部培养关注未代表群体、超越合规上限、设计异议机制与具备建设性策略的道德判断力。 (14:08)

Reading summary

According to Etsy’s Tomer Sabo and Beth Sebian, sustaining user trust in autonomous AI systems requires treating trust as an operational growth driver rather than a compliance checklist, underpinned by coordinated technical, organizational, and cultural infrastructure. Trust Erosion and Expanded User Delegation (0:04) Sabo highlights that technical precision does not guarantee user trust. In an Etsy seller tool built to optimize ad budgets, repeatedly recommending spend increases—even when mathematically sound—led sellers to suspect the agent was prioritizing Etsy’s corporate revenue over their personal business health. When users perceive conflicting incentives, they discard recommendations or abandon the agent entirely. Conversely, Sabo argues that high user trust functions as a growth asset: while tentative users will accept only isolated suggestions, deeply trusting users expand their adoption curve, delegating complex, multi-domain workflows to autonomous systems. Technical Risk Modeling and In-House Guardrails (5:33) Borrowing software engineer Robert C. Martin’s principle of constraining autonomous code generators with strict automated tests, Etsy decouples capability benchmarks from trustworthiness evaluations, which assess honesty, consistency, and cohort equity. Sabo details an internal taxonomy where cross-functional experts identified 70 failure modes across 12 risk categories. Each mode is scored across frequency, severity of harm, reach, and financial impact to assign risk tiers ranging from low to catastrophic. Mitigations are tailored to these tiers: low risks receive descriptive banners or adjusted terms, whereas critical risks—such as cross-seller data leakage—trigger hard deterministic runtime controls. Sabo notes that every mitigation requires dedicated evaluations to measure baseline frequency and control efficacy. Because external vendors lack context on platform-specific user relationships, Etsy maintains that trust evaluations cannot be outsourced. Gated Scaling Across Staged User Cohorts (11:15) To prevent rapid scaling from outpacing risk controls, Sebian outlines Etsy’s gated deployment framework, which balances feature velocity against risk exposure. Features advance across four sequential cohorts: a research alpha, an initial scaled test of 1,000 users, a broader validation group of 25,000 users, and general platform release. Every release gate enforces two evaluation criteria: verifying whether the feature delivers expected customer value, and confirming that the targeted risks are mitigated at that cohort size. Advancement requires explicit release artifacts, including a clear impact hypothesis, open questions on user friction, empirical exit criteria for engagement, and risk guardrails tracking error frequencies and negative user signals. Cultural Practices for Moral Attention and Dissent (14:08) Because engineers and product managers make subjective decisions daily while crafting prompts and evaluation sets, Sebian explains that static rules are insufficient and can lead teams to treat compliance as a ceiling. Borrowing case-based ethics models from medicine, law, and the military, Etsy instituted a curriculum to develop 'moral attention as a team practice.' The curriculum develops four core capabilities: identifying unrepresented stakeholders who might experience secondary harms, recognizing systemic failure modes beyond legal mandates, designing for dissent through formal premortems that empower staff to pause rollouts, and practicing productive tact by proposing concrete engineering solutions alongside flagged concerns. Sebian and Sabo conclude that investing in trust infrastructure acts as an enabler of speed rather than an impediment. Because repairing broken user trust is the slowest and most expensive process in product development, relying on empirical automated validation rather than gut instinct provides teams the necessary confidence to deploy complex generative AI features safely at scale.

精炼阅读摘要

Etsy的人工智能产品经理Tomer Sabo与工程经理Beth Sebian指出，智能体AI（Agentic AI）的规模化落地不能仅依赖基础模型能力或直觉判断，而必须构建贯穿技术、组织与文化的三层信任基础设施；用户信任不仅是防范合规与声誉风险的底线，更是拓展产品功能边界与驱动业务增长的核心资产。 信任重构：从下行风控转向增长资产 (0:04) Sabo以Etsy卖家广告预算智能体为例阐明，智能体的决策即便在数学与技术逻辑上完全最优（例如在广告投资回报率良好时反复建议增加预算），若持续向用户推送，也会诱发用户对平台动机的怀疑，促使卖家担忧平台究竟是在协助其经营还是借机推高自身营收。这种细微的信任侵蚀往往潜藏在边缘场景中，一旦累积爆发就会导致用户拒绝采纳建议甚至彻底停用工具，从而抹杀智能体的实际效用。 Sabo进一步提出，必须改变将信任仅视作类似网络安全等“防御性下行风险”的传统视角。相反，信任具备显著的业务上行价值：随着用户信任阈值的逐步提高，其使用场景将从单次的被动建议采纳，逐步延伸至跨领域的主动战略咨询与复杂任务委托。因此，企业需要将信任视为一项需要长期、递进式培育的增长资产，而非仅在出现危机后才被动补救的合规负担。 技术层：细分风险建模与防御性验证 (5:33) 当前AI行业常将模型在数学或基准测试上的“原始能力”与“可信度”混为一谈。Sabo引用《Clean Code》作者Robert C. Martin（Uncle Bob）通过严密自动化测试来约束不可全读代码的做法，强调工程团队必须为自主智能体设立严格的验证防线。Etsy组织跨职能专家开展头脑风暴，针对卖家智能体归纳出12个核心风险类别与70种具体故障模式，并依据预期发生频率、危害严重性、波及用户范围以及潜在财务影响进行多维度量化评级。 在缓解策略上，Etsy实行与风险等级相匹配的分级防御：针对低风险项目采用横幅提示或补充声明等轻量级方案；对涉及跨卖家数据泄露等致命级风险，则在智能体运行时直接嵌入确定性的工作流控制机制。Sabo强调，由于外部供应商与咨询机构缺乏对特定平台用户关系与专有风险背景的深刻理解，核心信任评估无法对外转包，必须由企业内部全权负责，且每项防御措施均需配套自动化评估来验证其在生产环境中的真实拦截效果。 组织层：基于门控机制的渐进式发布 (11:15) Beth Sebian介绍了将风险评估与功能拓展相连接的组织管理框架。为平衡快速交付与信任维护，Etsy设立了与风险成熟度严格绑定的门控发布机制。每一个门控检查点都必须硬性回答两个根本问题：“该功能是否交付了符合预期的用户价值？”以及“团队是否在该阶段人群中有效缓解了既定风险？”。 团队在推进功能跨越门控时必须提交四项核心交付物：明确的用户影响假设、野外测试中待解的未知问题、支撑扩量的数据退出标准，以及监控故障率和负面用户信号的风险护栏。Sebian透露，Etsy近期AI功能的上线划分为四个明确阶段：从早期收集深度反馈的研究性内测（research alpha），平稳过渡至1,000名用户规模，随后扩容至25,000名用户进行广泛验证，最终在确认防御体系完备后才推向全平台常规发布（general release）。 文化层：道德敏锐度与制度化异见机制 (14:08) Sebian认为，单纯依靠静态规章和检查清单会导致一线团队产生思维惰性并放弃自主判断，然而提示词调试与参数配置等重大决定每天都在底层发生。为此，Etsy借鉴医学、法律与军事领域的案例研讨教学，推行旨在建立“团队实践式道德敏锐度”的培训课程，重点培育四项核心能力：主动识别并顾及未在场的相关利益群体（特别是边缘化用户）；打破将“法律合规作为安全上限”的反模式，正视立法滞后于生成式AI所造成的现实盲区；推行正式的发布前预检（premortems）等制度化机制，赋予团队成员叫停发布的权力；以及培养“建设性策略”，要求员工在指出风险的同时提供兼顾业务目标的替代方案，避免沦为单纯的流程阻碍者。 Sebian与Sabo总结认为，建立完善的信任基础设施并非拖慢研发节奏的障碍，而是实现持续、稳定交付速度的催化剂。在部署高敏感度的生成式AI产品时，团队绝不能依赖直觉假设，而必须依赖客观实证数据与自动化校验来审视系统表现。及早捕捉并纠正异常至关重要，因为在用户信任破裂之后再行修补，是产品研发中耗时最长且成本最高的环节。

Detailed notes — zh

## Background 本视频由 Ai4 频道于 2026 年 8 月 18 日发布在 YouTube 平台。内容为 Etsy 的 AI 资深产品经理 Tomer Sabo 与工程经理 Beth Sebian 的联合演讲，深入探讨了为何“用户信任本身就是一种产品”，并分享了 Etsy 如何通过构建技术与组织基础设施来系统化维护信任、赋能 AI 产品的快速创新。 ## TL;DR 演讲围绕 AI 与智能体时代如何系统化构建与守护用户信任展开，强调信任不仅是防范下行风险的底线，更是解锁更深层交互与核心价值的增长飞轮。Etsy 提出了涵盖技术、组织与文化的三层信任基础设施：在技术层，从单纯关注模型能力转向针对 70 种失效模式的信任度与护栏评估；在组织层，推行绑定明确退出阈值与风险护栏的四阶段门控发布流程；在文化层，摒弃机械合规教条，通过案例教学法培养一线工程师的“道德注意力”与异见表达机制，用严密的工程证据与护栏底座赋能更大胆的业务创新。 ## Atmosphere & style 整体氛围务实深邃、理性从容且极具工程实操感。两位讲者分工默契，Tomer 借由生动幽默的现场互动与逼真的智能体冲突案例破题，Beth 则以沉稳细腻的管理思维抽丝剥茧地解析团队文化与发布机制；整场分享既有对行业沉迷跑分与合规上限的敏锐警示，又充盈着将抽象伦理落地为扎实工具链的工程师自信，为观众带来了清醒克制却又充满创新底气的沉浸式现场体验。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=jekXJWBLI-I&t=4s) 即使 AI 智能体给出的建议在数学和技术逻辑上完全正确（如持续建议卖家增加广告预算），若缺乏对用户立场的关照，也会悄然侵蚀用户信任并导致智能体最终被停用。 - [02:52](https://www.youtube.com/watch?v=jekXJWBLI-I&t=172s) 用户信任不仅是防范用户流失的“下行风险”，更是解锁更深交互、拓展高价值场景授权边界的巨大“上行收益”。 - [04:44](https://www.youtube.com/watch?v=jekXJWBLI-I&t=284s) 系统化守护与培育用户信任需要三层基础设施：技术基础设施层（评测与护栏）、组织管理层（门禁式发布）以及文化支撑层（负责任 AI 课程体系）。 - [05:33](https://www.youtube.com/watch?v=jekXJWBLI-I&t=333s) AI 评估体系必须跳出单纯的“模型能力”评测局限，转向覆盖诚实性（Honesty）、一致性（Consistency）、行为边界（Boundaries）以及公平性与群体影响（Fairness & Impact）的信任度评测。 - [08:03](https://www.youtube.com/watch?v=jekXJWBLI-I&t=483s) 建立评估体系必须基于严密的跨职能风险梳理与量化评分，并为每一项缓解策略配置专属的 Eval 来检验实际拦截有效性；平台特有的信任风险与评测必须坚持内部自建。 - [11:15](https://www.youtube.com/watch?v=jekXJWBLI-I&t=675s) 门控发布（Gated Releases）机制将风险优先级、理论假设、开放问题与准出护栏深度绑定，在快速迭代交付与持续建立用户信任之间达成自动平衡。 - [11:15](https://www.youtube.com/watch?v=jekXJWBLI-I&t=675s) 扩量过程应划分为明确的阶段性阶梯（如深度 Alpha 实验、1,000 名用户、25,000 名用户直至全量上线），每个门禁节点均需基于严格的风险化解指标进行放行判定。 - [14:08](https://www.youtube.com/watch?v=jekXJWBLI-I&t=848s) AI 场景下的关键决策往往由一线工程师在编码和评测瞬间决定；组织应摒弃死板的规章灌输，采用“案例教学法”将道德注意力（Moral Attention）与审慎判断力内化为工程实践。 - [16:06](https://www.youtube.com/watch?v=jekXJWBLI-I&t=966s) 培养工程团队审慎判断力需聚焦四项核心技能：识别被忽视的利益相关者、失效模式识别、为异见建立机制（如事前剖析会），以及兼具风险雷达与解决思路的建设性沟通智慧。 - [16:06](https://www.youtube.com/watch?v=jekXJWBLI-I&t=966s) 必须警惕“以合规作为上限（Compliance as a ceiling）”的失效陷阱；法律法规具有天然滞后性，仅满足合规标准无法覆盖生成式 AI 技术前沿的实际伤害与安全盲区。 - [18:45](https://www.youtube.com/watch?v=jekXJWBLI-I&t=1125s) 信任基础设施并非拖慢研发的枷锁，而是赋能大胆原型实验的安全底座；必须用坚实的工程证据和真实数据来校验主观直觉，方能应对生成式 AI 的极端复杂性。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=jekXJWBLI-I&t=4s) 开场介绍与用户信任的微妙侵蚀 来自 Etsy 的 AI 资深产品经理（Staff Product Manager）Tomer Sabo 与工程经理（Engineering Manager）Beth Sebian 共同开场。Tomer 幽默地调侃在漫长会议的一天接近尾声时，大家还得“乘上整整五层扶梯走到走廊尽头”来到这里，肯定是对这个主题怀有极大的热情。在正式开讲前，Tomer 做了一个快速的现场互动调查：有多少人认为自己团队决定何时规模化推广（Scale）一个 AI 功能的流程对客户来说已经足够好了？结果台下无一人举手。 ```mermaid flowchart TD accTitle: 卖家广告预算智能体中的信任侵蚀循环 subgraph OptimizationLoop["广告预算推荐闭环"] A["卖家拥有良好广告回报率 (ROAS)"] --> B["智能体给出技术上正确的建议：<br/>增加广告预算"] B --> C["连续多次推荐增加预算"] end subgraph TrustErosion["用户心理与信任崩溃"] C --> D{"卖家产生怀疑：<br/>'它在为我着想，还是在帮 Etsy 创收？'"} D -->|信任被侵蚀| E["不再采纳广告优化建议"] E --> F["彻底停用卖家 AI 智能体"] end ``` Tomer 进而引入核心议题：在 AI 产品开发、特别是 Agentic AI（智能体 AI）应用中，用户信任（User Trust）究竟意味着什么？他以 Etsy 为平台卖家打造的 AI 智能体为例展开说明。该智能体能够为卖家优化广告预算并提出操作建议。从算法角度看，它经过优化以提升卖家销售额；当一位卖家的广告投资回报率（ROAS）非常出色、每投入一美元就能赚回数倍时，在技术和商业逻辑上“最正确”的优化建议显然是“追加广告预算”。然而，如果这个智能体一次又一次地向卖家推送“增加预算”的建议，卖家必然会心生疑窦：“等等，这个智能体是真的在为我着想、关心我的利益，还是仅仅在以牺牲我的利益为代价为 Etsy 赚取更多广告收入？”即使智能体的表现完全符合预期且建议在数学上是正确的，长此以往也会悄然侵蚀卖家的信任，导致卖家下一次不再采纳建议，甚至彻底停用该智能体。 --- ### [02:52](https://www.youtube.com/watch?v=jekXJWBLI-I&t=172s) 用户信任不仅是下行风险，更是巨大上行收益 Tomer 强调，在 AI 与智能体产品开发中，必须将用户信任提升为“一等公民”（First-class consideration）来对待。很多人把信任仅仅看作类似网络安全的“下行风险”（Downside risk）——认为一旦失去信任，用户就会流失。但实际上，信任更蕴含着巨大的“上行收益”（Upside）： - **更深度的交互与价值解锁**：当卖家对智能体产生高度信任时，不仅会采纳基础的广告建议，还会主动提出更多追问、寻求更复杂的业务指导，并将其应用到更多核心场景中。 - **信任决定了用户授权的边界**：Tomer 举例提到，无论你自认为对 AI 多么熟悉，总有人比你更信任 AI。例如有人在分享“如何用 150 个智能体来打理我的人生”，甚至直接将信用卡号、社会安全号（SSN）交给智能体，授权其进行金融交易和开设银行账户。普通人之所以不敢这么做，根本原因在于对 AI 的信任程度不同。 因此，信任不是非黑即白的二元状态，而是一个可以持续建立、维护与深化的连续体。 --- ### [04:44](https://www.youtube.com/watch?v=jekXJWBLI-I&t=284s) 构建用户信任的三层基础设施框架 为了系统化地构建和守护用户信任，Etsy 提出了一个涵盖技术、组织和文化的三层基础设施架构： ```mermaid flowchart TD accTitle: 用户信任的三层基础设施 Layer3["文化层 (Cultural Layer)<br/><b>负责任 AI 课程体系 (Curriculum of Responsible AI)</b><br/>在组织内部建立负责任 AI 的文化认知与共识"] --> Layer2 Layer2["组织层 (Organizational Layer)<br/><b>门禁式发布机制 (Gated Releases)</b><br/>建立阶段性放量标准与上线审核机制"] --> Layer1 Layer1["技术基础设施层 (Technical Layer)<br/><b>全面评估体系 (Evaluations & Guardrails)</b><br/>超越功能测试，覆盖信任、公平与防护栏评测"] ``` 1. **技术基础设施层（Evaluations / 评估体系）**：针对信任与护栏建立严格的评测系统； 2. **组织管理层（Gated Releases / 门禁式发布）**：在产品走向生产与放量过程中设立严密把关的准入机制； 3. **文化支撑层（Curriculum of Responsible AI / 负责任 AI 课程体系）**：在公司内部普及负责任 AI 的理念，确保上述两层获得组织文化的充分支持。 --- ### [05:33](https://www.youtube.com/watch?v=jekXJWBLI-I&t=333s) 重新审视评估体系：从“能力评测”转向“信任度评测” 在解析技术层时，Tomer 引用了著名软件工程先驱、《代码整洁之道》（*Clean Code*）作者及《敏捷宣言》合著者 Robert C. Martin（Uncle Bob）近期的一段话： > “我的年龄比你们大得多，我从 60 年代末就开始写代码。我目前的策略是：**绝不阅读任何由我的智能体编写的代码**。这是我能够利用它们生产力的唯一途径。作为替代，我所做的是给这些智能体施加极端的约束条件和各种测试。最终，我对它们生成的代码拥有极高的信心，因为这些代码必须通过我设置的所有约束与测试的残酷考验（Run the gauntlet）。” Tomer 指出，目前 AI 智能体应用最前沿、最成熟的领域正是软件开发。因此，借鉴顶级程序员约束智能体的方法至关重要。行业内虽然都在大谈评估（Evals）与护栏（Guardrails），但绝大多数讨论都局限在**能力（Capability）**层面——即关注模型是否更聪明、能否证明新的数学定理、输出是否满足既定功能。 然而，真正关乎用户信任的维度往往被忽视： - **诚实性（Honesty）**：智能体是否如实反映信息，不产生欺骗性诱导？ - **一致性（Consistency）**：在不同场景和时间下是否表现稳定？ - **边界控制（Boundaries）**：智能体是否严格遵守设定的行为边界？ - **公平性与群体影响（Fairness & Impact）**：智能体是否公平对待所有卖家？不同背景的用户群体是否会受到差异化的负面影响？ --- ### [08:03](https://www.youtube.com/watch?v=jekXJWBLI-I&t=483s) 风险评估与护栏落地的实战流程 Etsy 将建立信任评估体系拆解为一套严密的方法论，绝不依赖凭空臆测，而是从详尽的跨职能风险评估入手： ```mermaid flowchart LR accTitle: Etsy 的 AI 风险评估与评估闭环流程 A["风险头脑风暴<br/>(跨部门领域专家)"] --> B["风险分级与打分<br/>(发生频率 × 影响严重度)"] B --> C["制定缓解策略<br/>(低成本提示语 vs 严密工作流校验)"] C --> D["构建针对性 Eval<br/>(监测发生率 & 检验拦截有效性)"] D -.->|持续迭代| A ``` - **全方位风险分类与细化**：召集全公司的领域专家（Domain Experts）与业务专家，全面梳理可能危及用户的风险点。在 Etsy 的卖家智能体项目中，团队共梳理出 **12 个核心风险类别**，并在其下细分出 **70 个具体的失效模式（Failure Modes）**。 - **多维度风险量化评分**：对每个风险点从“预期发生频率”、“潜在影响广度（波及多少用户）”、“影响深度及财务损失”等多维度打分，最终判定为低（Low）、中（Medium）、高（High）乃至灾难性（Catastrophic）风险等级。 - **匹配差异化的防御策略**： - *低/中风险*：采用轻量级缓解措施，例如界面提示横幅（Banners）或在服务条款中增加审慎免责声明（Hedged language）。 - *高危/灾难性风险*：例如**跨卖家数据泄露（Cross-seller data leakage）**，则必须在整个智能体工作流中植入多重硬性控制与拦截检查点。 - **为每项缓解机制配置专属评估（Eval for each mitigation）**： - 必须验证风险在现实中到底以多大频率发生； - 必须评测所设立的防护栏是否真正拦截了风险，还是存在漏网之鱼。 - **坚持内部自建而非完全外包**：虽然市面上有像 Sierra 这样的第三方工具或咨询机构擅长搭建智能体和评测基础能力，但外部团队绝不可能比内部更理解平台卖家对信任的敏感点与特有业务风险。因此，风险建模与信任评测必须由内部团队常态化主导推进。随后，Tomer 将演讲交接给工程经理 Beth Sebian。 ### [11:15](https://www.youtube.com/watch?v=jekXJWBLI-I&t=675s) 门控发布策略：在风险与规模扩展中寻找平衡 在转向组织与文化层面的基础设施建设时，工程经理 Beth Sebian 重点阐述了 Etsy 控制产品发布的关键机制——门控发布（Gating）。作为工程管理者，她始终在权衡一个核心痛点：团队如何在任何时刻都能最有效地调配工程资源，为客户创造真正的影响力。门控策略的本质，正是为了在“快速迭代交付”与“持续建立用户信任”这双重目标之间建立自动平衡。这套流程将团队对风险的理解、对“何为对买家有价值”的未解之问，与具体的发布阈值和扩量节点进行了精细的序列化对应。 团队将风险进行优先级排序，并映射到特定的目标用户群体中；因为最丰富、最真实的认知往往发生在“产品实际进入真实卖家环境的野外场景”。在每一个设定的发布门禁（Gate）前，团队都必须强制回答两个核心问题：“产品是否达到了预期的价值？”以及“我们是否在这个规模层级上，成功化解了针对该目标人群设定的特定风险？”为此，每个门槛都明确绑定了四项要素：基于用户影响的理论假设、待验证的开放问题、进入下一阶段的准出标准（Exit Criteria），以及作为特殊准出标准的风险护栏（Risk Guardrails）。 ```mermaid flowchart LR accTitle: 门控发布四个阶段流程图 subgraph S1["阶段 1：阿尔法研究"] A["高纵深 Alpha 实验<br/>深度定性反馈"] end subgraph S2["阶段 2：小规模验证"] B["1,000 名用户<br/>基础风险排查"] end subgraph S3["阶段 3：规模化测试"] C["25,000 名用户<br/>放量压力测试"] end subgraph S4["阶段 4：全面上线"] D["General Release<br/>全量正式发布"] end A -->|"验证退出标准与护栏"| B B -->|"达成指标门槛"| C C -->|"风险化解验证"| D ``` - **四阶段扩量路径**：在最近上线的功能中，Etsy 采用了清晰的四级跳跃——从极其精细、高深度研究的 Alpha 实验起步，验证核心假设；随后放量至 1,000 名用户；平稳后进一步扩展到 25,000 名用户；最终在各项指标完全达标后推进至全量公开发布（General Release）。 - **门禁的核心控制要素**：每一个扩量节点并非机械式的时间推进，而是严格依据“假设验证、开放问题解答、准出阈值、风险护栏”四位一体的评估结果来决定是否放行。 --- ### [14:08](https://www.youtube.com/watch?v=jekXJWBLI-I&t=848s) 文化基础设施：基于案例教学培养团队的“道德注意力” 除了技术与流程层面的发布门禁，构建用户信任的另一大核心支柱是文化建设。随着团队在 AI 领域的探索逐步深入，管理者意识到一个严峻的现实：大量对产品走向和用户体验产生深远影响的决定，实际上并不是在管理层会议上做出的，而是由一线工程师在现场写代码、构建评估方案（Evaluations）的细微瞬间决定的。AI 场景下的陷阱往往极其隐蔽，没有任何规则清单能够穷尽所有可能出现的边界状况，团队必须具备从混沌中提炼风险模式的直觉。 为了将这种应对风险的能力培养为一种专业技艺（Craft），团队建立了一套内部培养课程，其核心理念是“将道德注意力（Moral Attention）内化为团队的日常实践”。团队摒弃了传统的规章制度灌输或乏味的线上自学模块，转而采用类似医学伦理、法律教学和军事道德培训中广泛使用的“案例教学法”（Case-based approach）。Sebian 深刻指出，单纯给员工定死规则往往会导致“决策权向规则的转嫁”——一旦超出既定条目，人们便无所适从；而在现实的 AI 研发中，信息永远是不完备的，团队需要一线工程师有底气在关键时刻运用个人判断力，真正意识到“这就是需要你挺身而出的时刻，我们平时的训练正是为了让你在这一刻发挥决定性作用”。 - **集体研讨而非单向灌输**：培训完全依托于有引导员主持的社区化集体讨论（Facilitated conversations），而非冷冰冰的合规测试题库。 - **判断力作为可习得的技能**：实践证明，审慎的工程与道德判断力并非不可捉摸的玄学，而是可以通过高频案例推演和建立共同语言来系统性培养的复利技能。 --- ### [16:06](https://www.youtube.com/watch?v=jekXJWBLI-I&t=966s) 审慎判断的四项核心技能与“以合规为上限”的失效陷阱 在文化培训与日常工程实践中，Etsy 重点打磨团队成员的四项专业技能，旨在构建一个既敏锐又富有建设性的研讨环境： 1. **识别被忽视的关键利益相关者（Unrepresented Stakeholders）**：这极其考验团队的共情与想象力，促使工程师在写下每一行评估逻辑时自问：还有哪些群体可能因该模型受到负面影响，却从未出现在当前的用户画像中？ 2. **失效模式识别（Failure Mode Recognition）**：对常见的系统崩溃、偏见放大或幻觉模式建立高度敏锐的雷达。 3. **为异见建立机制（Designing for Dissent）**：团队在发布流程中推行了“事前剖析会”（Premortems），从制度层面赋予每个人叫停的权力，公开鼓励并支持有成员站出来提出：“请稍等一下，让我们先认真评估一下这么做的潜在后果。” 4. **建设性的处事智慧（Tact）**：警惕讨论风险时的反面模式——即成为那个“只抛出问题、泼冷水、强行终止对话的阻碍者”。团队追求的理想人选，是所有人“都极度希望他出现在会议室里”的角色：他不仅雷达敏锐、能提前揪出深层隐患，更能凭借经验带来解决思路，协助整个团队高超地化解棘手抉择。 在谈到失效模式识别时，Sebian 着重警示了“以合规作为上限”（Compliance as a ceiling）的严重误区。当前整个科技行业普遍倾向于将法律法规或外部监管条文作为衡量工作合格的标准线；然而在生成式 AI 发展日新月异的背景下，法律条文天生具有滞后性，根本无法预判技术前沿可能衍生的所有实际伤害。如果团队仅仅将“符合法规”视作终极安全线，必然会在未知水域中漏掉大量的安全盲区，最终导致用户信任的急剧崩塌。 --- ### [18:45](https://www.youtube.com/watch?v=jekXJWBLI-I&t=1125s) 信任基础设施的飞轮效应：用数据与证据校验直觉 在演讲的尾声，Sebian 总结了这整套兼顾技术工具、发布门禁与团队文化的信任基础设施所带来的长远复利。在项目初期就搭建好这套防护体系，并非减慢研发节奏的枷锁，反而是让团队在面对突发失误和全新未知时可以随时倚赖的安全底座——团队能够借由同样的护栏快速吸取教训、加固系统，并毫无后顾之忧地大步向前。 这种基础设施最终反哺的是研发效能与创新魄力。有了坚实的信任防护网，团队反而敢于进行更大胆的原型实验，以更强烈的底气推进发布，并在问题演变成昂贵、致命的公关或业务灾难之前将其提前消灭。“一旦信任被彻底打破，事后修补信任的代价将是全天下最缓慢、最痛苦的过程。” 在最后与现场观众的互动中，Sebian 呼吁从业者不要仅仅依赖主观直觉来推测防护体系是否完备，而必须学会用铁打的工程证据和真实数据来校准直觉；唯有通过规模化、严谨的基础设施建设，团队的应对手段才能真正匹配现代生成式 AI 产品的极端复杂性。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=jekXJWBLI-I&t=4s) 等等，这个智能体是真的在为我着想、关心我的利益，还是仅仅在以牺牲我的利益为代价为 Etsy 赚取更多广告收入？ — Tomer Sabo > [02:52](https://www.youtube.com/watch?v=jekXJWBLI-I&t=172s) 如何用 150 个智能体来打理我的人生 — Tomer Sabo > [05:33](https://www.youtube.com/watch?v=jekXJWBLI-I&t=333s) 绝不阅读任何由我的智能体编写的代码。这是我能够利用它们生产力的唯一途径。作为替代，我所做的是给这些智能体施加极端的约束条件和各种测试。最终，我对它们生成的代码拥有极高的信心，因为这些代码必须通过我设置的所有约束与测试的残酷考验。 — Robert C. Martin (Uncle Bob) > [11:15](https://www.youtube.com/watch?v=jekXJWBLI-I&t=675s) 产品是否达到了预期的价值？ — Beth Sebian > [11:15](https://www.youtube.com/watch?v=jekXJWBLI-I&t=675s) 我们是否在这个规模层级上，成功化解了针对该目标人群设定的特定风险？ — Beth Sebian > [14:08](https://www.youtube.com/watch?v=jekXJWBLI-I&t=848s) 将道德注意力（Moral Attention）内化为团队的日常实践 — Beth Sebian > [14:08](https://www.youtube.com/watch?v=jekXJWBLI-I&t=848s) 这就是需要你挺身而出的时刻，我们平时的训练正是为了让你在这一刻发挥决定性作用 — Beth Sebian > [16:06](https://www.youtube.com/watch?v=jekXJWBLI-I&t=966s) 请稍等一下，让我们先认真评估一下这么做的潜在后果。 — Beth Sebian > [16:06](https://www.youtube.com/watch?v=jekXJWBLI-I&t=966s) 以合规作为上限 — Beth Sebian > [18:45](https://www.youtube.com/watch?v=jekXJWBLI-I&t=1125s) 一旦信任被彻底打破，事后修补信任的代价将是全天下最缓慢、最痛苦的过程。 — Beth Sebian ## Mentioned Etsy, Tomer Sabo, Beth Sebian, Robert C. Martin (Uncle Bob), 《代码整洁之道》（Clean Code）, 《敏捷宣言》, Sierra, Ai4 ## Video description Tomer Sabo - AI Staff Product Manager, Etsy Beth Sebian - Engineering Manager, Etsy User Trust *is* a product. Building and keeping user trust, in the context of AI product development, unlocks new types of user engagement and opens up new opportunities to connect with and create value for customers. User trust is not built and maintained by happenstance. Safeguarding and nurturing it is a systematic process that requires investment in specific technical and organizational infrastructure. We want to share our approach and tactics to making those investments, and how doing so allowed our team to move more quickly, ambitiously, and confidently as we develop AI enabled products for our customers. [Source](https://www.youtube.com/watch?v=jekXJWBLI-I)

Source quotes

[0:04] Tomer Sabo: 等等，这个智能体是真的在为我着想、关心我的利益，还是仅仅在以牺牲我的利益为代价为 Etsy 赚取更多广告收入？ [2:52] Tomer Sabo: 如何用 150 个智能体来打理我的人生 [5:33] Robert C. Martin (Uncle Bob): 绝不阅读任何由我的智能体编写的代码。这是我能够利用它们生产力的唯一途径。作为替代，我所做的是给这些智能体施加极端的约束条件和各种测试。最终，我对它们生成的代码拥有极高的信心，因为这些代码必须通过我设置的所有约束与测试的残酷考验。 [11:15] Beth Sebian: 产品是否达到了预期的价值？ [11:15] Beth Sebian: 我们是否在这个规模层级上，成功化解了针对该目标人群设定的特定风险？ [14:08] Beth Sebian: 将道德注意力（Moral Attention）内化为团队的日常实践 [14:08] Beth Sebian: 这就是需要你挺身而出的时刻，我们平时的训练正是为了让你在这一刻发挥决定性作用 [16:06] Beth Sebian: 请稍等一下，让我们先认真评估一下这么做的潜在后果。 [16:06] Beth Sebian: 以合规作为上限 [18:45] Beth Sebian: 一旦信任被彻底打破，事后修补信任的代价将是全天下最缓慢、最痛苦的过程。

Source quotes

[0:04] Tomer Sabo: Hey, wait a second. Does this agent actually look out for me? Does it care about what I'm doing? Or is it... really just looking out for Etsy and trying to drive more revenue for Etsy at my expense? [5:33] Robert C. Martin ("Uncle Bob"): to surround the agents with extreme constraints, all kinds of tests [11:15] Beth Sebian: Does it deliver the value that we expect? [11:15] Beth Sebian: Are we mitigating the risk that we've wanted to tackle at that level with that population? [14:08] Beth Sebian: moral attention as a team practice [14:08] Beth Sebian: compliance as a ceiling [18:45] Beth Sebian: repairing trust after you've broken it is the slowest [19:56] Beth Sebian: how confident are you that your safeguards are strong enough? [19:56] Beth Sebian: using evidence to actually check your gut is going to save you [19:56] Beth Sebian: I think I'll leave it there. Thanks everyone.

Original transcript

[0:04] And thanks everyone for joining. I know  it's sort of the end of a long day and [0:07] you have to go all the way up all the five  escalators and come to the end of the hallway. So [0:11] I know you all are really excited  about us specifically, right? [0:15] Um so yeah, I'm Tor. I'm a product manager  at Etsy. I'm joined by my colleague Beth. [0:20] She's an engineering manager at Etsy. And we  want to talk talk to you a little bit about [0:23] user trust and how to build infrastructure  to ensure that you're taking care of it. [0:30] Um, and before getting started, just a quick  survey. Uh, want to see just a hands raised for if [0:37] you believe that your process for deciding when to  scale an AI feature is good enough for customers. [0:44] No hands. Okay. Well, let's see if uh you  feel any differently when we're done uh [0:50] talking to you about how we think about it. Um,  cool. So I want to start by just talking about [0:57] what we mean when we talk about user trust  in the context of AI product development, [1:02] AI application development and specifically  agentic AI product and application development. Um [1:11] I want to give an example about how we think about  it at Etsy that I think illustrates exactly how we [1:16] think about user trust. So we work on a agent for  our sellers that among other things can optimize a [1:23] seller's ad budget um take action and uh you know  make recommendations around what they should be [1:30] doing with their ads [snorts] and it's designed to  give good recommendations right that will increase [1:35] sales for sellers. It's optimized to increase  sales for our sellers. But you can imagine a [1:41] situation where the correct recommendation, the  correct optimization for a seller's ad budget is [1:46] to increase their budget, right? Maybe they  have great return on ad spend and every dollar [1:51] they put in gets many dollars back, the correct  recommendation would be, hey, add more money. [1:56] Um, but if over and over and over again the  agent keeps recommending uh adding more budget, [2:02] keeps increasing the seller's ad budget, you can  imagine the seller saying, "Hey, wait a second. [2:08] Does this agent actually look out for me? Does it  care about what I'm doing? Or is it and care about [2:12] my well-being or is it really just looking out for  Etsy and trying to drive more revenue for Etsy at [2:17] my expense? Um, and so there's these subtle ways  that uh even if the agent is behaving as expected, [2:25] right, it's doing what it was optimized to  do, it is providing the right recommendations, [2:29] taking the right actions. Um, you can imagine  eroded trust from a seller where the next time [2:34] the recommendation comes in, they don't take it  anymore, right? and maybe they don't even use [2:38] the agent anymore. Um, and so there's  these long tale of sort of subtle [2:44] kind of ways in which trust can be eroded even  when the outcome is what you expected it to be. [2:52] Um, and I want to talk a little bit  about why we think user trust should [2:55] be a first class consideration in AI product  development and agentic product development. [3:00] And it's largely about user trust not  being something that's just downside risk, [3:06] right? It's not something like cyber security  where you say, well, if you lose user trust, [3:10] that's a big deal and um you know, sellers in  our case won't be using a product anymore. It's [3:15] also something that you can get a lot of upside  from, right? more seller trust, more user trust, [3:22] uh allows for even more deep engagement, even  more um new opportunities, and we can unlock [3:28] even more value for our customers, right?  And so, going back to the ad budget example [3:34] if the seller trusts uh the agent, maybe it  takes the recommendations. If it really trusts [3:40] the agent, trusts it even more. Um, you can  imagine it taking uh, you know, asking for [3:46] even more recommendations, following up, using the  agent for all kinds of other use cases as well. [3:51] And I think another idea that really illustrates  this is no matter how sort of AI forward you think [3:56] you are, how agentic, you know, fluent you think  you are, there are people who are more uh, AI [4:05] fluent or interested, right, or trusting. Um, I  saw a talk earlier that said how I use 150 agents [4:12] to run my life, right? You have people who are  giving agents credit card numbers, social security [4:16] numbers, authorizing it to make transactions,  authorizing it to open bank accounts. Um, [4:21] it's unlikely that you're doing that. And the  difference is that rightly or wrongly, those [4:25] people believe uh, you know, they trust the agent  more than you do, right? Otherwise, maybe you [4:31] would be open to doing something similar. Um, and  so again, it's not a binary thing. you can always [4:37] uh build and maintain and deepen uh trusts  that you have with um with your users. [4:44] Cool. So hopefully this all sounds really obvious  uh and intuitive and if [snorts] not hopefully my [4:49] two slides and little spiel has convinced you  uh otherwise that this is you know something [4:55] important and worth investing in. And so there's  three layers to how we think you can do this. The [4:59] first is evaluations which I'm sure you all heard  about at Nauseium but I think we have a slightly [5:05] different angle on it. Um and this is really the  technical infrastructure layer for user trust. [5:11] We have gated releases which is the organizational  layer of um the infrastructure for user trust and [5:19] we have a curriculum of responsible AI which is  really the cultural layer within your organization [5:24] that ensures that um you know all of the above  can be supported. Cool. So, evaluations. Um, [5:33] there's a long- winded tweet here that I think  is worth kind of reading out loud, and I'll [5:37] explain why I think it's relevant in a moment.  Um, see which side it's a little easier. U, okay, [5:44] so I'm significantly older than you. I started  coding in the late '60s. My current strategy is [5:48] not to read any of the code written by my agents.  That's the only way I can take advantage of their [5:52] productivity. What I do instead is to surround  the agents with extreme constraints, all kinds [5:57] of tests. In the end, I have very high confidence  in the code they produce because they've had to [6:01] run the gauntlet of all my constraints and tests.  Okay, so this is Robert Martin. He's the author of [6:08] seminal coding textbook clean code. He's the  co-author of the agile manifesto, a legendary [6:13] programmer. And why is this relevant? um because  the domain the use case in which agents have been [6:21] most widely adopted the frontier of agentic use is  really in writing software right and so what we we [6:27] it's instructive to look at people at the  forefront of using agents for writing software [6:34] and learn from them about best practices standards  uh and approaches to ensuring that you can [6:40] uh you know take advantage of agents to  the fullest extent within your domain Um, [6:46] and you can see as he talks about it's uh eval and  guardrails, which again I know we probably heard [6:54] all day today, eval guardrails, eval guardrails,  and you'll continue to hear about it. But I think [6:59] often when we talk about eval guardrails, we're  talking about capability, right? Making sure [7:03] that the agents are doing what we expect,  they're doing the right thing. And we are [7:07] talking not about is the agent doing what we  think, but is the agent trustworthy, right? So [7:13] often we're thinking about how smart is this, how  capable is this. The headlines when a new model is [7:19] released on one of the frontier labs is all about  how much smarter the new uh AI is, right? The new [7:26] math theorem that it's able to solve, right? All  about capability. And we don't see headlines, [7:30] although I'm sure there are people within the  labs who might disagree with this about uh [7:35] trustworthiness, right? Is the agent honest?  Um is the agent consistent? Does it stay [7:41] within boundaries? Um, does it treat all all  users fairly? Do people think it treats all [7:46] users fairly? What are the different impacts to  different groups of people from the agent um or [7:51] the AI? And so these are all things that we think  are actually really important in the evaluation [7:58] process and in the guardrail process and not  just thinking about outcomes and capabilities. [8:03] So how do you do this? Uh it's not rocket science.  You start with a thorough risk assessment. [8:09] um you bring together subject matter experts,  domain experts across your organization and you do [8:13] a brainstorm of what the possible risks are uh  to your users. Um we do it by first coming up [8:21] with a list of categories that we think exhaust  possible risk types and then brainstorming what [8:26] are all the risks within that. And so just to give  you a sense of scale for us in our seller agent, [8:31] we have 12 risk categories and 70  distinct failure modes within them. [8:36] Cool. And then each risk is scored uh across a  number of dimensions like how frequently do we [8:41] expect this to occur if it were to occur how big  of a deal would it be? How many people would be [8:45] impacted by it? What's the extent of that impact?  You know you could think about what the financial [8:50] implications are etc. Um and altogether you can  kind of come up with some sort of label right [8:57] low, medium, high or in you know this  hypothetical catastrophic. Um and then based on [9:05] that label, you have to think about what is the  appropriate guardrail or mitigation strategy for [9:11] uh ensuring that that risk doesn't come to be. Um  you know, for something that's low or medium risk, [9:16] maybe you can ignore it. Uh in our case, you  know, we don't want to ignore anything. And [9:21] so we think about what are some very low lift  ways to mitigate, right? Things like banners, [9:27] hedged language within terms and conditions,  right? That could be something that's fine for [9:31] a lowrisk uh low frequency kind of item, but  something that's as big a deal as potentially [9:37] cross- seller leakage. You really want to make  sure that you have lots of controls throughout [9:41] the agent workflow, checking in, making sure  that this is not something that can happen. [9:50] Cool. And then every one of these  mitigations needs its own evaluation, [9:53] right? So you've come up with the ris, you've  identified the risk, you've scored the risk, [9:56] you've come up with a mitigation strategy,  you've implemented a mitigation strategy, [10:00] and now you need to know, well, how often is  this risk actually happening? Did we make it up? [10:05] Were we right about what the potential frequency  is? Does do we need to pay more attention to it? [10:10] And also, is the mitigation catching it? Um, you  know, the the controls you put in place, you felt [10:15] really good about, are they actually catching  it or are lots of issues still coming through? [10:25] Cool. And uh we don't think you should skip or  outsource this. I think you know tools like Sierra [10:31] outside consultancies are great for even building  your agent, evaluating capability and outcome, [10:36] but they're not going to be experts on your  users, how they think about trusting your product, [10:41] what they think about the risks, what you know of  the risks in the product. And so we really think [10:45] this is something you should do in-house regularly  catching up with um all sorts of experts and [10:51] do in domain experts within your company. Uh  and with that I want to hand it to Beth who will [10:57] talk about the uh organizational and cultural  infrastructure layers that support this work. [11:15] Yeah, I'm going to start off talking about  gating, which is uh one of the strategies that we [11:21] used to control releases. Um, we're trying to  understand, we wanted to design and develop a [11:29] process for how we could map our understanding  of risk to populations where we can learn and [11:34] and mitigate risk. Um, a a question that's  always on my mind as an engineering manager is [11:41] how do I um am at any given time are we using  our team's resources most effectively to deliver [11:48] impact for customers? And so we want a release  strategy that is both um setting us up to do [11:55] that by default um and then managing this balance  that Tor has been talking about of moving quickly [12:03] while um simultaneously building trust and and  developing those two things in concert. Uh the [12:13] process that we developed with gating um is really  one of sequencing and mapping our understanding of [12:20] risk and the open questions that we have about  what actually is impactful and valuable to [12:25] to buyers into different thresh um like  release thresholds and um scaling moments. [12:33] In order to develop that strategy, we're doing  a prioritization and sequencing uh process where [12:39] we're ranking risks. We're then matching um and  mapping those to uh the um different populations [12:48] that we can target. And then, you know, all of  the richest learning that we have is when products [12:52] are actually out in the wild with sellers.  And so that's where our learning um happens. [12:59] each release uh moment, each gate um is forcing  two questions. Does it deliver the value that [13:08] we expect? And then are we mitigating the risk  that we've wanted to tackle at that level with [13:14] that population? Uh and so for every gate that  we've defined, we articulate what we the hypo [13:22] hypothesis, which is basically like our theory  of of user impact. Um the open questions that [13:27] we want to answer. We define exit criteria that  are provide like a bar for when we can expand. Um [13:34] and then uh risk guard rails which are really  kind of a specialized type of um exit criteria. [13:42] And what that's looked like for us is um  in in our most recent feature like four [13:48] four phases of scaling. We started really  small um with like a very rich research um [13:54] alpha experiment. Then we  scale to a thousand users [13:59] um and we cycle up through 25 um thousand  um eventually to like a general release. [14:08] The last pillar um of our uh like  strategy for building trust and developing [14:14] infrastructure to support that is is more on  the cultural side through a curriculum that [14:19] we've developed and work with our team um use  uh to work with our team. The problem that we [14:26] were trying to solve here is that as we got more  familiar with this domain, it became clear how [14:32] how many consequential decisions were falling  to people that were like on the ground making um [14:39] decisions in code as they're writing evaluations.  Um and we wanted to equip those folks to use um [14:47] the best judgment possible. Um those moments,  the gotchas are subtle. uh you can't um [14:56] account for everything, but you need to be able  to lift up patterns that are going to help you [15:00] strategically address the risk that you're  trying to account for. Um and it's really like [15:05] the development of a craft to do this well.  And [sighs] so we anchored our curriculum [15:11] around this idea that we're trying to build moral  attention as a team practice. Uh the goal is to [15:16] build the habit of flagging risk, assessing  it skillfully, uh developing shared language [15:21] for hard calls with the idea that this compounds  over time and becomes part of how the team works. [15:29] Uh we use a case-based approach. The premise  here is that uh when you give people rules, [15:37] they delegate their decision-m to a rule. Um,  but we want people exercising their judgment in [15:46] situations where information is imperfect and we  want them to feel empowered and really explicitly [15:51] understand that like this is their moment and  we've been training for this, you know, and um, [15:57] lean into it and and um, like we're counting on  you showing up uh, meaningfully in this moment. [16:06] There's a precedent for this across a bunch  of industries and and spaces, medicine, law, [16:10] military ethics. Uh some important um features  like this is happening in community. These are [16:17] facilitated conversations. They're not just like  e-learning modules. Um and so far it's been uh [16:25] like a pretty rich process. Uh I think  we were pretty excited to learn that [16:31] judgment is actually like a skill that you can  help develop. The qualities that we've focused on [16:38] the like the specific skills that we've  focused on are um the skill of identifying [16:43] under unrepresented stakeholders who matter um  which is really a question of like imagination [16:50] and being ready and willing to ask that  question. Um failure mode recognition which [16:56] I'm going to talk about in a second. Designing  for disscent which is really a like team practice. [17:02] We have implemented premortems around  releases um and are trying to kind of sanction [17:08] the work uh not sanction like u um promote the  work and role of someone saying hey let's pause [17:17] for a second and think about the consequences of  that we want to design for those conversations to [17:23] happen and then the last is one that I found  quite interesting it's the idea of like um [17:29] tacted maybe I think The anti-attern is like that  when we're talking about risk, you're the person [17:35] that shuts down the conversation. And what we  really want everyone on the team to be equipped to [17:40] do is be the person like you every team like wants  you in the room because you are going to be um [17:48] have a good radar for things that need to  be discussed. And you're not just going to [17:52] drop them on the table as problems, but you're  going to really um bring solutions to and bring [17:58] a wealth of experience that helps the team  navigate a difficult decision um skillfully. [18:04] And then the failure mode recognition um just  going to call out one of these like given time [18:10] compliance as a ceiling. This is another learning  like interesting learning for us. I think so much [18:16] of the industry right now is anchoring on  regulation and laws to define for themselves [18:23] what good work is, but um this technology is  developing so quickly uh that those things don't [18:32] give you a good understanding of what can actually  go wrong. And so if that's the main thing you're [18:36] anchoring to, you're going to be missing a lot of  possibility for harm um and and erosion of trust. [18:45] So wrapping this all up, um we think this  suite of practices and infrastructure position [18:55] us to really be proactive in understanding  what can go wrong and and and mitigating it. [19:01] Um and doing that while we're working with urgency  to deliver meaningful uh impact to our users. [19:09] The real like selling point here is that when you  build this infrastructure from the beginning that [19:15] it's just there for you to fall on and lean on. Um  when you learn new things, when you make mistakes [19:20] and and when you encounter them, you use those  same rails, fortify them, um and move forward. [19:28] And this ultimately allows you to ship faster.  So you can experiment more boldly, release more [19:33] confidently, um, and find issues first before  you burn them in ways that are going to be really [19:38] expensive and painful to repair. Um, repairing  trust after you've broken it is the slowest. [19:48] Um, I mean, uh, like skip the vote. I think the  ma the main thing that we were really wanting to [19:56] like elevate with this poll of how how confident  are you that your um safeguards are are strong [20:02] enough is that like using evidence to actually  check your gut is like going to um save you [20:10] and all the infrastructure that we've talked  about as a way to actually do that at scale um and [20:16] with a degree of rigor that can match the  complexity of the products that we're building. [20:24] I think I'll leave it there. Thanks everyone.