Tech Infrastructure Drives AI Child Abuse Surge, Zero Abuse Warns

AI Harm Is Everyone's Business: Why Every Industry Needs to Understand Where Its Responsibility — Ai4

Ai4

Tech companies must actively police their hosting, distribution, and payment rails because systemic handoff failures across the commercial supply chain enable bad actors to generate and monetize illicit material before law enforcement can intervene.

AI-assisted summary of AI Harm Is Everyone's Business: Why Every Industry Needs to Understand Where Its Responsibility — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 15, 2026, this presentation features Stephanie Hall, Technology & Innovation Lead at the Zero Abuse Project. The session description explains that AI-facilitated harm moves through an interconnected commercial supply chain—exemplified by AI-generated child sexual abuse material—and offers a practical framework and three core questions to help organizations in any industry locate their responsibility.

## TL;DR Stephanie Hall argues that AI harm does not occur in a silo, but across a multi-stage commercial supply chain spanning model creators, hosting platforms, wrapper applications, payment processors, and downstream responders. Focusing on the exponential surge of AI-generated child sexual abuse material—which surged to 1.5 million NCMEC reports in 2025—she explains how synthetic media breaks traditional forensic hash-matching and strains law enforcement triage. Through real-world case studies like the San Francisco app store enforcement and direct cross-sector collaboration with Cash App, she illustrates how upstream and intermediate platforms hold immense leverage to disrupt harm. Ultimately, she challenges enterprise leaders to evaluate their product touchpoints, build necessary child safety capabilities, and establish clear operational handoffs.

## Atmosphere & style The presentation carries a grounded, pragmatic, and urgent tone, balancing an appreciation for AI's productivity benefits with the sobering realities faced by child protection investigators. Rather than engaging in theoretical AI alignment debates, Hall delivers an authoritative, clear-eyed examination of real-world operational bottlenecks and systemic vulnerabilities. Listening to the talk feels like an actionable masterclass in digital forensic accountability, transforming complex multi-stakeholder failures into concrete, structured governance challenges.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=iyrrI86EYlY&t=4s) AI harm operates across an interconnected commercial supply chain—spanning foundation models, cloud hosts, wrapper tools, app marketplaces, payment gateways, and triage teams—rather than in isolated silos. - [03:53](https://www.youtube.com/watch?v=iyrrI86EYlY&t=233s) Generative AI tools have lowered the technical barrier to offending, enabled instantaneous bulk creation of abusive material, and produced novel cryptographic hashes that evade traditional perceptual hash databases. - [03:53](https://www.youtube.com/watch?v=iyrrI86EYlY&t=233s) Investigators are forced to spend critical operational hours discerning synthetic imagery from real-world photography, directly competing with the time needed to locate live children in immediate danger. - [07:36](https://www.youtube.com/watch?v=iyrrI86EYlY&t=456s) Reports of GenAI-facilitated exploitation sent to NCMEC exploded exponentially from 4,700 in 2023 to 67,000 in 2024 and 1.5 million in 2025, overwhelming human triage and investigative queues. - [07:36](https://www.youtube.com/watch?v=iyrrI86EYlY&t=456s) AI-enabled child exploitation extends beyond static generation to active abuse vectors, such as coercive sextortion, peer-to-peer schoolyard deepfakes, abuse by trusted authority figures, and synthetic grooming personas. - [10:53](https://www.youtube.com/watch?v=iyrrI86EYlY&t=653s) Intermediaries that govern discovery, hosting, and monetization—such as app stores and payment processors—hold actionable responsibility in the supply chain even if they did not build the underlying AI algorithm. - [14:43](https://www.youtube.com/watch?v=iyrrI86EYlY&t=883s) Data handoff failures impede legal accountability, as over 10% of compliance reports submitted by tech platforms lack basic jurisdictional and identifying data necessary for law enforcement action. - [14:43](https://www.youtube.com/watch?v=iyrrI86EYlY&t=883s) Building direct connective tissue between corporate investigation teams and law enforcement, as shown by Cash App and the Zero Abuse Project, can turn opaque telemetry into actionable leads. - [19:01](https://www.youtube.com/watch?v=iyrrI86EYlY&t=1141s) Leadership across all tech sectors must address three governance questions: whether products facilitate or disrupt harm, what link-specific safety capabilities are required, and who owns the operational handoff when abuse occurs. - [19:01](https://www.youtube.com/watch?v=iyrrI86EYlY&t=1141s) Effective child protection in the age of generative AI demands proactive upstream architectural guardrails, data preservation, and cross-industry signal sharing rather than reactive post-incident remediation.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=iyrrI86EYlY&t=4s) Introducing the Zero Abuse Project and the AI Supply Chain of Harm

Stephanie Hall introduces herself as the Technology and Innovation Lead at the Zero Abuse Project, a national nonprofit organization dedicated to ending all forms of child abuse and sexual exploitation. A core operational pillar of the organization is training the investigators and prosecutors who handle technology-facilitated crimes against children—a role that, over the past three years, has increasingly pivoted toward investigating artificial intelligence-facilitated offenses. Hall sets expectations early by clarifying that she is not anti-technology; in fact, she relies on AI every single day to give her nonprofit team the bandwidth and reach they could never afford with limited funding and staffing alone. The presentation is not about painting AI as inherently evil, but about understanding what occurs when AI harm—specifically AI-generated child sexual abuse material (AIC-CSAM)—travels through digital ecosystems.

The core premise of the talk is that just as every commercial product has a supply chain, "harm has one too." When AI is weaponized or misused to inflict harm, that damage does not occur within a single siloed company. It moves across an entire sequence: one organization trains the foundational model, another hosts it, a third builds a wrapper application, app marketplaces list it, payment gateways process transactions, platforms circulate the output, and downstream agencies are left triaging the aftermath. In traditional AI safety tracks, the conversation fixates heavily on the upstream model itself—evaluating guardrails, alignment, and jailbreak behavior during misuse. However, real-world law enforcement cases rarely stem from an autonomous model acting maliciously on its own; they involve ordinary bad actors using accessible tools across a commercial chain of companies that enable, distribute, monetize, detect, and report.

```mermaid flowchart LR accTitle: The Supply Chain of AI Harm

subgraph Upstream["Upstream Tech"] M["Model Developers<br/>& Hosting"] W["Wrapper Apps<br/>& Tool Builders"] end

subgraph Distribution["Distribution & Commerce"] A["App Stores &<br/>Search Engines"] P["Payment Processors<br/>& Platforms"] end

subgraph Downstream["Response & Triage"] TS["Trust & Safety<br/>& NCMEC"] LE["ICAC Investigators<br/>& Prosecutors"] end

M --> W W --> A A --> P P --> TS TS --> LE ```

---

[03:53](https://www.youtube.com/watch?v=iyrrI86EYlY&t=233s) Anatomy of the Harm Supply Chain and Changing Vectors of Exploitation

When an AIC-CSAM case lands on the desks of Internet Crimes Against Children (ICAC) task force investigators trained by the Zero Abuse Project, it has already traveled across multiple commercial touchpoints. The typical pipeline initiates when ordinary, non-sensitive source material—such as an innocuous student portrait scraped from a high school website or Facebook profile—is fed into an image-manipulation tool powered by cloud infrastructure. Search engines and mobile app stores facilitate discovery, online platforms host the generated output, and fintech payment rails monetize the transactions. When Trust & Safety teams eventually flag and report the material to the National Center for Missing & Exploited Children (NCMEC), the reports get routed to ICAC investigators. Because these investigators sit at the very end of the line, AI has introduced staggering volume and nuance into an investigative system that was already severely overburdened.

The introduction of generative tools has fundamentally altered the threat landscape across three operational dimensions:

- **Who can offend:** Abuses that historically required direct physical access to a child can now be orchestrated by anyone scraping publicly available photos with off-the-shelf consumer software, causing technical barriers to virtually vanish. - **Volume of generation:** Rather than possessing a static, finite batch of illicit media, a single offender can generate thousands of unique, synthetic abusive images in minutes. - **Forensic detection criteria:** Traditional detection workflows rely on matching perceptual file hashes of known illegal imagery against law enforcement databases. First-generation synthetic images produce novel cryptographic hashes that yield no historical matches.

Consequently, investigators are forced to open every new case by answering a time-consuming question: "Is this a real child?" Every operational hour an investigator spends discerning synthetic imagery from real-world photography represents critical time lost in locating a live child who may be facing immediate physical peril.

---

[07:36](https://www.youtube.com/watch?v=iyrrI86EYlY&t=456s) Exponential Surge in GenAI Exploitation and Emerging Offense Patterns

While AI-driven harm also manifests across domains like financial fraud, corporate impersonation, and non-consensual deepfakes of adults, child exploitation carries immediate, life-altering fallout. The explosive growth of GenAI-related reports submitted to NCMEC illustrates an unprecedented operational bottleneck:

- **2023:** 4,700 GenAI-related reports received - **2024:** 67,000 GenAI-related reports received - **2025:** 1.5 million GenAI-related reports received

In daily operational terms, the report volume surged from roughly a dozen reports per day to well over a thousand each day, flooding the human review queues responsible for triaging, routing, and initiating investigations.

Furthermore, AI-facilitated offenses manifest in diverse, deceptive ways across the field:

- **Financially Motivated and Coercive Sextortion:** When a minor refuses to produce explicit images, bad actors create synthetic, compromised media from benign photos and use the fabricated material to extort the victim for money, additional real imagery, or sexual compliance. - **Peer-to-Peer Schoolyard Abuse:** In March, two Pennsylvania teenagers were sentenced after orchestrating the creation of approximately 350 AI-generated explicit images depicting 59 female classmates. - **Abuse by Trusted Authority Figures:** In Mississippi, a former middle school teacher scraped social media photos of his own students to produce sexually explicit synthetic videos of eight girls aged 14 to 16, resulting in a guilty plea and a five-year prison sentence. - **Synthetic Grooming and Impersonation:** Predators deploy synthetic personas—leveraging AI-generated profile photos, clone voices, and coordinated accounts—to impersonate peer-aged children and groom minors online.

---

[10:53](https://www.youtube.com/watch?v=iyrrI86EYlY&t=653s) Real-World Case Studies: UK School Extortion and the SF App Store Actions

To illustrate the chain of responsibility, Hall examines a cyber extortion case that unfolded in May at a UK secondary school. A criminal gang scraped ordinary student photos from the school's public website and social media channels, processed them through an AI generator, and produced roughly 150 sexually explicit images. Because the media mapped directly onto identifiable, living children, the UK Internet Watch Foundation (IWF) classified the outputs as child sexual abuse material under UK law—a standard similarly upheld for prosecution under United States federal law. The perpetrators sent samples directly to school administrators alongside a ransom ultimatum: pay money or the images would be published across the public web. The school immediately alerted law enforcement, who engaged the IWF to generate digital perceptual hashes and distribute them to major tech platforms to preemptively block dissemination. In this instance, distinct organizations held the source material, the generative software, the extortion channel, the response triage, and the downstream blocking, proving that no single entity managed the pipeline from end to end.

Two months later, on July 17, an enforcement action demonstrated what intervening upstream on the supply chain looks like. The San Francisco District Attorney issued formal cease-and-desist letters to Apple and Google regarding 13 face-swapping and "nudify" applications distributed across their app marketplaces. The legal notices underscored that the tech giants were not merely hosting downloadable software; they were processing the underlying in-app payments and extracting platform revenue shares.

While neither Apple nor Google wrote the underlying algorithms or rendered the illicit images, they held absolute authority over storefront access, algorithmic discovery, and monetization rails. Both companies acted swiftly ahead of the statutory deadline by terminating offending developer accounts, removing the apps, and suppressing search keywords. Hall emphasizes the core takeaway: "You don't have to be the one who created the harm to be standing in the supply chain."

---

[14:43](https://www.youtube.com/watch?v=iyrrI86EYlY&t=883s) Data Quality Failures, the Cash App Partnership, and Broken Handoffs

Reaching the tail end of the supply chain rarely yields a tidy, ready-to-prosecute dossier. Instead, investigative fragments are scattered across corporate silos: one vendor maintains the suspect account credentials, a payment processor stores the billing metadata, and a hosting service logs the network telemetry. Preserving and translating this data across organizational boundaries is critical. Of the 1.5 million GenAI-related reports submitted to NCMEC in 2025, over 400,000 were actionable, and more than 182,000 specifically involved the creation, attempted creation, or possession of AIC-CSAM. Hall notes that forensic trends reveal individuals generating synthetic CSAM frequently harbor real-world abusive material of actual children on their physical hardware.

Alarmingly, more than 10% of reports submitted by technology companies lacked foundational identifying data, such as basic jurisdictional information, leaving law enforcement unable to route them. Compliance cannot be reduced to merely checking a regulatory reporting box; reports must provide sufficient fidelity for investigators to take legal action.

```mermaid flowchart TD accTitle: Cash App and Zero Abuse Project Handoff Integration

subgraph Problem["The Handoff Gap"] T["Cash App Files ~450<br/>CyberTips / Month"] I["Investigators Struggle<br/>to Interpret Complex Data"] T -. Friction & Delay.-> I end

subgraph Solution["Direct Coordination"] D["Dan Berry (Zero Abuse Project)<br/>Initiates Contact"] S["Block Inc. Dedicated<br/>Special Investigations Team"] D <==> S end

subgraph Outcome["Operational Result"] W["Joint Training, Webinars &<br/>Actionable Financial Tracing"] end

Problem --> Solution Solution --> Outcome ```

A positive example of closing this handoff gap occurred with Block Inc., the parent entity of Cash App. Handling approximately 450 NCMEC CyberTips monthly, Cash App sat squarely on the monetization link. At ICAC training events, law enforcement personnel consistently voiced frustration that the financial cyber tips were structurally opaque and difficult to interpret forensically. Zero Abuse Project's lead ICAC investigative specialist, Dan Berry, reached out directly to Block's leadership.

The outreach revealed that Block already maintained an internal special investigations team dedicated to combating CSAM on their rails, but field investigators were unaware of its existence. By bridging this operational divide through direct collaboration, webinars, and specialized training, the two groups turned raw payment telemetry into an effective tool for identifying suspects and locating victims. As Hall points out, a handoff fails not necessarily because an individual link failed its isolated duties, but because the connective tissue between links was neglected.

---

[19:01](https://www.youtube.com/watch?v=iyrrI86EYlY&t=1141s) Three Critical Questions for Leadership and Strengthening the Chain

Hall leaves tech leaders, engineers, and platform operators with three fundamental governance questions to bring back to their executive teams:

- **Could our products enable, facilitate, detect, or disrupt harm?** Organizations must look past intended feature design and evaluate how bad actors might repurpose their tools inside an adversarial pipeline. - **What child safety capabilities do we need at our link?** Depending on where a company sits, this might require explicit terms-of-service policies, red-teaming and adversarial misuse testing, safety-by-design UI constraints, robust automated detection, or enhanced data-retention workflows. - **When harm surfaces, who owns the next action?** Teams must define explicit operational triggers, specify forensic artifacts that must be legally preserved, and identify exact law enforcement contact pathways so that down-funnel recipients can act immediately.

Cross-industry initiatives like the Tech Coalition's Lantern program demonstrate the power of collaborative intervention by enabling real-time, cross-platform signal sharing to disrupt child exploitation networks across distinct platforms. Hall reminds the audience that even if an enterprise does not cater directly to children, its infrastructure can still be co-opted into a pipeline of abuse. The Zero Abuse Project actively assists companies by auditing internal child safety policies, providing specialized AI-exploitation training to operational teams, and coordinating corporate sponsorships for investigator and prosecutor education. In closing, Hall reinforces that child safety is not merely a reactive post-incident cleanup exercise; it demands designing upstream architectures, policies, and operational handoffs that systematically dismantle the supply chain of harm before it begins.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=iyrrI86EYlY&t=4s) Every product has a supply chain, and it turns out harm has one too. — Stephanie Hall

> [00:04](https://www.youtube.com/watch?v=iyrrI86EYlY&t=4s) When AI is weaponized or misused to inflict harm, that damage does not occur within a single siloed company. — Stephanie Hall

> [03:53](https://www.youtube.com/watch?v=iyrrI86EYlY&t=233s) Is this a real child? — Stephanie Hall

> [03:53](https://www.youtube.com/watch?v=iyrrI86EYlY&t=233s) Every operational hour an investigator spends discerning synthetic imagery from real-world photography represents critical time lost in locating a live child who may be facing immediate physical peril. — Stephanie Hall

> [10:53](https://www.youtube.com/watch?v=iyrrI86EYlY&t=653s) You don't have to be the one who created the harm to be standing in the supply chain. — Stephanie Hall

> [19:01](https://www.youtube.com/watch?v=iyrrI86EYlY&t=1141s) Could our products enable, facilitate, detect, or disrupt harm? — Stephanie Hall

> [19:01](https://www.youtube.com/watch?v=iyrrI86EYlY&t=1141s) What child safety capabilities do we need at our link? — Stephanie Hall

> [19:01](https://www.youtube.com/watch?v=iyrrI86EYlY&t=1141s) When harm surfaces, who owns the next action? — Stephanie Hall

## Mentioned Stephanie Hall, Dan Berry, Zero Abuse Project, National Center for Missing & Exploited Children (NCMEC), Internet Crimes Against Children (ICAC), Internet Watch Foundation (IWF), Tech Coalition, Lantern, Apple, Google, Block Inc., Cash App, Facebook

## Video description Stephanie Hall - Technology & Innovation Lead, Zero Abuse Project

Every product has a supply chain, and it turns out harm has one too.

When AI is used to cause harm, it moves through a chain of companies: one builds the tool, one hosts it, one sells it, one processes the payment, one's platform spreads it, and someone far downstream responds to the damage. This talk maps that chain using one of the clearest and most urgent examples available: AI-generated child sexual abuse material.

I serve as Technology and Innovation Lead at Zero Abuse Project, a national child protection nonprofit, where I develop AI training for the investigators, prosecutors, and child protection professionals who respond at the end of that chain. In my field we say that child abuse prevention is everyone's business, and this session extends that same principle to AI: attendees will learn a practical framework for locating their organization's link on any chain of AI harm, plus three questions to bring back to their leadership.

Whatever your industry, your organization sits somewhere on one of these chains. This session will help you find your link and understand what responsibility looks like there.

[Source](https://www.youtube.com/watch?v=iyrrI86EYlY)

中文简报

Zero Abuse警告：科技基础设施助推AI儿童虐待激增 科技公司必须主动监管其托管、分发和支付渠道，因为商业供应链中系统性的交接脱节使不法分子得以在执法部门介入前生成非法内容并获利。 Stephanie Hall指出，AI危害并非孤立发生，而是贯穿于由模型创建者、托管平台、封装应用、支付处理商及下游响应方组成的多阶段商业供应链中。针对AI生成的儿童性虐待材料呈指数级激增（2025年向NCMEC提交的报告激增至150万份），她解释了合成媒体如何打破传统的司法哈希比对并加剧执法分流筛选的负担。通过旧金山应用商店执法行动以及与Cash App开展跨行业直接合作等现实案例，她阐明了上游与中介平台在阻断危害方面具备的巨大影响力。最后，她呼吁企业领导者评估其产品触点，构建必要的儿童安全能力，并建立明确的业务交接流程。 AI 造成的危害跨越了相互关联的商业供应链——涵盖基础模型、云托管、包装应用、应用商店、支付通道和应急响应团队，而非局限于单一孤岛。 (0:04) 生成式 AI 工具大幅降低了犯罪的技术门槛，使批量即时生成虐待材料成为可能，并产生全新的哈希值以规避传统的已知非法图像感知哈希数据库。 (3:53) 提交给美国国家失踪与受虐儿童中心（NCMEC）的生成式 AI 相关举报呈指数级激增，从 2023 年的 4700 起猛增至 2024 年的 6.7 万起，并在 2025 年激增至 150 万起，使人工分流系统不堪重负。 (7:36) 应用商店和支付处理商等掌控发现、托管和变现的中间服务商，即使并未构建底层 AI 算法，也对其在供应链中所处环节承担着切实的责任。 (10:53) 数据交接缺陷严重阻碍了司法追责，科技平台提交的合规举报中有超过 10% 缺乏执法行动所需的基本管辖权和身份识别数据；而 Cash App 与 Zero Abuse Project 的合作表明，打通企业调查团队与执法部门的直接联系能将模糊的遥测数据转化为有效线索。 (14:43)

Reading summary

Stephanie Hall of the Zero Abuse Project argues that artificial intelligence-facilitated child exploitation operates across a commercial 'supply chain of harm' linking model developers, wrapper applications, app stores, and payment rails. Because abuse relies on this multi-layered ecosystem rather than isolated models acting autonomously, Hall contends that every participating industry must take active responsibility rather than leaving detection and triage to downstream law enforcement. The Commercial Pipeline Enabling Synthetic Child Abuse (0:04) Traditional AI safety debates concentrate heavily on upstream foundation models, such as guardrails and jailbreaks. In real-world investigations, however, harms such as AI-generated child sexual abuse material (AIC-CSAM) materialize through a chain of commercial actors. A bad actor typically scrapes benign public imagery, runs it through cloud-hosted wrappers, discovers tools via search engines or app stores, and monetizes transactions through fintech payment gateways. Downstream trust and safety teams and law enforcement agencies are left to absorb the resulting crisis. Intervention can occur anywhere along this pipeline. For example, in July, the San Francisco District Attorney issued cease-and-desist notices to Apple and Google targeting 13 'nudify' and face-swapping applications. While neither platform developed the code or rendered the abusive imagery, both acted as critical gatekeepers by facilitating storefront discovery, executing in-app billing, and collecting revenue shares. Both companies responded ahead of statutory deadlines by terminating developer accounts and removing the applications, illustrating how intermediate platforms hold the leverage to sever distribution. Operational Bottlenecks and Surging Synthetic Report Volumes (7:36) Generative AI has fundamentally shifted exploitation patterns across three dimensions: it eliminates barriers to offending by allowing individuals without physical access to children to manufacture abuse; it enables single perpetrators to produce thousands of synthetic images in minutes; and it disrupts traditional forensic workflows. Because newly generated synthetic images produce unique cryptographic hashes that do not match existing law enforcement databases, investigators must inspect every file to answer whether it depicts a real, living child, draining critical hours away from locating children in immediate physical danger. This dynamic has triggered an unprecedented surge in reporting. Reports of GenAI-related exploitation submitted to the National Center for Missing & Exploited Children (NCMEC) jumped from 4,700 in 2023 to 67,000 in 2024, and reached 1.5 million in 2025—moving from roughly a dozen reports per day to more than a thousand daily. Offense models have expanded beyond traditional abuse to include sextortion rings targeting schools (such as a UK extortion scheme generating 150 explicit student images), peer-to-peer classroom abuse, exploitation by authority figures, and synthetic grooming personas. Data Quality Deficits and Inter-Organizational Handoffs (14:43) A critical vulnerability in mitigating exploitation occurs during the handoff of digital evidence between corporate platforms and law enforcement. Of the 1.5 million GenAI reports filed with NCMEC in 2025, over 400,000 were actionable, and more than 182,000 involved the creation, attempted creation, or possession of AIC-CSAM. Hall emphasizes that individuals generating synthetic abuse often also possess real-world abuse material on their hardware. However, over 10% of reports submitted by tech platforms lacked basic identifying or jurisdictional details, rendering them unrouteable. A collaborative resolution to this handoff failure occurred between the Zero Abuse Project and Block Inc. (parent company of Cash App). Cash App was filing approximately 450 NCMEC tips per month, but investigators struggled to interpret the raw financial telemetry. When Zero Abuse Project's investigative specialist reached out, the two organizations aligned field investigators directly with Block's internal special investigations unit. Through joint training and tailored workflows, complex payment telemetry was converted into actionable forensic evidence, proving that broken handoffs can be repaired through operational coordination. Operational Governance and Cross-Industry Intervention Frameworks (19:01) Hall urges enterprise executives and engineers to integrate child safety directly into system architectures by addressing three operational questions: whether products could enable, facilitate, detect, or disrupt abuse; what baseline child-safety capabilities must exist at their specific link in the chain; and who holds explicit ownership of handoffs and legal evidence preservation when abuse is detected. Capabilities range from adversarial red-teaming and UI constraints to cross-platform signal sharing initiatives like the Tech Coalition's Lantern program. The Zero Abuse Project actively assists companies by auditing internal safety policies, delivering exploitation training to technical teams, and coordinating corporate sponsorships for law enforcement education. Hall stresses that companies whose products do not directly target children are still vulnerable to having their infrastructure weaponized, making upstream preventative architecture a core governance requirement. Hall concludes that child safety can no longer be treated as a reactive, downstream cleanup task left to overwhelmed investigators. Because modern digital harm relies on multi-vendor commercial infrastructure, preventing exploitation requires tech companies at every tier to design adversarial guardrails, ensure high-fidelity reporting, and maintain coordinated handoffs across the supply chain.

精炼阅读摘要

Zero Abuse Project 技术与创新主管 Stephanie Hall 指出，人工智能生成的儿童性虐待材料（AIC-CSAM）等危害并非孤立存在，而是依托于由模型训练、工具封装、应用分发、支付结算及平台传播构成的商业“危害供应链”；企业即便未直接开发有害算法，只要身处该链条的分发或变现环节，就对遏制技术滥用承担关键责任。 危害供应链模型与安全重心的错位 (0:04) 传统的人工智能安全讨论通常聚焦于上游的基础模型开发，重点评估模型对齐、安全护栏以及越狱攻击。然而，在实际执法案例中，数字侵害极少源于自主恶意运行的模型，而是不良行为者利用各行业成熟的商业基础设施链条发起攻击。这一链条涵盖底层模型开发者、云托管服务商、应用封装工具、应用商店与搜索引擎、金融支付通道以及内容传播平台，最终由信任与安全（Trust & Safety）团队及下游执法机构负责处置应对。 Hall 强调，由于危害在企业间的流转具备连续性，单纯审查技术源头无法解决问题。任何提供技术发现、软件下载、交易抽成或内容托管的节点，实际上都在客观上赋能、分发或变现了非法产出。因此，将 AI 风险置于端到端供应链中重新审视，是界定跨行业治理责任的基础。 生成式技术导致犯罪门槛骤降与取证瘫痪 (3:53) 生成式 AI 从三个操作维度重构了儿童剥削的威胁态势：第一，施害者门槛大幅降低，以往需要接触实物儿童的犯罪行为，现在任何人仅需抓取社交媒体等公开照片，便可通过消费级软件生成虐待图像；第二，产出数量呈现爆炸式增长，单个施害者在数分钟内即可生成数千张全新图像；第三，传统基于感知哈希值匹配已知违法图像数据库的刑侦工作流失效，初代合成图像生成的全新加密哈希值无法产生历史匹配。 这导致互联网侵害儿童（ICAC）特别工作组调查员陷入严峻的操作瓶颈。执法人员在立案时不得不耗费大量时间辨别“图像中是否包含真实受害者”，直接挤占了用于搜寻处于现实危险中儿童的关键救援时间。根据美国国家失踪与被剥削儿童中心（NCMEC）的数据，涉生成式 AI 的报告数量呈现指数级飙升：从 2023 年的 4,700 起、2024 年的 6.7 万起，激增至 2025 年的 150 万起，相当于单日人工审核队列从十余起飙升至千起以上。现实中的犯罪形态已延伸至经济勒索性敲诈、校园同侪侵害、权威人士利用职务之便合成学生图像，以及利用合成人设诱导未成年人的恶意行为。 中游分发与变现通道的问责实践 (10:53) 以 5 月英国一所中学的网络勒索案为例，犯罪团伙抓取该校公开的学生肖像，经 AI 工具生成约 150 张露骨虐待图像并向校方勒索赎金。因图像明确映射到在世未成年人，英国互联网观察基金会（IWF）与美国联邦法律均将其归类为违法 CSAM。该案表明，素材来源、生成软件、勒索渠道与拦截网络分属完全不同的商业主体，没有任何单一机构主导整条流水线。 7 月 17 日，旧金山区检察官向苹果（Apple）和谷歌（Google）发出正式制止令，针对其应用商店中分发并提供内购变现的 13 款换脸及“脱衣（nudify）”应用。尽管两家科技公司并未开发底层算法，但作为平台方，它们直接掌控应用发现、分发审核并抽取内购分成。两家公司随后在法定截止日期前清除了违规应用、关停开发者账号并屏蔽了相关搜索关键词。该事件凸显出，在危害供应链中拥有分发权与支付控制权的中游企业，具有直接实施前端阻断的法定与操作能力。 数据流转断层与政企协同修复机制 (14:43) 当案件流转至下游执法机构时，关键证据往往碎片化地分散在各个企业孤岛中：账户凭证、支付元数据与网络日志分属不同厂商。2025 年提交给 NCMEC 的 150 万起生成式 AI 相关报告中，有超过 40 万起具有执法可操作性，超过 18.2 万起直接涉及 AIC-CSAM 的制作、企图制作或持有。然而，科技公司提交的报告中有超过 10% 缺乏最基本的管辖权标识数据，导致执法人员无法落地立案。Hall 指出，仅完成合规性报告远远不够，必须保证数据的可用性与保真度。 Block Inc. 旗下的 Cash App 针对这一交接断层提供了改善范例。Cash App 每月产生约 450 条网络线索（CyberTips），此前一线刑警普遍反映其支付线索在取证层面晦涩难懂。通过 Zero Abuse Project 与 Block 专门的特别调查团队展开直接对接与跨界培训，执法人员得以有效解读底层支付遥测数据，成功将碎片化交易记录转化为定位嫌疑人与受害者的有效线索。这证明跨机构协作必须修补节点之间的连接机制，而非仅依赖各方孤立履行职责。 Stephanie Hall 总结指出，企业领导层需就自身产品能否被武器化、各环节所需的儿童安全能力以及风险触发后的响应归属展开战略自审；面对被恶意管道裹挟的技术架构，企业应通过早期安全设计、标准化证据留存及行业间信号共享（如 Lantern 计划），在危害供应链的前中端建立主动阻断能力，而非被动依赖下游执法部门清理后果。

Detailed notes — zh

## Background 本视频由 Ai4 于 2026 年 8 月 15 日发布在 YouTube 平台，主讲人为“零虐待项目”（Zero Abuse Project）的技术与创新主管斯蒂芬妮·霍尔（Stephanie Hall）。页面简介指出，如同实体产品一样，AI 造成的伤害同样存在完整的企业供应链；演讲以 AI 生成儿童性虐待材料（AIC-SAM）这一紧迫危机为例，阐述了各行业应如何定位自身在伤害链条上的责任节点，并提供了提交给管理层自查的三大核心问题。 ## TL;DR 斯蒂芬妮·霍尔指出，AI 伤害并非孤立存在，而是沿着由模型研发、云端托管、应用包装、支付结算、平台分发到司法响应构成的完整“伤害供应链”流转。以 AI 生成儿童性剥削内容为例，技术不仅急剧降低了犯罪门槛、颠覆了传统数字指纹检测逻辑，更使 2025 年 NCMEC 接收的相关线索激增至 150 万起。许多企业因“打勾式合规”而提交缺乏关键定位信息的碎片化报告，导致下游一线执法人员无法采取实质侦办行动。为此，企业管理层必须通过三大核心问题审视自身产品在防范与阻断伤害中的角色，建立安全能力并确保责任交接环节的有效连通。 ## Atmosphere & style 整场演讲风格务实冷静、紧迫而极具建设性，演讲者开篇即表明自己热爱并频繁使用 AI，从而奠定了非妖魔化技术、而是聚焦系统工程的专业基调。叙事从一线儿童保护调查的残酷现实切入，结合触目惊心的司法案例与呈指数级爆发的数据，为观众带来了强烈的警示感。听众能够深刻感受到各环节各自为政导致的系统性失效，同时也能从 Block 等积极协作案例中获得加固防线、跨界担责的清晰行动力量。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=iyrrI86EYlY&t=4s) 非营利组织的视角：AI 的巨大潜力与“伤害供应链”的提出 Stephanie Hall 首先介绍了自己的身份与工作背景。作为全美非营利组织“零虐待项目”（Zero Abuse Project）的技术与创新主管，她的核心使命是终结一切形式的儿童虐待与性剥削。该组织的一项关键职责，是为全美一线调查员和检察官提供专业培训，指导他们如何侦办和处理借助技术手段实施的侵害儿童犯罪。而在过去大约三年中，这项工作日益演变为应对“由 AI 辅助与促成的侵害儿童犯罪”。 在演讲开篇，Stephanie 坦诚地亮明了自己对 AI 的态度：“我每天都在使用 AI，我热爱它。”她强调 AI 对非营利领域展现出了不可估量的潜力，让她的团队得以完成过去因缺乏资金或人手而根本无法企及的工作。因此，这场演讲并非是要将 AI 技术本身妖魔化，或宣称 AI 是一切危害的罪魁祸首。她的核心目标非常清晰：向大家展示当 AI 造成的危害——特别是 AI 生成的儿童性剥削内容——在各个系统之间流转时究竟发生了什么，并呼吁每一家使用或构建 AI 产品的机构与科技公司，都应认清并承担起属于自己的那份责任，共同守住链条上的关键节点。 --- ### [01:31](https://www.youtube.com/watch?v=iyrrI86EYlY&t=91s) 超越模型本身：AI 伤害的供应链机制与责任延伸 ```mermaid flowchart LR accTitle: AI 伤害供应链流转图 subgraph upstream["上游：研发与托管"] A["模型训练与构建"] --> B["云端托管与 API"] end subgraph midstream["中游：封装、分发与变现"] B --> C["套壳应用包装"] C --> D["应用商店上架与支付处理"] D --> E["社交平台传播与输出"] end subgraph downstream["下游：检测与响应"] E --> F["信任与安全检测"] F --> G["调查员与司法响应"] end ``` Stephanie 提出了贯穿全场的核心理念：“每件产品都有供应链，事实证明，AI 造成的伤害同样存在供应链。”当 AI 被滥用来制造危害时，它从来不会只经过单一一家公司，而是穿透一整条企业链条：一家公司训练基础模型，一家公司提供云端托管，另一家公司在此之上构建应用程序；随后，应用商店上架该应用并处理支付结算，社交平台负责分发和扩散生成的内容，最终在链条的最末端，由一线人员出面应对和收拾这些破坏带来的恶果。 这正是“AI 伤害与每个人息息相关”的根本原因。企业的责任绝不能因为危害离开了自身系统就戛然而止。当 AI 被用于侵害儿童时，涉及的每一个环节、每一家公司，要么能够成为阻断危害扩散的拦截点，要么就会沦为推波助澜的纵容者。当前的 AI 安全与对齐（AI Safety and Alignment）讨论往往过度局限于模型底层本身——比如训练数据清洗、护栏机制以及模型在对抗滥用时的表现。然而，本次演讲关注的是产品部署后的现实世界：从执法实践来看，侵害案例极少是因为模型产生了自主意识或“失控行动”，绝大多数都是恶意人员利用普通通用的 AI 工具来实施伤害，而这一伤害随即沿着由各家公司构成的链条不断赋能、分发、变现、检测并流转。 --- ### [03:53](https://www.youtube.com/watch?v=iyrrI86EYlY&t=233s) 伤害供应链的具体流转：从网络爬虫到执法调查 当一起涉及“AI 生成的儿童性虐待材料”（AIC-SAM）的案件最终递交到接受培训的“针对儿童的网络犯罪”（ICAC）调查员手中时，它早已穿透了供应链中的多个环节。在常规流程中，科技公司如果在其平台上发现了疑似儿童性剥削内容，会依法向美国国家失踪与受剥削儿童保护中心（NCMEC）报告，随后这些线索会被分流、分派给全美各地的 ICAC 调查网络专员进行侦查。 一条典型的 AI 侵害儿童案件路径往往始于极其日常的场景： - **源头输入（Source）**：不法分子从公开的 Facebook 个人主页或学校官网上抓取普通儿童的生活照片。 - **生成与云服务（Generate & Hosting）**：利用消费级 AI 工具对照片进行脱衣或面部替换，底层则由各大云服务厂商提供算力支撑。 - **分发与变现（Distribution & Monetization）**：搜索引擎或应用商店帮助用户找到这些工具，应用内支付渠道协助开发者获取收益。 - **传播与检测（Spread & Detection）**：社交平台承载并扩散这些生成的违规文件，直到平台的信任与安全（Trust & Safety）团队通过审查机制捕获并上报。 - **最终响应（Respond）**：案件进入最下游的司法与保护系统，由早已不堪重负的线上一线调查员接手处理。 Stephanie 指出，这条链条精准复刻了科技行业每天都在打交道的现代技术栈（模型、API、套壳应用、托管、分发、支付和平台）。普通素材从一端进入，链条上的每个环节都为其叠加了功能、收益或传播范围，最终在另一端演变为实质性的侵害。没有任何单一组织能完全控制整条链条，但每一个环节都必须对自己掌控的这部分承担责任。 --- ### [06:01](https://www.youtube.com/watch?v=iyrrI86EYlY&t=361s) AI 带来的三大范式转变与指数级增长的危机数据 AI 的出现不仅为执法人员增加了全新的案件类型，更从根本上颠覆了危害在供应链中流转的规模、速度与复杂性，具体体现在三个维度的范式转变： 1. **犯罪门槛急剧降低（Who can offend）**：过去某些形式的严重虐待必须依赖于对儿童身体的直接接触；而现在，任何人只需从网上随意爬取一张普通生活照，配合普通的消费级应用即可作恶。技术门槛不仅是降低了，对许多施害者而言几乎是彻底消失了。 2. **生成规模呈现爆炸式增长（Scale of generation）**：以往不法分子往往只能在固定的违规素材库中流转交换；如今单一施害者动动手指就能瞬间批量合成成千上万张新型虐待图像。 3. **检测与侦办逻辑被颠覆（Detection & Triage）**：传统的儿童性虐待材料（CSAM）检测高度依赖已知图像的数字指纹（哈希值匹配，如 PhotoDNA）。但对于初次由 AI 生成的全新图像，系统根本不存在历史匹配记录。这迫使一线调查人员在接手每一起案件时都必须先回答一个前所未有的棘手问题：“这究竟是不是一个真实存在的孩子？” 甄别合成图像与真实受害者图像所耗费的大量时间和司法资源，直接挤占了用于紧急定位和解救处于现实危险中真实儿童的宝贵黄金时间。 - **2023 年**：NCMEC 接收到的涉及生成式 AI 的报告仅为 **4,700 起**（平均每天约十几起）。 - **2024 年**：该类报告激增至 **67,000 起**。 - **2025 年**：报告数量呈现几何级爆发，突破了 **150 万起**（相当于每天涌入超过 **1,000 起**报告）。每一份报告都需要人工与系统的审查、分流和研判，这给后台审核和司法资源带来了难以承受的巨大压力。 --- ### [08:42](https://www.youtube.com/watch?v=iyrrI86EYlY&t=522s) 伤害如何具体浮现：性勒索、同伴侵害与熟人犯罪案例 AI 辅助的儿童侵害在现实中往往不会贴着清晰的标签出现，而是隐蔽地渗透在各种复杂的犯罪场景中： - **新型性勒索（Sextortion）**：这是当前儿童保护领域极其严峻的热点。NCMEC 记录了多起典型案例：当未成年人拒绝向敲诈者发送真实的私密照片时，犯罪分子便利用 AI 工具伪造该未成年人的裸照，反过来以此威胁索要金钱、索取更多真实私密照片或逼迫其进行性顺从。施害者不再需要拿到真实的受害者把柄即可启动敲诈链条。 - **校园同伴侵害（Peer-to-Peer Abuse）**：今年 3 月，美国宾夕法尼亚州两名青少年被正式宣判，他们承认利用 AI 生成了约 350 张涉及本校 59 名女同学的露骨色情图片并在校内传播。 - **受信任成年人的熟人犯罪（Trusted Adult Abuse）**：侵害并非总是来自网络上的陌生人。密西西比州一名初中前教师，通过其任教学生的社交媒体账号下载照片，利用 AI 制作了涉及 8 名 14 至 16 岁女学生的性剥削视频。该教师随后认罪并被判处 5 年监禁。 - **身份伪造与线上诱导（Impersonation & Grooming）**：不法分子利用 AI 合成账号、虚拟头像、逼真语音甚至整套虚构身份伪装成同龄人接近未成年人，以此建立信任并实施诱导侵害。 这些案例的表现形态各异，但其底层共享着完全一致的运作链条：普通素材输入 -> AI 算法加工变形 -> 外部数字系统承载传播 -> 真实的孩子在现实中承受毁灭性的创伤后果。 --- ### [10:53](https://www.youtube.com/watch?v=iyrrI86EYlY&t=653s) 真实链条追踪：英国学校勒索案与科技巨头的平台责任 Stephanie 通过两起对比鲜明的真实案例，剖析了企业在供应链中所处的位置与应对方式： 第一起案例发生在今年 5 月的英国。一个犯罪团伙从一所英国中学的官方网站和社交媒体主页上爬取了大量在校学生的普通照片，随后导入 ### [13:54](https://www.youtube.com/watch?v=iyrrI86EYlY&t=834s) 伤害供应链中的碎片化证据与交接困境 当科技巨头面临监管压力时，行动往往十分迅速——应用被下架、开发者账号被封禁、“一键去衣（Nudify）”等搜索词遭到限制，并且全部赶在法定截止日期前完成。然而，这绝非仅仅是苹果或谷歌等个别巨头面临的难题。任何负责分发工具或处理支付业务的企业，在伤害事件曝光时，都必须清楚自己在链条中实际控制着哪一个环节。正如斯蒂芬妮·霍尔（Stephanie Hall）反复强调的核心观点：“你不需要亲自制造伤害，也可能身处伤害供应链之中。” 更为棘手的问题在于，当危害沿着供应链传递到最下游的执法与响应端时，调查人员面对的绝不是一份完整的案卷。真实的执法现状极其碎片化：一家公司可能只掌握犯罪嫌疑人的注册账户，另一家公司持有支付交易记录，第三家公司则保留着底层的技术与网络日志。每个组织都只握有案件拼图中的一块碎片，整个应对链条能否运转，完全取决于各环节能否将这些关键证据妥善保全并清晰地交接给下一环。 ```mermaid flowchart TD accTitle: AI Harm Evidence Fragmentation Flow subgraph S1["上游与中游：证据碎片分布"] A["开发者/分发平台<br/>(保留账户与设备信息)"] B["支付结算渠道<br/>(保留交易与资金记录)"] C["云/网络服务商<br/>(保留访问与技术日志)"] end subgraph S2["下游：执法与处置响应"] D["执法部门 / ICAC 调查员<br/>(依赖完整证据链拼凑破案)"] end A -->|"碎片化数据交接"| D B -->|"碎片化数据交接"| D C -->|"碎片化数据交接"| D ``` --- ### [15:10](https://www.youtube.com/watch?v=iyrrI86EYlY&t=910s) AI 带来的规模危机：NCMEC 数据与“无效合规”隐患 生成式 AI 的爆发使这一交接挑战变得空前紧迫。2025 年提交给美国国家失踪与被剥削儿童中心（NCMEC）的线索报告中，与生成式 AI 相关的报告高达 150 万起。其中超过 40 万起具备实质可查证性（actionable），更有超过 18.2 万起直接涉及制作、企图制作或持有 AI 生成的儿童性虐待材料（CSAM）。一线办案经验表明，在生成 AI 虐待材料的嫌疑人设备中，极大概率也同时存有真实受害儿童的 CSAM 内容。 然而，更令人担忧的数据是其中超过 10% 的科技行业报告由于缺失核心定位信息，导致根本无法判断所属管辖区域，难以分发给对应的执法机构。许多科技企业在发现违规后仅仅是“打勾式合规”，按部就班地填报系统，但报告内容对于下游调查人员而言毫无用处。企业在产品数据、安全策略与运营流程上的设计，目标绝不能仅止于“提交一份报告”，而是必须确保提供足够充分的信息，让下一个环节有能力采取行动。针对这一痛点，科技联盟（Tech Coalition）等行业组织正为企业提供同行交流与对齐最佳实践的协作平台。 - 2025 年 NCMEC 接收到涉及生成式 AI 关联的线索报告达 150 万件。 - 超过 40 万件报告具备可追查性，18.2 万余件涉及 AI CSAM 的生成、企图生成或持有。 - 超过 10% 的科技企业报告因缺乏基础定位信息而沦为无法派发至具体管辖区的“无效报告”。 --- ### [16:36](https://www.youtube.com/watch?v=iyrrI86EYlY&t=996s) 修复断裂的交接链：Block（Cash App）与专案团队的合作案例 在金融支付环节，Block Inc. 旗下的 Cash App 正好处于有害交易变现的核心链条上。仅去年一年，Cash App 平台每月就向 NCMEC 产生大约 450 条网络举报线索（CyberTiPs）。在以往的培训中，一线调查人员常常向 Zero Abuse Project 抱怨：“这些来自 Cash App 的线索极难阅读，缺少关键细节，难以展开实质性侦查。”针对这一堵点，Zero Abuse Project 的首席 ICAC 调查专家丹·贝里（Dan Berry）直接拨通了 Block 公司的电话进行沟通。 这一通电话揭开了一个意想不到的现实：Block 内部其实早已设立了一支高水平的专项调查团队，其全职工作就是协助调查人员解析线索并打击平台上的 CSAM。双方此前之所以脱节，纯粹是因为外部调查人员根本不知道该团队的存在。一次直接沟通打破了信息壁垒，此后双方展开了长期合作，通过系列研讨会培训执法人员如何利用支付流水锁定嫌疑人身份并解救受害儿童。 与此形成鲜明对比的是各环节各自为政的系统性溃败。在许多失败案例中，链条上的每个参与者都可以理直气壮地宣称自己完成了份内之事——“我们没开发这个工具，只是上架了它”、“我们只是处理了支付”、“我们已经按规定提交了报告”。每个声明都属实，但整体应对依然全面崩溃：工具下架了却未保全证据，报告提交了却无法分发，专业团队设立了却无人知晓如何联系。“你的责任可能已经结束，但交接环节必须有人负责。” --- ### [19:01](https://www.youtube.com/watch?v=iyrrI86EYlY&t=1141s) 组织自查的三大核心问题与系统性防御行动 为帮助各类企业识别自身在伤害链条上的位置，斯蒂芬妮·霍尔提出了企业管理层必须自问的三个核心问题： 1. **我们的产品是否可能助长（enable）、促进（facilitate）、检测（detect）或阻断（disrupt）伤害？** 不要只看产品的设计初衷，要审视它在更广泛的链条中会被如何滥用。 2. **在所处的环节上，我们需要具备哪些儿童安全能力？** 这涵盖了制定更清晰的安全政策、开展逼真的防滥用压力测试、推行安全导向的产品设计、强化检测算法以及优化报告提交流程。 3. **当伤害浮出水面时，下一个处置动作由谁负责？** 必须明确触发机制是什么、哪些数据与证据必须被冻结保全、以及具体应该联系外部的哪个人或机构。 科技联盟推出的“提灯计划”（Lantern program）便是跨平台协同的范例，通过在不同企业间共享风险信号，实现全网跨平台协同打击儿童性剥削。即便一家企业的产品并不直接面向儿童群体，其掌控的底层系统依然可能沦为伤害链条的一部分。儿童安全不仅是在惨剧发生后的事后处置，更是在产品构建、策略制定与日常运营中构建防线，从源头增加作恶成本。各行各业都应认清自身所处的环节，加固该环节的能力，并确保为下一环节提供足以采取行动的坚实支撑。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=iyrrI86EYlY&t=4s) 我每天都在使用 AI，我热爱它。 — Stephanie Hall > [01:31](https://www.youtube.com/watch?v=iyrrI86EYlY&t=91s) 每件产品都有供应链，事实证明，AI 造成的伤害同样存在供应链。 — Stephanie Hall > [13:54](https://www.youtube.com/watch?v=iyrrI86EYlY&t=834s) 你不需要亲自制造伤害，也可能身处伤害供应链之中。 — Stephanie Hall > [06:01](https://www.youtube.com/watch?v=iyrrI86EYlY&t=361s) 这究竟是不是一个真实存在的孩子？ — Stephanie Hall > [16:36](https://www.youtube.com/watch?v=iyrrI86EYlY&t=996s) 我们没开发这个工具，只是上架了它 — Stephanie Hall > [16:36](https://www.youtube.com/watch?v=iyrrI86EYlY&t=996s) 我们只是处理了支付 — Stephanie Hall > [16:36](https://www.youtube.com/watch?v=iyrrI86EYlY&t=996s) 我们已经按规定提交了报告 — Stephanie Hall > [16:36](https://www.youtube.com/watch?v=iyrrI86EYlY&t=996s) 你的责任可能已经结束，但交接环节必须有人负责。 — Stephanie Hall > [19:01](https://www.youtube.com/watch?v=iyrrI86EYlY&t=1141s) 我们的产品是否可能助长（enable）、促进（facilitate）、检测（detect）或阻断（disrupt）伤害？ — Stephanie Hall > [19:01](https://www.youtube.com/watch?v=iyrrI86EYlY&t=1141s) 在所处的环节上，我们需要具备哪些儿童安全能力？ — Stephanie Hall > [19:01](https://www.youtube.com/watch?v=iyrrI86EYlY&t=1141s) 当伤害浮出水面时，下一个处置动作由谁负责？ — Stephanie Hall ## Mentioned Stephanie Hall, Zero Abuse Project, NCMEC, ICAC, PhotoDNA, Facebook, Block Inc., Cash App, Dan Berry, Tech Coalition, Lantern program, Apple, Google ## Video description Stephanie Hall - Technology & Innovation Lead, Zero Abuse Project Every product has a supply chain, and it turns out harm has one too. When AI is used to cause harm, it moves through a chain of companies: one builds the tool, one hosts it, one sells it, one processes the payment, one's platform spreads it, and someone far downstream responds to the damage. This talk maps that chain using one of the clearest and most urgent examples available: AI-generated child sexual abuse material. I serve as Technology and Innovation Lead at Zero Abuse Project, a national child protection nonprofit, where I develop AI training for the investigators, prosecutors, and child protection professionals who respond at the end of that chain. In my field we say that child abuse prevention is everyone's business, and this session extends that same principle to AI: attendees will learn a practical framework for locating their organization's link on any chain of AI harm, plus three questions to bring back to their leadership. Whatever your industry, your organization sits somewhere on one of these chains. This session will help you find your link and understand what responsibility looks like there. [Source](https://www.youtube.com/watch?v=iyrrI86EYlY)

Source quotes

[0:04] Stephanie Hall: 我每天都在使用 AI，我热爱它。 [1:31] Stephanie Hall: 每件产品都有供应链，事实证明，AI 造成的伤害同样存在供应链。 [13:54] Stephanie Hall: 你不需要亲自制造伤害，也可能身处伤害供应链之中。 [6:01] Stephanie Hall: 这究竟是不是一个真实存在的孩子？ [16:36] Stephanie Hall: 我们没开发这个工具，只是上架了它 [16:36] Stephanie Hall: 我们只是处理了支付 [16:36] Stephanie Hall: 我们已经按规定提交了报告 [16:36] Stephanie Hall: 你的责任可能已经结束，但交接环节必须有人负责。 [19:01] Stephanie Hall: 我们的产品是否可能助长（enable）、促进（facilitate）、检测（detect）或阻断（disrupt）伤害？ [19:01] Stephanie Hall: 在所处的环节上，我们需要具备哪些儿童安全能力？ [19:01] Stephanie Hall: 当伤害浮出水面时，下一个处置动作由谁负责？

Source quotes

[0:04] Stephanie Hall: Every product has a supply chain, and it turns out harm has one too. [0:04] Stephanie Hall: When AI is weaponized or misused to inflict harm, that damage does not occur within a single siloed company. [3:53] Stephanie Hall: Is this a real child? [3:53] Stephanie Hall: Every operational hour an investigator spends discerning synthetic imagery from real-world photography represents critical time lost in locating a live child who may be facing immediate physical peril. [10:53] Stephanie Hall: You don't have to be the one who created the harm to be standing in the supply chain. [19:01] Stephanie Hall: Could our products enable, facilitate, detect, or disrupt harm? [19:01] Stephanie Hall: What child safety capabilities do we need at our link? [19:01] Stephanie Hall: When harm surfaces, who owns the next action?

Original transcript

[0:04] I'm Stephanie Hall. I'm the technology  and innovation lead at Zero Abuse Project. [0:09] Zero Abuse Project is a national nonprofit  organization and our mission is to end all forms [0:14] of child abuse and sexual exploitation. We do  that in a number of ways um through our programs, [0:20] but one of those ways is that we are responsible  for training the investigators and the prosecutors [0:27] who train and handle technology facilitated  crimes against children. And as of the last [0:34] three years or so, that has increasingly  meant AI facilitated crimes against children. [0:40] Um and so I'll I'll be honest with you upfront,  I use AI every single day. I love it. Um, [0:47] I think it has so much potential for the nonprofit  sector. Um, it has given my team the ability to do [0:54] work that we never would have had the funding or  the staff to do. So, this is not a talk about how [0:59] AI harm or AI is the the main problem. U, my  goal is simple. It's just to walk you through [1:06] um what happens when AI harms specifically  um AI generated child sexual exploitation [1:13] moves through systems and why every institution  and technology company who is using AI products [1:23] should own their responsibility and um help move  it to the next chain, move it to the next link. [1:31] So here's the idea I want to kick us off  with. Every product has a supply chain [1:36] and it turns out AI harm has one too.  Uh when AI is misused to cause harm, [1:44] it doesn't pass through one company. It passes  through a chain of them. One trains the model, [1:50] one hosts it, one builds an app on top of it. Uh  one lists the app and pro processes the payments. [1:59] One platform spreads the output and  eventually at the response end of that [2:04] chain is someone dealing with the damage.  That's why AI harm is everyone's business. [2:11] Responsibility doesn't stop when the harm leaves  your system. Um when AI is used to target a child, [2:19] every link, so every company involved can either  interrupt that harm or it can allow it to spread. [2:28] So here is where this talk fits in  the um AI safety and alignment track. [2:34] A lot of AI safety talks and focus is  mostly on the model itself. So training, [2:39] guard rails, safeguards, um how the model  behaves during misuse. But today's focus [2:46] is on what happens after the product is deployed  and the harm begins moving through other systems. [2:53] The cases we see in the field working with IAC  investigators and prosecutors are rarely a result [3:00] of a model acting on its own. Um they are about  people using ordinary AI tools to cause the harm [3:10] and that harm moving through a chain of  companies that enable, distribute, monetize, [3:16] detect and report. And so I'm using AI generated  child sexual abuse material, AIC SAM for short, [3:24] as the case study, not only because it's the case  that I know best, but it is also one of the um [3:30] leading and growing AI harms as it relates to  online safety. And so I'm going to show you [3:36] what this looks like through real cases um  from the past year. And my goal is simple. [3:43] um help you see where your  organization may enter that chain [3:47] and what it can do at that point and  what the next link needs from you. [3:53] So by the time an AIC SAM case reaches  the IAC investigators that my team trains, [4:00] it's probably crossed several of the  links on this slide. For context, um, IAK, [4:06] Internet Crimes Against Children, is a national  um, network of inst of investigators handling [4:11] technology facilitated crimes against children.  And so when companies report uh, suspected [4:18] child exploitation to NickMech, the National  Center for Missing and Exploited Children, [4:23] the hope is that those reports eventually  get triaged to an IAC investigator. [4:29] Um, and here's one common path an AI  generated child exploitation case can take. [4:35] The chain can begin with something as ordinary as  a child's photo being scraped from a Facebook page [4:42] or a school website. An AI tool transforms it.  Cloud services power that tool. App stores are [4:51] the ones or search engines help people find that  tool. Platforms carry the output. and payment [4:58] systems may help monetize it. Eventually, a trust  and safety team detects the file and reports it. [5:05] But the last link, respond, is where  those investigators I work with sit. [5:11] And AI is adding more cases and complexity to  an already overloaded investigative system. [5:20] I call this the supply chain of harm uh  because it follows the same stack many of you [5:24] work with every day. models, APIs, rapper apps,  hosting, distribution, payments, and platforms. [5:32] Ordinary material enters one end. Each  link can add capability or profit or reach [5:41] and then harm emerges at the other end. No  one organization controls that whole chain. [5:48] But each organization can be accountable for  the link that it controls. Your organization [5:54] may hold more [clears throat] uh one or more  of these links even if you've never named it. [6:01] An AI didn't just create a new kind of case  for investigators to work. It changed the [6:06] scale and the speed and the complexity of  how the harm is moving through this chain. [6:14] So one of the first things it changes is who  can offend. Some forms of abuse that we see [6:20] once require direct access to a child now can  begin with an ordinary image someone scrapes [6:27] off the internet and they use a consumer app.  The technical barrier did not just fail for many [6:37] offenders it nearly vanished. Second, how much one  offender can generate thousands of images of CSAM [6:46] um rather than working with a fixed set which  is what they typically had to do before. [6:52] And third, the detection that uh what detection  requires. So traditional CSAM detection relies [6:58] very heavily on known image hashes, but many  first generation AI images have no known match. [7:06] So investigators have to begin a case now by  asking a brand new question. Is this a real child? [7:15] Triage, investigative priorities, and charging  decisions can all depend on that answer. [7:21] So to put it into perspective, every hour spent  sorting through synthetic images from real images [7:28] is time not spending locating a  child who may be in immediate danger. [7:36] Child exploitation is not the only  AI harm, right? We know this. Fraud, [7:40] impersonation, adult deep fakes, other abuses  move through the same kind of chain. But when [7:48] we look at AI generated child exploitation, its  consequences are immediate. Reports to Nickcheck [7:55] involving generative AI went from 4,700 in  2023 to 67,000 in 2024 to 1.5 million in 2025. [8:07] In daily terms, two years ago, that's about  a dozen reports to NickMech a day. Last year, [8:14] that's more than a thousand reports a day. Each  report enters a queue that must be reviewed, [8:21] triaged, and routed. And so just think of  the massive resources that that now takes. [8:28] So for the time we have together, I want to follow  one real chain and keep asking the same question, [8:34] which is what does your organization control  and what does the next link need from you. [8:42] So before we follow one case um across the full  chain, I want to show you the different ways that [8:48] AI facilitated harm can surface for investigators  because it rarely arrives with a clear label. [8:56] Sometimes it appears as sextortion which  is a very hot topic right now in our field. [9:01] Nickmick has documented cases of sextortion where  a child refused to send an explicit image. So, the [9:09] offender created a fake one with AI and used it to  demand money, more images, or sexual compliance. [9:19] The offender no longer needs a real image to  begin this threat. So, it's obviously going to [9:25] amplify it even more. Sometimes the cases we  see are peer-to-peer abuse. This past March, [9:32] two Pennsylvania teenagers were sentenced after  admitting involvement in the creation of roughly [9:38] 350 AI generated sexual images depicting 59  uh females, female students from their school. [9:47] This can also involve a trusted adult. Last year,  a Mississippi uh in Mississippi, a former middle [9:54] school teacher used his students social media  accounts and created sexually exploitive AI videos [10:04] of eight girls between the ages of 14 and 16. He  later pleaded guilty and was sentenced to 5 years. [10:11] This was not a stranger targeting children online. [10:15] This was an adult who already had  access, direct access to them. [10:19] And in impersonation and in grooming cases,  offenders can use synthetic accounts, photos, [10:26] voices, or entire identities to approach children  and pretend to be a peer of theirs. We see this a [10:34] lot. So the cases all kind of look different, but  the same chain exists, right? ordinary material [10:43] enters, AI transforms it, other systems carry it  on, and a child is left um with the consequences. [10:53] So, here's the case I want us to follow across the  chain. Um, in a case that came to light this May, [10:59] a criminal game gang took ordinary photographs  of students from a UK secondary school, [11:06] scraped them from their website  and their social media pages, [11:09] ran them through an AI tool, and created  sexually explicit images of those children. [11:15] Because these images used the faces of real  children, the IWF, the Internet Watch Foundation, [11:22] classified those, about 150 of them, as  child sexual abuse material under UK law. [11:30] Similar imagery involving real  identifiable children can also [11:33] be prosecuted in the US. It is illegal. The  offenders took their crime a step further [11:40] and sent examples to the school with a demand.  pay us or we put these on the internet. [11:48] Um, the school did the right thing. They  called the police. The police involved the IWF. [11:54] The IWF converted the images into digital  fingerprints, otherwise known as hashes, [11:59] and shared them with the major tech  platforms so that they could be blocked upon. [12:05] Now, if we look at that chain, school photographs  entered at the source link, right? An unidentified [12:12] AI tool occupied the generate link. Extortion  brought the harm back to the school. [12:18] Police and the IWF held the response link and  platforms, tech platforms were asked to block [12:27] the distribution link. No one controlled  that whole chain. But several organizations [12:34] controlled a point where the harm could have  been interrupted or handed forward better. [12:41] So, hold on to this case because about two  months later, we got to see a different outcome. [12:46] About two months after the school case um went  public, something different happened. This time [12:51] in the US on July 17th, the San Francisco  attorney sent cease and desist letters [12:57] to both Apple and Google over 13 of their face  swapping apps that could turn ordinary photos [13:04] into non-consensual nude images. And the letters  made a very sharp point. Apple and Google weren't [13:11] just distributing these apps. They were processing  the purchases and taking a share of the revenue. [13:18] So they were occupying two links, right? Not one.  Now Apple and Google did not build those apps. [13:24] They did not create a single image. But they did  control which apps entered their marketplaces, [13:31] how users found them, and  whether the money kept moving. [13:37] And the letters ask for exactly that. Remove  the apps, cut the payment relationships, [13:42] and review the stores so the same tools don't come  back under new names, which happens every day. [13:49] The requested actions matched the  links that Apple and Google controlled. [13:54] And they responded. Apps were removed,  developer accounts were terminated, [13:58] nutify searches were restricted, and they did it  all ahead of the legal deadline. Yay for them. [14:05] This is not only an Apple or a Google problem.  Any company that distributes tools or processes [14:12] payments should understand what it controls when  harm is discovered. In the school case, the harm [14:18] crossed several systems without the commercial  links being publicly identified for the privacy [14:25] of the children involved. Here, two companies  were named along with the exact links that they [14:31] controlled. I just want to emphasize this quote.  You don't have to be the one who created the harm [14:38] to be standing in the supply chain and that's  really the whole point of this presentation. [14:43] So what reaches the response end of the chain?  Not a complete case. I will tell you that [14:50] one company may hold the account information  of the perpetrator. Another company may hold [14:56] the payment record. Another one may hold the  technical data. Each organization has one piece [15:04] and the handoff only works when those pieces  are preserved and passed forward clearly. [15:10] The scale that AI has created makes this urgent.  In 2025, 1.5 million reports sent to NickMick had [15:19] a generative AI nexus. More than 400,000 of those  were actionable and more than 182,000 involved [15:27] creating, attempting to create or possessing  AI generated child sexual abuse material. [15:35] And typically what we find is  when someone's generating AI CSAM, [15:40] they are very likely to have real CSAM  on their devices of real children. [15:47] But the scarier number is the 10 10%. more than  10% of the technology industry reports still [15:53] lacked enough information to identify the right  jurisdiction to route these two. So basically [15:59] technology companies were sending in  reports. They were check marking the box [16:03] but the reports didn't have anything in  it that was useful for investigators. [16:08] That is where product data  policy and operations all matter. [16:13] The goal is not just to file a report. It is to  give the next link enough information to act. [16:19] And companies don't have to figure this out  by themselves. The tech coalition is a great [16:24] example of an organization that gives technology  companies a community to learn from their peers, [16:31] compare practices, and strengthen  child safety systems together. [16:36] Now, I want to tell you about a handoff that  got fixed because I actually got to watch this [16:40] one happen. Block Inc. is the financial technology  company behind Cash App, which puts them squarely [16:45] at the monetized link of this chain. Uh, for  scale, Cash App as of last year alone generated [16:52] roughly 450 cyber tips a month to NickMeck. That's  how much harm was going through their platform. [16:59] At our trainings, investigators kept coming  up to us and saying the same thing. Uh, [17:04] these cash app tips, um, cyber tips are  really hard to read. They're hard to act [17:08] on. We need more information. And so, um, our  lead IACK investigative specialist, Danberry, [17:14] he picked up the phone and called him and let  let Block know what was going on. Um, and what [17:21] we found was that they actually already had a  special investigative team whose entire job was [17:27] helping investigators use these reports that they  were sending and combat SESAM on their platform. [17:34] So, the team at this company already existed,  but our investigators just didn't know about it. [17:41] One phone call connected them. We've partnered  with that team ever since through webinars um [17:46] and helping people understand how payment  data identifies suspects and locates victims. [17:51] A payment company looked at the link  it controlled, built a team around it. [17:56] A training organization heard what heard where  the handoff was failing and reached across [18:03] Neither side controlled that whole chain, but  together they strengthened the connection between [18:09] the two links. And that is what owning your link  can look like before anyone forces you to do [18:14] it. And here's the opposite of the Cash App story.  Every organization can truthfully say that its own [18:22] part was completed, right? We didn't build it. We  only listed it. We only processed the payment. Uh [18:30] we filed the required report. Every statement can  be true and the overall response can still fail. [18:38] So a product can be removed without evidence being  preserved. A report can be filed without enough [18:43] information to route it. A specialized team can  exist without investigators knowing how to reach [18:49] them. The failure is often not inside one link.  It is often the connection between the links. [18:56] And your responsibility may end, but  your handoff still needs an owner. [19:01] So here are the three questions. Um I'm going  to try and say in the last 39 seconds first [19:06] uh take this back to your or to your organization  first. Could our products enable facilitate detect [19:12] or disrupt harm? Do not only look at what the  product was designed to do. Look at how it could [19:18] fit into the wider chain. Second, what child  safety capabilities do we need at our link? [19:26] That may be clearer policy. That may be realistic  misuse testing. That may be safer product design [19:32] or better detection or even  stronger reporting processes. [19:37] Third, when harm surfaces, who owns the next  action? Name that trigger, what must be preserved, [19:44] and who gets contacted? A handoff is only complete  when the next person can act. The tech coalition's [19:51] lantern program is a perfect example of what  shared responsibility can look like. Lantern is [19:57] a pioneering cross-platform signal sharing program  that enables tech companies to work together [20:04] to identify and address online child sexual  exploitation and abuse across platforms. [20:11] Even if your community, excuse me, even if  your company does not serve children, the [20:15] products and systems that you control can still  become a link in the chain that harms children. [20:22] So, if today made you wonder about  what your organization's link is, [20:26] that's a conversation I'm here for. Zero Abuse  Project can review your child's safety policies, [20:31] we can train your team on AI enabled exploitation  and response. And if you want to get involved in [20:37] the effort, we can help your company sponsor  one of our trainings uh for investigators and [20:42] prosecutors. And the QR code on this slide  is how you can start that conversation. [20:48] So, I'll just end with this. I know I'm a minute  over. Not every organization has the same role. [20:53] That is not what uh we're trying to say. But every  company should ask whether its products enable, [20:59] facilitate, detect, or disrupt harm. Child safety  is not only about responding after abuse happens. [21:08] It's about building products, policies, and  operations that make abuse harder in the first [21:13] place. So know your link, strengthen it, and make  sure that the next link has what it needs to act. [21:20] I'm Stephanie Hall. Thank you so much. [applause]