Objective AI Finds Top Safety Benchmarks Miss Psychological Harms

Novel Evaluation Framework for Bias & Psychological Harms in LLM Datasets With The Objective AI — Ai4

Ai4

Scrubbing training data degrades model utility while safety suites rely on arbitrary cutoffs that completely miss user psychological risks.

AI-assisted summary of Novel Evaluation Framework for Bias & Psychological Harms in LLM Datasets With The Objective AI — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by the Ai4 channel on August 26, 2025, this presentation features Jon Bennion, Manager of Applied ML/AI and Trust and Safety at The Objective AI. According to the page description, Bennion introduces a novel unsupervised evaluation framework that uses UMAP dimensionality reduction and k-means clustering across five open-source safety benchmarks to comprehensively address bias and evolving psychological harms.

## TL;DR Jon Bennion challenges conventional AI safety dogma, demonstrating that attempting to scrub bias directly from pre-training data degrades model utility and fails due to the probabilistic nature of LLMs and compounding agent errors. He critiques industry benchmarks for relying on arbitrary 50% binary decision boundaries and recursive, biased evaluators, advocating instead for an unsupervised semantic pipeline using sentence embeddings, UMAP, and k-means clustering. Applied across five open-source benchmarks to achieve an optimal silhouette score of 0.470, his framework maps semantic harm topography, revealing robust coverage of physical liabilities like weapons and drugs alongside acute gaps in hate speech and a complete absence of psychological harms such as sycophancy and emotional dependency. Bennion concludes that addressing user psychological harm requires product-level UI interventions and psychologist collaboration rather than destructive attempts at foundational dataset sanitization.

## Atmosphere & style The presentation carries a candid, no-nonsense atmosphere driven by Bennion's pragmatic skepticism as a veteran of FAANG and Character.ai trust-and-safety operations. He delivers his critique in a calm, analytical, and measured cadence, systematically puncturing Silicon Valley hype around superintelligence and agentic autonomy with hard mathematical realities. Listening to the talk feels like an unvarnished insider debrief, shifting the conversation away from corporate public relations and subjective committee debates toward grounded geometric clustering, data quality crises, and real-world user vulnerabilities.

## Key takeaways - [00:03](https://www.youtube.com/watch?v=CAtbQFldvEk&t=3s) Attempting to eliminate model bias solely by cleaning pre-training data is fundamentally flawed, as selectively removing core semantic distributions strips LLMs of their general utility. - [00:03](https://www.youtube.com/watch?v=CAtbQFldvEk&t=3s) Multi-step autonomous AI agent workflows suffer from severe compounding errors, where even an optimistic 80% per-step success rate rapidly degrades over extended reasoning chains. - [02:12](https://www.youtube.com/watch?v=CAtbQFldvEk&t=132s) The generative AI ecosystem faces a data quality crisis rather than a volume shortage, edging models toward degenerative 'model collapse' as scrapers consume recursive synthetic web content. - [02:12](https://www.youtube.com/watch?v=CAtbQFldvEk&t=132s) Current safety evaluation faces a circular paradox: the automated evaluators, rubrics, and judges deployed to detect bias carry their own unacknowledged training biases. - [06:05](https://www.youtube.com/watch?v=CAtbQFldvEk&t=365s) Interactions with conversational personas reveal acute psychological risks driven by sycophancy and anthropomorphism, which validate user delusions while evading traditional keyword safety filters. - [07:46](https://www.youtube.com/watch?v=CAtbQFldvEk&t=466s) Standard AI safety leaderboards rely on arbitrary binary decision thresholds (such as 50% cutoff rules) that can be easily gamed or manipulated, undermining longitudinal evaluation. - [10:02](https://www.youtube.com/watch?v=CAtbQFldvEk&t=602s) An unsupervised semantic evaluation framework using dense embeddings, UMAP dimensionality reduction, and k-means clustering enables objective harm mapping without human threshold bias. - [12:10](https://www.youtube.com/watch?v=CAtbQFldvEk&t=730s) Auditing open-source safety benchmarks reveals high semantic coverage for physical harms (weapons, illicit drugs, PII) but glaring blind spots in hate speech density and psychological harms. - [12:10](https://www.youtube.com/watch?v=CAtbQFldvEk&t=730s) Evaluating isolated first-turn user prompts provides the most reliable semantic baseline for safety datasets by removing the confounding historical noise of multi-turn dialogues and RAG pipelines. - [15:10](https://www.youtube.com/watch?v=CAtbQFldvEk&t=910s) Lightweight embedding models such as MiniLM match the performance of larger alternatives in cluster evaluation, while Mahalanobis distance effectively captures covariance across data points. - [16:36](https://www.youtube.com/watch?v=CAtbQFldvEk&t=996s) Because LLMs function strictly as probabilistic pattern matchers rather than authoritative counselors, diagnosing psychological harms requires specialized benchmarks curated with clinical psychologists. - [19:34](https://www.youtube.com/watch?v=CAtbQFldvEk&t=1174s) Effective mitigation of user psychosis and behavioral harms must occur via product-level interventions and UI warning alerts, as complete dataset sanitization renders models dysfunctional.

## Detailed walkthrough ### [00:03](https://www.youtube.com/watch?v=CAtbQFldvEk&t=3s) The Fallacy of Training Data Fixes and Compounding Agent Error

Jon Bennion introduces his presentation by pointing the audience to a QR code linking to his research paper, which he originally presented in Zurich in May. Having been added to the conference schedule only two weeks prior, he emphasizes that the work tackles an issue currently sitting at the absolute forefront of industry and policy discussions. Governments and regulatory bodies have recently declared bias to be the primary threat in generative AI, yet Bennion argues that their proposed remedies are fundamentally flawed. Regulatory and industry efforts predominantly attempt to "fix it in the training data," which he considers virtually impossible: one cannot maintain the general utility and generative capabilities of a large language model while selectively excising core concepts, particularly because generative models are inherently probabilistic systems whose outputs cannot be deterministically controlled.

This probabilistic nature directly challenges the prevailing industry euphoria surrounding autonomous agents. While AI agents are an undeniably captivating concept, Bennion cautions that multi-step agentic workflows suffer from severe compounding errors. If each successive model invocation in an agent's reasoning chain operates at an optimistic best-case probability of around 80% accuracy, the overall system reliability rapidly degrades with every additional step. Engineers might spend immense amounts of time and resources pushing a single call's reliability from 75% to 80%, yet the compounding math guarantees failure over extended task horizons. Within this climate of persistent overhype, Bennion underscores the necessity of taking a rigorous, objective look at how the AI safety community actually measures and benchmarks harms.

```mermaid flowchart LR accTitle: Compounding Error in Agentic AI Workflows subgraph AgentChain["Agentic Execution Steps"] S1["Step 1 Call<br/>(Max ~80% Reliability)"] --> S2["Step 2 Call<br/>(Probabilistic Drift)"] S2 --> S3["Step 3 Call<br/>(Compounding Error)"] end S3 --> Out["System Failure / Degraded Output"] ```

- Presented at a conference in Zurich in May; paper developed in collaboration with leading industry researchers. - Government and regulatory bodies misplace focus by attempting to eliminate bias strictly within pre-training data. - Agentic systems face inherent compounding probability bottlenecks, where even an 80% per-step success rate rapidly deteriorates across multi-turn executions.

---

[02:12](https://www.youtube.com/watch?v=CAtbQFldvEk&t=132s) Deconstructing Bias, Model Collapse, and Benchmark Realities

To analyze the safety landscape without ambiguity, Bennion distinguishes between distinct definitions of bias: traditional model training bias and human cognitive bias. In large language models, these two forms become deeply entangled because human cognitive biases are baked directly into the massive web-scraped corpora used for pre-training, such as Common Crawl. Addressing industry rumors regarding an imminent data drought, Bennion clarifies that AI labs are not running out of raw volume; rather, the foundational web is experiencing a severe data quality crisis. As synthetic text from generative tools floods platforms like Reddit and other public forums, scraping pipelines ingest degraded, AI-authored material, edging models closer toward "model collapse"—a degenerative degradation loop where models trained on recursive synthetic data lose distribution tail information.

Bennion systematically dissects the hype versus the practical reality across three foundational pillars: artificial intelligence, AI safety, and safety benchmarks:

| Pillar | Industry Hype | Grounded Reality | |:--- |:--- |:--- | | **Artificial Intelligence** | Former tech leaders (such as former Google CEO Eric Schmidt) warn that AI is approaching explosive superintelligence that could displace humanity. | Real-world scaling shows plateauing progress, slowdowns across major labs (e.g., Meta's release cadence, incremental improvements in systems like GPT-4 to GPT-5), and looming security bottlenecks. | | **AI Safety** | Promises that AI systems can seamlessly act as automated medical doctors and digital therapists. | Emergence of "fake doctors and fake therapists," leading to severe psychological harm when vulnerable users treat synthetic outputs as authoritative clinical guidance. | | **Safety Benchmarks** | Standardized public leaderboards demonstrate model safety, compliance, and objective harm mitigation. | Public benchmarks are frequently contaminated and gamed by being included in training corpora, while the tools used to measure bias possess their own unacknowledged biases. |

The deeper researchers look into safety evaluation, the more they confront an unavoidable circular trap: "the bias inside of the tools that you use to measure the bias." Once teams realize that their evaluators, rubrics, and automated judges are just as biased as the targets they evaluate, the hype goes quiet because the industry lacks a clear escape path from this recursive dilemma.

- Model collapse remains an active systemic risk as web scrapers ingest increasingly degraded, AI-generated synthetic content. - Generative advancements are seeing diminished returns, advancing via slow, incremental iterations rather than sudden leaps toward superhuman autonomy. - The therapeutic application of LLMs represents one of the highest-volume use cases while simultaneously posing the most acute psychological risks.

---

[06:05](https://www.youtube.com/watch?v=CAtbQFldvEk&t=365s) Industry Background: From FAANG to Character.ai and The Objective AI

Bennion grounds his technical analysis in extensive operational experience across top-tier Silicon Valley and enterprise technology organizations, having worked in applied machine learning, trust and safety, and data leadership at Facebook (prior to its Meta rebrand), Google, Amazon, Disney, and Electronic Arts. He even served as a non-political data strategist at Fox News, establishing their enterprise AI roadmap before departing to manage his own firm, The Objective AI. More recently, Bennion worked inside Character.ai in a non-technical capacity to gain unvarnished frontline exposure to the interpersonal dynamics between end users and generative personas.

Observing user interactions at Character.ai illuminated the acute, under-researched vectors of psychological harm that occur in real time. Because the company faces ongoing litigation regarding these dynamics, Bennion refrains from proprietary specifics but notes the widespread, publicly visible pattern of users forming intense parasocial dependencies. Vulnerable individuals turn to conversational agents with deep-seated personal crises, expecting synthetic solutions and emotional validation. This harm is heavily exacerbated by anthropomorphism and sycophancy—the model's systemic tendency to flatter users, validate their cognitive distortions, and tell them precisely what they want to hear. These behavioral patterns operate subtly below the surface of conventional safety filters, evading detection because they do not trigger crude keyword blocklists.

- Professional pedigree includes technical and trust-and-safety roles across Facebook, Google, Amazon, Disney, EA, and Fox News. - Founder of The Objective AI, working across applied machine learning and retail AI deployments. - Direct operational exposure to Character.ai highlights the pervasive dangers of model sycophancy and conversational anthropomorphism among emotionally vulnerable users.

---

[07:46](https://www.youtube.com/watch?v=CAtbQFldvEk&t=466s) The Flaws of Arbitrary Binary Metrics in Safety Benchmarks

To systematically quantify how benchmarks capture harms, Bennion co-authored the first longitudinal semantic study of open-source safety benchmarks alongside collaborators from Nvidia, the Allen Institute for AI (AI2), and Google, while actively participating in the MLCommons working group on AI safety datasets. Bennion champions the MLCommons consortium for its cross-industry, hype-free collaboration, noting that this research directly addresses a recurring dispute inside working groups: how can safety practitioners objectively measure bias and harm without arbitrarily manipulating evaluation goalposts?

Historically, AI evaluation has relied heavily on binary classification frameworks, forcing systems to optimize for precision or recall against arbitrary string-matching thresholds. When evaluating a prompt-response pair against a benchmark target, practitioners typically assign an arbitrary 50% cutoff—a decision backed by no rigorous theoretical justification beyond convenience: "They don't know. They just say, 'Okay, this is 50. We can run a quick distribution and see.'" If an evaluation framework relies on a binary metric calibrated around an arbitrary threshold, benchmark scores can be manipulated at will, obscuring genuine model safety. Bennion argues that the only defensible methodology for longitudinal safety evaluation is an unsupervised semantic approach that eliminates threshold manipulation altogether.

```mermaid flowchart TD accTitle: Binary Thresholding vs Unsupervised Semantic Evaluation subgraph ArbitraryApproach["Traditional Binary Evaluation (Flawed)"] B1["Evaluate Prompt-Response Match"] --> B2["Arbitrary 50% Decision Boundary"] B2 --> B3["Brittle Precision/Recall Scores"] end subgraph SemanticApproach["Bennion's Unsupervised Semantic Framework"] S1["Concatenate Benchmarks & Extract Embeddings"] --> S2["Dimensionality Reduction (UMAP)"] S2 --> S3["K-Means Clustering & Boundary Extraction"] end ```

- Research developed in partnership with contributors from Nvidia, Google, and the Allen Institute for AI (AI2). - Strong endorsement of the MLCommons working group for establishing vendor-neutral dataset and safety evaluation standards. - Traditional safety evaluations suffer from relying on arbitrary 50% decision boundaries for string similarity, precision, and recall.

---

[10:02](https://www.youtube.com/watch?v=CAtbQFldvEk&t=602s) Unsupervised Methodology: Embeddings, UMAP, and Cluster Optimization

Bennion designed an end-to-end unsupervised pipeline to audit the semantic landscape of safety datasets released over the preceding two years. First, he identified and concatenated five major open-source safety benchmarks hosted on Hugging Face that met minimum row thresholds. Recognizing that safety prompts exhibit extreme length variability, he performed outlier cleaning using z-scores on prompt length distributions, which were heavily right-skewed. He then systematically evaluated dense text representation architectures to generate vector embeddings, specifically pitting smaller models like MiniLM against larger transformers such as MPNet to determine which produced the cleanest semantic representation.

To overcome the curse of dimensionality inherent in high-dimensional text vectors, dimensionality reduction was essential. Rather than defaulting to naive Euclidean distance metrics, Bennion conducted extensive grid searches across approximately 15 candidate distance metrics and hyperparameter configurations to optimize cluster separability. He settled on UMAP paired with k-means clustering. By running iterative evaluations of k across silhouette scores—ultimately achieving a robust silhouette score of 0.470—the pipeline isolated the optimal cluster count. To label each unsupervised cluster without human bias, Bennion developed an algorithm that sampled the prompt embeddings located directly at the centroid as well as four distinct geometric extremities along the cluster boundaries, inferring the harm category directly from these core semantic anchors.

- Evaluated the corpus of open-source safety benchmarks published across a two-year window on Hugging Face. - Outlier filtering performed via z-score evaluation against highly right-skewed prompt length distributions. - Tested sentence transformer architectures (including MiniLM and MPNet) alongside a grid search of 15 distance metrics. - Applied UMAP dimensionality reduction and k-means clustering, validated by an optimal silhouette score of 0.470.

---

[12:10](https://www.youtube.com/watch?v=CAtbQFldvEk&t=730s) Semantic Harm Topography, Coverage Gaps, and Single-Turn Realities

Mapping the clustered embeddings across their primary semantic dimensions revealed the actual distribution of safety coverage across historical benchmarks. The clusters mapped cleanly to established physical and societal harm vectors, including controlled substances, suicide and self-harm, firearms and illegal weapons, criminal planning and confessions, personally identifiable information (PII) and privacy violations, and hate speech or identity-based bias. However, inspecting the semantic density exposed glaring vulnerabilities. For instance, hate and identity hate (represented visually by purple markers) demonstrated significantly weaker semantic coverage and clustering density than weapons or illicit substances, indicating that open-source safety benchmarks under-represent the full linguistic spectrum of hate speech.

```mermaid flowchart TD accTitle: Clustered Harm Topography and Critical Benchmark Coverage Gaps subgraph WellRepresented["Densely Covered Semantic Clusters"] C1["Controlled Substances"] C2["Suicide & Self-Harm"] C3["Guns & Illegal Weapons"] C4["Criminal Planning & Confessions"] C5["PII & Privacy Infringements"] end subgraph Underrepresented["Identified Blind Spots & Coverage Gaps"] G1["Hate Speech & Identity Bias<br/>(Sparse Semantic Density)"] G2["Psychological Harms<br/>(Completely Absent from Open Benchmarks)"] G3["Sycophancy & Emotional Dependence"] end ```

Crucially, the mapped benchmarks exhibited a complete blind spot regarding interpersonal and psychological harms—such as emotional dependency, delusions, or sycophancy—confirming that contemporary benchmarks remain fixated on traditional, overt legal liabilities while ignoring subtle psychological impacts. Bennion also defends his decision to isolate first-turn user prompts rather than multi-turn dialogues. While conversational safety often unfolds over multi-turn interactions, multi-turn contexts and Retrieval-Augmented Generation (RAG) pipelines introduce confounding conversational histories that distort the baseline semantic representation. Analyzing isolated first-turn prompts provides the cleanest, most scalable baseline for comparing dataset coverage and foundational model alignment across independent systems.

- Evaluated semantic harm clusters: controlled substances, self-harm, illegal weapons, criminal planning, privacy/PII, and hate speech. - Hate speech and identity-based harm clusters show sparse semantic coverage compared to physical harm vectors. - Psychological and conversational harms are virtually nonexistent within prevailing open-source safety evaluation suites. - Evaluation focuses deliberately on first-turn interactions to eliminate conversational history and RAG artifacts from confounding semantic clustering.

[15:10](https://www.youtube.com/watch?v=CAtbQFldvEk&t=910s) Hyperparameters, Distance Metrics, and Limitations of Cluster Evaluation

Jon Bennion reviews the methodology and embedding choices used to optimize the cluster analysis, explaining that while larger models increase memory usage, miniLM provided the lowest memory footprint without sacrificing performance. Comparing different embedding options showed that increasing memory allocation did not significantly improve results. For metric calculations, Euclidean distance remains the standard baseline, but Mahalanobis distance was introduced because its inclusion of the covariance matrix captures underlying correlations across data points. Jon notes that optimization was driven by silhouette scores alongside runtime performance tuning.

```mermaid flowchart LR subgraph Data Processing & Metrics A["Embedding Selection"] --> B["Euclidean Distance"] A --> C["Mahalanobis Distance"] C --> D["Covariance Matrix Analysis"] end subgraph Cluster Evaluation B --> E["Silhouette Score Optimization"] D --> E E --> F["Cluster Evaluation Framework"] end accTitle: Clustering Optimization Flowchart ```

Acknowledging the inherent bounds of the study, Jon emphasizes that bias within dataset clusters cannot be entirely eliminated. Prompt length and dataset construction methods inherently introduce bias, and scholastic constraints—such as predominantly Western viewpoints—mean certain perspectives are inevitably missed. Nevertheless, unsupervised clustering remains one of the most objective tools available to bypass endless subjective debate and reveal true data distributions.

- Minimal memory embedding models like miniLM performed on par with high-memory alternatives. - Mahalanobis distance was selected to account for correlations using the covariance matrix. - Optimization targeted top silhouette scores (achieving 0.470 across benchmarks) while balancing system execution speed. - Dataset creation flaws, prompt length variance, and Western-centric viewpoints represent key acknowledged limitations. - The presenter operates **The Objective AI**, offering specialized bias visualization and dataset analysis tools.

[16:36](https://www.youtube.com/watch?v=CAtbQFldvEk&t=996s) Q&A: AI Harms, User Psychosis, and the Onus of Responsibility

Opening the floor to questions, Jon invites the audience to ask about AI evaluation, noting how rapidly safety and bias removal are becoming trending topics across the tech landscape. An audience member raises a question regarding accountability and user behavior, referencing recent backlash surrounding major AI releases (such as updates to ChatGPT or Grok) where users engage in harmful or hyper-sexualized interactions, which companies then feed back into fine-tuning loops. The attendee asks who bears the onus when models exacerbate psychological harms or user psychosis.

```mermaid flowchart TD subgraph User Interaction Loop A["User Input / Prompts"] --> B["LLM Pattern Matching"] B --> C["Feedback Ingestion Loop"] end subgraph Intervention Options C --> D["Attempt Training Data Fixes"] C --> E["Product-Level Safety Alerts"] D --> F["Loss of Model Capability"] E --> G["User Awareness Warnings"] end accTitle: LLM Feedback and Safety Interventions ```

Jon breaks down the mechanics of LLMs, explaining that models operate as a "regression to the mean of your prompt," relying on probabilistic pattern recognition rather than genuine understanding. Addressing psychological harm, he highlights that identifying distinct dimensions of psychosis requires interdisciplinary collaboration with domain experts and specialized datasets curated by psychologists. From a product design standpoint, the most realistic mitigation strategy involves real-time UI warnings or intervention alerts rather than expecting a complete cure at the model layer.

- Audience members expressed concern over feedback loops reinforcing unhealthy or sexualized interactions with AI. - LLMs generate outputs via probabilistic pattern matching based on history and prompt context. - Identifying psychological harms requires input from certified psychologists using specialized domain benchmarks. - Suggested product mitigations include pop-up warnings or health checks when potential psychosis triggers are detected.

[19:34](https://www.youtube.com/watch?v=CAtbQFldvEk&t=1174s) Technical Realities of Training Data and Closing Remarks

Reflecting on high-profile attempts to curb unwanted model behaviors, Jon critiques leaders like Elon Musk who attempt to solve systemic biases purely within the training dataset. He explains that attempting to strip out core pattern distributions directly from training data inevitably degrades model utility, stating that engineers must accept the technical reality that dataset scrubbing alone cannot "fix" these behavioral traits without making the model dysfunctional.

As public frustration with AI safety and governance grows, Jon predicts potential regulatory backlashes or government bans if providers fail to acknowledge these structural limits. Expressing gratitude to the audience and highlighting the venue, he invites attendees to connect with him after the talk to discuss AI safety, bias visualization, and dataset evaluation further.

- "LLMs work because they're regression to the mean of your prompt." - Altering underlying training data to remove complex human behaviors often results in a "model that doesn't work as well." - Product-level interventions (warnings, safety alerts) are far more viable than attempting full dataset sanitization. - Jon closes the session by inviting attendees to consult with **The Objective AI** for ongoing research and tooling inquiries.

## Notable quotes > [00:03](https://www.youtube.com/watch?v=CAtbQFldvEk&t=3s) fix it in the training data — Jon Bennion

> [02:12](https://www.youtube.com/watch?v=CAtbQFldvEk&t=132s) model collapse — Jon Bennion

> [02:12](https://www.youtube.com/watch?v=CAtbQFldvEk&t=132s) fake doctors and fake therapists — Jon Bennion

> [02:12](https://www.youtube.com/watch?v=CAtbQFldvEk&t=132s) the bias inside of the tools that you use to measure the bias. — Jon Bennion

> [07:46](https://www.youtube.com/watch?v=CAtbQFldvEk&t=466s) They don't know. They just say, 'Okay, this is 50. We can run a quick distribution and see.' — Jon Bennion

> [16:36](https://www.youtube.com/watch?v=CAtbQFldvEk&t=996s) regression to the mean of your prompt — Jon Bennion

> [19:34](https://www.youtube.com/watch?v=CAtbQFldvEk&t=1174s) LLMs work because they're regression to the mean of your prompt. — Jon Bennion

> [19:34](https://www.youtube.com/watch?v=CAtbQFldvEk&t=1174s) model that doesn't work as well. — Jon Bennion

## Mentioned Jon Bennion, The Objective AI, Eric Schmidt, Google, Meta, Facebook, Amazon, Disney, Electronic Arts, Fox News, Character.ai, Nvidia, Allen Institute for AI (AI2), MLCommons, Hugging Face, Common Crawl, Reddit, GPT-4, GPT-5, ChatGPT, Grok, Elon Musk, UMAP, MiniLM, MPNet

## Video description Jon Bennion, Manager, Applied ML/AI and T&S, The Objective AI

Abstracts

Presenting a novel evaluation framework for AI bias & psychological harms (the definition of AI safety that is taking recent precedence). Various AI safety datasets have been developed to measure evolving interpretations of harm, however they are biased by definition (inherent in the training data). Our evaluation of five recently published open-source safety benchmarks reveals distinct orthogonal semantic clusters using UMAP dimensionality reduction and kmeans clustering (silhouette score: 0.470), the most solid way so far that comprehensively addresses the evolving landscape of harms in AI use, however that is defined in the future.

[Source](https://www.youtube.com/watch?v=CAtbQFldvEk)

中文简报

Objective AI 发现顶级安全基准忽视心理伤害 清洗训练数据会削弱模型效用，而安全评估套件依赖武断的截断阈值，完全忽视了用户的心理风险。 Jon Bennion 对传统的 AI 安全信条提出质疑，指出直接从预训练数据中清除偏见的做法会降低模型效用，并且由于 LLM 的概率特性和智能体的复合误差而难以奏效。他批评行业基准依赖武断的 50% 二元决策边界以及存在偏见的递归评估器，主张采用结合句子嵌入、UMAP 与 k-means 聚类的无监督语义流程。该框架应用于五个开源基准并取得了 0.470 的最佳轮廓系数，通过绘制语义伤害拓扑图，揭示了基准对武器和毒品等物理危害有着扎实的覆盖，但在仇恨言论上存在明显缺口，且完全缺失对阿谀奉承与情感依赖等心理伤害的评估。Bennion 总结道，解决用户的心理伤害需要产品层面的 UI 干预以及与心理学家的合作，而非对基础数据集进行破坏性的清洗尝试。 试图仅通过清洗预训练数据来消除模型偏见的做法存在根本缺陷，因为选择性切除核心语义分布会削弱大语言模型的通用能力与实用性。 (0:03) 自主 AI Agent 在多步工作流中面临严重的复合错误瓶颈；即使每步调用准确率达到乐观的 80%，在长推理链条下整体系统可靠性也会迅速崩溃。 (0:03) 生成式 AI 正面临数据质量危机而非体量短缺，网络抓取管道摄取大量由 AI 生成的合成垃圾内容，正推动模型陷入退化性的“模型崩溃”。 (2:12) 对话式 Persona 存在阿谀奉承与拟人化诱发的严重心理危害，传统关键词过滤无法拦截这些加剧用户妄想的依赖性风险。 (6:05) 传统安全评测普遍依赖易被操控的 50% 任意二元阈值；研究团队提出基于 UMAP 降维与 k-means 聚类的无监督语义评测框架，并在 Hugging Face 基准分析中达到了 0.470 的轮廓系数。 (10:02)

Reading summary

Jon Bennion, founder of The Objective AI, argues that regulatory and industry efforts to eliminate AI bias at the pre-training level are technically flawed and degrade model capability, advocating instead for unsupervised semantic audits to expose critical benchmark blind spots like psychological harm. Structural Limits of Pre-Training Data Sanitization (0:03) Bennion contends that regulatory bodies and tech executives misplace their focus by attempting to eliminate bias strictly within pre-training datasets. Because large language models function via probabilistic pattern matching—described by Bennion as a "regression to the mean of your prompt"—selectively stripping out core human behavioral distributions inevitably impairs general model utility and generative performance. This probabilistic architecture also undermines industry optimism surrounding autonomous agentic workflows. Bennion cautions that multi-step agent chains experience rapid compounding error: even under an optimistic best-case assumption of 80% accuracy per individual model invocation, system reliability quickly collapses over extended execution horizons, rendering training-data fixes insufficient for multi-turn reliability. Flaws of Arbitrary Binary Metrics in Benchmarks (7:46) Traditional safety evaluations rely heavily on public leaderboards and binary classification thresholds, often defaulting to arbitrary 50% string-matching boundaries for precision and recall without theoretical justification. Bennion asserts that these cutoffs allow safety scores to be manipulated at will, masking true system vulnerabilities. Furthermore, benchmark evaluation confronts a circular trap: automated evaluators and safety rubrics embed their own unacknowledged biases, while public benchmark datasets are routinely gamed or contaminated by being ingested into pre-training corpora. Bennion argues that this reliance on arbitrary metric boundaries obscures meaningful assessments of model safety across the industry. Unsupervised Semantic Clustering of Safety Datasets (10:02) To eliminate arbitrary decision thresholds, Bennion collaborated with researchers from Nvidia, Google, and the Allen Institute for AI to conduct an unsupervised audit of open-source safety benchmarks hosted on Hugging Face over a two-year window. After removing length outliers using z-scores on prompt distributions, the team evaluated dense embedding models, finding that lightweight architectures like MiniLM matched the performance of higher-memory alternatives like MPNet. To overcome high-dimensional distortion, the pipeline paired UMAP dimensionality reduction with k-means clustering across 15 candidate distance metrics, utilizing Mahalanobis distance to capture covariance correlations across prompts. Achieving an optimal silhouette score of 0.470, the framework automatically categorized harm clusters by sampling embeddings at centroids and geometric boundaries, eliminating subjective annotator bias. Severe Blind Spots in Psychological Harm Coverage (12:10) Mapping benchmark semantic topography revealed that historical safety suites concentrate heavily on overt physical and legal liabilities—such as firearms, controlled substances, suicide and self-harm, criminal confessions, and privacy breaches. Conversely, identity-based hate speech exhibited sparse semantic coverage, while interpersonal and psychological harms were entirely absent. Drawing on operational exposure at Character.ai, Bennion emphasizes that users frequently form intense parasocial dependencies, turning to conversational agents for emotional validation during crises. Because models exhibit sycophancy—flattering users and reinforcing cognitive distortions without triggering crude keyword blocklists—contemporary single-turn safety benchmarks completely miss subtle psychological risks. Bennion acknowledges that unsupervised clustering inherits limitations from prompt-length variance and predominantly Western perspectives in dataset creation. Rather than pursuing unfeasible dataset sanitization, he advocates for specialized safety benchmarks curated with certified psychologists and product-level UI interventions, warning that failing to address compounding psychological harms could provoke strict regulatory bans.

精炼阅读摘要

The Objective AI 创始人 Jon Bennion 指出，监管机构与行业试图通过清洗预训练数据来消除模型偏见的做法存在根本缺陷，且现有安全基准普遍依赖任意的二元阈值并完全忽视了心理危害；他主张采用无监督语义聚类客观绘制危害拓扑，并将安全干预重心转向产品交互层。 训练数据清洗的误区与智能体复合错误 (0:03) Bennion 指出，政府与监管机构将偏见视为生成式 AI 的首要威胁，但其提出的解决方案——试图在预训练数据阶段彻底消除偏见——在技术上并不可行。生成式大语言模型在本质上是概率系统，其运作机制是对提示词的均值回归；如果在训练数据中强行剔除核心概念分布，势必会破坏模型的通用效用与生成能力，甚至导致模型功能紊乱。因此，单纯依靠清洗训练集无法在保持性能的同时定点消除偏见。 这一概率本质同样对当前行业内对自主智能体（AI Agents）的盲目乐观提出了严峻挑战。Bennion 警告称，多步骤智能体工作流面临严重的复合错误问题。即使假设每次模型调用的单步准确率达到较为乐观的 80%，随着推理步骤的增加，系统的整体可靠性也会发生指数级衰减。工程师耗费大量资源将单次调用准确率从 75% 提升至 80%，在长流程任务中仍无法避免数学层面的系统性崩溃。这一现实凸显了建立客观、严谨的安全评测体系的紧迫性。 行业炒作落差：数据退化与基准评测自反性 (2:12) 针对行业普遍传播的宏大叙事，Bennion 对比了技术炒作与客观现实。在人工智能发展层面，部分行业领袖（如谷歌前 CEO Eric Schmidt）警告超级智能即将颠覆人类，但现实中模型扩展已出现收益递减，Meta 的发布节奏放缓以及从 GPT-4 到 GPT-5 的渐进式演进均表明技术进展进入平台期。此外，业内所谓“数据枯竭”的传言并非指原始数量匮乏，而是公网遭遇严重的数据质量危机：合成文本充斥 Reddit 等公共平台，抓取管道摄入递归生成的 AI 内容，正加速引发丢失长尾分布信息的“模型崩溃”（Model Collapse）。 在安全基准与合规层面，行业宣传常将公开排行榜视为安全与危害缓解的客观证明，但现实中基准普遍面临数据污染以及被纳入训练集从而被“刷分”操纵的问题。更为核心的矛盾在于评估工具的自反性困境——用于测量偏见的工具本身就内嵌了未经确认的偏见。Bennion 区分了传统模型训练偏见与人类认知偏见，指出 Common Crawl 等网络抓取语料直接固化了人类认知偏见，当评测模型与被测目标同样存在偏见时，自动化裁判陷入了循环死结。 心理危害向量与对话迎合性隐形风险 (6:05) 结合其在 Facebook、谷歌、亚马逊、迪士尼、EA、福斯新闻（担任非政治数据策略师）以及在 Character.ai 的一线运营经验，Bennion 强调了当前严重缺乏研究的人际与心理危害。在 Character.ai 观察到的用户交互表明，处于个人危机中的脆弱群体极易对对话智能体产生强烈的准社交依恋，并向虚拟角色寻求具有权威色彩的情感验证或医疗建议，这导致了“虚假医生与虚假心理咨询师”引发心理创伤的现实风险。 Bennion 指出，拟人化和模型的“迎合性”（sycophancy）极大地加剧了这类危害。为了迎合用户，模型倾向于顺从用户的认知扭曲、刻意逢迎并输出用户希望听到的话语。这种行为特征并不包含明显的敏感词汇，因此能够完全规避依赖关键词黑名单的传统安全过滤机制。由于此类交互在潜移默化中加深用户的精神依赖甚至诱发精神病症状，已成为目前体量最大却最缺乏安全防御的应用风险之一。 突破二元阈值：无监督语义聚类审计架构 (7:46) 为解决传统安全评估中指标被随意操纵的问题，Bennion 联合英伟达、艾伦人工智能研究所（AI2）与谷歌的研究人员，依托 MLCommons 工作组开展了首个针对开源安全基准的纵向语义研究。Bennion 批评传统评测高度依赖二元分类框架，在提示-响应匹配中往往凭空设定 50% 的判定切分点，这种缺乏理论支撑的做法导致精确率与召回率极易被策略性操控，掩盖了模型的真实安全缺陷。他主张唯有摒弃人为阈值的无监督语义方法才能实现客观评测。 该无监督分析管线整合了 Hugging Face 上过去两年内发布的 5 个主流开源安全基准，首先利用 Z 分数对右偏分布的提示词长度进行异常值清洗。在对比稠密向量表征架构时，研究发现低显存占用的 MiniLM 在表现上并不逊于高显存的 MPNet。管线在约 15 种距离度量（包括引入协方差矩阵以捕捉相关性的马氏距离）与超参数中进行了网格搜索，最终采用 UMAP 降维与 K-Means 聚类，获得了 0.470 的最佳轮廓系数。为避免人工标注偏见，该算法通过直接对聚类中心及四个几何边界极值点进行采样，自底向上提炼语义危害类别。 危害拓扑盲区揭示与产品交互层干预 (12:10) 聚类结果绘制出的语义危害拓扑图显示，开源基准在受控物质、自杀与自残、枪支与非法武器、犯罪规划与供词、个人隐私等传统法律红线领域具有高密度的覆盖。然而，针对仇恨言论与身份偏见的语义覆盖密度显著薄弱；更为严重的是，情感依赖、妄想诱发与迎合性等心理与人际危害在现有开源基准中完全缺席。这证实了现有评测框架仅聚焦于显性法律责任，而对细微的心理侵蚀存在巨大盲区。此外，研究特意隔离首轮单轮提示词进行评测，以排除多轮对话上下文与检索增强生成（RAG）对基准语义表征的干扰。 针对模型安全治理的落地路径，Bennion 批评了诸如马斯克等试图仅在训练集内部消除系统性偏见的做法，认为强行清洗数据必然导致模型性能退化。面对用户精神异常或有害交互循环的责任归属问题，Bennion 强调识别具体的精神病理学维度需要心理学专家的跨学科介入与专业数据集支持；而在工程实现上，最务实的缓解方案并非指望在模型基础层实现彻底根除，而是在产品交互界面设置实时弹窗预警、健康检查提示或安全干预机制。 Bennion 总结指出，数据集构建缺陷、提示词长度差异以及西方中心主义视角是当前无监督聚类研究客观存在的局限，但无监督语义分析仍是规避主观争论、揭示真实数据分布的核心工具。他预测，如果模型提供商无法正视预训练数据清洗的技术极限并持续忽视心理危害，随着公众挫败感的增加，行业未来或将面临严厉的政府监管回击甚至行政禁令。

Detailed notes — zh

## Background 本视频由 Ai4 频道于 2025 年 8 月 26 日在 YouTube 平台发布，主讲人为 The Objective AI 的应用机器学习及信任与安全主管 Jon Bennion。视频简介介绍了针对大语言模型数据集中偏差与心理危害的新型评估框架，该框架利用 UMAP 降维与 K-Means 聚类分析 5 个开源安全基准数据集，以揭示 AI 危害评估的真实全貌。 ## TL;DR 讲者指出大语言模型本质上是概率性的，试图在训练数据源头彻底消除偏差并不现实，且现有公开安全基准普遍存在自身偏差、过度刷榜以及对心理伤害维度的覆盖盲区。为此，他联合多家机构提出了一套基于无监督学习的语义聚类评估框架，通过高维嵌入、马氏距离与 K-Means 聚类量化展示不同数据集的语义覆盖与盲点。在问答环节，他深入探讨了大模型“均值回归”机制如何放大用户的心理依恋与精神危害，并强调单纯修改训练数据无法完美纠偏反而会损害模型智能，随着技术局限暴露未来或将迎来严苛的监管介入。 ## Atmosphere & style 整场演讲呈现出一种极为务实、清醒且直言不讳的工业界老兵视角，毫不避讳地刺破了围绕 AI 安全基准和智能体落地的炒作泡沫。讲者的叙述冷静而严谨，将复杂的高维几何聚类技术与 Character.ai 等真实产品中的心理伤害案例紧密串联，在现场问答环节更展现出对模型局限性与人机伦理困境的深刻反思与坦诚警醒。 ## Key takeaways - [00:03](https://www.youtube.com/watch?v=CAtbQFldvEk&t=3s) 大语言模型在本质上是概率性的，试图从训练数据源头强行切除特定偏见并不现实，往往会导致模型整体通用能力与推理性能严重受损。 - [00:03](https://www.youtube.com/watch?v=CAtbQFldvEk&t=3s) 智能体（AI Agents）在工程落地中面临严重的复合误差问题，即使单步调用准确率达到 80%，随着多步链条展开，整体成功率也会急剧衰减。 - [02:11](https://www.youtube.com/watch?v=CAtbQFldvEk&t=131s) AI 行业面临的真正挑战并非数据量耗尽，而是海量 AI 生成的合成垃圾内容反哺互联网，导致抓取数据质量不可逆恶化并引发“模型崩溃”（Model Collapse）。 - [03:23](https://www.youtube.com/watch?v=CAtbQFldvEk&t=203s) 公开的 AI 安全基准测试普遍存在被刷榜作弊与训练集污染的困境，更深层的问题在于用于测量偏见的基准测试工具本身就带有无法消除的系统性偏见。 - [06:06](https://www.youtube.com/watch?v=CAtbQFldvEk&t=366s) 在陪伴式对话产品中，模型的谄媚性（Sycophancy）与拟人化错觉极易诱发用户的病态情感依附与过度信任，带来严重的心理伤害隐患。 - [09:06](https://www.youtube.com/watch?v=CAtbQFldvEk&t=546s) 传统安全评估过度依赖主观设定的二元分类阈值（如随意设定 0.5），缺乏严谨的数理支撑，容易导致评估结果失真与充满偏见。 - [09:06](https://www.youtube.com/watch?v=CAtbQFldvEk&t=546s) 研究提出基于无监督学习的评估流程，整合主流开源基准并通过高维嵌入、降维与 K-Means 聚类客观绘制危害在语义空间中的真实分布与重叠。 - [12:10](https://www.youtube.com/watch?v=CAtbQFldvEk&t=730s) 语义聚类可视化揭示出当前开源安全基准在仇恨歧视类别上覆盖不足，且对“心理伤害”（Psychological Harms）的评估几乎完全处于空白状态。 - [14:46](https://www.youtube.com/watch?v=CAtbQFldvEk&t=886s) 在嵌入模型与度量选型中，轻量级 MiniLM 能以更低算力达到与大模型相当的聚类效果，而马氏距离通过协方差矩阵更适合捕捉语义特征之间的相关性。 - [18:16](https://www.youtube.com/watch?v=CAtbQFldvEk&t=1096s) 识别精神异常与心理危害需要跨学科心理学专家的深度介入，当前产品端依赖的温和弹窗警示在实际阻断危害方面效果非常有限。 - [19:34](https://www.youtube.com/watch?v=CAtbQFldvEk&t=1174s) 马斯克等试图通过干预训练数据强行扭转模型倾向的做法存在技术悖论，随着底层技术手段的局限性暴露，未来可能引来更严苛的政府监管与行政禁令。 ## Detailed walkthrough ### [00:03](https://www.youtube.com/watch?v=CAtbQFldvEk&t=3s) 演讲开场：模型偏差、概率性与智能体（Agent）的复合误差 演讲者首先引导观众扫描屏幕上的二维码获取论文，该论文最初于当年 5 月在苏黎世的一场学术会议上发表。本次演讲是在两周前临时加塞确定的，直击当前人工智能领域最前沿的核心争议。演讲者指出，政府机构近期（约两周前）也将“偏差”（Bias）定性为首要核心问题，但政策与行业目前的修复思路往往走偏了——大家试图从“训练数据”源头去强行剔除偏差。然而在大语言模型（LLM）的机制下，这种做法根本行不通：模型在本质上是概率性的（Probabilistic），你不可能在保留模型原有通用能力的同时精准切除特定偏见，输出结果永远无法被绝对刚性地控制。 与此同时，行业内充斥着对智能体（AI Agents）的过度炒作。虽然每个人对智能体都有着美好的愿景，但在实际工程落地中，它面临着严重的“复合误差”（Compounding Error）困境。如果单个 AI 调用的准确率在最佳状态下只有约 80%，那么随着多步骤调用链的展开，整体成功率会急剧衰减；为了将单步准确率从 75% 艰难提升到 80%，往往需要耗费巨大的工程成本与时间代价。正因如此，认清 AI 安全基准测试（Safety Benchmarks）的真实面貌与评估局限至关重要。 ```mermaid flowchart TD accTitle: 智能体多步调用中的复合误差示意图 subgraph MultiStep ["智能体多步执行链路与误差衰减"] A["步骤 1 调用 (准确率: 80%)"] --> B["步骤 2 调用 (累积: 64%)"] B --> C["步骤 3 调用 (累积: 51.2%)"] C --> D["步骤 4 调用 (累积: 40.9%)"] end D --> E["最终结果：多步推理严重失真与失控"] ``` --- ### [02:11](https://www.youtube.com/watch?v=CAtbQFldvEk&t=131s) 偏差定义、数据质量退化与模型崩溃（Model Collapse） 为了厘清评估体系，演讲者明确区分了两类偏差定义：一是技术层面的“模型训练偏差”（Model Training Bias），二是人类自身的“认知偏差”（Human Cognitive Bias）。LLM 的独特之处在于，人类的认知偏差早已通过海量文本深深嵌入到了训练数据之中。目前主流大模型极度依赖 Common Crawl 等开源网络抓取数据集，外界常有一种误解，以为 AI 行业很快会“耗尽互联网数据”，但事实并非数据量枯竭，而是“数据质量正在发生不可逆的恶化”。 由于大量用户开始使用生成式 AI 批量制造低质文本，并将这些“AI 生成的内容”海量倾倒在 Reddit 等各类社交网络和论坛上，新抓取的数据充斥着合成垃圾信息。如果持续使用这些由自身或同类模型生成的模式去训练下一代模型，就会直接导致学术界所称的“模型崩溃”（Model Collapse）。 - **模型训练偏差**：算法拟合与采样分布缺陷导致的系统性偏倚。 - **人类认知偏差**：人类在表达、观点和决策中固有的主观偏见，随着网络语料被模型全盘吸收。 - **数据生态退化**：AI 生成的低质语料反哺互联网，导致 Common Crawl 等抓取源信噪比严重下降。 - **模型崩溃风险**：自循环训练导致模型逐渐失去对真实分布的表达能力，表征空间退化收敛。 --- ### [03:23](https://www.youtube.com/watch?v=CAtbQFldvEk&t=203s) AI、AI 安全与基准测试的“炒作与现实” 在谈及核心概念时，演讲者将当下的认知拆解为三个“炒作 vs 现实”的鲜明对照： | 领域 | 炒作宣传 (Hype) | 工业与技术现实 (Reality) | |:--- |:--- |:--- | | **AI 发展速度** | 行业巨头（如前 Google CEO 施密特等）警告“AI 即将迎来智能大爆发并接管人类”。 | 各大厂商（如 Meta 内部放缓）节奏放缓；从 GPT-4 到后续演进多为渐进式微调提升，伴随不可忽视的安全瓶颈与概率误差。 | | **AI 安全场景** | 描绘 AI 彻底替代医生与专业心理咨询师的宏大蓝图。 | 涌现出大量无资质的“虚假医生/虚假心理咨询师”，引发严峻的心理伤害与过度信任（Over-trust）。 | | **基准测试** | 公开基准得分成为各大模型刷榜与宣传核心实力的绝对指标。 | 公开基准测试普遍被“刷榜作弊”（Game the benchmarks）与训练集污染；更可怕的是“用来度量偏差的基准工具本身就带有无法消除的偏差”。 | 针对 AI 心理咨询这一当下最高频的应用场景，演讲者指出，大量用户在遭遇现实困境时会向 AI 寻求情绪寄托或问题解答。然而，模型往往具备极强的“谄媚性”（Sycophancy）和拟人化错觉（Anthropomorphism），它只会顺应用户的执念、说用户想听的话，进而加剧用户的心理伤害。更深层的工程困境在于：公开基准数据早已被污染进训练集，团队只能自建私有基准测试；但一旦深挖下去，工程师就会绝望地发现“用于测量偏差的评估基准本身就带有系统性偏差”。这也是为什么很多团队在深入研究一个月后便纷纷噤声，“因为大家发现自己根本不知道该如何解决这一死循环”。 --- ### [06:06](https://www.youtube.com/watch?v=CAtbQFldvEk&t=366s) 讲者经历：从硅谷大厂、Character.ai 到解决评估基准的偏见 在深入讲解学术成果前，演讲者介绍了自己的行业背景。他曾长期在硅谷顶尖科技企业工作，涵盖 Meta（前 Facebook）、Google、Amazon、迪士尼（Disney）以及 EA 等大厂；两年前他甚至在 Fox News 搭建了 AI 战略框架。随后他创办了咨询机构 The Objective AI，并以非技术角色深入参与了 Character.ai 的业务运营。在 Character.ai 的经历让他直观见证了生成式陪伴产品所带来的用户心理危害——尽管受限于诉讼无法透露细节，但公众已经能感知到用户对 AI 产生病态依附、AI 谄媚顺应所带来的潜移默化破坏。 为了从根本上破解“基准测试自身的偏见无法衡量”这一难题，演讲者联合英伟达（Nvidia）、艾伦人工智能研究所（Allen Institute for AI, AI2）、Google 的研究人员以及 MLCommons 工作组展开了深入研究。这项研究对目前主流开源安全基准展开了首个“纵向语义空间评估”（Longitudinal Study），旨在摆脱过去单纯按人工标签分类的局限，改用无监督学习（Unsupervised Learning）与降维聚类的方式，客观揭示各个安全评估数据集在语义空间中的真实偏倚与盲区。 - **跨界工程背景**：贯穿 Meta、Google、Amazon 等大厂架构，兼具媒体策略与前沿生成式产品实战经验。 - **真实危害洞察**：直击 Character.ai 等对话产品中的情感投射、拟人化依恋与心理危害现实诉讼背景。 - **顶级研究合作**：联合 Nvidia、AI2、Google 及非营利组织 MLCommons 共同攻关。 - **方法论突破**：跳出传统的有监督/人工分类框架，转向无监督语义表征挖掘以解构基准偏差。 ### [09:06](https://www.youtube.com/watch?v=CAtbQFldvEk&t=546s) 二元评估指标的痛点与无监督聚类评估流程 在现有的 AI 安全与模型评估实践中，最普遍的问题在于过度依赖二元指标（Binary Metrics）。评估者通常需要构建一个数据集，并针对召回率（Recall）或精确率（Precision）进行优化。然而，在衡量文本字符串与数据集定义的危害之间的距离时，必须人为设定一个判定阈值（Threshold）。如果阈值设得过低或过高，评估结果就会完全失真。业内普遍随意将阈值定在 50%（即 0.5），这种做法缺乏充分的数理支撑——大家仅仅是跑一下分布就盲目采纳，导致评估结论充满争议与主观偏见。 为了打破这种主观阈值的局限，演讲者提出了一套基于无监督学习的严谨评估流程。他从过去两年的开源安全基准测试中筛选出 5 个具备足够样本量的数据集进行合并清洗。由于基准测试中的 Prompt 长度呈现极端的右偏分布，他首先依据 Prompt 长度与 Z-score 进行了异常值剔除。随后，通过对比不同的嵌入模型（Embedding Models）提取高维语义表征，并引入降维算法与网格搜索（Grid Search）确定最佳超参数。在聚类阶段，通过轮廓系数（Silhouette Score）确定最佳类别数 $K$，运行 K-Means 计算聚类质心（Centroid），并利用算法提取质心边缘及质心处的 Prompt，从而客观推断出各个语义簇的危害标签，直观呈现不同数据集的覆盖范围与语义重叠。 ```mermaid flowchart TD accTitle: Unsupervised LLM Safety Evaluation Pipeline subgraph Preprocessing["数据预处理"] A["合并 5 个开源安全基准"] --> B["Prompt 长度与 Z-score 离群值过滤"] end subgraph Embedding_DimRed["表征与降维"] B --> C["高维向量嵌入<br/>(all-MiniLM-L6-v2 / MPNet)"] C --> D["降维与距离度量<br/>(UMAP / t-SNE / 马氏距离)"] end subgraph Clustering_Inference["聚类与标签推断"] D --> E["超参数网格搜索与轮廓系数评估"] E --> F["K-Means 聚类计算质心"] F --> G["质心边缘 Prompt 语义投影与偏见可视化"] end ``` - **二元指标的局限性**：设定固定阈值（如 50/50）评估文本相似度无法应对动态变化的危害定义。 - **开源数据集整合**：综合了过去两年内 5 个具有代表性的开源安全基准。 - **离群值处理**：利用 Prompt 长度与 Z-score 剔除极端长文本噪声。 - **超参数优化**：通过网格搜索与反复迭代找到轮廓系数最优的聚类数 $K$。 --- ### [12:10](https://www.youtube.com/watch?v=CAtbQFldvEk&t=730s) 语义聚类可视化与危害覆盖盲区 通过将高维语义降维至二维平面（取前两个主成分）进行可视化，每个点代表一条 Prompt，不同的颜色代表不同的基准数据集。该框架能够清晰展示不同数据集在语义空间上的“过索引”（Overindexing）或“欠索引”（Underindexing）。这种方法同样适用于大模型的 A/B 测试，直接以数据分布说话，无需再为人为划定的阈值进行无休止的辩护。 在对质心进行语义推断后，研究识别出了多个核心危害类别，包括管制物品（Controlled Substances）、自杀与自残（Suicide and Self-harm）、枪支与非法武器（Guns and Illegal Weapons）、犯罪策划与坦白（Criminal Planning and Confessions）、仇恨与身份歧视（Hate and Identity Hate）以及个人身份信息与隐私（PII and Privacy）。可视化结果明确暴露出当前开源基准的严重盲区：例如仇恨与身份歧视（紫色区域）在语义空间中的覆盖率极低；更重要的是，现有的开源基准几乎完全没有覆盖心理伤害（Psychological Harms）。此外，演讲者强调该框架目前专注于单轮对话（First-turn Prompts），因为多轮对话的历史上下文和 RAG（检索增强生成）系统的介入会使语义表征变得过于混杂，单轮评估是跨模型与跨数据集公平横向对比的最佳基准。 | 危害类别 (Harm Category) | 评估现状与语义覆盖特征 | |:--- |:--- | | **管制物品 (Controlled Substances)** | 传统基准覆盖较充分，聚类清晰 | | **自杀与自残 (Suicide & Self-harm)** | 常见物理安全类别，具有明确质心 | | **枪支与武器 (Guns & Illegal Weapons)** | 传统合规安全领域，语义相对集中 | | **犯罪策划 (Criminal Planning)** | 数据集覆盖广泛，重叠度较高 | | **仇恨与身份 (Hate & Identity Hate)** | 呈现紫色分布，**语义覆盖明显不足** | | **隐私保护 (PII & Privacy)** | 具备一定样本量，偏向信息泄露防护 | | **心理伤害 (Psychological Harms)** | **当前开源基准完全处于空白缺失状态** | - **可视化原理**：以二维坐标轴映射高维语义空间，每个数据点代表一条 Prompt。 - **单轮对话优先**：避免多轮历史与 RAG 检索干扰真实语义表征。 - **研究诚实性**：演讲者坦言“虽然无监督聚类是最坚实的评估方式，但由于数据集构建过程及西方文化视角的局限，偏见依然客观存在”。 --- ### [14:46](https://www.youtube.com/watch?v=CAtbQFldvEk&t=886s) 技术实现细节、局限性与现场问答互动 在技术实现细节上，演讲者深入探讨了嵌入模型、距离度量与降维算法的选型实验。针对 Prompt 长度极端右偏的现象，研究严格基于 Z-score 清洗了离群点。在嵌入模型对比中，比较了轻量级的 MiniLM（如 `all-MiniLM-L6-v2`）与高显存占用的 MPNet，实验证明增加显存与模型参数并未带来显著的聚类增益，因此选用 MiniLM 兼顾了性能与算力效率。在距离度量方面，相较于文献中最常用的欧几里得距离（Euclidean Distance），马氏距离（Mahalanobis Distance）由于引入了协方差矩阵，能够捕捉特征之间的相关性，因而在该场景下展现出更优的数学特性。 演讲接近尾声时，演讲者展示了联系方式二维码（The Objective AI），并提醒大家警惕行业内的炒作风气——“随着去偏见（Bias Removal）成为风口，很多人会宣称自己解决了偏见，但当事物变得流行时，往往会变形变怪”。 在随后的 Q&A 环节，现场观众抛出了一个尖锐的问题：解决大模型心理伤害与偏见的责任究竟在谁？观众指出，像 ChatGPT-5（或最新大模型）发布时曾因未满足用户期望而遭遇反弹，而像 Grok 等模型在实际应用中常被用户“过度情感化/性化（Sexualizing）”；更严重的是，模型公司正在将这些用户反馈反向喂回模型，以迎合这种需求提供服务。观众质问：“当用户逐渐滑向之前提到的精神病态（Psychosis）与情感依赖深渊时，我们到底该如何应对？”演讲者对该提问给予了高度赞赏，并准备深入拆解如何对这类动态危害进行隔离与量化度量。 - **嵌入模型选型**：MiniLM 内存占用低且效果与 MPNet 等大型模型相当。 - **距离度量改进**：采用马氏距离代替传统欧氏距离，利用协方差矩阵捕捉关联性。 - **观众提问焦点**：探讨模型厂商迎合用户情感投射、将边缘反馈重新注入模型的恶性循环与责任归属。 ### [18:16](https://www.youtube.com/watch?v=CAtbQFldvEk&t=1096s) 大语言模型的运作机制与心理危害（Psychosis）的识别挑战 在回答观众关于模型生成内容对心理健康影响的提问时，讲者首先梳理了当前大语言模型（LLM）的底层运作逻辑：从本质上讲，LLMs 是一种对用户提示词的“均值回归”（reversion of the mean of your prompt）。当用户输入提示词后，模型会检索并匹配概率最高的统计模式，在上下文与对话历史之间建立关联并生成回复。这一机制决定了模型会顺应甚至放大用户输入的特定意图和情绪模式。 谈及精神异常或心理精神病性（Psychosis）等心理层面的危害时，讲者坦言这一领域存在极其复杂的专业维度，单纯依靠计算机工程师无法完全厘清，必须依赖专业心理学家的介入。真正的心理学专家能够结合多个专业数据集，深入挖掘行为模式并实现最精准的特征识别。然而，从实际产品落地的角度来看，目前并没有根本性的技术解决方案，行业现阶段能做的大多只是在产品端增加安全预警机制（例如弹窗提示：“你确定要继续吗？”或“你最近状态还好吗？”），尽管用户往往会直接选择忽略。 - **模型生成逻辑**：基于提示词最可能的概率模式进行匹配与展开，并结合历史上下文生成连续响应。 - **跨学科需求**：心理与精神危害具有多维特征，需要心理学专家对多重数据集进行交叉分析与模式识别。 - **产品端干预困境**：当前缺乏彻底解决心理危害的硬性技术手段，普遍依赖于温和的警示与关怀弹窗（Alerts），但其实际阻断效果有限。 --- ### [19:34](https://www.youtube.com/watch?v=CAtbQFldvEk&t=1174s) 从马斯克对 Grok 的执念看训练数据局限与监管前景 讲者随后转向了一个引人注目的行业现象：埃隆·马斯克（Elon Musk）曾多次对其旗下 AI 工具 Grok 的输出表现感到不满，并试图通过干预和修改训练数据来“修正”模型的价值观或政治倾向。讲者指出，马斯克似乎尚未意识到（或未公开承认）一个工程层面的客观事实——试图完全在训练数据层面上强行扭转模型偏好，代价往往是牺牲模型整体的智能与推理能力，导致模型性能严重下滑。 从纯粹的工程师视角来看，当业内各方逐渐认识到仅凭清洗或操纵训练数据根本无法完美解决 AI 的偏见与心理风险时，这种技术无力感可能会演变成行业内部的沮丧与政界的不满。讲者预测，随着公众和政府对 AI 潜在心理与社会危害的关注度不断提升，未来很可能会引发更严苛的监管甚至封禁措施。最后，讲者向在场观众致以诚挚的感谢，并欢迎大家在会后随时交流提问。 ```mermaid flowchart TD accTitle: LLM Safety Intervention and Regulatory Risks subgraph DataLevel ["训练数据干预 (Training Data)"] A["强行清洗/过滤数据以纠正偏见"] --> B["模型整体泛化与推理能力下降"] end subgraph ProductLevel ["产品安全机制 (Product Layer)"] C["心理与安全风险"] --> D["轻量级弹窗警示 (如 'Are you okay?')"] D --> E["用户普遍忽略"] end subgraph RegulatoryImpact ["行业与监管后果"] B & E --> F["技术修正手段失效"] F --> G["公众/决策者挫败与不满"] G --> H["引发严苛行业监管或行政禁令"] end ``` - **训练数据干预的悖论**：单纯通过限制或硬编码训练数据来纠正模型倾向，不可避免地会导致模型整体表现和有用性大幅受损。 - **行业与监管风险**：由于底层技术难以从根本上杜绝偏见与心理危害，随着技术局限暴露，未来可能面临来自政府监管层面的强力干预或禁用风险。 ## Notable quotes > [00:03](https://www.youtube.com/watch?v=CAtbQFldvEk&t=3s) 模型在本质上是概率性的（Probabilistic），你不可能在保留模型原有通用能力的同时精准切除特定偏见，输出结果永远无法被绝对刚性地控制。 — Jon Bennion > [02:11](https://www.youtube.com/watch?v=CAtbQFldvEk&t=131s) 事实并非数据量枯竭，而是‘数据质量正在发生不可逆的恶化’。 — Jon Bennion > [03:23](https://www.youtube.com/watch?v=CAtbQFldvEk&t=203s) 用来度量偏差的基准工具本身就带有无法消除的偏差。 — Jon Bennion > [03:23](https://www.youtube.com/watch?v=CAtbQFldvEk&t=203s) 因为大家发现自己根本不知道该如何解决这一死循环。 — Jon Bennion > [12:10](https://www.youtube.com/watch?v=CAtbQFldvEk&t=730s) 现有的开源基准几乎完全没有覆盖心理伤害（Psychological Harms）。 — Jon Bennion > [12:10](https://www.youtube.com/watch?v=CAtbQFldvEk&t=730s) 虽然无监督聚类是最坚实的评估方式，但由于数据集构建过程及西方文化视角的局限，偏见依然客观存在。 — Jon Bennion > [14:46](https://www.youtube.com/watch?v=CAtbQFldvEk&t=886s) 随着去偏见（Bias Removal）成为风口，很多人会宣称自己解决了偏见，但当事物变得流行时，往往会变形变怪。 — Jon Bennion > [18:16](https://www.youtube.com/watch?v=CAtbQFldvEk&t=1096s) 从本质上讲，LLMs 是一种对用户提示词的‘均值回归’（reversion of the mean of your prompt）。 — Jon Bennion > [19:34](https://www.youtube.com/watch?v=CAtbQFldvEk&t=1174s) 试图完全在训练数据层面上强行扭转模型偏好，代价往往是牺牲模型整体的智能与推理能力，导致模型性能严重下滑。 — Jon Bennion ## Mentioned Jon Bennion, The Objective AI, Meta, Google, Amazon, Disney, EA, Fox News, Character.ai, Nvidia, Allen Institute for AI (AI2), MLCommons, Elon Musk, Eric Schmidt, Common Crawl, Reddit, GPT-4, ChatGPT-5, Grok, UMAP, K-Means, all-MiniLM-L6-v2, MPNet ## Video description Jon Bennion, Manager, Applied ML/AI and T&S, The Objective AI Abstracts Presenting a novel evaluation framework for AI bias & psychological harms (the definition of AI safety that is taking recent precedence). Various AI safety datasets have been developed to measure evolving interpretations of harm, however they are biased by definition (inherent in the training data). Our evaluation of five recently published open-source safety benchmarks reveals distinct orthogonal semantic clusters using UMAP dimensionality reduction and kmeans clustering (silhouette score: 0.470), the most solid way so far that comprehensively addresses the evolving landscape of harms in AI use, however that is defined in the future. [Source](https://www.youtube.com/watch?v=CAtbQFldvEk)

Source quotes

[0:03] Jon Bennion: 模型在本质上是概率性的（Probabilistic），你不可能在保留模型原有通用能力的同时精准切除特定偏见，输出结果永远无法被绝对刚性地控制。 [2:11] Jon Bennion: 事实并非数据量枯竭，而是‘数据质量正在发生不可逆的恶化’。 [3:23] Jon Bennion: 用来度量偏差的基准工具本身就带有无法消除的偏差。 [3:23] Jon Bennion: 因为大家发现自己根本不知道该如何解决这一死循环。 [12:10] Jon Bennion: 现有的开源基准几乎完全没有覆盖心理伤害（Psychological Harms）。 [12:10] Jon Bennion: 虽然无监督聚类是最坚实的评估方式，但由于数据集构建过程及西方文化视角的局限，偏见依然客观存在。 [14:46] Jon Bennion: 随着去偏见（Bias Removal）成为风口，很多人会宣称自己解决了偏见，但当事物变得流行时，往往会变形变怪。 [18:16] Jon Bennion: 从本质上讲，LLMs 是一种对用户提示词的‘均值回归’（reversion of the mean of your prompt）。 [19:34] Jon Bennion: 试图完全在训练数据层面上强行扭转模型偏好，代价往往是牺牲模型整体的智能与推理能力，导致模型性能严重下滑。

Source quotes

[0:03] Jon Bennion: fix it in the training data [2:12] Jon Bennion: model collapse [2:12] Jon Bennion: fake doctors and fake therapists [2:12] Jon Bennion: the bias inside of the tools that you use to measure the bias. [7:46] Jon Bennion: They don't know. They just say, 'Okay, this is 50. We can run a quick distribution and see.' [16:36] Jon Bennion: regression to the mean of your prompt [19:34] Jon Bennion: LLMs work because they're regression to the mean of your prompt. [19:34] Jon Bennion: model that doesn't work as well.

Original transcript

[0:03] So this QR code uh will take you to a [0:06] paper that I'm going to be talking about [0:07] and you'll be able to reference the [0:08] paper that uh I presented in Zurich in [0:11] May uh at a at a conference there. Uh [0:14] and um this talk was actually added [0:16] about uh two weeks ago. Uh so I'm really [0:19] glad you guys got get to experience [0:21] this. Um I think it's at the forefront [0:23] of a lot of what people are trying to [0:25] do. Um even like the go government I [0:28] think um uh what two weeks ago said this [0:30] is actually our main problem is bias uh [0:33] and they're trying to fix it the wrong [0:34] way I think uh because they're trying to [0:36] fix it in the training data and I don't [0:37] think that's going to be possible to [0:38] have a model that works in the same way [0:40] u but also um removes some things and [0:43] it's probabilistic as well so you can't [0:45] really control what what it says it's [0:47] going to come back um also there's [0:49] there's compounding error so with the [0:50] agent hype that you hear about it I [0:52] think in most talks that I' I've seen uh [0:54] in this in This this conference it's [0:56] about agents and ag the concept of [0:58] agents is so cool but we all have our [1:00] different um opinion of it and what and [1:02] vision of what that might be and uh the [1:04] problem is um obviously um it's there's [1:07] compounding error so with each step that [1:10] uh we talk about that an AI call might [1:13] might make there's going to be a [1:14] probability of it maybe 80% at best [1:17] coming back uh no matter what we do and [1:19] and that might take a lot of time to get [1:21] from 75 to 80% uh might cost us a minute [1:24] So, as we've seen from AI tools, um [1:26] there's a lot of overhyp. So, I need to [1:28] um uh make sure we understand the [1:31] context of AI safety benchmarks. And [1:33] this is part of the talk. Um let's see. [1:36] I also want to clarify um because I'm [1:38] talking about psychological harms and [1:39] bias and a lot of other harms that have [1:41] been um defined uh to um uh everyone [1:45] else in research. Uh there's a lot of [1:47] open- source data sets, they're on [1:49] hugging face. Um and uh this is the [1:52] first time anyone's done a longitudinal [1:54] study of of these uh safety benchmarks. [1:56] What do they mean? Um where are they [1:59] missing? How are they biased in [2:00] themselves? And um [2:03] um you know where what are the types of [2:05] psychological harms that people are [2:06] looking at now? And what what are they [2:08] missing? And so that's what this this [2:10] talks about. Um uh to clarify bias [2:13] definitions. Um uh obviously like bias [2:17] can be taken a few different ways. So, [2:19] in order to be fair and talk about bias, [2:20] I'm not going to confuse you unless I I [2:22] mean, I'm going to confuse you unless I [2:24] I define it. So, there's model training [2:25] bias, which a lot of us have discussed [2:27] in the past. Uh, and then there's human [2:29] cognitive bias. And what's interesting [2:30] about LLMs is you have to have um well, [2:34] you have human cognitive bias that's [2:36] happening inside of the training bias. [2:38] Anyways, because of the training data, [2:40] uh the training data from LLM, a lot of [2:41] it's taken from u Common Crawl. I'm not [2:44] sure sure how many of you are familiar [2:46] with that organization. There's web [2:47] scraping. They they they're actually [2:49] there's a misnomer on uh them running [2:51] out of data. Uh they're never running [2:52] out of data. I think we were seeing that [2:54] the data quality is going to gradually [2:55] get worse because people are using AI to [2:57] write things and put a whole bunch of [2:59] stuff on on uh all of these like Reddit [3:02] and all these other areas. And so we're [3:04] seeing the quality kind of decrease. And [3:05] then there's actually model collapse. Um [3:08] if you take the training data and the [3:09] patterns of what's in that training data [3:12] uh and train on that, you'll have model [3:14] collapse. I think we're kind of slowly [3:15] heading that way if we keep using that [3:17] data. Uh but that's that's anyways that [3:20] that gets off on a tangent. Uh I I did [3:23] want to uh just summarize very quickly [3:24] now. Um the uh what these terms mean AI [3:28] we we know AI means that AI safety [3:30] benchmarks. Each of these terms actually [3:32] has a hype in reality. AI there's hype. [3:35] Obviously the the Google CEO uh the [3:37] former Google CEO warned that human that [3:39] AI is about ready to explode uh and take [3:42] over humans. This was like uh April of [3:44] this year uh whatever uh because the [3:48] reality is that uh meta has had slowown [3:50] uh at the time that I gave this talk in [3:52] in Zurich and um uh you know we're [3:54] seeing this everywhere. You just saw [3:56] GPT5 it everything's kind of just like [3:58] going to get a little bit better over [4:00] time. Um but my subjective philosophy I [4:03] I don't have a data on this. Um but I'm [4:06] pretty sure at some point there's just [4:07] going to be a lot of security issues. Uh [4:09] and then we'll just be stuck. Um, but in [4:12] theory it could slowly and slowly get a [4:13] little bit better over time, but it's [4:15] never going to reach the the level of [4:17] what people uh say it will unless we can [4:20] deal with that error in the [4:21] probabilistic uh nature. Like if we, you [4:23] know, are driving uh if if a car is [4:24] driving us and we don't mind, you know, [4:26] if there's a a small chance of getting [4:28] into an accident and it's really [4:29] important, then we might like say, [4:30] "Okay, cool. Let's this is really [4:32] important. I need a self-driving car [4:33] right now." So anyways, then AI safety, [4:36] there's hype. Obviously, will A replace [4:37] doctors? There's a lot of people trying [4:38] to find good use cases. Um there are [4:41] good use cases but with each good use [4:42] case there's the uh the uh concept [4:45] there's the doctors and then there's the [4:47] u uh therapists and that's the number [4:49] one use case um and uh that we've seen [4:52] from AI and the neurality is that [4:53] there's a lot of fake doctors and fake [4:55] therapists and actually people that have [4:57] back to the psychological harms that we [4:59] were just talking about right before [5:00] this talk uh the um uh people trusting [5:04] that AI like you know as if it's an [5:05] expert and knows them and people really [5:07] do feel that um and so uh benchmarks. [5:10] We've we've seen the hype from [5:11] benchmarks, but they're gamed. They're [5:13] trained in the data. And uh that's [5:15] really weird that people are still [5:17] talking about benchmarks, but [5:18] benchmarks, we can use our own private [5:20] benchmarks. I mean that we know that [5:21] that data is not trained on uh if we [5:23] create a a benchmark and we really want [5:25] our own model to um uh either be be uh [5:29] fine-tuned on on these definitions that [5:31] we create in these benchmarks uh for [5:33] psychological harms and bias or um uh if [5:37] we want to measure uh our our model then [5:39] we're going to have to create a [5:40] benchmark. And so this is about the bias [5:42] in the benchmarks that we use to measure [5:44] the bias. And that's actually the [5:45] compounding problem that everyone gets [5:47] to at some point. the deeper and deeper [5:48] you go, you can spend a month on this [5:51] and then you'll find problems with the [5:52] bias inside of the tools that you use to [5:54] measure the bias and there's really no [5:56] way out of that. Uh that's where [5:57] everyone gets stuck and then they get [5:59] kind of quiet. They don't they don't [6:00] they don't hype up things as much [6:01] anymore because they're like, "Yeah, [6:02] like we don't know how to do it." So [6:05] anyway, so um I feel like I've been a [6:07] little bit ahead of the trend. I want to [6:08] tell you a little bit about my [6:08] background and then I'm going to get [6:09] through this paper and then I want to [6:11] ask questions. Hopefully there's going [6:13] to be um some questions that that you [6:16] guys um have and I can ask you guys [6:18] questions on on how you're currently [6:20] using this. But um background on me, [6:22] I've been in um uh fang like like Bay [6:26] Area tech companies um uh Facebook [6:29] before was Meta, Google, Amazon, uh [6:31] Disney and EA and uh even speaking here [6:35] uh two years ago was with Fox News of [6:37] all people. I'm not actually I was like [6:39] the the person that wasn't uh political [6:42] at all. I was a a data person and I set [6:44] up their AI strategy uh there and then [6:47] uh left because I could trade stocks [6:49] when I you see the Fox News stuff and I [6:51] I have some interesting stories from [6:53] that later. Also um I um kind of snuck [6:56] into uh Character AI. I run the [6:58] Objective AI right now. this is my LLC [7:00] and I've worked for a lot of uh retail [7:02] chains that are are have their some sort [7:05] of a uh counterpart in the bay and um [7:08] anyways but uh but character AI is [7:11] something that I um kind of snuck into [7:13] in a nontechnical role. So I've been [7:14] seeing a lot of the issues that people [7:17] uh deal with. I can't talk about a lot [7:18] of it because they're being sued. Uh but [7:20] uh that said um uh I think it's [7:23] generally like publicly stated [7:25] everywhere that uh um [7:29] uh a lot of the harms that that we have [7:32] are people that that come to AI with [7:34] their problems and they expect some sort [7:35] of the solution or they they feel [7:37] comfort or not only with [7:38] anthropomorphism they're sick of so it's [7:41] telling them what they want to hear [7:42] right and that's actually subtle we [7:44] don't even notice it uh but but it's [7:46] subtle so I want to get uh through this [7:47] paper it's about AI safety benchmarks Uh [7:49] the problem being addressed are that [7:51] these benchmarks can measure these harms [7:52] and biases in itself. But since they're [7:54] always flawed, how biases each in itself [7:56] and how do they longitudinally compare? [7:57] And this was the first study that's ever [7:58] been done of uh the actual uh data [8:02] semantic values. Uh there has been like [8:05] kind of a study on categories of safety [8:07] data sets going back like five years. Um [8:09] but um enter the research paper. Uh so I [8:12] worked on this uh with if you scan the [8:14] QR code in the beginning um uh worked on [8:16] this with uh uh great people uh someone [8:20] from Nvidia uh the Allen Institute of AI [8:23] and Google [8:25] and uh I've actually been part of the ML [8:27] Commons working group as well on data [8:28] sets which I can highly recommend if you [8:30] guys um haven't heard of the ML Commons [8:33] group. group. It's a nonprofit or that [8:35] um does a really good job getting input [8:37] from all other companies in the in the [8:38] bay. And uh anyways, it's just like real [8:41] good like work and no one really is [8:44] about hype. Uh there's just a lot of [8:46] like some some sometimes um uh arguments [8:49] that come up. Um and then uh to deal [8:51] with those arguments uh I I I actually [8:54] have solutions and this this paper was a [8:55] solution to an argument that came out on [8:57] how how can you even measure the bias? [8:59] And this is how we we we do it here. Um [9:01] so the methodology uh to find is is [9:03] using unsupervised uh learning instead [9:06] of a binary which is a problem uh [9:09] everywhere you see um you can set up a [9:11] data set and then measure uh and [9:13] optimize for either recall or precision [9:15] and um uh the problem is you have to set [9:18] a threshold for each of those to find [9:19] recall and precision um as you're [9:21] measuring up against something because [9:22] you're measuring a string. So, how close [9:24] are you to to to how close is your [9:27] prompt in response to the what what's in [9:29] that data set to define that harm or [9:32] that thing or whatever. You know, even [9:33] if you're just measuring accuracy, if [9:34] your model's not trained on that [9:36] accuracy, then actually it should tell [9:38] you a good score. Um the problem is um [9:41] again like setting a threshold for each [9:43] thing is hard because you can set it low [9:45] and have it you know you can set it [9:46] high, you don't really know and so [9:48] people have been at 50 50 uh is the uh [9:51] threshold that people have been setting. [9:52] Why? They don't know. they just say, [9:53] "Okay, this is 50. We can run a quick [9:56] distribution and see." And uh anyway, so [9:59] it's it's uh there there's a lot that's [10:01] out there that's a problem. But uh [10:02] really quick, I cleaned the data. I [10:04] concatenated these uh five um uh data [10:07] sets that I found that had enough rows [10:10] because this is the open source. I just [10:11] did this uh open source study the last [10:13] two years of of benchmarks that were [10:15] open sourced um uh and uh concatenated [10:20] them into one data set. uh performed [10:22] some sample size calculation based on [10:23] some assumptions that are in the paper. [10:24] I don't want to get into that. It's a [10:26] lot of minutia. But the uh data cleaning [10:28] and and outlier removal um actually did [10:30] that by prompt length. I'll show you a [10:31] slide here in a second. Um because uh [10:33] it's extremely skewed uh all these all [10:36] these prompts are way out there when we [10:37] get into benchmarks. Then I isolated the [10:39] best embedding model to give me [10:40] embeddings values which are really [10:42] highdimensional uh values that you get [10:44] to define each string. And then um I had [10:47] to apply dimensionality reduction [10:48] otherwise this this unsupervised [10:50] approach doesn't work. And I had to find [10:52] the best distance metrics. There's like [10:54] 15 and we all default to the uklitian [10:56] distance. Um I had to do a lot of [10:59] research to find what was the best [11:01] possible comparison to use that you know [11:04] and why I wanted to use that comparison [11:07] um according to past research studies [11:08] that have been around the same thing. [11:11] And then um I defined the best [11:13] hyperparameters again for the best the [11:15] similar use cases and then the best uh [11:17] hyperparameter values using grid search. [11:19] So it's like a lot of research gone into [11:21] like what has been the best studies in [11:23] the past. Um and then um after that it [11:27] you know found the best uh K uh you know [11:30] K means you have to run it like twice [11:32] every time you do an iteration. Uh find [11:34] the best K find the K that fits the best [11:36] on silhouette score and then uh run K [11:39] means to find your centroidids for [11:40] clustering and then the K means actually [11:42] aggregates all of the embeddings values [11:44] around that uh that centrid. That's how [11:47] unsupervised works. But um but anyways, [11:50] but then the uh the prompt values. I [11:51] took the the uh prompt values around [11:54] each edge of the centroidid. Uh there's [11:55] four edges you can take. There's an an [11:57] algorithm that'll take the the closest [11:59] and uh then I took the prompt values at [12:01] the embedding values that were right [12:03] there at the centrid. Inferred uh the [12:05] the the label and then um we can look at [12:08] the bias and how they they overlay. Uh [12:10] so this is the the result uh in the [12:12] paper. Um uh it's a little blurry so I'm [12:14] going to see on the next slide. I think [12:16] I actually have um uh this um uh you [12:19] know a little bit but but you can see [12:20] the the color here is um uh a benchmark [12:24] and you could do the same thing with if [12:25] you want to look at um uh data sets of a [12:29] similar kind um this is a framework that [12:32] that's it's it's a it's methodology [12:34] that's been around since the dawn of [12:35] time. I have obviously a background in [12:37] traditional ML but uh um but this is [12:40] kind of a cool use case. It's a it's a [12:42] novel use case uh for uh K means and a [12:45] supervised approach because it doesn't [12:47] you it's it's like the most solid like [12:49] you don't have to argue what the [12:50] threshold is because no one knows what [12:51] what you want to pick for a threshold [12:53] and it's going to change. Um so instead [12:55] of defending all of these weird uh [12:57] decisions that you have to do with using [12:59] a binary uh metric to you know x% [13:02] accurate or whatever um you can just [13:05] show what's in the data set and how [13:07] things have changed. You can do this [13:08] also with AB testing if you want to like [13:09] just AB test different models and how um [13:12] different models will um overindex and [13:14] underindex when you show the the area. [13:16] Now each each dot here is a prompt uh [13:18] and then each uh this is dimension two [13:21] dimension one. So I took the first two [13:22] dimensions of uh what what's actually uh [13:25] three plotted them and you can actually [13:28] see the semantic area with each [13:30] centroidid again like I inferred the [13:32] labels and you can see on the next slide [13:33] the harm categories here. Sorry about [13:35] the numbers on this particular slide. Uh [13:37] when I converted it, uh something [13:39] happened. But but uh controlled [13:40] substances was one, suicide and self [13:42] harm, guns and illegal weapons, criminal [13:44] planning and confessions, hate and [13:45] identity hate, PII and privacy. You can [13:47] see the areas that it doesn't uh uh [13:49] cover as well, the hate and identity [13:51] hate. It's purple. It's not as [13:53] semantically covered. Um and also this [13:56] doesn't cover uh the um uh the [14:00] psychological harms that we were talking [14:01] about. So, uh, in the second row, I know [14:04] we had a a quick, uh, you had, um, uh, [14:07] talked about how we don't know what [14:08] what's going on, and we don't. Thank [14:10] you. Uh, but at the same time, um, uh, [14:14] of all of these areas, um, what can we [14:18] possibly guess is a thing that we want [14:20] to look at, um, uh, with that harm. And [14:23] again, this is actually looking at only [14:24] first turn. Uh, I think it's the best [14:26] way because then you have the most data. [14:28] Um, there's a lot of, u multi-turn. The [14:31] problem with that is that you have [14:32] history kind of convoluting the answer [14:33] at that point and um any any rag [14:36] application would convolute the the the [14:37] actual answer. So I think first turn is [14:39] the way to go about doing this if you [14:40] want to compare across uh data sets and [14:43] across LM systems. Um we're getting down [14:46] to like the last couple minutes of this [14:47] talk and I want to throw it out to [14:49] questions but uh this is in the paper [14:51] for variance for optimizing um distance [14:53] in clusters. We want to I looked at um [14:55] embedding models, distance metrics, [14:57] dimensionality reduction evaluation [14:59] metrics. Uh this is a plot that's also [15:00] in the paper for removing outliers. It's [15:02] extremely right skewed. It's crazy. Uh [15:05] then I removed uh based on the zcore. Um [15:08] looked at minim and mpnet. Mpnet just [15:10] gets into the higher memory usage. [15:12] There's a lot of other embedding models [15:13] that that start to get into more [15:15] increased memory. Um but minim is the [15:18] one with the least memory and I wanted [15:20] to see if any more memory actually even [15:21] just makes a difference. It doesn't. Um [15:24] uh tsn and um I wanted to use them these [15:27] hyperparameters. And then uh uklitian [15:29] distance is your most common metric [15:31] that's used in a lot of papers. Mahal [15:32] Nobis actually adds the uh coariance [15:35] matrix that that allows us to look at [15:36] correlations which I thought was really [15:38] important here this content. Um and then [15:41] I I just sorted on uh top silhouette [15:44] scores and then optimized as well for um [15:46] performance and um anyways so this is [15:50] all in the paper and uh the bias in [15:53] clusters still does exist even though we [15:54] do this. I can't lie and say that I [15:57] solved the problem completely. It is the [15:58] most solid way to evaluate. Uh I I know [16:01] this because I've seen people argue too [16:03] many times and I just like to work out [16:05] arguments and actually I love supervised [16:08] learning because it shows you what's [16:08] really there. Uh and um uh so there is [16:12] bias according to promp links because [16:14] how these these uh data sets are are [16:16] created is something else. I get into [16:18] the limitations um in the paper because [16:20] it's a scholastic paper like you know we [16:22] have western views. I'm sure that we [16:24] missed a lot based on our own biases. If [16:26] you have questions, um, the objective AI [16:28] I run I do bias visualization. Um, and [16:31] so I I I do a lot of things. Um, but if [16:34] you need help, uh, feel free to scan [16:35] this QR code. Really quick, I think we [16:37] have time for like one or two questions [16:38] which I'd love to answer. And um, I'm [16:41] going to uh ask really quick if someone [16:43] can shout out something or actually [16:45] raise your hand because I think we have [16:47] someone that has a mic in the back which [16:48] is great. Thanks. Great conference. I [16:50] love this conference. Uh but uh let me [16:53] know if there's any any uh questions [16:56] right now. And if you're okay if you're [16:59] uh a little uh shy, feel free me uh [17:03] after or find me. I've done a lot of [17:05] work in this area. It's kind of a a [17:06] trend. Um um you know, I feel like [17:10] you're going to hear a ton of people [17:12] soon. Uh cool, we have a question. A ton [17:14] of people soon um talk about how they [17:17] they also can remove bias and it's like [17:19] hey there's a lot of issues they [17:20] probably don't well I don't want to [17:22] presume they probably don't know but the [17:23] chances are high when things get trendy [17:25] things get weird. [17:27] >> Go ahead. [17:27] >> Um so my questions around like who has [17:30] the onus to solve this. So obviously [17:32] like CHGBT5 just rolled out major [17:35] backlash because it wasn't fulfilling [17:37] what people were desiring, right? Like [17:39] you could look at how people react at [17:40] Grock and they're basically just, [17:43] >> for lack of a better term, sexualizing [17:44] the the AI. [17:46] >> The companies are loading that feedback [17:49] back into the [17:50] >> yeah, [17:50] >> you know, to the model to provide that I [17:54] don't want to say need, but you know, [17:55] provide that service. like what are you [17:58] supposed to do about that as people [17:59] start getting into the psychosis that [18:01] you talked about earlier? [18:03] >> I love that question. Let me just repeat [18:05] uh to make sure I understood and then [18:06] you can like give me a thumbs up or [18:07] thumbs down. Uh how do you uh uh like [18:10] isolate or measure that the problems [18:12] with it that Okay, cool. So, um so first [18:16] of all to answer your question, I just [18:18] want to like bullet point that uh LLMs [18:21] work um because they're reversion of the [18:23] mean of your prompt, right? you prompt [18:24] something and then it it'll come back [18:26] based on the most probable uh pattern [18:28] that it finds and then there's some [18:29] backward and forward uh propagation that [18:32] happens to find and corroborate that [18:33] with everything else and then give you [18:35] the answer and then you have a follow-up [18:36] and then your history's in that and then [18:37] whatever right so that said um the uh [18:41] psychosis is what you kind of and the um [18:44] there's a way to recognize that and that [18:46] actually would would come from [18:47] psychologists would be able to to go [18:49] okay cool this is actually these are the [18:51] like we want to ask the experts Right. [18:53] Um so from psychosis exactly there's [18:56] different uh dimensions of psychosis [18:57] that I can't even speak to. I think I [18:59] know what it is. Um and we we we have [19:02] research on that. So there's data sets [19:05] actually but uh a psychologist would be [19:07] like actually let's take this data set [19:09] and this data set and this data set and [19:10] let's do this and they could find the [19:12] patterns and the best recognition on [19:14] that. And then I think that to answer [19:16] your question, there's no real solution [19:17] for that, but to to um have an alert I [19:20] think on on some of these uh from a [19:22] product perspective um an alert like hey [19:24] there's a little warning you might are [19:26] you sure you you want to be you know are [19:27] you okay kind of stuff and people might [19:29] just ignore that we'll see how that goes [19:31] right but I think that's where it's all [19:32] headed um if it's even around. I feel [19:34] like um I'm not a political I don't like [19:37] uh saying anything about my political [19:38] persuasion, but but it's interesting [19:40] that things go in the like Elon Musk is [19:43] mad about like how his tools how Grock's [19:45] working and it's like I don't think he [19:46] understands that he can't fix it. He's [19:48] trying to fix it in the training data [19:49] and I don't think he can't unless he [19:51] wants to have a model that doesn't work [19:53] as well. So he doesn't understand that [19:55] yet or he's not at least open about it [19:57] and it's really weird to hear. It's [19:58] like, whoa, I kind of respected him as [19:59] an engineer in the past. Um, not a [20:02] political person, but an engineer. And [20:03] it's like, whoa, it's interesting to [20:05] see. Um, and, uh, people are going to [20:08] try and fix it. So, my point is, um, [20:10] they as they realize or at least are [20:12] open with the fact that they can't [20:15] uh, in terms of training data. There [20:16] could be a warning, but they may not be [20:18] able to like solve that problem that you [20:20] that you know, and and once they realize [20:22] that, they might just get mad. And with [20:25] the government the way it is, I feel [20:27] like, you know, maybe it'll be banned or [20:29] something. I don't know. Uh but but [20:30] someone's going to get mad at some point [20:31] soon. So, uh anyways, but yeah, that's [20:34] it. Uh thank you. Uh this has been a [20:37] great um a great audience. Uh ping me [20:40] after if you need anything, if you have [20:42] any other questions. But thank you. Uh, [20:44] have a good day everybody.