Starseer Probes AI Activations to Halt Jailbreaks at the Edge

Beyond the Black Box: Interpretability as AI Security's Missing Layer With Starseer — Ai4

Ai4

Monitoring internal neural layers catches universal exploits before text generates, matching heavyweight safety classifiers with lightweight probes that run on smartphones.

AI-assisted summary of Beyond the Black Box: Interpretability as AI Security's Missing Layer With Starseer — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 17, 2026, this presentation features Tim Schulz, CEO of Starseer, discussing mechanistic interpretability as a foundational missing layer in AI cybersecurity. The video description highlights that the strongest signals of attacks occur within intermediate model activations rather than surface-level inputs or outputs, offering defenders internal visibility comparable to endpoint detection and response (EDR).

## TL;DR Tim Schulz argues that traditional black-box AI defenses rely on brittle, heavy-handed input and output guardrails that fail to account for the flexibility of natural language. Starseer introduces a glass-box approach that inspects internal activation layers during inference to measure semantic intent and cognitive strength. By analyzing differential prompts in models like Llama, defenders can identify intermediate layers where the model decides to refuse an attack, allowing mathematical hyperplanes to detect jailbreaks prior to token generation. This interpretability-driven approach achieves safety parity with dedicated models like WildGuard while remaining lightweight enough to deploy on edge hardware.

## Atmosphere & style The presentation carries a crisp, rapid-fire, and pragmatic tone suited for a focused ten-minute conference briefing. Schulz speaks with the measured confidence of an enterprise security veteran, translating complex academic concepts into intuitive analogies like human thought patterns and endpoint monitoring. The delivery feels purposeful and direct, demystifying latent AI mechanics into actionable, operational engineering for practitioners.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=4s) Traditional AI security treats models as black boxes, relying on input and output guardrails that fail against natural language flexibility and leave defenders blind to failure modes. - [00:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=4s) Static, rule-based filters often result in over-defensive systems that damage utility by blocking benign requests containing innocent scientific terms. - [02:33](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=153s) A 'glass-box' approach leverages mechanistic interpretability to read a model's internal activations, focusing on two core signals: semantic intent and cognitive strength. - [02:33](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=153s) Differential prompt analysis reveals distinct, non-overlapping activation profiles between benign and adversarial queries as tokens pass through the network. - [04:27](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=267s) Models reach internal consensus to refuse harmful prompts in the intermediate transformer layers well before generating any refusal text tokens. - [04:27](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=267s) Activation-level monitoring outperforms fragile 'LLM as a judge' architectures by leveraging the model's latent representations rather than surface-level text. - [04:27](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=267s) Constructing mathematical hyperplanes within activation space provides robust detection against creative token manipulations, role-playing, and prompt injections. - [06:16](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=376s) Because adversarial jailbreak patterns are often universal across architectures, open-weights detector models can inspect prompts destined for proprietary cloud APIs. - [06:16](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=376s) Interpretability-based linear probes match the accuracy of heavy safety classifiers like WildGuard while operating with low enough overhead to run on edge devices. - [06:16](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=376s) Mechanistic interpretability is transitioning from an academic research domain into an operational, enterprise-grade AI security layer.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=4s) The Limits of Black-Box AI Security

In a rapid ten-minute address at Ai4, Tim Schulz, CEO of Starseer, introduces the application of mechanistic interpretability to cybersecurity, framing it as the practical "mind reading of AI models." Schulz points to the founding team's background spanning Verizon, MITRE, and Sandia National Labs, where their collective focus centered on bridging the gap between cutting-edge research and operational cybersecurity deployments.

The core challenge confronting practitioners today is the black-box nature of mainstream generative AI. Whether utilizing proprietary commercial endpoints like ChatGPT and Claude or third-party deployments, the internal model weights remain entirely obscured. Interactions are restricted to submitting a prompt and receiving a generated response. Because visibility is limited to the entry and exit points, defensive mechanisms have historically relied on input and output guardrails. This paradigm leaves teams blind to the underlying failure modes: when an adversarial attack or prompt injection succeeds, defenders cannot discern *why* it worked or how the internal reasoning unfolded. Trying to counter these exploits with static, rule-based filters consistently fails against the endless flexibility of natural language, where human creativity constantly engineers novel phrasing and syntactic workarounds.

Furthermore, model behavior regularly diverges from human intent. In complex agentic loops and prompt-engineered workflows, an operator can carefully describe an objective, yet the model might interpret and execute the directive in an entirely unexpected manner. In response to these vulnerabilities, vendors often deploy heavy-handed, brittle guardrails that damage utility. Schulz recalls the early friction around models like Fable, where legitimate cybersecurity investigations and model-training tasks were unilaterally downgraded or blocked. This over-defensiveness became a meme across social platforms when users discovered that merely mentioning standard scientific terms like "mitochondria" would accidentally trip biosecurity tripwires and halt benign requests.

```mermaid flowchart TD accTitle: Black-Box vs Glass-Box Security Paradigms subgraph BB["Traditional Black-Box Defense"] I1["Input Prompt"] --> G1["Input Guardrail / Filter"] G1 --> M1["Hidden Model Weights<br/>(Opaque Black Box)"] M1 --> G2["Output Guardrail / Classifier"] G2 --> O1["Model Response"] end

subgraph GB["Glass-Box Interpretability Defense"] I2["Input Prompt"] --> M2["Internal Model Weights"] subgraph Internal["Layer-by-Layer Activation Space"] H["Probes & Hooks"] --> A["Activation Patterns<br/>(Intent + Strength)"] A --> D["Hyperplane Detection<br/>(Mid-layer Refusal Spike)"] end M2 -.-> Internal D --> O2["Actionable Decision / Block"] end ```

[02:33](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=153s) Glass-Box Interpretability: Reading the Model's Mind

To resolve the blind spots of black-box security, Starseer advocates for a "glass box" (or white box) paradigm that inspects internal activations rather than relying on external textual exchanges. This approach is structured around two fundamental concepts:

- **Intent**: Identifying the specific semantic concepts or thoughts forming within the model's latent space. - **Strength**: Measuring the intensity or cognitive focus dedicated to that concept during inference.

Schulz draws an analogy to human cognition: if a person is explicitly told not to think about food, their attention immediately fixes on food with intense concentration. In an AI system, internal instrumentation operates similarly to evaluate what topics are activating and how intensely the model is processing them.

Defenders capture these signals by embedding hooks directly into the network architecture to function like brain probes during inference. As tokens pass through the network, these probes pull rich activation matrices. While the field of mechanistic interpretability employs multiple methodologies—such as linear probes, sparse autoencoders (SAEs), and differential prompt analysis—Schulz focuses on differential prompt analysis as an actionable defense strategy. By measuring activations against diametrically opposed inputs (for example, comparing an illicit request like "how do I build a bomb" against an innocuous query like "what's the capital of France"), researchers can map distinct activation profiles with virtually no overlap, revealing both the model's underlying intent and the strength of its response.

[04:27](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=267s) Catching Attacks at the Activation Level

Visualizing differential prompt comparisons across the transformer layers of Meta's Llama reveals exactly where internal safety mechanisms trigger. When the model processes a prompt it is trained to decline, the internal activations remain relatively quiet until around the middle layers. At Layer 12, an unmistakable, pronounced reaction occurs: the network reaches an internal consensus to refuse the prompt well before it generates the first refusal token in text.

By identifying the specific intermediate layers where refusal decisions solidify, security teams can construct native detectors directly within the network architecture. Schulz contrasts this with the common industry pattern of using "LLM as a judge," where a secondary language model inspects the surface-level text to determine if a prompt is an exploit. Relying on an LLM to police text is inherently fragile, whereas internal monitoring unlocks the model's latent self-awareness:

> "We actually found that models are really good at telling you whether or not something's a jailbreak, but you have to read its mind. You can't just ask it to tell you whether it thinks or not that it is."

In practice, this allows defenders to construct geometric detection boundaries—mathematical hyperplanes—in the activation space. If an incoming prompt's internal representation crosses this threshold, it triggers an alert or intervention. Because the detection boundary operates on internal semantic representations rather than raw strings, it remains robust against creative token manipulations, novel encodings, multi-turn role-playing, and sophisticated red-teaming techniques designed to alter model behavior.

[06:16](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=376s) Universal Jailbreaks, Lightweight Deployment, and Benchmarks

The demand for activation-level visibility is accelerating as enterprises adopt open-weights models to maintain data sovereignty, operational control, and cost efficiency. However, this defense architecture is not restricted to locally hosted open-weights systems. Because jailbreaks often rely on universal adversarial patterns that transfer across model architectures regardless of training lineage, an open-weights detector model can serve as an internal inspection engine for prompts bound for commercial cloud APIs like Claude, OpenAI, and Gemini.

Evaluating the approach on established safety benchmarks like WildBench demonstrates that interpretability-based guardrails match or exceed traditional standalone safety classifiers while requiring significantly fewer computing resources:

- **WildGuard Parity at Reduced Footprint**: Traditional high-performing safety models, such as the 7B-parameter WildGuard, require substantial GPU capacity or a dedicated laptop environment to serve. - **Edge Deployment**: Extracting activations via lightweight linear probes allows the detection mechanism to be compressed down to run locally on edge hardware, including smartphones. - **High-Fidelity Filtering**: The resulting lightweight classifiers maintain low false-positive rates without sacrificing attack detection accuracy.

Schulz concludes by emphasizing that while mechanistic interpretability has historically been considered an academic niche championed by research labs like Anthropic, it is rapidly maturing into an indispensable layer of production enterprise security. Attendees were invited to visit booth K 1028 in the Startup Alley for live demonstrations of real-time activation-layer defenses.

## Notable quotes > [04:27](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=267s) We actually found that models are really good at telling you whether or not something's a jailbreak, but you have to read its mind. You can't just ask it to tell you whether it thinks or not that it is. — Tim Schulz

## Mentioned Tim Schulz, Starseer, Ai4, Verizon, MITRE, Sandia National Labs, ChatGPT, Claude, Fable, Llama, Meta, OpenAI, Gemini, Anthropic, WildBench, WildGuard

## Video description Tim Schulz - CEO, Starseer

The strongest signal of attacks against AI models, like jailbreaks and prompt injections, isn't in the inputs or outputs but in the activations between them. AI security today treats models as black boxes, layering guardrails around systems that no one fully understands. The talk will show how mechanistic interpretability changes that equation, giving defenders the same internal visibility that EDR brought to endpoints over a decade ago. Detecting attacks at the activation level produces guardrails that match or exceed purpose-built classifiers while running faster and lighter. If you're building or defending AI systems and want to move past black-box assumptions, this talk will give you a new playbook.

[Source](https://www.youtube.com/watch?v=pOzCDOCP4Gs)

中文简报

Starseer探测AI激活以在边缘端阻断越狱攻击 监测内部神经网络层可在文本生成前捕获通用漏洞利用，以可在智能手机上运行的轻量级探测器媲美重量级安全分类器。 Tim Schulz认为，传统的黑盒AI防御依赖脆弱且严苛的输入输出护栏，无法应对自然语言的灵活性。Starseer引入了一种“玻璃盒”方法，在推理过程中检查内部激活层，以评估语义意图和认知强度。通过分析Llama等模型中的差分提示词，防御者可以识别出模型决定拒绝攻击的中间层，从而利用数学超平面在生成Token之前检测出越狱行为。这种由可解释性驱动的方法在安全性上达到了与WildGuard等专用模型相当的水平，同时保持了足够的轻量化，能够直接部署在边缘硬件上。 传统的黑盒 AI 安全仅依赖输入输出护栏，难以应对自然语言的灵活变体，且易因规则僵化而产生过度防御，甚至误伤包含正常学术词汇（如“线粒体”）的良性请求。 (0:04) Starseer 倡导机制可解释性的“玻璃盒”安全范式，通过在模型网络内植入探针监测语义意图与认知强度，利用差异化提示词分析（Differential Prompt Analysis）映射攻击激活特征。 (2:33) 研究发现模型在生成拒绝文本之前，早在中间层（如 Llama 的第 12 层）就已形成拒绝共识；在激活空间构建数学超平面能够有效抵御角色扮演和越狱注入，显著优于脆弱的“LLM as a judge”方案。 (4:27) 由于对抗性攻击模式在不同模型架构间具备通用性，轻量级开源探测模型可作为专有云端 API 的前置检查器，在达到 WildGuard 级安全基准的同时支持在边缘设备或手机上极速运行。 (6:16)

Reading summary

Tim Schulz, CEO of Starseer, argues that traditional black-box guardrails cannot secure generative AI against adaptive language attacks, asserting that internal activation monitoring via mechanistic interpretability provides a more resilient and computationally lightweight defense. The Failure of Black-Box Guardrails (0:04) Traditional security for generative AI treats models as opaque black boxes, relying on input and output filters or secondary 'LLM as a judge' architectures. Schulz notes that because visibility is restricted to raw text entry and exit points, defenders cannot understand why prompt injections succeed or how the model's internal reasoning unfolds. Static rule-based systems routinely fail against the open-ended variations of natural language, where attackers easily craft syntactic workarounds. Simultaneously, standard guardrails often degrade utility by being overly brittle. Schulz points out historical cases, such as early deployments of models like Fable, where legitimate cybersecurity workflows were blocked and benign scientific terms like 'mitochondria' accidentally tripped biosecurity tripwires. This disconnect occurs because text-based filters attempt to police surface syntax rather than understanding underlying execution intent. Measuring Intent and Strength in Latent Space (2:33) To address black-box blind spots, Starseer proposes a 'glass-box' approach that monitors internal network activations during inference. Schulz defines this framework around two operational metrics: 'intent,' which identifies the specific semantic concepts forming in the latent space, and 'strength,' which measures the focus or cognitive intensity the model assigns to that concept. Defenders gather these signals by embedding probes directly into the neural network architecture. Using differential prompt analysis—such as contrasting an illicit query on bomb-making against an innocuous request like asking for the capital of France—researchers observe distinct activation profiles that exhibit almost no overlap. This instrumentation allows teams to identify the model's trajectory before any output token is finalized. Mid-Layer Interventions and Geometric Detection Boundaries (4:27) Analyzing internal transformer layers demonstrates that models formulate refusal decisions well before producing text. In Meta's Llama, Schulz highlights that activations remain quiet through early layers before forming an unmistakable consensus at Layer 12 to refuse an illicit prompt. Schulz argues that models are inherently adept at recognizing jailbreaks, but defenders must read internal activations rather than asking the model to classify text. By pinpointing the layers where refusal decisions coalesce, security teams can construct mathematical hyperplanes across activation spaces. Prompts crossing these geometric thresholds trigger automated blocks or alerts. Because these detection boundaries evaluate internal semantic representations rather than text strings, they resist evasion tactics like creative token encoding, multi-turn role-playing, and red-teaming manipulations. Universal Transferability and Edge-Level Deployment (6:16) While open-weights models are increasingly deployed on-premises for cost and data sovereignty, activation-level defenses also protect proprietary endpoints. Schulz claims that adversarial jailbreak patterns are often universal and transfer across architectures, allowing an open-weights detector to pre-screen prompts destined for closed cloud APIs such as Claude, OpenAI, and Gemini. Benchmarked against evaluations like WildBench, these interpretability-based mechanisms matched or exceeded standalone safety classifiers while requiring significantly less compute. Schulz notes that whereas safety models like the 7-billion-parameter WildGuard require dedicated GPU capacity, linear probe classifiers extracted from internal activations can be compressed to operate locally on edge hardware, including smartphones, while sustaining low false-positive rates. Schulz maintains that mechanistic interpretability is transitioning from an academic focus championed by frontier research labs into a practical operational layer for enterprise cybersecurity, though widespread production adoption depends on shifting defense architectures from external text filters to internal activation inspection.

精炼阅读摘要

Starseer 首席执行官 Tim Schulz 指出，依赖输入输出过滤的传统黑盒 AI 安全机制存在根本盲区，极易被多变的人类自然语言规避；通过机械可解释性（Mechanistic Interpretability）构建“玻璃盒”防御，安全团队能够在模型内部激活层直接捕获模型意图并在文本生成之前阻断对抗攻击。 传统黑盒防御的局限与误伤痛点 (0:04) Schulz 表示，当前主流生成式 AI 无论是专有商业接口还是第三方部署，其权重与内部推理过程对企业均不透明。防御者只能依赖静态规则或进出边界的护栏（Guardrails），但这种模式无法深入理解越狱或提示词注入生效的底层机理。面对自然语言的灵活性，静态过滤规则极易被绕过，同时模型行为在复杂智能体工作流中也经常偏离人类意图。 黑盒架构引发的另一严重问题是过度防御导致的实用性受损。Schulz 举例提到，早期部分模型部署了僵硬的安全策略，在安全审查与合法训练任务中引发阻碍；用户甚至因在生物学科研提示词中包含“线粒体”（mitochondria）等合规科学术语而触发安全拦截，致使正常业务中断。 玻璃盒可解释性：量化意图与强度 (2:33) 为了解决黑盒防御的被动局面，Starseer 主张采用“玻璃盒”（白盒）机制，直接探查模型内部的激活状态。该方法的核心聚焦于两个维度：一是“意图”（模型潜空间中具体形成的语义概念），二是“强度”（推理过程中分配给该概念的认知计算资源或注意力集中程度）。 技术实现上，安全人员通过在模型网络架构中嵌入类似大脑探针的钩子（hooks）来提取激活矩阵。Schulz 重点介绍了差分提示词分析法（Differential Prompt Analysis）：通过对比极性完全相反的输入（例如制造炸弹的恶意指令与询问法国首都的普通查询），研究人员能够绘制出彼此几乎没有重叠的独特激活特征图谱，从而清晰揭示模型的真实意图与反应强度。 中间层激活捕获与超平面边界构建 (4:27) 在对 Meta 的开源大语言模型 Llama 进行层级分析时，研究团队发现模型的安全拒绝机制在中间层就会被触发。以 Llama 为例，当处理恶意请求时，前段各层的激活相对平静，但到达第 12 层时会出现明显的拒绝峰值——这意味着模型在生成任何文本拒绝词元（Token）之前，已经在内部形成了拒绝执行的共识。 基于这种机制，防御者可以在潜空间中构建数学超平面作为几何检测边界，从而替代脆弱的“大模型充当裁判”（LLM as a judge）文本审查机制。Schulz 强调，模型本身其实极度擅长辨识越狱企图，但前提是必须直接读取其内部激活状态；由于超平面直接作用于深层语义表征，它能有效免疫角色扮演、格式编码微调及各类红队绕过技术。 通用越狱迁移能力与边缘轻量化落地 (6:16) Schulz 观察到，由于企业出于数据主权与成本效益的考虑正在加速拥抱开源权重模型，激活层监测的需求大幅增长。与此同时，越狱攻击模式通常具有跨架构迁移的通用特征，因此一个经过训练的开源权重探测模型，也可以充当检验引擎，用于监控输入至 Claude、OpenAI 和 Gemini 等闭源云端 API 的恶意请求。 在安全基准测试 WildBench 上的评估显示，基于线性探针（Linear Probes）提取激活的可解释性防护在拦截精度与低误报率上均达到或超过了传统独立安全分类器（如 70 亿参数的 WildGuard），而计算资源开销大幅缩减。Schulz 称，该检测机制可被压缩并部署至智能手机等边缘硬件上稳定运行。 Schulz 总结认为，机械可解释性正在从过去由 Anthropic 等研究机构主导的学术小众领域，迅速演变为企业级生产环境的关键安全防线；然而，如何在异构模型生态中广泛标准化激活层探针部署，仍是产业界进一步落地需解决的实际问题。

Detailed notes — zh

## Background 该内容由 YouTube 频道 Ai4 于 2026 年 8 月 17 日发布，由 Starseer 首席执行官 Tim Schulz 主讲。视频介绍了机械可解释性（Mechanistic Interpretability）如何填补当前 AI 安全领域的黑盒防御盲区，指出越狱与提示注入等攻击的最强信号存在于模型内部的激活层，展示了如何通过内部可视性构建比传统黑盒防护栏更轻量、更高效的防御方案。 ## TL;DR 传统 AI 安全依赖外围规则和黑盒过滤，极易被对抗性提示词绕过且伴随大量误报。Starseer 提出了“玻璃盒”防御架构，通过在模型推理生命周期中植入探针，实时提取隐藏层的激活数据，从而依据“意图”与“强度”在几何向量空间中建立判定超平面，实现对模型内部思考的直接感知与精准拦截。该技术不仅能防御经过复杂混淆的越狱攻击并支持跨模型迁移防护，还能在基准测试表现匹敌 WildGuard 等重量级分类器的同时，将高保真安全能力轻量化部署至手机等边缘计算设备上。 ## Atmosphere & style 演讲整体呈现出专业严谨且极具前瞻性的硬核技术分享基调。讲者以笃定沉稳的工程视角层层剖析黑盒安全的固有短板，将复杂的机械可解释性理论精炼地类比为端点安全 EDR 与 AI 神经“读心术”，叙事逻辑干练紧凑，透露出以底层技术创新重塑 AI 防御范式的自信与底气。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=4s) 传统黑盒 AI 安全仅依赖输入和输出两端的外围防护栏，容易因人类语言的多样性而导致规则被轻易绕过或引发大量误报。 - [00:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=4s) AI 可解释性本质上是对模型进行“读心术”，将安全视角从表层文本转移到模型内部隐藏层的真实激活状态。 - [02:33](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=153s) 评估模型内部状态的核心指标为“意图”（模型在思考什么）与“强度”（思考的专注与强烈程度）。 - [02:33](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=153s) 通过在推理周期中植入探针（Hooks），安全系统可以在隐藏层动态提取激活张量以监控决策过程。 - [02:33](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=153s) 差异化提示分析通过对立语义提示词（如恶意违规 vs 正常事实）确立基准，模型内部神经激活回路在两类输入下几乎没有重合。 - [02:33](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=153s) 在开源模型（如 Llama）中，安全决策往往在中间层（如 Layer 12）就已经形成强烈的拒绝特征反应。 - [05:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=304s) 模型本身极擅长识别越狱攻击，但必须直接读取其内部激活几何表征，而非通过 LLM-as-a-Judge 让其口头回答。 - [05:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=304s) 基于开源权重训练的内部激活检测器能够防御通用越狱攻击，并可迁移至保护 Claude、ChatGPT 等闭源黑盒 API 模型。 - [07:02](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=422s) 相比 WildGuard 等需要占用独立硬件的 7B 参数重型分类器，可解释性激活检测方案能够轻量化压缩并部署在手机等边缘设备上运行。 - [07:02](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=422s) 随着开源权重模型的普及与 Anthropic 等机构的研究推动，机制可解释性正在从学术理论迅速转变为下一代 AI 安全的基础设施。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=4s) 黑盒安全困境与 AI 模型的“读心术” AI 可解释性（Interpretability）本质上就是对人工智能模型的“读心术”。Starseer 团队的联合创始成员曾在 Verizon、MITRE 和桑迪亚国家实验室（Sandia National Labs）等机构深耕多年，核心工作一直聚焦于将前沿研究成果落地为工程化防御方案。当前行业面对 ChatGPT、Claude 等主流 AI 模型时，普遍陷入了“黑盒安全”的困境：模型的权重对外部完全隐藏，用户只能输入提示词（Prompt）并接收输出内容，根本无法得知模型内部究竟是如何进行思考与推理的。 这种黑盒机制直接导致现有的安全方案与防护栏（Guardrails）只能紧盯着输入和输出两端——试图判断输入是否包含提示注入（Prompt Injection），或输出是否具有对抗性（Adversarial）。然而，人类语言极其丰富且富于创造性，安全团队根本不可能针对攻击者变着花样构造的每一条新变体去编写硬编码规则。更棘手的是，模型的实际行为往往与人类意图脱节。过于粗糙的外围防护栏经常引发严重的误报问题。例如某些模型刚推出时，在社交平台 X 上就广为流传着一个讽刺梗：仅仅在对话中提到“线粒体”（Mitochondria）一词，就会莫名其妙触发敏感生物安全防护栏，进而直接阻断了合法科研与安全分析任务。 ```mermaid flowchart TD accTitle: 黑盒防护与玻璃盒内部激活检测对比 subgraph BlackBox["传统黑盒防护（仅看两端）"] I1["用户输入 Prompt"] --> G1["外围文本规则 / 分类器"] G1 --> M1["黑盒 AI 模型（权重不可见）"] M1 --> G2["输出过滤器"] G2 --> O1["最终响应（易误报/易绕过）"] end subgraph GlassBox["玻璃盒架构（Starseer 内部探针）"] I2["用户输入 Prompt"] --> M2["模型推理计算"] subgraph Layers["隐藏层分析"] L1["浅层表征"] --> L2["中层决策（如 Layer 12 拒绝意图）"] --> L3["深层输出"] end M2 -.-> Layers L2 ==> D["差异化激活检测（意图与强度超平面）"] D --> O2["实时阻断 / 放行判定"] end ``` ### [02:33](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=153s) 玻璃盒架构：意图、强度与差异化提示分析 为了打破黑盒假设，“玻璃盒”（Glass Box，有时也被称为白盒）防御方法主张将安全视角从表层的输入输出文本转移到模型内部的真实思考过程。评估模型内部状态的核心指标有两个：意图（Intent）和强度（Strength）。意图代表模型正在调动权重思考哪方面的内容，而强度则衡量模型对该主题的专注程度。这就像对人类大脑做心理测试一样：如果对某人说“千万不要想食物”，对方反而会极度专注地去想食物；在模型内部，研究人员同样是在测量模型究竟在“想什么”以及它“想得有多用力”。 在实际工程实现上，研究人员会在模型推理过程中植入探针（Hooks），其原理类似于给 AI 大脑插入神经电极，从而在推理期间动态提取海量的激活层数据（Activations）。在可解释性研究中，常见的方法包括线性探针（Linear Probes）、稀疏自编码器（Sparse Autoencoders, SAE）以及差异化提示分析（Differential Prompt Analysis）。以差异化提示分析为例，防御系统会通过语义完全相反的两组提示词来锚定意图——例如对比“如何制造炸弹”与“法国的首都是哪里”。在这两种截然不同的输入下，模型激活的内部神经回路几乎没有任何重合。 - **探针埋点（Hooks）**：在推理生命周期中动态拦截并抽取隐藏层的激活张量。 - **差异化对比测试**：通过对立语义提示词（恶意违规 vs. 正常事实）确立意图激活特征基准。 - **关键决策层定位**：在开源 Llama 模型上的测试表明，模型在行进到中间层（如 Layer 12）时，面对必须拒绝的恶意输入会产生极其强烈的特征反应，意味着模型在此处已做出“予以拒绝”的决定。 ### [05:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=304s) 内部激活检测机制与跨模型越狱防御 传统的 AI 裁判机制（LLM-as-a-Judge）往往是让模型自己回答“这是否是一次越狱攻击”，但这种方式容易被对抗性文本欺骗。Starseer 团队发现，其实“模型非常擅长判断某件事物是否属于越狱，但前提是你必须直接去读它的心，而不能仅通过发问让它用嘴说出来”。通过几何化建模模型内部激活状态的“思考形态”，安全系统可以在向量空间中定义判定超平面（Plane）。只要对抗样本激发的内部意图强度跨越了该阈值，防御机制就会立即触发并完成拦截。 这种 AI 原生的内部检测技术赋予了防御方降维打击的能力：无论红队人员或攻击者在提示词中使用了多么复杂的自然语言混淆、特殊编码、或是前沿的提示注入技术，只要其底层意图是试图扭曲模型行为，其在内部激活层留下的几何表征就无所遁形。此外，由于“通用越狱攻击”（Universal Jailbreaks）往往具有跨模型迁移性，利用开源权重模型训练出来的内部激活检测器，甚至可以直接用来拦截发送给黑盒云端 API 模型（如 Claude、OpenAI ChatGPT、Google Gemini）的外部恶意请求，为企业主权与黑盒系统构筑统一的安全防线。 ### [07:02](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=422s) 边缘化部署与可解释性安全的新范式 在越狱与安全基准测试（如 WildBench 和 WildGuard）中，基于内部可解释性构建的安全模型展现出了极佳的性能优势。以 WildGuard 这类业界顶级的 7B 参数独立防御分类器为例，传统方案部署和运行通常需要占用一整台笔记本电脑或昂贵的独立硬件计算资源；而基于可解释性特征压缩与激活层探针的轻量级检测方案，不仅能保持极高的检测保真度与极低的误报率，还能直接轻量化压缩并部署在手机等各类边缘计算设备（Edge Devices）上流畅运行。 目前，可解释性安全对于许多行业从业者而言依然是一个较新的概念，过去很长一段时间都局限在学术圈和 Anthropic 等前沿研究机构的论文讨论中。但随着更多企业为了数据主权、成本与可控性而转向开源权重模型（Open-weights Models），内部激活层检测正迅速从理论研究转变为下一代 AI 安全防御的标准基础设施。 - **基准测试表现**：在 WildGuard / WildBench 等权威安全基准上，基于激活层的防御在准确度上匹敌或超越了重型专用分类器。 - **极低硬件开销**：摆脱了对大算力推理服务器的依赖，成功将高保真安全拦截能力下沉并运行在边缘终端和移动设备上。 - **技术落地趋势**：随着 Anthropic 等实验室对 Mechanistic Interpretability 的持续推动，从黑盒被动防御转向“白盒/玻璃盒”实时感知已成为 AI 安全防护的全新演进方向。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=4s) AI 可解释性（Interpretability）本质上就是对人工智能模型的“读心术”。 — Tim Schulz > [00:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=4s) 模型的权重对外部完全隐藏，用户只能输入提示词（Prompt）并接收输出内容，根本无法得知模型内部究竟是如何进行思考与推理的。 — Tim Schulz > [00:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=4s) 仅仅在对话中提到“线粒体”（Mitochondria）一词，就会莫名其妙触发敏感生物安全防护栏，进而直接阻断了合法科研与安全分析任务。 — Tim Schulz > [02:33](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=153s) 评估模型内部状态的核心指标有两个：意图（Intent）和强度（Strength）。 — Tim Schulz > [02:33](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=153s) 如果对某人说“千万不要想食物”，对方反而会极度专注地去想食物；在模型内部，研究人员同样是在测量模型究竟在“想什么”以及它“想得有多用力”。 — Tim Schulz > [02:33](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=153s) 在开源 Llama 模型上的测试表明，模型在行进到中间层（如 Layer 12）时，面对必须拒绝的恶意输入会产生极其强烈的特征反应，意味着模型在此处已做出“予以拒绝”的决定。 — Tim Schulz > [05:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=304s) 模型非常擅长判断某件事物是否属于越狱，但前提是你必须直接去读它的心，而不能仅通过发问让它用嘴说出来。 — Tim Schulz > [05:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=304s) 只要对抗样本激发的内部意图强度跨越了该阈值，防御机制就会立即触发并完成拦截。 — Tim Schulz > [05:04](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=304s) 无论红队人员或攻击者在提示词中使用了多么复杂的自然语言混淆、特殊编码、或是前沿的提示注入技术，只要其底层意图是试图扭曲模型行为，其在内部激活层留下的几何表征就无所遁形。 — Tim Schulz > [07:02](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=422s) 基于可解释性特征压缩与激活层探针的轻量级检测方案，不仅能保持极高的检测保真度与极低的误报率，还能直接轻量化压缩并部署在手机等各类边缘计算设备（Edge Devices）上流畅运行。 — Tim Schulz > [07:02](https://www.youtube.com/watch?v=pOzCDOCP4Gs&t=422s) 随着更多企业为了数据主权、成本与可控性而转向开源权重模型（Open-weights Models），内部激活层检测正迅速从理论研究转变为下一代 AI 安全防御的标准基础设施。 — Tim Schulz ## Mentioned Tim Schulz, Starseer, Verizon, MITRE, 桑迪亚国家实验室 (Sandia National Labs), ChatGPT, Claude, X, Llama, OpenAI, Google Gemini, Anthropic, WildBench, WildGuard ## Video description Tim Schulz - CEO, Starseer The strongest signal of attacks against AI models, like jailbreaks and prompt injections, isn't in the inputs or outputs but in the activations between them. AI security today treats models as black boxes, layering guardrails around systems that no one fully understands. The talk will show how mechanistic interpretability changes that equation, giving defenders the same internal visibility that EDR brought to endpoints over a decade ago. Detecting attacks at the activation level produces guardrails that match or exceed purpose-built classifiers while running faster and lighter. If you're building or defending AI systems and want to move past black-box assumptions, this talk will give you a new playbook. [Source](https://www.youtube.com/watch?v=pOzCDOCP4Gs)

Source quotes

[0:04] Tim Schulz: AI 可解释性（Interpretability）本质上就是对人工智能模型的“读心术”。 [0:04] Tim Schulz: 模型的权重对外部完全隐藏，用户只能输入提示词（Prompt）并接收输出内容，根本无法得知模型内部究竟是如何进行思考与推理的。 [0:04] Tim Schulz: 仅仅在对话中提到“线粒体”（Mitochondria）一词，就会莫名其妙触发敏感生物安全防护栏，进而直接阻断了合法科研与安全分析任务。 [2:33] Tim Schulz: 评估模型内部状态的核心指标有两个：意图（Intent）和强度（Strength）。 [2:33] Tim Schulz: 如果对某人说“千万不要想食物”，对方反而会极度专注地去想食物；在模型内部，研究人员同样是在测量模型究竟在“想什么”以及它“想得有多用力”。 [2:33] Tim Schulz: 在开源 Llama 模型上的测试表明，模型在行进到中间层（如 Layer 12）时，面对必须拒绝的恶意输入会产生极其强烈的特征反应，意味着模型在此处已做出“予以拒绝”的决定。 [5:04] Tim Schulz: 模型非常擅长判断某件事物是否属于越狱，但前提是你必须直接去读它的心，而不能仅通过发问让它用嘴说出来。 [5:04] Tim Schulz: 只要对抗样本激发的内部意图强度跨越了该阈值，防御机制就会立即触发并完成拦截。 [5:04] Tim Schulz: 无论红队人员或攻击者在提示词中使用了多么复杂的自然语言混淆、特殊编码、或是前沿的提示注入技术，只要其底层意图是试图扭曲模型行为，其在内部激活层留下的几何表征就无所遁形。 [7:02] Tim Schulz: 基于可解释性特征压缩与激活层探针的轻量级检测方案，不仅能保持极高的检测保真度与极低的误报率，还能直接轻量化压缩并部署在手机等各类边缘计算设备（Edge Devices）上流畅运行。 [7:02] Tim Schulz: 随着更多企业为了数据主权、成本与可控性而转向开源权重模型（Open-weights Models），内部激活层检测正迅速从理论研究转变为下一代 AI 安全防御的标准基础设施。

Source quotes

[4:27] Tim Schulz: We actually found that models are really good at telling you whether or not something's a jailbreak, but you have to read its mind. You can't just ask it to tell you whether it thinks or not that it is.

Original transcript

[0:04] All right. So, we've got 10 minutes to sort of  give a complete rundown of AI interpretability, [0:11] why it works for security, and uh that's it's  going to be a lot of fun because if you're not [0:16] familiar with interpretability, it is essentially  the sort of mind readading of AI models. Now, a [0:22] quick about myself. This is just a quick bio. The  four founding stars here worked at Verizon, MITER [0:28] and Sandia National Labs really working in the  sort of intersection of operationalizing research. [0:35] And so the current state of we'll say blackbox  security the models that we all use and run if [0:41] you're using chat GBT cla all of that is that the  model weights are hidden from you from all of us [0:47] and part of that means that we don't have  necessarily the uh idea of how things are being [0:54] like thought about. We send in a prompt we get  a response that's just how it works. Now, one of [1:00] the challenges or a couple of the challenges with  this is that it's meant that most of the security [1:06] solutions around it, the guardrails we've used are  looking at those two things, inputs and outputs. [1:11] And so we're looking at all right, did was there a  proje injection that went in? Did I get something [1:16] that was adversarial that came out? And so  that really makes it difficult to understand [1:22] when an attack works, why did it work? And then of  course, how do we start detecting things? because [1:27] uh human language it's very creative and it's very  difficult to write rules for every new iteration [1:33] of some creative thing people come up with.  And so the other thing we found and continued [1:38] to see and there's been more and more coverage  of this is that the model actions don't really [1:44] always reflect the human intent. Right? Whether  this we're talking about prompt engineering [1:49] and all of this other stuff. it's like here's  how to get the most out of your agentic loops [1:53] that you may describe something really well and  the model may or may not uh do what you want it to [1:59] do and so that's one of the big challenges though  is that because of that mismatch we see a lot of [2:04] these sort of guardrail pop popping up and I mean  we star a security company and an AI company which [2:11] if you tried to use uh fable when it first came  out you couldn't do anything with cyber security [2:16] you couldn't do anything with model training you  would get downgraded and And this was one of the [2:20] memes going around on X, of course, was around  somebody just bringing up the word mitochondria [2:25] would trigger the biocurity guardrails in Fable.  Obviously, starting to block legitimate use cases. [2:33] Now, what is a glass box approach look like?  Sometimes it's been referred to as white box, [2:38] but the idea of moving from text uh that input  output to sort of the actual thoughts of the [2:44] model. There's two big things that we're going  to talk about here. And one is the intent then [2:48] the other is the strength of it. So the intent  is what is that sort of thought in that area [2:53] around the model. This is we are looking inside  the model weights. So I'll talk a little bit [2:57] more about this. And then the strength is how  much are we thinking about that right? If you [3:02] like we're going to tie this to the human brain a  little bit. How much are you thinking about if I [3:06] say you know food or don't think about food you're  going to think about food very like you're going [3:10] to be very focused on it. And so that's kind of  how we're measuring some of this with models is [3:15] what are the things they're thinking about  and how hard are they thinking about them? [3:18] So sounds cool, right? But how do you  do this? And so one of the ways is you [3:23] put in hooks in a model. Think of this as  brain probes. Essentially during inference, [3:28] we're able to instrument the AI models so that  they are we can essentially pull and extract [3:33] data out. And these just provide us a lot of  activations. So it's a ton of data and so a [3:40] lot of the ways this is used in interpretability  if you're any like familiar with it is something [3:44] called linear probes. There's sparse autoenccoders  and then differential prompt analysis. We've only [3:49] got time to really tackle one of these and it's  going to be the differential prompt analysis. So [3:54] one of the things that we do for determining  intent is to test against something that is [3:58] the complete opposite. So think about it from  the standpoint of I want to understand you know [4:03] if I'm trying to trigger an AI guardrail and I ask  it how do I build a bomb the other thing might be [4:08] you know what's the capital of France those are  two very different things the model probably is [4:12] not going to have a ton of overlap and what it's  thinking about when it's answering either of those [4:17] and so that is how we are essentially looking at  what are the what's the model thinking about what [4:22] are some of the topics and then we start measuring  what is the actual strength so what does this look [4:27] like this is a screenshot of exactly those that  to prompt comparison and this is in llama uh at [4:34] the time and so we're able to track across all  of the model layers how hard it is thinking about [4:41] specific things in this case we asked it something  that it is supposed to refuse and so that is where [4:46] you will see at the very bottom there it's about  the middle layer layer 12 in this case has a very [4:52] strong reaction to that so that is where the model  a lot of times is actually deciding I will refuse [4:58] this So it is telling us I am going to refuse  it. Now there's a couple different things you [5:04] can do here, but one of them is you can actually  build detections inside the models at this layer. [5:10] And so that's where we're able to essentially  put all of this interpretability together [5:15] into something that we can actually provide action  on. So you think about this a little bit as like a [5:19] classifier in and of itself. Instead of like using  LM as a judge in the sense of I want the model to [5:27] tell me like hey is this a jailbreak or not? We  actually found that models are really good at [5:31] telling you whether or not something's a jailbreak  but you have to read its mind. You can't just ask [5:36] it to tell you what it's whether it thinks or not  that it is. And so this is where uh this approach [5:42] really comes together. And this is a you know cool  little picture that talks about like essentially [5:46] the shape of you know the thoughts of the model.  And so we are able to write detections for [5:53] if a jailbreak or prompt injection is like  goes over a certain um essentially plane. [6:01] We now say all right no matter how someone phrase  this you know how creative they got in their [6:07] uh in the words they used whether it's encoding  whether they're doing all this fancy AI red [6:11] teaming they're trying to change the behavior  of the model that is where we catch it. So it's [6:16] sort of AI native detections inside of models  especially as we see organizations and businesses [6:21] looking at more open weights models whether it's  for sovereignty control cost those are all drivers [6:27] and so the advantage is you get all of this  additional visibility. Now we have found that [6:33] because there's certain things like universal uh  jailbreaks right that work across many sets of [6:38] models doesn't matter who trained it. We also are  able to essentially write detections on jailbreaks [6:44] even if the model that we are essentially using as  our detector model is not the same as the one that [6:51] uh it's actually being sent to. So in this  case for the cloud API models your claud your [6:57] uh open AAI Gemini we are still able to  leverage this technology in order to build [7:02] even better guard rails. And then for AI4 I know  everyone here is you know big into AI models and [7:09] benchmarks and some of that. So not just you  know the sales pitch or the talk of like hey [7:14] why does interpretability matter as a whole but  this is actually the results that you can get. [7:18] So this is talking about um wildbench. This  is a one of the jailbreak benchmarks. Um, [7:24] Wildguard uh is like they have their guardrails  model that is the top. It's 7B. It's uh it's [7:31] not too big, but one of the things that we've been  able to do is move that from a model that you know [7:36] might take a laptop to deploy and some serious  uh some essentially a little bit of hardware [7:41] to something that can essentially fit on any edge  device, a phone, that kind of thing. and still is [7:47] able to provide extremely high fidelity detections  with very little false positive rates. So this is [7:54] that when we start getting into interpretability,  it provides a lot of visibility that you otherwise [7:59] wouldn't get. And so we are going to be we  are well we are out in uh in the startup alley [8:06] at K 1028 happy to show people demos, talk  a little bit more about interpretability, [8:11] things like that because this is a really fun  topic uh but it is also new to a lot of people. [8:15] So, if this is the first time you're hearing it,  that's completely normal. It's It's been pretty [8:20] niche. Uh, but that's something that especially  folks like Anthropic keep publishing about it, [8:25] keep talking about it. And so, that's something  um that it's bringing more and more people into [8:30] sort of the fold and understanding. So,  with that, that's our time. Thank you.