Beyond the Black Box: Interpretability of Llms in Finance With Barclays Investment Bank

Ai4

AI-assisted summary of Beyond the Black Box: Interpretability of Llms in Finance With Barclays Investment Bank — original source. Check the original for full context.

Source published: 2025-08-28

Detailed notes

## Background This video was published on YouTube by Ai4 on August 28, 2025. It features Hariom Tatsat, Vice President of AI Quant at Barclays Investment Bank, discussing how mechanistic interpretability can open the black box of large language models to ensure transparency, regulatory alignment, and precise output control in financial applications.

## TL;DR Hariom Tatsat addresses the critical enterprise challenge where 86% of Generative AI projects remain stuck in the proof-of-concept phase due to a lack of model explainability. He details how Barclays leverages mechanistic interpretability and Sparse Autoencoders (SAEs) to peek inside an LLM's residual stream and resolve feature polysemanticity. The talk illustrates three practical financial use cases: steering models for credit risk sentiment analysis, using internal feature activations as signals for predictive stock trading, and dynamically triggering RAG to prevent hallucinations when domain-specific activation is low. Ultimately, these techniques allow financial institutions to control, explain, and safely deploy LLMs without expensive model retraining.

## Atmosphere & style The presentation carries an authoritative, pragmatic, and intellectually grounded tone tailored for financial technologists and AI practitioners. Tatsat delivers a clear, structured narrative that smoothly connects high-level AI research concepts to concrete, real-world banking challenges. Utilizing intuitive analogies—such as comparing traditional interpretability to tracking a diet while mechanistic interpretability acts as direct brain intervention—the delivery feels insightful, precise, and forward-looking.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=aZbXdcbx93E&t=4s) About 86% of enterprise generative AI projects remain stuck in the proof-of-concept (PoC) phase due to a lack of explainability, which 68% of industry leaders view as critical for production deployment. - [03:24](https://www.youtube.com/watch?v=aZbXdcbx93E&t=204s) Traditional AI interpretability evaluates models externally, whereas mechanistic interpretability looks directly at internal neurons, layers, and residual streams to reverse-engineer model representations. - [03:24](https://www.youtube.com/watch?v=aZbXdcbx93E&t=204s) Mechanistic interpretability enables direct model steering, allowing teams to manipulate internal activations and alter model outputs without fine-tuning or retraining. - [05:11](https://www.youtube.com/watch?v=aZbXdcbx93E&t=311s) Sparse Autoencoders (SAEs) address the issue of polysemanticity by decomposing mixed neuron activations into distinct, human-interpretable feature vectors. - [07:40](https://www.youtube.com/watch?v=aZbXdcbx93E&t=460s) Barclays applied SAE feature steering to credit risk sentiment analysis, making models selectively sensitive to credit terms and improving benchmark classification accuracy. - [10:18](https://www.youtube.com/watch?v=aZbXdcbx93E&t=618s) Internal SAE feature activations generated from financial news can be fed into decision trees to create transparent, explainable stock movement signals. - [12:25](https://www.youtube.com/watch?v=aZbXdcbx93E&t=745s) Monitoring internal financial feature activation levels acts as a guardrail against hallucinations, automatically triggering Retrieval-Augmented Generation (RAG) when activation falls below a threshold. - [14:23](https://www.youtube.com/watch?v=aZbXdcbx93E&t=863s) Mechanistic interpretability provides a viable framework for highly regulated financial institutions to safely move Generative AI applications from PoC to production.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=aZbXdcbx93E&t=4s) Introduction and the Imperative for Interpretability in Finance

Hariom Tatsat, Vice President on an AI team at Barclays in New York, opens the talk by introducing his team's focus on applied research and generative AI. A couple of months prior, they published a paper titled "Beyond the Black Box: Interpretability of LLMs in Finance," which generated significant interest across both buy-side and sell-side institutions. The core objective of their research is to look behind the scenes of large language models to make them practical and transparent for financial applications, leveraging foundational research from organizations like OpenAI and Anthropic.

Tatsat frames the problem by pointing out a major bottleneck in enterprise AI adoption: while there is massive experimentation, an estimated 86% of generative AI projects remain stuck in the proof-of-concept (PoC) phase and fail to transition to production. Surveys indicate that roughly 68% of industry leaders view explainability as critical to building the confidence required for deployment. In financial settings where real capital is at stake, common workarounds like trial-and-error prompt engineering are insufficient and risky.

``` [ Traditional AI Interpretability ] ( Examines Inputs & Outputs Externally) │ ▼ ┌─────────────────────────────────────────────────┐ │ Black Box LLM │ └─────────────────────────────────────────────────┘ ▲ │ [ Mechanistic Interpretability ] ( Peeks Inside Residual Stream & SAEs) ```

- **Speaker:** Hariom Tatsat, VP, AI Quant Team at Barclays Investment Bank. - **Paper Inspiration:** OpenAI's 2017 discovery of a single "sentiment neuron" that retained sentiment classification ability even when isolated from the rest of the network. - **Production Gap:** 86% of enterprise GenAI projects are stuck in PoC due to a lack of explainability. - **Leadership Sentiment:** ~70% (specifically 68%) of leaders cite explainability as essential for deploying AI models.

[03:24](https://www.youtube.com/watch?v=aZbXdcbx93E&t=204s) What is Mechanistic Interpretability?

Tatsat explains mechanistic interpretability (which he colloquially abbreviates as "mech interp") by drawing an analogy to neuroscience. Unlike conventional machine learning interpretability techniques (such as feature relevance or SHAP values) that treat models as black boxes and observe them strictly from the outside, mechanistic interpretability looks directly inside the model's "brain" at the neuron and layer levels.

To illustrate the distinction, Tatsat uses a medical analogy: if a patient suffers from high blood sugar or diabetes, traditional interpretability acts like tracking their external diet to infer patterns. Prompt engineering is like telling the patient "don't eat sugar." Mechanistic interpretability, by contrast, goes directly into the brain to locate the specific neural feature driving the sugar craving and actively turns down its activation. This ability to go beyond observation to direct model manipulation is known as "steering."

- **Definition:** Analyzing and reverse-engineering the internal representations (neurons, layers, and residual streams) of language models. - **Key Advantage:** Enables precise control ("steering") of model outputs without retraining or fine-tuning.

[05:11](https://www.youtube.com/watch?v=aZbXdcbx93E&t=311s) Unraveling Concepts with Sparse Autoencoders (SAEs)

Modern LLMs possess billions or trillions of parameters, yet they must represent an infinite array of real-world concepts. This leads to a phenomenon known as **polysemanticity**, where a single neuron or feature vector fires in response to multiple unrelated concepts (for instance, firing simultaneously for Michael Jordan, astrophysics, and general geography). To isolate individual concepts, researchers apply **Sparse Autoencoders (SAEs)**.

An SAE functions by tapping into a specific layer's residual stream within the LLM. It projects the dense internal representations into a much wider, sparse neural network layer, effectively decomposing mixed representations into interpretable feature directions. For instance, when asking a model "What is the capital of France?", the SAE uncovers separate, decoupled features representing concepts like "Eiffel Tower," "wines," and "Paris." To make these mathematical features human-readable, researchers pass sample text corpora through the model and assign labels based on which semantic patterns trigger maximum feature activation.

- **Polysemanticity:** The challenge where individual neurons respond to multiple, distinct concepts. - **Residual Stream:** The internal vector space through which information flows between LLM layers. - **Sparse Autoencoder (SAE):** A wider neural network applied to an internal LLM layer to isolate orthogonal, monosemantic features. - **Feature Labeling (Neuronpedia):** Automated attribution tools (such as DeepMind's Neuronpedia) run sentences through features to assign human-readable labels based on maximum activation.

[07:40](https://www.youtube.com/watch?v=aZbXdcbx93E&t=460s) Use Case 1: Credit Risk Sentiment Analysis & Model Steering

Turning to practical applications, Tatsat outlines how Barclays applied mechanistic interpretability to credit risk sentiment analysis. Using DeepMind's open-source Neuronpedia platform, the team identified feature vectors that selectively activate on credit risk concepts. By artificially boosting the activation of these credit risk features (a process called "steering"), they made the LLM "obsessive" about credit risk indicators without altering its weights or modifying the base prompt.

When evaluated on credit sentiment scoring tasks (scaling sentiment from -10 to 10), the steered model naturally prioritized credit-relevant keywords, altering its score and providing clearer domain-specific rationale. When tested across thousands of human-annotated financial sentences, the credit-steered model consistently outperformed the non-steered base model across five distinct sentiment categories.

- **Tooling:** Leveraged Google DeepMind's *Neuronpedia* to locate pre-trained SAE features related to credit risk. - **Mechanism:** Steering directly modifies internal activations to increase sensitivity to specific domains. - **Result:** Improved benchmark accuracy on credit risk sentiment classification over standard, non-steered baseline LLMs across thousands of test sentences.

[10:18](https://www.youtube.com/watch?v=aZbXdcbx93E&t=618s) Use Case 2: Extracting Financial Signals for Trading

In trading environments, black-box recommendations are unacceptable—traders require transparent rationale before acting on model output. The team tested whether internal SAE activations could serve as explicit, explainable signals for predictive trading models. They processed financial news headlines through SAEs to monitor 50 to 100 domain-specific features (such as Named Entity Recognition tags and financial term detectors).

The team then fed these feature activation levels into a decision-tree model to predict stock price movements (up versus down). The decision tree highlighted highly intuitive and interpretable features as the strongest predictors: 1. **Locations and Geographical Entities:** Named Entity Recognition (NER) for places proved surprisingly powerful in signal generation. 2. **Financial Terminology:** Specific vocabulary related to corporate actions and market metrics. 3. **Stock Ticker Symbols:** Explicit company identifiers.

Tatsat notes that ongoing work includes scaling up dataset sizes, incorporating more feature dimensions, and applying SAEs to larger base LLMs.

- **Methodology:** Passed financial news through SAEs to capture ~50–100 active features, using their activation states as inputs for a decision-tree classifier. - **Primary Feature Predictors:** Locations/NER, financial terms, and stock tickers.

[12:25](https://www.youtube.com/watch?v=aZbXdcbx93E&t=745s) Use Case 3: Conditional Activation for Hallucination Reduction

The final application addresses hallucination detection and mitigation in financial chatbots. The underlying principle is simple: when a user asks a finance-specific question (e.g., regarding earnings reports or regulatory filings), finance-related features within the LLM should naturally activate. If a query yields abnormally low activation across relevant domain features, the system detects a potential domain disconnect or lack of knowledge.

Barclays developed a "first line of defense" dynamic guardrail: if the proportion of active financial features falls below a configurable threshold (e.g., 20%), the system automatically flags the response and triggers Retrieval-Augmented Generation (RAG) to inject verified external document context. Comparing non-augmented outputs with conditionally activated RAG outputs showed marked improvements in context, citations, and factual accuracy.

- **Detection Threshold:** Monitored average feature activation on domain prompts; low activation indicates insufficient internal knowledge. - **Remediation:** Dynamically triggers RAG retrieval when internal domain activations fall below set thresholds. - **Outcome:** Significantly reduced hallucinations on complex tasks like evaluating earnings reports and regulatory filings.

[14:23](https://www.youtube.com/watch?v=aZbXdcbx93E&t=863s) Concluding Thoughts on Transparency in Financial AI

Closing the presentation, Tatsat re-emphasizes that in highly regulated sectors like finance—where substantial capital is on the line—explainability is non-negotiable. While frontier model providers routinely launch larger foundation models (referencing releases like GPT-5 or Anthropic's Claude 3.5/4 series), interpretability research often receives less attention than raw scale.

Referencing statements by Anthropic CEO Dario Amodei on the importance of AI safety and interpretability, Tatsat concludes that Sparse Autoencoders and mechanistic techniques provide a viable path toward transparent, trustworthy AI in financial services.

- **Core Takeaway:** Mechanistic interpretability using SAEs enables granular inspection, domain-specific steering, and robust guardrails without expensive retraining. - **Industry Alignment:** Aligns with safety-first research paradigms to bring Generative AI from experimental PoC to production-grade compliance.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=aZbXdcbx93E&t=4s) Beyond the Black Box: Interpretability of LLMs in Finance — Hariom Tatsat

> [03:24](https://www.youtube.com/watch?v=aZbXdcbx93E&t=204s) mech interp — Hariom Tatsat

> [03:24](https://www.youtube.com/watch?v=aZbXdcbx93E&t=204s) don't eat sugar — Hariom Tatsat

> [03:24](https://www.youtube.com/watch?v=aZbXdcbx93E&t=204s) steering — Hariom Tatsat

> [05:11](https://www.youtube.com/watch?v=aZbXdcbx93E&t=311s) polysemanticity — Hariom Tatsat

> [05:11](https://www.youtube.com/watch?v=aZbXdcbx93E&t=311s) Eiffel Tower — Hariom Tatsat

> [05:11](https://www.youtube.com/watch?v=aZbXdcbx93E&t=311s) wines — Hariom Tatsat

> [05:11](https://www.youtube.com/watch?v=aZbXdcbx93E&t=311s) Paris — Hariom Tatsat

> [07:40](https://www.youtube.com/watch?v=aZbXdcbx93E&t=460s) obsessive — Hariom Tatsat

> [12:25](https://www.youtube.com/watch?v=aZbXdcbx93E&t=745s) first line of defense — Hariom Tatsat

## Mentioned Hariom Tatsat, Barclays Investment Bank, OpenAI, Anthropic, DeepMind, Neuronpedia, GPT-5, Claude 3.5, Claude 4, Dario Amodei, Sparse Autoencoders (SAEs)

## Video description Hariom Tatsat, Vice President, AI Quant, Barclays Investment Bank

Abstracts

Can we open the black box of large language models and make AI in finance truly transparent?

As financial institutions adopt LLMs for tasks like trading, compliance, and advisory, the need for transparency is more critical than ever. Understanding how these models internally reason about financial topics is key to ensuring trust and regulatory alignment.

This session covers how mechanistic interpretability can reveal internal patterns in LLMs related to financial reasoning—such as risk and credit events—and how modifying these patterns can steer model outputs without retraining. Practical examples will illustrate how this approach enhances control and explainability.

Join the session to explore cutting-edge interpretability techniques that bring clarity to Generative AI behavior in finance.

[Source](https://www.youtube.com/watch?v=aZbXdcbx93E)

中文简报

超越黑盒：巴克莱投资银行探索金融大模型可解释性 Hariom Tatsat 探讨了企业面临的关键挑战：由于缺乏模型可解释性，86% 的生成式 AI 项目仍停留在概念验证阶段。他详细阐述了巴克莱如何利用机制可解释性与稀疏自编码器（SAE）深入大语言模型的残差流，从而化解特征的多义性。演讲列举了三个金融实战用例：引导模型进行信用风险情绪分析、将内部特征激活作为股票预测交易的信号，以及在领域特定激活度较低时动态触发 RAG 以防止幻觉产生。最终，这些技术使金融机构能够在无需昂贵模型重新训练的前提下，有效控制、解释并安全部署大语言模型。巴克莱银行指出，由于缺乏可解释性，约86%的企业生成式AI项目停留在概念验证（PoC）阶段，而68%的行业领导者视可解释性为投产的关键。 (0:04) 机制可解释性（Mechanistic Interpretability）深入模型神经层与残差流，借助稀疏自编码器（SAE）化解多义性表征，实现无需重新训练即可调控输出的模型引导（Steering）。 (3:24) 巴克莱将SAE特征引导应用于信用风险情感分类并提升了基准准确率，同时将财经新闻激活特征输入决策树，构建出透明可解释的交易信号。 (7:40) 系统通过监控金融特征激活水平构建动态防幻觉护栏，一旦领域激活度低于设定阈值，便会自动触发RAG以注入可靠上下文并降低差错率。 (12:25)

Reading summary

Barclays AI researcher Hariom Tatsat argues that mechanistic interpretability and Sparse Autoencoders offer a viable path to move financial generative AI beyond the proof-of-concept stage by making internal model representations transparent, steerable, and auditable. The Explainability Bottleneck in Enterprise AI (0:04) Enterprise deployment of generative AI faces a significant implementation gap. Tatsat points out that an estimated 86% of enterprise generative AI projects remain stalled in the proof-of-concept phase, with approximately 68% of industry leaders citing explainability as a prerequisite for deployment. In capital-sensitive financial environments, standard approaches like heuristic prompt engineering carry substantial operational risks because they treat underlying models as opaque black boxes. Mechanistic Interpretability and Sparse Autoencoders (3:24) Traditional machine learning explainability relies on external input-output observations like SHAP values. In contrast, mechanistic interpretability analyzes internal layers, neurons, and the residual stream—akin to observing neural activity directly rather than inferring habits from external behavior. A fundamental hurdle in large language models is polysemanticity, where a single neuron activates for multiple unrelated concepts, such as sports figures, astrophysics, and geography simultaneously. To resolve this overlap, researchers attach Sparse Autoencoders (SAEs) to the LLM's residual stream. The SAE projects dense internal states into an expanded, sparse layer that decomposes tangled vectors into isolated, monosemantic feature directions. Tools such as DeepMind's Neuronpedia then map sample text against peak activations to assign human-interpretable labels to individual feature vectors. Internal Steering for Sentiment and Trading Signals (7:40) Barclays applied mechanistic interpretability to credit risk sentiment analysis without modifying base model weights or re-engineering prompts. By identifying credit risk feature vectors via Neuronpedia and artificially boosting their activation levels—a technique termed steering—the model prioritized credit-relevant factors. Across benchmark evaluations encompassing thousands of human-annotated financial sentences, the credit-steered model consistently exceeded non-steered baselines across five sentiment categories on a scale from -10 to 10. For predictive trading applications where black-box outputs are impractical, the team extracted activations from 50 to 100 domain-specific SAE features after passing financial news headlines through the model. These activation states served as direct inputs to a decision-tree model forecasting stock price movements. The decision tree isolated three primary predictive signals: geographic entities and named entity recognition tags, specialized financial terminology, and stock ticker symbols. Conditional Activation to Mitigate Hallucinations (12:25) Tatsat's team developed an internal monitoring guardrail to detect hallucinations in financial chatbots evaluating earnings reports and regulatory filings. The methodology presumes that domain-specific prompts should consistently trigger corresponding financial feature activations within the network. A low activation level across these vectors signals that the model lacks sufficient parametric knowledge to answer reliably. The system enforces a configurable activation threshold. If active financial features register below a predefined benchmark (such as 20%), the query is automatically flagged, triggering Retrieval-Augmented Generation (RAG) to source verified external documentation. Comparative testing confirmed that this conditional intervention improved factual precision, citation fidelity, and context alignment over unaugmented model outputs. Tatsat concludes that while frontier model providers emphasize raw parameter scaling, mechanistic interpretability via SAEs provides the granular inspection and dynamic control required for regulatory compliance and production deployment in finance. Barclays plans to scale this framework across larger base LLMs, broader datasets, and expanded feature dimensions.

精炼阅读摘要

Barclays 投资银行 AI 量化团队副总裁 Hariom Tatsat 指出，缺乏可解释性导致约 86% 的企业级生成式 AI 项目停滞在概念验证阶段。该团队提出借助稀疏自编码器等机制可解释性技术深入大模型内部，通过神经元级控制实现模型引导、信号提取与幻觉抑制，从而为金融高风险场景提供透明可信的应用路径。企业落地瓶颈与机制可解释性的引入 (3:24) Tatsat 指出，当前约 86% 的企业生成式 AI 项目因无法建立生产级信任而停留在概念验证（PoC）阶段，同时约 68% 的行业高管将可解释性视为部署 AI 的关键先决条件。在涉及真实资金的金融环境中，仅依靠反复试错的提示词工程（Prompt Engineering）伴随着极高风险。传统的机器学习解释方法（例如特征相关性分析或 SHAP 值）通常将模型视作黑盒，仅在外部观察输入与输出的关系，无法触及模型内部复杂的推理逻辑。针对这一痛点，Barclays 团队借鉴了 OpenAI 和 Anthropic 的基础研究，引入了机制可解释性（Mechanistic Interpretability）。Tatsat 将其比作神经科学层面的直接干预：传统方法如同通过记录饮食习惯来推断糖尿病成因，提示词工程好比告诫患者不要摄入糖分，而机制可解释性则是直接进入大脑定位引发糖分渴望的具体神经特征，并主动调低其激活水平。这种跳过重训或微调直接操控内部表征的技术被称为“模型引导”（Steering）。稀疏自编码器解耦神经元多义性 (5:11) 现代大语言模型拥有数十亿乃至万亿级参数，但仍需表达近乎无限的现实概念，这导致了严重的“多义性”（Polysemanticity）问题，即单个神经元或特征向量会同时对多个毫无关联的概念（例如迈克尔·乔丹、天体物理学和普通地理名词）产生激活反应。为了在数学上解耦这些混合表征，研究人员应用了稀疏自编码器（Sparse Autoencoders, SAE）。 SAE 的运作机制是接入大模型特定层级的残差流（Residual Stream），将密集的内部表征投影至更高维度且稀疏的神经网络层，从而分离出相互正交的单一语义特征。例如，在回答“法国的首都是什么”时，SAE 可以分离出代表埃菲尔铁塔、葡萄酒和巴黎的独立特征。为了使这些特征具备人类可读性，团队结合 DeepMind 的开源工具 Neuronpedia，将样本语料输入模型并根据触发最大激活程度的语义模式为特征自动打上标签。信用风险情绪分析与模型内部引导 (7:40) 在实际落地层面，Barclays 将机制可解释性应用于信用风险情绪分析。研究团队通过 Neuronpedia 定位出专门对信用风险概念产生响应的特征向量。在不修改基础模型权重、也不改动提示词的前提下，团队通过人为提高这些信用风险特征的激活程度，使语言模型在分析文本时对信用风险指标表现出极高的敏感性与关注度。在将情绪评分量化为 -10 至 10 的评测任务中，经过内部引导的模型展现出了自发聚焦信用风险关键词的能力，给出了更为清晰的专业推理依据。在涉及数千条经人工标注的金融句子基准测试中，该引导模型在五个不同的情绪类别上均持续优于未进行内部引导的基准大模型，证明了特征级干预在特定领域的有效性。交易信号提取与动态防幻觉护栏构建 (10:18) 在量化交易场景中，黑盒交易决策难以被交易员采纳。Barclays 团队将金融新闻标题输入 SAE，监测 50 到 100 个领域特定特征（包括命名实体识别与金融术语检测器）的激活状态，并将这些激活数据输入决策树模型以预测股价涨跌。决策树分析显示，地理位置实体、企业行动及市场指标等专业金融术语、以及股票代码等显式标识符构成了最具预测能力的解释性信号。团队目前正计划将该方法拓展至更庞大的数据集及更大规模的基础模型上。此外，机制可解释性还被用于解决金融对话机器人的幻觉问题。团队观察到，当模型处理财报或监管文件等金融特定问题时，相关金融特征应正常激活；若激活程度异常偏低（例如低于 20% 的预设阈值），系统便会将其识别为领域知识缺失并触发动态防御机制。一旦检测到特征激活不足，系统将自动触发检索增强生成（RAG）流程注入经过验证的外部权威文档，从而在复杂财务分析中显著降低模型幻觉并提升引用准确率。 Tatsat 总结表示，在资本密集且监管严格的金融行业，AI 的可解释性与安全性是不可妥协的底线要求。尽管前沿 AI 机构持续推出参数规模更大的基础模型，但基于稀疏自编码器的机制可解释性研究为金融机构开辟了一条切实可行的技术路径，使机构能够在无需承担高昂微调成本的前提下实现精细化控制与合规落地。

Detailed notes — zh

## Background 本内容由 YouTube 频道 Ai4 于 2025 年 8 月 28 日发布。视频记录了巴克莱投资银行（Barclays Investment Bank）AI Quant 副总裁 Hariom Tatsat 的主题演讲，探讨如何利用机理解释性（Mechanistic Interpretability）打开大语言模型的黑盒，使其在交易、合规与风险管理等严监管金融场景中实现真正透明与受控。 ## TL;DR 巴克莱团队针对 86% 的生成式 AI 项目因不可解释性而困在 PoC 阶段的痛点，借鉴了 OpenAI 与 Anthropic 的机理解释性思路，通过稀疏自编码器（SAE）解耦大模型残差流中的多义性特征。演讲展示了三大落地实战：一是通过特征转向（Feature Steering）在免微调和免复杂 Prompt 的前提下提升信用风险情感分析准确率；二是从财经新闻中提取可解释特征作为量化交易信号并驱动价格预测；三是通过监控金融特征激活阈值实现动态触发 RAG，有效抑制模型幻觉。这套方法为金融级高风险场景下构建合规、可控的透明 AI 提供了切实路径。 ## Atmosphere & style 演讲兼具硬核量化金融的技术严谨度与一线实战者的从容幽默，Hariom 用“脑神经扫描”和“嗜糖冲动”等接地气的医学比喻拆解深奥的前沿架构，极易引发工程与风控人员的共鸣。叙事节奏层层推进，从调侃业内狂加 Prompt 碰运气的无奈乱象，直抵深入模型脑内神经元干预特征的底层突破，展现出对金融真金白银风险的敬畏感。整场分享散发着务实求真的技术先锋气息，在当下盲目追逐模型尺寸狂飙的行业狂热中，冷静而有力地为模型透明度与确定性发声。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=aZbXdcbx93E&t=4s) 巴克莱银行 AI Quant 团队发表论文探索大语言模型在金融领域的可解释性，旨在打破黑盒限制以满足买卖方机构的合规与风控需求。 - [00:04](https://www.youtube.com/watch?v=aZbXdcbx93E&t=4s) 全行业高达 86% 的生成式 AI 项目受困于概念验证（PoC）阶段无法投产，近 70% 的金融高管认为缺乏可解释性是核心阻碍。 - [03:24](https://www.youtube.com/watch?v=aZbXdcbx93E&t=204s) 机理解释性（Mechanistic Interpretability）如同面向大语言模型的神经科学，能够直接侵入模型神经元和网络层级分析内部信号流动，超越了传统黑盒评估与外围提示词工程。 - [05:11](https://www.youtube.com/watch?v=aZbXdcbx93E&t=311s) 大语言模型的多义性问题导致单个神经元混杂承载多种无关概念，稀疏自编码器（SAE）能够将残差流中的纠缠概念有效放大并解耦为单义特征。 - [05:11](https://www.youtube.com/watch?v=aZbXdcbx93E&t=311s) 借助 Google DeepMind 的 Neuronpedia 平台，团队通过文本最大激活值分析为解耦后的神经元特征打上人类可读的语义标签。 - [07:40](https://www.youtube.com/watch?v=aZbXdcbx93E&t=460s) 在信用风险情感分析任务中，通过特征转向（Feature Steering）直接增强特定神经元激活，无需微调或繁杂 Prompt 即可显著提升模型推理准确度并引导其专注关键风险术语。 - [10:19](https://www.youtube.com/watch?v=aZbXdcbx93E&t=619s) 量化交易无法容忍不可解释的黑盒决策，巴克莱通过 SAE 从实时财经新闻中提取 50 至 100 个高激活金融特征，构建透明且具预测力的交易信号。 - [11:02](https://www.youtube.com/watch?v=aZbXdcbx93E&t=662s) 将 LLM 内部潜在特征（如地理实体、股票代码及专业术语）的激活值接入决策树模型，可有效预测资产市场价格走势并符合交易员直觉。 - [12:25](https://www.youtube.com/watch?v=aZbXdcbx93E&t=745s) 通过监控金融特征的条件激活程度建立幻觉防御机制，当金融神经元激活比例低于阈值（如 20%）时自动触发 RAG 检索研报与合规文件以确保回答精准。 - [14:25](https://www.youtube.com/watch?v=aZbXdcbx93E&t=865s) 金融领域对合规与资产安全高度敏感，相比盲目追逐前沿模型规模，深挖稀疏自编码器等底层机理解释性研究才是推动生成式 AI 迈入生产环境的关键基石。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=aZbXdcbx93E&t=4s) 开启大模型“黑盒”：巴克莱银行在金融可解释性上的探索巴克莱银行（Barclays）纽约团队的人工智能副总裁（VP）兼 AI Quant 负责人 Hariom Tatsat 登台开场。他所在团队日常深耕于人工智能与生成式 AI（Generative AI）的各类业务落地与前沿研究。几个月前，团队发表了一篇题为《超越黑盒：金融领域大语言模型的可解释性》（*Beyond the Black Box: Interpretability of LLMs in Finance*）的论文，在买方与卖方机构中引发了极为广泛的关注与讨论。团队之所以决定公开分享这项成果，是因为他们选择深入大语言模型的底层内部，“去探究幕后究竟在发生什么，并将这些底层机制真正应用到金融实际场景中”。这项研究借鉴了 OpenAI 与 Anthropic 最新的前沿探索。Hariom 概述了当天的核心议程：首先剖析金融业务中“可解释性”（Explainability / Interpretability）生死攸关的重要性；接着深入介绍“机理解释性”（Mechanistic Interpretability，他幽默地自嘲这个词实在拗口，平时更习惯简称为“Mechinp”）；随后解析其核心工具——稀疏自编码器（Sparse Autoencoders, SAE）；最后通过三大落地场景进行验证：包括用于信用风险的情感分析、从大模型内部挖掘交易信号的量化交易应用，以及通用层面的幻觉消除（Hallucination Reduction）。 Hariom 带着技术人员特有的调侃展示了一个由大模型生成的 PPT 二维码，随后将技术溯源拉回到了 2017 年。当时 OpenAI 在训练模型执行“下一个词预测”（Next-word prediction）任务时，偶然发现模型内部竟然自发演化出了某个神经元（或特定特征），它能精准预测文本的情感倾向。这意味着即便剥离整个模型、仅保留这几个神经元，它依然能稳定识别情感。OpenAI 官方博客将其命名为“情感神经元”（Sentiment Neuron），而这也正是巴克莱这项金融研究最初的灵感源头与创生起点。时光切回当下，生成式 AI 的落地遭遇了残酷的现实瓶颈：全行业绝大多数 GenAI 项目都深陷在概念验证（PoC）阶段裹足不前，根本无法推向生产环境——这个滞留在 PoC 阶段的比例高达 **86%**。造成这一局面的核心根源正是“无法解释”。模型如同黑盒，团队缺乏投产的确定性与信心。一项面向高管层的行业调查显示，高达 **68%**（近七成）的决策者明确指出“可解释性是迈向生产环境的决定性前提”。Hariom 点出了业内屡见不鲜的无奈现状并引发台下共鸣：大家在面对模型失效时，往往病急乱投医，“如果一个 Prompt 没起效，就疯狂加 Prompt，直到试出看似正确的答案为止”。然而在真金白银面临风险的金融世界里，这种试错式、碰运气的做法绝对不可接受。 - **86%** 的生成式 AI 项目目前被困在 PoC 阶段，无法安全上线投产。 - **68%**（约 70%）的金融机构业务与技术领导者认为模型可解释性是落地的核心阻碍。 - 研发灵感直接继承自 2017 年 OpenAI 发现的“情感神经元”（Sentiment Neuron）机制。 --- ### [03:24](https://www.youtube.com/watch?v=aZbXdcbx93E&t=204s) 什么是机理解释性？给大模型做“神经科学脑部扫描” 面对传统黑盒的困境，Hariom 详细拆解了机理解释性（Mechanistic Interpretability）的本质。简单来说，它就相当于**“面向生成式 AI 的神经科学”**。研究者不再只盯着输入与输出，而是真正侵入大语言模型的“大脑”深处，直接在神经元层级（Neuron level）和网络层级（Layer level）进行微观解剖，查明内部信号究竟是如何流动与计算的。 Hariom 犀利地对比了新旧技术的本质差异。在传统的机器学习与经典 AI 领域，人们虽然也谈可解释性（如特征重要性评分、相关性分析等），但所有这些传统方法都只是“站在门外从外部审视输入与输出”，鲜有手段能真正切入内部运作。机理解释性则是目前所知唯一能直接探入 LLM“脑内结构”的途径。为了让听众直观理解，他打了一个极其形象的医学比方： - **传统可解释性**：就好比诊断一个糖尿病患者，医生只能在体外记录患者每天吃了什么食物、分析进食模式，进而制定外部控糖方案。 - **机理解释性**：则是直接给大脑做开颅级神经探测，深入患者脑部找到具体负责诱发“嗜糖渴望”（Sugar craving）的神经元或特征回路；更关键的是，不仅要定位它，还要在后续步骤中抑制或调节该神经元的激活强度。 - **提示词工程（Prompt Engineering）**：则仅仅相当于站在门外对着患者干喊：“喂，你别吃糖了！”这种外部喊话根本无法触及模型内部的表征机制。机理解释性不仅能够定位内部特征，更为下一步的主动干预奠定了基础——这种无需重新训练模型、直接在内部干预神经元激活的技术被称为“转向”（Steering）。 --- ### [05:11](https://www.youtube.com/watch?v=aZbXdcbx93E&t=311s) 稀疏自编码器（SAE）与多义性解耦进入底层技术核心，Hariom 重点讲解了实现机理解释性的杀手锏——**稀疏自编码器（Sparse Autoencoders, SAE）**。当前尖端大语言模型虽然动辄拥有万亿级参数（Trillions of parameters），但人类现实世界中的概念数量却不可胜数。这就导致了一个长期困扰深度学习的根本难题：**多义性（Polysemanticity）**。在大模型的隐空间中，单颗神经元或一组特定特征往往会被迫同时承载多种截然不同的概念，可能同一个神经元既对“迈克尔·乔丹”产生响应，又同时负责表征“天体物理学”或某些日常事物，所有概念彻底纠缠混杂在一起。如何将这些纠缠的概念独立剥离并解耦？稀疏自编码器正是解开这个结的利器。Hariom 将其比喻为“在大模型的某一特定层凿开一个观察孔，通过训练一个更宽的人工神经网络（Wider ANN）将其内部特征强行放大与解耦”。 ```mermaid flowchart LR accTitle: 稀疏自编码器解耦残差流多义性 accDescr: 展示大语言模型残差流中混杂的概念如何通过稀疏自编码器被放大并分离为单一可解释特征。 subgraph LLM["大语言模型内部"] R["残差流 (Residual Stream)<br/>概念高度混杂 (Cats, Dogs,<br/>Astrophysics, Jordan)"] end subgraph SAE["稀疏自编码器 (SAE)"] W["更宽的自编码网络<br/>(Wider Hidden Layer)"] end subgraph Features["解耦后的单义特征"] F1["概念特征 A: 葡萄酒 / 埃菲尔铁塔"] F2["概念特征 B: 天体物理学"] F3["概念特征 C: 信用风险 / 违约"] end R --> W W --> F1 W --> F2 W --> F3 ``` 在大模型的残差流（Residual Stream）中，猫、狗、天体物理、体育明星等所有信息以矩阵形态高度混杂交织。SAE 挂载到残差流后，通过扩展的高维稀疏层将这些混合信号投影开来。例如当询问“法国的首都是哪里”时，模型输出“巴黎”，而在 SAE 的解耦视窗中，可以清晰观测到底层被独立激活并剥离的细粒度特征——埃菲尔铁塔、法国葡萄酒等。完成了特征解耦后，如何为这些无序的神经元编号赋予人类能读懂的标签（Human-readable labels）？巴克莱团队利用了业界通用的测试方法：将提取出的特定特征运行在大量文本语料上，观察该特征在哪些词汇或语境下出现最大激活值（Maximum activation）。这就像人类脑科学研究一样，“左脑面对科学逻辑激活更高，右脑面对艺术感知激活更强”，通过激发模式最终赋予每个特征直观的语义标签。在这一环节中，团队大量使用了 Google DeepMind 开源的 **Neuronpedia** 平台，利用其预训练并标注好的 SAE 库进行探索。 --- ### [07:40](https://www.youtube.com/watch?v=aZbXdcbx93E&t=460s) 金融落地实战一：信用风险的情绪分析与激活转向（Steering）在完成了理论与工具铺垫后，Hariom 展开了巴克莱的实操落地案例。第一个核心场景是**信用风险（Credit Risk）的情感分析**。传统的做法通常依赖复杂的微调或大量的 Prompt 设计，而巴克莱团队则另辟蹊径：他们借助 DeepMind 的 Neuronpedia 平台，输入“Credit Risk Related”（信用风险相关）关键词，直接在大模型内部挖掘出专门响应信用风险的特定底层神经元与特征集。找到特征后，真正的技术突破在于**“特征转向”（Feature Steering）**。Hariom 解释称，转向的本质就是在推理过程中强行提高特定特征的激活幅度（Increasing activation），“让模型的大脑在信用风险维度上变得极度专注、甚至达到‘强迫症’（Obsessive）般的敏感度”。为了验证其威力，团队输入测试语句并给出了相同的提示词规则：“请在 -10 到 10 的区间内为本句的情感打分”： - **未激活转向（Baseline / Non-steered）**：模型给出的基准评分为 **3 分**。 - **实施信用风险转向（Steered）**：模型评分转变为 **4 分**。 - **推理逻辑（Rationale）对比**：查看模型生成的判定理由时发现，进行了转向的模型自动将全部注意力紧紧锚定在关键的信用风险术语与风险逻辑上。值得注意的是，整个过程**没有添加任何繁复的 Prompt 诱导，也没有执行任何参数微调（Fine-tuning）**，纯粹依靠直接干预模型内部特征激活即可重塑其认知焦点。团队随后将这一实验推广到了成千上万个真实金融文本样本中，与资深专业人员的人工标注（Human-annotated ground truth）进行全量比对。统计结果显示，在横跨五个情感梯度的评估中，经过特征转向的模型在准确率（Accuracy）上全面且显著地超越了未转向的基础模型。 --- ### [10:19](https://www.youtube.com/watch?v=aZbXdcbx93E&t=619s) 金融落地实战二：从黑盒交易到透明信号提取紧接着，Hariom 抛出了第二个极具吸引力但又争议巨大的场景：**二级市场量化交易（Trading）**。当前金融圈许多交易员兴奋地呼吁：“我们想直接用 ChatGPT 和大语言模型来做交易决策！”但现实中存在一个致命逻辑漏洞：假设有交易员跑来向管理层或风控汇报：“我用大模型赚了大钱，但千万别问我背后的赚钱逻辑是什么（Don't ask me why）。”在正规金融机构里，这种无法解释盈利与亏损来源的黑盒交易机制是绝对被禁止且无法容忍的。这也正是深入模型内部探寻明确特征信号的价值所在。巴克莱团队将成熟的 SAE 特征解耦能力复用到量化交易信号挖掘中： 1. 他们将高密度的实时财经新闻资讯连续输入大语言模型中； 2. 经由稀疏自编码器（SAE）实时监测模型深层的神经激活模式； 3. 动态捕捉并筛选出 50 至 100 个具有明确金融业务标签（Human-readable labels）、且对特定市场动态产生高强度激活的神经特征，以此构建出兼具高预测性与完全可解释性的透明交易信号。 ### [11:02](https://www.youtube.com/watch?v=aZbXdcbx93E&t=662s) 特征激活驱动的价格走势预测与模型验证巴克莱团队进一步将大语言模型内部提取出的稀疏特征直接应用于下游金融任务。研究人员提取了与财经新闻及命名实体识别（NER）相关的隐藏特征，并将这些特征的激活值作为输入特征，构建了一个预测资产价格涨跌的决策树模型。通过分析新闻句子与即时快讯在模型中触发的激活模式，决策树成功学习到了特征激活与市场价格波动之间的关联性，其实际预测表现令人印象深刻。更为关键的是，模型筛选出的高权重特征具备极强的直觉可解释性。排名最靠前且高度相关的特征涵盖了地理位置与地点名称（即经典的命名实体识别能力），这在各类金融交易场景与地缘分析中本身就是核心监控指标；紧随其后的是专业金融术语以及股票代码（Ticker Symbols）。这些在可解释性分析中自然涌现的高贡献特征，与量化交易员的基本直觉高度契合。团队目前正计划推进后续研究，包括引入更大规模的语料与数据集、扩增特征维度，并将稀疏自编码器（Sparse Autoencoders, SAEs）迁移应用至参数规模更大的基础大模型之上。 - **核心预测特征类型**：地理地点与位置实体（NER）、标的股票代码、专业金融与财务术语。 - **建模范式**：以 LLM 内部潜在特征的激活程度作为输入变量，接入树形模型预测标的走势。 - **后续演进方向**：在更庞大的金融语料集上扩展特征规模，并在更大参数级的基础模型中训练 SAE。 ```mermaid flowchart TD accTitle: Feature Activation Market Prediction Pipeline accDescr: 金融文本特征激活值输入决策树预测价格涨跌的流程架构 subgraph FeatureExtraction["特征提取层"] A["金融财经新闻与快讯"] --> B["LLM + 稀疏自编码器 (SAE)"] B --> C["实体特征: 地理位置/股票代码"] B --> D["语义特征: 金融术语/财务指标"] end subgraph PredictionEngine["走势预测层"] C --> E["特征激活强度矩阵"] D --> E E --> F["树形分类模型 (Tree Model)"] F --> G{"市场价格走势预测<br/>(上涨 / 下跌)"} end ``` ### [12:25](https://www.youtube.com/watch?v=aZbXdcbx93E&t=745s) 基于特征条件激活的幻觉抑制与动态 RAG 机制针对大模型在金融问答中频发的“幻觉”问题，团队设计了一种基于内在神经元特征激活监控的防御机制。该方案背后的逻辑类似人类大脑的神经反应：正如物理学家在思考物理问题时大脑中负责物理概念的神经元理应被激活一样，当用户向金融聊天机器人抛出财报或监管问题时，模型内部对应的“金融特征神经元”也必须呈现活跃状态。团队通过向 LLM 输入海量金融语料，统计并标记了各特征的激活均值（Mean Activation），并使用自然语言为其生成直观的特征标签，以此明确特征是否与金融语义高度挂钩。在实际推理阶段，系统引入了“条件激活”（Conditional Activation）监控策略。如果用户的提问未能在模型内部充分唤醒对应的金融特征（例如设定激活比例低于 20% 的警诫阈值），系统便会判定当前模型的上下文知识或内在激活不足，从而自动触发检索增强生成（RAG）管道，动态检索补充相关的研报与监管文件。对比测试表明，在未触发条件激活的原生应答中，关于财报与合规文件的回答较为松散；而启用条件激活联动 RAG 后，模型输出的答案在上下文精准度、专业相关度以及原始引证追溯上均有质的提升，该机制被视为抵御模型幻觉的“第一道防线”。 - **脑神经激活映射机制**：通过对比金融提问下的神经元激活分布，诊断模型是否真正进入专业推理状态。 - **特征可解释标注**：为 SAE 提取的特征赋予白话标签与激活阈值基线，便于设定监控规则。 - **动态触发 RAG 补强**：当金融特征激活率低于设定门槛（如 20%）时，自动挂载外部权威财报和监管文书辅助生成。 ### [14:25](https://www.youtube.com/watch?v=aZbXdcbx93E&t=865s) 可解释性在金融前沿的应用价值与未来展望演讲最后着重强调了可解释性对于金融体系的核心必要性。金融行业处于高度监管之下，每一次决策都直接牵涉真金白银的资产安全，因此算法的透明度与可追溯性绝非可有可无的点缀，而是关乎系统生死的底线要求。实践证明，稀疏自编码器（SAEs）是剖析大型语言模型黑盒内部机制、洞察其深层表征结构的最有力工具，为构建安全、合规且值得信赖的金融级 AI 提供了清晰的技术路径。面对当前业内盲目追逐模型尺寸和迭代速度的风气，演讲者引用了 Anthropic 首席执行官 Dario Amodei 的观点指出，尽管业界每周都在高调发布诸如 GPT-5 或新一代 Claude 4 等前沿大模型，但对于决定模型能否落地的机械可解释性研究，获得的公众与资源关注却依然远远不够。特别是在金融这类对确定性要求极为严苛的垂直领域，深挖模型内部表征的透明度研究才是推动生成式 AI 走向生产环境的真正基石，而巴克莱团队当下的探索正是迈向该目标的关键一步。 - **核心行业痛点**：金融场景对资金风险与合规监管的高度敏感，决定了黑盒系统无法直接承担高风险决策。 - **技术突破核心**：稀疏自编码器（SAEs）被确立为解构大模型隐藏表征、构建透明金融 AI 的关键利器。 - **行业趋势反思**：在不断追逐前沿模型参数狂飙的同时，亟需将更多资源与目光倾斜至可解释性底层基础研究。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=aZbXdcbx93E&t=4s) 去探究幕后究竟在发生什么，并将这些底层机制真正应用到金融实际场景中 — Hariom Tatsat > [00:04](https://www.youtube.com/watch?v=aZbXdcbx93E&t=4s) 可解释性是迈向生产环境的决定性前提 — Hariom Tatsat > [00:04](https://www.youtube.com/watch?v=aZbXdcbx93E&t=4s) 如果一个 Prompt 没起效，就疯狂加 Prompt，直到试出看似正确的答案为止 — Hariom Tatsat > [03:24](https://www.youtube.com/watch?v=aZbXdcbx93E&t=204s) 面向生成式 AI 的神经科学 — Hariom Tatsat > [03:24](https://www.youtube.com/watch?v=aZbXdcbx93E&t=204s) 喂，你别吃糖了！ — Hariom Tatsat > [05:11](https://www.youtube.com/watch?v=aZbXdcbx93E&t=311s) 在大模型的某一特定层凿开一个观察孔，通过训练一个更宽的人工神经网络（Wider ANN）将其内部特征强行放大与解耦 — Hariom Tatsat > [07:40](https://www.youtube.com/watch?v=aZbXdcbx93E&t=460s) 让模型的大脑在信用风险维度上变得极度专注、甚至达到‘强迫症’（Obsessive）般的敏感度 — Hariom Tatsat > [10:19](https://www.youtube.com/watch?v=aZbXdcbx93E&t=619s) 我们想直接用 ChatGPT 和大语言模型来做交易决策！ — Hariom Tatsat > [10:19](https://www.youtube.com/watch?v=aZbXdcbx93E&t=619s) 我用大模型赚了大钱，但千万别问我背后的赚钱逻辑是什么（Don't ask me why） — Hariom Tatsat > [12:25](https://www.youtube.com/watch?v=aZbXdcbx93E&t=745s) 第一道防线 — Hariom Tatsat ## Mentioned Hariom Tatsat, Dario Amodei, 巴克莱银行 (Barclays Investment Bank), OpenAI, Anthropic, Google DeepMind, Ai4, Neuronpedia, ChatGPT, GPT-5, Claude 4, 《超越黑盒：金融领域大语言模型的可解释性》（Beyond the Black Box: Interpretability of LLMs in Finance） ## Video description Hariom Tatsat, Vice President, AI Quant, Barclays Investment Bank Abstracts Can we open the black box of large language models and make AI in finance truly transparent? As financial institutions adopt LLMs for tasks like trading, compliance, and advisory, the need for transparency is more critical than ever. Understanding how these models internally reason about financial topics is key to ensuring trust and regulatory alignment. This session covers how mechanistic interpretability can reveal internal patterns in LLMs related to financial reasoning—such as risk and credit events—and how modifying these patterns can steer model outputs without retraining. Practical examples will illustrate how this approach enhances control and explainability. Join the session to explore cutting-edge interpretability techniques that bring clarity to Generative AI behavior in finance. [Source](https://www.youtube.com/watch?v=aZbXdcbx93E)

Source quotes

[0:04] Hariom Tatsat: 去探究幕后究竟在发生什么，并将这些底层机制真正应用到金融实际场景中 [0:04] Hariom Tatsat: 可解释性是迈向生产环境的决定性前提 [0:04] Hariom Tatsat: 如果一个 Prompt 没起效，就疯狂加 Prompt，直到试出看似正确的答案为止 [3:24] Hariom Tatsat: 面向生成式 AI 的神经科学 [3:24] Hariom Tatsat: 喂，你别吃糖了！ [5:11] Hariom Tatsat: 在大模型的某一特定层凿开一个观察孔，通过训练一个更宽的人工神经网络（Wider ANN）将其内部特征强行放大与解耦 [7:40] Hariom Tatsat: 让模型的大脑在信用风险维度上变得极度专注、甚至达到‘强迫症’（Obsessive）般的敏感度 [10:19] Hariom Tatsat: 我们想直接用 ChatGPT 和大语言模型来做交易决策！ [10:19] Hariom Tatsat: 我用大模型赚了大钱，但千万别问我背后的赚钱逻辑是什么（Don't ask me why） [12:25] Hariom Tatsat: 第一道防线

Source quotes

[0:04] Hariom Tatsat: Beyond the Black Box: Interpretability of LLMs in Finance [3:24] Hariom Tatsat: mech interp [3:24] Hariom Tatsat: don't eat sugar [3:24] Hariom Tatsat: steering [5:11] Hariom Tatsat: polysemanticity [5:11] Hariom Tatsat: Eiffel Tower [5:11] Hariom Tatsat: wines [5:11] Hariom Tatsat: Paris [7:40] Hariom Tatsat: obsessive [12:25] Hariom Tatsat: first line of defense

Original transcript

[0:04] So my name is Haram. Uh I'm a VP at uh [0:07] Barclays in New York. U I work in an AI [0:10] team. Uh what we do is we work on [0:13] several use cases related to AI and [0:15] generative AI. Uh there are some [0:17] research that we do as well. And couple [0:19] of months back uh we wrote a paper which [0:21] was called beyond the blackbox [0:23] interpretability of LLMs in finance. Uh [0:26] it got a lot of attention from buy side [0:28] and sell side. we thought uh why not go [0:30] and present to the people. So uh the [0:34] idea is uh what we did was we went [0:36] deeper into the large language models uh [0:39] tried to understand what's happening [0:41] behind the scene and use it for [0:43] financial use cases. There's a lot of [0:45] research going on in open AI and [0:48] anthropic and some of the work uh we [0:50] leveraged for this paper. So in terms of [0:52] the agenda, I'll quickly talking about [0:54] the importance of explanability or [0:57] interpretability in finance. Uh then [1:00] I'll go deeper into something called [1:01] mechanistic interpretability. Uh it's [1:04] kind of difficult for me to pronounce so [1:05] I call it mechinp. Uh we'll talk about [1:08] some tools of mechan as well uh which is [1:10] pars autoenccoder followed by a couple [1:12] of use cases. [1:14] uh the use cases are in credit risk [1:16] where we have used it for sentiment [1:18] analysis and then for trading uh where [1:21] we have tried uh to go inside the LLMs [1:23] and find out uh the signals uh for [1:26] trading and uh the third one for the [1:28] hallucination reduction I mean it can be [1:31] generic not just for finance [1:34] and that's the link of uh the paper I [1:38] love taking QR code or using LLM for QR [1:40] code so I just did that so in 2017 [1:44] OpenAI [1:46] uh they were training a model and they [1:48] were training the model for the next [1:49] word prediction. What they realized was [1:52] that there was one neuron or few [1:54] features which were able to predict the [1:57] sentiment. What I mean by that is that [2:00] if you remove the entire model just keep [2:02] those uh neurons the features it was [2:05] still able to predict the sentiment. So [2:07] the open AI there's a page from them uh [2:11] which is mentioned below you can go [2:13] there and they call it sentiment neuron [2:15] and our research was pretty much uh kind [2:18] of motivated by that uh the genesis of [2:21] our project was uh or the paper was uh [2:24] uh this openai sentiment neuron. [2:27] Now cut to uh 2025. We all have talked [2:31] about uh the adoption [2:34] uh most of the the projects or the geni [2:38] use cases are in PC state. It's very [2:40] difficult to move them to the [2:41] production. 86% is the number. Uh then [2:45] why they're not moving to the production [2:47] uh because we can't explain it. It's [2:49] difficult to explain. We don't have the [2:50] confidence. There was a survey that was [2:52] done which uh which which the number uh [2:56] from the leader is 68%. So around 70% of [3:00] the leaders think that explanability is [3:02] very very important. And another reason [3:05] why it's important I'll let you guys uh [3:07] uh look at it spend some time we all [3:10] have done it. If a prompt is not working [3:12] you put more prompts till the answer is [3:15] right. Not the right thing to do where [3:17] money is at stake. And that's why uh it [3:19] might be very very relevant uh in the [3:21] financial context. Uh now I'll talk [3:24] about mechanistic interpretability meant [3:26] in short. Uh what is it exactly? Uh so [3:30] it's pretty much like neuroscience [3:32] for generative AI. You go deeper into [3:35] the brain uh of the LLMs uh look at the [3:39] neuron level the layer level and try to [3:41] figure out what's happening behind the [3:43] scene what's happening inside. So we [3:45] have seen couple of interpretability [3:47] techniques before as in in the [3:49] conventional machine learning and AI [3:51] space. There have been lot of [3:53] interpretability techniques like feature [3:55] uh relevance and whatnot. But all of [3:58] these they look at the models or the [4:00] input from outside. There are very few [4:02] which go inside and mechanistic [4:04] interpretability in my knowledge is the [4:06] one that goes inside and looks into the [4:09] brain of the LLM. So for example there's [4:11] a person who is suffering from diabetes. [4:13] How will you use conventional like the [4:16] older types of uh interpretability to [4:19] find some pattern? So you look at what [4:22] person eats finds the pattern and then [4:25] try to figure out a game plan for that. [4:26] Right? You try to figure out what foods [4:28] are causing the problem. What what's the [4:30] reason now interp is slightly different. [4:34] What's happening here is you are getting [4:35] inside the brain of the person. you're [4:38] trying to find out the feature which is [4:40] responsible or the neuron which is [4:42] responsible for uh the sugar craving and [4:45] then you try to minimize it. It's not [4:48] just finding the feature but it's also [4:49] going to the next step. I'll talk about [4:52] it. It's called steering but you're also [4:53] trying to minimize the activation of [4:56] that particular feature. Prompt [4:58] engineering again it's looking at things [5:00] at uh external level. You say hey don't [5:02] eat sugar and uh not going inside the [5:05] model. So the the mech interf is the [5:07] only thing which goes uh inside uh the [5:09] model. [5:11] Now uh let's talk about uh sparse [5:14] autoenccoders. This is one of the key [5:16] thing in mechanistic interpretability. [5:19] What if you look at the number of [5:22] parameters in an LLM? Even if you talk [5:24] about the most recent LLM, trillions of [5:26] parameters, but how many concepts are [5:28] there in the world? I mean there are [5:30] many concepts in the world. So there's a [5:32] term called polysemanticity. What does [5:34] this mean? This one neuron or one set of [5:37] features might be responsible for [5:39] multiple concepts. So one neuron might [5:42] be responsible for Michael Jordan, [5:43] astrophysics and whatnot. How do we know [5:47] that what feature or how can we isolate [5:50] or decouple the feature and that's where [5:51] the sparse autoenccoders comes comes [5:53] into picture. What you're doing is [5:56] basically you take one layer of the LLM [5:58] you make a hole and try to zoom into [6:02] what's happening by having a wider model [6:04] by having a wider machine learning model [6:07] or wider uh artificial neural network [6:09] which you train. So in this example you [6:12] are asking a question what's the capital [6:14] of France uh it answers Paris but uh the [6:18] sparse autoenccoders you can see there [6:20] are a couple of features which gets [6:22] isolated decoupled those are wines and [6:25] Eiffel Tower and whatnot right an easier [6:29] way to explain would be uh this is how [6:32] the information flows in the LLM uh [6:34] there's something called residual stream [6:36] there are a lot of information [6:38] intermingled [6:40] mixed So the cats, dogs, astrophysics, [6:43] Michael Jordan, everything is kind of [6:45] mixed in the matrix and that's something [6:48] you're trying to decouple using the [6:51] sparse autoenccoder. So in this one you [6:52] see a wider yellow uh neural network [6:55] which uh decouples uh different features [6:58] which are dogs, cats and whatnot. So [7:01] very relevant and this is what we kind [7:03] of leveraged for all the use cases. Now [7:06] another question I'll quickly go through [7:09] this slide. Uh you have the feature [7:11] number. How do you know that what does [7:13] this feature represent? So having some [7:16] name in normal English would be [7:18] important. How do you find the human [7:21] readable label? You take a feature you [7:23] pass or run it by couple of sentences [7:26] and where it sees the maximum activation [7:29] that's what the feature name is. Uh it's [7:32] like looking into the brain. The left [7:33] brain activates more on science and [7:36] right brain on arts. That's how you kind [7:38] of name those features to some label. [7:40] Now coming to the use cases. Now all [7:43] these is good but how do we use it in [7:45] finance. So the first use case is uh [7:47] sentiment analysis for credit risk. What [7:50] we did was uh we tried to find out uh [7:55] what can be the features for credit [7:57] risk. How we did that? So there's a [8:00] website I'll encourage you guys that's [8:01] there in the bottom. It's called [8:03] neutedia. This is from Google's deep [8:05] mind. So they have trained some sparse [8:08] autoenccoders labeled them and put it [8:10] there. You can play around with it. So [8:12] we put this keyword credit risk related [8:15] and it gave us a couple of features that [8:17] might be activated uh on credit risk [8:21] right and this the second thing was it's [8:26] it doesn't only give you the name of the [8:28] feature you can also steer it think of [8:30] it that you find some neuron in the [8:32] brain you are making it obsessive on [8:34] credit risk you are make you're making [8:36] it more activate as as you're trying to [8:39] activate it more on couple of features. [8:43] So in this example if you see uh we have [8:46] taken one feature [8:49] uh and we activated it uh we steered it. [8:52] Steered it steering is basically [8:54] increasing the activation. You make it [8:56] obsessive on credit risk. Then we sent [9:00] or passed two sentences. The prompt was [9:02] uh [9:04] give the sentiment for this sentence on [9:06] a scale of - 10 to 10. [9:09] Now without steering the sentiment was [9:11] three. Uh there are a lot of sentences [9:14] but uh just focus on that number three. [9:16] And with steering the sentiment was [9:19] four. And if you look at the rational [9:22] for why the sentiment was four, the LLM [9:24] is focusing on couple of credit risk [9:26] related keywords. Right? So it's [9:28] becoming more activated on credit risk [9:32] sentiment without any prompting without [9:34] any fine-tuning. it's just coming from [9:36] inside the model. [9:38] And this this one was kind of something [9:41] we used for multiple use cases as well. [9:43] Really impressive uh feature of uh LLMs. [9:49] Now I mean we kind of extrapolated the [9:52] same thing. We ran it uh through [9:53] multiple sentences, thousands of [9:55] sentences. And what we found out that [9:58] with steering with the LLMs being [10:00] focused on the credit risk the accuracy [10:03] with the human annotated sentence was [10:05] much higher as compared to uh just the [10:08] non- steered model across the board for [10:10] five sentiment. Uh a lot of numbers [10:14] there but the bottom line is that the [10:16] accuracy was much higher with the [10:17] steered model. The next use case is [10:19] trading. There are a lot of traders who [10:22] are like hey we want to use chat GPT and [10:25] LLMs for trading and there are few let's [10:28] say somebody comes to you and says I've [10:30] made money using LLMs but don't ask me [10:33] why right so that's not something which [10:36] is acceptable and that's why going [10:38] deeper inside and finding some neurons [10:40] finding the features is important that's [10:42] something we tried the idea there was [10:45] that we know how to find the features so [10:48] we took financial news ran it by the [10:52] sparse autoenccoders tried to find out [10:55] what features activate. So we make made [10:58] a list of around 50 to 100 features. We [11:00] also had the feature label uh these are [11:02] financial news related features and [11:05] named and entity recognition features [11:07] and then we made a prediction model [11:10] of feature and their activation versus [11:13] price going up or down. Right? So those [11:16] features and their activation was used [11:18] as input to predict whether the price [11:20] will go up or down. Uh right side you [11:23] can see the examples just some new sent [11:26] uh sentence some headlines and the [11:28] features activating and the result was [11:31] quite impressive. [11:32] We just ran a tree model and what we saw [11:35] was that some of the features which came [11:39] out to be very very relevant were very [11:43] intuitive. The first one is name of [11:45] places and location. Something we call [11:47] as named entity recognition. Uh that [11:50] that's a common thing common model that [11:52] we use for a lot of use cases in [11:54] finance. Then there was financial terms [11:56] which was a second one stock ticker [11:58] symbol. If you look at it I mean most of [12:00] these are pretty relevant and impressive [12:03] one was uh names of places and location. [12:05] I mean any feature that clearly comes [12:07] out uh when we train it against uh the [12:10] trading model. So really impressive [12:13] result. Uh we are working on some next [12:16] step on this. Uh adding more data, [12:18] adding more features, making bigger [12:20] models and trying the sparse [12:21] autoenccoders on bigger models. [12:25] The next one hallucination reduction uh [12:28] not just for finance I mean this can [12:30] also be used for other sectors as well. [12:32] Uh the idea was that uh you try to see [12:37] so let's say there's a financial chatbot [12:40] and you were asking some questions [12:42] related to finance. So ideally the few [12:46] some of the features should activate. So [12:49] in the brain let's say there there are [12:50] few neurons related to physics and if [12:52] you're asking question related to [12:54] physics those neurons should ideally [12:56] activate. If they're not activating [12:58] maybe there's something wrong. So what [13:00] we did we sent a lot of sentences uh [13:04] financial sentences to the LLM looked at [13:07] what all features are getting activated. [13:08] So if you look at the right side those [13:10] are the feature number mean activation [13:12] and feature label. I mean feature label [13:14] is something that's important as well in [13:16] plain English it uh kind of mentions why [13:19] it should be if it should be related to [13:22] finance or not. [13:24] And once we found those features, then [13:27] we did some kind of conditional [13:29] activation. If you're giving a prompt, [13:32] if those features are not activated, you [13:34] can keep a threshold. Again, that's kind [13:36] of subjective. But let's say 20% of the [13:38] features, financial features are not [13:40] activated. It's not something wrong, but [13:42] we need to augment it with more [13:44] information. So, rag maybe you can use [13:46] it. You can add more document in that. [13:48] And that's something we tried. [13:50] So without rag or let's say without [13:54] activation is the result on left hand [13:56] side when we ask some questions related [13:57] to earning reports and regulatory [14:01] documents on the right hand side with [14:03] that conditional activation [14:05] uh the reply was much better with [14:08] context with relevance uh with with uh [14:11] with some references right so this is [14:15] something probably can be used as first [14:18] line of defense in pro uh controlling [14:20] against hallucination. So I think that [14:23] was pretty much it. Uh in terms of final [14:26] thought uh I would say that I mean for [14:30] finance sector which is uh highly [14:33] regulated money is at stake having [14:36] something explainable interpretable is [14:39] very important also covered in some of [14:41] the previous sessions [14:43] ses which are sparse autoenccoders uh [14:46] which is something we found as the most [14:49] impactful thing when it comes to [14:51] understanding the LLM structure going [14:53] deeper and understanding it uh they can [14:56] definitely help build trustworthy [14:59] transparent AI and uh this this is a [15:02] sentence from Dario who is the CEO of [15:04] Enthropic not endorsing him but it kind [15:06] of uh is very much aligned to the paper [15:09] that we wrote the work that we did uh [15:12] you see a lot of models coming every day [15:14] GPT5 last week Anthropic released claude [15:17] 4.1 and whatnot right but [15:20] interpretability is something that gets [15:21] less attention especially for sectors [15:24] like finance that's uh something which [15:27] we should focus on and uh our paper the [15:30] work that we did was an attempt in that [15:32] direction. [15:34] I think that's it and again [15:36] [Applause]