Cal Poly AI Pipeline Solves $1.5M Speaker Identification Challenge
Ai4
Combining lightweight decision trees with transformer embeddings enables rapid, low-cost legislative speaker identification by extracting self-introductions directly from initial utterances.
## Background
Published on YouTube by Ai4 on August 18, 2026, this presentation features Sofija Dimitrijevic from California Polytechnic State University presenting a lightweight machine learning pipeline designed to identify unknown speakers in legislative testimony. The talk explores how detecting self-introductions in opening sentences across 1.54 million state legislative utterances enables cost-effective speaker attribution by comparing classical tree models, feature engineering, and fine-tuned BERT transformers.
## TL;DR
To address the expensive bottleneck of identifying public speakers in legislative hearings where facial and audio records do not exist, Cal Poly's Digital Democracy Project developed a system that detects opening self-introductions. Using a corpus of 1.54 million utterances labeled at zero annotation cost through weak supervision, the team evaluated baseline decision trees, XGBoost, and fine-tuned BERT architectures. Stacking BERT embeddings into an XGBoost classifier achieved a 97.8% F1-score, reducing test errors while keeping inference compute lightweight. A subsequent error audit revealed that a quarter of the model's apparent false positives were actually correct detections that exposed noise in the original heuristic labels.
## Atmosphere & style
The presentation carries a crisp, pragmatic, and intellectually grounded tone characteristic of applied civic technology research. Dimitrijevic delivers her findings with methodical clarity and modest confidence, turning a niche data-labeling bottleneck into an engaging narrative of resource-conscious machine learning. The viewing experience feels rewarding and instructional, punctuated by the satisfying revelation where the algorithmic model proved more accurate than the noisy training heuristics.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) Associating real names with audio labels for unknown public speakers is the most time-consuming and expensive bottleneck in civic transcription pipelines.
- [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) Because public citizens almost invariably state their names in their opening sentence, identifying a single self-introduction allows diarization tools to attribute all subsequent utterances to that speaker.
- [02:22](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=142s) Basic keyword searches fail on legislative speech due to false positives on common phrases and false negatives on condensed introductions lacking explicit intro markers.
- [04:13](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=253s) The project applied weak supervision across 50,000 candidate utterances by matching speaker names verbatim against spoken words, generating a robust training set at zero manual annotation cost.
- [08:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=484s) Stacking predictions from a fine-tuned BERT model into an XGBoost classifier allowed the pipeline to leverage deep transformer embeddings while retaining lightweight tabular efficiency and interpretability.
- [09:42](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=582s) Model performance scaled from a 93.0% baseline F1-score with a simple decision tree up to 97.8% F1 with the stacked XGBoost and BERT architecture.
- [09:42](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=582s) A manual audit revealed that 25% of the model's supposed false positives were genuine self-introductions that the automated weak-supervision heuristic had mislabeled.
- [12:29](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=749s) Engineers working under compute constraints should test cheaper classical ML models first, as gradient boosted trees can rival expensive transformer architectures.
- [12:29](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=749s) Auditing misclassifications is essential when using automated labeling, as model 'errors' often expose underlying noise in the pseudo-ground truth.
- [12:29](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=749s) Civic technology initiatives can eliminate substantial manual labeling overhead by creatively repurposing existing institutional records and transcript metadata.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) The Challenge of Identifying Speakers in Legislative Hearings
In the past two decades, local and state political reporting has suffered massive funding cuts. To help fill this journalistic gap, California Polytechnic State University developed the Digital Democracy Project, which provides AI-assisted coverage and transcription of state legislative hearings. While modern speech-to-text engines and speaker diarization algorithms reliably segment audio into generic labels such as "speaker one" or "speaker two," associating real human names with those labels remains a persistent bottleneck. Previous research at Cal Poly identified name attribution as the single most time-consuming and expensive step in the entire transcription pipeline, even when high-resolution audio and video feeds are accessible.
Identifying established legislators is relatively tractable because they already possess facial recognition profiles, voice prints, committee rosters, and prominent desk nameplates. However, members of the general public who testify have none of these digital records. The core breakthrough of this project lies in recognizing that "the data can label itself": unknown public speakers almost invariably introduce themselves in their very first spoken sentence (e.g., stating "My name is..." or "I'm here on behalf of..."). If a system can pinpoint that single self-introduction utterance and extract the speaker's name, diarization can then automatically propagate that name across every subsequent utterance the speaker makes throughout the entire hearing transcript.
- **Project Lead & Collaborators**: Sofija Dimitrijevic, working under faculty advisor Dr. Foaad Khosmood at Cal Poly's Digital Democracy Project, alongside student collaborators Paula Vidal and Casey Liu.
- **Economic Value**: Solved speaker name attribution for approximately the cost of a single afternoon of human labeling effort.
- **Problem Statement**: Determining whether a lightweight machine learning pipeline can catch the single introductory sentence and accurately identify the speaker among millions of raw utterances.
---
[02:22](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=142s) Self-Introduction Edge Cases and Linguistic Complexity
In legislative video footage, on-screen graphics display metadata like bill numbers, discussion topics, and presiding legislators, but provide zero contextual information about testifying public citizens beyond their raw video feed. Distinguishing genuine self-introductions from ordinary speech patterns requires nuanced natural language processing. Simple keyword matching fails because speech patterns vary wildly between straightforward declarations, condensed introductions, and incidental phrase collisions.
```mermaid
flowchart TD
accTitle: Self-Introduction Linguistic Classification Examples
subgraph EasyCase["Obvious Self-Introduction"]
A["'Hello and thank you... My name is Kendra Scott...'"] --> A1["Explicit Keyword Pattern + Name"]
end
subgraph HardCase["Condensed Introduction"]
B["'Good morning, Glenn Brekes for the Drug Policy Alliance.'"] --> B1["No explicit 'My name is' marker"]
end
subgraph FalsePositive["Keyword False Positive"]
C["'I am against this bill.'"] --> C1["Matches 'I am' keyword without introducing a person"]
end
```
Initial baseline attempts using basic keyword searches (such as scanning for `"My name is"` or `"I am"`) readily flagged explicit declarations like *"Hello and thank you for having me here today. My name is Kendra Scott..."*, but caused false alarms on statements like *"I am against this bill."* Meanwhile, subtle real-world introductions such as *"Good morning, Glenn Brekes for the Drug Policy Alliance"* completely lacked introductory keywords and were missed by basic rule sets. While human listeners can instantly differentiate these cases, automating this distinction across 1.5 million utterances demands robust statistical classification.
---
[04:13](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=253s) Data Pipeline and Weak Supervision Heuristic
To construct the dataset, the team gathered 1.54 million raw utterances spanning five state legislative sessions: California (2015–2016 and 2017–2018), Texas (2015–2016), Florida (2015–2016), and New York (2015–2016), all published publicly on Hugging Face. The pipeline filtered out approximately 700 known legislators appearing on official legislative and committee rosters, reducing the corpus to 520,000 utterances. A stratified sample of 100,000 utterances was extracted across states to prevent California from disproportionately biasing the dataset, and an off-the-shelf spaCy named-entity recognition (NER) model isolated utterances containing detected person names, leaving 50,000 candidate utterances.
```mermaid
flowchart LR
accTitle: Data Filtering and Feature Pipeline
subgraph Ingestion["Corpus Ingestion & Filtering"]
A["1.54M Raw Utterances\n(CA, TX, FL, NY)"] --> B["Drop Roster Legislators\n(~520k remaining)"]
B --> C["Stratified Sample\n(~100k utterances)"]
C --> D["spaCy NER Filter\n(50k Candidates)"]
end
subgraph Supervision["Weak Supervision Labeling"]
D --> E{"Does candidate name\nmatch speaker name?"}
E -- "Yes" --> F["Label: Self-Intro (1)"]
E -- "No" --> G["Label: Non-Intro (0)"]
end
subgraph Modeling["Feature Extraction & Training"]
F & G --> H["Contextual Features\n(Words before/after, token position)"]
H --> I["Decision Tree / Random Forest / XGBoost"]
H --> J["BERT Classifier Feature"]
J --> I
end
```
Rather than undertaking costly manual annotation, the project utilized weak supervision. Because historical transcripts already contained known speaker identities, a deterministic rule checked whether the speaker's assigned name appeared verbatim within the words they uttered. This single programmatic rule labeled all 50,000 examples at zero annotation cost. Feature extraction captured structural and contextual cues, including word positions, lexical tokens appearing immediately before and after the detected name entity, and sentence-level positional offsets.
- **Candidate Models Tested**:
- Baseline Decision Tree
- Random Forest Classifier
- Gradient Boosted Trees (XGBoost)
- Fine-Tuned Bidirectional Encoder Representations from Transformers (BERT)
---
[08:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=484s) Pipeline Architecture and Feature Stacking
The classification pipeline integrates both classical machine learning and transformer architectures. The classical models evaluate discourse context features, lexical n-grams, and named-entity positioning, while Paula Vidal and Casey Liu trained an auxiliary fine-tuned BERT transformer on a separate gold-standard set of 600 hand-labeled utterances (evaluated against a held-out gold set of 30).
Instead of relying solely on the transformer as an end-to-end replacement, the team fed the output predictions of the fine-tuned BERT model back into the tabular feature set of the XGBoost pipeline. This stacking technique allowed the lightweight tree models to leverage deep transformer embeddings while preserving high interpretability, rapid training times, and low inference compute overhead.
---
[09:42](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=582s) Model Performance and Error Analysis
Evaluating the models on an unseen 20% test partition (~10,000 candidate utterances) demonstrated high baseline performance across all architectures. A simple decision tree achieved an initial baseline F1-score of 93.0%. Transitioning to an XGBoost model using contextual discourse features elevated the F1-score to 97.6% (a 4.6% improvement). Integrating the fine-tuned BERT transformer prediction as an auxiliary feature further boosted the overall F1-score from 97.5% to 97.8%, eliminating 34 remaining classification errors (dropping total misclassifications from 241 down to 207).
| Model Architecture | F1-Score | Performance Notes |
|:--- |:--- |:--- |
| **Decision Tree Baseline** | 93.0% | Simple, interpretable baseline using structural features |
| **XGBoost (Standard)** | 97.5% – 97.6% | Driven primarily by positional discourse context features |
| **XGBoost + Stacked BERT Feature** | 97.8% | Reduced total test errors from 241 to 207 (34 errors eliminated) |
A manual audit of the remaining classification errors revealed an unexpected finding: 25% of the instances flagged as "false positives" by the evaluation metrics were actually correct self-introductions that the automated weak-supervision heuristic had mislabeled as negative examples. This confirmed that the machine learning model had outperformed the imperfect pseudo-ground-truth labeling heuristic.
---
[12:29](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=749s) Key Takeaways and Civic Technology Applications
Sofija concluded the talk with three core recommendations for engineering teams building machine learning systems under resource constraints:
1. **"Try the cheaper thing first"**: Classical algorithms such as gradient boosted decision trees can match or approximate expensive deep transformer architectures for a fraction of the compute and labeling cost.
2. **"Audit your answer key"**: When deploying automated labeling or weak supervision, manually inspect misclassifications to verify whether the model is failing or exposing latent noise in the underlying ground truth.
3. **"Find the labels that you already have"**: Exploit existing structured data, meeting minutes, and metadata to generate programmatic training labels creatively without paying large manual annotation fees.
The presentation concluded with acknowledgments to research advisor Dr. Foaad Khosmood, project peers Paula Vidal and Casey Liu, and the Cal Poly Institute for Advanced Technology and Public Policy for sponsoring the research.
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) the data can label itself — Sofija Dimitrijevic
> [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) My name is...
> [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) I'm here on behalf of...
> [02:22](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=142s) Hello and thank you for having me here today. My name is Kendra Scott...
> [02:22](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=142s) I am against this bill.
> [02:22](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=142s) Good morning, Glenn Brekes for the Drug Policy Alliance
> [12:29](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=749s) Try the cheaper thing first — Sofija Dimitrijevic
> [12:29](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=749s) Audit your answer key — Sofija Dimitrijevic
> [12:29](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=749s) Find the labels that you already have — Sofija Dimitrijevic
## Mentioned
Sofija Dimitrijevic, Dr. Foaad Khosmood, Paula Vidal, Casey Liu, California Polytechnic State University, Digital Democracy Project, Institute for Advanced Technology and Public Policy, Ai4, Hugging Face, spaCy, BERT, XGBoost, Drug Policy Alliance, Kendra Scott, Glenn Brekes
## Video description
Sofija Dimitrijevic, Student, California Polytechnic State University
What if we could identify unknown speakers in government hearings by only using the words they say? As local journalism declines and reporters increasingly rely on incomplete transcripts, automatically matching legislative testimony to the correct speaker has become an expensive and critical problem. Yet, the answer is usually sitting in the speaker's very first sentence, when they introduce themselves. This talk will walk through a lightweight machine learning pipeline that detects these self-introductions and extracts the speaker's name from 1.54 million utterances across five state legislative sessions, comparing decision trees, random forests, XGBoost, and a fine-tuned BERT classifier. Attendees will learn how careful feature engineering can rival transformer models while remaining interpretable and compute-efficient, and how combining the two approaches yields the strongest results. The talk will also share a surprising finding: some of our model's "errors" were actually correct, revealing hidden noise in the source data itself. For attendees interested in machine learning, natural language processing, civic technology, and building practical systems that make institutions more transparent, this talk will show how a resource-conscious solution can tackle a deceptively difficult problem.
Researchers at California Polytechnic State University's Digital Democracy Project demonstrated that speaker name attribution in state legislative hearings can be solved cost-effectively by training lightweight machine learning models to detect opening self-introductions from members of the public.
Solving Speaker Attribution for Public Testifiers (0:04)
State and local political journalism has experienced severe funding declines over the past two decades. To address this gap, Cal Poly developed the Digital Democracy Project to automate transcription of legislative proceedings. Although modern diarization tools successfully segment audio into generic tags like 'speaker one,' assigning real human identities to those segments remains the most expensive and time-consuming stage of transcription. While elected officials possess known rosters, facial recognition profiles, and voiceprints, public citizens testifying before committees leave no digital footprint.
Project lead Sofija Dimitrijevic, working with Dr. Foaad Khosmood, Paula Vidal, and Casey Liu, capitalized on the insight that the data can label itself. Because public witnesses almost always introduce themselves in their opening sentence, locating this single self-introduction allows diarization tools to propagate that name across every subsequent utterance the speaker makes, resolving the attribution bottleneck for the cost of a single afternoon of manual work.
Linguistic Complexities Beyond Simple Keyword Rules (2:22)
On-screen legislative metadata displays bill numbers and committee rosters but provides no context for testifying citizens. Automated detection cannot rely on simple keyword matching such as 'My name is' or 'I am,' which fails across real-world linguistic variations. Keyword matching triggered false alarms on standard statements like 'I am against this bill,' while completely failing to capture condensed, marker-free introductions such as 'Good morning, Glenn Brekes for the Drug Policy Alliance.' Distinguishing genuine introductions across 1.5 million utterances required robust statistical classification of sentence structure and discourse context.
Weak Supervision and Transformer Feature Stacking (4:13)
To build an annotated corpus without prohibitive manual labeling costs, the team sampled 1.54 million raw utterances from California, Florida, New York, and Texas legislative sessions. After filtering out 700 roster legislators and isolating person entities via spaCy named-entity recognition, the researchers used a weak supervision heuristic on 50,000 candidate utterances. The deterministic rule labeled an utterance as a self-introduction if the speaker's historically recorded name appeared verbatim in the spoken text, generating training labels at zero annotation cost.
Rather than relying purely on an expensive end-to-end deep learning model, the pipeline extracted structural features, token offsets, and lexical n-grams for classical tree models. The team fine-tuned a BERT transformer on 600 hand-labeled utterances, then stacked BERT's output predictions back into an XGBoost classifier as an auxiliary tabular feature, preserving low compute overhead and fast inference while retaining deep contextual representations.
Model Performance and Exposing Labeling Noise (9:42)
Testing across an unseen evaluation split of approximately 10,000 utterances showed strong baseline performance, with a standard decision tree achieving a 93.0% F1-score. Transitioning to XGBoost with contextual discourse features raised the F1-score to 97.6%. Incorporating the stacked BERT feature further increased the F1-score to 97.8%, reducing test misclassifications from 241 to 207.
A manual audit of the remaining errors revealed that 25% of the purported 'false positives' were actually valid self-introductions that the automated weak-supervision heuristic had mislabeled as negative examples. This confirmed that the hybrid machine learning model had successfully surpassed the precision of the underlying pseudo-ground-truth labeling heuristic.
The project concludes that resource-constrained civic technology initiatives should prioritize cheaper, interpretable tabular models before deploying expensive architectures, leverage existing structured records to generate programmatic training labels, and consistently audit evaluation errors to catch latent noise in automated ground truth.
精炼阅读摘要
加州理工州立大学（Cal Poly）的数字民主项目团队由 Sofija Dimitrijevic 主导，该研究提出了一种低成本的弱监督与机器学习流程，旨在通过自动识别立法听证会中公众发言人的单句自我介绍，解决转录流水线中说话人身份归因的高昂成本难题。
听证会说话人归因瓶颈与自标签洞见 (0:04)
在过去二十年间，美国地方与州级政治报道面临资金严重削减的困境。为填补这一新闻监督缺口，加州理工州立大学开发了数字民主项目，利用人工智能辅助州立法听证会的转录与报道。尽管现有的语音转文本与说话人日志化技术能够稳定切分语音段落并赋予通用代号，但将真实人名与发言片段关联仍是整个流程中最耗时且昂贵的步骤。在听证场景中，知名立法委员因具备委员会花名册、人脸特征、声纹以及座位名牌，身份较为容易确定；而参与作证的普通公众则没有任何现成的数字记录。
该研究的核心突破基于一项关键洞见：数据本身能够实现自我标注。研究团队发现，未知的公众发言人几乎无一例外会在发言的第一句话进行自我介绍（例如说明姓名或代表机构）。系统若能精准捕捉这一句自我介绍并提取姓名，说话人日志化系统便可自动将该身份推及至该发言人在整场听证会中的所有后续发言，从而以极低的人工干预成本完成全篇归因。
语言歧义挑战与弱监督数据过滤流程 (2:22)
立法听证现场的屏幕字幕仅显示法案编号和议程，无法提供公众证人的背景信息。通过自然语言处理识别自我介绍面临显著的语言多义性：简单的关键词匹配（如“My name is”或“I am”）极易引发误报，例如将表达立场的“I am against this bill”（我反对该法案）误判为自我介绍；同时，规则检索也容易遗漏没有引导词的紧凑表达（如“Good morning, Glenn Brekes for the Drug Policy Alliance”）。在数以百万计的语音切片中，区分这些细微差异必须依靠鲁棒的统计分类模型。
为构建训练集，团队从加利福尼亚州、得克萨斯州、佛罗里达州和纽约州的立法会议中收集了154万条原始发言。数据清洗流程首先剔除了官方花名册上的约700名议员，使语料缩减至52万条；随后在各州间进行分层抽样以提取10万条语句，防止加州数据主导样本；最后利用现成的 spaCy 命名实体识别模型筛选出包含人名实体的5万条候选发言。研究未采用高成本的人工标注，而是设计了弱监督启发式规则：若候选发言中逐字出现了历史记录中该发言人的既定姓名，则自动标记为自我介绍，以零标注成本完成了5万条样本的标签构建。
特征堆叠架构与渐进式分类性能 (8:04)
研究团队提取了包含语篇上下文、实体前后词汇标记及句级位置偏移在内的结构特征，并对决策树、随机森林和梯度提升树（XGBoost）进行了对比评估。此外，项目成员 Paula Vidal 与 Casey Liu 基于600条人工标注语料微调了一个辅助 BERT 模型。团队并未将 BERT 作为端到端解决方案，而是创新地将微调 BERT 的预测概率作为新增特征堆叠注入 XGBoost 的表格型特征集中。这种特征堆叠架构既汲取了深度语言模型的语义表征能力，又兼顾了树模型的低计算开销与可解释性。
在未参与训练的20%独立测试集（约10,000条发言）上，基线决策树模型取得了93.0%的 F1 分数；引入语篇上下文特征的 XGBoost 模型将 F1 分数显著提升至97.6%；进一步融合微调 BERT 预测特征后，集成模型的 F1 分数最终达到97.8%。该堆叠方案将整体测试错误样本数从241个缩减至207个，成功消除了34个顽固分类错误。
伪标签误差审计与资源受限工程启示 (9:42)
在对剩余错误分类进行人工抽检时，团队发现了一个关键现象：在评估指标判定的“假阳性”错误中，有25%实际上是模型正确识别出的真实自我介绍，其之所以被归为错误，是因为最初用于弱监督的自动化规则未能匹配出该标签。这表明在特定边缘案例中，统计机器学习模型的泛化表现已经超越了充当黄金标准的弱监督启发式规则本身。
基于这些发现，研究负责人 Sofija 针对资源受限环境下的机器学习工程提出了三项实践准则：首先，优先尝试轻量级方案，梯度提升树等经典算法往往能以极低的算力和标注成本逼近昂贵的大型模型；其次，审慎审计基准答案，出现错误预测时应排查是否为伪标签规则自身的系统性噪声；最后，充分挖掘现有数据资产，利用已有会议记录和结构化元数据程序化构建标签，避免盲目承担高昂的人工标注开销。
该研究证实了利用弱监督与特征堆叠技术解决公民科技转录难题的可行性，以单日下午的人力成本替代了繁重的逐行人工校对。然而，弱监督固有的标签噪声也凸显了在缺乏高质量人工验证集时，模型评估指标可能存在的失真风险。
Detailed notes — zh
## Background
该内容由 YouTube 频道 Ai4 于 2026 年 8 月 18 日发布，主讲人是来自加州理工州立大学（Cal Poly）数字民主项目的 Sofija Dimitrijevic。视频简介指出，本演讲展示了一套轻量级机器学习流水线，旨在仅通过立法听证会中发言人的第一句自我介绍来精准识别未知公众证人，以低算力成本解决地方新闻报道中转录与发言人匹配难题。
## TL;DR
在立法听证会转录中，缺乏先验数据的公众证人身份标注一直极为耗时，研究团队发现发言人通常会在第一句话进行自我介绍。团队从五个州立法会议的 154 万条发言中筛选样本，利用弱监督与特征工程对比了决策树、随机森林、XGBoost 及微调 BERT 分类器。实验表明，仅需低廉算力成本的 XGBoost 与 BERT 融合后可达到 97.8% 的 F1 值，并在错误分析中发现 25% 的假阳性实为标注基准本身的错误，由此提炼出优先选用轻量方案、深入审计真值数据与善用既有标签的工程经验。
## Atmosphere & style
整场演说展现出严谨务实且条理清晰的学术工程风格，主讲人语调沉稳笃定，将宏观的公民科技愿景与微观的数据工程细节紧密串联。在拆解“四分之一的预测错误其实是真值标签出错”时，叙事洋溢着侦探解谜般的探索乐趣，生动呈现出在盲目追逐大模型的时代，以低成本经典机器学习和精细数据清洗解决复杂现实问题的优雅与从容。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) 面对美国地方与州级政治新闻报道资金匮乏的困境，加州理工州立大学数字民主项目致力于利用 AI 辅助听证会报道，填补媒体监督空白。
- [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) 在立法听证会转录流程中，为缺乏先验声纹或面部档案的公众发言人匹配真实姓名，是当前流水线中最耗时的瓶颈环节。
- [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) 核心洞察在于数据具备自标注能力：公众证人通常在作证的第一句话中自我介绍，只要捕获该语句并提取姓名，即可通过 Speaker Diarization 将身份传播至整场发言。
- [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) 由于自我介绍句式多样且容易产生歧义，简单的关键词匹配极易失效，必须依赖机器学习进行语义与语境分类。
- [04:13](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=253s) 研究团队从 154 万条跨州立法发言中过滤出 5 万条样本，并结合 spaCy 实体识别与已知发言人信息实现了零人工标注成本的弱监督标注。
- [06:45](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=405s) 基于词汇前后上下文、实体相对位置及会话语境的特征工程，使轻量级梯度提升模型 XGBoost 取得了 97.5% 以上的 F1 表现，且推理无需重型 GPU。
- [06:45](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=405s) 将仅用 600 条人工样本微调的 BERT 预测概率作为额外特征输入 XGBoost，模型 F1 达到 97.8%，错误数进一步削减了 34 例。
- [11:14](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=674s) 在深度错误分析中，团队发现 25% 的假阳性错误实际上是模型预测正确而弱监督真值标签出错，证明实际模型性能优于表面评测指标。
- [11:14](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=674s) 工程实践应优先尝试低成本的经典机器学习方案，盲目部署昂贵的大语言模型往往并不必要且不够经济。
- [11:14](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=674s) 在使用自动化或弱监督构建数据集时，必须定期抽样人工审计真实标签，并充分发掘现有元数据中潜藏的标注线索。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) 立法听证会中的发言人识别困境与“自我介绍”核心洞察
加州理工州立大学（Cal Poly）数字民主项目（Digital Democracy Project）的 Sophia Dimitrijevic 与 Foaad Khosmood 博士、Paula Vidal 及 Casey Liu 合作开展了题为《刚才发言的是谁？》（"Who Just Testified?"）的研究，旨在解决立法听证记录中的自我介绍检测问题。过去二十年间，美国地方和州级政治新闻报道面临严重的资金缩减，数字民主项目致力于利用 AI 辅助听证会报道来填补这一空白。
```mermaid
flowchart TD
accTitle: 听证会转录与发言人姓名匹配工作流
subgraph S1["现有成熟技术"]
A["语音转文字 (Speech-to-Text)"]
B["说话人日志化 (Speaker Diarization)<br/>划分 Speaker 1, Speaker 2..."]
end
subgraph S2["瓶颈与核心洞察"]
C{"发言人类型"}
D["已知议员：声纹/面部/名牌直接匹配"]
E["公众证人：无先验数据 (主要难点)"]
F["关键洞察：证人第一句通常自我介绍<br/>'My name is...' / 'I'm here on behalf of...'"]
end
subgraph S3["自动化传播"]
G["检测单句自我介绍并提取姓名"]
H["通过 Diarization 将姓名传播至整场听证会发言"]
end
A --> B
B --> C
C -->|议员| D
C -->|公众证人| E
E --> F
F --> G
G --> H
```
在现有的自动化听证会转录流程中，语音转文字（Speech-to-Text）和说话人日志化（Speaker Diarization，即将音频切分为 Speaker 1、Speaker 2 等）已基本解决。然而，为匿名标签赋予真实姓名仍然是整个转录流水线中最耗时的一步。议员等已知发言人拥有声纹记录、面部数据以及现场名牌，但参与作证的普通公众则没有任何先验档案。研究团队的核心洞察在于：**数据本身包含自我标注能力**。公众发言人通常会在作证的第一句话中进行自我介绍（例如 “My name is...” 或 “I'm here on behalf of...”）。只要机器能够精准捕获这一句话并提取出姓名，说话人日志化技术就能将该姓名顺畅传播至该发言人在整场听证会中的所有发言。
- 视频画面中通常包含法案编号、讨论议题以及关联议员等元数据，但对公众证人往往仅有一张面孔，记者处理时极为棘手。
- 简单的关键词匹配极易失效：例如短语 “I am against this bill” 虽包含 “I am”，但并非自我介绍；而 “Good morning, Glenn Brekes for the Drug Policy Alliance” 没有出现 “My name is”，却是货真价实的自我介绍。
---
### [04:13](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=253s) 数据预处理、弱监督标注与特征工程
研究团队收集了五个州级立法会期的 154 万条原始发言话语（Utterances），涵盖加利福尼亚州（2015–2016、2017–2018）、德克萨斯州、佛罗里达州及纽约州（均为 2015–2016 年）。为了降低计算复杂度并聚焦公众发言，团队设计了一套严格的数据筛选与弱监督标注流程。
```mermaid
flowchart LR
accTitle: 数据过滤与弱监督标注管线
A["154 万原始发言<br/>(5 个州级立法会期)"] --> B["剔除约 700 位在册议员<br/>(剩余约 52 万条)"]
B --> C["分层抽样 (Stratified Sampling)<br/>(平衡各州比例，保留 10 万条)"]
C --> D["spaCy 命名实体识别过滤<br/>(保留含人名发言，剩余约 5 万条)"]
D --> E["弱监督规则标注 (Weak Supervision)<br/>发言文本是否包含其已知姓名"]
```
在数据标注阶段，团队没有采用昂贵的全量人工标注，而是利用转录本中已知的事后发言人身份，结合现成的自然语言处理工具包 spaCy 进行弱监督（Weak Supervision）标注：判断发言人所说的话语中是否包含其自身姓名。这一规则以零人工标注成本为 50,000 条样本完成了自动标注。
- **特征工程提取的信号包括：**
- 姓名实体前后的词汇上下文（Words before and after the name）。
- 姓名在整句话中出现的相对位置（Position of the name in the sentence）。
- 话语中是否存在人名实体的判别标志。
- 会话语境特征（Discourse context features）。
---
### [06:45](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=405s) 模型对比：经典机器学习与微调 BERT 的结合
团队构建了多种分类模型进行对比实验，包括决策树基线（Decision Tree）、随机森林（Random Forest）以及梯度提升树（XGBoost），并由团队成员 Paula Vidal 与 Casey Liu 基于 600 条人工标注样本（加 30 条黄金测试集）微调了一个 BERT 双向编码器转换器模型。
| 模型 / 组合方案 | 准确率 / F1 表现 | 核心特点与代价 |
|:--- |:--- |:--- |
| **决策树 (Decision Tree Baseline)** | ~93.0% F1 | 简单易解释的基础基线 |
| **XGBoost (特征工程驱动)** | 97.5% - 97.6% F1 | 依靠语境特征与梯度提升，无需 GPU 即可高效推理 |
| **XGBoost + BERT 特征融合** | **97.8% F1** | 将 BERT 输出作为额外特征输入 XGBoost，总错误数从 241 降至 207 |
```mermaid
flowchart TD
accTitle: 模型特征融合与分类架构
subgraph Feat["特征提取层"]
F1["语境与位置特征工程"]
F2["600 条样本微调 BERT<br/>(提取预测概率特征)"]
end
subgraph Model["集成模型"]
XGB["XGBoost 梯度提升分类器"]
end
subgraph Eval["评估指标 (20% 测试集 / 10,000 条)"]
OUT["输出分类结果<br/>(F1: 97.8%, 错误数削减 34 例)"]
end
F1 --> XGB
F2 --> XGB
XGB --> OUT
```
在 80/20 的训练/测试集切分下（测试集包含约 10,000 条未见样本），XGBoost 展现出卓越的性能，将决策树基线的准确率提升了逾 4%。引入微调后的 BERT 模型输出作为特征后，模型错误数减少了 34 个，F1 评分达到 97.8%。Sophia 指出，微调 Transformer 带来了一定的分数提升，但需要额外的 GPU 训练与推理成本，在工程实践中它更适合作为经典机器学习流水线的“补充”，而非完全替代者。
---
### [11:14](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=674s) 审计真值标签与核心经验总结
在对模型预测错误进行深度错误分析时，研究团队发现了一个令人意外的现象：在模型被判定为“假阳性”（False Positive）的错误样本中，有 **25% 实际上是模型预测正确，而弱监督自动标注的真值（Ground Truth）标签出现了错误**。由于自动化规则未能将某些真实自我介绍识别出来，导致基准标签本身带有噪声。团队通过人工抽样复审审计了真值质量，证实了实际模型表现优于表面指标。
- **给工程与研发团队的三点核心建议：**
1. **优先尝试成本低廉的方案（Try the cheaper thing first）**：仅耗费相当于一个下午的人工标注成本，借助经典 ML 算法即可解决绝大部分问题，无需一开始就盲目部署昂贵的大语言模型与重型 GPU 集群。
2. **全面审计你的答案基准（Audit your answer key）**：在使用弱监督或自动化标注构建数据集时，必须抽样进行人工审核，防止数据噪声掩盖模型的真实能力。
3. **充分利用已有标签线索（Find the labels you already have）**：在会议记录、附注等现有元数据中寻找结构化信息，发掘创新的自监督或弱监督切入点。
演说最后，Sophia 感谢了导师 Foaad Khosmood 博士以及赞助该项目的加州理工州立大学先进技术与公共政策研究所（Cal Poly Institute for Advanced Technology and Public Policy），并邀请对会议与听证会档案智能化处理感兴趣的从业者进一步交流合作。
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) 为匿名标签赋予真实姓名仍然是整个转录流水线中最耗时的一步。 — Sophia Dimitrijevic
> [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) 数据本身包含自我标注能力。公众发言人通常会在作证的第一句话中进行自我介绍。 — Sophia Dimitrijevic
> [00:04](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=4s) 简单的关键词匹配极易失效：例如短语 'I am against this bill' 虽包含 'I am'，但并非自我介绍；而 'Good morning, Glenn Brekes for the Drug Policy Alliance' 没有出现 'My name is'，却是货真价实的自我介绍。 — Sophia Dimitrijevic
> [04:13](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=253s) 利用转录本中已知的事后发言人身份，结合现成的自然语言处理工具包 spaCy 进行弱监督标注，以零人工标注成本为 50,000 条样本完成了自动标注。 — Sophia Dimitrijevic
> [06:45](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=405s) 微调 Transformer 带来了一定的分数提升，但需要额外的 GPU 训练与推理成本，在工程实践中它更适合作为经典机器学习流水线的'补充'，而非完全替代者。 — Sophia Dimitrijevic
> [11:14](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=674s) 在模型被判定为'假阳性'的错误样本中，有 25% 实际上是模型预测正确，而弱监督自动标注的真值（Ground Truth）标签出现了错误。 — Sophia Dimitrijevic
> [11:14](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=674s) 优先尝试成本低廉的方案（Try the cheaper thing first）：仅耗费相当于一个下午的人工标注成本，借助经典 ML 算法即可解决绝大部分问题。 — Sophia Dimitrijevic
> [11:14](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=674s) 全面审计你的答案基准（Audit your answer key）：在使用弱监督或自动化标注构建数据集时，必须抽样进行人工审核，防止数据噪声掩盖模型的真实能力。 — Sophia Dimitrijevic
> [11:14](https://www.youtube.com/watch?v=T5l2YvqPQX0&t=674s) 充分利用已有标签线索（Find the labels you already have）：在会议记录、附注等现有元数据中寻找结构化信息，发掘创新的自监督或弱监督切入点。 — Sophia Dimitrijevic
## Mentioned
Sofija Dimitrijevic, Foaad Khosmood, Paula Vidal, Casey Liu, Glenn Brekes, 加州理工州立大学 (California Polytechnic State University), 数字民主项目 (Digital Democracy Project), 先进技术与公共政策研究所 (Institute for Advanced Technology and Public Policy), Drug Policy Alliance, Ai4, spaCy, BERT, XGBoost, Random Forest, Decision Tree
## Video description
Sofija Dimitrijevic, Student, California Polytechnic State University
What if we could identify unknown speakers in government hearings by only using the words they say? As local journalism declines and reporters increasingly rely on incomplete transcripts, automatically matching legislative testimony to the correct speaker has become an expensive and critical problem. Yet, the answer is usually sitting in the speaker's very first sentence, when they introduce themselves. This talk will walk through a lightweight machine learning pipeline that detects these self-introductions and extracts the speaker's name from 1.54 million utterances across five state legislative sessions, comparing decision trees, random forests, XGBoost, and a fine-tuned BERT classifier. Attendees will learn how careful feature engineering can rival transformer models while remaining interpretable and compute-efficient, and how combining the two approaches yields the strongest results. The talk will also share a surprising finding: some of our model's "errors" were actually correct, revealing hidden noise in the source data itself. For attendees interested in machine learning, natural language processing, civic technology, and building practical systems that make institutions more transparent, this talk will show how a resource-conscious solution can tackle a deceptively difficult problem.
[Source](https://www.youtube.com/watch?v=T5l2YvqPQX0)
Source quotes
[0:04] Sophia Dimitrijevic: 为匿名标签赋予真实姓名仍然是整个转录流水线中最耗时的一步。
[0:04] Sophia Dimitrijevic: 数据本身包含自我标注能力。公众发言人通常会在作证的第一句话中进行自我介绍。
[0:04] Sophia Dimitrijevic: 简单的关键词匹配极易失效：例如短语 'I am against this bill' 虽包含 'I am'，但并非自我介绍；而 'Good morning, Glenn Brekes for the Drug Policy Alliance' 没有出现 'My name is'，却是货真价实的自我介绍。
[4:13] Sophia Dimitrijevic: 利用转录本中已知的事后发言人身份，结合现成的自然语言处理工具包 spaCy 进行弱监督标注，以零人工标注成本为 50,000 条样本完成了自动标注。
[6:45] Sophia Dimitrijevic: 微调 Transformer 带来了一定的分数提升，但需要额外的 GPU 训练与推理成本，在工程实践中它更适合作为经典机器学习流水线的'补充'，而非完全替代者。
[11:14] Sophia Dimitrijevic: 在模型被判定为'假阳性'的错误样本中，有 25% 实际上是模型预测正确，而弱监督自动标注的真值（Ground Truth）标签出现了错误。
[11:14] Sophia Dimitrijevic: 优先尝试成本低廉的方案（Try the cheaper thing first）：仅耗费相当于一个下午的人工标注成本，借助经典 ML 算法即可解决绝大部分问题。
[11:14] Sophia Dimitrijevic: 全面审计你的答案基准（Audit your answer key）：在使用弱监督或自动化标注构建数据集时，必须抽样进行人工审核，防止数据噪声掩盖模型的真实能力。
[11:14] Sophia Dimitrijevic: 充分利用已有标签线索（Find the labels you already have）：在会议记录、附注等现有元数据中寻找结构化信息，发掘创新的自监督或弱监督切入点。
Source quotes
[0:04] Sofija Dimitrijevic: the data can label itself
[0:04] My name is...
[0:04] I'm here on behalf of...
[2:22] Hello and thank you for having me here today. My name is Kendra Scott...
[2:22] I am against this bill.
[2:22] Good morning, Glenn Brekes for the Drug Policy Alliance
[12:29] Sofija Dimitrijevic: Try the cheaper thing first
[12:29] Sofija Dimitrijevic: Audit your answer key
[12:29] Sofija Dimitrijevic: Find the labels that you already have
Original transcript
[0:04] Hi everyone. My name is Sophia Dimitrievich. I
work with Dr. Vlad Kackmood on Cal Poly's Digital
[0:11] Democracy Project. And so, my presentation
is called "Who Just Testified?" And our paper
[0:17] was detecting self-introductions in legislative
testimony. I just want to give a big shout-out to
[0:23] Paula Vidal and Casey Liu for also their for
their input on this project and their work.
[0:30] They're our peers, so yeah. Um Yeah, cool. Um
Yeah, every organization that records meetings
[0:38] has the same problem that we do. You get a
transcript, but you don't get names. And so,
[0:43] we solved it for roughly the cost of one afternoon
of human labeling, and that's what I'm going to
[0:49] show you. But yeah, so in the past two decades,
local and state political reporting has seen a lot
[0:58] less funding, and the Digital Democracy Project
exists to fill that gap with AI-assisted coverage
[1:04] of hearings. And so, the speech-to-text part is
essentially solved, and so is speaker diarization,
[1:12] meaning that the split of audio into, you
know, speaker one, speaker two, speaker three.
[1:17] But what is not solved is putting a real name on
speaker one or speaker two. And so, prior work at
[1:23] Cal Poly found this to be the most time-consuming
step in the whole transcription pipeline,
[1:29] even with the audio and the video available. And
so, known speakers are tractable. So, legislators
[1:36] have voice prints, face records, and nameplates.
And the people that we care about, which are the
[1:42] members of the public giving testimony, have none
of that. And so, the key insight here is in this
[1:50] setting, the data can label itself. So, unknown
speakers introduce themselves, um, actually,
[1:56] and it happens usually in their first sentence.
So, they say, "My name is." or "I'm here on
[2:01] behalf of." And if you can reliably spot that one
utterance, um, when someone says their own name,
[2:08] um, the diarization propagates that
name across the entire transcript,
[2:12] um, and so everything you say later.
And so, yeah, the problem that we're
[2:16] trying to solve is can a machine catch
that one line and get the name right.
[2:22] And so, this is just an example of, um, you
know, in the California legislator what, um,
[2:28] journalists can see like in the video. Um, so,
we have the bill number, the discussion number,
[2:34] as well as the legislator associated with it, and
then maybe some information regarding, um, the
[2:40] bill. Um, but the person testifying, um, a member
from the public, um, there's no information,
[2:47] um, regarding them other than, you know, the face
shown on the screen. And so, this is just to show
[2:53] like what, um, journalists are working with. And
so, I just want to take, um, a quick example of,
[3:00] um, a few different, um, cases. Um, so, try
catching like which of these is a name. So,
[3:06] um, the first two are genuine self-introductions
from California hearings. The third is what our
[3:11] first attempt, which was a keyword search for
phrases like "My name is." and "I am." confidently
[3:16] flagged as someone introducing themselves.
So, the easy case is line one. Um, we have
[3:23] Uh, what does it say? "Hello and thank you for
having me here today. My name is Kendra Scott
[3:28] Kendra Scott, and I've been here for 10 years."
So, that is a very, um, obvious self-introduction.
[3:33] The second one is "Good morning, Glenn Brekes, um,
for the Drug Policy Alliance." Um, this is also a
[3:41] self-introduction, and, um, this is a uh more a
harder case to catch. And then, the third one,
[3:47] um, is "I uh am against this bill." So, this
is very obviously not a self-introduction.
[3:53] So, you were able to do this, you know, within,
you know, 2 seconds of like looking at it.
[3:58] But now try, you know, having a machine do that
1.5 million times, which is the size of our
[4:05] data set. But yeah, just wanted to set the scene
here. And so, for our method, um first one,
[4:13] first step is to narrow down our data set. So,
we start with 1.4 million utterances across five
[4:19] state legislative sessions. So, we have California
2015 to 2016, 20 also California 2017 to 18,
[4:27] and then Texas, Florida, and New York all from
2015 to 16. And this is all available online
[4:33] publicly in our hugging face data set. And so,
then we're going to drop anyone on the legislative
[4:41] or committee rosters. So, that's around 700
legislators, leaving about 520,000. So, we
[4:49] basically cut it into like a third of our data
set. And so, this is because legislators, like we
[4:54] already have that information about them on file,
so we don't really need that in our training set
[4:59] for self-introductions. And then we're going to
take a stratified sample of about 100,000, so
[5:05] no one state is dominating our data set since
we see that California is two out of the five,
[5:11] um you know, let state legislative sessions. And
then we're going to keep utterances where a named
[5:18] detector can find a person. And so, that leaves
us with around 50,000 candidates for our data set.
[5:25] In our step two, we're going to label it. So,
um we already know who was actually speaking
[5:30] from the transcript. It's already in there. And
so, we just ask, does the speaker's own name
[5:39] appear in what they just said? And so,
that one rule can label all 50,000 examples
[5:46] at a zero annotation cost. And so, you know, in
supervised learning, like when you're working with
[5:53] you know, data sets, most of them are hand-labeled
or all of them are hand-labeled usually,
[5:58] but we were able to, you know, get rid of that
cost of hand-labeling by running an off-the-shelf
[6:05] space spacing model. So, this is an NLP model
that can just label it for us. And so, this
[6:11] is actually called weak supervision. And so, for
our third step, we have our features. So, this is
[6:21] everything that the model actually looks
at, like signals from the sentences. And so,
[6:27] this includes like words that
appear before and after the name,
[6:32] where in the sentence the name appeared, is
there like a name in the sentence? And so,
[6:39] these are just like some examples of features
that we look at. And then, for our fourth step,
[6:45] we classify. So, we ran three different
classification models. So, we have a decision tree
[6:51] baseline, which is a very like simple baseline
that we chose. Then, we also had a random forest
[6:58] for our second machine learning model. And
then, our third one, which is the one that
[7:02] actually produced the best outcome, was gradient
boosting. So, this is called XGBoost. And then,
[7:09] we also, this is the part that Paul and Casey
worked on. They worked on a BERT fine-tuned
[7:16] model. So, BERT is bidirectional encodings.
And so, that is kind of like the logic behind
[7:26] a lot of these like newer transformers, so just
ChatGPT and Claude. It's built on off of like,
[7:33] you know, transformers like BERT. And so,
this was kind of like a added like um,
[7:41] like step two um, you know, our our our best
um, result. Yeah. And so, we were able to get,
[7:50] you know, 1.5 for raw utterances down to 50k um,
for our data set that we worked with um, and um,
[7:58] you know, uh, run those in our model. And so,
for the pipeline, um, this is just like a more
[8:04] like detailed um, version of what I just talked
about. Um, and so, we can see every single step
[8:11] um, lined out. So, we have the left side and then
that goes into to um, that middle column right
[8:17] there. And then in the end, I just want to briefly
talk about how we evaluated this. Um, and so,
[8:24] um, in these like classical like for evaluating
classical machine learning models, there's,
[8:28] you know, overall accuracy as well as precision
and recall, which is like catching how many
[8:33] false positives and false negatives the model
um, did. So, if like a name was actually a
[8:38] self-introduction, but we didn't catch it, we want
to capture that information. As well as, you know,
[8:43] if we weren't able to catch a self-introduction.
Um, and also like an overall F1 score,
[8:49] which combines those two um, values. And we can
see on the side how the BERT fine-tuning um,
[8:56] went into our um, pipeline. So, they actually
became every pipeline every sorry, BERT um,
[9:04] you know, result became a feature in our pipe
our pipeline. And so, that I was actually able
[9:10] to increase the XGBoosted score um, you know,
I think like half a percentage point. Um,
[9:16] so, that's just something worth noting. Um, and I
also want to point out that the BERT model was um,
[9:22] trained on 600 hand-labeled um, data points.
So, Casey and Pallavi hand-labeled all of those
[9:29] utterances. And um, they had, you know, a gold
set of around 30 that they evaluated on. And so,
[9:37] um, they didn't do any of that automatic labeling
Um, in their part of the project that we did.
[9:42] And so, for our results, what we found is that
97.6% of an F1 overall accuracy for the XGBoost
[9:51] on, you know, we we split our data into 80%
training and 20% test. So, on that 20% test
[9:58] that the model had never seen before, so
around 10,000 utterances, it performed,
[10:03] you know, 97.6% accuracy. And so, you know,
the decision tree also did pretty well like our
[10:11] baseline of 93%. But, we were able to increase the
accuracy by 4% using boosted trees. And so, yeah,
[10:20] most of that gain comes from the Discord context
features plus that boosting and not from anything,
[10:25] you know, exotic or or I guess like that we
wouldn't expect. And so, the BERT result,
[10:33] when we added that fine-tuned transformer, it
moved our F1 from 97.4% to around, or sorry, 97.5%
[10:42] to around 97.8. And it reduced our total errors
down from, I think, 241 to 207. So, 34 errors like
[10:51] got cut out. So, you know, that's just I
guess like a fraction of a point exchange for
[10:58] you know, a training pipeline and a GPU for,
you know, training these bigger models. Um
[11:03] So, that is like an extra inference cost that
you wouldn't have to add on. Um but yeah,
[11:09] the transformer is definitely like a complement
and not or like a replacement for our pipeline.
[11:14] Um and so, yeah, our entire human budget was
those 600 hand-labeled um hand-labeled utterances.
[11:22] And so, everything else came from that
name-matching heuristic. And you know,
[11:27] the most interesting result for this paper is that
actually out of all of our, you know falsely false
[11:33] positives that like you know errors that the
model didn't catch 25% of them were actually
[11:40] you know flagged the model flagged them
the model was correct in the end but our
[11:47] labeler was actually wrong so we labeled the
data as not being an introduction even though
[11:52] it was and so our overall accuracy you know
we accounted for that by you know looking
[12:00] taking like a sample and labeling it ourselves to
see like around how what percentage of our data
[12:06] might be mislabeled and so that's like I guess
the caveat that comes with that automatic
[12:12] labeling like you're going to have way less human
cost but in the end your accuracy might not be
[12:18] as like like it might not reflect the accuracy
as um as well as if we had hand labeled it um
[12:29] but you know just what to take back to
your team I think the first thing is
[12:34] you know try the cheaper thing first so you
know running these three classical machine
[12:39] learning models you know it was you know the
cost of like one afternoon of labeling um
[12:45] we didn't like have that like expensive like
GPUs that we would have had to use for you know
[12:52] those bigger transformers like chat GPT and Claude
another one for sure is audit your answer key so
[13:00] you know that's what we did in the end like we
we ran like a full audit on seeing how well how
[13:05] correct was our ground truth since we had like an
auto labeler run it um and then finally yeah for
[13:12] that like find the labels that you already have so
maybe like in meeting transcripts like someone's
[13:17] name someone wrote like notes about like a naming
written down so like really like look look broadly
[13:22] for and try to be creative with your solution but
yeah if you have any like need for like meeting or
[13:31] like you know hearing archives or want to talk
about us like come find us. Like we'll be around.
[13:36] But, yeah. I just want to acknowledge
the team again. So, Dr. Floyd Cashmood,
[13:40] our advisor, and then Paul Dawson Casey as
well. And I also want to shout out the Cal
[13:45] Poly Institute for Advanced Technology and Public
Policy you know for all of their work that they
[13:51] do and you know sponsoring this project. Yeah. And
yeah. Contact us. So, we have our emails up there.
[13:59] Paul and Casey and I we just recently graduated
from Cal Poly. So, we'll have access to these
[14:04] emails for about another I think like 3 months.
But, yeah. Definitely reach out to Dr. Cashmood.
[14:10] That's the email on the left if you're if you're
interested. But, yeah. Thank you. [applause]
