Cal Poly Parser Beats Neural Networks on Legislative Transcripts

Panel 4 - 2: Discourse Analysis of Legislative Bill Discussions — Ai4

Ai4

Constraining classification search spaces with procedural grammar allows simple logistic regression to outperform complex neural networks across legislative debates.

AI-assisted summary of Panel 4 - 2: Discourse Analysis of Legislative Bill Discussions — original source. Check the original for full context.

Source published: 2026-08-18

Detailed notes

## Background Published on YouTube by the channel Ai4 on August 18, 2026, this talk is presented by Isaac Rudnick, an alumnus from California Polytechnic State University, San Luis Obispo. According to the page description, it covers an end-to-end pipeline that recovers the structured lifecycle of legislative committee hearings using a formal grammar to constrain classification across over 7,000 California hearings.

## TL;DR The presentation introduces a discourse parser designed to extract the structured rhetorical lifecycle of legislative committee hearings, from chair introductions and expert testimony to public commentary and final votes. Leveraging transcripts from the Digital Democracy project and adapting semantic ontologies from the UK Parliament, the team used role-based transition rules to prune the classification search space. A simple logistic regression model with TF-IDF features outperformed a multilayer perceptron across most section categories, successfully constructing complete parse trees for 55% of hearings. Rudnick emphasizes key engineering insights, highlighting the utility of LLMs for initial annotation bootstrapping while advocating for disciplined data modeling before reaching for complex neural architectures.

## Atmosphere & style The presentation maintains an engaging, pragmatic, and intellectually honest academic tone grounded in real-world data science. Rudnick delivers the findings with technical clarity and humility, candidly breaking down edge-case failures, unscripted transcript noise, and the surprising victory of simple linear models over deep learning. Watching the talk feels like a reflective engineering retrospective that cuts through AI hype to focus on domain modeling, procedural realities, and sensible system architecture.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=4s) Legislative committee hearings follow a structured lifecycle—from bill introduction and expert testimony to public comment and voting—that can be modeled as a formal discourse grammar. - [02:23](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=143s) To address the lack of searchable public records, researchers utilized over 7,000 California committee hearing transcripts from the Digital Democracy project (2015–2018) as a training corpus. - [04:09](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=249s) Procedural debate rules enforce strict constraints on 'who speaks and when,' which the pipeline uses to determine valid discourse phases. - [04:09](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=249s) The research adapted the UK Parliament's 2016 Oral Contributions ontology to define role-based speaker attributes and transition rules for legislative debate. - [06:29](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=389s) Grammar-based masking prunes invalid class labels upfront and resolves missing or ambiguous speaker identities in noisy automated transcripts. - [06:29](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=389s) A simple TF-IDF and logistic regression model outperformed a multilayer perceptron neural network on 7 of 8 section types, achieving a 52% F1 score versus 40% accuracy. - [06:29](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=389s) The handcrafted grammar rules alone accounted for the full structural parse tree in 55% of the analyzed legislative hearings. - [09:33](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=573s) The primary ceiling on classification accuracy was annotation ambiguity, especially transitional interjections by chairs and unscripted floor clashes during public comment periods. - [09:33](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=573s) The pipeline incorporates a graceful fallback mechanism that assigns the most probable valid transition rules per section when a full-sequence parse tree cannot be built. - [11:47](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=707s) LLMs were effective for early-stage exploratory bootstrapping, though annotation accuracy varied substantially between models (GPT-3.5 at ~50% vs. Claude at ~70%). - [11:47](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=707s) System architecture and data modeling must precede LLM-assisted code generation; generating code before formalizing schemas produced fragile implementations. - [11:47](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=707s) Pragmatic NLP engineering requires starting with traditional baselines, cleaning data, and leveraging lightweight models before adopting complex neural architectures.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=4s) Anatomy of a Legislative Committee Hearing

In democratic governance, a proposed bill cannot become law without navigating a rigorous committee hearing process where lawmakers, experts, and the public debate its merits and flaws. Isaac Rudnick, presenting collaborative work with Nipun Das and Dr. Foaad Khosmood from California Polytechnic State University, San Luis Obispo, details the creation of a committee hearing discourse parser designed to extract and analyze the rhetorical structure of these proceedings. Committee hearings follow a structured lifecycle: the presiding chair opens the session and introduces the bill's author or designated presenter; the presenter delivers an introductory summary of the proposed legislation; invited experts present formal testimony; members of the public voice their stances; and the committee concludes with closing remarks and a vote to determine whether the bill requires amendments or can advance to another committee or to the Assembly or Senate floor.

```mermaid flowchart TD accTitle: Legislative Committee Hearing Lifecycle subgraph S1["Opening & Presentation"] A["Introduction by Chair"] --> B["Bill Presentation by Author"] end subgraph S2["Testimony & Debate"] B --> C["Expert Testimony"] C --> D["Public Comments & Stances"] D --> E["Legislator Discussion"] end subgraph S3["Resolution"] E --> F["Closing Remarks & Voting"] F --> G["Floor Progression or Revision"] end ```

- **Hearing Stages**: Bill introduction, formal bill presentation, expert witness testimony, public commentary, legislative deliberation, and closing votes. - **Outcomes**: Deciding whether legislation progresses to floor debate or requires substantial redrafting.

---

[02:23](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=143s) Data Scarcity and the Digital Democracy Dataset

Despite legal expectations—such as ADA accessibility guidelines requiring governments to record and maintain public deliberations—many state and local bodies fail to preserve searchable transcripts of legislative debate. To bridge this transparency gap, the researchers leveraged data from the Digital Democracy project, which compiled legislative committee transcripts across states such as California, Texas, Florida, and Hawaii between 2015 and 2018. Because California offered continuous and complete transcript coverage across all four years, this corpus of over 7,000 hearings formed the primary training and evaluation foundation for the parser pipeline.

The study centered on two fundamental research questions: - Can committee transcripts be standardized to facilitate systematic comparisons, evaluate legislative work, and hold elected officials accountable? - Can researchers derive a formal discourse grammar for legislative debate analogous to natural language parts of speech (nouns, verbs, adjectives, and determiners)?

---

[04:09](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=249s) Speaker Roles and Adapting Ontological Frameworks

The core intuition behind the discourse grammar is establishing "who speaks and when." Parliamentary proceedings enforce strict procedural rules governing speaking turns. During expert testimony, dialogue is predominantly limited to recognized witnesses and the presiding committee chair, with rare interruptions from committee members or the committee secretary resolving procedural confusion. Conversely, during roll call votes or procedural motions, the committee secretary speaks to establish a quorum and record votes.

Because these procedural patterns were not formally digitized into a machine-readable schema, the researchers drew inspiration from external semantic data models. Specifically, they adapted an ontology developed by the UK Parliament in 2016 designed to model oral contributions in legislative debates. This adaptation allowed the team to formalize role-based speaker attributes and define the strict operational boundaries of each hearing section.

```mermaid flowchart LR accTitle: UK Parliament Ontology to Discourse Grammar Mapping subgraph Source["External Semantic Models"] UK["UK Parliament Oral Contributions Ontology (2016)"] end subgraph Adapt["Pipeline Rules"] SR["Speaker Role Constraints"] TR["Section Transition Rules"] end subgraph Target["Grammar Parser"] G["Constrained State Classification"] end UK --> SR & TR SR & TR --> G ```

---

[06:29](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=389s) Grammar Constraints and Model Performance

By enforcing structural grammar constraints, the pipeline significantly prunes the classification search space for any given utterance. Because legislative rules prohibit a member of the public from presenting a bill or a committee member from testifying as an expert witness, the parser eliminates invalid class labels upfront. This grammar-based filtering also resolves ambiguous or missing speaker identities in noisy automated transcripts by narrowing down permissible roles based on the active discourse phase.

```mermaid flowchart TD accTitle: Hearing Discourse Classification Pipeline A["Raw Hearing Utterance"] --> B["TF-IDF Feature Extraction"] B --> C["Grammar Role & Section Masking"] C --> D{"Classification Model"} D -->|Simpler & Faster| E["Logistic Regression (52% F1)"] D -->|Overfitting / Complex| F["Multilayer Perceptron (40% Acc)"] E --> G["Full Parse Tree Generation (55% of Hearings)"] ```

When comparing predictive architectures, a simple logistic regression model applied to the restricted label set surprisingly outperformed a complex multi-layer perceptron (MLP) neural network across 7 out of 8 section types:

---

[09:33](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=573s) Dataset Annotation Challenges and Discourse Ambiguity

The performance ceiling was primarily constrained by dataset annotation noise rather than model capacity. Initial utterance labels were bootstrapped using large language models alongside extracted feature sets, followed by manual review and label correction. However, genuine semantic ambiguities in the raw transcripts created persistent edge cases that confounded both automated classifiers and human annotators.

A major source of confusion stemmed from transitional utterances—such as brief interjections by the presiding chair or committee secretary between expert testimony and public comments. Similarly, unscripted floor clashes introduced severe noise, particularly when legislators engaged in heated arguments during scheduled public comment intervals. In instances where the grammar parser failed to construct a valid full-sequence parse tree, the system gracefully fell back to assigning the most probable valid transition rules for each individual section.

---

[11:47](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=707s) Future Directions and LLM Engineering Takeaways

Reflecting on feature engineering and language model utility, the team identified several avenues for future research, including upgrading from basic term-frequency inverse document frequency (TF-IDF) embeddings to dense semantic representations, incorporating formal procedural dictionaries like *Robert's Rules of Order* to capture standardized motions, and implementing fine-grained stance classification to track whether lawmakers are actively debating or aligning on specific policy points.

The project provided clear methodological lessons on the role of LLMs in structured NLP workflows: - **LLM Exploration**: LLMs are exceptionally useful for early-stage brainstorming, exploration, and bootstrapping initial training annotations, though annotation accuracy varied substantially between models (GPT-3.5 attained ~50% accuracy, while Claude attained ~70%). - **Architecture Precedes Code Generation**: Generating pipeline components with LLMs without an established, solid data model produced fragile, messy code; once the architecture and ontologies were finalized, LLM-assisted code generation accelerated development. - **Pragmatic Modeling Order**: Teams should "start with traditional methods" such as TF-IDF and logistic regression, clean and understand their underlying data, and leverage LLMs to bootstrap lightweight models before blindly adopting complex neural networks.

Isaac concluded by inviting collaborations on semi-structured natural language datasets and acknowledging contributors Sophia, Pallavi, and Casey for their foundational research on classifying speaker self-introductions and latent corporate features.

## Notable quotes > Committee hearings, board meetings, earnings calls, and claims reviews all follow a kind of script: an opening, a presentation, expert input, public back-and-forth, a decision. — Isaac Rudnick

> Most transcript parsing tools ignore this structure even though it’s what you actually want. — Isaac Rudnick

> This information is especially important for analyzing the work and power relationships within democratic bodies, so we built an end-to-end pipeline that recovers it. — Isaac Rudnick

> The grammar alone accounts for the full structure of 55% of hearings and plain logistic regression outperformed a shallow MLP model on 7 of 8 section types with a fraction of the complexity. — Isaac Rudnick

> In this talk, I'll walk through what worked, what broke, and how to decide when a handmade model beats a neural one. — Isaac Rudnick

> [00:04](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=4s) In democratic governance, a proposed bill cannot become law without navigating a rigorous committee hearing process where lawmakers, experts, and the public debate its merits and flaws. — Isaac Rudnick

> [02:23](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=143s) Can committee transcripts be standardized to facilitate systematic comparisons, evaluate legislative work, and hold elected officials accountable? — Isaac Rudnick

> [02:23](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=143s) Can researchers derive a formal discourse grammar for legislative debate analogous to natural language parts of speech (nouns, verbs, adjectives, and determiners)? — Isaac Rudnick

> [04:09](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=249s) The core intuition behind the discourse grammar is establishing "who speaks and when." — Isaac Rudnick

> [11:47](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=707s) start with traditional methods — Isaac Rudnick

## Mentioned Isaac Rudnick, Nipun Das, Dr. Foaad Khosmood, California Polytechnic State University, San Luis Obispo, Ai4, Digital Democracy project, UK Parliament, Robert's Rules of Order, GPT-3.5, Claude, TF-IDF, Multilayer Perceptron (MLP), Logistic Regression, Americans with Disabilities Act (ADA), Sophia, Pallavi, Casey

## Video description Isaac Rudnick, Alumnus, Department of Computer Science and Software Engineering, California Polytechnic State University, San Luis Obispo

Committee hearings, board meetings, earnings calls, and claims reviews all follow a kind of script: an opening, a presentation, expert input, public back-and-forth, a decision. Most transcript parsing tools ignore this structure even though it’s what you actually want. This information is especially important for analyzing the work and power relationships within democratic bodies, so we built an end-to-end pipeline that recovers it.

Working with over 7,000 California legislative hearings, we wrote a formal grammar describing the order these stages can occur in and used it to constrain what a classifier is allowed to predict at each point in the discussion. The grammar alone accounts for the full structure of 55% of hearings and plain logistic regression outperformed a shallow MLP model on 7 of 8 section types with a fraction of the complexity.

In this talk, I'll walk through what worked, what broke, and how to decide when a handmade model beats a neural one.

[Source](https://www.youtube.com/watch?v=BRxOWFs2IC0)

中文简报

Cal Poly立法文本解析器表现超越神经网络通过程序性语法约束分类搜索空间，简单的逻辑回归模型在立法辩论分析中的表现超越了复杂的神经网络。该演讲介绍了一种话语解析器，旨在提取立法委员会听证会的结构化修辞生命周期，涵盖从主席介绍、专家证词到公众意见和最终表决的全过程。团队利用“数字民主”（Digital Democracy）项目的转录文本并借鉴英国议会的语义本体，采用基于角色的转换规则来修剪分类搜索空间。结合 TF-IDF 特征的简单逻辑回归模型在大多数环节类别上的表现均优于多层感知机，成功为 55% 的听证会构建了完整的解析树。Rudnick 强调了关键的工程见解，突出了大语言模型在初始标注引导中的实用价值，同时倡导在采用复杂的神经网络架构之前，先进行严谨规范的数据建模。加州理工州立大学研究团队利用 Digital Democracy 项目中超过 7,000 场加州立法听证会转录文本，构建出立法辩论话语分析解析器，用于建模法案从引入到表决的全流程。 (0:04) 该研究借鉴英国议会 2016 年的口头发言本体论建立话语语法，严格约束发言人角色与辩论阶段转换，仅凭语法规则就成功解析了 55% 听证会的完整结构树。 (4:09) 基于语法掩码限制标签后，简单的 TF-IDF 与逻辑回归模型在 8 类发言阶段中的 7 类上优于浅层 MLP 神经网络，取得了 52% 的 F1 分数（MLP 准确率为 40%）。 (6:29) 分类性能的主要瓶颈来自转录文本中的过渡性插话及公众评论期的即兴交锋等语义歧义；团队建议在构建结构化 NLP 流水线时应先明确数据架构，并优先采用传统模型。 (9:33)

Reading summary

Researchers from California Polytechnic State University developed a legislative discourse parser that applies formal procedural rules to standardize and analyze committee hearing transcripts. Presenting the research, Isaac Rudnick demonstrated that constraining classification through institutional discourse grammar allows a lightweight classical model to outperform complex neural architectures. Formalizing Parliamentary Rules into Discourse Grammars (4:09) State and local legislative bodies frequently fail to maintain searchable debate transcripts despite legal accessibility mandates. To bridge this transparency gap, the researchers leveraged over 7,000 California committee transcripts recorded between 2015 and 2018 by the Digital Democracy project. Seeking to standardize these records for systematic comparison, the team designed a discourse grammar analogous to natural language parts of speech, mapping chronological hearing stages from bill introductions and expert testimony to public commentary and final voting. To formalize speaking turns, the team adapted a 2016 UK Parliament ontology on oral contributions. Because parliamentary procedure strictly defines speaking privileges—such as restricting testimony to invited witnesses and limiting roll call interactions to the committee secretary—the grammar establishes hard boundaries for permissible speakers within each section. Imposing these structural rules prunes invalid section labels and resolves ambiguous or omitted speaker identities in noisy automated transcripts. Performance Gains from Constrained Classical Models (6:29) By applying grammar-based role and section masking to TF-IDF feature extractions, the pipeline significantly reduces the label search space. Experimental comparisons revealed that a simpler logistic regression model outperformed a multilayer perceptron neural network across seven out of eight hearing section types. The logistic regression architecture attained a 52% F1 score, compared to approximately 40% accuracy for the overparameterized neural network and a 12% dominant-class baseline. Across the California corpus, the constrained pipeline generated a fully valid, grammatically compliant parse tree for 55% of the hearings. In cases where the parser could not generate an unbroken sequence tree, the system deployed a fallback mechanism that assigned the most probable valid transition rules to individual sections. Transcript Ambiguity and Procedural Edge Cases (9:33) The researchers observed that performance limitations were driven primarily by dataset annotation noise and conversational ambiguity rather than model capacity. Although initial training annotations were bootstrapped using large language models and refined by human annotators, genuine transcript anomalies created classification bottlenecks. Frequent interjections by presiding chairs or committee secretaries between witness statements and public testimony blurred section boundaries. Additionally, unscripted floor clashes introduced severe noise, particularly when legislators engaged in spontaneous debates during time reserved for public comment. These disruptions violated expected sequential patterns and forced the system into fallback transition assignments. Methodological Lessons for LLM-Assisted Pipelines (11:47) The project yielded practical methodological insights regarding the use of large language models in structured NLP tasks. While LLMs proved effective for exploratory work and bootstrapping training labels, baseline annotation accuracy varied significantly between systems, with GPT-3.5 achieving around 50% accuracy compared to approximately 70% for Claude. Rudnick also noted that prompting LLMs to write code produced fragile results until the data ontologies and system architecture were strictly finalized. The researchers concluded that workflows should begin with classical approaches like TF-IDF and logistic regression on clean data before adopting complex neural models. Planned improvements include incorporating dense semantic representations, integrating formal procedural dictionaries like Robert's Rules of Order, and training fine-grained stance classifiers to detect whether lawmakers are aligning or clashing on policy issues. While conversational interjections and procedural anomalies limit complete parse tree generation to 55% of hearings, the study confirms that codifying parliamentary constraints into structural grammars substantially improves automated parsing without requiring heavy neural networks.

精炼阅读摘要

加州理工州立大学圣路易斯奥比斯波分校研究团队针对立法委员会听证会提出了一套话语语法解析管线，通过将议会程序规则与本体模型相结合，有效解决了听证文本的结构解析难题，并证明传统轻量模型在受限分类任务中显著优于复杂神经网络。立法听证生命周期与数字民主数据集 (0:04) 在代议制民主中，立法委员会听证会是法案成为法律的核心审议环节，其流程具有高度程序化特征：由主席开场并介绍提案人，提案人进行法案概述，随后进入受邀专家证词陈述、公众发表支持或反对意见，最后由委员会成员展开辩论、发表总结陈词并投票表决。该流程直接决定法案是被修订推进，还是止步于委员会阶段。尽管《美国残疾人法案》（ADA）等法规要求政府记录并维护公共审议内容，但多数州和地方机构长期缺乏具备可检索性的听证文本。为突破数据透明度瓶颈，Isaac Rudnick及其合作者利用了“数字民主”（Digital Democracy）项目在2015年至2018年间收集的加利福尼亚州、得克萨斯州、佛罗里达州和夏威夷州委员会转录数据。其中加州拥有跨越完整四年的连贯记录，研究团队以其包含的7,000多场听证会语料为核心训练与评测基准，旨在验证能否将非结构化转录本标准化，并构建类似于自然语言词性（名词、动词等）的立法辩论形式化话语语法。引入英国议会本体构建话语语法约束 (4:09) 立法话语语法的核心逻辑在于规范“何人在何时发言”。议会程序对发言轮次设定了刚性限制，例如在专家作证阶段，对话主要集中于证人与主席之间，委员会秘书仅在处理程序性混乱或点名表决时发声。由于现存听证文本缺乏机器可读的结构化模式，团队借鉴了英国议会于2016年制定的口头辩论贡献本体模型（UK Parliament Oral Contributions Ontology），以此形式化定义参会者的角色属性与辩论环节的运作边界。该研究将议会规则转化为分类管道中的先验约束规则。例如，系统在规则层面直接禁止将普通公众识别为提案陈述者，或禁止委员会成员作为专家证人作证。这种语法掩码机制不仅前置剔除了非法的类别标签，大幅压缩了分类搜索空间，还能在自动语音识别转录存在高噪声或发言人身份丢失的情况下，依据当前话语阶段快速推断并补全合理的角色属性。语法剪枝赋能与轻量分类模型优势 (6:29) 在特征工程与建模方面，研究团队基于词频-逆文档频率（TF-IDF）提取特征，并结合语法约束对发言进行环节分类。评测结果表明，在施加状态转移约束后，结构简单的逻辑回归模型展现出意想不到的稳健性。在8种听证环节类别中，逻辑回归模型在其中7种上的表现均优于复杂的多层感知机（MLP）神经网络。具体实验数据显示：简单分配最高频类别的基线准确率仅约为12%；多层感知机因在稀疏类别上面临过度参数化问题，准确率仅为40%左右；而采用语法过滤的逻辑回归模型取得了52%的F1分数，较基准分类性能提升逾四倍。在整场听证会的宏观序列重建中，该系统能够为全州55%的听证会生成完全符合语法规则且逻辑闭环的完整解析树（Parse Tree）。话语转折歧义与解析器容错回退机制 (9:33) 研究分析指出，制约当前解析性能进一步提升的主要瓶颈在于数据集的标注噪声与真实政治互动的语义模糊性，而非模型本身的参数容量。团队最初采用大语言模型结合特征集进行冷启动弱标注，并辅以人工复核。然而，听证会中频发的边缘情况给模型与人类标注员均带来了显著干扰。转折性话语构成了最主要的混淆来源，典型表现为主席或秘书在专家证词与公众意见过渡期间穿插的简短程序性插话。此外，议员在指定公众评论时段爆发的即兴激烈争论，严重打破了预设的辩论秩序。针对这类无法构建完整全局语法树的极端场景，解析管线设计了容错机制，自动回退到为各个独立区间分配最可能有效转移规则的局部最优解。 LLM辅助开发启示与后续优化方向 (11:47) 回顾技术路线，团队总结了将大语言模型集成至垂直自然语言处理（NLP）工作流的三项工程实践经验：首先，LLM极适合用于项目早期的方案探索与初始样本自举标注，但标注质量差异巨大（GPT-3.5初始准确率约50%，而Claude可达约70%）；其次，系统架构与底层数据模型必须先于代码生成确立，脱离本体模型直接让LLM编写管线极易产生脆弱混乱的代码；最后，工程落地应坚持务实顺序，优先使用TF-IDF与逻辑回归等传统基准清洗并吃透数据，再借助LLM蒸馏轻量模型，而非盲目上线复杂深度网络。针对后续演进，团队计划将稀疏的TF-IDF特征升级为稠密语义向量，引入《罗伯特议事规则》（Robert's Rules of Order）等标准程序词典以精准捕获程序性动议，并研发细粒度立场分类器以跟踪议员的政策阵营。研究最后致谢了Sophia、Pallavi与Casey等合作者在发言人自我介绍分类及潜在企业特征提取领域的基础贡献。该项研究证实了将程序本体规则注入统计分类模型对于半结构化法律文本解析的高效性。尽管目前完整解析树的覆盖率受制于非受控政治争论与标注噪声仍停留在55%，但其基于英国议会本体所确立的约束架构与自底向上的务实工程范式，为自动化立法问责与政治话语分析提供了极具参考价值的基准管线。

Detailed notes — zh

## Background 本内容由 YouTube 频道 Ai4 于 2026 年 8 月 18 日发布，主讲人为加州理工州立大学圣路易斯奥比斯波分校校友 Isaac Rudnick。视频介绍了如何通过构建形式文法与端到端解析管线，从包含 7,000 多场加州立法听证会的非结构化转录文本中还原程序化讨论结构。 ## TL;DR 研究团队针对立法委员会听证会高度程序化的特点，借鉴英国议会本体模型推导出一套形式文法，利用议事规则与发言人角色约束大幅剪枝候选分类空间。实验表明，经过文法约束的传统逻辑回归模型在 8 类章节中的 7 类上击败了浅层多层感知机（MLP）神经网络，取得了 52% 的 F1 分数，并成功为 55% 的听证会生成了完整的语法解析树。作者深入剖析了由环节过渡模糊导致的标注噪音，并给出了大语言模型冷启动标注、架构先行以及优先构建轻量传统基线的工程实践建议。 ## Atmosphere & style 整个演讲呈现出严谨、务实且极具工程实战导向的学术分享氛围。讲者条理清晰、语调沉稳，通过翔实的对比数据和坦诚的失败案例复盘，生动拆解了经典 NLP 规则系统在特定结构化场景中战胜复杂神经网络的过程。听众能真切感受到研究者在面对脏数据与模糊标注时抽丝剥茧的钻研精神，以及对大模型工程落地中“返璞归真、基线先行”理念的深刻推崇。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=4s) 加州理工州立大学团队针对立法委员会听证会开发了一套端到端话语解析器，旨在标准化转录文本以实现立法者问责与深度分析。 - [00:04](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=4s) 立法委员会听证会遵循固定的结构化流程，涵盖法案引入、法案陈述、专家证言、公众意见表达以及结案讨论与投票表决等环节。 - [04:09](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=249s) 研究借鉴英国议会的本体数据模型，基于“谁在何时发言”的议程规则为法案讨论构建了形式文法（Formal Grammar）。 - [04:09](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=249s) 形式文法利用议事规则对不可能的角色和场景进行严格剪枝，大幅缩小了分类器的候选标签空间并提升了运行效率。 - [08:03](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=483s) 在章节分类任务中，受文法约束的传统逻辑回归模型取得了 52% 的 F1 分数，性能达到主导类别基线的 4 倍以上，全面超越了浅层神经网络（MLP 的 40% 准确率）。 - [08:03](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=483s) 该形式文法解析器在无先例可循的情况下，成功为数据集中 55% 的加州立法听证会构建了完整的形式文法解析树。 - [08:03](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=483s) 解析分类失败的主要原因在于各环节间天然存在的过渡模糊地带与议员突发辩论，导致标注数据存在内在歧义而非算法缺陷。 - [11:47](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=707s) 系统后续可引入文本嵌入（Embeddings）捕捉深层语义，并结合罗伯特议事规则（Robert's Rules of Order）与立场分析模块识别动议与辩论态度。 - [11:47](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=707s) 大语言模型（LLM）非常适合用于冷启动数据标注与头脑风暴，但模型选型对初始标注质量影响极大，仍需人工校验清洗。 - [11:47](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=707s) 在系统研发中应坚持“架构先行”与“传统基线优先”原则，优先利用 TF-IDF 与逻辑回归等轻量算法构建坚实基线，而非盲目套用复杂的重型神经网络。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=4s) 立法听证会话语解析的研究背景与流程规范加州理工州立大学圣路易斯奥比斯波分校（California Polytechnic State University, San Luis Obispo）的 Isaac Rudnick 与合作者 Nepun Das 及 Foaad Khosmood 博士共同开发了一套针对立法法案委员会听证会的话语解析器（Discourse Parser）。在一项法案正式成为法律之前，必须经过各个委员会的审议，立法委员与公众可以在此表达观点、陈述立场并讨论法案的优劣。这些听证会通常遵循高度结构化的程序，包括：主持委员会主席介绍法案作者或指定陈述人的法案引入环节、法案作者或委员会成员对法案进行概述的法案陈述环节、支持方传唤专家提供知情证言的专家证言环节、公众表达立场的公众意见环节，以及最终由委员会成员决定法案是需要进一步修改修正还是提交下一委员会的结案发言与投票表决环节。全部流程结束后，法案才会送交州众议院或参议院全体会议进行投票。 ```mermaid flowchart LR accTitle: Legislative Hearing Stages accDescription: Flowchart illustrating the sequential stages of a legislative committee hearing. subgraph S1["前期准备与陈述"] A["委员会主席开场与引入 (Bill Introduction)"] --> B["法案作者/委员概述 (Bill Presentation)"] end subgraph S2["证据与公众审议"] B --> C["专家提供知情证言 (Expert Testimony)"] C --> D["公众发表立场意见 (Public Comment)"] end subgraph S3["决议与后续"] D --> E["结案讨论与委员会表决 (Closing & Voting)"] E --> F["全体院会投票 (Floor Vote)"] end ``` 尽管依据美国残疾人法案（ADA）合规要求，各级政府理应保存记录或转录文本，但许多州政府和地方政府并未完整保存讨论的转录稿。所幸在“数字民主项目”（Digital Democracy Project）的支持下，研究团队获取了加利福尼亚州、得克萨斯州、佛罗里达州以及夏威夷州在 2015 至 2018 年间的委员会听证会转录文本。其中加利福尼亚州的数据最为完整连续，研究团队便以此为基础构建了端到端的数据处理管线。该研究旨在回答两个核心问题：一是能否将法案听证会转录文本标准化，以便更好地进行对比评估并对立法者进行问责；二是能否像自然语言中存在名词、动词、形容词那样，为法案讨论推导出一套形式文法（Formal Grammar）。 ### [04:09](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=249s) 形式文法的引入与发言角色约束机制构建该文法与处理管线的核心基石在于明确“谁在何时发言”。例如在专家证言环节，发言者通常仅限于受邀专家和主持主席，极少数情况下会有委员会成员或负责维持秩序与澄清疑问的秘书介入；而在投票表决和法案引入环节，委员会秘书则因程序性需求获准发言，例如点名确认法定人数（Quorum）或处理特定动议。为了构建抽象的数据模型与本体结构（Ontologies），团队参考了英国议会（UK Parliament）在 2016 年针对类似立法审议任务制定的成熟数据模型，并将其关于立法口头发言（Oral Contributions）的本体概念适配到本研究中，从而确立了章节划分与发言人属性特征。形式文法之所以能大幅提升分类效率，关键在于它利用议事规则对候选标签空间进行了严格剪枝。参议院的议事规则决定了某些场景绝对不可能发生——例如公众成员绝不会主持法案陈述，委员会成员也不会作为受邀外部专家作证。通过这套文法规则进行约束： - **候选类别剪枝**：系统无需在每个时间点都穷举所有可能的章节分类标签，有效消除了不合规的逻辑分支。 - **角色反推与降噪**：面对未识别身份的发言人或低质量噪音数据时，文法约束能够根据当前所处的议程阶段大幅缩小发言人的可能角色范围。 - **管线运行效率提升**：剪枝机制大幅降低了模型的搜索空间与计算开销，使整体流程更加轻量、高效。 ### [08:03](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=483s) 传统模型与神经网络的性能对比及标注误差分析在模型评估阶段，研究团队对比了在缩减类别集上运行的逻辑回归（Logistic Regression）模型与多层感知机（MLP）浅层神经网络模型。令人意外的是，结构更简单的逻辑回归模型在性能上全面超越了神经网络。在章节分类任务中，逻辑回归取得了 52% 的 F1 分数，而基线模型（若盲目预测出现频率最高的主导类别）仅有约 12% 的准确率，逻辑回归实现了基线 4 倍以上的性能；相比之下，神经网络方法的章节分类准确率仅为 40%。此外，在没有任何前人先例的情况下，该文法解析器成功为数据集中 55% 的听证会生成了完整的形式文法解析树（Parse Tree）。 | 模型 / 方法 | 章节分类准确率 / F1 | 相对主导类别基线 (12%) | 说明 | |:--- |:--- |:--- |:--- | | **主导类别基线 (Majority Baseline)** | ~12% 准确率 | 1.0x | 仅指定出现频率最高的类别 | | **浅层神经网络 (MLP)** | 40% 准确率 | ~3.3x | 表现不及传统线性模型 | | **受约束逻辑回归 (Logistic Regression)** | **52% F1** | **> 4.0x** | 在 8 类章节中的 7 类上表现更优 | 分析分类失败的原因发现，绝大多数错误并非算法本身的缺陷，而是源自数据集标注中的歧义与噪音。研究团队最初使用大语言模型（如 ChatGPT）根据每段发言提取的特征生成初始标注，随后辅以人工审核纠错。然而，听证会各环节之间存在天然的过渡模糊地带，例如专家证言与公众讨论之间的过渡发言，或秘书与主持主席穿插的串场用语，很难严格划归为前后某一特定章节；此外，在公众发表证言过程中如果两名立法委员之间突然爆发辩论，极易导致公众意见环节与立法者讨论环节发生混淆。遇到无法成功生成语法树的异常情况，系统会回退至各章节最通用的默认规则集。 ### [11:47](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=707s) 特征工程拓展方向与大语言模型工程落地建议针对未来的演进方向，研究团队指出当前系统仅采用了词频-逆文档频率（TF-IDF）来比较词汇在特定章节发言与全局发言中的分布差异，后续可引入更先进的文本嵌入（Embeddings）技术来提取深层语义特征。此外，系统未来还可以深度整合罗伯特议事规则（Robert's Rules of Order），利用标准化的议事程式用语来识别动议与程序性操作，并进一步开发立场分析模块，细粒度识别议员之间是处于支持同盟还是争锋相对的辩论状态。 ```mermaid flowchart TD accTitle: Recommended Pipeline Architecture accDescription: Practical development lifecycle recommending traditional baselines before scaling to complex LLM and neural methods. subgraph Phase1["阶段一：基础与原型设计"] D1["LLM 辅助头脑风暴与研究调研"] --> D2["确立固化数据模型与系统架构"] D2 --> D3["LLM 生成初始标注与代码原型"] end subgraph Phase2["阶段二：基线与传统 NLP"] D3 --> D4["人工清洗与消除标注歧义"] D4 --> D5["基于传统方法建立坚实基线 (TF-IDF + 逻辑回归)"] end subgraph Phase3["阶段三：进阶优化"] D5 --> D6["引入高级嵌入与复杂神经模型"] end ``` 在总结大语言模型（LLM）的工程实践经验时，Isaac Rudnick 强调了以下几项核心建议： - **模型选型对数据冷启动影响巨大**：LLM 非常适合前期调研、头脑风暴以及冷启动初始标注，但标注质量受具体模型影响极深。实验中 GPT-3.5（听转录提及 GPT-5.5 / GPT-3.5）仅达到约 50% 的标注准确率，而 Claude 模型（听转录提及 Claude Fable）的标注准确率达到了约 70%。 - **架构先行于代码生成**：使用 LLM 辅助编写代码的前提是必须预先确立清晰的数据模型与架构体系；若在数据模型尚未稳固前仓促生成代码，只会导致代码混乱失控并拖慢进度。 - **坚守传统基线优先原则**：工程落地应遵循 "从传统方法起步" 的策略，优先采用 TF-IDF 与逻辑回归等简单高效的算法构建基线并深入清洗数据，而不是一遇到困难就盲目套用复杂的重型神经网络，小巧轻量的模型在文法约束下往往能发挥出意想不到的强大效能。 Isaac Rudnick 在演讲末尾表达了对 Sophia、Pallavi 和 Casey 等同学在基于潜在特征识别与分类自我介绍方面所做贡献的感谢，并向拥有非结构化或半结构化自然语言文本数据的团队发出了合作邀请。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=4s) 在一项法案正式成为法律之前，必须经过各个委员会的审议，立法委员与公众可以在此表达观点、陈述立场并讨论法案的优劣。 — Isaac Rudnick > [00:04](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=4s) 能否像自然语言中存在名词、动词、形容词那样，为法案讨论推导出一套形式文法。 — Isaac Rudnick > [04:09](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=249s) 构建该文法与处理管线的核心基石在于明确“谁在何时发言”。 — Isaac Rudnick > [04:09](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=249s) 参议院的议事规则决定了某些场景绝对不可能发生——例如公众成员绝不会主持法案陈述，委员会成员也不会作为受邀外部专家作证。 — Isaac Rudnick > [04:09](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=249s) 面对未识别身份的发言人或低质量噪音数据时，文法约束能够根据当前所处的议程阶段大幅缩小发言人的可能角色范围。 — Isaac Rudnick > [08:03](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=483s) 在没有任何前人先例的情况下，该文法解析器成功为数据集中 55% 的听证会生成了完整的形式文法解析树。 — Isaac Rudnick > [08:03](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=483s) 绝大多数错误并非算法本身的缺陷，而是源自数据集标注中的歧义与噪音。 — Isaac Rudnick > [11:47](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=707s) 使用 LLM 辅助编写代码的前提是必须预先确立清晰的数据模型与架构体系；若在数据模型尚未稳固前仓促生成代码，只会导致代码混乱失控并拖慢进度。 — Isaac Rudnick > [11:47](https://www.youtube.com/watch?v=BRxOWFs2IC0&t=707s) 优先采用 TF-IDF 与逻辑回归等简单高效的算法构建基线并深入清洗数据，而不是一遇到困难就盲目套用复杂的重型神经网络，小巧轻量的模型在文法约束下往往能发挥出意想不到的强大效能。 — Isaac Rudnick ## Mentioned Isaac Rudnick, Nepun Das, Foaad Khosmood, 加州理工州立大学圣路易斯奥比斯波分校 (California Polytechnic State University, San Luis Obispo), 数字民主项目 (Digital Democracy Project), 英国议会 (UK Parliament), ChatGPT, GPT-3.5, Claude, 罗伯特议事规则 (Robert's Rules of Order), 美国残疾人法案 (ADA), Sophia, Pallavi, Casey ## Video description Isaac Rudnick, Alumnus, Department of Computer Science and Software Engineering, California Polytechnic State University, San Luis Obispo Committee hearings, board meetings, earnings calls, and claims reviews all follow a kind of script: an opening, a presentation, expert input, public back-and-forth, a decision. Most transcript parsing tools ignore this structure even though it’s what you actually want. This information is especially important for analyzing the work and power relationships within democratic bodies, so we built an end-to-end pipeline that recovers it. Working with over 7,000 California legislative hearings, we wrote a formal grammar describing the order these stages can occur in and used it to constrain what a classifier is allowed to predict at each point in the discussion. The grammar alone accounts for the full structure of 55% of hearings and plain logistic regression outperformed a shallow MLP model on 7 of 8 section types with a fraction of the complexity. In this talk, I'll walk through what worked, what broke, and how to decide when a handmade model beats a neural one. [Source](https://www.youtube.com/watch?v=BRxOWFs2IC0)

Source quotes

[0:04] Isaac Rudnick: 在一项法案正式成为法律之前，必须经过各个委员会的审议，立法委员与公众可以在此表达观点、陈述立场并讨论法案的优劣。 [0:04] Isaac Rudnick: 能否像自然语言中存在名词、动词、形容词那样，为法案讨论推导出一套形式文法。 [4:09] Isaac Rudnick: 构建该文法与处理管线的核心基石在于明确“谁在何时发言”。 [4:09] Isaac Rudnick: 参议院的议事规则决定了某些场景绝对不可能发生——例如公众成员绝不会主持法案陈述，委员会成员也不会作为受邀外部专家作证。 [4:09] Isaac Rudnick: 面对未识别身份的发言人或低质量噪音数据时，文法约束能够根据当前所处的议程阶段大幅缩小发言人的可能角色范围。 [8:03] Isaac Rudnick: 在没有任何前人先例的情况下，该文法解析器成功为数据集中 55% 的听证会生成了完整的形式文法解析树。 [8:03] Isaac Rudnick: 绝大多数错误并非算法本身的缺陷，而是源自数据集标注中的歧义与噪音。 [11:47] Isaac Rudnick: 使用 LLM 辅助编写代码的前提是必须预先确立清晰的数据模型与架构体系；若在数据模型尚未稳固前仓促生成代码，只会导致代码混乱失控并拖慢进度。 [11:47] Isaac Rudnick: 优先采用 TF-IDF 与逻辑回归等简单高效的算法构建基线并深入清洗数据，而不是一遇到困难就盲目套用复杂的重型神经网络，小巧轻量的模型在文法约束下往往能发挥出意想不到的强大效能。

Source quotes

Isaac Rudnick: Committee hearings, board meetings, earnings calls, and claims reviews all follow a kind of script: an opening, a presentation, expert input, public back-and-forth, a decision. Isaac Rudnick: Most transcript parsing tools ignore this structure even though it’s what you actually want. Isaac Rudnick: This information is especially important for analyzing the work and power relationships within democratic bodies, so we built an end-to-end pipeline that recovers it. Isaac Rudnick: The grammar alone accounts for the full structure of 55% of hearings and plain logistic regression outperformed a shallow MLP model on 7 of 8 section types with a fraction of the complexity. Isaac Rudnick: In this talk, I'll walk through what worked, what broke, and how to decide when a handmade model beats a neural one. [0:04] Isaac Rudnick: In democratic governance, a proposed bill cannot become law without navigating a rigorous committee hearing process where lawmakers, experts, and the public debate its merits and flaws. [2:23] Isaac Rudnick: Can committee transcripts be standardized to facilitate systematic comparisons, evaluate legislative work, and hold elected officials accountable? [2:23] Isaac Rudnick: Can researchers derive a formal discourse grammar for legislative debate analogous to natural language parts of speech (nouns, verbs, adjectives, and determiners)? [4:09] Isaac Rudnick: The core intuition behind the discourse grammar is establishing "who speaks and when." [11:47] Isaac Rudnick: start with traditional methods

Original transcript

[0:04] Hi, my name is Isaac Rnik and in cooperation with uh Nepundas and Dr. Ford Kosmood from California [0:14] Polytenic State University in San Los Abyispo. uh we developed a a rhetoric a rhetoric parser [0:23] uh particularly a committee hearing discourse parser for legislative uh bill bill bill hearings. [0:33] Now, in order for in order for a bill to become law, it has to pass through various committees [0:41] in which legislators and members of the public can offer their opinions, positions [0:48] and discuss the various merits of of the bill uh being discussed. And this can include [0:58] uh different sections such as an introduction in which the presiding chair of the committee [1:04] will introduce a member of a separate committee or of the current committee. And [1:13] who is a bill author or a designated presenter of the bill? And there's an there are other sections [1:21] such as bill presentation in which the bill a bill author or a member of the committee or in some [1:29] uh rare cases the presiding chair will present the bill by giving a overall summary [1:35] and then this can progress onto expert testimony uh in which in which members of the committee in [1:43] favor may call experts to offer their uh stance. es and informed testimony. Members of the public [1:52] can also provide their stances and opinions. And then finally, it goes on to vote, closing remarks [2:00] and voting in which members of the committee will decide whether or not the bill needs further [2:08] revisions or amendments or whether it can progress onto the next committee for discussion. [2:14] And after all that's completed, then it'll go on to uh the the assembly or senate floor for voting. [2:23] Unfortunately, a lot of state governments and even local governments don't keep transcripts [2:30] uh of these discussions even though under ADA compliance they're supposed to um or at least keep [2:38] recordings of them. Thankfully, as part of the digital democracy project, uh various transcripts [2:45] have been created for committee hearings for California, Texas, Florida, and I believe Hawaii [2:53] for the years 2015 through 2018. Now, not all of those uh in not all of those states have hearings [3:03] from all those years, but for California, they did have hearings for 2015 through 2018. And so [3:12] that was the data set with which we constructed our pipeline or upon which we based our pipeline. [3:21] And the main goal of this research was to answer two big questions. Basically, can uh [3:29] bill hearing transcripts be standardized for better comparison and evaluation [3:35] and holding our legislators and politicians accountable? And the other question is can we [3:42] derive a grammar for these bill discussions. So just as there are nouns, verbs, adjectives and [3:50] uh and even determiners and punctuation, can we find something analogous for these bill hearings? [3:58] And our answer to this has been a resounding or more or less a resounding yes. So the basis [4:09] of this is the basis of this grammar and our pipeline is determining who speaks and when. [4:17] Now during expert Now during expert testimony it's typically only experts and the presiding chair [4:24] and occasionally a member of the committee or uh the secretary in the rare case where [4:32] there's confusion or some non-standard uh noise so to speak in the discussion. Whereas during [4:41] uh the voting bill introduction and very and a couple other sections, the committee secretary [4:48] is allowed to to speak typically for uh procedural reasons such as introdu such as uh calling role in [4:58] passing particular motions uh ensuring a quorum is present and various other uh procedural things. [5:10] Uh, and while this may sound straightforward, a lot of these all none of these rules [5:18] were determined beforehand. And so we had to dig into various uh related research uh such as [5:28] onto such as data models and various ontologies or ways of representing different concepts. uh [5:38] more abstractly in use by other organizations and governments. It turns out that the UK parliament [5:45] has had a similar problem back in 2016. And so they developed a very robust and detailed uh [5:55] data model for modeling similar tasks. Not this particular one, [6:01] but we were able to adapt one of their ontologies uh particularly surrounding oral contributions [6:09] uh to legislative discussions for this task. And that motivated some of the section and [6:17] some of the speaker attributes used throughout our process. Going back to the uh pipeline, however, [6:29] the reason a grammar is useful in this case is because just as there are only certain speakers [6:36] during certain sections, you don't want you don't need to consider all possible class labels [6:44] for a particular section or utterance uh at a given time as you're never going to find, [6:52] for example, a member of the public presenting a bill or a member of the committee uh offering [6:59] expert testimony. It's just not allowed by the uh the Senate regulation or Senate procedures, [7:07] nor is it standard uh in any of the related work as far as we've seen. And so if you're able to [7:15] restrict these classifications and uh section labels by these criteria and roles, then you can [7:25] vastly narrow down uh which ones to assign as well as what possible roles someone might have [7:33] in the case of an un unidentified speaker uh or in noisy data. And this is extremely useful uh for [7:45] this is extremely useful beyond just the pipeline as in other cases you can use this sort of [7:54] winnowing down of the possibilities to construct a more efficient and faster pipeline. uh our PI for [8:03] our pipeline we compared two models a a logistic regression model on the reduced class class uh set [8:13] associated with each utterance as well as a multi-layer perceptron or a neural network [8:20] and we actually found that a logistic regression model outperformed the neural methods and this was [8:28] quite surprising to us uh especially considering ing the the size and complexity of the task. But [8:36] we were able to achieve oh we were able to achieve uh 52% F1 uh in terms of section classification as [8:47] well as a 55% of all the hearings in our data set having a corresponding gram uh grammar parse tree. [8:57] And while that may not sound like much, this was this is this was non-existent before. [9:05] And against uh against a baseline of assigning the dominant class, it had [9:14] assigning the dominant class would leave you with a baseline of roughly [9:17] 12% accuracy. Uh so this achieves more than four times that with logistic regression. [9:25] And our neural method actually achieved only 40% accuracy in terms of sectional classification. [9:33] And most of these most of the failure cases associated with these classifications turned [9:39] out to actually be mistakes in or confusion associated with our data set annotations. [9:47] For our data set annotations, we used Chad JPT to generate some initial uh annotations [9:54] based off of extract and extracted feature set associated associated with each utterance. [10:02] And then we manually went through and corrected uh some of it some of the [10:09] incorrect labels and poor annotations. And a lot of the confusion and errors in those annotations [10:19] come from ambiguities in is there a way to go back? uh come from ambiguities within [10:26] the sections themselves, such as whether or not to classify something in between an expert testimony [10:34] and the public discussion or an utterance by the secretary or presiding chair that [10:42] represents a transition between the two as a member of one or the other. And [10:49] uh when I've got additional slides if uh you're curious later, but a lot of the confusion and [10:57] uh and mistakes bringing those numbers down were due to exactly those [11:04] and public comment section. Some of the biggest uh most common of this kind of mistake turned out to [11:13] be public comment sections being confused with legisla uh legislator discussions such as when [11:20] there's some argument between legislators in the midst of somebody giving public testimony. uh and [11:28] if no parse was generated, we just fell back to uh the figure I showed before where it was the most [11:39] common rules associated with each section and the allowed rules uh in each section. uh [11:47] some future work could focus on extracting additional feature information with more [11:54] advanced embedding techniques. For this we just used term frequency inverse document frequency uh [12:02] which is just comparing how frequently a term appears in one utterance [12:08] associated with a particular section as opposed to all other sections uh in which that term appears. [12:16] We could also integrate Robert's rules of order which are formal phrases and statements used by [12:22] these committees to denote different motions as well as different procedures taking place. [12:29] And we could also tag and further classified uh speakers arguments into into not just sections but [12:38] whether for example two legislators are arguing or agreeing with each other on a particular point. [12:46] Uh some recommendations and our biggest takeaways from this in terms of our usage of large language [12:54] models were that it's fairly good for research and brainstorming different directions in which uh to [13:04] take the project and uh and pivoting upon different approaches. It's it's all right for [13:15] assigning initial labels to things, but the initial annotations were heavily variable [13:23] or the quality of the initial annotations were heavily variable based off of the model we used. [13:29] So we found that GPT5.5 got approximately 50% of the annotations correct whereas Claude uh [13:41] Claude Fable uh managed to achieve roughly 70% AC uh accurate labeling [13:50] and we also we also found that code generation was a lot more efficient using this method. [13:59] but only after the architecture and data model had been solidified. [14:06] Attempts to generate various sections of the implementation without these without a solid [14:15] data model and prior to a complete architecture resulted in messy code that slowed us down and uh [14:24] essentially just held us back. Uh but once we had those solid the solid data model and architecture, [14:32] we're able to breeze through this uh the grammar parser as well as the classification uh [14:39] evaluation harness as well as testing different models. Um and so our [14:47] our biggest takeaway is to start with traditional methods such as uh logistic regression model or TF [14:57] uh text embeddings and then only once you've thoroughly analyzed and cleaned your data [15:06] uh should you move on to more advanced methods. And you can also use large language [15:11] models to bootstrap uh these smaller and simpler and more efficient methods [15:18] as opposed to immediately turning to them uh when met with difficult. Uh if you or your company [15:26] has a similar data set of unstructured documents or uh semistructured natural language data, [15:34] I'd love to take a look at it. uh you can contact me uh my prof uh professor Kosmood [15:42] and Nepun at these emails and I'd also like to thank Sophia and Pelvi and Casey for uh their [15:51] input on uh on identifying self introductions and class for classifying self introductions [16:00] as well as uh organizing self- introductions based off of latent feed features uh within the corpora. [16:08] [applause]