Virginia Tech Deploys AI Models to Audit Essays and Assessments

弗吉尼亚理工部署AI模型审核文书与评估 将生成式AI整合至云基础设施中，简化了学术评估，并对本科招生中的人工评分进行验证。 Rob Hopkins介绍了弗吉尼亚理工大学自2022年11月起经历四个不同演进阶段的AI应用历程。他重点阐述了由数据科学家、云工程师、开发人员以及跨部门合作伙伴组成的跨学科团队如何协同构建AI基础设施。该演讲重点聚焦于两个关键的行政用例：开发用于项目与行政评估的GenAI工具，以及利用LLM通过验证人工评分和标记异常值来精简本科招生文书审核。 弗吉尼亚理工学院分析与机构效率办公室助理副教务长 Rob Hopkins 分享了学校如何利用 AI 工具提升机构分析与行政运作的效率与效能。 (0:04) 该项目由数据科学家、云工程师和全栈开发团队共同推动，并与招生管理部门及理学院教师建立了紧密的跨部门协作机制。 (0:04) 弗吉尼亚理工学院的 AI 演进历经四个阶段，自 2022 年 11 月 ChatGPT 发布起，最初通过低风险的日常行政实验（如起草脚本、修改演示文稿等）逐步展开探索。 (2:13) 本次分享重点探讨两大核心落地场景：面向机构有效性（IE）评估的生成式 AI 工具，以及辅助本科招生文书评审的大语言模型应用。 (0:04)

Rob Hopkins, Assistant Vice Provost for Analytics and AI at Virginia Tech, outlines how the university leverages artificial intelligence across administrative departments to improve institutional analytics and operational efficiency. Cross-Departmental Collaboration and Program Objectives (0:04) Virginia Tech's artificial intelligence implementations rely on an interdisciplinary team within the Office of Analytics and Institutional Effectiveness, combining data scientists, cloud engineers, and full-stack developers with campus partners in Enrollment Management and faculty in the College of Science. Hopkins emphasizes that the technical deployment is a collective effort rather than individual leadership, focusing on sharing practical case studies across administrative units. Phased Adoption and Early-Stage Risk Mitigation (2:13) The university's generative AI adoption began following the public release of ChatGPT in November 2022 and evolved through four overlapping phases. In the initial stage, personnel restricted their exploration of web-based large language models to low-risk administrative tasks—such as drafting scripts, revising presentation decks, and composing emails—ensuring that proprietary and sensitive institutional data were not exposed. The presented roadmap targets practical operational applications, including specialized tools for the Institutional Effectiveness group and streamlined undergraduate admissions essay reviews, anchored in cautious, phased infrastructure development.

弗吉尼亚理工大学分析与机构效能办公室副教务长助理 Rob Hopkins 阐述了该校如何依托跨学科团队与跨部门协作分阶段引入生成式人工智能，旨在提升机构效能并支持本科招生等具体管理流程。 跨部门技术协作与核心应用规划 (0:04) Rob Hopkins 强调，相关技术落地并非个人成果，而是由分析与机构效能办公室（OAIE）旗下的数据科学家、云工程师和全栈开发人员共同承担。除技术团队外，该校还与招生管理部门以及理学院的教师展开了深度跨部门协作。 根据 Hopkins 的规划，弗吉尼亚理工大学的分享重点聚焦于三大方向：首先是梳理该校的人工智能基础设施演进与采用时间线；其次是展示专为机构效能（IE）团队定制开发的生成式 AI 工具；最后是介绍大语言模型在招生管理领域的实际落地，重点在于优化和精简本科入学申请文书的评审流程。 四阶段演进路径与低风险初步探索 (2:13) 该校的生成式人工智能建设经历了四个相互重叠、非纯线性的发展阶段。Hopkins 追溯指出，探索始于 2022 年 11 月 ChatGPT 公开发布之后，团队由此逐步熟悉并评估大语言模型能力。 在最初的探索阶段，工作人员仅将网页版大语言模型应用于低风险的日常行政支持，包括起草脚本、修改幻灯片文稿以及撰写电子邮件等。为防范数据安全与合规风险，该阶段明确规定不得将任何专有或敏感的机构数据上传至外部模型。 目前的记录展现了弗吉尼亚理工大学在初期通过限制敏感数据、聚焦基础行政任务来稳步探索 AI 的路径，但由于材料在此截断，后续各发展阶段的演进机制、工具实际效能以及招生文书评审方案的量化结果仍有待进一步补充。

## Background 该视频由 Ai4 于 2025 年 8 月 28 日发布在 YouTube 平台，主讲人为弗吉尼亚理工大学分析与人工智能助理副教务长 Rob Hopkins。内容主要介绍了弗吉尼亚理工大学如何将 AI 技术与 Amazon Bedrock 基础模型集成到校级数据平台中，以赋能机构效能评估并优化本科招生文书审核流程。 ## TL;DR 演讲首先回顾了弗吉尼亚理工大学从早期商业 LLM 探索、本地实验到最终落地校内 ARC 超算集群与 Amazon Bedrock 云端架构的演进路线。随后，Hopkins 重点拆解了两个实际落地案例：一是利用 Claude 3.5 Sonnet 实现学术与行政项目评估报告的自动化提取与指标判定，将耗时从 300 多个工时压缩至 3 小时以内；二是针对年均 6 万多份本科申请，利用多开源模型集成构建人机协同文书打分机制，使人工审核工作量减少 46% 并年均节省 8,000 小时。整套方案在保障数据合规、解释性与评审质量的同时，为高校行政与业务智能化转型提供了可复制的范式。 ## Atmosphere & style 整场演讲呈现出严谨务实、数据驱动且极具工程落地导向的专业氛围。讲者 Hopkins 态度谦逊真诚，开场便将成果归功于背后的技术团队与跨部门协作伙伴，随后的分享则紧扣业务痛点、系统架构与详实的数据回测指标展开。整体叙事平实从容、逻辑严密，没有任何浮夸的炒作感，给观众带来一种高度可信、扎实且富有启发性的技术分享体验。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=luXGCIQp8os&t=4s) 弗吉尼亚理工大学展示了如何将生成式 AI 深度整合进高校运营与机构效能分析中，核心在于数据科学、云架构与各业务部门的紧密协作。 - [02:13](https://www.youtube.com/watch?v=luXGCIQp8os&t=133s) 学校的 AI 基础设施经历了从早期 Web 端探索、本地轻量开源模型验证，到目前“校内高性能计算集群（ARC）+ Amazon Bedrock 云平台”共存的双轨生产架构。 - [05:04](https://www.youtube.com/watch?v=luXGCIQp8os&t=304s) 在机构效能场景中，团队精准锁定了繁重的“低垂果实”——每年夏季需耗费 3-4 名员工 2-3 周（超 300 工时）手动审查的院系评估报告。 - [07:00](https://www.youtube.com/watch?v=luXGCIQp8os&t=420s) 通过在自研数据平台（UDC）中结合 Amazon Bedrock 上的 Claude 3.5 Sonnet 模型，实现了对学生学习成果（SLO）与项目成果（PO）的自动化提取与达成度判定。 - [09:20](https://www.youtube.com/watch?v=luXGCIQp8os&t=560s) 为了消除业务人员对黑盒模型的疑虑，系统在前端直接展示 AI 的推理过程与判定依据，通过可解释性迅速建立起人机信任。 - [11:30](https://www.youtube.com/watch?v=luXGCIQp8os&t=690s) 评估报告自动化将原本 300 多个工时的手动处理压缩至不到 3 个机器工时，使业务人员得以将精力转向实质性的院系教学与运营改进探讨。 - [13:15](https://www.youtube.com/watch?v=luXGCIQp8os&t=795s) 在本科招生文书评审中，每年需审核超 6 万名申请者的 4 篇短文书，传统双盲人工评审带来超过 50 万篇次的庞大审核与培训负担。 - [14:40](https://www.youtube.com/watch?v=luXGCIQp8os&t=880s) 文书评审系统构建了基于开源模型（LLaMA、Qwen、Phi-2）的多模型集成（Ensemble）架构，通过多数投票机制生成 AI 评分并与第一位人类评审员打分比对。 - [16:00](https://www.youtube.com/watch?v=luXGCIQp8os&t=960s) 系统设立了比人类之间（4 分）更严苛的比对阈值（差值 ≤ 2 分），分值吻合时即确立成绩，仅有分歧超过 2 分的文书才会触发第二轮人工复审。 - [17:20](https://www.youtube.com/watch?v=luXGCIQp8os&t=1040s) 历史回测显示近 92% 的文书人机分值误差在 2 分以内，实际仅约 8% 进入人工复审，成功减少了 46% 的人工审核量，年均节省约 8,000 个工时。 - [18:38](https://www.youtube.com/watch?v=luXGCIQp8os&t=1118s) 人机协同（Human-in-the-loop）模式在保障审核质量与公平性的前提下消除了招生瓶颈，为高等教育领域的 AI 智能化转型提供了成熟范式并获主流媒体认可。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=luXGCIQp8os&t=4s) 演讲开场与团队致谢 弗吉尼亚理工大学（Virginia Tech）分析与人工智能助理副教务长（Assistant Vice Provost for Analytics & AI）Rob Hopkins 登台开场。他所在部门为“分析与机构效能办公室”（Office of Analytics & Institutional Effectiveness）。Hopkins 明确指出，本次演讲的核心目的是通过两个具体的实际案例，展示学校如何利用人工智能（AI）工具来提升运营效率和工作效能。 在深入技术细节之前，Hopkins 特别向背后的幕后团队致谢。他坦言自己虽然站在台上演讲，但这并不是他一个人的成果，而是由他有幸带领的一群极具天赋的数据科学家、云架构工程师、全栈开发人员以及分析与 AI 基础设施团队共同完成的。此外，该项目还与招生管理部门（Enrollment Management）以及理学院（College of Science）的教职员工展开了跨部门合作。正如他所强调的：“我在这里负责宣讲，但实际工作是他们做的（I'm here. I'm doing the talking, but they did the doing）。” - **演讲大纲主要包含三大板块**： - 弗吉尼亚理工大学 AI 基础设施与资源演进的时间线总结； - 机构效能（Institutional Effectiveness）团队的评估自动化落地案例； - 面向本科招生录取中申请文书审核的 AI 与大语言模型（LLM）落地案例。 --- ### [02:13](https://www.youtube.com/watch?v=luXGCIQp8os&t=133s) 弗吉尼亚理工大学的 AI 技术演进路径 ```mermaid flowchart LR accTitle: AI Infrastructure Evolution Timeline subgraph S1["阶段一: 早期探索 (2022年11月)"] A["基于 Web 的商业 LLM<br/>(ChatGPT)"] A1["脚本编写与演示稿修改"] end subgraph S2["阶段二: 本地部署验证"] B["本地轻量级开源 LLM"] B1["探索文书审核可行性与 Prompt 工程"] end subgraph S3["阶段三 & 四: 双轨生产架构 (当前)"] C["校内 ARC 超算集群<br/>(运行开源 LLM / 连接 SIS)"] D["Amazon Bedrock 云平台<br/>(托管 100+ 基础模型 / Claude 3.5 Sonnet)"] end S1 --> S2 --> S3 ``` Hopkins 梳理了团队从 2022 年底至今的 AI 探索历程。这一演进并非严格的单向替代，而是多个阶段的技术在当前共存： 1. **早期探索（2022 年 11 月）**：随着 ChatGPT 的问世，团队开始接触并了解 LLM 的能力边界。当时主要使用基于 Web 的大模型进行脚本编写、演示文稿润色和电子邮件起草等日常辅助工作，严禁向系统上传任何敏感数据，重点在于理解新兴工具的本质。 2. **本地部署实验**：随着轻量级开源 LLM 的发展，开发人员开始在本地机器部署模型。团队借此展开深度测试：验证模型是否能用于文书评估、建模输出是否可靠、Prompt 工程与架构设计该如何规范等。 3. **超算集群生产落地（ARC）**：利用学校内部的高性能计算超算集群（Advanced Research Computing, ARC），运行开源 LLM 并执行自动化脚本，将计算结果无缝写回学生信息系统（SIS）。文书审核案例正是依托此架构运行。 4. **云端托管生态（Amazon Bedrock）**：针对前瞻性应用，团队选用了 Amazon Bedrock 作为 AI 基础设施平台。由于学校长期使用 AWS 处理存储、计算集群和数据库，Bedrock 提供了熟悉、合规且安全的云环境。 --- ### [05:04](https://www.youtube.com/watch?v=luXGCIQp8os&t=304s) 案例一：学术与行政项目评估自动化 ```mermaid flowchart TD accTitle: Academic Program Assessment Workflow subgraph InputStage["1. 数据输入与预处理"] A["用户上传评估报告 (PDF/Word)"] --> B["Python 脚本分段解析<br/>(提取标题、分类、结果指标)"] B --> C["组装为 JSON 并经 REST API 发送"] end subgraph BedrockStage["2. Amazon Bedrock 智能分类"] C --> D{"Prompt 路由器"} D -->|"学习成果"| E["SLO 专用 Prompt 模板"] D -->|"项目成果"| F["PO 专用 Prompt 模板"] E --> G["Claude 3.5 Sonnet 模型推理"] F --> G G --> H["输出标准化 JSON 结果"] end subgraph OutputStage["3. 结果呈现与置信验证"] H --> I["University DataCommons (UDC) 平台解析"] I --> J["前端渲染: 判定达成情况 (Met / Partially Met / Not Met)"] I --> K["输出 AI 判定依据与推理摘要 (用于人工校验与信心建立)"] end ``` 在与机构效能（IE）部门对接时，技术团队首先寻找业务流程中的“低垂果实（low-hanging fruit）”，即业务人员最繁重且不愿重复的手工劳动。痛点迅速被锁定在每年夏季的评估报告审查上： - **传统审查流程的痛点**： - 每年夏季需审查 150 至 200 份各院系提交的项目评估报告（如生物系等专业报告）。 - 依赖 3 至 4 名全职员工及研究生助理，需耗费整整 2 至 3 周的全职劳动时间。 - 人工逐页查阅并提炼“学生学习成果（Student Learning Outcomes, SLO）”与“项目成果（Program Outcomes, PO）”，判定指标是否达成（Met / Not Met），并将所有数据手工录入共享电子表格。 为彻底解决该痛点，团队在学校自研的 Python/JavaScript 数据生态平台 **University DataCommons (UDC)** 中构建了智能流水线： 1. **解析与路由**：用户上传报告后，后台 Python 脚本对文档进行结构化切片与解析，结合附带的元数据打包成 JSON，通过 REST API 发送至 Amazon Bedrock。系统依据指标属性自动匹配 SLO 或 PO 的专用 Prompt 模板。 2. **模型选择**：在 Bedrock 提供的上百种基础模型中测试后，选定 **Claude 3.5 Sonnet** 作为核心推理模型。模型精准输出指标判定（已达成、未达成、部分达成等）及对应的佐证摘要。 3. **可解释性与信任机制**：在系统试点初期，为了消除业务人员对黑盒模型的疑虑，团队专门设计了推理溯源机制，在前端直接展示 AI 给出该判定的“思维推理过程（AI reasoning）”，供业务团队抽检验证，快速建立起对自动化流程的信任。 - **应用成效**： - 项目文档分类与标注的人工耗时从原本的 **300 多个人工工时直接缩减至不到 3 个 LLM 机器工时**。 - 业务人员彻底从机械的数据搬运中解脱出来，将省下的时间投入到与各学院、系所的实质性沟通中，针对评估中发现的问题探讨改进方案，极大提升了高校运营与育人工作的核心价值。 --- ### [13:15](https://www.youtube.com/watch?v=luXGCIQp8os&t=795s) 案例二：本科招生文书的“人机协同”智能评审 ```mermaid flowchart TD accTitle: Admissions Essay Scoring and Verification Workflow A["申请者提交 4 篇文书<br/>(每篇上限 120 词，总计超 60,000 名申请者)"] --> B["人工评审员 1 打分 (Human 1 Score)"] A --> C["多模型集成推理 (Ensemble LLMs)<br/>- LLaMA<br/>- Qwen 3<br/>- Phi-2"] C --> D["多数投票机制 (Majority Vote) 汇总 AI 分数"] B --> E{"人机分值比对 (总分差值)"} D --> E E -->|"分值差异 <= 2 分<br/>(判定一致，约占 92%)"| F["采纳初审分，直接生成最终录取评分"] E -->|"分值差异 > 2 分<br/>(触发异常标记，约占 8%)"| G["派发第二位人工评审员 (Human 2 Review)"] G --> H["综合两次人工评分取平均值作为最终得分"] ``` 在演讲的最后阶段，Hopkins 详细介绍了 AI 在本科招生管理（Enrollment Management）文书评审中的创新落地。 - **业务规模与审核负担**： - 弗吉尼亚理工大学每年收到超过 **60,000 份**申请。 - 每名申请者需提交 4 篇各 120 词的短文书（Ut Prosim 提示文书，旨在考察学生的非认知能力与价值观）。 - 在传统流程下，每篇文书必须至少由 2 名人工评审员双盲打分；若两人评分差异达到或超过 4 分，则必须引入第 3 位人工评审裁决。 - 这导致每个招生周期全校工作人员及志愿者需要审核**超过 50 万篇次文书**，评审与志愿者培训负担极其沉重。 - **人机协同确认机制（Confirmatory Framework）**： - 团队设计了基于多模型集成（Ensemble）的验证架构，采用包含 **LLaMA、Qwen（通义千问）以及 Phi-2** 在内的多款开源模型，对文书进行独立打分并通过多数投票（Majority Vote）聚合出模型端分数。 - 申请文书首先由第一位人类评审员（Human 1）打分，系统随后比对人类打分与 AI 集成打分： - 若 4 篇文书总分差异 **≤ 2 分**，系统判定 AI 成功确认了人类评分的有效性，直接以此确立最终成绩，免去第二次人工审核； - 若总分差异 **> 2 分**，系统自动打上异常标记，文书转交第二位人类评审员（Human 2）进行复审，最终得分为两次人工打分的平均值。 - Hopkins 特别说明，针对人类评审员之间的分歧容忍阈值为 4 分，而模型与人类之间的比对阈值被严格限定为 2 分，这意味着系统对大模型设立了比人类之间更严苛的标准。 - **历史数据回测表现与量化效益**（以 2025 年秋季申请数据回测）： - **37%** 的文书 AI 评分与人类评审员完全一致； - **74.5%** 的文书评分误差在 ±1 分以内； - **近 92%（约 91.8%）** 的文书评分误差在 ±2 分以内； - 实际仅有约 **8%** 的文书需要流转至第二轮人工复审。 - 全校文书评审总量从 50 多万篇次直接骤降至 27 万篇次，**人工审核工作量减少了 46%**，每年直接**节省约 8,000 个工时**。 --- ### [18:38](https://www.youtube.com/watch?v=luXGCIQp8os&t=1118s) 成果发布与社会反响 在 2026 年秋季招生周期全面上线该机制前，学校于校内发布了新闻通告，明确宣布将在招生文书评审中引入这套“人机协同（Human-in-the-loop）”AI 支持模型。该举措迅速引发外界关注，《福布斯》（Forbes）杂志随后刊文进行了专题报道。 Hopkins 引述了《福布斯》报道中的积极评价：尽管外界曾普遍担忧 AI 介入录取决策，但弗吉尼亚理工大学的实践证明，“AI 深度整合进高校申请流程的时代已经到来，但这绝非焦虑的申请者所担心的‘AI 招生末日’”。Hopkins 总结道，这一方案不仅显著消除了招生季的业务流瓶颈、加快了录取通知发放节奏，更在保障公平与质量的前提下，为高等教育领域探索 AI 智能化转型开拓了全新道路。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=luXGCIQp8os&t=4s) I'm here. I'm doing the talking, but they did the doing. — Rob Hopkins > [18:38](https://www.youtube.com/watch?v=luXGCIQp8os&t=1118s) AI 深度整合进高校申请流程的时代已经到来，但这绝非焦虑的申请者所担心的‘AI 招生末日’ — Forbes > [05:04](https://www.youtube.com/watch?v=luXGCIQp8os&t=304s) 项目文档分类与标注的人工耗时从原本的 300 多个人工工时直接缩减至不到 3 个 LLM 机器工时 — Rob Hopkins > [13:15](https://www.youtube.com/watch?v=luXGCIQp8os&t=795s) 全校文书评审总量从 50 多万篇次直接骤降至 27 万篇次，人工审核工作量减少了 46%，每年直接节省约 8,000 个工时 — Rob Hopkins > [13:15](https://www.youtube.com/watch?v=luXGCIQp8os&t=795s) 针对人类评审员之间的分歧容忍阈值为 4 分，而模型与人类之间的比对阈值被严格限定为 2 分，这意味着系统对大模型设立了比人类之间更严苛的标准 — Rob Hopkins > [13:15](https://www.youtube.com/watch?v=luXGCIQp8os&t=795s) 37% 的文书 AI 评分与人类评审员完全一致；74.5% 的文书评分误差在 ±1 分以内；近 92% 的文书评分误差在 ±2 分以内 — Rob Hopkins > [05:04](https://www.youtube.com/watch?v=luXGCIQp8os&t=304s) 每年夏季需审查 150 至 200 份各院系提交的项目评估报告……依赖 3 至 4 名全职员工及研究生助理，需耗费整整 2 至 3 周的全职劳动时间 — Rob Hopkins > [02:13](https://www.youtube.com/watch?v=luXGCIQp8os&t=133s) 严禁向系统上传任何敏感数据，重点在于理解新兴工具的本质 — Rob Hopkins ## Mentioned Rob Hopkins, Virginia Tech (弗吉尼亚理工大学), Office of Analytics & Institutional Effectiveness (分析与机构效能办公室), Amazon Bedrock, Claude 3.5 Sonnet, LLaMA, Qwen (通义千问), Phi-2, ChatGPT, University DataCommons (UDC), Advanced Research Computing (ARC), Student Information System (SIS), Enrollment Management (招生管理部门), College of Science (理学院), Forbes (福布斯) ## Video description Rob Hopkins, Assistant Vice Provost, Analytics & AI, Virginia Tech Abstracts In this session we will discuss Virginia Tech's strategy for integrating AI applications into our University DataCommons to enhance operations in the Office of Analytics & Institutional Effectiveness. Leveraging Amazon Bedrock and foundational models, we create and implement AI solutions while ensuring data security and privacy. We will begin with a summary of our AI experimentation and exploration journey, from local LLMs to an overview of our current infrastructure, then present two use cases demonstrating the utility of AI technologies for institutional research and effectiveness. First, we will discuss the GenAI tools we are developing for program and administrative assessment. From selecting a foundation model to building Knowledge Bases for Retrieval Augmented Generation and end-user interfaces, we aim to show how AI can streamline academic program and administrative assessment processes. The second use case focuses on how we use large language models to support the undergraduate admissions process at Virginia Tech. These models help streamline the applicant essay review process by validating human scores, identifying outliers, and flagging essays for additional human review. In sum, we hope the audience will learn general strategies, steps, and tools to deploy an AI framework, setting the stage for the next wave of creativity and innovation in institutional analytics and effectiveness. [Source](https://www.youtube.com/watch?v=luXGCIQp8os)

[0:04] Rob Hopkins: I'm here. I'm doing the talking, but they did the doing. [18:38] Forbes: AI 深度整合进高校申请流程的时代已经到来，但这绝非焦虑的申请者所担心的‘AI 招生末日’ [5:04] Rob Hopkins: 项目文档分类与标注的人工耗时从原本的 300 多个人工工时直接缩减至不到 3 个 LLM 机器工时 [13:15] Rob Hopkins: 全校文书评审总量从 50 多万篇次直接骤降至 27 万篇次，人工审核工作量减少了 46%，每年直接节省约 8,000 个工时 [13:15] Rob Hopkins: 针对人类评审员之间的分歧容忍阈值为 4 分，而模型与人类之间的比对阈值被严格限定为 2 分，这意味着系统对大模型设立了比人类之间更严苛的标准 [13:15] Rob Hopkins: 37% 的文书 AI 评分与人类评审员完全一致；74.5% 的文书评分误差在 ±1 分以内；近 92% 的文书评分误差在 ±2 分以内 [5:04] Rob Hopkins: 每年夏季需审查 150 至 200 份各院系提交的项目评估报告……依赖 3 至 4 名全职员工及研究生助理，需耗费整整 2 至 3 周的全职劳动时间 [2:13] Rob Hopkins: 严禁向系统上传任何敏感数据，重点在于理解新兴工具的本质

[0:04] Rob Hopkins: I'm doing the talking, but they did the doing.

[0:04] Thank you all very much for coming and [0:08] um to introduce myself here. I'm Rob [0:10] Hopkins. I'm the assistant vice provost [0:13] for analytics and AI within the office [0:17] of analytics and institutional [0:19] effectiveness at Virginia Tech. [0:23] And the main goal of today's [0:25] presentation is for us to give a couple [0:27] of case studies on how we're using um AI [0:32] tools to improve [0:35] uh the efficiency and the effectiveness [0:38] of what we do and we're going to that in [0:40] two rather broad areas and case studies. [0:46] So the first thing that I want to do is [0:48] acknowledge the fact that while I'm up [0:50] here on the stage, this is not my work. [0:52] I get the great privilege of leading an [0:55] extremely talented group of data [0:58] scientists and cloud engineers and for [1:00] stack developers and all the folks that [1:03] really do the stuff that you're seeing [1:05] here. Um and in addition to those folks [1:09] within our data and analytics [1:10] infrastructure group analytics and AI [1:13] institutional effectiveness [1:15] um also cross out from us in u across [1:18] the college in terms of in enrollment [1:21] management and uh we also work with the [1:23] faculty in the college of science. So I [1:26] want to make that one of the things that [1:28] are pretty clear up front is that I'm [1:30] here. I'm doing the talking, but they [1:32] did the doing. [1:35] So here's a bit of an outline for what I [1:38] have planned. Uh three points here. One, [1:42] a summary of the AI infrastructure and [1:45] the resources that we used in our office [1:48] and a little bit of timeline about what [1:49] that looked like. [1:51] The second is a demonstration of what we [1:54] do for the U group of institutional [1:56] effectiveness which is also nested in [1:58] our office. And third is an application [2:02] of we use AI and LOM uh for enrollment [2:07] management in particular in our [2:09] admissions process. [2:13] So we'll start with the timeline. Like [2:15] most people uh over on the left, we [2:18] started our group was introduced in [2:21] November 2022 to chat GPT what it could [2:24] do, what it could not do. And so we we [2:27] were using those web-based LOMS to [2:29] explore scripting uh you know revision [2:33] of presentations and uh to generate [2:35] emails and so on and so forth. Nothing [2:38] that involved us actually uploading data [2:42] into those systems or doing any sort of [2:43] serious work but trying to understand [2:46] the nature of the tool that we that was [2:48] clearly coming onto the stage. [2:51] Second stage there locally deployed and [2:54] these are sort of a generalized timeline [2:58] really specific. It's not that we went [3:00] from one to two to three to four. Some [3:02] of these have coexisted over time. But [3:04] the second one, you know, we got to a [3:06] point where some of the developers had [3:09] um these LOMs that were small enough to [3:12] be locally deployed on machines. And at [3:15] that point, we could start to uh [3:17] research exactly what these LOMs could [3:20] do in terms of what we needed them to [3:23] do. Could they review essays? Could they [3:25] output? Could they model? Did they [3:27] produce reliable results? Did they not [3:29] produce reliable results? What did the [3:31] prompting need to look like? What did [3:32] the engineering need to look like? [3:35] The third stage over there on the right. [3:38] Um and actually the third and fourth I [3:40] would add at this point. Uh that is [3:42] where that is where we are now. Some of [3:44] our processes are still um housed on a [3:47] high uh on our internal arc resources [3:51] which is our supercomputing cluster. So, [3:54] we're able to use LOMs, uh, open open [3:56] source LOMs, uh, write various scripts [3:58] that we can execute and run on those [4:00] clusters to do what it is that we need [4:02] them to do. And we're able to tether [4:04] those to other resources and put those [4:05] back into our student information [4:07] system, get what we need out of them. [4:09] And that's actually case study two of [4:11] what I'm going to profile here in just a [4:12] moment. [4:14] And for but for more of the forward [4:16] looking stuff that we're thinking about, [4:18] uh, we're using Amazon Bedrock. that's [4:21] their current AI infrastructure uh [4:23] provider and in that case we're doing [4:25] that because we've used AWS for quite a [4:27] long time in terms of the general [4:29] storage [4:31] computational clusters for most of the [4:33] work that we do the progressive [4:34] databases and so this [4:38] being Amazon provided that familiar [4:41] environment that we needed um and we and [4:44] we were comfortable enough with to move [4:46] on through our process. [4:50] So now to get down to some details here, [4:54] uh the first example I'm going to give [4:56] is an overview of a tool that the team [4:59] built out institutional effectiveness. [5:04] So we had our team work with that team [5:07] and we really asked them the question of [5:10] what is the lowhanging fruit? What is [5:13] the one thing you do uh that you really [5:15] wish you didn't have to do? And what [5:18] that came down to is this group had to [5:21] review and look at about 150 or 200 um [5:26] program repro uh review reports over [5:30] each summer. And for each of those uh [5:33] review reports, they had to look through [5:35] they had to look at the outcomes. They [5:37] had to classify the outcomes. They had [5:39] to figure out whether the outcomes were [5:40] met, whether the outcomes were not met. [5:43] And this was a very tedious task for [5:46] that group and that was one task that [5:49] they very quickly identified where they [5:51] thought that the implementation of AI [5:54] would be very useful. [5:57] So this is what the current or past [6:00] current process sort of look like. Uh [6:03] there would be many documents [6:06] 150 or so. There would be three to four [6:09] individuals within that office in uh [6:12] including some graduate assistants that [6:14] would work more or less full-time for [6:16] two to three weeks reviewing the [6:19] documents [6:20] and they would identify the student [6:22] learning outcome, the program outcomes [6:25] and they would try to classify those [6:27] outcomes and they would try to [6:29] categorize those outcomes as met, not [6:31] met, so on and so forth and then they [6:33] would put all this into a shared [6:35] spreadsheet. So this involved three to [6:38] four folks for two to three weeks of [6:41] full-time work. So keep that in mind. [6:45] So what we designed was a process and [6:48] this is within Amazon bedrock of course [6:51] where uh within our UDC which is a [6:55] homegrown analytics ecosystem that we [6:57] built. It's a Python and JavaScript [7:00] based system uh where the individuals in [7:04] the IE office can upload their document. [7:07] We wrote some Python scripts that pi uh [7:10] that would parse out the documents into [7:11] the relevant segment that we needed [7:14] whether it was um the title or the [7:16] classifications or the outcomes and so [7:18] on and so forth. and then outcome data [7:22] and is combined with any additional [7:24] request details that may be passed along [7:27] by the group. Then all of that is passed [7:30] uh through a JSON file through a REST [7:34] API to Amazon Bedrock. [7:40] So once it's within that system there, [7:43] the data is um analyzed to determine [7:46] which of the prompts that we designed [7:48] that should be used whether we're [7:50] looking at a student learning outcome [7:52] and in that case we'd use the top prompt [7:54] or a program outcome in that case the [7:57] bottom prompt and all this is predicated [7:59] by the parsing that we done in the prior [8:01] step. [8:03] And those predefined prompts then are [8:05] combined with these segment details and [8:09] ultimately after the step here [8:12] they're passed through the LOM and the [8:15] LOM is where the work is done [8:18] and that's where the data are passed to [8:20] the instructions are passed to. [8:22] Currently we're using claw 3.5 the [8:25] sonnet model um that seem to be we've [8:28] we've explored a few and that seem to be [8:30] the best with this particular task but [8:32] we noticed that there are and and also [8:35] and mostly most likely know within [8:37] Amazon Bedrock there over a 100 [8:39] foundational models that are available [8:41] some of those are really good for [8:42] certain things and some of those are [8:44] really good for others this tint be [8:46] really good for this task [8:48] so we pass that through [8:50] uh just to recap here. So we go from the [8:53] JSON file, we have it's parsed, we pass [8:56] the instructions, the LM identifies [8:58] whether we're going to use the SLO [9:00] prompt or the PO prompt. And now we're [9:03] identifying and we're going to be [9:04] classifying and categorizing those [9:06] particular outcomes. [9:14] And once the LM does the work that it [9:16] needs to do, it provides a JSON [9:18] formatted response which we then push [9:22] back to UD to see application that we [9:24] host on our website and that parses the [9:27] response and then that renders the [9:29] classification data and descriptions and [9:31] the output for the users themselves. So, [9:34] so at this point it's just the diagram [9:36] but in the next stage here we'll have a [9:38] look at what that looks like. [9:40] So we'll spend a little time on this [9:42] point here which is what it actually [9:44] looks like. So on the top left there the [9:46] users would upload a document and that [9:49] document would of course contain [9:51] whatever they needed to look at for a [9:53] program a department department of [9:55] biology or so on and so forth. Uh a [9:57] Python script would up would parse that [10:00] document into the relative segments. We [10:02] pass that through the LOM and the LOM [10:05] would then output whether it be the SLO [10:08] classifications on the top or the PO [10:10] classifications on the bottom and it [10:13] would let us know whether those findings [10:14] were met, they were not met, they were [10:17] almost or partially met or fully met and [10:20] then uh there would be an AI summary of [10:24] why that particular conclusion well [10:26] maybe not why but an AI summary of what [10:30] was involved in that document to uh [10:32] validate that particular outcome. [10:42] Now for this group, what we also did [10:44] because it's an early stage of the [10:45] piloting process, um for a certain [10:49] fraction of the data that were passed [10:52] through, uh we flag those [10:56] so that we then output to the users [10:58] themselves. [11:00] what um the AI reasoning was behind each [11:03] of the classifications. It's one thing [11:05] to design the system up front, plug it [11:07] in blind, and then run with it, but [11:10] these folks wanted to be able to better [11:12] understand what was happening, why what [11:14] was happening, and could they rely on [11:16] what was happening. So in this case uh [11:18] we designed an avenue where they could [11:20] look at the so classifications what was [11:23] going on what the AI reasoning was and [11:27] they could validate whether or not that [11:29] the the AI would working in the [11:31] particular way that they had hoped it [11:33] would work in terms of classifying the [11:35] outcomes, classifying the um criteria [11:38] whether it was met or not met or so on [11:40] and so forth. and it would essentially [11:43] um a step that we put in there to [11:45] engender confidence [11:48] in using a particular process going [11:51] forward. [11:58] So for this project what that meant was [12:02] uh that the the UDC tool university data [12:05] common tool that we developed [12:08] um it reduced the time spent class [12:11] reading and classifying and labeling [12:14] from about 300 labor hours maybe a [12:17] little more than that to less than three [12:19] LOM hours. [12:22] So what that meant for these folks is [12:24] not I don't have a job but it meant okay [12:26] now I have all this done so I can reach [12:29] back out to the departments to the [12:31] programs and I can interact with them. I [12:34] can talk about what we saw what we [12:36] didn't see and how to improve things [12:37] going forward. So that huge chunk of [12:40] time that was spent just trying to [12:42] classify and put into that spreadsheet [12:44] was now freed up dramatically for them [12:47] to do much more added value work to [12:49] really advance the mission of the [12:51] university [12:53] and what that's also done for us [12:55] fortuitously was it primed the path [12:57] forward for more advanced AI application [12:59] work within that IE atmosphere which is [13:02] which is in my opinion uh one particular [13:05] area of higher education that is replete [13:08] with opportunity ities for LOMS and [13:11] applications thereof. [13:15] All right. Now, we're going to shift [13:16] gears a little bit here uh to the the I [13:20] just got a few minutes here, about six [13:22] minutes. So, with the use of Loom as a [13:25] tool to support enrollment management [13:27] efforts. [13:30] So, each year at Virginia Tech, all of [13:33] our applicants have to submit um four [13:36] different essays. And these essays are [13:39] called the uprom which were basically [13:41] measuring the non-cognitive attributes [13:43] of the students themselves. Um each of [13:45] these essay questions u can have up to a [13:49] 120word response and that 120word [13:52] response is then reviewed and at least [13:54] in the past by a minimum of two human [13:57] reviewers [13:59] and [14:01] if needed a third. So the con so the [14:04] context for this is a lot of applicants, [14:07] a lot of essays, a lot of stuff to [14:10] review. [14:13] So we have 60,000 applicants per year or [14:17] a little more than that. We have four [14:19] essays per applicant at 120 words per [14:22] essays. We have to have a minimum of two [14:25] human reviews per essays right now. [14:28] And that really results in about half a [14:32] million or more essays that have to be [14:35] read by humans each year. Now, I say the [14:38] two plus human essays because we because [14:40] we have a threshold as we score these [14:43] essays out. If the difference between [14:46] two humans is four or more, we kick it [14:49] to a third human. So, that's why the two [14:51] plus point. But the point is some folks [14:54] out there on staff and volunteer are [14:58] reviewing in excess of half a million [15:01] essays every single admission cycle. So [15:04] the question we were asking is can we [15:06] use LOMs in a confirmatory framework to [15:09] reduce to one human raider when [15:13] when possible. [15:17] So this is what the decision flow looked [15:19] like. [15:20] Um, we figured we could do this. We had [15:23] some evidence up front with other work [15:24] we were doing, but we thought there [15:26] would be the essay submission which is [15:28] on the uh the left [15:31] the top left and then we have the human [15:34] score which is our human one score. Uh [15:37] we have since uh had uh de three models [15:42] uh the llama model there at top, the [15:44] Quinn 3 model, you see the parameters [15:47] there and a fine to model. And after [15:50] much iteration over the last year or so, [15:52] we develop uh we decided that an [15:54] ensemble approach would necessary to [15:57] where these models output a score and we [16:00] take the majority vote uh vote of their [16:02] scores and then we compare that with the [16:05] human score. So that's the middle part [16:07] of score comparison. Now once the score [16:10] comparison happens, uh that's where [16:12] there's a modification. Um if it's less [16:16] than or equal to a twopoint difference [16:18] overall for the applicant, then we take [16:21] that result because it's confirmed the [16:23] initial human score and we proceed with [16:25] the final score. If it's greater than [16:28] two points uh for that applicant, then [16:31] we actually flag that for the second [16:33] human review. And so that second human [16:35] review would um would be conducted and [16:39] then we would average those out for the [16:41] final score. Now I do want to reiterate [16:44] the point was the threshold here is two. [16:47] So we're being much more stringent with [16:48] the LOM for the for the um humans [16:51] themselves. The threshold is four. So [16:54] we're making the LOM [16:57] align more closely with the human than [16:58] we did the humans with the humans [17:00] themselves. [17:03] And so what that prompt sort of looks [17:04] like is the highle overview. So we have [17:06] the description of the task. Uh we have [17:09] some rubric information, some example [17:12] scoring, a little bit guidance there. [17:14] We're passing those uh those essays [17:16] through and the framework that we're [17:18] working within and then we're outputting [17:20] that score. Again, we're putting [17:22] outputting the score per essay and then [17:24] we're summing those up across those four [17:26] different essay per each applicant. [17:31] Uh this is the uh overview here. Um so [17:35] if we look back at fall 2025, [17:38] 37% of the time we were exactly the same [17:41] as the raider one score which was the [17:44] human 74.5% [17:47] of the time plus or minus one and almost [17:50] 92% of the time plus or minus two at the [17:54] rate of two. So what that means is if we [17:57] look retrospectively only about 8% of [18:00] the essays would have been flagged for [18:02] that second human review. [18:04] Now if we do the calculations out we [18:06] consider how much time it takes to read [18:08] these uh we reduce that workload from [18:11] over 500,000 to 270,000 [18:14] at the 46 uh 46% reduction in workload [18:18] saves about 8,000 labor hours per year. [18:21] So that for us that means a lot of [18:24] volunteer hours and volunteer training [18:26] hours that we don't have to do and it [18:29] results in an improvement in the [18:30] accuracy and elimination of the workflow [18:33] uh bottleneck for those emission [18:35] decisions themselves. [18:38] So late last month we released an [18:40] internal um news article um highlighting [18:44] that we were going to ship this for fall [18:46] 2026 applicants that we were going to [18:49] have a new approach. we going to we [18:51] would evaluate these essays with the [18:53] human in the loop uh artificial [18:55] intelligent supported model [18:59] and about a week or so after we did that [19:01] for magazine uh wrote an article about [19:04] that process [19:06] and uh to um to our advantage. Uh they [19:12] said that um [19:15] everything everyone had feared was not [19:17] so and that the day that AI is [19:20] integrated into the application process [19:22] has arrived for V for Virginia Tech in [19:25] the green there. But it is not the AI [19:27] admission apocalyptic [19:29] among anxious applicants. And so we hope [19:32] that in the end that this will allow us [19:35] to not only produce better decisions for [19:38] the applicant to get that out to the [19:40] student or the prospective student [19:42] suitor, but also um more quality overall [19:46] and unleash an avenue of potential on [19:50] what we can learn for this going [19:51] forward. [19:53] And thank [19:55] [Applause]