Allen AI Introduces TutorMoments to Evaluate How AI Tutors Balance Scaffolding and Rigor

Allen AI Introduces TutorMoments to Evaluate How AI Tutors Balance Scaffolding and Rigor The Allen Institute for AI has unveiled a preview of TutorMoments, an open evaluation framework designed to test whether large language models can navigate the pedagogical balance between offering assistance and encouraging students to solve problems independently. TutorMoments evaluates LLMs using 462 de-identified math tutoring transcripts from U.S. students in grades 2–7, featuring more than 1,500 key moments annotated by 27 math teachers. The replay framework pauses a transcript at critical pedagogical decision points and tasks an LLM tutor with managing five simulated dialogue turns opposite an LLM student. Default prompting leads models to over-help by providing excessive support, but prompting that explicitly outlines trade-offs between scaffolding and rigor improves performance across seven tested models. While models improve with enhanced prompting, they rely heavily on asking students to explain answers, whereas human tutors employ a wider range of strategies and more frequently allow students to work independently. The project currently measures model behavior rather than student learning outcomes and is limited to elementary and middle school mathematics. The Pedagogical Tension in Automated Tutoring Effective math tutoring requires diagnosing what a student knows and deliberately preserving productive struggle, rather than immediately volunteering explanations or solutions. Because language models are trained primarily to be helpful assistants, their default tendency is to do the challenging intellectual work for the user. Existing benchmarks often miss this nuance by rewarding static rules—such as never providing the answer or always supplying a hint—rather than evaluating whether a tutor makes the right contextual judgment call between assisting a struggling student and pushing for deeper conceptual reasoning. The TutorMoments Evaluation Framework The benchmark relies on TutorMoments-Preview, a dataset of 462 text-based transcripts gathered from a high-dosage tutoring program serving primarily Title I schools. Experienced math teachers identified moments where tutors had to choose between scaffolding to lower problem difficulty or pushing for rigor to demand deeper thinking. TutorMoments pauses each transcript at these decision points, instructs an LLM to take over the tutor role for five turns against a simulated student model, and uses an automated scoring pipeline validated against teacher consensus to assess whether the model scaffolded, pushed for rigor, or over-scaffolded. Model Performance and Prompting Effects Testing across seven language models demonstrated that default instructions to tutor well lead to frequent over-scaffolding and rare demands for rigorous thinking. Providing an evaluation-aware prompt that clearly outlines the balance between scaffolding and rigor improved scores across every tested model, though significant variance remained. The authors noted that human tutors scored lower than models under evaluation-aware prompts, not because AI outperforms humans, but because the human baseline data intentionally concentrated on transcripts where tutoring decisions could have been improved. Furthermore, human tutors displayed a wider variety of pedagogical tactics than models, which leaned heavily on asking students to explain their reasoning. Limitations and Planned Development The researchers note several constraints in the preview release, foremost being that simulated replays evaluate tutor behavior at single decision points rather than genuine student learning outcomes. The data also exhibits an imbalance, containing 738 scaffolding moments compared to 260 rigor moments, alongside higher measurement noise when classifying rigor pushes. The current scope remains restricted to U.S. elementary and middle-school mathematics, with future iterations planned to expand toward multimodal inputs, broader datasets, and more refined scoring pipelines.

Allen AI 推出 TutorMoments：评估大语言模型何时提供辅导与适时放手 Allen AI 推出了 TutorMoments 评测框架预览版，利用真实的美国中小学一对一数学辅导记录和重放机制，评估前沿大语言模型能否在适度提供帮助与推动学生自主深入思考之间取得教学平衡。 TutorMoments 基于 462 份美国 2 至 7 年级真实数学辅导文本，由 27 位资深数学教师标注了 1500 多个教学关键时刻。 评测采用重放机制，让大模型在关键决策点接管导师角色，与模拟学生互动 5 轮，评估其适度引导、严格要求及避免过度帮助的能力。 测试显示大模型默认倾向于“过度帮助”，即便在提示词中明确权衡策略，模型的辅导手段仍显单一且各模型表现差异较大。 该框架当前存在依赖模拟学生、未测量真实学习效果、推进严谨性样本偏少且噪声较大等局限。 辅导平衡难题与重放评估机制 良好的数学辅导需要在降低问题门槛与推动学生独立深度思考之间进行权衡。大语言模型由于默认被训练为提供帮助的助手，往往会替学生解决最困难的步骤，从而剥夺了有益于学习的探索过程。TutorMoments 采用重放评测方法，在教师标记的关键决策点切断真实辅导记录，交由待测大语言模型接替导师角色，与扮演学生的另一个模型进行 5 轮对话，并由评分流水线评估模型的教学应对是否恰当。 数据集构建与教师基准标注 公开发布的 TutorMoments-Preview 包含 462 份经过脱敏的纯文本数学辅导对话，来源于主要面向 Title I 学校学生的辅导项目。27 位美国资深数学教师对其标注了 1500 多个决策关键时刻，评估导师当时应当搭建脚手架降低难度还是推动严谨思考。评分系统以教师标注的多数意见作为判定基准，重点衡量模型在需要支持时是否提供脚手架、在时机成熟时是否推进深度思考，以及是否有效避免过度降低问题难度。 实验表现与教学策略差异 测试评估了 7 个大语言模型在无具体指引的常规提示词与阐明权衡取舍的评估感知提示词下的表现。结果显示，常规提示词下模型倾向于过度提供支持，极少引导深度思考；明确权衡取舍后所有模型得分均有提升，但各模型之间差异显著。此外，大模型在策略丰富度上不及人类导师，往往仅依赖要求学生解释答案，而人类导师更倾向于采用多元化教学策略并适时放手让学生独立解题。 局限性与后续研究方向 TutorMoments 目前仍处于早期预览阶段。由于重放使用的是模拟学生，评分仅反映模型在决策点的行为，无法代表真实学生的实际学习效果。此外，推进严谨思考的关键时刻样本数量（260 个）少于脚手架样本（738 个），检测噪声相对更高；且数据集仅涵盖美国中小学数学领域。开发团队未来计划构建更大规模的多模态数据集，并进一步完善评分流水线。

📄 Tech Report: https://tutormoments.allen.ai/static/paper/tutormoments-preview.pdf | 📊 Data: https://huggingface.co/datasets/allenai/tutormoments-preview | 💻 Code: https://github.com/allenai/tutormoments Today we're introducing a preview of TutorMoments , a framework to measure whether cutting-edge LLMs can balance one of the hardest trade-offs in education: when to step in and help a student and when to hold back and let the student do more of the work. TutorMoments is a replay-based evaluation built off real one-on-one math tutoring sessions. Experienced math teachers go through transcripts collected from a U.S. tutoring program and flag the moments where a tutor had to choose between making a problem easier to get started on and pushing the student to do more of the reasoning themselves. TutorMoments then takes the transcript up to that decision point, hands it to a language model, and has the model take over as the tutor in a simulated session – with the student played by another language model – to see what the LLM tutor does. Told only to "tutor well," we find that models tend to over-help by giving too much support and rarely pushing students to do deeper thinking. Spelling out the trade-off (when to help versus when to hold back) in the tutor's prompt improves performance, but it doesn't close the gap to human tutoring that consistently fits the moment, and LLMs still differ widely in how reliably they make that call. As part of our commitment to open research, we're releasing a dataset of de-identified tutoring transcripts , the code for running our replay pipeline , and the model tutor replays of the key moments we evaluated in those transcripts for reproducibility. We hope TutorMoments gives educators, researchers, and the teams building AI tutors a sharper way to ask how a model handles the pedagogical decisions that matter most—and helps the field build tutors that adapt to each student instead of doing the work for them. What makes a good tutor? Ask a good math tutor for help and you'll likely get a question back like, "What do you know about what the problem is asking?" That isn't unhelpfulness–part of strong teaching is diagnosing what students do know and providing the right support for them in the moment. Immediately volunteering support would rob a student of the intellectual work that helps them learn. Sometimes support is needed; other times what's most effective is a push to solidify understanding by explaining a correct answer. Language models, though, are trained to be helpful, and a helpful assistant tends to do the hard part for you—explaining the concept, laying out the steps, and guiding you to the answer. In a tutoring session, that can cut short the productive struggle—the effortful, sometimes frustrating problem-solving that learning research has long tied to stronger understanding. Most benchmarks for language models acting as tutors don't capture this tension. They tend to reward one behavior in particular – never giving away the answer to a problem, say, or always offering a hint – without accounting for whether that was the right move for where the student actually was in their understanding. But good tutoring isn't a single fixed behavior you can identify across the board. It's a judgment call: what does this student need, right now, on this problem? How TutorMoments works TutorMoments is built on real tutoring data. The dataset we're releasing, TutorMoments-Preview , is 462 de-identified, text-only transcripts of real one-on-one math tutoring with U.S. students in grades 2-7, with more than 1,500 teacher-annotated key moments and several thousand free-text annotations from 27 U.S.-based teacher annotators. The transcripts come from a high-dosage tutoring program whose students mostly attend Title I schools, shared under a research clause agreed to by parents and guardians; all data was stripped of identifying details, first by the provider and then through an additional math-aware pipeline. All annotations came from experienced math teachers, whom we asked to read the transcripts and mark key learning moments—noting what was going on, what the tutor did, and how it landed for the student. Each key moment is a decision point where the tutor had to weigh scaffolding (making a problem more accessible) against pushing for rigor (encouraging the student to do harder thinking). TutorMoments runs by pausing a transcript at one of those key moments and handing the session to a language model, which takes over as the tutor for five turns with a simulated student. We call each of these model-generated continuations a replay . An LLM-based scoring pipeline then rates each replay on three things: whether the model (1) scaffolded when the student needed support, (2) pushed for rigor when the student was ready for more challenge, and (3) avoided over-scaffolding (reducing the challenge more than the moment called for). The scoring pipeline starts from a teacher-defined ground truth: for each key moment, whether it called for scaffolding or for a push for rigor. Several teachers annotated each moment, and when they disagreed we took the majority label—if three teachers annotated a moment and two called for rigor while one called for scaffolding, the ground truth is rigor. A separate LM classifier validated against teacher annotations then decides whether the tutor's actual move matches what the moment called for—an "appropriate" turn means the tutor's classified action (scaffold, push for rigor, or over-scaffold) lines up with what teachers judged the moment to call for. Preliminary results We ran seven LLMs through TutorMoments using two prompts: a plain prompt that gives no real guidance – it only tells the model to use what it knows about good tutoring to respond to the student – and an evaluation-aware prompt that spells out the trade-off between scaffolding, over-scaffolding, and pushing for rigor. Each model was scored over key moments drawn from the tutoring transcripts, split evenly between moments where scaffolding was the right approach and moments that called for rigor. Every number in the table is a rating between 0 and 1 – the share of the relevant moments where the model did the appropriate thing – so a higher score means the model made the right call more often. A 0.50 on appropriate rigor, for instance, means the model pushed for rigor in half of the moments that called for it. A few things to keep in mind when reading the scores: Human tutors are a naturalistic reference, not a ceiling. We don't treat human tutors as a model of ideal practice—even experienced tutors make less-than-optimal choices in the moment. Scored the same way at the same decision points, the human tutors in our transcripts get 0.458 (appropriate scaffolding), 0.182 (appropriate rigor), and 0.496 (avoids over-scaffolding)—all below the models' evaluation-aware scores and around the range of their plain-prompt scores. But this isn't a claim that AI tutors outperform human teachers. Annotators specifically looked for moments where tutoring could have gone better, so the dataset concentrates on missed opportunities rather than ideal practice. The scores measure tutor behavior, not learning. Replays use a simulated "oracle" student, so the numbers reflect how a model acts at a decision point—not whether a real student learned. Rigor is noisier than scaffolding. The scoring pipeline detects rigor pushes less reliably, and there are fewer rigor moments (260) than scaffolding moments (738) in the underlying annotations. The clearest pattern in the table is how much the prompt matters: every model scores higher under the evaluation-aware prompt than under the plain one. That suggests a model's default "helpful assistant" behavior isn't enough on its own to tutor well. But spelling out the trade-off in the prompt only goes so far—while it lifts every score, models still differ widely in how they interpret the enhanced prompt and even the best scorers have plenty of room to improve. We also break down the moves that tutors made under each scenario. While prompting encourages models to push for rigor, they use fewer strategies than humans do, often relying on asking students to explain their answers. In contrast, human tutors employ more varied strategies and are much more likely to step back and let students work independently. Limitations and next steps TutorMoments is still early in its development, and it has several limitations at this stage. The biggest is that automated evaluation gives us signal about how a model behaves at a decision point, but it can't stand in for studies with real students and real learning outcomes. The dataset is also narrow: U.S.-based, mostly elementary and middle-school math, annotated by a single pool of educators. Our findings may not generalize to other subjects, grade levels, or settings. We're sharing this preview to gather feedback as we build toward a larger, multimodal dataset, a stronger scoring pipeline, and deeper analysis. Acknowledgments This project has been made possible in part through support from the Gates Foundation and Learning Commons.