Jane Street Pivots Intern Training to Review as AI Writes Code

AI代写代码之际，Jane Street将实习生培训重心转向代码审查 由于人工智能使代码生成变得平价化，却让代码验证依然代价高昂，开发者教育必须放弃基础语法编写教学，转而聚焦架构、测试与代码审查。 Aaron Bauer回顾了自己从科学计算和卡尔顿学院的高校教学，到塑造Jane Street混合型开发者教育者角色并为公司内部编辑器团队做出贡献的职业历程。讨论深入探讨了开发者工具的运行机制，解释了为何代码审查和版本控制依然与编辑器紧密相连而非依赖外部网页门户，以及遥测数据如何显示随着工程师转向代码验证，语言服务器操作呈现上升趋势。Bauer与Minsky剖析了智能体AI带来的教育困境，强调逆向课程设计、认知负荷管理，以及暂不向学员提供辅助工具以建立扎实心智模型的必要性。最后，他们详述了Jane Street如何重构其实习生项目以应对肤浅的“氛围编码”（vibe coding），在语法生成问题已基本解决的时代，着力提升测试严格度、设计评审和代码审查。 Jane Street在发现以往需要四周才能完成的入门项目如今仅需一条AI提示词即可搞定后，重构了软件工程实习生课程体系，将教学重点转向架构设计、需求收集和严格测试。 (85:10) 来自Jane Street OCaml语言服务器的遥测数据显示，引入大语言模型后，开发者的“跳转到定义”操作有所增加，这表明工程师正把更多时间花在阅读、导航和验证代码上，而非单纯编写语法。 (48:05) Jane Street将开发者教育者的角色在实际生产工程与课程设计之间进行均等划分，以应对涵盖OCaml、定制版本控制和专用编辑器扩展的公司定制化软件栈。 (21:29) 课程设计者观察到教学的中间地带已经消失：对初学者而言足够简单的编程练习在现代大语言模型面前微不足道，而能够抵御AI直接求解的任务却往往超出学生的时间和认知能力。 (59:15)

Jane Street software engineer and developer educator Aaron Bauer and host Ron Minsky argue that the rise of agentic AI upends traditional programming education, requiring curricula to move away from isolated syntax towards backward-designed learning goals, code comprehension, and rigorous verification. Backward Design and Explicit Learning Goals (76:15) Bauer argues that effective technical training requires starting from concrete, observable learning goals rather than abstract conceptual topics. Following the pedagogical formula 'After completing this, students will be able to...' instructors define precise technical behaviors—such as writing a property-based test targeting nested record types—before assembling code scaffolds, exercises, or lectures. Minsky observes that explicit learning goals shift the evaluation focus in corporate settings from grading students to measuring the efficacy of the instructional program itself. Additionally, introducing tools in context to solve immediate architectural obstacles during application development proved far more effective than teaching generalized conceptual frameworks in isolation. Managing Cognitive Load Across Tools and Languages (82:08) To prevent cognitive overload, where the volume of novel concepts overwhelms student working memory, Bauer decoupled Jane Street's introductory OCaml onboarding into distinct sequential phases. Learners initially explore language syntax via the utop REPL, then receive isolated instruction in text editors and version control tooling, and only combine them into unified workflows once individual foundations are established. This structured isolation addresses the friction of teaching inside a bespoke proprietary environment while training engineers to be multi-language practitioners across both OCaml and Python rather than siloing themselves into single-language identities. The Disappearing Beginner Problem Space (62:12) Minsky and Bauer observe that modern language models have collapsed the traditional intersection of programming exercises: tasks simple enough for a beginner to grasp are now trivial for an LLM to one-shot with a single prompt. Early attempts to prevent automated shortcuts led to assignments that were too sprawling for students to complete in the allotted time. Minsky warns that educational efficacy relies on deliberately withholding specific affordances—such as writing code away from screens or programming on paper—because the absence of struggle prevents students from forming coherent mental models. To preserve that friction, Bauer advocates for interactive scaffolding like Claude Code's learning mode, which inserts TODO markers and asks diagnostic questions rather than generating unbroken code files. Adapting Internships Toward Review, Architecture, and Testing (85:10) Because code generation is increasingly commoditized, Jane Street overhauled its intern curriculum to focus heavily on specification, architectural design, and evaluation. Rather than spending weeks writing introductory client-server applications, interns conduct stakeholder interviews with full-time engineers, critique flawed design documents, and complete dedicated multi-day training on code review. Minsky highlights that software testing has grown more critical because LLMs rely on fast, rigorous feedback loops to catch subtle bugs. After early iterations suffered from unfocused 'vibe coding' and cognitive overload caused by overly large diffs, the program adopted four-person pods to actively analyze prompt outcomes and handle the non-deterministic variability of model outputs. The Editor as an Interactive Engineering Anchor (49:07) Despite predictions that agentic tools would replace standard workflows, Bauer maintains that the text editor remains the software engineer's primary operational base. With developers writing less manual syntax, editor usage has shifted toward diagnostics, code comprehension, and reviewing stacked diffs, leading to increased rates of jump-to-definition queries and OCaml Language Server type lookups. Jane Street integrates code reviews directly into editor buffers via in-source comments called CRs and direct-edit feature branches, avoiding fragmented browser interfaces while actively engineering editor internals to handle colossal monorepo diffs without freezing. While AI tutors and automated feedback can support scalable learning, technical educators still face the unresolved dilemma of how to balance real-world AI tooling workflows with the unassisted cognitive struggle necessary to cultivate foundational engineering competence.

Jane Street 开发者教育专家 Aaron Bauer 与 Ron Minsky 指出，大语言模型正在颠覆传统的计算机教学模式，使编程教育的核心从单纯的代码敲写转向需求分析、系统设计、严格测试与代码审查。为了避免初学者在 AI 生成的代码中丧失深层心智模型，课程设计必须依靠逆向教学设计与认知负荷管理，精准把控自主练习与工具辅助的边界。 逆向教学设计与认知负荷管理 (76:15) 在将专业教学法引入企业培训的过程中，Aaron Bauer 强调明确“学习目标”（Learning Goals）是设计有效课程的基石。学习目标必须以“完成本模块后，学员将能够……”为主干，并使用具体、可检验的动词（例如“编写针对嵌套记录类型的基于属性的测试”），而非“理解测试”等模糊表述。清晰的目标能倒推必须安排的练习，并剔除无关的背景知识与脚手架代码。同时，根据认知负荷理论，初学者的工作记忆极度有限。Jane Street 过去的 OCaml 入门培训曾让新员工同时应对语法、陌生编辑器与内部版本控制工具，导致认知过载；改进后课程将要素解耦，先在 REPL 环境中学习纯语法，再隔离练习编辑器与版本控制，最后才整合进完整工作流。 AI 工具对代码理解与心智模型的重构 (49:07) 随着生成式 AI 的普及，工程师的日常工作从手动敲击语法大幅转向代码阅读、代码审查与类型检查。编辑器遥测数据显示，尽管实际输入的代码行数减少，但针对语言服务器（LSP）的跳转到定义和类型推断请求反而激增。这表明工程师在处理 AI 生成的高质量样板时，必须投入更多精力去逆向重构系统的逻辑心智模型。正如审阅他人的 Pull Request 远比自己编写代码更耗费心力，直接审阅大段 LLM 生成的代码容易导致理解脱节。Aaron 尝试了类似 Claude Code 的“学习模式”（Learning Mode）——即模型在生成脚手架时主动暂停并留下 TODO 占位符，强制人类开发者完成关键逻辑，从而在交互过程中有效建立了对代码架构的认知。 问题空间萎缩与实习生课程重塑 (62:12) 大语言模型的强大能力使教学面临严峻困境：既简单到适合新手上手、又复杂到无法被模型单次提示直接攻破的练习题，其存在区间已近乎消失。过去需要实习生做四周的入门项目，如今单次 Prompt 即可搞定。为此，Jane Street 彻底重塑了工程实习课程，不再单纯评估代码编写，而是侧重架构决策、设计文档批判、扮演利益相关者进行需求访谈，以及为期三天的代码审查实训。课程特别强化了基于属性的测试与数据库自动化测试，因为在 AI 辅助开发中，模型无法单次保证逻辑正确，严密的测试集构成了验证与纠正生成代码不可或缺的反馈闭环。 摆脱氛围编码与小组协作实践 (90:25) 在实施新课程的过程中，教学团队初期曾过度鼓励多任务并发与大量代码生成，陷入所谓的“氛围编码”（vibe coding），导致实习生无暇打磨代码细节，违背了对代码负责的教学初衷。团队随后调整节奏，将测试规范与严谨性置于模型并发之前，并严格控制代码审查 Diff 的规模以防认知过载。此外，培训从过去完全自驱的单人模式，转为由四名实习生与全职导师组成的学习小组（Pods）。由于大语言模型具有非确定性，相同的提示语也会生成不同结果，小组定期研讨模型提示策略、分析生成测试中的缺陷，成为帮助新人摸索 AI 协作边界的核心支柱。 编辑器核心地位与工具链张力 (31:18) 即便代理型 AI（Agentic AI）迅速普及，Jane Street 依然坚持将文本编辑器（VS Code、Emacs、Neovim）作为工程师的主阵地，拒绝将审查与代码索引割裂到浏览器门户中。由于公司采用定制的版本控制与代码审查系统 Iron，工程师直接在源文件内通过内联注释进行审查并直接提交修复。这种独特的超大规模代码库也给通用工具带来了巨大挑战，例如庞大的机器生成配置和超大文件经常导致 VS Code 差异对比算法卡死。团队在持续构建内部工具（如基于 OCaml Web 框架的定制 Notebook）与积极向上游开源社区贡献（如逐步剥离 25 年来定制的 Emacs 黑客补丁）之间维持动态平衡，以避免因过度专用化而承担高昂的维护成本。 生成式 AI 的非确定性与即时求解能力，正在根本性地改变编程技能的传递路径。如何在日常研发高度依赖 AI 的现状下，既防止初学者丧失构建心智模型所需的挣扎过程，又避免人为设置脱离实际的教学禁令，仍是企业与高等教育需要持续迭代应对的开放课题。

## Background 该视频由 Jane Street 于 2026 年 9 月 23 日发布在其官方 YouTube 频道（节目《Signals and Threads》）上。视频简介介绍，本期对话由主持兼资深工程师 Ron Minsky 与 Jane Street 内部兼顾编码与教学的“开发者教育者”Aaron Bauer 展开，深入探讨在企业自研工具栈及大语言模型普及的背景下，如何重新构想工程教学、编辑器工作流与实习生培养方案。 ## TL;DR 对谈梳理了 Aaron 从高校计算机教学转向工业界“50/50 开发者教育者”的历程，并深入剖析了 Jane Street 将文本编辑器视为工程大本营的独特研发与代码审查文化。遥测数据表明，随着大模型低成本生成海量代码，工程师不仅没有减少操作，反而因审查需求激增而更频繁地使用“跳转至定义”等底层分析功能。为此，Jane Street 彻底重塑了培训体系，果断摒弃盲目的“氛围写代码”，转而将教学核心聚焦于编写确定性自动化测试套件、系统架构设计以及严格的代码审查训练上。 ## Atmosphere & style 整场对谈兼具理性克制的工匠气质与深邃的教育学洞察，对话氛围沉稳、诚恳且充满智性交锋。Ron 与 Aaron 的互动极具共鸣，既有资深工程师对工业级极端工程规模与自研系统痛点的相视一笑，也有前线教育者面对大语言模型瓦解传统题库时的焦虑与反思。听众能真切感受到一种直面技术剧变时的务实探求感——既不盲目神化 AI 的威力，也不墨守成规，而是在混乱的不确定性中执着寻找工程教育的坚固基石。 ## Bulletin - [21:29](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=1289s) Jane Street 设立了“开发者教育者”岗位，采用 50% 编写生产代码、50% 专职教学的独特架构，以兼顾一线工业工程实践与系统性技术传授。 - [50:33](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3033s) 编辑器遥测数据显示，AI 代码生成技术的普及并未降低“跳转至定义”的使用频率，反而因生成代码量激增而迫使工程师进行更多阅读与排查，导致调用率微升。 - [01:25:10](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=5110s) 面对 LLM 能轻易完成基础代码编写的现实，Jane Street 彻底重构其实习生课程，将评估与教学重心转移至系统设计、利益相关者沟通及代码审查上。 - [01:01:59](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3719s) 企业技术教学正面临“空集困境”：既适合新手入门又无法被大语言模型一次性给出完美答案的编程题目，其交集已趋近于零。 ## Key takeaways - [09:32](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=572s) 现场讲座具备录播无法替代的注意力和临场聚焦效应，但长时间单向讲授易致疲劳，必须穿插提问、讨论等主动学习环节。 - [11:49](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=709s) 物理屏幕连接了海量外界干扰，容易诱导初学者将编程当成盲目试错游戏；脱离屏幕手写或推演代码能更有效地建立稳固的心智模型。 - [21:29](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=1289s) Jane Street 的开发者教育者实行 50/50 分工，要求教育者既是合格的生产级软件工程师，又是深谙教学法并赋能全员教学实践的专业导师。 - [33:56](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=2036s) 公司将文本编辑器确立为日常开发、审查与交互的唯一“大本营”，坚持去 Web 化的键盘驱动工作流，这在极端代码规模下对工具性能提出了严苛挑战。 - [40:41](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=2441s) 在开源方案无法良好适配特定业务流（如 Python Notebook）且维护多个分裂系统时，团队果断选择基于 OCaml 与自研 Web UI 技术栈推进自研破局。 - [51:53](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3113s) 代码生成的边际成本已近乎归零，但验证与审查代码的成本不降反升，使得理解与审查成为整个研发流程中最显著的瓶颈。 - [53:27](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3207s) 类似 Claude Code 学习模式的“留白让人类补全”机制，通过在编码中途强迫人类介入，比起让模型一口气全量生成更能有效沉淀心智模型。 - [59:27](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3567s) 即使未来开发者大量依赖 LLM 生成的代码，早期亲自用纯手工方式深入实现关键机制至少一次，依然是形成代码评估能力的基石。 - [01:01:59](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3719s) 编程教学面临“空集困境”，即适合初学者的题目难度与大语言模型无法一次性解决的题目难度之间的交集正急剧萎缩。 - [01:09:15](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=4155s) 企业工程培训更应以“实用工具”为载体，在具体的系统开发情境中自然引出新工具，顺带渗透高阶架构概念，而非照搬大学的罐头式概念教学。 - [01:16:15](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=4575s) 规范的“学习目标”必须使用可度量的主动动词界定学员结业时能做到的行为，其核心价值在于指导逆向课程设计并果断剔除冗余内容。 - [01:22:08](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=4928s) 根据认知负荷理论，训练营应将语言语法、专用编辑器与版本控制工具拆分为梯度进阶模块，避免新手因多维度未知信息交织而过载。 - [01:28:10](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=5290s) 为应对 AI 冲击，Jane Street 增设了为期三天的代码审查专项训练，让实习生研讨参考答案并相互交叉评审，以分担真实项目中的审查职责。 - [01:32:47](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=5567s) 课程设计需坚决抵制“氛围写代码”的开局诱惑，必须首先建立起严谨且高标准的确定性自动化测试套件作为 AI 协作的反馈护栏。 ## Detailed walkthrough ### [00:02](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=2s) 小行星、Python 与海滩上的退休教师 对谈在 Ron Minsky 与嘉宾 Aaron Bauer 之间展开。Aaron 目前是 Jane Street 的一名软件工程师，同时也是公司内为数不多但至关重要的“开发者教育者”（developer educator）之一。在探讨企业级技术培训之前，Ron 询问了 Aaron 是如何走上软件工程、计算机科学与教学这三者交汇的职业道路的。 Aaron 回忆自己的“起源故事”时提到，他从小就对计算机深感着迷，但编程曾一度让他望而生畏。转折点发生在他高三前的暑假，当时他参加了一个名字极为质朴的暑期项目——“暑期科学项目”（Summer Science Program）。在为期六周的集训中，学生们前往新墨西哥州，集中学习数学、天文学、物理学以及一些 Python 编程。他们的核心任务是用望远镜对一颗小行星进行实测观测，随后编写 Python 代码处理这些观测数据并计算出该小行星的运行轨道。尽管项目涵盖了众多引人入胜的理工科领域，但 Aaron 结项后的感受极为明确：“编程不仅远比其他内容有趣得多，而且完完全全就是我的本命。” 当时 Aaron 就读的高中位于一座小城镇，校内并未开设 AP 计算机科学课程。为了继续学习，他通过北卡罗来纳州一项对非本州居民开放的在线课程项目找到了修读途径。巧合的是，教授这门网课的老师当时并不在任何学校实体任教，而是在北卡罗来纳州的一处海滩上过着退休生活。不久后，在进入研究生院前的那个夏天，Aaron 以助教（TA）身份重回“暑期科学项目”。正是在那里，他第一次尝试为学生编写习题并讲授了一堂关于二叉搜索树（binary search tree）的微课。当看到讲台下的反应时，Aaron 彻底被教学所吸引，自此确立了将教学作为长远追求的想法。 ```mermaid flowchart TD accTitle: Aaron 早期编程与教学兴趣的发展路径 accDescr: 展示 Aaron 从高中参加暑期科学项目、在线学习 AP 课程到担任助教并确定教学志向的演进流程。 subgraph S1["编程启蒙"] A["参加暑期科学项目<br/>(望远镜观测 + Python 测算小行星轨道)"] --> B["确立编程核心兴趣"] B --> C["寻找外援：北卡海滩退休教师的 AP 计算机网课"] end subgraph S2["教学觉醒"] C --> D["大学毕业前夕担任暑期项目 TA"] D --> E["首次编写练习题并讲授二叉搜索树"] E --> F["确立计算机科学教学志向"] end ``` 谈及天文学、物理学在计算机早期历史中的纽带，Ron 指出早期计算机科学与程序语言的发展往往源于高端物理模拟。当今研究人员在处理望远镜海量数据时，依然会涉足极为硬核的计算机科学底层技术（例如定制 FPGA 等黑客级操作）。科学计算领域既存在推动技术前沿的惊人创新，也伴随着大量对规范计算机科学缺乏系统理解的混乱代码。Aaron 补充道，他在本科阶段就曾接触过用 Fortran 编写的撞击模拟程序，用于模拟小行星或彗星撞击地表；耐人寻味的是，这套代码最早脱胎于早期计算机时代模拟原子弹爆炸的程序，因为两者的底层物理模型极为相近。Ron 进一步补充，现代机器学习依赖基于梯度下降的反向传播，因而极其依赖自动微分（automatic differentiation），而该技术早在早期 Fortran 编译器中就已出现：那时的编译器能生成一阶导数程序，但过程混乱不堪，若试图对生成的导数程序再求一次导数，“那就只能向上帝祈祷了”。Aaron 总结道：“计算机科学在很大程度上是一种达到目的的手段……大家并非出于对编程本身的热爱而编程，仅仅把它当作工具，因为无论如何它都比算尺好用得多。” ### [04:05](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=245s) 研究生院、图形渲染与 Foldit 本科最后一学期，Aaron 选修了一门计算机图形学课程，立刻被渲染（rendering）技术深深吸引。他当时认定这是自己一生想要追求的方向，而能够一辈子专注单一领域的最佳途径就是去读博士。然而，当他申请并进入研究生院、开始与学界人士深入交流后，才发现学术界的渲染研究已经基本停滞——该领域经过数十年发展已极度成熟，剩余的前沿课题几乎全被工业界接管。 在学术惯性的推动下，Aaron 留在了研究生院，逐步转向游戏设计研究与计算机科学教育研究，最终的博士论文聚焦于“人机协同问题求解”（human-computer problem solving）。他的研究对象是知名的科学发现游戏 Foldit（一款众包蛋白质折叠游戏）。在 AlphaFold 问世之前，学术界试图通过游戏机制借助大众智慧破解蛋白质三维折叠结构。人机结合的关键在于优势互补：人类玩家具备宏观的几何空间直觉，能敏锐看出氨基酸链的大致折叠趋势，并尝试在宏观上翻转某些链段；而当时的计算机算法则擅长局部的微小微调（wiggle），通过局部搜索寻找能量最低的局部最优解。凭借这种协同模式，许多毫无生物化学背景的普通游戏玩家成功解析出实验室多年未能攻克的蛋白质晶体结构，后期玩家甚至开始在游戏中从头设计全新的蛋白质。 Ron 询问 AlphaFold 等前沿模型是否使用了来自 Foldit 的数据，Aaron 认为 AlphaFold 必然训练在所有已知结构数据之上，因此毫无疑问吸收了这些成果。Ron 敏锐地指出，这里本身就蕴含着极为深刻的教育学命题：如何让完全不懂特定领域的门外汉快速掌握领域规律，并在某种程度上将学习与解题过程游戏化。Aaron 赞同这一看法，并表示自己研究生阶段的研究带有浓厚的认知心理学色彩，其核心就是理解人类是如何思考、学习并应用技能的。 - **Foldit 人机协同模式**：人类负责大局几何直觉与宏观结构翻转；计算机负责细粒度抖动计算与寻找局部能量最优解。 - **科研转向**：从成熟饱和的图形学渲染研究转向人机协同、游戏化机制与计算机科学教育认知。 - **关于纯科研工具属性的判断**：Aaron 强调很多科研人员“不是因为热爱编程而写代码，只是因为编程比算尺好用”。 ### [06:43](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=403s) 卡尔顿学院、疫情冲击与翻转课堂 完成博士学业后，Aaron 明确希望寻找一份以教学为首要职责的工作，最终加入了明尼苏达州知名的文理学院——卡尔顿学院（Carleton College）。然而，他在该校的任教历程开局便充满波折：在线下实体环境仅讲授了两个学期，2020 年春季新冠疫情爆发，随后他不得不连续四个学期完全在网络上开展教学。 对于刚走上讲台的新教师而言，这是极度严峻的考验。Ron 询问卡尔顿学院以及 Aaron 个人是如何适应线上教学限制的。Aaron 坦承自己经历了一段试错期：起初，他直接借鉴了高中时那位海滩退休教师所采用的模式——全文字教学，不安排任何视频讲授，学生只需阅读文字材料并完成编程练习。当时校方建议考虑到许多学生居家隔离时网络带宽极差，纯文本模式具有最高的无障碍性（accessibility）。然而，学生很快反馈这种模式难以适应，体验极差。正如 Ron 总结的那样，这种方式“在某些维度上具有无障碍性，在其他维度上却极度不可访问”。 面对挫折，Aaron 迅速转向“翻转课堂”（flipped classroom）模式，结合同步在线课堂将学生拆分成小组展开协作。为了厘清术语，Aaron 向 Ron 解释了翻转课堂的定义：“所谓翻转课堂，就是不再把课堂时间用来听教授讲课，而是由教授提前录制好视频讲义，学生在上课前自行观看，进而把面对面的课堂时间用来做练习或展开讨论，开展更有互动性的活动。” 这种模式并非疫情期间凭空诞生，在教育学界已存在多年，但在全员被迫线上化的特殊时期迎来了爆发式的普及。翻转课堂的核心逻辑，正是将学生原本被动接收知识的单向灌输过程前置，把珍贵的师生同步互动时间最大化留给高阶实践与即时反馈。 ### [09:32](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=572s) 混合教学与“翻转课堂”的反思 在回顾教学模式时，Aaron Bauer 提到，翻转课堂（flipped classroom）的典型设想是将例题讲解等内容移出课堂，利用面对面的时间进行一对一辅导或主导讨论——即那些只有共处一室才能进行的事情。但 Aaron 自己的最终落脚点并非纯粹的翻转课堂，而是将面对面的时间转变为互动环节与直接讲授的混合体。 Ron 对此提出疑问，想知道 Aaron 在多大程度上认同翻转课堂背后的假设。Ron 坦言自己始终认为人们严重低估了传统讲座（lecture）的能量：讲座本质上是一种极具感染力的“现场表演”（performance），与讲者共处一室的真实在场感是录播无法替代的。尽管从纯理论上看，把世界上讲得最好的一堂课录下来让所有人看似乎效率更高，但面对真实的具体环境与人类心理，这种论调并不成立。正如 Ron 所评价的那样：“I do feel like it under—like the whole thing of like oh there's just like the passive lecture is just like missing something about the human psychology there.” Aaron 表示赞同，认为讲者在面前的现场表演对学生的注意力有一种聚焦作用，而且正因为是现场，学生更倾向于动笔记笔记以加深理解。但他同时强调，整整一个小时单向输出的讲座同样会磨灭学生的专注度，甚至适得其反。因此，在实体课堂中穿插“主动学习”（active learning）至关重要。 - **课堂主动学习的形式**：在讲解过程中随时停下抛出问题，让学生与同桌讨论，或者留出简短时间让学生独立解题以检验理解程度。 - **面对面表演效应**：讲者的实时呈现能天然建立起一种现场张力，将学生的注意力和认知状态锁定在当前讨论上。 --- ### [11:49](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=709s) 笔记的魔力与屏幕干扰 讲到这里，Ron 和 Aaron 探讨起“记笔记”这一行为看似怪异却极其高效的学习机制。Ron 指出，哪怕你走出教室后把笔记一把火烧掉、此生再也不翻看它，大量研究依然表明记笔记能显著提升知识留存率。Aaron 解释道，原因在于大脑必须先对听到的内容进行处理，将其提炼、蒸馏，再转化为手写符号；将概念具象化写下的动作，会在记忆中建立起某种质的差异，效果远胜于单纯在听懂层面的被动输入。 两人还谈到手写与打字的差别。虽然研究多倾向于手写优于打字，但 Aaron 指出其在教学一线的体会或许与输入介质本身关系不大，核心在于打字往往意味着面前立着一块屏幕。屏幕连接着人类创造的所有信息与分心诱惑，直接横亘在学生与讲授者之间。Aaron 为此甚至刻意避免在计算机机房（computer lab）授课，因为一旦眼前有了屏幕，“哪怕一半的学生内心极想集中注意力，他们也根本做不到”。 ```mermaid flowchart TD accTitle: Screen Distraction vs Pen and Paper Focus accDescr: Comparison between screen-based learning which leads to distraction and trial-and-error, versus paper-based learning which forces deep cognitive processing. subgraph Screen["屏幕环境（机房 / 电脑）"] S1["大量信息与外界诱惑干扰"] --> S2["把写代码当成试错游戏"] S2 --> S3["任意微调代码直至能跑通<br/>（缺少深度理解）"] end subgraph Paper["离线环境（纸笔 / 面对面）"] P1["移开物理屏幕与外界干扰"] --> P2["提炼、蒸馏与手写记录"] P2 --> P3["强迫推演代码执行语义<br/>（形成稳固心智模型）"] end ``` Ron 对此深有共鸣，并分享了他教自己孩子编程时的家庭趣事。在餐馆等上菜时，为了哄孩子打发时间，他会拿出一张纸写下一小段使用混淆变量名的 Scheme 程序，让孩子们推演“函数 `flub` 到底干了什么”。Ron 强调，一旦把电脑摆在他们面前，孩子们就会陷入“让电脑跑出结果”的操作游戏，而不再思考背后的求值机制；把计算机挪开，反而更容易讲透真正的概念。Aaron 也证实，初学编程的人极易产生一种倾向：将代码视作给机器的神秘符咒，不断做细微扰动（perturb it arbitrarily）直到撞出正确结果。唯有让学生在纸上写代码并自行判断正确性，才能逼迫他们真正思考程序运行的内部机理。 关于初学者依赖屏幕试错的现象，Aaron 总结道：“And there is definitely a thing that people just learning programming will tend to do, which is this is kind of arcane instructions for the machine. I will perturb it arbitrarily until it does what I want.” --- ### [14:56](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=896s) 课后沉淀：录播定位与掌握学习 当疫情封控结束、重新回归 Carleton College 的实体课堂后，Aaron 将线上教学期间积累的有效工具带回了日常教学中。其中一项长期保留的做法便是自费且低成本地录制现场讲座。由于 Carleton 的常规教室没有现成的录像硬件，Aaron 每次上课都随身携带装在三脚架上的网络摄像头，插在笔记本上录制并上传至网络。 - **录像的作用机制**：尽管某些学校担心录课会导致到课率下降，但在 Carleton，学生依然愿意出勤。录制的主要价值在于灵活答疑——当学生因为田径比赛等事务请假，或在课后提问时，Aaron 可以直接发送带有精确时间戳（timestamp）的录像链接：“你看，我就是在这里解释这个问题的。” - **即时自动化反馈（Automated Feedback）**：取代“交作业—两周后助教批改—拿到已淡忘的反馈”的滞后链条。通过多选题每周自测、填空题，以及提交代码后立即运行测试用例的自动化判题软件（autograding software），使学生获得即时诊断。 - **掌握学习（Mastery Learning）**：借助自动化评估工具，学生可以在通过之前进行任意多次尝试，根据反馈反复迭代，直至证明自己确实掌握了该知识点才进入下一阶段。 针对这种“掌握学习”模式，Ron 分享了自己的反思体验。他提到在大学攻读高阶数学课时，常常感觉自己只是在勉强跟上、大脑处于半懂不懂的混乱状态；然而六个月之后，当下一届同学来请教相同知识点时，他却能豁然开朗、娓娓道来。Ron 认为，对概念感到极度困惑与在局部达成“掌握证明”，往往是学习过程中并存的状态。Aaron 对此表示赞同，指出严格意义上的“掌握学习”（如必须完全掌握长除法或加法算法才能进入下一单元）通常更适用于低龄阶段的自节奏（self-paced）基础数学教学；在大学等高难度课程中，由于课时与进度弹性有限，往往只能将其作为针对特定知识点的辅助手段，而无法全盘照搬。 ### [19:08](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=1148s) 掌握学习中的防刷题设计与 LLM 即时反馈的潜力 在讨论大学教学中的评估体系时，Aaron Bauer 解释了他所设计的测验机制。虽然他主张“掌握学习”（mastery learning）——即学生应不断迭代直到通过，但这种迭代必须发生在学期中的固定时间点内。他指出：“学生的拖延症是真实存在的；如果放任大家把所有任务堆积到学期末，那反而是害了他们。”因此，教学上必须设置阶段性截止日期，强迫学生按部就班推进。为了落实这一理念，他每周安排一次可重复作答的短测验，成绩按所有尝试中的最高分计算。为了防止学生用穷举法（brute-force）试探答案，题目和选项的顺序每次都会随机打乱，让死记硬背或投机取巧变得极为烦琐，学生最终会发现“老老实实弄懂全部知识点直接拿满分反倒最快”。 ```mermaid flowchart TD accTitle: 掌握学习测验机制与防刷题设计 accDescr: 展示学生在固定截止日期内多次尝试测验的闭环流程，以及随机化如何促成真正的知识掌握。 subgraph Iteration["周测验闭环 (固定截止期内)"] A["开始/重试周测验"] --> B["题目与选项顺序随机打乱"] B --> C{"作答并提交"} C -->|"穷举/盲猜"| D["耗时极长且繁琐"] C -->|"理解概念"| E["取得 100% 满分"] D --> A E --> F["记录历史最高分并按时推进"] end ``` 谈及当前的人工智能技术，Ron 询问这类对学生至关重要的“即时反馈”未来能否由 LLM 来自动生成。Aaron 对此持积极态度，认为当反馈的范围被严格框定时，LLM 的潜力尤为巨大。例如，在代码规范（code style）这类结构化更强的领域，LLM 可以很好地配合自动化测试用例和传统 Linter，在规整语法之外为学生提供更具可读性的代码风格建议。但他同时也给出了审慎的提醒：在实际使用中，LLM 经常会出现误报（false positives），给出极具误导性的建议。教师若要引入 LLM 辅助反馈，就必须谨慎设定提示词和使用边界，或者想方设法压低模型误导学生的概率。 - 测验反作弊设计：题目顺序与选项顺序均动态随机排列，从机制上瓦解穷举尝试的可行性。 - LLM 教学潜力：在代码测试用例评分、Linter 静态检查的基础上，叠加 LLM 进行代码风格审阅。 - LLM 反馈风险：存在不可忽视的误报率与误导性反馈，直接面向学生使用时需极为谨慎。 --- ### [21:29](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=1289s) 发明“开发者教育者”：50/50 拆分的特殊岗位 谈及如何从明尼苏达州的大学讲台来到 Jane Street，Aaron 分享了这段充满机缘巧合的经历。当时他的哥哥和嫂子搬到了纽约布鲁克林（嫂子来自湾脊区 Bay Ridge），为了与关系亲密的哥哥离得更近，他也决定搬往布鲁克林。这意味着他无法继续在明尼苏达教书，而他虽然喜欢学术界的部分特质，却早已做好尝试新事物的准备，且不想再经历折磨人的学术求职市场（academic job market）。就在他打算转行去当一名普通软件工程师时，恰逢 Jane Street 正在招聘一个极其罕见的复合岗位——“开发者教育者”（developer educator），二者一拍即合。 Ron 从公司的角度详细阐述了设立这一岗位的初衷。Jane Street 长期以来高度重视内部教学，一方面是因为交易业务专业壁垒极高，新人入职时对量化交易普遍缺乏认知，需要系统性培训；另一方面是因为公司的技术栈独树一帜——使用自成一体的编程语言、开发工具和版本控制生态，外部开发者完全无法开箱即用。过去这些教学工作全靠一线工程师兼职完成，但随着规模扩大，公司意识到需要引入具备长期专业教学经验的人才。然而，Jane Street 坚决反对设立纯粹脱离工程一线的“专职讲师”。 Ron 强调，新岗位的理想形态必须是“50/50 的双重角色”：一半时间用于编写代码，另一半时间投入教育工作。这半数教育时间包括直接开课、编写教学大纲，以及指导并帮助内部其他工程师提升教学技巧。Ron 解释道：“我们实际上不希望由少数几个开发者教育者包揽所有教学任务，而是希望通过他们来壮大这一实践，让更多人更擅长教学。” > “在最理想的情况下，我们真正想要找到的是这样的人：他们是极其出色的教育者，热爱这门手艺，倾注了大量时间钻研，并在该领域勇于尝试突破；同时，他们又必须是完全合格、货真价实的 Jane Street 软件工程师——而这两者的交集其实相当狭小。” —— Ron Ron 坦言最初的招聘过程令人沮丧：“我原本以为这是一个显而易见棒极了的职位，只要把招聘广告打出去，优秀的人选就会从天而降，因为这个机会实在太酷、太独特了。但事实证明，只要我们发广告的时间差上三个月，我们可能就根本遇不到 Aaron。”Aaron 也分析指出，这类复合人才之所以极度稀缺，是因为具备丰富教学经验的人大多深耕学术界，热爱学术并无意转向工业界；即便大学教授们在教编程，平日里也往往不从事高强度的工业级软件工程，甚至对把大量时间花在日常写代码上毫无兴趣。 --- ### [26:48](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=1608s) 内部技术变革与教育需求的持续膨胀 尽管招聘难度极大，但该角色的设立在内部取得了显著成功。Aaron 认为其关键在于“拥有将教育作为核心主业的专门人员，这在持续投入与关注度上带来了阶梯式的质变（step change）”。然而随之而来的现实是，公司内部需要被讲授的技术内容不仅没有减少，反而像滚雪球一样不断激增。 教育需求激增的核心原因在于 Jane Street 的技术底座始终在高速演进，教学对象早已不局限于刚入职需要学习 OCaml 的新人： 1. **语言底层的演进**：Jane Street 正在对 OCaml 语言本身进行深度魔改与扩展（例如引入 OxCaml 等特性），这意味着“公司里的每一个人，无论资历多老，都需要重新学习这门语言”。 2. **新兴技术与工具链**：伴随新型 AI 编程工具与大模型的涌现，全员都需要接受系统培训，掌握如何高效利用这些工具。 3. **技术栈边界的拓宽**：除了核心的 OCaml 之外，团队如今也大量采用 Python，必须让工程师们对 Python 保持系统理解，避免思维“被单一工具所束缚”。 基于这些迫切需求，Jane Street 正在进一步扩充开发者教育团队，并拆分出多个纵深方向。除了成功招揽到具备深厚编程语言（PL）背景的“OxCaml 教育者”之外，公司仍在积极物色负责前沿模型构建与训练的“机器学习教育者”（Machine Learning Educator），以及覆盖更广泛工程体系的通用教育者岗位。教育专业度在公司内部的推进，使得工程团队对高质量专项培训的胃口变得越来越大。 ### [29:03](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=1743s) 语言多样化、开发教育者角色与组织嵌入模型 随着公司技术栈逐渐扩展到包含两门、三门乃至四门编程语言，工程师队伍中自然出现了专业分工与技术层化（striation）。Ron 明确表示，自己并不希望公司被分割成狭隘的单一语言程序员群体，而是推崇真正的通用工程师素养：“I'd really love to be more in a world where we there are no like OCaml programmers and Python programmers, they're just like software engineers and they know how to use many tools and can figure out what's the right tool to use given the particular circumstance.” 现实情况是，虽然几乎所有人都懂得一点 Python，但能达到精通水准的人并不多，需要系统化地教学；此外，公司内部还在某些特定领域自研了专有编程语言。技术覆盖面的急速扩大，为技术培训创造了实打实带来业务价值的广阔空间。 针对这种日益膨胀的教学需求，Aaron 提出了“开发教育者”（Developer Educator）未来在组织架构中的嵌入模式。他借用设计团队（designers）的先例指出，未来教育者不会孤立存在，而是像设计师一样分散派驻到不同的业务团队中。例如在当前的 AI 辅助工具团队中，就迫切需要一位常驻教育者来专门负责相关的培训工作与资料产出。Ron 补充指出，这种派驻模式具有双重收益：一方面能精准对接特定领域的教育痛点；另一方面，如果把所有教育者都塞在同一个角落，会导致团队经验单一脆弱，而让教育者轮岗、深入不同的技术场景，能帮助他们积累多样化的技术背景，建立起更为健壮扎实的知识体系。 ### [31:08](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=1868s) 编译器与编辑器团队，以及 Jane Street 的自研开发工具链 Aaron 随后谈及自己在 Jane Street 承担的具体工程研发职责。他隶属于工具与编译器团队（Tools and Compilers Group）旗下的编辑器团队（Editors Team）。该团队的核心任务是维护全公司日常写代码所用的各类主流文本编辑器（VS Code、Emacs、Neovim），以及背后的核心配套基础设施，包括代码索引服务（code indexing service）和 OCaml Language Server。Aaron 个人在开发者工具领域的工程经历十分广泛，早期曾专注于为公司的核心特性管理系统 Iron 开发扩展，以简化代码审查切分流程。 在 Jane Street 的术语体系中，“feature”是一个核心概念，其功能大致对应外部世界的 Commit 或 Pull Request（PR）。Aaron 开发的工具允许工程师轻松将一整块庞杂的修改拆分成互相链接的逻辑链条（chain of relationships），并分别发起审查。Ron 指出，Jane Street 过去 25 年仿佛在一场开发工具的“平行宇宙冒险”中前行，形成了一套极其独特的工具取向： - **Iron 与外部 PR 工具的优劣差异**：在支持堆叠 PR（stacked PRs）这一底层机制上，Iron 的表现极其出色且浑然天成；但在过去，针对 Diff 本身进行交互式切分、来回挪动代码的过程却异常缓慢、痛苦且极其不友好，体验甚至远落后于外部开源世界中 Git 配合 Emacs Magit 的工作流。 - **技术路径分化**：朝着同一个宏观工程目标，往往能走出截然不同的技术路线，最终沉淀出完全不同的功能集与工程权衡。 ```mermaid flowchart TD accTitle: Feature and Code Review Splitting via Iron accDescr: Diagram illustrating how a large monolithic changeset is split into a dependency chain of distinct review units in Jane Street's Iron workflow. subgraph LargeChange["原始大型变更"] direction TB Monolith["包含配置、底层接口与业务逻辑的庞大代码改动"] end subgraph IronTool["Iron 切分扩展 (Aaron 早期研发)"] direction TB SplitAction["将改动解构为微观 Diff 节点"] end subgraph StackedFeatures["链式独立代码审查 (Stacked Features)"] direction TB F1["Feature A: 基础数据结构与接口"] F2["Feature B: 核心引擎逻辑实现"] F3["Feature C: 外部调用与配置更新"] F1 --> F2 F2 --> F3 end Monolith --> IronTool IronTool --> StackedFeatures ``` ### [33:56](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=2036s) 将文本编辑器作为大本营的研发与审查文化 不同的底层技术系统从根本上塑造了其上生长的工程师文化。Ron 抛出一个尖锐问题：既然市面上已有现成的开源文本编辑器，为什么 Jane Street 还要养一支专门的编辑器团队？Aaron 解释称，Jane Street 将文本编辑器确立为软件工程师的核心“大本营”（home base）。代码审查、版本控制以及周边系统的几乎所有交互都必须收敛在编辑器内完成，完全依赖键盘导航，而不需要在浏览器、命令行和桌面应用之间频繁来回横跳。由于公司底层采用的是 Iron 而非标准的 Git，必须依靠专门团队将这些独特系统无缝集成进编辑器。 不仅如此，Ron 对比了主流大型科技公司（Big Tech）将所有开发、审批与反馈动作全盘搬上浏览器 Web UI 的做法，指出 Jane Street 是一个“去 Web 化”（much less web-focused）的工作环境。编辑器不仅是写代码与交互的场所，更是团队沟通代码的唯一中枢： - **CR 内联注释规范**：在代码审查中，工程师不会去网页上点击某个提交的 Review 按钮，而是在编辑器内的代码行直接敲入特定格式的注释（CR）。 - **协同就地修改机制**：因为审查者始终留在编辑器这一生产环境中，比起留下长篇大论的 CR 评语，很多人会顺手直接修改被审阅的代码。针对这种与外部隔离代码权限完全相悖的文化，Ron 评价道：“It's like someone else is reaching in and modifying my PR. It feels unsanitary, but it just like for good or ill or maybe as one should say for good and ill. It's like just like a different modality, a different way of interacting within thinking about code and the editors like just have a really central piece in all of that.” ### [37:47](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=2267s) 规模化难题：巨型 Diff 与 VS Code 的性能瓶颈 在技术实现层面，将现有开源编辑器改装成满足 Jane Street 高强度工作流的大本营面临巨大的工程挑战。编辑器原本是为外部常规开发场景设计的，面对公司的特定工业规模往往会出现严重的适配问题。 Aaron 举例说明团队正在 VS Code 中排查攻坚的棘手案例：公司业务中经常出现尺寸极其恐怖的“巨型 Diff”（giant diffs），例如数十万行的超大配置文件或极为庞大的 OCaml 源文件。当遭遇如此规模的代码变动时，VS Code 内置的 Diff 算法会出现极其严重的性能崩溃与假死（performance problems），暴露出通用工具在极端工程规模下的底层架构局限。 ### [38:38](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=2318s) 规模化难题与编辑器作为核心枢纽 在大型代码库中，开发工具极易在极端规模下暴露出脆弱性。Aaron Bauer 讲述了一个典型的破坏性场景：当工程师试图在编辑器中打开一个超大 diff 时，界面发生卡死；如果用户强行关闭它，底层的 diff 查看队列就会被彻底锁死，导致后续所有的 diff 都无法查看。此外，内部系统与外部工具的深度整合也会打破原有软件的底层假设。例如，简街（Jane Street）将自研的代码审查系统 Iron 集成进 VS Code 时，破坏了 VS Code 对“一个 buffer 是什么”以及“buffer 的 URI 应该长什么样”的内置假设，导致编辑器的全局搜索功能出现离奇故障，团队不得不逐行排查并修复。 除了编辑器前端，编辑器背后的共享基础设施（如 OCaml 语言服务 `ocaml-lsp` 以及提供类型查找与跳转定义的底层系统 Merlin）同样面临严峻的性能挑战。Aaron 举例说明：曾有工程师为了在全仓库（monorepo）索引中更精准地查找模块，在编译器生成的文件中夹带了额外的元数据信息，结果导致索引进程暴涨数个吉字节（GB）的内存，以至于在开发机上根本无法完成构建。正如 Aaron Bauer 所总结的：“我意识到我们试图解决的大量问题，都与我们在简街所做事情的规模有关，我们需要让这些通用工具在极其特殊的用例下依然保持优秀的性能。” 针对这一现象，Ron Minsky 指出编辑器在开发体验中扮演着绝对的核心枢纽地位： > “编辑器拥有这种核心地位，它是许多其他事物触达用户的界面。因此，你遇到的问题可能出在类型查找系统或索引系统上——这些东西从字面意义上讲并不属于编辑器本身，但编辑器却是你看到问题的地方。” 编辑器团队因而被推向了整个开发栈的各个角落，既要修复代码审查系统的问题，又要优化 OCaml 底层语言基建，而近期 Python 工具链的扩张又带来了全新的复杂性。 ```mermaid flowchart TD accTitle: 编辑器中心架构与问题传导关系 accDescr: 展示下游语言和代码库设施的问题如何最终在编辑器界面集中爆发。 subgraph Base["底层系统与数据规模"] A["单体大仓库 Monorepo 规模"] B["编译器与 Merlin 类型查找元数据"] C["Iron 代码审查系统"] end subgraph Integration["中间件与通信协议"] D["OCaml 语言服务器 (ocaml-lsp)"] E["Buffer / URI 抽象映射层"] end subgraph EditorHub["开发者前台：编辑器"] F["VS Code / Emacs / Neovim 界面"] G{"用户可见故障"} end A --> D B -->|"元数据膨胀 / 内存耗尽"| D C -->|"非常规 URI 结构"| E D --> F E -->|"破坏搜索等原生假设"| F F --> G ``` --- ### [40:41](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=2441s) Python Notebook 的内部分裂与自研破局 随着交易与研究业务对 Python 依赖的急剧增加，编辑器团队感受到了全新的冲击。表面上看，业界普遍使用 Python，复用现成生态理应十分轻松，但现实却截然相反。在量化交易和投研部门，核心交互工具是 Python Notebook。外部开源社区虽然存在多种 Notebook 方案，但普遍存在严重的缺陷或局限性。内部团队往往希望引入某种新方案来替代旧方案，但新工具总是缺少旧工具的某项关键功能，最终导致团队不得不同时维护多个互不兼容的 Notebook 系统。 这种碎片化直接伤害了工程协同与代码共享。研究人员在 Notebook 中撰写的分析逻辑，往往需要抽离并沉淀到单体仓库的 Python 共享库中，以便供其他 Notebook 导入复用。为了实现这种平滑过渡，理想的工作流应当是让用户统一停留在 VS Code 这类环境中，既能写 Notebook 又能直接编辑普通 Python 文件。然而，VS Code 在处理简街特有的 Notebook 工作流时，频频遭遇严重的性能和可用性瓶颈。 面对外部工具的反复受挫，简街开始启动自研 Notebook 工具项目。这一决策折射出团队对外部方案与内部控制权的深层权衡： > “外部世界的所有 Notebook 工具都存在一堆问题。我们现在的态度是：我们要用 OCaml 编写一个轻巧美观的东西，结合我们自己的 Web UI 技术栈，让它平滑运行。我们目前的原型非常不错，我对它感到相当乐观——落地难度不算太高，同时能带给我们极高的控制权，使我们能够完全根据内部极其独特的诉求去定制 Notebook，并提供更好的体验。” --- ### [43:30](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=2610s) 上游标准化与双向技术拉锯：Emacs、ECaml 与跨语言并发困境 在构建内部工具时，简街时刻警惕“全盘自研”的过度工程化陷阱。Ron 强调，在编辑器领域实际上存在着截然相反的双向张力：一方面，对于外部严重不适用的领域（如 Notebook），团队果断推进 OCaml 自研；另一方面，团队正投入大量资源推进“去怪异化”（deweirdifying），系统性清理过去 25 年间在 Emacs 中积累的历史包袱和特化补丁，主动拥抱标准开源方案。 Aaron 指出，借力开源上游的好处是显而易见的： > “能够不写的代码就是最好的代码。不仅是你不用写，而且还有大量其他人在使用它并报告问题，那些问题也在被修复。因此我们不必通过自己踩坑去发现所有缺陷。” 然而，简街特有的技术栈带来了深层次的技术张力：团队习惯使用 OCaml 来为主流编辑器编写扩展插件（如 Emacs 中的 ECaml，以及针对 Neovim 和 VS Code 的 OCaml 绑定），而非纯粹使用宿主编辑器的原生语言。在 Emacs 中，原生 Elisp 与编译出的 ECaml 之间的交互边界极其别扭，尤其是在异步与并发场景下，极易引发双方互相等待的死锁问题。 - **并发与运行时缺陷**：Emacs 和 Vim 属于历史悠久的老牌工具，最初的架构设计并未考虑现代复杂的异步交互。上游 Emacs 社区虽在垃圾回收器（GC）和异步机制上持续演进，但相关特性尚未稳定合并到当前简街所采用的稳定版本中。 - **跨语言状态协同**：将两种具备独立运行时和并发原语的语言深度绑定（如 Elisp 与 OCaml、Python 与 OCaml）在架构上极为棘手，团队中长期的重构目标是厘清调用边界，彻底杜绝死锁。 面对跨语言绑定的痛点，对话随即转向了对现代工具链产生深远冲击的新变量——真正可用的人工智能 Agent（Agentic AI）。尽管 AI 正在全面重塑编程工作流，Aaron 依然明确表态：这些变革并未动摇团队的核心信念，文本编辑器依然是软件工程师不可替代的“大本营”（home base）。 ### [49:07](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=2947s) 编辑器角色的重塑：代码编写减少与代码阅读激增 随着智能体与 LLM 深度融入开发工作流，编辑器中正在形成全新的操作模式。然而，编辑器原有的核心支柱并没有消失，例如查看 Diff、进行代码审查（Code Review）以及阅读并理解代码。正在发生根本性转变的是工程师编写代码的方式。无论是整个软件工程界还是在 Jane Street 内部，许多工程师在日常工作中已经极少亲自敲出大段代码；他们的键盘输入越来越偏向提示词（Prompts）。但与此同时，工程师们依然频繁地在编辑器中阅读代码、检查类型定义和查阅变量与函数的定义。正如 Aaron Bauer 所总结的那样：“他们依然在做许多在编辑器里做的事情，只是单纯的代码编辑变少了。” 这种由编写到阅读的重心转移，给负责打造内部编辑器工具的团队带来了巨大的不确定性与困惑。大家最常面临的疑问是：在这个时代，究竟什么工具才是有价值去构建的？今天投入心血打造的功能，一年之后是否还会有人使用，抑或是会被全新形态的 AI 范式完全颠覆？ ### [50:33](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3033s) 利用遥测数据验证假设：跳转定义的意外上升 为了不再凭空猜测开发者的使用习惯，Jane Street 的编辑器团队近期大力投资了遥测（Telemetry）和可观测性基础设施，用以监控用户在编辑器中的具体行为以及他们与 OCaml Language Server（LSP）的交互细节。Aaron 针对开发行为的演变提出了一个直观的假设：如果大模型生成代码的质量很高，工程师不再需要自己写底层逻辑、查阅繁琐的 API，那么人均“跳转至定义”（Jump to Definition）的调用次数理应显著下降。 然而，遥测系统的真实数据却给出了完全相反的答案：工程师人均使用跳转定义的频率不仅没有减少，反而略微上升了。这一反直觉的结果揭示出一个关键事实：正是因为 LLM 能够瞬时生成大量结构复杂的逻辑，工程师被迫需要阅读和理解比以往更多的陌生代码。数据驱动的方法因此变得至关重要，它能直接指导工程资源的投入方向。比如，投入大量工程精力将跳转定义的响应速度再提高一个量级是否值得，完全取决于实际数据表明开发者到底是更频繁还是更少使用它。 ```mermaid flowchart TD accTitle: AI代码生成对开发者行为的影响分析 accDescr: 对比直觉假设与实际遥测数据，展示开发者在面对LLM生成代码时的真实交互路径 subgraph Hypothesis["直觉假设 (Theorizing)"] H1["LLM 生成高质量代码"] --> H2["工程师写代码/查API变少"] H2 --> H3["跳转定义使用率下降"] end subgraph TelemetryReality["遥测数据揭示的现实 (Telemetry Data)"] R1["LLM 极低成本生成大量代码"] --> R2["审查与理解陌生代码激增"] R2 --> R3["跳转定义使用率反而微升"] R3 --> R4["理解代码成为核心瓶颈"] end ``` ### [51:53](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3113s) 代码验证的困境与“平滑幻象” Ron Minsky 极为赞同使用真实数据指导工具建设的策略，而不是坐在办公室里凭空构想。他借用 Charlie Marsh 的观点指出了当下软件工程的核心痛点：“生成一个看起来合情合理的 PR 的成本几乎已经降到了零，但验证它的成本却维持不变，甚至变得更糟了。” 这一现象的危险之处在于 LLM 生成代码所带有的“不可思议感”（Uncanny）。这些由模型写出的功能表面上极其顺滑（Smooth）、排版美观且没有常规程序员写代码时留下的典型逻辑破绽或粗糙痕迹，但其深层逻辑很可能是完全损坏甚至存在隐蔽致命 Bug 的。这使得代码审查从根本上变得更加困难。Ron 强调，尽管可以通过提示词让 LLM 充当强大的解释工具，但绝不能低估那些传统的“硬核”程序分析能力，诸如基于编译器的类型推导（Inferred Types）、代码跳转、确定性测试结果等，这些才是在虚幻代码中提取事实的压舱石。 ### [53:27](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3207s) 交互范式反思：Claude Code 的学习模式与心智模型构建 面对理解代码的瓶颈，Aaron 开始从开发者与模型交互的源头寻找解决方案。他分享了自己在教学和日常工作中对 Claude Code 的“学习输出风格”（Learning Output Style）的探索。该模式通过调整系统提示词，让大模型在写代码的过程中主动停下来，在代码中留下一处 `TODO: human` 注释，并要求人类去补齐缺失的这一逻辑片段。 Aaron 发现，这种强迫人类在半途介入的交互体验意外地出色。工程师必须先阅读并理解模型在此之前写下的全部上下文，才能写出正确的片段。相比于让 LLM 一口气把整个工程全部生成完再去通读最终产物，这种共同构建（Co-authoring）的过程不仅体验更愉悦，而且能让开发者在最终产出时对整体代码拥有更加扎实、清晰的心智模型（Mental Model）。 - **手写笔记效应的再现**：Ron 联想到之前讨论的用铅笔手写笔记的认知价值——只有身体力行地参与到生产过程中，大脑才能真正内化知识。 - **当前工具设计的错位**：目前几乎所有 AI 辅助工具的交互都偏向于单一模式——“模型全部生成，人类审核接受/拒绝”，这一范式很可能从认知学角度是次优甚至错误的。 - **心智模型的构建机制**：Aaron 在面向 Jane Street 实习生的分享中提到，自己写代码之所以容易理解，是因为在边写边排错的过程中逐渐修正了心智模型；而代码审查之所以痛苦，是因为面对的是别人端出的既成事实（Finished Result）。大模型生成全量代码在本质上迫使开发者直接进入“痛苦审查模式”，严重阻碍了心智模型的形成。 ### [56:22](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3382s) 代码审查的速率限制步：可视化与预计算分块解释 生成代码的成本崩塌带来了全新的开发生态，也彻底重构了工程师的时间分配。在当前的开发日常中，审查和理解代码已经成为了整个软件研发周期中最为显著的“速率限制步”（Rate Limiting Step）。为了破解这一瓶颈，两位工程师讨论了利用模型能力重塑审查工具的几种创新方向： 首先是**即时生成的自定义可视化（Custom Visualizations）**。在过去的工程实践中，为了协助他人理解一个代码补丁（Diff）而专门编写一个专用的交互式可视化工具是极其奢侈且不切实际的；但在大模型时代，开发者可以在提交 Pull Request 的同时，让模型针对该补丁的逻辑即时生成一个专用的可视化展示面板，并在代码审查过程中直接渲染出来。 其次是**预计算的代码分块解释（Pre-computed Chunk Explanations）**。虽然工程师可以在编辑器中手动划词并让模型进行解释，但这伴随着上下文切换与等待延迟的摩擦力。Aaron 构想了一种机制：在功能提交时，系统已提前将代码切分为逻辑块，并为每个块预先生成精准的语义解释。审查者在编辑器中只需按下单个快捷键，就能瞬间呼出当前逻辑块的上下文解释，从而成倍提升消化海量待审查代码的速度。 随着讨论的深入，Ron 指出这一切关于代码开发与理解的剧烈变化，正在直接撞击工程教育的核心领域。既然大模型可以在几秒钟内完成课程作业并生成看似完美的实现，技术导师又该如何在 AI 浪潮的中心地带重新规划培训项目与学习目标？这一挑战构成了当下极具张力且棘手的新课题。 ### [59:27](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3567s) 测试研习营与“自己动手写一次”的价值 Aaron Bauer 介绍了 Jane Street 面向早期职业工程师的培训体系，其中一个核心形式被称为“研习营”（teach-in）。本次讨论聚焦于软件测试研习营，该课程主要教授工程师如何使用各种专用库进行“基于属性的测试”（property-based testing）以及在测试中实现对时间的确定性控制（deterministic control of time）。然而，面对大语言模型（LLM）几秒钟内就能写出调用这些复杂库的代码这一现状，教学团队必须反思：工程师真正需要在实践中掌握的核心概念究竟是什么？哪些只是可以被工具替代的样板代码（boilerplate）？ Ron Minsky 指出，大模型固然能够极其轻松地生成初始脚手架代码（initial scaffolding），但教育的关键在于“你必须传递给人们的理解究竟是什么”。Aaron Bauer 明确表示，他们目前得出的结论是：学习者亲自深入思考这些问题并动手实现至少一次，仍然是建立充分理解的关键前提。Aaron Bauer 解释道： > “And even if what they are doing subsequent to that is reading applications of this technique initially authored by an LLM, building that understanding by having done it themselves once is sort of a critical input to then being able to evaluate those things.” 教学团队目前正在深入思考如何在教学中划定界限：过去要求工程师做的某些琐碎、繁重且机械的任务，如今确实应当剔除；但更具挑战性的问题在于，如何系统性地教会人们有效地将这些 AI 工具融入日常工作流，并敏锐地察觉它们何时能提升效率、何时会带来误导。由于工具迭代极为迅猛，业内至今尚无定论。 ```mermaid flowchart TD accTitle: LLM 时代技术学习的双层模型 accDescr: 展示工程师通过亲自实现底层机制建立心智模型，进而具备审查和评估 LLM 生成代码的能力闭环。 subgraph DeepUnderstanding["核心理解建立阶段"] A["亲自思考测试边界与时序逻辑"] --> B["完全动手实现一次 (No LLM)"] B --> C["建立扎实的心智模型与评估基准"] end subgraph PracticalEvaluation["工作流应用与审查阶段"] C --> D["使用 LLM 生成脚手架与样板代码"] D --> E{"人工审查与代码评估"} E -->|"发现幻觉或隐蔽缺陷"| F["修正 Prompt 或介入重构"] E -->|"逻辑严密且符合规范"| G["合并入生产代码库"] end ``` --- ### [01:01:59](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3719s) 剥夺便利的教学法与“空集困境” 谈及 AI 带来的副作用，Ron Minsky 表达了强烈的忧虑。他观察到一些人已经开始陷入“过度依赖氛围感”（vibe it up too much）的陷阱——纯靠模型生成代码，自己完全放弃思考。更广泛地看，整个教育体系正面临结构性危机，因为学习的本质包含与困难材料的搏斗（struggling with the material）。当身边时刻存在一个有求必应的“神灯精灵”时，深层认知负荷便被轻易卸载了。 Ron Minsky 提出了当前技术教育面临的“空集困境”：那些既简单到适合初学者上手、又困难到 LLM 无法一次性直接生成完美答案（oneshot）的编程问题，如今几乎已经变成了空集。他联想到早期不借助电脑、纯靠纸笔推导算法的教学模式，强调教学的关键往往在于主动“扣留某些便利条件”（withholding certain affordances）： > “Sometimes an important thing about making the educational process work is withholding certain affordances. And I worry that we have like a whole generation of people who are like learning in this new environment where we don't know what to do.” Aaron Bauer 对此深有共鸣。他透露 Jane Street 在研发新课程时多次踩中这个陷阱：为了避开 LLM 的一步解答，他们设计的任务往往过于宏大，导致学员必须花费大量时间去调试和修改模型生成的代码才能达到预期，这在有限的培训课时内完全不可行。未来教育的必然趋势是在课程中人为设置分界线：某些环节完全禁止接触 LLM，另一些环节则针对性地训练人机协作。 - **空集困境（The Empty Set Problem）**：适合人类初学者的题目难度与大语言模型无法一次性解决的题目难度之间的交集正趋近于零。 - **课时超限矛盾**：试图设计大到足以让模型出错的项目，往往会压垮有限的课堂时间预算。 - **打字与编码直觉的断层**：未来的学员群体可能在进入职场前就没有积累过多少亲自敲击代码的经验，这种底层学习机制的缺失可能会彻底重塑教育需求。 --- ### [01:05:16](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=3916s) 苏格拉底式 AI 助教与智能辅导系统 针对 OCaml 训练营（OCaml Boot Camp）的新人，Jane Street 内部也在激烈讨论是否应该提供类似 Claude Code 学习模式（learning mode）的工具。如果让 LLM 替新人包揽一切代码编写，学员将一无所获；但如果完全与世隔绝，又脱离了日常工程实践。理想的折中方案是设计特定的提示词约束或模式，让模型扮演苏格拉底式的引导者——只搭建骨架代码，通过提问来检验学员的理解，或者在卡壳时提供启发性提示（hints），而不是越俎代庖直接给出最终答案。 Aaron Bauer 回忆起自己在研究生阶段所做的科研项目：当时他专注于开发一种自动化智能导师（automated tutor），该系统能够引导学生逐步推导任意示例算法： > “And with the idea that sure you had the one hour where you could go talk to a TA and get one-on-one help and that was the most effective thing you could do, but if you were at other times studying for an exam, it might be really useful to have this help like not as effective but still helpful sort of tutor that is infinitely patient and can walk you through any example.” 两人一致认为，这种“不知疲倦且拥有无限耐心”的辅助教学模式，虽然单次交互效果未必比得上一对一的人类助教（TA），但其随叫随到的特性将成为未来高等教育和企业内训的核心基石。 --- ### [01:07:26](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=4046s) Jane Street 经典新员工入职流水线 讨论随后转向 Jane Street 系统化的新人入职培训流程。一名新加入的软件工程师在入职头两周通常会经历密集的体系化训练： ```mermaid flowchart LR accTitle: Jane Street 工程师入职前两周培训流水线 accDescr: 展示新入职工程师从第一周的 OCaml 语法与风格训练，过渡到第二周的生产环境部署与配置实践。 subgraph Week1["第一周：语言与规范基石"] A["OCaml Boot Camp"] --> B["构建小型 C/S 架构应用"] B --> C["掌握 Jane Street 核心库生态"] C --> D["导师深度 Code Review 建立代码风格"] end subgraph Week2["第二周：生产环境工程化"] D --> E["Production Boot Camp"] E --> F["应用部署机制学习"] F --> G["系统监控与告警运维"] G --> H["分布式配置管理实操"] end ``` - **第一周：OCaml Boot Camp**：学员通过一系列层层递进的练习，在 OCaml 中从零搭建一个微型客户端/服务器（client-server）应用程序。在此期间，他们会熟悉公司专有的核心基础库，由资深导师进行逐行代码审查（code review），从而深刻理解 Jane Street 独有的代码规范和工程风格。 - **第二周：Production Boot Camp**：学员学习在 Jane Street 真实的生产环境中，软件系统是如何被打包配置、安全部署、实时监控与维护的。 Ron Minsky 回顾指出，OCaml Boot Camp 的最早雏形已有约 20 年的历史。近年来，在 Aaron Bauer 等人的主导下，这套培训机制被持续打磨和迭代，效率大幅提升。尽管新员工对内部某些处于“起起落落”状态的系统文档并不总是完全满意，但这两门经过千锤百炼的基础训练营，在近几年的员工反馈中几乎收获了一致的高评价。 ### [01:09:15](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=4155s) 动机驱动的教学：在具体情境中引入工具 在讨论基础培训时，Aaron Bauer 指出，在教授技术内容时，最关键的是在具体情境中引入新概念，即让新概念的出现直接由某个具体的应用场景或系统需求所驱动。他分析了很多人常犯的教学误区：往往是先抛出一个工具，然后再到处去寻找该工具能解决的问题；而更理想的方式应该是大家正在开发某个项目，突然产生了一个新需求，现有手段无法妥善处理，此时顺理成章地引出：“等等，其实有这么个现成工具，我们为什么不用它呢？”这种由实际问题倒逼工具出场的方式，能够让学员的大脑产生认知锚点，从而将新知识稳稳地挂靠在上面。这也是团队在重构入门教程时投入大量心力的核心原则。 随后，Aaron 梳理了新员工入职第一年所经历的“Teach-ins（专题研讨课）”培训体系。该体系由一系列兼顾深度与广度的模块组成，正逐步演化为一个成熟的“课程目录（Course Catalog）”。员工入职时会与主管或团队成员坐下来评估，根据各业务组的具体需求定制第一年的学习路径。不仅如此，这些教学材料全部被设计为支持自主学习的形式，便于工程师在工作现场真正遇到特定问题时随时查阅和实践。 ```mermaid flowchart TD accTitle: Jane Street 新员工第一年课程目录架构 accDescr: 展示从入门基础到核心专题，再到专项领域课程的学习路径。 subgraph Core ["核心技术进阶 (Core Teach-ins)"] T1["软件测试 (Testing)"] T2["高性能 OCaml 与分析工具 (Performance & Profiling)"] T3["系统级调试 (Systems Debugging)"] T4["高级函数式编程 (Advanced FP)"] end subgraph Specialized ["专项业务与领域 (Specialized Concepts)"] S1["Bonsai Web UI 框架与增量计算"] S2["市场数据系统 (Market Data Systems)"] end Start["新人入职 (Onboarding)"] --> Core Core --> Plan{"与经理定制学习方案\n或按需自主学习"} Plan --> Specialized ``` - **高性能 OCaml 开发与分析**：讲授如何编写高性能的 OCaml 代码，并熟练运用性能分析与性能度量工具定位瓶颈。 - **系统级调试**：系统梳理用于排查复杂线上系统故障的诊断工具与分析技巧。 - **高级函数式编程**：深入剖析 OCaml 的高级语言特性及其在工程实践中的精巧用法。 - **专项业务与领域库**：涵盖用于在 OCaml 中构建 Web 界面与前端 UI 的 Bonsai 框架，以及对特定交易组至关重要的市场数据（Market Data）系统内部架构。 ### [01:11:42](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=4302s) 实用工具优先：概念传授与大学教育的脱钩 Ron Minsky 回顾了工程团队的 Teach-ins 最初如何脱胎于交易员培训项目。最初的设想更偏向于高阶架构与思维模式：例如学 Bonsai 不仅是学前端框架，更是借此理解“增量计算（Incremental Computing）”；讲市场数据系统不仅是讲业务流，更是传授如何处理复杂的流式数据并设计高效模块化协议。但 Ron 坦言，这种纯概念化的顶层设计在实践中远不如预期成功，团队逐渐转向以提供“开箱即用、能立即解决实际问题的工具”为主线。面对 Ron 关于“概念与实用工具该如何平衡”的疑问，Aaron 对比了大学计算机教学与 Jane Street 内部教学的本质区别。 在大学里（例如讲授数据库课程），教师不仅传授领域知识，还要借此培养学生的通用问题解决能力以及应对不确定性的排错能力。然而在 Jane Street，这种模式并不适用。Aaron 直言同事们本身就是极其顶尖的问题解决者，根本不需要占用宝贵的工作时间来训练基础的解题思维，正如 Ron 调侃指出的：““他们本来就已经是极为优秀的问题解决者了。这背后显然存在某种筛选效应。””（Ron Minsky：“they're already really good problem solvers. It turns out... there's some selection effect there.”）当员工日常在工位上处理的都是最硬核的真实难题时，如果培训中安排大量脱离实际、耗费数小时去构想新解法的“罐头式虚拟难题（canned educational thing）”，体验会非常糟糕。 因此，教学策略演变为以工具为载体顺带输出概念。Ron 将其精准概括为：即便最终目标是传递核心架构理念，最有效的方法依然是聚焦于具体、有用的工程工具，因为““理解这个工具并能熟练运用它，本身就需要以理解那些底层原理为前提。””（Ron Minsky：“understanding the tool and being able to use it required the understanding of those lessons.”）Aaron 赞同这一观点，指出企业内的概念教学更适合采取代码演练（walkthrough）的形式：带领学员一步步看懂真实问题如何拆解解决，让他们在后续回到工位应对实际生产任务时，再自然而然地拼完深层理解的最后一块拼图。 ### [01:16:15](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=4575s) 什么是真正的“学习目标（Learning Goals）” 话题转向外部专业教育理念在企业内部的落地。Ron 忆及 Aaron 刚入职时接手的第一个重头项目——重构测试培训课程（Testing Teach-in）。在此之前，该课程推进得极为艰难，因为大家各自负责零散模块，缺乏全职推动者将其整合为一个高质量的交付物。当 Aaron 初次介入、向负责课程初始构想的 Ron 询问“这门课的学习目标（Learning Goals）是什么”时，Ron 甚至感到一阵错愕，坦承自己虽然知道这些英文单词组合在一起的意思，但完全不理解在教育学语境下它的具体指代。 Aaron 解释了系统化教学设计中“学习目标”的严格定义：当开启任何大型项目时，必须首先以书面形式明确要达成的成果；而在教学语境下，学习目标必须从**学员身上将发生何种改变**的角度来表述。 - **必须使用主动动词短语**：学习目标通常应当写成“完成此项培训后，学员将能够（Students will be able to...）”并紧跟具体的行动动词。 - **拒绝模糊的概念性表述**：诸如“理解基于属性的测试（Property-based Testing）”并不是一个合格的学习目标，因为表意宽泛且无法客观度量。 - **锚定具体复杂度与输出行为**：一个有效的目标应当是“学员能够针对嵌套记录类型（nested record type）的输入，编写出对应的基于属性的测试用例”。 - **明确边界与非目标（Non-goals）**：强迫自己用操作层面的动词拟定目标，其最大的价值不仅在于理清必须达成什么，更在于果断砍掉那些不属于核心目标、只会白白浪费学员时间的教学环节。 ### [01:18:50](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=4730s) 学习目标、教学自评与教育者的心智模型 明确界定学习目标（learning goals）不仅能够规范课程开发，还能深刻改变对教学成效的评估方式。Aaron Bauer 解释说，只要清晰写下期望学员在课程结束后能够独立做到的事情，后续课程活动的设计就会变得顺理成章：学员应当在培训中直接练习结业时需要展现的能力。以此为锚点，讲师便能精准推导出所需的前置知识背景、讲座安排与配套材料，同时果断剔除与核心目标无关的代码和冗余内容。Aaron 打趣道，自己在 Jane Street 的教育工作在某种程度上就是“走进不同的房间，面对不同的人，嘴里念叨‘学习目标’这个词”，推动团队建立起以目标为导向的严密开发流程。 Ron Minsky 补充指出，学习目标在“评估”这件事上提供了极具启发性的全新视角。在企业的内部培训中，组织的目的通常不是为了给员工打分或筛选淘汰学员，而是为了评估教学项目本身——即这门课是否真正教会了大家有用的东西。通过观察学员在结业时是否具备了预期的能力，教育者得以对自己和课程方案进行客观的自我审视。此外，Ron 还发现“学习目标”在实践中演化成了两层含义：一层是具体书写、推敲并用于搭建项目结构的规范工具；另一层则是宏观的心智框架，促使讲师时刻反问自己：“学员在学完之后会有什么不同？你今天为什么要来上班？你的终极诉求到底是什么？” 在物色与面试潜在的开发者讲师（developer educators）时，这种框架展现出了极大的辨别力。Ron 注意到，优秀的教学候选人往往能对课堂目标给出深思熟虑的回答，而有些人的初衷仅仅是“因为我自己喜欢这个主题”。尽管充满个人魅力、对知识满怀热忱的讲授也能成就许多伟大的课堂，但系统性的目标架构能为缺乏教学经验的新人提供至关重要的思考框架。对此，Aaron 强调道：“它迫使你关注那些在某种程度上不仅仅存在于学员脑海中的东西——我在这里试图达成的目标，绝非仅仅是我告诉学员一件事、然后他们知道了而已，而是他们是否以某种可被观察的方式发生了改变。” ```mermaid flowchart TD accTitle: 逆向教学设计与评估流程 accDescr: 展示从定义最终学习目标反推实操练习、背景知识和评估教学成效的逆向设计流程。 subgraph Design["课程设计阶段"] A["明确界定最终学习目标<br/>学员能够做什么？"] --> B["设计针对性实战练习<br/>学员直接演练目标技能"] B --> C["筛选精简先备知识<br/>仅提供必要讲义与核心代码骨架"] end subgraph Evaluation["效果评估阶段"] C -.-> D["结业能力观察与验证"] D --> E["反哺评估教学项目本身<br/>而非单向考核学员"] E --> A end ``` ### [01:22:08](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=4928s) 认知负荷理论与主动学习的实践 针对教育学研究庞杂且多有争议的现状，Ron 提问在浩瀚的教育学文献中，有哪些经过验证的理论真正指导了 Jane Street 的工程培训。Aaron 指出，自己最为刻意应用到 OCaml 训练营（OCaml bootcamp）中的理论之一便是“认知负荷”（cognitive load）或“工作记忆”（working memory）模型。人在学习全新知识时，大脑同时处理新概念的容量存在生理上限；一旦过载，任何单一知识点的吸收效率都会暴跌。在早期的 OCaml 训练营中，新人一入职就必须同时学习 OCaml 语法、一款此前可能从未用过的文本编辑器、以及一套内部自研的版本控制与代码审查系统。三者交织并行，极易导致认知超载。 为了解决这一问题，Aaron 对训练营进行了大刀阔斧的模块拆解，实行梯度渐进教学： - **阶段一：纯粹的语言基础**。学员最初仅在终端交互式解释器（REPL 工具 `utop`）或浏览器内的交互式代码框中编写简短的 OCaml 代码片段，专注于掌握语言的核心语法。 - **阶段二：专注工具链上手**。在掌握语法后，学员才被引入专用编辑器，并在孤立环境中单独演练 Jane Street 的版本控制系统。 - **阶段三：综合融会贯通**。只有当上述孤立要素都逐一掌握后，学员才进入将语言、编辑器与版本控制合为一体的综合工程任务中。 除了认知负荷控制，另一个核心支柱是“主动学习”（active learning）。教学不再局限于单向灌输信息，而是最大化交互性。在当年的暑期实习生培训中，Jane Street 设计了大量的小组讨论、系统设计推演，以及面向利益相关者（stakeholders）收集需求与沟通汇报的模拟练习。Ron 对此深表认同，并强调工程教育绝非只是冰冷的知识传递，学习与记忆的背后始终包含着强大的认知心理学与社交互动规律。 ### [01:25:10](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=5110s) 应对 AI 冲击：重塑实习生培养与测试基石 大型语言模型（LLM）能力的飞跃，迫使 Jane Street 对沿用多年的实习生早期培养项目展开了彻底的重构。谈及改革的根本导火索，Aaron 坦言：“最初的诱因是，哇，现在的 AI 工具实在是太出色了，以前我们让实习生起步做的那个为期四周的项目，如今可能一个 Prompt 就搞定了。”如果生成式 AI 能一键输出完整实现，传统的项目便彻底丧失了作为评估与培养工程素养工具的价值。 针对这一剧变，公司大幅提升了实习生项目的复杂度与现实度。代码编写能力——这一昔日衡量工程师最重要的考核指标——其权重正在显著下降。新的实习项目更加开放（open-ended），涵盖系统架构设计、跨团队需求挖掘以及与业务方的沟通。在教学准备上，教育团队并不赞成将新手直接扔进 AI 的深水区任其自生自灭，而是系统性地教导他们如何辨别模型的缺陷、掌握与 AI 协同的最佳模式，并熟悉内部可用的各类模型工具。在练习题设计上，团队必须拿捏好平衡：任务必须足够复杂，避免 LLM 一次性就能给出无瑕的代码，但也不能过于臃肿庞大，导致学员需要耗费整整一周去通读检查生成的代码。 作为这场课程重塑的关键落脚点，软件工程中的“测试”（testing）环节被提到了前所未有的战略高度。在全新的实习课程中，为基于数据库的应用系统构建一套极其严密且健壮的自动化测试套件（test suite）成为了核心教学模块。Ron 指出，测试在当下之所以远比过去更为关键，核心原因在于“模型极度依赖反馈（feedback）”：在应对现实中超越简单模板的复杂问题时，大模型根本无法做到一次性命中正确方案（one-shot），唯有依托高覆盖、确定性的测试套件源源不断地向模型回传运行时与逻辑反馈，整个 AI 辅助开发闭环才能真正有效运转。 ### [01:28:10](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=5290s) 重塑实习生培养：设计文档、利益相关者角色扮演与代码审查训练 在设计实习生培养计划时，平衡授课与实战项目始终是一道难题：如果把整个实习期的一半时间都花在教学上，实习生就很难有充分的时间投入到实际项目的开发中。Aaron Bauer 指出，在 AI 时代，公司一方面决定加大对教育培训的投入力度，但另一方面仍然必须尽早让实习生进入实际的项目开发节奏。为此，团队开发了一系列全新模块，将重心转移到软件生命周期中更为前置和后置的关键环节——系统设计与代码评审。 在设计与规划阶段，课程加入了分析设计文档（Design Docs）的全新练习：实习生需要判断什么是优秀的设计文档，审阅质量较差的案例并共同拆解其中的缺陷。为了模拟真实的工程沟通场景，团队还设计了一项角色扮演活动——由全职工程师分别扮演某个待开发应用程序的各类利益相关者（Stakeholders），实习生必须主动与他们沟通访谈，从而梳理、挖掘出设计系统时所需的完整需求。 更为重大的结构性变革在于代码审查（Code Review）。以往在 Jane Street，实习生通常不会参与审查他人的代码，但在大语言模型时代，审查能力已经成为软件工程的核心。这个夏天，团队不仅要求实习生开始审查代码，还在实习中期额外安排了整整三天的专项培训： ```mermaid flowchart TD accTitle: 实习生代码审查专项训练流程 accDescr: 展示 Jane Street 实习生在为期三天的代码审查培训中的学习闭环。 subgraph Training["3天代码审查专项培训"] A["审阅预设功能代码<br/>（向原有练习应用添加功能）"] --> B["撰写代码审查评论<br/>（提交 CR 意见）"] B --> C["对比标准参考答案<br/>（讨论分歧与盲区）"] C --> D["实习生互评代码<br/>（开发自制功能并交叉审查）"] end D --> E["实战项目阶段：<br/>导师向实习生提交功能请求 CR"] ``` 实习生首先会拿到一些预先做好的功能代码（直接作用于他们在前期测试和设计训练中接触过的同一个应用程序），并在其上撰写审查批注；随后，导师会给出一份标准参考答案（Answer Key），供大家以小组形式研讨哪些问题被遗漏了、哪些审查意见存在分歧；接着，实习生需要亲自动手实现新功能并互相审查彼此的代码，进行多轮实战演练（Reps），最终承接导师在真实项目中发给他们的审查请求。 - 传统的实习体系几乎不要求实习生参与评审他人的代码。 - 专项培训直接嵌入到实习中期，为期 3 天，紧密衔接设计与测试环节。 - 最终目标是让实习生能够在实际项目中帮导师分担真实的 Feature 审查工作。 --- ### [01:30:25](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=5425s) 规模失控、认知负荷与 4 人小组讨论的教学价值 对于这次实习生课程的大刀阔斧改革，Ron Minsky 询问效果究竟如何，Aaron 笑着坦言：“I think we're learning a lot.”（我想我们学到了很多）。在实际运行中，最大的挑战集中在**任务的范畴（Scope）**上。如果教学团队一边反复向实习生灌输“你必须全权负责并拥有这段代码，仔细检查大模型产出的每一个细节，绝不能闭眼盖章（Rubber stamp）”，另一边却要求他们用 LLM 生成远远超出润色时间限制的大量代码，两者的脱节就会向学生传递混乱的信息。同样的问题也发生在代码审查上：如果丢给实习生一个缺陷过多、体积过大的 Feature，巨大的认知负荷（Cognitive load）会让新手很难抓住真正需要关注的核心架构隐患。 不过，抛开任务规模控制上的初试磕碰，全新的高互动性教学模式展现出了巨大的优势。过去学习 OCaml 时，实习生大多各自按自己的节奏推进，与全职导师进行一对一审查交流，同伴之间几乎零横向沟通。而这套新课程将实习生划分为 4 人小组（Pods），并刻意穿插了密集的小组研讨： > “I think that those discussions, including kind of full-time mentors, have been really productive, and I'm excited about kind of anchoring a lot of our uh, at least intern education in that sort of model.” —— Aaron Bauer 在包含全职导师的小组讨论中，实习生们相互交流各自如何使用 LLM 工具、模型生成的测试用例究竟出现了哪些诡异的问题，这种同伴互助与经验共享成为了课程中最具生命力的部分。 --- ### [01:32:47](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=5567s) 抵制“氛围写代码”的诱惑：确定性测试驱动与应对不确定性 针对未来的课程迭代，Aaron Bauer 强调不仅需要削减单项任务的规模，还要调整教学节奏与重点。由于实习生完成任务的速率差异极大，课程进度表必须精准设计：既要为跑在最前面的学生提供高质量的扩展探索任务，又不能让落后的学生因为盲目追赶时间线而略过关键理念。 特别是在早期批次的实习生培训中，团队踩过一个坑——最初将重心放在了任务多线程并发以及所谓的“氛围写代码”（Vibe coding）上。这种让模型同时并行处理三件事的玩法虽然表面上“很酷”，但团队很快意识到这种开局方式定错了调子。经过反思与重组，他们果断调整了顺序：课程必须从**高标准的测试用例编写（Testing-focused piece）**切入，先立下严谨的质量红线，而不是一上来就炫耀模型有多能干。 | 教学维度 | 早期试验做法 | 修正与未来规划方向 | |:--- |:--- |:--- | | **开局导向** | 强调并发使用 LLM、快速试错的“氛围写代码” (Vibe Coding) | 转向以编写高质量自动化测试套件为核心基石 | | **工作流指导** | 仅罗列 LLM 的已知反常行为与防坑点，缺乏规范流程 | 建立更严密的引导体系，规范“人机定期 Check-in”的具体工程工作流 | | **同伴学习** | 缺乏结构化的实践经验交换机制 | 依赖 4 人小组交流各自摸索的 Prompt 与避坑技巧 | 面向未来的实习培训，团队计划提供更具指导性的特定工作流范式，明确开发者应当在哪些节点与 AI 产出进行检查（Check-in）。谈及给 AI 时代的工程师传授经验的难度，Ron Minsky 感叹技术环境迭代极快，今年的忠告到了明年可能彻底失效。Aaron 则补充指出了教学中更深层的挑战：LLM 天然具有高度不确定性，哪怕两个人使用完全相同的 Prompt 和方法，模型也可能给出截然不同的响应。将 LLM 引入教学实验，本质上是给以往高度确定、标准化的编程教学引入了一组巨大的不确定变量。最后，Ron 感谢了 Aaron 的分享，并在片尾播报中指出可在 signalsandthreads.com 查看全集文稿与拓展资料。 > “Not only is it changing fast but two people using it in the same way might have it do different things. So when you give people an exercise that involves using an LM, it just introduces much more uncertainty into what is going to happen when they do it than you would have had in the past.” —— Aaron Bauer ## Notable quotes > [43:30](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=2610s) 能够不写的代码就是最好的代码。不仅是你不用写，而且还有大量其他人在使用它并报告问题，那些问题也在被修复。因此我们不必通过自己踩坑去发现所有缺陷。 — Aaron Bauer > [38:38](https://www.youtube.com/watch?v=qN6OM1IzjIE&t=2318s) 编辑器拥有这种核心地位，它是许多其他事物触达用户的界面。因此，你遇到的问题可能出在类型查找系统或索引系统上——这些东西从字面意义上讲并不属于编辑器本身，但编辑器却是你看到问题的地方。 — Ron Minsky ## Mentioned Aaron Bauer, Ron Minsky, Jane Street, Carleton College, Summer Science Program, Foldit, AlphaFold, Python, Fortran, Scheme, OCaml, OxCaml, VS Code, Emacs, Neovim, Iron, Merlin, ocaml-lsp, ECaml, Bonsai, Claude Code, Charlie Marsh ## Video description Aaron Bauer is a software engineer and one of Jane Street's few developer educators—a role that splits his time between writing code and teaching other people how to write it. Before joining the firm, he taught computer science at Carleton College, including four straight terms online during the pandemic. In this episode, Ron and Aaron discuss what it takes to teach engineering inside a company with its own language, its own version control, and its own editors, and what changes when an LLM can do the exercise for you. Along the way, they consider the underrated power of a live lecture; why the text editor is still the software engineer's home base; using editor telemetry to find out how AI is actually changing developer workflows; and how Jane Street rebuilt its intern curriculum around testing, design, and code review now that producing the code is the easy part. 00:00:00 – Start 00:00:02 – Asteroids, Python, and a teacher on a beach 00:04:05 – Grad school, rendering, and Foldit 00:06:43 – Carleton, COVID, and the flipped classroom 00:10:02 – In defense of the lecture 00:14:54 – Feedback loops and mastery learning 00:21:29 – Inventing the developer educator job 00:26:48 – More and more to teach 00:31:18 – The Editors team, and what a "feature" is 00:34:10 – The editor as home base 00:37:47 – Giant diffs and other problems of scale 00:40:41 – Python notebooks 00:43:30 – Build it ourselves, or push it upstream? 00:48:05 – What agentic AI changes for editors 00:53:27 – Claude Code's learning mode and mental models 00:58:20 – Teaching when the LLM can do the exercise 01:07:26 – Bootcamps, teach-ins, and a course catalog 01:16:15 – What are the learning goals? 01:25:10 – Rebuilding the intern curriculum for AI For more episodes, visit: https://signalsandthreads.com/ [Source](https://www.youtube.com/watch?v=qN6OM1IzjIE)

[43:30] Aaron Bauer: 能够不写的代码就是最好的代码。不仅是你不用写，而且还有大量其他人在使用它并报告问题，那些问题也在被修复。因此我们不必通过自己踩坑去发现所有缺陷。 [38:38] Ron Minsky: 编辑器拥有这种核心地位，它是许多其他事物触达用户的界面。因此，你遇到的问题可能出在类型查找系统或索引系统上——这些东西从字面意义上讲并不属于编辑器本身，但编辑器却是你看到问题的地方。

[59:27] Ron Minsky: There are definitely people who run into problems where they just like kind of vibe it up too much and like use the LLM to generate stuff and do not think, and like that's a real problem. [92:47] Aaron: when you give people an exercise that involves using an LLM, it just introduces much more uncertainty into what is going to happen when they do it than you would have had in the past. [76:15] Aaron Bauer: learning goals are a specific way of writing down what you want to accomplish in terms of what is changing about the students as a result of the thing that you're doing.

[0:01] [music] [0:02] All right. Well, it's my great pleasure [0:04] to have Erin Bowerer uh with me. Erin is [0:07] a software engineer and one of Jane [0:10] Street's uh proud but few developer [0:11] educators. We'll talk a little bit more [0:13] about what that means in a bit, but [0:14] thanks for joining me. [0:15] >> I'm very excited to be here. [0:17] >> So, I like to always hear a little bit [0:19] about how people started out. Uh you've [0:21] done a lot of work which is kind of on [0:23] the overlap of both software engineering [0:25] and computer science and teaching. And [0:27] I'm kind of curious like how did you get [0:29] there? [0:30] >> Sure. So kind of my origin story you [0:34] might say is I've been uh I was sort of [0:37] fascinated with computers from from a [0:39] child. Computer programming was always [0:41] felt kind of intimidating. And then I [0:43] did this summer program before my senior [0:46] year of high school, creatively named [0:48] the summer science program, [laughter] [0:49] where went to New Mexico for six weeks [0:51] and they taught us a bunch of uh math, [0:54] astronomy, physics, and some Python. We [0:57] took telescope observations of an [0:58] asteroid and then wrote some Python code [1:01] to take those measurements and compute [1:02] an orbit for the asteroid. So really fun [1:06] and and interesting program, but I came [1:08] out of it or went into it being like all [1:10] these things seem interesting. And I [1:11] came out of it being like, wow, the [1:13] programming was just way more [laughter] [1:15] fun and like really my thing. Uh so at [1:18] that time, my small town high school [1:20] didn't offer AP computer science, but [1:23] found a retired teacher on a beach in [1:26] North Carolina who could take it from. [1:28] Uh and then sort of from there, [1:30] >> h how did you find a retired teacher on [1:33] the beach? Uh, so it's North Carolina [1:35] had a like online course program that [1:37] you didn't have to live in North [1:38] Carolina [1:39] >> to use. And it just turned out the guy [1:41] teaching this course was not currently [1:43] working in a school, but instead was on [1:45] the beach. [1:46] >> Oh, amazing. [1:47] >> And uh then [1:50] the teaching side comes in that before [1:53] the summer before I go into grad school, [1:55] I go back to the summer science program [1:57] as a TA. And that's sort of my first [1:59] exposure to writing an exercise for [2:02] students to do and giving a little [2:04] lecture on binary search trees. And I [2:06] was hooked and sort of from then knew [2:08] that I wanted to do something with [2:09] teaching. It's it's funny by the way [2:11] just like the the kind of science and [2:13] physics hook in programming. That's [2:15] actually like a deep thing in the [2:16] history of computer science, right? A [2:18] lot of the early programming and [2:19] programming languages and work was all [2:21] about like, you know, high-end physics [2:23] simulations. Um, I always every now and [2:25] then when I like run into like someone [2:27] who's like, you know, gathering [2:30] telescope data, just like the hardcore [2:33] like computer science of like, you know, [2:34] weird FPGA exploits and stuff that [2:36] people are doing there, I always find [2:38] very impressive. There's just like a lot [2:39] of interesting pushing of the [2:40] state-of-the-art and also a lot of messy [2:44] not understanding a lot about computer [2:46] science that I feel like comes out of [2:47] that like scientific computing world. [2:49] >> Uh, yes. Uh something I did in undergrad [2:52] was work with a uh forrans simulation of [2:57] uh impacts kind of asteroids or comets [3:01] hitting the surface and this came out of [3:03] code that was used to simulate atomic [3:05] explosions in the early days computing [3:08] because the the physics of those two [3:09] situations are not that different it [3:11] turns out. [3:12] >> Oh wow. [laughter] [3:13] >> Yeah. Yeah, and there's a bunch of like [3:14] super cool stuff in the early forrren [3:17] work like that, you know, we we care a [3:19] lot about automatic differentiation [3:21] these days, right? Because we write all [3:22] these machine learning models and you [3:24] need gradient descent and for that it's [3:25] nice to have derivatives. [3:26] >> Um, but automatic differentiation goes [3:29] back to like the early forrren days [3:31] where they, you know, they there was a [3:32] bunch of forran compilers that could do [3:34] this. It was messy. They could do first [3:36] derivatives but not second derivatives. [3:39] Like you would take the derivative and [3:40] you of one program, you get another [3:41] program and if you tried to take the [3:43] derivative of that second program, God [3:45] help you. Nothing nothing good would [3:47] happen there. [3:47] >> Yeah. It's it's very much computer [3:49] science is a a means to an end. It's the [3:53] like I'm not programming because I I [3:55] love programming. I'm using it just as a [3:57] tool because it beats a slide rule, I [4:00] suppose. [4:01] >> Right. [laughter] [4:02] >> Okay. But you loved programming then? [4:04] >> Yes, I certainly did. [4:05] >> And then you went to grad school. like [4:07] why did you go to grad school and what [4:08] did you study there? [4:09] >> Uh so I didn't know I was going to grad [4:12] school until I took a computer graphics [4:15] course my last year of undergrad and got [4:17] incredibly excited about rendering and [4:19] was like I would like to do this is [4:21] something I could do for the rest of my [4:22] life. What's a way that you get to do [4:24] one thing for the rest of your life? You [4:26] go get a PhD in that thing. Uh so I [4:29] applied to some programs, got into them [4:32] and then as I was talking to people uh [4:34] realized that people weren't so much [4:36] doing research and rendering anymore. A [4:39] lot of the field had sort of moved on. [4:41] It was very mature. The remaining work [4:42] was largely in industry. So, I kind of [4:45] inertia carried me into grad school [4:47] anyway and kind of did some game design [4:51] research, some computer uh science [4:53] education research uh and then ended up [4:56] uh uh writing a dissertation on what you [4:59] might call human computer problem [5:00] solving, studying how uh players of the [5:04] scientific discovery game Fold it uh [5:06] solved problems and collaborated and [5:07] sort of worked with the automated tools [5:09] available there. [5:10] >> All right. fold. It was the like the [5:12] early like here's how we're going to [5:13] solve protein folding by having people [5:16] go off and try and solve protein folding [5:17] as a game. [5:18] >> Exactly. It was humans have some like [5:20] big picture geometrical insight of oh I [5:23] sort of know how these amino acids tend [5:27] to fold up and oh what if we took this [5:29] part and flipped it over here and at [5:31] that time computers were really good at [5:32] the I'm just going to wiggle the protein [5:34] a lot to find the lowest energy kind of [5:36] local optima and those two things [5:38] together were able to resolve proteins [5:40] that had gone unresolved in the lab for [5:43] many years people playing this game had [5:46] often had no biochemistry background [5:48] found. So it was very successful and [5:50] eventually moved on to people actually [5:52] designing novel proteins in the game. Uh [5:55] but I think um alpha fold and and other [5:58] things have sort of eaten their lunch. [6:00] >> Do you know if some of the data that [6:03] went into AlphaFold and the like came [6:06] out of Folded? [6:07] >> I assume Alphafold was trained on all [6:11] known structures in which case [6:12] absolutely. [6:13] >> So there's a little bit of an [6:14] educational point in that, right? You're [6:17] talking about like how do you get a [6:19] bunch of people who do not know a domain [6:20] to learn it and like how does that [6:22] process work and the kind of gamifying [6:24] the education process essentially? [6:26] >> Yeah. A a lot of my work in grad school [6:30] was had this sort of cognitive [6:32] psychology element to it of how of [6:36] trying to understand how people think or [6:38] learn and then how they apply uh those [6:42] skills. [6:43] >> Okay. And then where did you go from [6:44] grad school? Uh so I I knew that I [6:46] wanted to go somewhere where teaching [6:49] was my primary responsibility. Uh and so [6:52] that's how I ended up at a small liberal [6:54] arts college uh Carlton College in [6:56] Minnesota. I went there and taught two [6:58] quarters in person and then it was [7:00] spring 2020 and then I taught four [7:02] quarters online. [7:05] Uh [7:05] >> so this is like a hard introduction to [7:08] teaching. [7:09] >> Uh yes it was it was challenging for me, [7:12] very challenging for the students. uh [7:14] but also like involved a lot of trying [7:17] new things and experimenting and [7:20] different formats um which I think is an [7:23] important part of of succeeding in [7:26] education. So how well do you feel like [7:29] Carlton did at adapting to these new [7:32] online constraints and how like how well [7:34] you just even forget Carlton I guess [7:36] like your own classes like how [7:38] successful do you feel like you were [7:39] actually able to be at this? Uh I think [7:42] I eventually got to the point where I [7:44] was fairly successful. Uh I started out [7:49] taking the model of that online AP [7:52] computer science course that I had taken [7:54] in high school which was entirely [7:56] textbased. There were no video lectures. [7:59] You just read some stuff and you did [8:01] some exercises. So I started out on that [8:03] in part due to guidance of like people [8:05] were at home. Not everyone had good [8:06] internet connections. Want to make this [8:08] very accessible. Uh but just immediately [8:10] got feedback that this really does not [8:12] work for all. It worked great for me [8:14] does not work for all students, [8:15] >> right? [8:16] >> Uh so then [8:16] >> accessible in some way deeply [8:18] inaccessible in others. [8:20] >> Yes. So I experimented with um sort of [8:23] flipped classroom approaches with [8:26] synchronous courses uh synchronous [8:29] classes uh splitting out into kind of [8:31] small working groups. [8:33] >> There's a bit of jargon that I bet not [8:34] everyone understands. Can you say a [8:35] little bit more about what a flipped [8:36] classroom is? [8:37] >> Yeah. So a flipped classroom is instead [8:39] of the class time being when uh the [8:43] professor would deliver a lecture, the [8:45] professor prepares a lecture, a recorded [8:49] video ahead of time. Students watch that [8:51] before class and then you use class time [8:53] to do exercises or discussions, [8:55] something more interactive, [8:57] >> right? And this is like not a new idea [8:59] that came up in COVID. People have been [9:00] talking about flipped flipped classrooms [9:02] for a long time. Yes, this is a [9:04] long-standing idea uh in education that [9:07] has been um implemented in in different [9:09] ways, but it gained a sudden relevance [9:12] and I think widespread adoption when [9:14] classes had to move online, [9:16] >> right? And I guess part of the point of [9:17] the flip classroom is is the in-person [9:20] time, right? Being able to like leverage [9:21] the inperson time to do things that are [9:23] better than just giving a lecture. [9:25] >> Yes. So you you take the time where [9:28] students are say passively kind of [9:30] receiving some some content or seeing [9:32] worked examples and you say that does [9:34] not need to be done in person and we're [9:37] going to leverage the instructor's [9:38] ability to tutor someone one-on-one or [9:41] to lead a discussion things that can [9:42] only happen when you're uh when you have [9:45] time together in the classroom. kind of [9:47] my where I sort of personally ended up [9:50] was just trying to make not not doing [9:53] the the flip classroom but trying to [9:56] make the the time together more of a mix [9:58] of interactive things and some [10:01] instruction. [10:02] >> Yeah. Like I'm curious how much you buy [10:05] the kind of argument behind the flipped [10:07] classroom cuz like my instinct about [10:10] this has has always been that people [10:13] underrate the power of lectures that [10:15] like lectures are kind of a form of [10:17] performance that is actually like really [10:19] compelling and like the really being in [10:21] person with the person doing the [10:22] lecture. Even though in some sense it [10:24] seems dumb and how could you possibly [10:26] need it and surely you should just like [10:28] record the best lecture that has been [10:30] given on the topic and watch it that [10:32] like actually with real human beings in [10:34] real environments this just kind of [10:35] isn't true and there's just something [10:38] more compelling about being there in [10:40] person [10:42] like it's not like I think the flip [10:43] classroom idea is totally wrong but I do [10:45] feel like it under like the whole thing [10:46] of like oh there's just like the passive [10:48] lecture is just like missing something [10:50] about the human psychology there. [10:52] >> Yes. So I I would tend to say that lines [10:54] up with my experience that there is [10:57] something about like having a person as [11:00] you say perform live in front of you [11:02] that I I think has a sort of focusing [11:04] effect of your attention and because it [11:07] is live I think you're more likely to [11:09] try and take notes which I think is good [11:12] for people understanding which is not to [11:13] say you couldn't do that when you're [11:15] watching a video in your dorm room but I [11:17] think it's less likely. I think you and [11:19] also I think a solid hour of someone [11:22] just talking at you that starts to lose [11:26] the engagement and become [11:27] counterproductive. So another bit of [11:29] jargon active learning is very important [11:32] for these sort of in-person uh things [11:35] where throughout the lecture I want to [11:37] stop and pose a question and have people [11:39] talk to their neighbors or have people [11:42] uh stop and and do a little uh problem [11:45] on their own to check their [11:47] understanding. things like this. By the [11:49] way, note-taking is like another great [11:51] example of like a super weird and also [11:53] strangely effective learning technique [11:56] where like just being in the classroom [11:58] and taking notes and then you if you [11:59] walk you just like light them on fire [12:01] like you never look at them again. Like [12:02] there's actually, as I understand it, [12:04] pretty good evidence that like this [12:05] actually helps retention in a serious [12:07] way even though the notes themselves are [12:08] like totally useless. having to to [12:11] process things and then sort of distill [12:13] distill them down to uh something that [12:16] you write down and then [12:19] some I agree the evidence seems to be [12:21] that the act of writing it down helps [12:23] set it in your memory in some [12:26] qualitatively different way than just [12:28] hearing it even if you understood it [12:31] >> right and apparently writing is better [12:33] than typing for this maybe I'm getting [12:35] that this is what I thought was true [12:36] about the research although it's not [12:37] like I remember the study or anything [12:39] That sounds familiar to me. I would say [12:42] my own experience in the classroom is [12:46] it's maybe less about the typing per se [12:48] than someone having a screen with all [12:52] the information and distraction that [12:54] humanity has created sitting between [12:57] them and the presenter uh tends to [13:00] undermine one's ability to pay [13:02] attention. [13:02] >> That seems very believable. Uh this is [13:05] also why I have generally tried to avoid [13:08] teaching in computer labs which is [13:10] something that people often do in CS [13:13] courses but I I just can tell as soon as [13:16] there's a screen in front of people like [13:18] 50% of people even if they want to pay [13:20] attention they just they can't. [13:22] >> Yeah, exactly. This this echoes a little [13:24] bit my own experience just like teaching [13:26] my kids how to program like one of the [13:28] things I did a lot of was have them [13:31] think about programs on paper, [13:33] >> right? [13:34] In fact, one of my one of uh one of the [13:37] things I conned my kids into thinking [13:38] was like an acceptable distraction like [13:40] when food arrive at a restaurant was [13:43] like I would write a little scheme [13:45] program on a piece of paper with like [13:47] obfiscated names and then they would [13:48] have the puzzle of trying to figure out [13:50] like what did flub do and [laughter] [13:53] somehow I convinced them this was like [13:54] an acceptable way to spend time. But I [13:56] do think like the thing of like talking [13:58] to them about what a program means and [14:01] starting out by being like, you know, [14:03] I'm going to write down on the board or [14:05] on a piece of paper like here's an [14:06] expression like what does this evaluate [14:09] to? I think the moment you gave them [14:11] like a computer to type it in and try to [14:12] evaluate it, like they were playing the [14:14] game of getting the computer to do thing [14:16] and not thinking about the question of [14:18] like what is actually happening and I [14:20] think that was just like there were [14:21] ideas you could get across much more [14:22] easily when you like got the computer [14:24] out of the way. [14:25] >> Yeah. Yeah. And there is definitely a [14:27] thing that people just learning [14:29] programming will tend to do, which is [14:32] this is kind of arcane [14:35] instructions for the machine. I I will [14:38] perturb it arbitrarily until it does [14:40] what I want. And this is very easy to do [14:43] when it's on a screen and you can run [14:44] it. But if I ask you to write out a [14:47] program on paper and then decide when [14:50] you think it's correct, it's more likely [14:52] to force you to think about what it's [14:54] actually doing. for sure. Okay. So, you [14:57] were at Carlton during this like very [14:59] complicated and disruptive COVID [15:01] pandemic. You did a bunch of experiments [15:03] and learned things. What happened when [15:06] you got back to the classroom and people [15:08] could be there? Like did you are there [15:10] things that you took away from that [15:11] experience that you kept? Like what what [15:13] was what what what was persistent that [15:15] you learned there? [15:16] >> Yeah, I think one of the persistent [15:19] things was I I just find it very useful [15:21] to have a lecture recording. I think [15:24] this varies by institution. I've heard [15:26] from other instructors that if they [15:27] record the lecture, no one shows up to [15:29] class because they can watch it. This [15:31] wasn't the case at Carlton. Students [15:33] wanted to come to class. And I tried to [15:35] make being in class with this sort of [15:37] interactive stuff actually useful. But I [15:40] the [15:41] >> it might be institution dependent. It [15:42] also might be lecturer dependent. [15:44] >> This this is fair. [laughter] This is [15:45] fair. The classrooms at Carlton were not [15:47] equipped generally with video equipment. [15:50] So, I was carrying around a webcam on a [15:52] tripod to each of my classes and [15:54] plugging into my laptop and recording [15:56] the lectures and posting them online. [15:59] And as I would get questions throughout [16:01] the seme like, oh, I have this track [16:04] meet or this obligation. I have to miss [16:06] this class. What should I do? And if I [16:09] have a recording, it's like, well, it's [16:11] it's there. You can watch it. This is or [16:13] someone asks a question, I can send them [16:15] a link with the timestamp of the [16:17] lecture. Here's where I explain this [16:18] thing. So that carried over. I think [16:22] I I think I may have arrived at this [16:25] even without uh covid lockdown. But [16:29] during that I experiment experimented a [16:31] lot with uh sort of automated feedback [16:34] in various ways. So, uh, weekly quizzes [16:38] that were multiplechoice and autograded [16:40] or different sorts of like [16:42] fill-in-theblank kind of things, [16:44] different, uh, autograding software [16:46] where students could submit homework and [16:48] get immediate feedback on whether it [16:50] passed tests, that sort of thing, and [16:54] carried that forward into my courses and [16:57] and found it like because getting [16:59] feedback on your work is an essential [17:01] part of learning and I was always [17:04] dissatisfied with okay you do the [17:07] homework you turn it in a week or two [17:09] later an instructor TA has graded it and [17:12] you give back some feedback but at that [17:14] point you've moved on and you may not [17:15] even read the feedback you get also that [17:19] incorporated a technique that is [17:21] sometimes called mastery learning though [17:24] I didn't do it in its fullest extent but [17:26] the idea is you have to demonstrate sort [17:29] of mastery or competence at a thing [17:31] before you can move on where you can [17:33] attempt it any number of times s and you [17:35] keep getting feedback and you sort of [17:37] reattempt until you actually um have [17:40] kind of gotten it right. And so I in [17:43] some of this automated assessment let me [17:47] take more of that approach. Interesting. [17:49] My first instinct about the mastery [17:51] thing is I would have done very poorly [17:52] without in when I took math classes in [17:54] college because I felt like the [17:55] experience of taking high-end math [17:57] classes was like I was like holding on [18:00] barely keeping up trying to understand [18:01] what happened and then 6 months later [18:05] someone else was taking the class and [18:06] asked me about it be like oh yeah I know [18:07] how that works but like just somehow [18:09] like I I did couldn't quite get my arms [18:11] around it at the time but somehow just [18:12] like the passage of time after and then [18:14] you look back you're like oh [18:16] >> yeah I know how that fits together [18:18] >> but I'm probably under under imagining [18:20] like probably I could have like [18:21] demonstrated some local mastery. This is [18:24] like I think the demonstration of [18:25] mastery and also feeling really confused [18:27] or probably like consistent experiences. [18:30] Yeah, I think this is often applied in a [18:32] sort of maximal extent with much younger [18:35] students. Like in elementary school math [18:37] class, you have to like it's sort of [18:39] self-paced in a way where the class [18:41] might be at different points through the [18:43] material and you have to like master [18:46] uh addition or kind of long the long [18:48] division algorithm before you move on to [18:50] the next unit. I think as you get higher [18:53] level, you start needing to apply this [18:54] in more targeted ways and that the sort [18:57] of the flexibility is not there to the [19:00] same extent in a in say a college [19:02] course. Did you actually do the thing of [19:04] like having people stay on older topics [19:06] until they demonstrated mastery or is it [19:08] more like there's a thing that you [19:10] should do which is like keep on [19:11] iterating until you pass and then but [19:13] that but that happens at a fixed time [19:14] and you kind of have to like do it at a [19:16] particular point in time in the [19:18] semester. [19:18] >> Yeah. it it would not have worked for [19:20] people to like progress through [19:23] different different rates. Um not least [19:26] of which because uh students can [19:30] procrastinate is a thing people do. Uh [19:34] and I think it would have been a [19:36] disservice to let people pile up all the [19:39] work until the very end. So you do have [19:42] to give people deadlines throughout for [19:44] their own for their own good uh and and [19:46] force people to move on. And so it was a [19:48] model of like you have this short quiz [19:49] to do this week. You can attempt it any [19:52] number of times. Uh but you are just [19:56] going to like get sort of the best [19:58] result of the times that you take it. So [20:01] and the questions are set up in a way [20:03] such that it's pretty annoying to brute [20:05] force. Like the order of them keeps [20:07] changing, the order of the answers keeps [20:08] changing. So it's just going to be [20:10] faster if you actually like try and [20:11] understand all of it and then get 100% [20:14] that way. Have you have you thought much [20:16] about whether in the modern world some [20:19] of this feedback the kind of instant [20:21] feedback that you think would be more [20:22] useful for students can be generated by [20:24] LLMs? [20:25] Yes, I think there is quite a lot of [20:27] potential for that particularly because [20:30] when you have say a more [20:33] structured thing to sort of uh narrow [20:37] the focus of of the LLM. So, for [20:41] example, uh feedback on on code style as [20:44] opposed to an essay. I think I see a lot [20:47] more promise in um in addition to like [20:51] some automated grading of um test cases [20:55] and maybe some llinter to give students [20:57] feedback of like you need to follow [20:59] these rules, some additional kind of [21:01] code style pass uh I can see being very [21:05] useful. [21:07] That being said, in my experience, [21:10] there's also plenty of false positives [21:13] of LLMs giving feedback that is [21:16] misleading. Um, and so I think at the [21:19] very least you would want to frame it [21:21] carefully to students at least and uh or [21:25] find ways to really reduce the rate at [21:27] which it's kind of leading them astray. [21:29] >> Sure. [21:30] >> Somehow you ended up here. How did that [21:32] happen? [21:33] >> Right. So, I was living and teaching in [21:36] Minnesota. Uh, and uh, my brother and [21:39] his wife had moved to Brooklyn. Uh, [21:42] she's from Bay Ridge and I decided that [21:44] I wanted to live near them be uh, my [21:47] brother and I are close. So, I decided [21:48] to move to Brooklyn. Was not going to be [21:50] able to keep teaching in Minnesota while [21:53] living in Brooklyn. Uh, and it was very [21:56] fortunate that at that exact time, Jane [21:59] Street was trying to hire for this weird [22:04] educator programmer combo. I had just [22:06] assumed like there were some things I [22:08] liked about academia, but I was ready to [22:11] try something new and didn't really want [22:13] to go through the academic job market. [22:15] So, I figured I would go be a software [22:17] engineer somewhere. Uh but it's been uh [22:21] amazing to continue doing education [22:23] stuff in addition to doing some [22:25] programming. [22:26] >> Yeah. And it's maybe worth saying a [22:27] little bit about like the idea that we [22:29] had in creating this job, right? We we [22:32] really wanted I mean to take a step [22:34] back, teaching has been an important [22:36] part of what we do at Jude for a very [22:38] long time. Uh in part this is because we [22:42] do some weird stuff that we've not done [22:44] so much in the outside world. Some of [22:45] that is just subject matter stuff of [22:47] like nobody kind of comes in knowing [22:49] much about trading. And so there's a a [22:51] big effort in teaching people to [22:53] understand how to think about trading. [22:55] And then also our tech stack is in a [22:57] bunch of ways weird. And so there's a [22:58] lot of things that we want to educate [22:59] people about to understand the specifics [23:02] of the programming language and the [23:04] environment and the tools, a lot of [23:05] which are very different from what you [23:06] see on the outside. Uh and so we've [23:09] spent a long time a lot we've spent a [23:11] lot of energy on education for a long [23:13] time now. And at some point we're like, [23:15] you know, it's great to have people who [23:18] are on the ground involved in things [23:20] diving in and doing the educational [23:21] work, but maybe we could get some people [23:24] who have some expertise in the area and [23:27] have like actually done it for a [23:28] material period of time and try and like [23:30] leverage that effectively. So, well, [23:31] let's just like actually create a job [23:32] that's just for this. And it it seemed [23:36] clear to us that we wanted the people [23:37] who were doing the teaching to also be [23:39] people who were practitioners, right? [23:41] again in part because there's lots of [23:44] things that are unique about our [23:45] environment and if the people involved [23:46] don't directly experience and work in [23:49] the same ecosystem. They're just kind of [23:51] not going to be effective or compelling [23:54] teachers there. And so we designed this [23:56] as a kind of 50/50 split where the idea [23:58] is like you're going to spend half your [23:59] time programming and half your time [24:01] focusing on educational work. Some of [24:03] which is going to be directly teaching [24:04] classes or creating curriculum and some [24:06] of which will be helping other people [24:09] grow as teachers, right? because we [24:10] actually don't want there to be like a [24:12] small number de of developer educators [24:14] who do all of the teaching work but to [24:16] like help grow the practice and make [24:18] people better at it. I guess the other [24:20] thing that I find uh interesting and a [24:22] little frustrating about your [24:24] description is uh that like this putting [24:27] up this job ad did not at all work in [24:28] the way I wanted it to [laughter] [24:31] which say I just sort of thought this [24:32] was like an obviously awesome job and we [24:35] will advertise it and like great [24:37] candidates will fall from the sky [24:38] because like it's a cool and unique job [24:41] opportunity and it turns out like if we [24:43] had put this out like 3 months [24:46] differently or something we would have [24:47] not gotten you at all and in Fact, in [24:50] fact, actually, as an ongoing thing, [24:51] it's actually been pretty hard to like [24:54] find the right people for this spot. [24:56] Like, it's like I think it is less the [24:58] case that people in general, people [24:59] don't fall from the sky. I think hiring [25:01] people is really hard. [25:02] >> Yes. And I I think that this job is, as [25:07] you say, very cool, but also very [25:09] unusual. And the kind of person that [25:13] would like be looking for this job is [25:16] also therefore kind of unusual. Uh in [25:19] particular, most people with a lot of [25:21] teaching experience are in academia and [25:23] most people chose academia because they [25:25] are passionate about what they get to do [25:27] there and often are not looking to [25:31] leave academia and uh work at a place um [25:35] in in industry. The job is also a kind [25:38] of unusual marriage of of skills. like [25:42] if if I'm teaching courses at at a [25:45] university, I'm not necessarily doing a [25:47] lot of day-to-day software engineering [25:49] and I may or may not even be interested [25:51] in that being a significant fraction of [25:54] what I do, [25:55] >> right? And we really want at least in [25:57] the ideal case to find people who are, [25:59] you know, really good educators who like [26:03] have a love for the craft and have spent [26:05] a lot of time doing it and really think [26:06] hard about, you know, what kind of [26:09] things might you do and want to [26:10] experiment in the space and learn how to [26:11] be really better and are also just like [26:14] fully credible Jane Street software [26:16] engineers and like that's like the [26:19] intersection of two things where like [26:20] that intersection is relatively small. [26:22] um is maybe worth saying that like we're [26:25] we're really excited with how having [26:27] people in this role has turned out. [26:29] There's some amount of like taking some [26:31] internal people and kind of converting [26:33] them into this role which is like [26:34] another path. Uh but we're also excited [26:36] to get more people from the outside to [26:38] do this and I think there are just like [26:40] more and more domains where we want it. [26:41] Maybe it's worth like talking through [26:42] like some of the other spots that we are [26:45] trying to find developer educators for. [26:48] >> Yeah. One one thing I would add about [26:50] the the role itself is that I think one [26:53] reason it has been successful despite [26:56] Jane Street having been doing doing a [26:58] lot of education before this role [27:00] existed is having someone where that is [27:02] a sort of first class focus yeah [27:05] >> of their job uh as sort of a a step [27:08] change and kind of how they're going to [27:10] be able to engage and the sustained [27:12] attention on on education. Uh but yeah, [27:15] somehow there's just more and more stuff [27:17] we need to teach. [laughter] Um and uh [27:21] not just to, you know, new people who [27:23] need to learn Okamel, but it turns out [27:26] we're changing the Okamel language. And [27:28] everyone at the company, no matter how [27:30] long they've been [laughter] here, needs [27:32] to learn about it. There are all these [27:33] new AI tools. We want to teach everyone [27:36] how to engage with these effectively. [27:40] And it's sort of the case that the the [27:43] more we have kind of put into this and [27:46] expanded and refined it, the more places [27:49] we go like, oh, we should make that [27:51] better, too. And so the kind of uh [27:55] appetite for this kind of focused [27:57] educational expertise uh seems to have [28:00] only grown, [28:01] >> right? And so like on the O camel side, [28:03] we had we've posted this like Ox camel [28:05] educator uh spot uh for people who kind [28:08] of have like a strong PL background and [28:10] are also excited about teaching and [28:11] we've actually had some good luck on the [28:13] hierarchy there. Um we're still looking [28:15] for great people to do a kind of machine [28:17] learning educator role. Uh where kind of [28:20] focusing on like you know we're doing [28:22] all this exciting new you know [28:25] generation and training of exciting new [28:27] ML models and people who understand that [28:30] space and can and can teach more people [28:31] to do it is a very valuable thing. Um, [28:34] and just a kind of the general version [28:36] of this role that hits all sorts of [28:38] things like you mentioned, you know, [28:40] having to like, you know, teach everyone [28:41] a new how to how to use the exciting new [28:44] corners of Okamel as they come into [28:45] existence is one thing. Also, we now do [28:48] a bunch of Python, right? And we kind of [28:51] want everyone to know something about [28:53] Python. In fact, one of the things I I [28:55] worry about is people getting captured [28:58] by their tools. One of the things that's [28:59] nice about having a small number of [29:00] programming languages is like you can [29:03] kind of go anywhere into any codebase [29:04] and do stuff and once you have like two [29:07] or three or four programming languages [29:09] there's a kind of natural like striation [29:11] where people kind of get good at the one [29:12] thing and I'd really love to be more in [29:14] a world where we there are no like [29:17] okamel programmers and python [29:18] programmers they're just like software [29:19] engineers and they know how to use many [29:22] tools and can figure out what's the [29:23] right tool to use given the particular [29:26] circumstance but for some of that we [29:27] actually have to teach people there [29:28] people who have been here for a long [29:29] time who've like I mean everyone knows [29:31] some Python but like there are lots of [29:33] people who don't who aren't any good at [29:34] it and there's real things to learn to [29:36] be good at it and so that's like another [29:38] another corner where I think there's [29:39] more teaching plus we also invent our [29:42] own programming languages in spots and [29:43] teaching people about that yeah and like [29:45] a million other things so yeah I do [29:47] think that there's just more surface [29:51] area to cover more technology for people [29:53] to understand uh and therefore more [29:55] opportunities to like create real value [29:57] by teaching people stuff. [29:59] >> Yeah. And to to say another thing about [30:00] this sort of like general dev educator [30:03] role, I think a a model that uh I feel [30:07] like we're we're heading toward and that [30:09] sort of mimics some of the ways that uh [30:12] say designers have been incorporated [30:15] into sort of different teams and spots [30:17] throughout the firm. I think that may be [30:19] where we're headed for educators as as [30:21] well. So I mentioned AI tools. I think [30:23] we would love to have an educator like [30:25] sitting on kind of our AI assistance [30:28] team to focus on educa the education [30:31] work that they're doing and I think more [30:33] and more places will want sort of an [30:35] educator embedded there um for all the [30:39] sort of teaching materials that need to [30:41] come out of that part of the world. [30:43] >> Right. And I actually feel like there's [30:44] two benefits there. One is the thing [30:46] you're saying of like there's education [30:48] that's there are education needs that [30:49] are focused on what's happening in a [30:51] particular area and then also I think [30:53] you want the people who are doing the [30:54] education work to have a variety of [30:56] different technical experiences so that [30:58] they this sort of more of a body of [31:00] experience if you like poke them all in [31:02] the same corner of the firm they'll like [31:04] kind of have all the same context and [31:05] that's just like less of a robust [31:07] foundation. [31:08] >> Yeah. [31:08] >> Let's pivot for a second to talk about [31:09] like the technical work that you do. So [31:11] like where in kind of James Street's [31:14] technical organization are you and what [31:15] kind of engineering problems do you [31:17] think about there? [31:18] >> Yeah, so I'm on the editors team within [31:21] the tools and compilers group. Uh so our [31:24] team maintains uh the text editors that [31:28] folks uh at Jane Street use to write [31:30] code. uh VS code, Emacs, Neovim, uh as [31:33] well as some related services like the [31:35] code indexing service that we have uh [31:39] the Okamel language server [31:42] and I have sort of worked across various [31:44] things in the kind of dev tools space uh [31:48] started out working on uh an extension [31:51] to our feature management software iron [31:56] to make it much easier to split up work [31:59] into different piec pieces of of code [32:01] review and and organize it in a way that [32:04] uh was sort of awkward and manual [32:06] before, but giving people a nice tool to [32:08] be like take this and split it over here [32:10] and arrange it in this nice of chain of [32:13] of relationships that all get uh [32:15] reviewed separately. [32:16] >> Right. It's maybe worth saying when we [32:17] say the word feature at Jane Street, we [32:19] mean something that's like roughly [32:20] analogous to like [32:23] >> sometimes analogous to commit and [32:24] sometimes analogous to PR in the other [32:27] world in in like the outside kind of [32:29] world's dev tooling. Um I think the [32:32] thing that really strikes me about all [32:34] of the tooling we've built in this kind [32:35] of code review and feature management [32:37] area is just like it's actually made a [32:40] very different set of choices. like [32:42] we've been off on a like, you know, 25 [32:44] year alternative universe adventure of [32:47] building development tools that just [32:49] like look pretty different. And if you [32:50] look at iron, which is the the tool [32:52] that's responsible for the for code [32:54] review and and releasing features and [32:56] all of that, [32:57] >> it's like way better than everything in [33:00] the outside world and also way worse [33:02] depending on exactly which thing you [33:04] want to look at. And the thing you're [33:05] talking about this like having nice ways [33:08] of carving up and breaking down like a [33:10] larger collection change into into a [33:13] sequence of smaller changes is like a [33:15] good microcosm of like first of all it [33:17] has really good support for what we [33:18] sometimes call stacked PRs in the [33:20] outside world. Uh, and that's like built [33:22] in in like a really nice foundational [33:25] piece. And also the process of [33:28] really editing diffs and moving things [33:30] back and forth was awful, like slow and [33:33] painful. And the tooling wasn't very [33:34] good. And you would have been much [33:36] happier like using like, you know, I [33:38] don't know, like the Emacs Maggot [33:39] extensions or whatever in git than using [33:41] the stuff that we've had. It's just it's [33:43] just interesting that like there are [33:44] lots of different technical paths you [33:46] can take towards the same big picture [33:48] goal and end up with really different [33:51] like sets of choices and really [33:53] different functionality as a result. [33:56] >> Yeah. And there's a whole engineering [33:59] culture that flows from that. How you do [34:01] code review, how you think about kind of [34:03] giving people feedback on on their work [34:06] is entirely sort of shaped by the [34:08] underlying technical system, [34:10] >> right? And and actually I think this [34:11] comes to the question of like why does [34:12] Jane Street have an editor's team? [34:14] Exactly. Right. Like you know text [34:17] editors can't you just like use one of [34:19] them? Um and I I think it's just the [34:21] case that like editors are much more [34:23] central to how we work and how we review [34:26] code than you might imagine. I know it's [34:28] worth a little bit saying like like how [34:30] how are editors here like different than [34:32] maybe people have seen in the outside [34:34] world in terms of the role that they [34:35] play in our software development [34:37] process? [34:38] >> Yeah. So there are a few different [34:40] things. One at a fundamental level, uh [34:44] Jane Street has approached the text [34:46] editor as the software engineer's home [34:48] base. Like all the interactions with uh [34:52] Jane Street's code review and version [34:55] control uh and all these other systems [34:58] live within the editor. You do [35:00] everything there. It's all easily [35:02] keyboard navigable [35:05] and you're not sort of jumping between [35:06] different applications. uh uh or or [35:09] pages. And so that means that because [35:13] Jane Street does a lot of weird [35:15] different stuff like not using git but [35:19] using iron, uh we actually need a team [35:22] to build all this integration into the [35:25] editors so that people can actually live [35:28] there and not you know do some things in [35:30] the command line and some in the browser [35:31] and then write code in the editor and [35:33] sort of jumping between these different [35:34] surfaces. [35:35] >> Yeah. And I and despite the fact that [35:37] we've built a lot of I think quite cool [35:40] web UIs over the last few years and [35:42] tooling for building web UIs, Gain [35:44] Street is just like a much less web [35:46] focused place certainly than like you [35:48] know the big tech firms which I think [35:50] have all ended up leaning quite heavily [35:52] in terms of taking all of these [35:53] interactions and sticking them into a [35:55] web browser. And for us, yeah, it's just [35:57] like you're in the editor, you make the [35:59] changes, and the the editor becomes both [36:01] the place in which you interact with all [36:02] these tools and also like the central [36:04] location for understanding code and for [36:07] communicating about code. Like I think [36:09] another interesting early design choice, [36:11] which has like ups and downs, but I [36:13] think has certainly cemented the [36:16] importance of editors, is that the way [36:17] in which you give feedback on a change [36:20] is in the editor, right? there's like [36:21] specialized comments that you write with [36:23] a particular format to like mark a piece [36:26] of feedback rather than like going off [36:28] into some separate web UI and like [36:30] clicking a button and leaving a note on [36:32] a kind of PR that somebody else owns. [36:34] Um, and part of it is like this like [36:37] formal feedback comes through the [36:39] editor, but also because you're in the [36:41] editor, some of what you do when working [36:43] on a feature with someone is you just [36:45] collaborate on the feature. instead of [36:47] leaving a CR, which is the name for [36:49] these comments that we put in, you might [36:51] just fix the thing. You might just make [36:52] a change, which I think is like a great [36:55] it's like a it's certainly I think it's [36:57] a great part of how development works [36:58] here. It's also like weird and [37:00] unsettling, I think, for people who are [37:02] used to the other thing. It's like [37:03] someone else is reaching in and [37:04] modifying my PR. It feels unsanitary, [37:07] but it just like for good or ill or [37:10] maybe as one should say for good and [37:11] ill. Uh it's like just like a different [37:13] modality, a different way of interacting [37:15] within thinking about code and the [37:17] editors like just have a really central [37:20] piece in all of that. [37:21] >> Yeah. And and as you're saying, this [37:23] really biases us to keep the editor as [37:26] the home base because if you're [37:28] interacting with these systems in a [37:30] surface where you can't just as easily [37:34] jump in and edit the code, you then are [37:36] sort of losing that uh property of this [37:40] code review system and and property of a [37:43] code review culture that I think we see [37:45] is pretty positive. [37:47] >> What is challenging about this at a [37:49] technical level? like what are what are [37:50] the things that make it like hard to [37:52] make the editors do the things that we [37:54] want them to do. So one aspect is that [37:57] uh the editors that we're using are uh [38:02] things that exist in the outside world [38:04] vs code Emacs [clears throat] [38:06] and [38:08] for example one problem that we're uh at [38:11] work solving in VS code right now is [38:14] that uh in J's work we sometimes have [38:17] incredibly large diffs so a giant [38:21] configuration file or just a massive [38:24] okamo file And it turns out that the [38:27] diff algorithm in VS Code has some [38:30] serious performance [laughter] problems [38:32] once you have diffs of a certain size. [38:35] And sort of there's furthermore an issue [38:38] of if you try and open a large diff and [38:40] it hangs and you close it, it's now like [38:43] locked up the queue for viewing diffs at [38:46] all and you can't look [laughter] at any [38:48] others. Uh and so there there are just [38:51] things like this where [38:54] either there's there's some [38:56] functionality that we want internally [38:58] that isn't accommodated by this external [39:00] tool that we're building in. Um there [39:04] are also challenges where like we have [39:07] implemented some integration with the [39:09] iron code view system that messes with [39:12] some assumptions that uh say VS code [39:15] makes about what a buffer is going to be [39:18] what the uh URI for that buffer is and [39:21] that say like makes the uh built-in [39:24] editor search misbehave in a strange way [39:26] and we have to track that down and uh [39:29] and fix it. Uh I think that there are [39:32] also challenging things about the kind [39:35] of shared infrastructure between the [39:37] editors like the uh Okamel language [39:40] server uh and uh the underlying system [39:43] Merlin that provides uh type lookups and [39:46] and jump to definition for Okamel. uh [39:49] and there there's kind of lots of uh [39:52] interesting performance problems and [39:54] questions about what are the optimal [39:55] data structures and if we want to build [39:58] a index of the uh entire monor repo. Oh, [40:05] someone has uh added extra information [40:07] to do better uh to the the files that uh [40:10] the compiler produces to do better [40:13] lookups of of modules using this index. [40:16] Well, now this index uses uh many [40:19] gigabytes more of RAM and we can't [40:21] actually build the index on [laughter] [40:22] our boxes anymore. So there, as I'm [40:26] saying this, I'm realized a lot of the [40:28] problems we're trying to solve relate to [40:30] this kind of the scale of certain things [40:32] that we're doing at at Jane Street and [40:34] making these general purpose tools like [40:37] perform really well at these kind of [40:39] idiosyncratic use cases, [40:41] >> right? Right. And I think one thing that [40:42] that comes out of what you're saying [40:44] you're describing here is that the [40:45] editor has this kind of central role [40:47] where it's like the surface through [40:48] which many other things come at the [40:51] users. And so maybe the problems you run [40:54] into are problems about like the type [40:57] lookup system or this indexing system, [40:59] things that are like not like literally [41:02] part of the editor, but the editor is [41:05] the place where you see the issues. And [41:07] so you're kind of driven to like fix [41:08] things sometimes in the code review [41:11] system and sometimes off in the kind of [41:13] Okamel language stuff. Increasingly the [41:15] Python world is like another place where [41:17] we're spending a bunch of time actually. [41:19] Maybe it's worth saying like what what [41:21] has changed as we've needed to do more [41:24] support for Python. I'm kind of curious [41:26] what kind of pressures that puts on the [41:28] editors because in some should all be [41:29] easy, right? The whole world uses [41:30] Python. Now we can just do what [41:32] everybody else does. I think the main [41:36] surface uh for uh sort of folks in in [41:42] kind of trading and and research who are [41:44] interacting with with Python is the [41:45] Python notebook and uh this has been I [41:50] think the area that's been very [41:52] challenging. There are there are [41:53] different notebook solutions. they all [41:56] have different significant flaws or or [41:59] limitations. And I think it's been kind [42:04] of one of the pressures is even [42:05] understanding like what the different [42:08] use cases are and why people use one [42:11] over the other. and a sort of [42:13] unfortunate trend of uh spinning up a [42:17] new one that we hoped would replace the [42:19] old one except it has a a nonf fully [42:22] like a uh it has it's missing some [42:25] functionality that the old thing has and [42:27] so now we're maintaining multiple of [42:29] them and one other pressure is that for [42:34] people who are writing a lot of code in [42:36] in Python notebooks that code is not [42:39] always easily sharable and for example [42:42] Well, you might want to take it and put [42:43] it in a Python library uh that lives in [42:47] the repo that can be imported and shared [42:49] between multiple notebooks. And for [42:51] that, you'd really want people in uh a [42:55] service like VS Code where it's sort of [42:58] just as accessible to edit Python files [43:01] as it is notebooks. uh but for the at [43:04] least the kind of uh Python notebooks [43:07] work that people do here vs code has [43:10] been very challenging and has had a lot [43:12] of performance and and usability [43:14] problems to the point where we're now uh [43:19] starting to work on kind of just [43:21] building our own solution to this um [43:24] which is uh I would say a theme of a lot [43:28] of the tools work at chain street [43:30] >> right and I do worry a lot that we like [43:33] overlean in the direction of building [43:34] our own things. Um at the same time like [43:39] you do there's a lot of power in having [43:41] control and having everything built in [43:44] an ecosystem that you kind of understand [43:45] from top to bottom. So I actually feel [43:47] like you see work happening especially [43:49] in the editor space that's like shearing [43:51] in both directions right there's some [43:52] places where like in the notebook work [43:54] where we're like ah yeah now we're going [43:56] to like instead of like these multiple [43:58] things on the outside world none of [44:00] which exactly work in the way that we [44:02] want. None of them have quite the design [44:03] goals that we want. You know some of the [44:05] ones that look closer aren't maintained [44:06] anymore. Whatever. There's like a whole [44:07] bunch of problems with the kind of [44:09] notebook stuff in the outside world. [44:10] We're like, we're going to write a nice [44:12] little like O camel thing, use our our [44:14] own web UI stuff and like it's just [44:17] going to work smoothly and we have a [44:18] prototype that's pretty good and like [44:20] like I feel pretty optimistic about this [44:22] like being both not that hard to land [44:25] and also giving us a huge amount of [44:27] control to be able to adapt the notebook [44:29] to the very specific things that we want [44:31] to do internally and have like a much [44:32] better experience. At the same time, I [44:34] feel like a lot of the other work that [44:36] you guys have been doing is about like [44:37] dew weirdifying various aspects of our [44:40] of our uh setup. Like we have like, you [44:43] know, 25 years of like weirdo special [44:46] case Emacs stuff that we've built up [44:48] over time. And we've just been spending [44:49] a lot of time funding work on Emacs, you [44:52] know, figuring out how can we can like [44:54] take the weird custom thing that we do [44:56] that we've done internally and replace [44:58] it with the standard open source thing. [45:00] So I feel like actually both of these [45:01] directions are good. You just kind of [45:03] have to pick your spot. [45:04] >> If you can find a way to make the [45:07] upstream thing do what you want, that's [45:10] just so much nicer, [45:12] >> right? [45:13] >> Other people improve it for you, [45:17] >> right? The code that you don't have to [45:18] write is the best code of all. [45:19] >> Not only you not writing, but like lots [45:21] of other people are using it and [45:22] reporting problems and those are getting [45:24] fixed. So you don't have to find we [45:27] don't have to find all the the problems [45:29] uh ourselves by running into them. [45:32] >> And it turns out the upstream Emacs [45:33] people really understand Emacs well and [45:36] they know how the system works and they [45:38] have like the design aesthetic down and [45:40] like in many ways you just get better [45:41] stuff than you would get from our like [45:43] custom creations. [45:45] >> Yeah. all the different pieces uh work [45:48] together better, that some external uh [45:51] package will play much more nicely with [45:55] our internal extension if our internal [45:56] extension does it the sort of native [45:59] Emacs way as opposed to some totally [46:02] different approach that then you know [46:05] breaks when it comes into contact with [46:07] the outside worlds. [46:08] >> Right? Although a thing that pushes in [46:09] the opposite direction is like a bunch [46:11] of the extensions that we've done we've [46:13] written in Okamel which is like super [46:16] not the standard way. In fact we do this [46:18] across multiple editors for like Neoim [46:20] and Emacs and VS code. We have Okamel as [46:24] our like at least one of the major [46:26] implementation languages. You also have [46:28] to use the kind of more native language [46:30] to do some of the pieces. So how does [46:32] that like play into the kind of [46:34] technical tension? Yeah, it definitely [46:37] complicates the story and in EMAC in [46:42] particular, we're still we have some [46:44] ideas about we haven't sort of put the [46:48] boundary between uh elisp the sort of [46:52] native EMAC implementation language and [46:54] ecamel the flavor of okamel that we use [46:57] to write Emacs things like the the [46:59] boundary there particularly for [47:01] asynchronous things is pretty awkward [47:04] and so I think A long a long-term goal [47:07] would be sort of rearchitecting that to [47:11] have sort of fewer potential for weird [47:14] deadlocks when kind of these two sides [47:17] are waiting on each other or uh things [47:19] like this. [47:20] >> Is part of the issue here is that like [47:22] just Emacs is like super ancient and [47:23] does not have a great concurrency story [47:26] at all. [47:27] >> Yes. So so I think to some extent that [47:30] that's true. I think upstream Emacs is [47:32] actually working on this. uh I don't [47:34] think they uh and in particular there [47:38] were some changes to the the garbage [47:39] collector um that I think uh may play [47:43] more nicely with uh asynchronous things [47:46] but unfortunately that kind of wasn't [47:48] ready for the kind of current cut of of [47:50] Emacs that were working on upgrading to [47:53] Emacs and and and Vim were both sort of [47:55] like tools that have been around for a [47:57] while and [sighs] like uh have have some [48:00] limitations in terms of the like kinds [48:02] of interactions that they were intended [48:04] to support, [48:05] >> right? And even beyond that, just like [48:07] marrying the concurrency primitives of [48:09] two different languages is really hard. [48:11] We actually have run into like very [48:12] similar challenges around Python and O [48:15] camel where I think we have like now [48:17] maybe started to work out a story that [48:19] makes more sense. Uh but it's you know [48:21] but it's a tricky it's a tricky thing [48:23] even when you have like pretty good [48:24] concurrency stories on both sides. So to [48:28] kind of pivot a little bit, like I think [48:31] a thing that has affected kind of [48:33] everything about developer tools is the [48:36] emergence of like actual good agentic AI [48:39] uh for for programming. And I'm curious [48:42] how that has affected the way you think [48:44] about work in the editor and like you [48:47] know what you prioritize and what are [48:48] the things that you want to build. [48:51] Yeah, there are a few different pieces [48:53] to this one. And I think it hasn't [48:56] really affected our belief that the [48:58] editor can still serve as uh an [49:01] excellent home base for the software [49:04] engineer and that we want to make the [49:07] new workflows that are emerging sort of [49:10] just as natural and accessible via the [49:13] editor as the ways people were working [49:15] before. Uh and then there are also some [49:19] things that seem [49:22] uh like they aren't going to necessarily [49:25] change about what people are doing in [49:26] the editors. This is uh looking at [49:29] diffs, doing code review, uh working to [49:32] understand and and read code whereas [49:35] kind of how people write code may be [49:37] changing and is changing significantly. [49:40] There are certainly plenty of engineers [49:43] all over the place, but also at Jane [49:44] Street who just like don't type code [49:46] very much anymore and most of the typing [49:48] of you know they type more in the way of [49:50] prompts and but they're still reading [49:51] code and they're still like looking at [49:53] the types of things and looking at the [49:54] definitions of things. So they're doing [49:55] lots of things you do in the editor but [49:57] like maybe less editing. [50:00] >> Yes. And this I think has created quite [50:02] a lot of uncertainty for people working [50:04] on editors in terms of [laughter] what [50:06] is it that is useful to build now. what [50:09] is it that is going to be useful a year [50:12] from now will will anything [laughter] [50:16] uh and one thing that I've gotten quite [50:19] interested in recently particularly [50:21] because the editors team has been [50:23] investing in improving the sort of [50:24] telemetry and observability of what [50:27] people are doing in editors and how [50:29] they're interacting with uh the oamel [50:32] language server [50:33] is trying to use this telemetry to [50:37] understand how is Is it that people's [50:39] work is actually changing? You know, I [50:43] looked the other day I I had a [50:45] hypothesis that well, if I look at the [50:49] number of times per user that people uh [50:52] jumped to definition in Okamel, if [50:55] people are if if the LLMs are producing [50:59] good code and people aren't sort of [51:01] writing and having to to look up APIs as [51:03] much, they'll be jumping to definition [51:05] less on a per user basis. But then in [51:08] the telemetry, it seemed if anything, [51:10] they're doing it slightly more, which [51:12] was quite surprising, but also maybe [51:14] relates to people are actually needing [51:16] to read and understand more code than [51:18] they were before. And so I'm excited for [51:22] this kind of data to guide what is it to [51:26] help us understand like what is it that [51:28] people are actually doing in editors [51:29] because it's kind of been quite [51:31] amorphous as things have been changing [51:33] very quickly. uh and using that to help [51:36] us decide what is important to invest in [51:40] and and what are the sort of workflows [51:43] to make better because questions like [51:45] should we try hard to make jump to [51:47] definition faster that depends if people [51:50] are using it more or less. [51:53] >> Yeah. No, it makes a ton of sense and [51:55] and the thing you're proposing of like [51:57] using the data to drive seems more right [51:59] than my instinct which is like [52:00] theorizing the air about it. But on the [52:03] theorizing the air part, it does feel [52:04] like understanding is like at the moment [52:08] the thing that's like most important, [52:10] right? You know what's the, you know, [52:12] the cost of creating a plausible looking [52:14] PR? I think this is I'm stealing [52:15] something from Charlie Marsh. I think [52:17] like the cost of making a plausible PR [52:18] has like dropped nearly to zero and the [52:20] cost of verifying it is about the same [52:22] or maybe worse, right? I think notably [52:25] like the there's something kind of [52:29] uncanny about a lot of the features [52:31] generated by LLMs in that they like [52:33] they're very smooth. They like look very [52:36] nice. A lot of like tells of things that [52:38] you might think of as like things that [52:39] are like indications of a problem just [52:42] kind of aren't there, but they're often [52:44] still super broken, [52:45] >> right? And so the process of reading [52:47] through and understanding is at a [52:49] minimum different and in some ways [52:51] certainly harder. And so the idea that [52:52] we need to like spend more energy and [52:56] engineering work making it easy for [52:58] people to understand what is happening [53:00] seems really important. I think some of [53:02] that understanding should come from [53:05] further leveraging of LLM. I think LLMs [53:07] are like a really powerful explanatory [53:08] tool. Um, but I I think it's easy to [53:11] underrate the value of kind of [53:14] traditional like hard ways of extracting [53:18] data from code and surfacing it in like [53:19] I think things like go to definition and [53:21] inferred types and test results and [53:24] things like that are all are all really [53:26] exciting. [53:27] >> Yeah. Uh, I've also been thinking more [53:30] about people's actual sort of workflows [53:33] when getting LLMs to write code and how [53:36] that relates to [53:39] uh sort of understanding it at the end [53:42] result. So, one thing that I have played [53:46] around with in some of my education work [53:48] is Claude codes learning output style, [53:52] which changes the system prompt to tell [53:54] it like stop sometimes and leave a to-do [53:57] human sort of comment in the code and [54:00] ask the human to go fill in that piece. [54:04] And in using this myself, I found that [54:06] having it stop partway through and then [54:09] I had to read what it had written. so [54:12] far and understand it enough to fill in [54:15] this piece. I felt like I came out at [54:18] the end sort of both have had a sort of [54:22] more pleasant experience of producing [54:24] this code but also a more pleasant [54:27] experience and I think effective and [54:29] actually understanding what it had [54:30] produced and I'm curious to explore [54:36] not just kind of on the output end [54:39] adding more explanatory power and tools [54:42] to the code that's already produced but [54:44] thinking about how do people interact [54:46] act with these models as they're [54:48] producing the code to most effectively [54:51] end up with a solid understanding of it [54:53] at the end and not just be guided. Well, [54:56] it feels more efficient to have it write [54:58] everything and then read it at the end [55:00] as I'm become a little skeptical that [55:03] that is actually sort of the optimal way [55:06] to approach it. [55:07] >> It's interesting. It reminds me a little [55:08] bit of the thing we were talking about [55:09] about like the utility of taking notes [55:11] while writing with a pencil. The idea [55:13] that like you are involved in the [55:15] production function will cause you to [55:16] better understand the thing at the end [55:18] and there's maybe a kind of infilicity [55:20] in the current way the tools are set up [55:22] in that they largely like kind of [55:25] everything is kind of tilted in the [55:26] direction of the model generates the [55:28] code and then you decide whether or not [55:30] to accept it and maybe that's wrong. [55:33] That's a fascinating idea. [55:34] >> Yeah. I was giving a talk to a group of [55:37] interns about code review recently and [55:40] had a slide. you know, code review is [55:42] hard. Uh, writing code, you're sort of [55:45] incrementally building your [55:46] understanding. And when you're catching [55:48] mistakes, you're sort of correcting your [55:49] mental model. And when you're reviewing [55:52] someone else's code, they're just [55:53] presenting you with the finished result. [55:55] And you kind of have to awkwardly kind [55:57] of build your own mental model of it. [56:00] And it immediately struck me, wait, this [56:02] is exactly what working with LLMs is [56:05] like in the way that it makes it harder [56:08] to have a mental model of the code you [56:10] are authoring. That sort of started me [56:12] more firmly down this path of yeah, [56:15] maybe we should be interacting with [56:17] these things differently at least some [56:19] of the time. That's really interesting. [56:22] There's just a lot of space for [56:24] innovation here and like new methods of [56:27] approaching like just like LLM suddenly [56:29] make the things that you can do. It's [56:31] like a very different cost structure and [56:32] there's a very different opportunity [56:34] set. And so like the thing you're [56:36] talking about with like you know the [56:37] cloud code learning mode thing is [56:40] something where you can like integrate a [56:42] kind of tutor like behavior into the [56:45] automation and like that's just like [56:47] dramatically cheaper of a thing to have [56:49] done than you know before you have to [56:51] like go off and like go and talk to [56:52] another person to get that experience to [56:55] happen. I also feel like in the editor [56:56] space in general, there's a lot of room [56:58] for custom integrated visualizations and [57:02] ways of understanding the data and that [57:04] like the models are really good at [57:05] jinning these things up and like the [57:07] idea that we should be able to ship [57:09] along with a feature a custom [57:11] visualization that you made for [57:13] understanding that diff like that just [57:15] kind of shows up in the middle of [57:16] review. Like that's a thing that we [57:18] could plausibly do now that would have [57:19] made no sense three years ago. [57:22] >> Yeah. And and one thing that I'm excited [57:25] to explore that connects to your point [57:27] about these things being very good at [57:29] explaining is that you could also [57:33] sort of bake into a feature some uh sort [57:39] of pre uh precomputed sort of [57:41] segmentation of the code into sort of [57:43] logical chunks and then a pre-baked kind [57:47] of explanation of each of those chunks [57:48] that you can just call up with a single [57:50] keystroke as you're reviewing the code. [57:53] Uh I mean as you're doing it, you can of [57:56] highlight it and ask a model to explain [57:59] the piece, but that's some additional [58:01] friction. The response doesn't [58:04] necessarily come back that quickly. And [58:06] if like for any given piece, you could [58:08] sort of call this up. I think that that [58:09] might also be a sort of powerful tool [58:12] for helping people [58:14] more easily understand the increasing [58:17] volume of of code they're being asked to [58:19] review, [58:20] >> right? And I think this is just like the [58:23] modern condition of a software engineer. [58:24] And by modern I mean for the last eight [58:27] months or something uh is that there's [58:31] just a lot more review to do as a [58:34] proportion of your time. You're just [58:35] spending much more time reading and [58:36] understanding and so suddenly the value [58:38] of making that better just like shoots [58:41] up by a lot. I mean it was already a lot [58:42] of work so it was already somewhat [58:43] valuable to do it but like it's it's now [58:46] much more clearly the rate limiting step [58:48] in all of this work. So, we're talking a [58:50] bunch about how AI affects the [58:54] experience of developing code and [58:56] understanding code. And this like [58:59] resonates, I think, a lot with the [59:00] question of like how does AI affect the [59:02] process of teaching people stuff? So, [59:04] I'm kind of curious what what that has [59:07] felt like as you've been kind of helping [59:10] to run education programs here amidst [59:12] the middle of this AI revolution. [59:15] >> Yeah. So this is I would say quite a [59:17] thorny problem because you're faced with [59:20] this tension of like one of the things [59:23] that I uh help work on and and maybe we [59:27] can talk more about this but uh this uh [59:30] workshop for early career engineers at [59:34] at Jane Street focused on techniques for [59:36] software testing. Um and we call these [59:40] workshops workshops uh teachins. [59:43] So this testing teach uh teaches people [59:46] how to interact with various libraries [59:48] for property based testing or for uh [59:52] deterministic control of of time in [59:55] tests. And [snorts] [59:58] when you're teaching someone how to use [60:00] an unfamiliar library, well an LLM could [60:04] just write that for them. So what is it [60:08] about this library that someone actually [60:11] needs to know in practice and what might [60:14] you consider sort of boilerplate that [60:17] you don't need? [60:18] >> The initial scaffolding can be made [60:19] really easy, right? The models can do a [60:21] decent job of that, but what's the [60:24] understanding that you have to give to [60:25] people? [60:26] >> Yes. And so I think so far we've come [60:30] down on the side of people having to [60:34] think through this these sorts of [60:36] problems and implement them implement [60:38] them themselves at least once is [60:41] actually pretty important for coming out [60:43] of it with a sufficient understanding [60:46] and even if what they are doing [60:48] subsequent to that is uh reading [60:52] applications of this technique initially [60:54] authored by an LLM um building that [60:57] understanding by having done it [60:59] themselves once is sort of a critical [61:00] input to then being able to evaluate [61:03] those things. That is one piece of it [61:06] and I think we are actively thinking [61:09] about like where to draw the line in [61:11] terms of there's some stuff that we were [61:12] probably asking people do in the past [61:14] because it was sort of uh stuff that [61:17] they had to do even though it was [61:18] somewhat toilome that yeah we should [61:20] probably not have kind of people try and [61:23] learn how to how to do those pieces [61:26] but there's also [61:28] I think the challenging question of how [61:30] do we teach people and even what do we [61:33] teach people about using these tools [61:35] effectively, about how to integrate them [61:39] with their workflow, about how to [61:42] understand the ways in which they can be [61:44] effective or lead you astray. And I [61:47] think this is still in very early days. [61:50] I don't think we have uh sort of settled [61:55] answers to this and the tools are [61:57] changing very quickly, [61:59] >> but you can already see problems, right? [62:00] There are definitely people who run into [62:01] problems where they just like kind of [62:03] vibe it up too much and like use the LLM [62:06] to generate stuff and do not think [62:08] >> and like that's a real problem. Like you [62:10] still need to think. I actually worry [62:12] more generally about what it means for [62:14] the overall education system. Like I [62:17] think a kind of critical component of [62:19] learning is struggling with the [62:21] material. And the world where you always [62:25] have a genie at your beg and call beck [62:26] and call that you can just like summon [62:28] up to solve the problem is itself like a [62:31] real problem. And the overlap of [62:35] problems that are easy enough that they [62:37] are suitable for someone who is learning [62:40] to do and hard enough that an LLM can't [62:43] just oneshot it is like close to the [62:45] empty set now. [gasps] [62:47] Um so it's like a really challenging [62:50] problem. I mean it really does make me [62:52] think back to the like doing stuff on [62:54] paper without a computer. Sometimes an [62:57] important thing about making the [62:59] educational process work is withholding [63:02] certain affordances. And I worry that we [63:04] have like a whole generation of people [63:06] who are like learning in this new [63:08] environment where we don't know what to [63:10] do. We don't know what affordances to [63:12] provide and which ones to withhold. And [63:14] like I worry that we're like not [63:16] building a good educational system for [63:19] people in this world. [63:20] >> Yeah. I I think you and pretty much [63:23] everyone else [laughter] involved in [63:24] education uh is is deeply worried about [63:28] this. Uh to your to your first point [63:31] about like how do you get something [63:33] that's uh small enough to be tractable [63:37] for students and large enough to not be [63:40] trivial with an LLM? like we have [63:42] [snorts] run into this exact problem [63:44] multiple times uh with new curriculum [63:46] we've built and we've sort of repeatedly [63:48] ended up the thing we're asking people [63:50] to do is too large for the time we have [63:53] like we are asking them to do something [63:54] where they do have to put a lot of work [63:56] into the LM code LLM code to make it [63:59] good and actually do what they want but [64:01] that just takes too much time to do well [64:04] given sort of the amount of time we have [64:07] on [snorts] the education piece and I [64:09] think it's going to take more [64:10] experimentation to figure out where that [64:13] where that lands. [64:15] But yes, I think that figuring out where [64:18] where to slot using these tools into [64:22] sort of traditional education of I it [64:26] feels inevitable that we're going to end [64:28] up with some sort of structure where you [64:30] do uh this piece or this set of [64:33] activities without the aid of LMS and [64:37] then you do some with and I think [64:41] figuring out kind of how to make that [64:44] pairing and what you're trying to get [64:46] out of each of them uh is something that [64:48] a lot of people are going to be [64:50] experimenting and figuring out. [64:52] >> Yeah. And I think it'll be interesting [64:54] over the next few years as we're doing [64:56] this increasingly with cohorts of people [64:58] who have not spent that much time typing [65:01] code themselves. [65:02] >> Right. they'll just be like they there's [65:04] like a core learning mechanism that [65:07] maybe they haven't had enough experience [65:09] with and yeah I wonder I wonder if that [65:11] will change what we have to teach people [65:13] in the end. Yeah, there's there's been a [65:16] lot of conversation about um [65:21] things along the lines of that cloud [65:23] code learning mode where what LLM tools, [65:29] if any, should we provide to folks going [65:33] through Okamel Boot Camp learning Okamel [65:35] for the first time? And if they're just [65:37] having LMS write all the code, well, [65:39] doesn't seem like they're going to learn [65:40] anything. [65:40] >> Yep. Uh, but it also [65:44] feels a bit odd if people are going to [65:46] leave this and then LMS are going to be [65:48] a core part of their day-to-day work. [65:51] And so not having those be at all a part [65:54] of this also feels like maybe not a [65:57] great match for what we want to teach [65:59] them. And so can we come up with prompts [66:03] something a mode where the LM will just [66:05] sort of ask you questions to check your [66:08] understanding or we'll leave you large [66:11] pieces to do yourself like we'll sort of [66:13] set up a kind of skeleton but actually [66:15] ask you to do most of the the [66:17] implementation and I think there's going [66:19] to be a lot of experimentation with that [66:20] as well [66:21] >> and maybe the the the the thing that's [66:23] been talked about a lot in educational [66:24] circles of setting up the LLMs to be [66:27] effective tutors right so that you can [66:29] there's a part you're supposed to write [66:30] yourself and maybe you get stuck on [66:31] something and maybe you can get the LLM [66:32] to give you a hint but not to go off and [66:34] write the code for you so that you do [66:36] have the opportunity to like actually do [66:37] the hard thinking yourself. [66:39] >> Yeah, this is actually a category of [66:41] thing that I did research on in grad [66:43] school of an automated tutor that could [66:45] walk you through kind of an arbitrary [66:48] example problem where you had some [66:50] algorithm that you wanted to teach [66:52] people to follow to solve this kind of [66:55] problem. and with the idea that sure you [66:59] had the one hour where you could go talk [67:00] to a TA and get one-on-one help and that [67:02] was the most effective thing you could [67:04] do, but if you were at other times [67:07] studying for an exam, it might be really [67:08] useful to have this help like not [67:12] [clears throat] as effective but still [67:13] helpful sort of tutor that is infinitely [67:16] patient and can walk you through any [67:18] example. Yeah, I I think that that is [67:21] likely going to become a large part of [67:24] of at least higher education. [67:26] >> Makes sense. So, we just talked about [67:29] this testing teaching, which is part of [67:31] like the kind of formal educational [67:33] program that we have for new engineers. [67:35] Maybe it's worth walking through like [67:37] what what do we do to kind of onboard [67:39] people and teach them about our world? [67:42] >> Right. So, uh, a new software engineer [67:46] joins and the first thing that they're [67:48] likely to do is Okamel Boot Camp, uh, a [67:52] sort of cumulative series of exercises, [67:55] building up a little, uh, client server [67:57] app in Okamel, uh, and learning the [68:00] different core libraries and, uh, [68:02] getting their code reviewed by a mentor [68:04] and starting to understand some of Jane [68:06] Street's style. And uh from there we [68:10] have something called a production boot [68:12] camp where go through a series of [68:15] exercises to learn how uh applications [68:18] are actually deployed and monitored and [68:21] configured uh at Jane Street and that [68:23] forms say like the first two weeks [68:26] someone is on the job and then and to me [68:29] we're [clears throat] saying like that's [68:30] really been refined a lot over the [68:32] years. I think like the O camel boot [68:33] camp itself like early versions of that [68:36] are something like 20 years old. Um uh [68:39] but there's been a lot of work uh in [68:41] fact you've been involved in a bunch of [68:42] of the recent stuff to kind of refine [68:44] and hone and make that better. And the [68:45] fact that like nowadays like someone can [68:48] arrive and get through that in about a [68:50] week and then like you a little more for [68:52] the production boot camp is already like [68:53] a really good improvement like I think [68:54] we've been able to make that process [68:55] more efficient and I don't know people [68:57] who come in to Jane Street there are [68:59] good and bad things. people aren't, for [69:01] example, always super enthused about [69:02] like the overall state of all of our [69:04] documentation, which, you know, has its [69:06] ups and downs, but I feel like I've [69:07] gotten, you know, especially in the last [69:09] few years, really positive reviews from [69:12] people of like these early boot camps of [69:14] like they really, I think, do a really [69:15] good job of like setting people up with [69:17] the basics that they need to do and [69:18] doing it in a really efficient way. So, [69:20] I think that's been a nice piece of [69:21] progress. [69:22] >> Yeah. One thing that we've worked hard [69:24] on which I think is important when [69:26] teaching this kind of material is [69:29] introducing new concepts in context [69:32] where it's motivated by some specific [69:36] application, some feature you want to [69:37] add to this system you're working on. [69:40] Uh, and I always frame this as the [69:42] mistake people make is they introduce a [69:45] tool and then go in search of a problem [69:47] that it solves rather than like we're [69:50] working on this thing now. Now, we want [69:52] to do this. We don't have a good way to [69:54] do this, but wait, here's this thing. [69:55] [laughter] [69:56] Why don't we use this? And yeah, I think [69:58] that makes it smoother. That helps [70:00] people like have sort of hooks that they [70:02] can hang their understanding on. Uh, and [70:04] so that's been a lot of the work to to [70:06] rework these intro materials. Cool. [70:08] >> From there, in [70:10] someone's first year or so at the firm, [70:14] they'll go through this series of [70:15] teachings. So, I mentioned the the [70:17] testing one. Uh there's one on uh sort [70:21] of uh writing performant uh Okamel code [70:25] and the tools to analyze the performance [70:27] of Okamel code. Uh there's uh one on [70:32] systems debugging all the different [70:33] tools we have for understand what might [70:35] be going wrong with some system. Uh [70:38] there's one on advanced functional [70:40] programming, a bunch of the advanced [70:41] features of Okamel and cool ways to [70:43] apply them. [70:45] Uh and then there are also some uh more [70:48] specialized concepts. Uh a teaching on [70:52] bonsai the library for uh writing uh web [70:56] pages and web UIs via o camel. Uh [70:59] there's one on market data how the the [71:01] systems for market data work at at Jane [71:03] Street which very important to to some [71:05] people but other desks not so much. And [71:08] I think as we add more, we're going to [71:10] be even more in the mode of it's sort of [71:11] a course catalog. And you and and your [71:14] manager or folks on your team sit down [71:17] when you when you start and sort of [71:18] think about which of these are relevant [71:20] to you to do when uh and kind of almost [71:24] devise a sort of course of study uh for [71:27] your first year at at the firm. and and [71:30] we also try and create these materials [71:33] in a way that's amendable to self-study [71:34] and you could also sort of go and work [71:37] through them on your own sort of at the [71:39] point where you actually need to learn [71:40] and and apply the material. [71:42] >> Yeah. One one of the interesting things [71:43] about the teachings and about how those [71:45] have evolved over time. Like they were [71:46] initially built as a kind of reflection [71:48] of a trading side education program also [71:50] called teachins where that was mostly [71:52] organized around basically going around [71:55] from trading desk to trading desk [71:57] learning the kind of core concepts that [71:59] were important for that desk. And my [72:02] mental model of it early on was like, [72:04] you know, there's some specific [72:06] technology you want to teach people [72:07] about, but the more important thing is [72:10] the kind of conceptual layer of like, [72:13] you know, there's, you know, bonsai is [72:14] like a web framework, but it's also an [72:16] introduction to incremental computing, [72:18] right? And you know market data is like [72:21] a particular domain you need but it's [72:23] also thinking about you know handling [72:27] like complex streaming data and building [72:30] modular and efficient protocols for kind [72:32] of consuming those in a good way. Like [72:34] there's a bunch of architectural lessons [72:35] from each one of these. And I feel like [72:37] that initial vision sounds good and has [72:40] been less successful than I expected to [72:42] be. and we've kind of over time gone in [72:44] a direction that's been more about [72:47] giving people like the practical tools [72:48] that let them go do things. I'm kind of [72:50] curious how like how do you think about [72:52] that? Like how does the conceptual piece [72:55] fit in? When and where should we be [72:57] focusing on that and to what degree [72:58] should we just giving people what feel [73:00] like the most practically useful tools? [73:02] Yeah, this is a very interesting [73:05] difference in the Jane Street context [73:08] compared to say college courses which in [73:13] a college degree you know maybe I'm [73:16] teaching a databases course and there's [73:17] some domain knowledge about databases I [73:20] want to impart but as part of their [73:22] university education I also want to [73:25] teach them to be capable problem solvers [73:27] and how to uh work through ambiguity or [73:32] try out different solutions. Kind of a [73:34] lot of these general skills and uh one [73:38] of the great things about working at [73:40] Chain Street is that my colleagues are [73:43] very smart and capable and I don't [73:46] really need to use their time to teach [73:49] them problem solving skills. Uh [73:51] >> they're already really good problem [73:52] solvers. It turns out [73:54] >> uh there's there's some selection effect [73:56] there. And [73:59] so this I think has biased the education [74:03] away from what you might see in [74:04] university of like we're going to have [74:06] you spend a bunch of time grappling with [74:08] this tricky problem and figuring out a [74:12] solution. Um because actually when [74:15] people are at their desk in their [74:17] day-to-day job that tends to be what [74:19] they are doing and it therefore can feel [74:24] pretty I guess unsatisfying to do that [74:27] on a kind of canned educational thing as [74:31] opposed to something that's more [74:32] directed at here are like useful tools [74:34] that we're going to like have you work [74:36] through and understand which you would [74:38] then go apply to like the truly tricky [74:40] problems. So I think that you're right [74:43] that there are some core conceptual [74:46] things like the uh of incremental [74:49] computing things like this which we do [74:52] want people to be exposed to and think [74:55] about but we tend not to do that by [74:59] asking them to like take on some like [75:01] particularly sort of thinky [snorts] [75:04] piece of it to kind of spend several [75:07] hours just kind of [75:10] uh having to come up with a a kind of [75:12] novel solution to it. [75:14] >> What you're saying here is even if a big [75:16] part of your goal is to convey the [75:17] conceptual lessons, [75:19] it will work better if the way you do [75:22] that is by teaching them like concrete [75:25] useful tools and focusing on those and [75:27] like some lessons will sneak in that [75:30] way, but not because you shaped it [75:33] explicitly about that, but because [75:34] understanding the tool and being able to [75:36] use it required the understanding of [75:37] those lessons. [75:38] >> Yeah, I I think that's right. And also [75:41] teaching this conceptual knowledge just [75:43] in a different way than you would in say [75:46] a university setting. And doing it more [75:51] often more in ways that look like a [75:53] walkthrough of let me take you through [75:56] this uh kind of problem and how you [75:59] would solve it. and you'll do some [76:01] pieces but how you will sort of build [76:06] the more complete understanding is when [76:08] you actually go and apply this to [76:12] the the problems you're sort of solving [76:14] on the desk. [76:15] >> So this is a little bit about some ways [76:17] in which teaching at Jane Street is [76:18] different from teaching the outside [76:19] world. I feel like a thing I learned [76:21] from you early on uh is about some of [76:23] the ways in which like lessons from the [76:25] outside world apply here. So we talked a [76:27] bunch about the testing teaching. This [76:28] was actually like the first real project [76:30] that you did here where the way I [76:32] remember it is we were trying to [76:33] organize this testing teaching before [76:35] you started um and it was going badly [76:38] like [laughter] [76:40] um a little bit because it was is this [76:42] problem of like it was hard to quite get [76:43] enough of anyone's time to like you know [76:45] people were doing you know good work and [76:46] focusing on various parts of it but [76:48] there wasn't like a real effort to kind [76:50] of bring the whole thing together and [76:51] like make the whole thing good and land [76:53] it. Um, and I remember you asking me [76:55] early on cuz like the shape of it was a [76:58] thing I had come up with. And so you [77:00] asked me, so like what are the learning [77:01] goals? And I was like, fascinating. [77:04] What are learning goals? Like it sounds [77:07] like, you know, this, you know, those [77:08] are like English words that kind of make [77:09] sense together, but clearly you mean it [77:11] in a more specific way. Um, and I found [77:14] that to be like in the end like a very [77:16] useful and insightful way of thinking [77:18] about teaching. Maybe can you like say a [77:20] few more words about like what are [77:22] learning goals and like [77:23] >> how should that affect how you design [77:25] curriculum and build courses? [77:27] >> Yeah. So when starting off on like any [77:31] larger project, it's good to write down [77:33] what you are trying to accomplish. [77:35] [laughter] [77:35] And in this case, learning goals are a [77:38] specific way of writing down what you [77:41] want to accomplish in terms of what is [77:43] changing about the students as a result [77:46] of the thing that you're doing. And [77:48] specifically uh I find it very useful to [77:51] to write them in the form after [77:54] completing this teach and say students [77:57] will be able to and then write down a [77:59] bunch of active verb phrases. So for [78:02] example like understand property based [78:05] testing is not a great learning goal [78:08] because that's very vague. It's not a [78:11] specific thing that they're going to be [78:13] able to do. [78:14] >> And how would you measure it? how it and [78:16] hard to know how you would measure it, [78:18] but students were able to like write a [78:20] property base test like a a sort of [78:23] nested record type input. So, sort of [78:26] some sort of concrete like scoping what [78:29] is the complexity and the sort of thing [78:31] they're going to be able to do. Um, one [78:34] that really forces you to structure your [78:36] thinking about what am I actually trying [78:39] to accomplish here and importantly, what [78:42] am I not trying to accomplish? like what [78:44] is not a good use of students time [78:46] because it's not actually a goal to have [78:48] them do this thing. Um but then in terms [78:50] of the overall design of the educational [78:53] program once you have written down [78:55] pretty specifically what you want people [78:57] to do or what you want people to be able [78:59] to do afterward. Well, it follows pretty [79:02] directly what you should have them do in [79:05] the thing that you're running because [79:08] they should practice the things that [79:09] they're going to be able to do when [79:11] they're at the end. And okay, now I know [79:14] the sort of exercises or activities I'm [79:16] going to have them do. Okay, what is the [79:18] background that they need to know for [79:20] this? So what lectures or kind of other [79:24] materials or what parts of code would we [79:26] want to provide them for which writing [79:28] those is not relevant to these goals? [79:30] And so the whole [79:33] uh process of developing some new piece [79:36] of curriculum sort of flows nicely out [79:39] of having been sort of disciplined in in [79:42] writing down these goals. And I feel [79:44] like to some extent my education work at [79:47] Chain Street has been going into [79:49] different rooms with different people [79:51] and saying the word learning goals. [79:53] [laughter] [79:54] We should do this. [79:56] >> Right. And the other the other thing [79:57] that I found interesting about the [79:59] learning goal idea is it tells you [80:00] something about how you should think [80:01] about evaluation. [80:02] >> And here like we mostly are not so [80:06] interested in our teaching stuff to [80:07] evaluate the students. [80:09] >> But you really want to evaluate the [80:10] program of like did the thing you do [80:12] actually teach people anything useful. [80:14] And so the idea that you could like see [80:16] whether or not at the end that people [80:18] are able to do the thing that you wanted [80:20] them to do is like a way of evaluating [80:22] yourself in your own work as an [80:24] educator. Mhm. [80:25] >> And I guess the other thing that I think [80:26] it was striking about is I think that [80:28] like learning goals has now showed up [80:29] for me as like in some sense two [80:31] different things. One is there's this [80:34] like somewhat more specific way of [80:36] writing them out and thinking them [80:37] through and using them to structure the [80:39] program. And then there's also just the [80:40] frame of mind of like how are the [80:43] students going to be different at the [80:44] end, right? Like just what are you even [80:46] doing here? Like why did you show up to [80:48] work today? What are you trying to [80:49] achieve? Um, and I found actually that [80:51] framing to be very useful in talking to [80:56] like potential developer educators, like [80:58] talking to people who we want to hire to [80:59] do education work is like, you know, I [81:01] think some people have really thoughtful [81:02] answers what they were trying to achieve [81:04] in the classes that they they taught. [81:07] And sometimes it was like, oh, I just [81:08] like this material, so I taught a course [81:09] about it. And like, I don't want to [81:12] complain too much. I think a lot of [81:14] great classes are just like someone is [81:16] excited about material and they teach [81:17] it. And like an enormous amount of good [81:19] education is not about process but is [81:21] about like charisma and ideas and being [81:24] able to like just carry the performance [81:26] through and really caring about the end [81:28] results. Um but I do feel like the [81:31] structure helps and especially helps [81:34] people who haven't done it before right [81:36] people who are trying to figure out how [81:37] to teach classes for the first time. I [81:38] think it's like very helpful kind of way [81:40] of organizing your work and your [81:42] thought. [81:43] >> Mhm. And and it forces you to focus on [81:47] things that are sort of not just in the [81:50] students head to some extent of of what [81:54] am I trying to accomplish here that [81:56] isn't just sort of like [81:59] I tell the student a thing and now they [82:01] know it. uh but yeah kind of how are [82:05] they changed in some sort of observable [82:07] way. [82:08] >> So are there other aspects of like the [82:11] outside like learning curricula and [82:14] approach like you know the whole world [82:17] of like education the whole world of [82:19] research on education is like [82:21] complicated and messy and also like [82:23] includes lots of conclusions that like [82:25] probably aren't true and also lots of [82:29] conclusions that we've known for a while [82:30] that somehow don't get applied. Like [82:31] there's all this crazy stuff about like [82:34] uh ways of teaching reading where like [82:37] we had pretty good research telling us [82:38] what to do and then a lot of like not [82:40] doing it for all sorts of messy reasons [82:42] and yet I think like the learning goals [82:43] is a nice example of there being real [82:45] insight that comes out of [82:46] >> the kind of whole like intellectual stew [82:49] of the educational world. Are there like [82:50] other pieces that you think of as like [82:52] really important and valuable that that [82:55] you try and leverage here? Yeah, one [82:57] that I've tried very deliberately to [83:00] apply to the Okamel boot camp is uh what [83:04] is sometimes called cognitive load or [83:06] you might call it working memory. But [83:08] just when you are trying to learn [83:10] something new, there's actually a limit [83:12] to like how many new things you can sort [83:14] of handle at the same time before it [83:17] becomes harder to learn any of them. And [83:20] so in an older version of the Okamel [83:23] boot camp, right away you were learning [83:26] Okamel. You were learning an editor that [83:29] perhaps you hadn't used before. And you [83:31] were learning a version control and code [83:32] review system that you also hadn't used [83:35] before. And you were doing all of these [83:37] from the jump. And one thing that I I [83:40] did was sort of split these apart. So [83:43] you start off just learning Okamel [83:46] syntax in a sort of uh in in Utop the [83:49] sort of command line rippler and now we [83:51] have nice kind of in browser kind of [83:53] text editor boxes where you can run [83:56] little snippets of Okamel and then you [83:58] go to a piece where you get introduced [84:00] to your editor and you do some [84:02] activities just with the version control [84:04] system and only after you've done each [84:06] of these sort of introductory pieces do [84:08] you get to the part where you're [84:09] bringing them all together. So kind of [84:10] applying this this cognitive load idea [84:13] in that way. And I think the other thing [84:15] that that comes to mind uh which I [84:17] mentioned before is this idea of active [84:19] learning of trying to make the [84:22] educational activities um uh not just [84:26] someone telling you things but as [84:29] interactive as possible. [84:32] And I think that uh some of the the new [84:35] material that we have developed for the [84:37] internship uh this summer is really [84:41] applying this in some interesting ways [84:44] of lots of small group discussions or [84:46] kind of interactive activities around [84:49] design or practicing talking to [84:51] stakeholders. Yeah, sort of uh applying [84:54] this idea of people actually interacting [84:56] with other students uh as an important [84:59] part of the educational process. [85:01] >> Right. Again, leading into like this [85:03] isn't just about conveying information. [85:05] There's a big psychological element to [85:07] people learning and retaining. [85:08] >> That's right. [85:10] >> So, maybe it's worth actually talking [85:11] for a second about this point about the [85:12] internship. So, you've been involved in [85:14] >> what has been a pretty serious rework of [85:16] the kind of educational kind of intro [85:19] that people get to the firm. uh can you [85:22] say a little bit more about like what [85:23] motivated that and and what we've what [85:25] we've tried to do in that space? [85:26] >> Yeah, so the sort of precipitating [85:28] motivation was wow AI tools are really [85:31] good now [laughter] and you know maybe [85:34] the the four-week project we used to [85:37] have interns start off by doing like [85:39] that's one prompt and you're done. And [85:42] so that's not a useful way to like [85:46] evaluate uh an intern or like productive [85:51] work to ask an intern intern to do. And [85:55] so okay, we're going to change the [85:57] nature of the projects that interns are [86:00] due are going to do to be more [86:01] open-ended to maybe involve more [86:03] elements of design or gathering [86:05] requirements, things like this. Uh well, [86:08] >> basically making them like harder and [86:09] more realistic. [86:11] >> Yes. and more sort of going beyond like [86:15] can you produce a reasonable amount of [86:18] good code is just sort of just producing [86:20] the code like that used to be maybe [86:23] among the most important signals and [86:25] it's less important now I would say um [86:29] and so aligning the projects with like [86:31] these are the things we sort of want to [86:33] sort of see people do successfully and [86:37] well to set them up for success we [86:39] should try and teach them some more uh [86:42] about how to approach this, how we think [86:45] about this at at Jane Street and in [86:48] addition uh not just sort of dump them [86:50] into the deep end of like you have [86:52] access to LLMs, good luck, but actually [86:54] like try and train them some on uh what [86:58] to keep an eye out for, what are [87:00] effective ways of working with these [87:01] tools, what tools even exist at Jane [87:03] Street and how do people use them. [87:05] So the sort of new materials have been a [87:09] combination of uh giving people and this [87:14] is the sort of balancing act that I was [87:15] talking earlier tasks that are like [87:17] sufficiently [87:19] um complicated or large so the LM [87:21] doesn't just give you a perfectly fine [87:24] version on the first try uh but not too [87:27] large that you know it's going to take [87:29] you a week to work through all the code [87:31] that produ is produced [87:33] >> and also O taking this as an opportunity [87:35] to focus on some new um software [87:38] engineering topics that maybe didn't get [87:40] a lot of formal training at the start of [87:42] the internship like testing kind of [87:45] writing a very good test suite for this [87:47] databasedbacked application is kind of [87:50] part of the new curriculum and that's [87:53] >> and testing has always been important [87:54] but like in many ways more important now [87:56] because the models thrive on feedback [87:59] right you really need to give them ways [88:01] of figuring out whether or not the thing [88:03] they are doing is right they're not [88:04] going to like when you get beyond [88:05] relatively simple things, they're just [88:06] not going to oneshot good answers. [88:08] >> For sure. Yeah, it's it's been [88:10] important, but there there's always a [88:12] balance of like, well, we could spend [88:14] half of the internship teaching people's [88:16] things, but we actually want them to [88:17] spend most of their time working on a [88:18] project. So, sort of in in this new [88:21] world, we're like, okay, we're going to [88:23] definitely invest more in education, but [88:25] we still at some point need to get them [88:27] starting to work on their projects. Um, [88:30] so there's also been new exercises [88:32] around looking at uh design docs and [88:35] judging what is a good design doc, what [88:37] is uh looking at some bad ones and [88:40] discussing the the flaws um and kind of [88:44] more on this sort of planning side of [88:46] things. There's been uh we developed a [88:48] new activity where there are various uh [88:51] full-timers sort of role- playinging [88:53] different stakeholders for this app that [88:55] the interns are working on and they talk [88:56] to them and sort of try and gather what [88:58] are the requirements for for what I [89:00] would design. And then the the final [89:02] piece that I would mention is that we've [89:05] talked about code review becoming a sort [89:07] of more central part of the job. And so [89:10] we traditionally have not had interns [89:13] review other people's code. uh at least [89:16] that would be pretty uncommon, but this [89:18] summer we this is something that we're [89:20] going to ask interns to do and that we [89:23] want to actually understand how well [89:26] they perform at it. But to that end, we [89:29] actually have three additional days of [89:32] training in the middle of the internship [89:34] on code review where uh the interns will [89:37] see kind of pre-made features actually [89:41] um adding functionality to the same [89:45] application that they were doing the the [89:46] testing and design work [89:48] >> on earlier in the system. and uh they [89:51] are leaving uh code review comments and [89:53] then they are seeing a reference of like [89:55] here is the set of kind of the answer [89:58] key here's a set of code review comments [90:00] that someone might have left and [90:02] discussing as a group sort of what they [90:04] saw there ones that they disagree with [90:06] uh and then also implementing their own [90:10] features and in turns reviewing each [90:12] other's code to give them some reps [90:15] doing this with the idea that then their [90:18] mentors and their actual project will [90:21] send them some some features for review. [90:25] >> So, this is like a really big change [90:27] both to like the educational part of the [90:29] internship and really to how the [90:30] internship as a whole runs. [90:31] >> How do you think it's gone? [90:33] >> I think we're learning a lot. [laughter] [90:37] Uh I think a lot of the the challenges [90:40] that have come up are a lot related to [90:43] the the scope of what we're asking [90:45] people to do. So if we want people to [90:48] generate with code with LMS and really [90:50] hammer the message in you own this code, [90:52] you need to review it carefully. You [90:54] need to get it into a good state, you [90:56] don't just uh s stamp whatever the LLM [90:59] produced. Well, if we ask people to [91:03] produce much more code than they have [91:06] time to polish, some sense that doesn't [91:10] quite line up and we're not kind of [91:12] communicating the the lesson uh that we [91:15] want. [91:17] Um, so I think we've sort of learned a [91:19] lot about like what is the right scope [91:21] for these different pieces and similar [91:22] with the code review like sending people [91:25] sort of too large a feature with too [91:27] many problems and it's hard to sort of [91:31] for people to to to kind of spot the [91:34] issues that we actually want them paying [91:36] attention to. [91:37] >> It maybe goes to like the cognitive load [91:39] issues you were talking about before. [91:40] >> Yes. So, I think there's been some [91:42] challenges there, but I also think it [91:44] has been [91:46] uh really great to see this sort of more [91:51] interactive [91:53] um [91:55] form of of education compared to say the [91:58] Omel [92:00] the interns went through because in that [92:02] they're going at their own pace. They're [92:04] getting code reviewed by a full-timer [92:06] and sort of discussing with them, but [92:08] they're not necessarily doing a lot of [92:10] uh talking to other interns about the [92:12] material, but this new curriculum [92:15] interns are in these pods of four people [92:17] and it's explicitly structured with [92:20] these small group discussions frequently [92:22] throughout it both like how are you [92:23] using uh the LLM tools like what [92:27] problems are you seeing with the sort of [92:28] tests that it's throwing that it's [92:31] spitting out to you? Uh, and I think [92:33] that those discussions, including kind [92:35] of full-time mentors, have been really [92:37] productive, and I'm excited about kind [92:41] of anchoring a lot of our uh, at least [92:44] intern education in that sort of model. [92:47] >> So, it sounds like next time you run [92:49] this, that we run this, you'll want to [92:51] like reduce the scope of some of the [92:54] tasks people get. [92:55] >> Are there any other are there any other [92:57] changes you think you should make? So I [92:59] think that reducing the the scope but [93:03] also uh there are adjustments to like [93:06] emphasis and pacing. So one uh thing [93:12] that is that is always a challenge with [93:14] programs like this is that people go [93:16] through them at very different paces and [93:20] how do you accommodate someone who's [93:22] like racing way ahead? you want useful [93:25] things for them to do and of people are [93:27] going through it more slowly. What do [93:29] you actually have them focus on? So some [93:31] of this is about being careful with the [93:34] the schedule and also kind of how we are [93:36] messaging about what the expectations [93:38] are uh such that people are actually [93:41] taking the time to learn from the things [93:44] that we think are most important and not [93:46] trying to rush ahead to sort of keep to [93:48] some schedule. And I think there was uh [93:52] at least there are different groups of [93:54] interns and this has been this whole [93:56] curriculum has been evolving rapidly and [93:59] significantly between each batch of [94:01] interns as they start throughout the the [94:02] summer. But in an early one, we had an [94:06] issue of starting out with emphasis a [94:09] lot on a lot of multitasking and sort of [94:12] vibe coding stuff. I think we felt [94:15] afterward that that was not sort of the [94:17] right place to start. to sort of set a [94:19] bad kind of tone for the rest of the [94:22] exercise and there's been some [94:23] rearranging to it. We're actually going [94:24] to start with um like the testing focus [94:28] piece and making this testing suite like [94:30] really high quality and sort of framing [94:32] it around that rather than oh you can [94:34] have an LM do three different things at [94:37] once and like it's cool. [94:39] >> It is cool. [laughter] [94:40] >> Yes, it's Yeah. And I would want like I [94:43] was talking earlier about thinking about [94:46] different ways of of working with these [94:50] models and having people sort of kind of [94:54] checking in with what they're doing more [94:56] often. And I think sort of by next [95:00] summer I think we might have like a lot [95:04] more considered ideas about this or [95:06] people have tried different things. I [95:09] think it's very possible that we'll want [95:10] to like build more sort of guidance into [95:15] like here are specific workflows to [95:18] follow whereas I think now it was yeah [95:20] we have some things to know about ways [95:24] in which these systems can misbehave and [95:26] things to to keep an eye out for but not [95:29] a lot about like we think this is the [95:32] like good way to prompter or interact [95:36] with with these systems and this is [95:38] another reason why these small group [95:39] discussions were great because people [95:40] were sharing the sort of different [95:42] techniques that they were playing around [95:43] with and and what was working well and [95:45] what sort of wasn't serving them [95:47] >> right and I guess one of the challenges [95:49] of this particular aspect of the world [95:52] is it's just changing fast and so [95:54] knowing what advice to give it's like we [95:55] can give some advice now next year might [95:57] need to be different advice. [95:59] >> Not only is it changing fast but two [96:02] people using it in the same way might [96:04] have it do different things. So when you [96:07] give people an exercise that involves [96:09] using an LM, it just introduces [96:13] much more uncertainty into what is going [96:16] to happen when they do it than you would [96:19] have had in the past. [96:21] >> Right. [96:22] All right. Well, maybe that's a good [96:24] place to stop. Thank you so much for [96:25] joining me. This is great. Thanks so [96:27] much. [96:28] >> You'll find a complete transcript of the [96:30] episode along with show notes and links [96:32] at signalsandthreads.com.