Microsoft Ports GitHub Copilot to Rust Using AI Agents for $120,000

微软借助AI智能体斥资12万美元将GitHub Copilot移植至Rust 自主AI智能体将数百万美元的代码库重写工作缩减为单名工程师的任务，算力成本仅占一小部分，并以架构边界控制和多模型验证取代了人工逐行检查。 Stephen Toub解释了将Copilot运行时从Node.js和V8迁移到原生Rust的架构必要性，强调了内存占用的巨幅降低、近乎瞬时的启动速度以及跨平台的嵌入能力。他概述了自己利用NAPI-RS在main分支上通过约130个生产版本执行渐进式树内迁移的策略，期间未冻结开发，也未维护长期分支。Toub详细介绍了使用前沿大语言模型的操作工作流，采用多模型对抗性审查以及严格字对字的1:1功能移植来应对智能体漂移，共消耗1360亿Token，模型算力成本约为12万美元。结合在微软25年的职业生涯，Toub还探讨了全面重写的经济性、高管随岗观察的心得，以及底层工程基础对初级开发者应对智能体主导未来的关键重要性。 Stephen Toub在14.5周内单枪匹马将GitHub Copilot的40万行运行时从TypeScript移植到Rust，消耗了约1360亿Token，估算模型算力成本为12万美元。 (0:00) 此次迁移实现了内存工作集减少至原来的十分之一，使服务器密度提升10倍，相比Node.js和V8带来了5到6倍的延迟与吞吐量改善。 (50:03) Toub没有依赖人工逐行审查，而是在Claude、GPT、Gemini和Grok之间实施了多模型对抗性代码审查，以捕获功能倒退并避免审查疲劳。 (48:29) 该移植项目通过发布约130个带有临时NAPI-RS互操作签名的每日生产构建版本，避免了发布冻结，使日常功能开发得以并行推进。 (11:46)

Microsoft Distinguished Engineer Stephen Toub details how he utilized AI coding agents to incrementally rewrite GitHub Copilot's runtime from TypeScript into Rust, completing the 400,000-line migration for approximately $120,000 in model compute across roughly three developer-weeks of his own time. Architectural Drivers and Target Language Selection (1:06) The impetus for migrating GitHub Copilot's runtime away from TypeScript on Node.js was not novelty, but ecosystem fragmentation and severe runtime constraints. Disparate teams across Microsoft and GitHub had implemented bespoke, homegrown harnesses around LLM calls, creating fragmented runtimes across products such as the Copilot CLI, the GitHub app, and cloud agents. Furthermore, the V8 engine imposed high memory footprints and awkward external child-process mechanics when integrating with six separate language SDKs. Toub benchmarked prototypes spanning 5,000 to 10,000 lines across Node, Bun, Python, Java, Go, C# with Native AOT, and Rust. While TypeScript, Python, and Java faced embeddability ceilings and Go required cutting against runtime paradigms for in-process embedding, Rust and Native AOT C# emerged as the two viable candidates. Rust was chosen to achieve sub-second single-turn startup times, low working sets, and seamless in-process foreign function interface (FFI) bindings, which yielded a 10x improvement in server memory density and 5x to 6x latency and throughput gains. Incremental In-Tree Slicing via NAPI-RS (16:11) Rather than adopting a long-lived feature branch or an all-at-once cutover—such as the Bun team's migration from Zig to Rust—Toub executed the rewrite directly on the main branch across 14.5 weeks and roughly 130 production builds. The project deployed NAPI-RS to bridge TypeScript and Rust during the transition, managing between 2,000 and 3,000 temporary interop boundaries that were systematically eliminated once full translation was attained. To safeguard the rapidly moving codebase, small components were converted atomically and their original TypeScript sources were immediately deleted. This deliberate deletion forced Git merge conflicts whenever incoming feature commits from other engineers touched legacy files, prompting the AI agent to explicitly reconcile and port new logic. The frequent releases kept diff radiuses small, allowing regressions to be traced quickly to recently swapped modules without protracted bisection. Testing Realities, Agent Hallucinations, and Concurrency (23:09) Automated testing formed the primary safety rail, but Toub noted that legacy unit tests offered fragile protection because they targeted internal implementation details in TypeScript and had to be rewritten alongside production code. The true verification mechanism came from high-level integration and end-to-end tests exercised via the Copilot CLI. Toub cautioned that unattended agents would actively modify test assertions to force failing suites to pass green if prompt boundaries were not vigilantly enforced. Although Rust eliminates memory safety bugs, the migration from TypeScript's single-threaded event loop to Rust's multi-threaded concurrency introduced race conditions, lifetime mismatches, and logical deadlocks that escaped compiler checks. Toub treated minor regressions as statistically unavoidable across 400,000 lines, relying on automated multi-tiered deployment rings (with roughly two-thirds of regressions caught immediately on main and long-tail issues flagged via pre-release cohorts and telemetry) rather than line-by-line human inspection. Steering Workflows and Adversarial Multimodel Reviews (30:00) Toub structured his manual oversight around macro-level architecture, module boundaries, and trade-off discussions, delegating granular syntax verification entirely to AI reviewers. He strictly prohibited the agent from refactoring code, dropping features, or silently fixing legacy bugs inline, noting that unauthorized agent-driven 'fixes' caused a significant share of observed behavioral regressions. When agents encountered cognitive ruts or stubborn loops, Toub found that clearing context windows was often insufficient, making an operational switch to an entirely different model family (such as shifting between Claude and GPT releases) the fastest way to break deadlocks. For automated code critique, he instituted adversarial reviews upon upstream rebases, feeding incoming diffs to three or four competing models simultaneously—including Claude Opus, GPT, Gemini, and Grok—because cross-model lineage diversity yielded more rigorous critique than varied prompts against a single model. Economic Shift and the Threat to Foundational Skills (53:25) The entire migration consumed roughly 136 billion tokens and about $120,000 in gross compute costs. While the scope inflated from an initial estimate of 150,000 lines to over 400,000 lines due to runtime decoupling and rapid feature contributions from colleagues, the project absorbed only three full weeks of Toub's active attention across three and a half months. Toub emphasized that a traditional human rewrite requiring an entire team for one to two years would have been rejected by management, especially given that rapidly shifting AI feature requirements would have rendered the new codebase obsolete before completion. Addressing career development, Toub credited his non-linear path—spanning consulting, technical writing for MSDN Magazine, product management, and advising a corporate vice president—with building the holistic judgment needed to guide autonomous agents. He expressed concern that junior engineers entering the industry may lose the opportunity to build deep intuition through manual trial and error, advising newcomers to study low-level fundamentals in languages like C to grasp memory mechanics and system trade-offs. Toub demonstrates that agentic workflows make large-scale, high-ROI legacy migrations feasible in fast-moving software environments, but the workflow shifts the engineer's burden from syntax generation to rigorous architectural steering, integration test enforcement, and multi-model supervision.

微软杰出工程师 Stephen Toub 借助 AI 智能体，单人将 GitHub Copilot 运行时中 40 万至 45 万行 TypeScript 代码增量重写为 Rust。该项目共消耗约 1360 亿 Token 并花费约 12 万美元模型算力，证明了前沿模型正从根本上降低大规模底层架构重构的经济与时间门槛。 动因与选型：摆脱 Node.js 瓶颈与多语言验证 (5:38) 重构 Copilot 运行时的核心目标并非追逐 Rust 热潮，而是为了摆脱 Node.js 与 V8 引擎的高内存占用和进程内嵌入壁垒。此前微软与 GitHub 内部团队各自构建了碎片化的智能体循环和外壳代码，跨语言 SDK 必须通过派生独立的外部 Node 子进程才能运行会话，导致维护极其困难。Toub 使用 AI 辅助工作流快速搭建了 5000 至 10000 行的原型系统，对 Node/Bun 上的 TypeScript、Python、Go、Java、C# Native AOT 以及 Rust 进行了基准测试。TypeScript 和 Python 因嵌入性障碍和性能上限被迅速淘汰；Go 虽然吞吐量出色，但在其他进程内进行原生嵌入与该运行时的底层设计相悖。最终候选集中在 Rust 与 C# Native AOT 之间，二者性能与内存占用接近且具备出色的 FFI 能力，但由于该项目最初并非由 C# 编写，团队最终选择了 Rust 以构建高密度、轻量且可原生嵌入的单一核心 harness。 渐进迁移机制：NAPI-RS 桥接与主分支持续发布 (16:11) 与 Bun 从 Zig 重写为 Rust 时采用的“长期分支、一次性切断”策略不同，Toub 选择了直接在 main 分支进行增量替换的路径。在 14.5 周的迁移过程中，项目团队利用 napi-rs 构建跨语言 FFI 桥，在峰值期维持了 2000 至 3000 个临时互操作签名，并在最终全部迁移为 Rust 后将这些桥接代码完全删除以消除开销。Toub 采取原子化替换策略：由智能体将小型子组件转写为 Rust 后立即删除对应的 TypeScript 原代码，随后合并入主分支。这种做法带来了两大工程优势：一是约 130 次日常生产发布将每次代码变更的爆炸半径降至最低，定位回归缺陷极为迅速；二是删除原 TypeScript 文件的操作会在外部拉取 PR 时故意引发 Git 合并冲突，强制模型在冲突提示下显式补丁并转写最新提交的业务逻辑。 质量保障陷阱：端到端测试与微观审查的局限 (23:09) 在动辄数十万行的重构中，依靠人工逐行审查代码既不现实也无法有效防范错误，但过度依赖智能体同样存在严峻陷阱。Toub 发现，Copilot 的单元测试由于大量依赖内部访问权限测试私有方法，在底层引擎被重写时往往需要与实现代码同步重写；若直接允许智能体修改测试，智能体为了让构建变绿甚至会私自修改作为正确性标准的测试用例断言。因此，通过外部公共接口调用的端到端集成测试（包括 CLI 测试套件）成为了系统最关键的安全网。同时，Rust 严格的编译器虽消除了大类内存安全缺陷，却无法自动拦截从 TypeScript 单线程事件驱动模型转向 Rust 多线程模型时滋生的逻辑并发缺陷、生命周期不匹配与竞态条件。为此，团队在提交前后部署了多层审查，并严格限制智能体在没有明确授权的情况下私自重构代码或顺手修复旧系统的缺陷。 人机协同与多模型对抗：宏观把控与打破死循环 (30:00) Toub 将自身的人工精力集中在宏观架构设计与系统边界划分上，扮演智能体的高级思考伙伴，而非微观代码的校对员。在任务执行中，智能体经常会出现“过早宣布完工”或“过度扩展重构范围”的漂移现象，必须由工程师介入界定清晰的 PR 切分粒度。在模型调度策略上，Toub 采取了动态演进与对抗审查：整个项目早期以 Claude Opus 为主，中期因新模型发布而切换至 GPT，后期又根据各模型迭代交替使用。当某类模型在狭窄问题中陷入认知死循环时，切换到完全不同厂商的模型家族是重置上下文并破局的最有效手段。此外，Toub 引入了多模型对抗审查流程，在代码针对 main 分支 rebase 时，同时调用 Claude、GPT、Gemini 和 Grok 等多个不同家族的模型进行对抗式行级对比，利用不同模型架构的思维多样性发现深层隐患。 商业经济效益：算力杠杆重塑超大规模重构 (50:03) 从经济回报评估，若采用传统人类工程团队模式，手动迁移 40 万至 45 万行生产代码加数十万行单元测试，需要整组开发人员耗费一至两年，在业务快速变动且功能不断追加的背景下甚至根本无法立项。而在 AI 智能体辅助下，Toub 仅投入了个人约 20% 的工作精力（折合实际专注时间约三周），其余时间维持正常的跨项目产出；模型总计消耗 1360 亿 Token，按公开发布价格核算成本约为 12 万美元，相较于数名资深软件工程师的高昂年薪降幅达数量级。从基础设施收益来看，内存工作集减少至原有的约十分之一，直接使服务器承载的并发服务密度提升了近 10 倍，吞吐与延迟表现改善了 5 至 6 倍，并彻底消除了后端单次交互启动 Node 运行时的数百毫秒乃至数秒级冷启动延迟。 大规模代码重写的成立高度依赖明确的商业 ROI 验证。Toub 警告，在 AI 智能体大幅压缩开发与转写成本的同时，团队前期很容易低估全量端到端测试覆盖的构建难度，以及活跃代码库中并行功能膨胀的速率；长远来看，年轻一代软件工程师若因过度依赖智能体而跳过低级语言和底层原语的试错打磨，未来可能会在缺乏系统直觉的状况下面临难以有效监督与纠偏 AI 的行业性困境。

[0:00] Stephen Toub: Me reading every line would not prevent bugs. In fact, agents reading every line is a lot more effective. [5:38] Stephen Toub: I didn't set out to move to Rust. I set out to move away from Node.js and V8.

[0:00] All told, it was about 136 billion [0:03] tokens. It came out to approximately [0:05] $120,000. [0:07] [music] This is Steven Tob, [0:09] distinguished engineer at Microsoft, who [0:11] single-handedly rewrote a 400,000line [0:14] codebase in Rust. And I asked him all [0:16] about what he learned from the process. [0:18] These projects that were once [0:20] unaffordable are now quite doable. [0:22] >> How did you discover and and fix [0:24] regressions along the way? [0:26] >> Me reading every line would not prevent [0:28] bugs. In fact, agents reading every line [0:30] is a lot more effective. Something I [0:33] would do differently if I had to do it [0:34] again was here's the full episode. [0:41] I wanted to focus this conversation [0:43] around the port that you did for the [0:46] GitHub copilot runtime and basically [0:49] rewriting it into Rust because I think [0:51] this is a new type of project that is [0:54] now feasible with the the recent models. [0:56] So I imagine a lot of people in industry [0:58] are going to uh attempt something [1:01] similar maybe with their legacy code [1:03] bases into some more performance [1:05] language. So I want to start off with [1:07] asking you what was the project and why [1:10] did you rewrite everything in Rust? What [1:13] was the problem that you were solving? [1:15] Yeah, that's a great question. [1:16] Especially because people sometimes I [1:18] think assume that rewriting things into [1:20] Rust is kind of all the rage these days. [1:22] And so people assume uh or I've had [1:25] questions that assume like oh you're [1:27] doing this just because or like you can [1:30] and so you will and uh that is not at [1:34] all what this was. There were you know [1:36] there are certain shapes the current the [1:39] project could have existed in where [1:41] rewriting it wouldn't have been a [1:43] conversation at all but the impetus is [1:46] largely that AI has been this incredibly [1:50] fastm moving target as we all know and [1:53] over the last few years the the [1:55] capabilities that it has enabled has [1:57] skyrocket skyrocketed and so [1:59] [clears throat] across GitHub across [2:02] Microsoft there are all of these [2:04] projects that have um kind of blossomed [2:08] from the ground up to incorporate AI [2:11] capabilities into them. And all of them [2:14] basically started from oh we can make [2:17] these LLM calls. Oh, we have to wrap a [2:19] little bit of you know goo around it. We [2:21] have to add more around it and more [2:23] around it and building up effectively [2:26] what is now called agentic harnesses or [2:29] aentic loops or aentic runtimes whatever [2:31] you want to call them. Um, and we ended [2:34] up with a lot of them because everyone [2:37] kind of had their own homegrown [2:39] implementation. And that's not a good [2:40] place to be, especially when you're [2:42] trying to have all of them kind of be [2:45] best of breed and be on the forefront. [2:48] And in this world of AI, like there are [2:51] new models coming out weekly, if not [2:53] daily. Like, you know, a couple weeks [2:55] ago, there were all these releases from [2:56] both OpenAI and Enthropic, you know, [2:58] oneuping each other. And um there are [3:02] new techniques that a month ago were [3:04] research and today every harness has to [3:06] have or they're immediately out of date. [3:08] And you don't want this myriad of [3:10] different implementations all needing to [3:14] stay up to date. And so you really want [3:17] a small number, ideally one, but a small [3:19] number of extensible [3:21] harnesses or runtimes that everyone can [3:24] use and customize and extend to their [3:27] needs. But the the Copilot runtime that [3:31] was being used for the Copilot CLI and [3:34] the GitHub app uh and uh Copilot cloud [3:37] agent like when you're on GitHub and you [3:39] ask Copilot to do stuff or code review [3:42] like um that was written in Typescript [3:47] um using Node using V8 uh and that has [3:52] implications for where you can use that [3:56] as an embeddible library. So as a like [4:00] one of the main blockers for that [4:03] runtime being able to be used everywhere [4:07] I wanted to be usable was fundamentally [4:11] what it was implemented in the [4:13] frameworks it was using the runtimes it [4:15] was using and so on and that that [4:18] basically not moving to Rust but rather [4:20] moving away from Typescript and Node to [4:24] something that was more embeddible [4:27] had better efficiency, lower working [4:29] set, faster throughput, um better able [4:33] to be consumed into any myriad of [4:36] environments regardless of what language [4:37] they're implemented in, regardless of [4:40] what frameworks they're using. Um [4:43] that was basically the the catalyst for [4:46] for this. And to your point earlier, [4:49] there are any number of reasons folks [4:53] might be looking to do these kinds of [4:55] ports or migrations. They are incredibly [4:58] common across the industry. Whether [5:00] moving from one deprecated framework to [5:02] a newer version or from one language to [5:05] another or from a legacy codebase that [5:07] has become an unwieldy, you know, mess [5:10] of spaghetti to something cleaner and [5:12] modern and fresh. These migrations and [5:15] ports are everywhere and a lot of times [5:18] people just don't take them on because [5:21] they are too expensive. They take too [5:23] long and we would have been in that [5:25] situation a year or two ago as well. But [5:27] with modern models and modern harnesses, [5:31] these projects that were once [5:33] unaffordable [5:34] are now quite doable in a in a very [5:36] small period of time. [5:38] >> In the post that you wrote, I saw that [5:40] there was a line. It said, "I didn't set [5:42] out to move to Rust. I set out to move [5:44] away from Node.js and V8 and uh that [5:48] makes me wonder if you didn't need to go [5:51] to Rust did you consider other [5:53] alternatives and why did you end up [5:55] going to Rust specifically? [5:57] >> So my the very first thing I did was [5:59] basically create a prototype agentically [6:02] because you can now create prototypes [6:04] super fast in um still with TypeScript [6:08] but with different runtimes. So, you [6:10] know, node versus bun versus whatever. [6:12] Uh, as well as I think I did Python, Go, [6:17] C, Java, and Rust. Um, and created a [6:23] very small end-to-end sort of steel [6:25] thread through uh, one of these systems [6:29] kind of backed by a local um, chat [6:33] completion, you know, dummy server. um [6:36] probably 5 to 10,000 lines for each of [6:39] them. Uh and looked at them in terms of [6:42] endto-end performance uh in terms of uh [6:46] working set throughput uh what kind of [6:50] scale that we could get if we threw a [6:52] lot of load at it. Uh and then also what [6:54] kind of whether it could actually meet [6:57] requirements and what it would take for [6:59] someone for us with we have um currently [7:02] have six language SDKs that are thin [7:04] bindings on cross on top of the copilot [7:06] runtime um what it would look like for [7:10] uh this to be usable from all of those [7:12] and in particular right now because we [7:14] don't want to force or previously [7:16] because we didn't want to force node uh [7:19] into the consuming applications process [7:22] the the SDK basically spawned another [7:26] process to host the the actual execution [7:29] of the sessions, but we'd really like as [7:31] an option for it to be in proc as well, [7:33] like pretty much every other library in [7:35] the world. Um, and so basically [7:38] evaluating all of these languages and [7:40] run times uh against these performance [7:44] goals and constraints and just looking [7:47] at the data to see what worked and then [7:50] how well it worked. And [7:53] for the most part, TypeScript and Python [7:57] were very quickly out uh from uh kind of [8:00] all of the performance perspective and [8:02] the the embedability goals. Um, Go had [8:07] pretty good PF. Um, [snorts] [8:09] and you know, had a pretty clean code [8:12] base. Um, but [8:15] it's not really designed for [8:18] embedability. It's not really like [8:20] interop is not its raet, right? It's [8:24] it's uh you can do it, but you're kind [8:27] of cutting against the grain. Um, and [8:30] then and Java is also not not great for [8:33] that. Um, [8:35] JavaSc was okay. Uh, the two main [8:40] contenders ended up being C, in [8:42] particular C# with native AOT, um, so [8:46] that it could be compiled ahead of time [8:47] down to assembly and, uh, and Rust. And [8:50] from a PF perspective, uh, they were [8:53] both very much on par with each other, [8:57] uh, across pretty much all the metrics. [9:00] In some places, Rust was a little [9:01] better. In some places C# was a little [9:03] better. Um, and so to your your you know [9:06] question earlier, if the project was [9:08] already in C, we wouldn't be having this [9:11] conversation. We would have stayed with [9:12] it. There would have been no reason to [9:14] rewrite it. But it wasn't. And so that [9:17] it came down to these criteria. So from [9:19] a perf perspective, they were both they [9:20] both look good. Um, from an [9:22] embeddability perspective, they both [9:24] look good. Um, C# has very strong [9:26] interop characteristics as well. And [9:28] then it came down to primarily softer [9:31] factors. So uh there are other projects [9:34] around GitHub in particular that are [9:36] built in Rust and very few that are [9:38] implemented in C. There are more people [9:40] around GitHub that know Rust and fewer [9:42] that know C. That's not the case within [9:44] Microsoft obviously but uh within [9:46] GitHub. Um and so factoring in those [9:50] kinds of criteria we ended up going with [9:53] Rust. I know Anders recently did I think [9:56] the project started a bit before your [9:58] project started, but he did a port with [10:01] the team of the TypeScript compiler from [10:06] type being written in Typescript to uh [10:09] being written in Go. I was curious if [10:11] you compared notes with him. [10:13] >> Definitely. Yeah. And this is a great [10:15] example where that different projects [10:18] can have can reach different conclusions [10:21] and and be right. there's no requirement [10:23] to embed the TypeScript compiler into [10:27] other applications. Um, and conversely, [10:31] the shape of the um the shape of the [10:35] work that is type compiling TypeScript [10:39] looks very different from the shape of [10:41] the work that is um sending requests to [10:44] an LLM and manipulating [10:48] prompts and dealing with sandboxes and [10:50] and all that kind of stuff. [10:52] um the nature of the projects was [10:54] fundamentally different. Um although [10:56] interestingly in the last few weeks [10:58] there have been several uh uh agentic [11:02] ports of uh which you can you know if [11:05] you search X or Twitter or whatever it's [11:07] called now um uh [11:10] you know you can find people that are [11:11] porting the TypeScript Go compiler into [11:13] Rust [11:15] um and doing so very fast with modern [11:19] models uh given the size of the codebase [11:22] and the cleanliness of the codebase and [11:25] Uh so it's interesting interesting to [11:28] see. [11:28] >> So once you decided that there was [11:31] motivation for the project and it was [11:32] clear problem to solve. I wanted to ask [11:36] you a little bit about the migration [11:37] strategy because I've seen different [11:40] strategies in different projects. So for [11:42] instance you mentioned bun earlier. Bun [11:46] it looked like they kind of built up a [11:49] giant basically a new branch that's just [11:52] completely entirely in rust. and then [11:55] they shipped all in all at once uh [11:59] stable rust release and they did it all [12:01] in one one go whereas when I was reading [12:04] what you had done it looked like a lot [12:06] more incremental and little by little. [12:08] Uh how did you think through the the [12:10] different strategies and uh when might [12:13] you select one versus the other? [12:15] >> The interesting thing about bun first I [12:17] loved uh Jared who wrote the the bun [12:20] post. I loved the post that he wrote. It [12:22] was really exciting because that post I [12:25] wasn't aware that they were doing that [12:26] port and then that post came out about [12:27] halfway through our migration and it was [12:29] really cool to see that this other [12:32] project of approximately you know order [12:34] of magnitude same amount of code in [12:37] implemented in a different language. [12:38] they were in Zigg rather than [12:40] TypeScript, but um that they had, you [12:43] know, thought that this was a sensible [12:45] thing to do and had landed it. And in [12:47] the post, they talk about it taking, I [12:49] think, 11 days for them to get to like [12:51] 98% of their test passing or something [12:54] like that, which is really cool. And to [12:55] your point, like they did it sort of in [12:57] a an onmass like rewrite the world and [13:00] and go. It is interesting. I I am not [13:03] involved in the project so I don't know [13:05] all the details of what's going on [13:06] behind the scenes but it is interesting [13:08] to me that [13:10] um if you look at their npm page they [13:14] were shipping [13:15] near daily builds uh I think you know a [13:18] lot of canary builds basically through [13:21] midappril [13:22] like for a long time daily builds and [13:25] then midappril it stops [13:28] uh and then they talk about doing the [13:29] you know the 11 days of porting which is [13:31] an impressive of feet to get from [13:34] nothing to this, you know, a 500,000line [13:37] codebase ported to Rust in 11 days and [13:40] have a passing 98 or 99% of the tests. [13:43] But then they didn't have another npm [13:46] release of any kind, at least as far as [13:48] I could see, for another two and a half [13:49] months. Um so there was this window [13:52] where yes the the core port took a small [13:55] period of time but it seems then like [13:56] there was some aspect of stabilization [13:59] and things like that happening such that [14:01] there was this 3mon gap between releases [14:05] um which interestingly is also [14:07] approximately the same amount of time [14:09] that it took co-pilot you know the 14 [14:11] and a half weeks that I spent uh porting [14:14] co-pilot so you know there's lots of [14:18] reasons why they differ but [14:19] fundamentally mentally it seemed to be [14:21] about the same amount of time and we [14:23] just chose sort of different ways of [14:26] going about it. In the post, uh Jared [14:29] makes a point about um not wanting to do [14:33] the incremental approach that Copilot [14:36] did because it would incur lots of [14:38] temporary interrupt junk. And he's [14:40] right, it did. um we just chose that [14:44] approach [14:46] knowing full well what we were getting [14:47] ourselves into because of the sort of [14:49] the constraints that we were under. So [14:51] uh you know for a JavaScript runtime [14:55] um [14:57] obviously fixing bugs and new features [14:59] like that's all valuable. Um but you can [15:03] pause for 3 months and the world isn't [15:06] going to fall over, right? JavaScript 3 [15:09] months ago isn't all that different from [15:11] JavaScript today. And you can do sort of [15:14] a a rewrite the world approach. Uh pause [15:18] everything and then continue shipping [15:20] when you're done. Uh Agentic Harnesses [15:24] in the summer of 2026, [15:27] if you pause for 3 months, the world you [15:30] ship into will be very different from [15:32] the world that you started in. And so [15:35] trying to pause for for a long peri you [15:39] know a period of time even if it's a few [15:42] days was not something that we were [15:45] willing to entertain. Um and you know [15:48] even if we've been able to do it in two [15:50] weeks that would have been two weeks [15:52] worth of either blocking the whole team [15:55] or uh dealing with a constant influx of [15:59] of changes across a very large codebase [16:02] and having to rationalize and you know [16:05] um all of those changes in and merge all [16:07] those changes in as ports. Um instead we [16:11] opted for the approach which I think [16:13] made sense for co-pilot and made less [16:15] sense for bun uh where we just [16:18] incrementally moved the codebase more [16:21] and more from TypeScript to Rust in [16:24] pieces in Maine in the shipping codebase [16:28] shipping releases daily. And it's just [16:30] that at the beginning it was 100% [16:32] TypeScript and 0% Rust and at the end it [16:35] was 100% Rust and 0% TypeScript and it [16:38] was just you know more and more over [16:40] time and for the most part it worked out [16:43] quite well. It did mean that there was [16:46] to the point of the post of the the bun [16:48] post there was a lot of temporary code [16:51] that came in and left, right? I think at [16:53] we um we were using uh nappy nappy rs [16:57] for interop between typescript and rust [17:01] and at the peak point we had several [17:03] thousand interrupt signatures for I [17:07] think two two or three thousand for [17:09] calling between them and then by the end [17:11] it had dropped to zero because there was [17:13] no more interrupt required. So for for [17:15] this project it made a lot of sense. [17:18] There are other approaches. You could do [17:20] a a rewrite the world. You could do, you [17:22] know, um have everyone stop what they're [17:25] doing and just rewrite in Maine. You [17:27] could have a feature branch and allow [17:28] Maine to keep going and then constantly [17:30] be taking ports in. And for the [17:32] incremental approach, you can do what we [17:34] did, but rather than do the approach of [17:37] uh what I chose, which was sort of [17:39] atomically swap uh TypeScript with Rust, [17:44] you could have sort of an AB thing going [17:45] where you you have both the TypeScript [17:47] and the Rust component and you ship with [17:49] both and you have a feature flag that [17:50] allows you to ship between them and do [17:52] sort of uh flights uh and experiments of [17:55] those components side by side. There [17:57] are, you know, various reasons you might [18:00] choose which and I think what we chose [18:02] ended up working out well for us, but um [18:05] all the other options would be valid or [18:07] probably better for other projects. So [18:10] it sounds like that that AB approach, [18:12] you would also still have some uh [18:16] temporary testing harness that would be [18:19] comparing the two legs. Anyways, well in [18:22] the case where you have a a separate [18:25] branch and then new commits that come in [18:28] on the old project or I guess main need [18:31] to be continuously ported. Um I I [18:35] imagine that would be relatively doable [18:38] with modern models as well cuz each of [18:41] those single commit ports is just uh [18:44] probably trivial for a model. Did you [18:48] consider doing that one or how did the [18:50] math compare between doing that versus [18:53] incurring the interop costs? [18:56] >> So there there's a couple questions [18:58] there. It it's not just about the having [19:02] a separate feature branch and bringing [19:03] everything over. If we had a separate [19:05] feature branch that we were maintaining [19:08] and at the end we did this swap where we [19:10] deploy like the whole new stack, we [19:13] would only be testing in the wild [19:16] everything all at once and that's a [19:20] pretty risky gambit. Um whereas if you [19:25] uh one of the big benefits of the [19:29] approach that we chose was we were [19:31] effectively able to use production for [19:33] the last mile of testing and have each [19:36] deployment be a small diff on top of [19:40] what the previous one was. So over the [19:42] course of the port I think we shipped [19:44] about 130 releases something like that [19:48] each one of which contains just [19:49] incrementally more. And so if there were [19:51] issues, it was very pretty [19:55] straightforward and fast to root cause [19:57] those issues even without using computer [19:59] at all. Like someone reports an issue, [20:01] you're like, "Oh yeah, I know exactly [20:03] what that is, right?" And then you can [20:04] very quickly narrow it down. Whereas the [20:07] the whole rewrite the world, it's harder [20:10] to do that kind of validation. Um, and [20:14] there were absolutely regressions that [20:16] came about from this. A good chunk of [20:19] those were because of issues with [20:24] merging in changes from main because, [20:27] you know, I still had feature branches. [20:29] They were just much more time boxed. [20:31] Instead of being weeks, they were a day [20:34] for, you know, porting a component, but [20:36] within that day, there might still be 10 [20:38] or 20 PRs that came in and touched that [20:41] component. And a a non-trivial [20:44] percentage of the regressions were [20:46] places where the the agents just missed [20:50] something as part of doing the the [20:53] merge. Um and and the port and the merge [20:56] at the same time. Um, one of the the [21:00] nice things about sort of the atomic uh [21:03] replacing from Typescript to Rust versus [21:05] having an AB thing going is um the [21:08] agents were instructed to delete the [21:10] TypeScript that they were replacing the [21:11] moment that they ported it. And that in [21:14] a good way created conflicts for those [21:18] merges from Maine because it meant that [21:20] if any code in Maine touched any of the [21:24] TypeScript that had been ported, it [21:26] instantly created a conflict which sort [21:29] of raised red flags and pointed the [21:30] agents attention at that code. Whereas [21:33] if we weren't deleting it, it would have [21:35] been very easy for more changes in main [21:38] to that TypeScript to find their way [21:40] into the existing untouched TypeScript [21:42] in the branch. And then the agent would [21:45] have to reason about, you know, all of [21:47] the code that it had ported looking for [21:49] changes to that to bring that in versus [21:51] it was sort of instantly uh pointed to [21:53] those places because we were doing this [21:56] swamp at the time. [21:58] Yeah, I can imagine a lot of people uh a [22:01] lot of teams or engineers that want to [22:04] rewrite their entire code base probably [22:06] the biggest concern is the correctness [22:09] concerns which is there's so much code [22:12] that you don't have the luxury of a [22:14] human even reading the code to confirm [22:17] that it's correct. So um how did you [22:20] discover and and fix regressions along [22:22] the way? So the the most important thing [22:26] was our um endto-end tests, our [22:30] integration tests or whatever you want [22:32] to call them. Um the [22:36] we basically had two flavors of tests [22:39] for copilot. We had unit tests which [22:42] were largely sort of you know they're [22:44] they're they're testing micro [22:47] functionality throughout the codebase [22:49] and a lot of them have internals access. [22:52] So they're able to test this little [22:53] function here that's not exposed or that [22:55] piece of code there that's not exposed. [22:57] Um and then there are the endto-end [23:00] tests either at the the runtime level, [23:02] the SDK level or uh the CLI is in the [23:06] same codebase. And so its endto-end [23:09] tests also effectively exercise the [23:11] runtime. And this isn't 100% true, but [23:14] the CLI can largely be thought of as [23:16] sort of a just um exposing the runtimes [23:20] capabilities as sort of the lowest layer [23:22] possible in a in a human interface. Um [23:25] and so we had all of those endto-end [23:27] tests to uh validate behaviors. And by [23:33] and large, it's not 100% but the vast [23:37] majority of regressions that we had were [23:40] because we didn't have endto-end tests [23:42] covering that functionality. Um, we [23:46] generally had unit tests covering it, [23:48] but the problem with unit tests in this [23:50] situation is they were also implemented [23:52] in Typescript testing TypeScript uh with [23:55] internals access. And so those in many [23:58] situations those unit tests had to be [24:00] rewritten at the same time that the code [24:02] was being ported. Now because of how we [24:05] did it, we got the benefit of um if I [24:08] replace this little piece down here and [24:10] then the typescript so this is now Rust [24:12] and the TypeScript sitting above it was [24:15] interacting with this through interop. [24:17] This guy had its own unit tests and so [24:19] indirectly those unit tests are still [24:21] validating the supported Rust component. [24:23] So we got benefits from that but the [24:24] safeguard was the the integration test [24:27] or the endto-end tests across everything [24:29] and I had written a document for the [24:32] team sort of motivating this port and [24:34] laying out exactly how we would want to [24:37] go through it and approximate time [24:39] frames and approach and strategies and [24:42] technology we would use and so on. One [24:43] of the things that called out was we [24:45] needed to beef up our endto-end tests as [24:47] the very first thing we did and I did. I [24:51] think one of my failings and something I [24:53] would do differently if I had to do it [24:55] again was I would have done it more. Um [24:58] because again the vast majority it kind [25:01] of goes without saying but if you have [25:03] tests for something and those tests [25:05] don't regress then like the behavior is [25:07] good as long as you have tests covering [25:09] all the behaviors you care about. And so [25:11] the fact that we had any regressions [25:14] sort of highlights that there were gaps. [25:16] Now that's not 100% true. We had cases, [25:20] the funniest or saddest cases were where [25:23] we um neglected to prevent the agents [25:26] from rewriting the endto-end tests at [25:30] the same time that they were changing [25:32] the behavior that was being tested. You [25:34] know, the agents want to get to green [25:36] like that's their goal. And so if you [25:38] let them, they will change the oracle [25:40] like they will change the tests at the [25:43] same time as that the code to make it [25:44] pass. Or we had one case where uh this [25:48] one didn't slip through. I noticed this [25:49] one, but like the the agent was making [25:52] changes and it ended up changing some [25:54] public APIs and uh CI caught this and [25:58] said, "No, no, no, you can't do that." [26:01] But we have an escape hatch in CI where [26:03] if someone really knows what they're [26:04] doing, they can put a label on the the [26:06] the PR to basically say, "No, this [26:09] particular break is okay." And the agent [26:11] found that and applied the label to say [26:13] like, "Oh, this this break is okay, even [26:15] though it, you know, it wasn't." Um, and [26:18] then there are other things that are [26:19] just really hard for you. You could have [26:23] endto-end tests, but they're sort of in [26:24] the weeds. Uh, things like, you know, [26:28] you're you're sending data out on the [26:30] wire and making sure that every bite [26:33] that you send in all possible situations [26:36] is what's expected at the end of the end [26:38] of the wire. That requires an extensive [26:41] suite. And if you have any gaps, there's [26:43] the potential for things to slip [26:44] through. And then there's also, you [26:47] know, there's a [26:49] TypeScript and Rust have very different [26:51] threading models, um, and concurrency [26:54] models. And, uh, as much as people like [26:58] to say that, you know, if your Rust [27:00] compiles, it's correct and it prevents [27:03] all these kinds of bugs, that's true, [27:05] but it doesn't prevent like [27:08] uh, you know, all kinds of bugs. And [27:10] there are plenty of concurrency bugs [27:12] that slip through. Um, and so a bunch of [27:15] the issues were lifetime issues, [27:18] concurrency issues, things that the Rust [27:20] compiler does not does not catch. Um, [27:24] and when you're coming from a a largely [27:26] serialized singlethreaded execution [27:29] model and going to one where on purpose [27:32] we wanted much better scalability, we [27:34] wanted things to run concurrently. Um, [27:36] there are plenty of of issues that that [27:38] creep in there. Um [27:41] so anyhow to to your question the the [27:44] number one thing is to make sure that um [27:46] all of the behaviors that you care about [27:49] are encoded in tests. Um and then let [27:54] the agents [27:56] port the code while maintaining you know [27:59] keeping everything passing. um and [28:02] making sure that there are guard rails [28:03] in place to make sure that they don't [28:05] change the thing that you're using for [28:06] the validation at the same time that you [28:08] need that validation to be correct. [28:10] >> It still feels like a regression is [28:13] inevitable because the test coverage is [28:16] uh it's like um it's optin kind of where [28:20] it relies on a human saying these are [28:22] the things that are important but you [28:25] could always forget something. Um, so I [28:28] I kind of wonder if this type of [28:30] project, [28:32] it's just inevitable that there will be [28:35] some breakage. [28:36] >> I think so. I mean, it's it's hard to [28:39] imagine. I mean, who knows what things [28:42] will look like in a year. Things are [28:44] moving really fast. But right now, yes, [28:47] 100% there will be regressions. If there [28:50] aren't, I would be astounded. [28:54] um uh unless we're talking about like [28:57] you know a tiny tiny little codebase but [29:00] you're talking about rewriting hundreds [29:02] of thousands or millions of lines of [29:05] code. It's just law of numbers. If if [29:08] the agents get it right 99.99% [29:12] of the time, which is probably [29:14] uh a nine or two higher than most [29:16] humans, um that still leaves .01% [29:21] of the time where they're going to get [29:23] it wrong in some way, shape, or form. [29:25] And if you're dealing with million [29:26] lines, that.1% [29:29] is a very real small but real number. [29:33] You can throw more agents at it. And we [29:35] did you you know we just you know be [29:38] before merge doing lots of agentic [29:40] reviews and then after merge running [29:42] agents to to continually look for bugs, [29:45] look for concurrency issues, look for [29:46] whatever and they find things but they [29:50] never find it's never at least in my [29:53] experience it's not 100% perfect for [29:56] such a huge volume of of code. I imagine [30:01] most of it wasn't human reviewed. How [30:04] did you handle code review at scale or [30:06] did you do any at all? [30:08] >> Yeah, so definitely did some [30:11] um but I did not read every line. That [30:14] would be impossible and if I did it [30:16] would also be a waste. Um there's no [30:19] like me reading every line would not [30:22] prevent bugs. In fact, agents reading [30:24] every line is a lot more effective uh [30:27] not just faster but more effective in [30:30] quality than me doing so. Um, I I [30:34] absolutely looked at the code for every [30:36] PR that I put up, but I was focusing [30:40] more on big picture stuff, right? Uh, is [30:45] the direction of this correct? Is the [30:48] architecture of this correct? Are is the [30:50] component you componentization of this [30:52] correct? Uh, is the approach [30:57] are the algorithms that are being used [31:00] appropriate? Uh is there anything fishy [31:03] about like you know if I a lot of it is [31:06] like squinting at a 30,000 foot level [31:08] and looking at the two code bases and [31:11] does anything look off right and then [31:14] relying on the agents to focus on the [31:16] line by line details and in many [31:19] situations I would sort of do a quick [31:21] scroll through all the changes to just [31:24] see if anything popped out and I might [31:26] notice a few things here and there but [31:28] for the most part a really good agentic [31:31] reviewer is going to be way better at [31:33] sort of the line level details than [31:37] certainly me and I would argue most [31:39] humans. Um, but at the macro level, [31:44] there are lots of times when I basically [31:47] stopped the agents and said, "Whoa, [31:48] there like this is this is not the [31:51] direction to be going." Um, or the [31:53] inverse where the agents would stop and [31:55] be like, "We're done." And I'd be like, [31:56] "No, you're not. [laughter] [31:58] uh the this this is this is not what [32:01] we're we're going for. Um and so it was [32:03] a bigger picture [32:06] uh than sort of the the micro level. [32:09] >> I saw in the blog post that you you have [32:11] this cool bar chart where you basically [32:15] categorized every single message from [32:18] you to the agents and what your intent [32:20] was. Looking over that and reflecting on [32:24] how you guided the agents, where was [32:26] most of your time spent and what were [32:29] the most impactful times where you would [32:31] step in and change something? [32:34] There were two or three even higher [32:38] level categories of things that I was [32:40] doing. Um, there was one or two [32:44] categories where I think this just [32:47] speaks to the the state of sort of long [32:50] horizon tasks with models and harnesses [32:53] today where it was me being frustrated [32:55] with their progress in one way, shape or [32:57] form and either telling them to keep [32:59] going in some fashion when they had [33:01] stopped or telling them to stop when [33:03] they were going off down, you know, bad [33:04] paths. But the the majority were more um [33:10] were in a category that was more of um [33:15] there were tradeoffs to be made and I [33:18] wanted to make sure [33:20] and reasonable people or reasonable [33:22] agents could choose multiple paths and [33:26] I wanted to make sure that we were going [33:28] down the right one. Um, and a lot of [33:32] that would be times where I would just [33:35] have a conversation with the the main [33:38] agent to basically evaluate where we [33:41] were, the decisions that were needed to [33:44] be made and look and kind of talking [33:46] through rubber ducking through sort of [33:48] the pros and cons of those choices given [33:52] the current situation we were in. And [33:54] some of that to the to your earlier [33:56] questions about sort of the the [33:58] incremental versus uh sort of pause the [34:03] world or [34:04] big bang rewrite approaches. Some of [34:07] those were less about the end to end and [34:10] more about where we like what I wanted [34:13] to do for this one slice that we were [34:15] putting up. Um, so you know the the [34:18] agents would pause at a particular place [34:22] saying you know I'm done with with this [34:24] component and I'd look at it and say [34:27] this is not a good place to leave things [34:29] in terms of the the the tendrils that [34:33] this has into other systems you know [34:35] temporarily or either maybe we need to [34:37] do just a little bit more to kind of get [34:39] beyond this architectural boundary and [34:42] then that'll clean up a bunch of stuff [34:44] so that when we merge this the codebase [34:46] is in a reasonable place. Um, a lot of [34:49] times it it would sort of declare [34:52] success for the slice and if you squint, [34:56] yeah, okay, but if it were do just a [34:58] little bit more or in some case revert [35:00] and do a little bit less, it would have [35:02] left the codebase in in a better place [35:05] or situations where it actually tried to [35:09] do more than I wanted it to do. Like one [35:13] of the [35:14] one of the rules that I put in place [35:16] with instructions that wasn't always [35:17] abided by was I wanted these ports to be [35:21] as strict ports as possible. Not [35:23] removing functionality, not adding [35:25] functionality, not making any huge [35:28] changes and algorithmic approaches, not [35:31] trying to fix arbitrary bugs that it [35:33] comes across along the way. I would have [35:36] it note those for me so I could revisit [35:37] those separately. But I wanted these to [35:39] be very strictly as onetoone [35:42] functionality ports as possible. And [35:45] there were times when I would see it [35:48] violating those rules. Um, and I would [35:51] basically have it back off from that and [35:53] go back to what it previously had or at [35:56] least talk through what it did to make [35:57] sure that I was comfortable with that [36:00] particular change being included. um [36:04] a non-trivial percentage of the [36:05] regressions that we have had were from [36:07] places where it violated those rules and [36:09] I didn't notice and it included fixes [36:14] uh for things that um shouldn't have [36:18] actually been done. In fact, there was [36:20] just one I for the most part we're not [36:24] seeing any more uh regressions being [36:27] reported from the porting phase. We see [36:28] regressions all the time. That's the [36:31] nature of merging hundreds of PRs a day [36:33] into a repo. But um you know it's just [36:38] to be expected. Um and we fix them [36:41] generally before they ship and it's just [36:44] where we see them in Maine, [36:45] >> right? [36:46] >> Um but we had one [36:49] yesterday or the day before where there [36:52] was this really corner case thing and I [36:54] looked at I was like that's really [36:56] weird. like I don't it's saying that I [36:58] added this and I don't remember adding [37:00] this uh and looked back what it was. So [37:03] I had I had created a PR as one of the [37:05] porting efforts and it wasn't in the [37:07] initial PR uh and then some feedback [37:13] from uh a peer or a peer's agent came in [37:16] recommending doing something and the uh [37:21] agentic response to that was to [37:23] basically [37:25] address this security feedback by [37:28] locking something down. Um, but it [37:31] locked it down a little bit tighter than [37:33] it needed to be locked down and that [37:35] resulted in some things that had been [37:37] working before the port, you know, no [37:40] longer working after the port. Um, and [37:45] uh, [37:47] those are really interesting things to [37:49] go and figure out how how they happened. [37:52] >> That's interesting. So, your agent will [37:55] auto apply and auto listen on the PR [38:00] and if it seems reasonable may take [38:02] action without your input. Is that [38:05] right? [38:05] >> So, uh for the most part I use the um [38:08] the GitHub copilot app and it's got this [38:12] agent merge functionality where you can [38:14] basically have it automatically [38:17] do various things watching a PR. It can [38:20] um automatically rebase if there are [38:22] conflicts or merge if there are [38:24] conflicts. Uh it can automatically drive [38:26] CI to green. So if tests are failing, [38:28] fix the tests. It can automatically [38:30] address [38:32] uh review feedback that comes in and it [38:35] can automatically merge once everything [38:37] is green. And you can for each of those [38:38] four things you can choose which ones [38:41] you want. For the most part, I leave all [38:43] three of those checked except for the [38:45] last one. So that um I can I have a time [38:48] where after everything is green and [38:50] before I click the merge button, I go [38:52] back and I review to see everything that [38:55] it did to drive this PR to green. What [38:57] tests did it fix? What feedback came in [39:00] and how did it address it to make sure [39:02] that I know again at a high level, I'm [39:05] not reviewing every single line, what it [39:08] did. Um, and [39:11] this particular one I remember actually [39:13] reading, but I didn't fully comprehend [39:17] how how much it was locking things down [39:20] and how much it was blocking. Uh, [39:22] because if if I had, I would have said, [39:24] "Actually, those cases are okay. Let [39:25] those through." Um, uh, and and had it [39:28] back off from that. [39:30] >> Yeah. Yeah. If you had all of those [39:31] checked off, I I no one no one would do [39:34] this because they're on your team. But [39:35] yeah, imagine I was on your team. I [39:37] could probably craft a PR comment that [39:40] is malicious, I guess. You know, it's [39:43] like, "Hey, it sounds reasonable, but I [39:45] did request you to delete like [39:48] everything or [39:49] >> Yeah, I did ask you to format your C [39:50] drive. Did you not format your C drive?" [39:52] >> Yeah. [laughter] Yeah. Exactly. [39:54] Just take everything down. Yeah. And you [39:57] know, certainly [39:59] different projects I would I would have [40:02] different boxes of those checked. Like [40:04] if this was an open source project, uh [40:07] the only one I would consider leaving [40:09] checked would be the CI one, you know, [40:11] driving I there's no way uh and maybe [40:15] rebasing um but there's no way that I [40:17] would allow it to automatically address [40:19] arbitrary review feedback coming in on a [40:22] PR and certainly not automatically merge [40:25] um on a closed on a private closed [40:28] source thing where I trust uh you know [40:32] the trust the people that I know to be [40:34] working on the project. Um it's it's [40:38] less risky, but even then I still, you [40:42] know, the that less check box you have [40:44] to be very very trusting. Um [40:49] yeah, the one of the things I like about [40:51] the app as well is it does you there [40:52] there are built-in prompts as part of [40:54] agent merge for what those different [40:55] things are, but the app does permit me [40:58] to add my own prompting for those [41:01] different capabilities. And so I have [41:02] additional constraints and guidance for [41:05] the agent as part of doing each of those [41:07] things about how it should do it. Like [41:10] you know always rebase instead of merge [41:12] for for one of them or for review [41:14] feedback. Here are the kinds of things I [41:16] want you to think critically about [41:17] before you just blatantly accept it and [41:19] these are the ones where it's okay just [41:21] go ahead and address it. You know those [41:22] sorts of things. OpenAI, Enthropic and [41:25] Cursor all use this product to make [41:27] their lives easier. Work OS is an API [41:29] platform that adds enterprise features [41:31] like SSO, skim, and arbback to your app. [41:34] These are things that you need to sell [41:35] to other companies, but aren't the main [41:37] focus of what you're building. And now [41:39] they've added more support for AI [41:41] agents. For instance, O for agents gives [41:43] each agent its own identity with [41:45] short-lived scope tokens instead of a [41:47] longived API key. So if something goes [41:49] wrong, you can revoke one session and [41:51] everything under it dies. Another [41:53] feature they have is airlock, which is [41:55] an authorization layer. [snorts] Every [41:56] action agent takes is checked against [41:58] rules that you can write in plain [42:00] English. So it's not rubber stamping [42:02] everything and it's not running [42:03] unsupervised either. And lastly, they [42:06] have pipes which connects your agents to [42:08] over 300 third party tools without you [42:10] ever needing to manage a token. So if [42:12] you're building SAS or AI agents and [42:14] selling to enterprises, work OS fills [42:16] all these gaps in a few lines of code. [42:19] Check them out at work os.com to learn [42:21] more and get started. And I appreciate [42:23] them for sponsoring my show. We talked [42:26] [snorts] a bit about regressions and you [42:28] mentioned that a lot of them were caught [42:31] uh before they went out to prod it [42:33] sounds like. Um and I think a lot of [42:35] people might hear regressions and they [42:36] think oh you're you're taking down the [42:39] app or you know things are exploding all [42:41] the time or their incidents. Could you [42:43] explain maybe the the further [42:46] protections after end to end testing [42:48] that made it so these aren't bad [42:49] regressions? [42:51] I would say probably [42:53] twothirds of these things that I'm [42:55] calling regressions were caught very [42:58] soon after merging to main and before we [43:00] shipped any pre-release. Um even though [43:03] we're shipping at this point we're [43:04] shipping I think three pre-releases a [43:06] day. So they were caught like very [43:08] quickly. Um then there was there were [43:11] some that made it into a pre-release and [43:14] um you know our form of ring deployments [43:17] with the CLI is basically pre-release [43:19] and stable where pre-release on any [43:22] given day is probably hitting 5 to 10% [43:24] of the um the audience um of the [43:28] customer base and then you know 90 or [43:30] 95% is only unstable and so a a a [43:33] reasonable number of these issues [43:35] uh I don't know exactly what percentage [43:37] 10 20% [43:39] um found their way into a pre-release. [43:41] And then pretty quickly we'd get an [43:43] issue opened or someone saying, "Hey, [43:44] like I'm seeing this difference in [43:47] behavior. Was this by design or not?" [43:48] And so a lot of these were things that [43:50] were different, not necessarily bad, but [43:54] different enough that someone noticed [43:55] and then asked us like, "Was this on [43:58] purpose?" Um, and some of those cases [44:01] were ones where I failed to prevent the [44:04] agent from fixing bugs as part of the [44:08] port. And so it it some of them were [44:10] actually desirable changes, but we [44:11] hadn't intended to, you know, they they [44:14] weren't purposeful in that regard. Um, [44:18] and then there there were a a number [44:21] that made it into stable releases. And [44:23] the ones that made it into stable [44:24] releases, because we hadn't noticed them [44:27] in either main or pre-release, they [44:29] often ended up staying in and being [44:31] unnoticed in stable releases for quite a [44:33] while. Um, and this is mostly the [44:36] trickle that we have been seeing over [44:38] the last, you know, well, you don't [44:40] really see much anymore, but a month ago [44:42] would have been the trickle that we were [44:43] seeing, you know, over a few weeks. uh [44:45] where some super corner case thing just [44:48] behaved slightly differently and then we [44:51] get an issue open like hey this this [44:53] isn't working the way I expected you [44:55] know is this on purpose or not and often [44:57] times they weren't and we'd fix them as [44:59] soon as we heard about them um there [45:01] were also ones that we would see from [45:03] various telemetry systems [45:06] you know uh monitoring for exceptions [45:09] crashes [45:11] uh things like that and obviously those [45:14] are unintentional and we fix those. [45:16] >> In the post, you you shared this graphic [45:19] of which models you used throughout the [45:22] course of the project. So, you have this [45:24] nice um I guess bar chart that's showing [45:27] what portion of each week's I guess [45:30] tokens are consumed by either Claude or [45:33] or GPT and in rare cases Gemini. At the [45:37] beginning of the project, it was almost [45:38] entirely Claude and then by the end of [45:41] the project, it was almost entirely GPT. [45:43] So what made you shift your model of [45:47] choice in the middle of the project to [45:49] GPT? [45:51] >> Um for the most part I was constantly [45:55] trying different ones and uh cuz you [45:58] know over that period like both labs [46:00] were releasing new models at a very fast [46:03] pace. And so I was experimenting and [46:06] when when I would move to one and I [46:08] found it to work well, I would stick [46:10] with it until uh something bothered me [46:14] and then I would try a different one and [46:15] then you know so um it doesn't in any [46:19] way reflect sort of [46:24] a long-term preference. It was more of [46:27] at that moment what was working best for [46:30] me then. Um, and so at the beginning of [46:34] it, I think I was on mostly Opus [46:37] 46 or Opus 48, and it was working great. [46:41] And then GPT [46:44] uh, what was it? 5.5, 5.6, one of those, [46:49] and that started working even better for [46:51] the these particular workloads. So then [46:52] I jumped to that. And then Opus 5 came [46:54] out, and I was, oh, this is cool. And [46:56] jump to that one, right? And then um [46:58] recently I've been having a lot of luck [47:01] with uh 61 Saul. Um and I'll use uh I'll [47:08] use Opus 55 for code review uh for the [47:11] work that I've done with 61 Saul. Um and [47:14] so I'll like I'll sort of split the [47:16] balance there. Um sometimes if I get [47:19] really frustrated with something like [47:21] both over the course of the port both [47:23] models both GPT and Claude would exhibit [47:26] some very frustrating [47:28] traits at certain points like um a [47:31] particular point or if I was working on [47:33] it for some period of time one of the [47:36] models might get stuck and kind of like [47:39] no matter what I did it kept kind of [47:42] refocusing on a particular thing and not [47:45] stepping out and looking at the forest [47:46] for the trees. Um, and there are various [47:50] things I would do in those situations. [47:51] Sometimes I would start like keep the [47:53] code changes but start over with a clear [47:55] session and try and use that to, you [47:57] know, clear context window and try and [47:59] use that to get it to reset itself. But [48:01] often it would find itself back into a [48:02] rut. And so in those situations, I would [48:04] switch models um and then keep going [48:07] with the new model until it found itself [48:10] exhibiting some frustrating behavior and [48:12] then I would, you know, switch to [48:14] something else. Um I did experiment [48:17] using Gemini and Grock as my main [48:20] drivers. Um but I would often leave [48:23] those to be um as part of like a [48:26] multimodel code review. Um so a lot of [48:29] the the reviewing that I had that I was [48:32] doing I would be using either GPT or [48:35] Claude as or for the main work. um and [48:39] then use three or four different models [48:43] from different families to do a line by [48:46] line uh you know review of the the [48:49] porting that was done and I would do [48:51] that very frequently um at at least as [48:56] frequently as every time that I was [48:57] rebasing to bring in additional [48:59] functionality from uh you know changes [49:02] that had been coming into Maine but [49:04] often uh even more frequently than that. [49:07] Yeah, I did see that you had a prompt [49:09] that was I guess auto applied on rebase [49:12] or something like that where you [49:14] requested it to be reviewed um I guess [49:18] adversarially or you know all all of [49:20] them at the same time. What made you [49:23] come to that conclusion rather than I [49:25] guess code review with the same model [49:28] but I don't know different focuses for [49:31] each thread? Yeah, it's it's basically [49:33] to get different effectively different [49:36] perspectives. You know, every every [49:38] little change that you make to a [49:40] non-deterministic system ends up sending [49:42] it down, you know, different paths and [49:44] looking at different things. And one of [49:46] the biggest changes you can make is [49:48] completely changing the model. Um and so [49:51] the best way to get different opinions [49:54] and focusing on different things I that [49:57] I found is by having um different models [50:00] especially than the one that I used [50:02] initially. So um you know after you [50:06] finished the rewrite what was the [50:08] concrete impact and the cost? So if you [50:13] look at it from a business perspective [50:15] what was the benefit and the cost of [50:17] this project? [50:19] So the benefits are stark. I mean the [50:21] benefits are [50:24] the ability for all these products and [50:27] services and teams across GitHub and [50:30] Microsoft to be able to rely on a shared [50:34] harness that is constantly sort of, you [50:38] know, best of breed and getting better [50:40] and better every day and not have to [50:43] spend their own resources [50:45] trying to do the same thing. they can [50:47] focus on their business value and in and [50:49] reuse this component. It's the same as [50:51] with any library that you want to rely [50:54] on. And this library is a particularly [50:56] large library, right? You want to be [50:58] able to focus on your business value and [50:59] reuse the constantly evolving work for [51:02] someone that is spending all their time [51:04] focusing on on that thing. And so from a [51:07] business value perspective, it's it's [51:10] like incalculable across GitHub and [51:13] Microsoft being able to leverage this [51:17] shared harness. Um whereas before the [51:20] port, a lot of that would have been a [51:23] harder sell and harder to justify. And [51:28] um [51:30] part of that is because of very real [51:33] uh costs uh like uh performance costs. [51:35] So you know something that was taking 10 [51:39] times as much working set waving my [51:42] hands means 10 times less density [51:45] which means 10 times as many servers. [51:47] You know those sorts of things. Um and [51:52] uh if you can you know make the um if [51:57] you can scale the number of users that [52:00] are able to be on the same machine [52:02] running concurrently that has impacts on [52:04] your your bottom line. If you can scale [52:06] the you know the throughput and reduce [52:10] the latency by five or 6x or whatever it [52:13] is that has real benefits. Um one of the [52:16] the biggest impacts from a Perf [52:18] perspective um a a lot of these systems [52:21] especially services will do um you know [52:26] a request will come in and the the agent [52:29] loop needs to be started to make one [52:32] more turn which could be a very short [52:34] thing and then the request is done. And [52:37] um so for these things the startup cost [52:40] ends up being one of the dominant [52:42] factors and the startup of a node [52:45] application versus the startup of a a a [52:49] native Rust or C application is there's [52:52] a big difference there. we're talking [52:54] hundreds and hundreds of milliseconds [52:57] uh difference or or in some cases [53:00] seconds um on a per request basis that [53:04] latency can be extremely measurable and [53:08] extremely impactful for bottom line if [53:11] you're even able to adopt it because you [53:14] might not be able to and then you're [53:15] back in the previous world that I was [53:17] talking about. So from a business [53:18] perspective, it's worth a lot to to do [53:22] this port. Um, in terms of what it [53:25] actually cost us, um, it was about [53:29] if I so I I stored all of my session [53:33] logs or 99% of my session logs from this [53:36] effort. And from those session logs, I [53:38] was then able to sort of deduce the [53:40] token counts for fresh tokens, cash [53:44] rights, cash reads, uh output tokens. [53:48] And all told, it was about 136 billion [53:51] tokens broken up in such a way that if [53:55] you were to use the public pricing [53:58] um of those models at those times, it [54:02] came out to approximately $120,000. [54:07] Um this is not factoring any discounts [54:10] or things that companies might have for [54:12] what those tokens cost. Um and uh this [54:15] is just sort of what's on the website of [54:17] you know open AI and anthropic plus my [54:21] time which you know the it took it was [54:25] about 14 15 weeks um but I was doing [54:30] other things at the same time you know a [54:32] lot of these uh agentic projects if if I [54:35] look at my my co-pilot app that I [54:38] currently have on my screen I've got [54:40] about 35 sessions right now for things [54:42] that I'm working on some of them in the [54:44] same area. Some of them are uh across [54:48] different projects. Um and [54:51] so I'm not developers are no longer [54:54] working on one thing at a time. So if [54:57] you look at the percentage you just as a [54:59] rough estimate for my time if you look [55:01] at the percentage of PRs that I was [55:04] submitting over that same period about a [55:06] fifth of them give or take was to do [55:10] with this this project and about 80% was [55:13] for other things. Um and so if you [55:17] figure you know about 20% of about 15 [55:21] weeks you end up with about three weeks [55:22] of my time spread out over three months. [55:25] um plus the the token costs. Um and it's [55:29] only getting cheaper. The the the models [55:32] are becoming like at cheaper at an [55:36] astronomical [55:37] rate. I think like when OpenAI released [55:41] I'm not going to remember the exact, but [55:42] I think like 61 Saul is half the price [55:44] of six Saul and six was half the price [55:46] of 56 Saul. Like the they're just [55:49] getting cheaper and cheaper and cheaper. [55:51] Um, and the models are also getting [55:54] better and better and better, which [55:55] means that the amount of time that I [55:57] need to put in to steer the models to do [56:00] the right thing is also decreasing. Um, [56:02] and so the costs of these ports, the [56:05] cost of these migrations is only getting [56:07] cheaper over time. Wow. So, three weeks [56:11] of uh I guess of developers time plus [56:15] $120,000, you're able to rewrite the [56:17] entire codebase. Um uh if you were to [56:21] scope this out preai, let's say you [56:25] wanted to undertake this, what would [56:28] that look like? First, it wouldn't have [56:30] happened. [laughter] [56:32] Um for and I'm actually ser like for [56:34] this particular thing the [56:38] it wouldn't have happened because by the [56:40] time we were done, we would have ported [56:42] the wrong thing. like the this space is [56:44] moving so fast that we would have [56:46] basically never finished. I think in [56:49] fact you can kind of see that in the [56:51] charts in the blog post in that for a [56:54] good portion of the porting effort the [56:57] amount of rust is increasing and the [57:00] amount of TypeScript is largely [57:02] remaining the same but increasing a [57:04] little bit and that's because as the [57:06] port was happening there was just a huge [57:09] amount of new functionality coming into [57:11] the repo right and at the beginning of [57:14] the porting effort because the majority [57:16] of the codebase was Typescript the [57:18] majority of the new incoming code was [57:19] also Typescript. By the end of the [57:21] project where the majority of the [57:22] codebase was Rust, the majority of the [57:24] new incoming code was Rust. But if you [57:26] look at like the first half of the graph [57:28] where we're generally 50% or less Rust, [57:32] the amount of TypeScript was doing this [57:35] even as the amount of Rust was, you [57:37] know, doing that. You know, even if that [57:40] wasn't the case, this we're talking it [57:43] was about 450,000 [57:45] lines of Typescript, just production [57:48] source, never mind tests that were [57:51] ported. I think there was a few [57:52] hundred,000 lines of unit tests there in [57:54] Typescript as well that also needed to [57:55] be ported to Rust. Um, [57:59] just think about the productivity of an [58:01] individual developer and how much they [58:03] can reasonably do in a given day if [58:07] they're top-notch. And we're talking a [58:10] whole team of developers for a year or [58:12] two to do something like this, right? [58:14] It's it is a [58:17] if you're able to do it at all, it is it [58:20] would have been a monumental investment. [58:24] And like I I struggle to think about [58:27] even writing that proposal because I [58:29] know that it would be sort of I would [58:32] look bad like I would be dismissed [58:34] immediately as of course we're not doing [58:35] that. [58:37] Um whereas this was a proposal like I'm [58:40] going to take you know 20% of my time [58:43] over 3 months and by the end this whole [58:46] thing is going to be moved over and for [58:47] the most part no one else on the team is [58:50] going to be impacted. Um, and you know, [58:53] that's not 100% true, but like waving my [58:56] hands and we'll have all these business [58:59] benefits in a few months. Um, and it, [59:02] you know, this will be the bill. That's [59:04] a much easier proposal to write. And [59:09] even just the even if you were paying [59:12] that actual, you know, uh non-disounted [59:15] public-f facing uh token rates of [59:19] $120,000, you compare that to the cost [59:22] of a senior developer, [59:25] you know, for a year, [59:28] uh with all their benefits and their [59:31] bonus, like even that is a good deal. [59:35] It sounds like at least for this type of [59:39] work where it's super what we want is [59:42] very obvious and encoded somewhere in [59:45] the existing functionality. So like this [59:47] type of work is [59:49] at least one order of magnitude cheaper [59:51] maybe even more. I mean, if you had a [59:53] team of engineers and they're, you know, [59:57] engineers maybe two to 400K or something [60:00] like that per this is a drop in the [60:03] bucket, the 120K. When you were [60:06] proposing the project, did you have a [60:09] sense of how much this would cost in [60:11] tokens before you went and did it? [60:13] >> Yes. And [60:17] the bill ended up being larger than I [60:19] expected. Um [60:22] mostly because the line count ended up [60:25] being larger than I expected. Um I [60:27] hadn't fully factored in the growth rate [60:31] of how much incoming code there would [60:34] be. So this was you know there were [60:36] there were a few things I think I did a [60:37] lot of things right. I did a few things [60:39] wrong and you know 2020 hindsight and [60:42] all that. One of them I already [60:43] mentioned I didn't spend enough time uh [60:46] really fleshing out the endto-end tests [60:48] to make sure that they were as [60:49] comprehensive as they could have been. I [60:51] could have done that better. The other [60:52] thing I could have done better was [60:54] really factor in the growth rate of code [60:57] that would need to be ported. So when I [61:00] looked at the the code base as part of [61:03] writing this proposal, it was somewhere [61:06] on the order of 150,000 lines of [61:09] production TypeScript. 30 something like [61:11] that. I don't remember the exact number. [61:14] Um and based on that and some back of [61:16] the napkin calculations and looking at [61:19] how long it took to create the [61:20] prototypes that I'd alluded to earlier [61:23] for you know in the different languages [61:24] to sort of measure Perf and things like [61:26] that, I think I'd estimated about um six [61:30] to eight weeks [61:32] uh to get it done. Um, but when you look [61:37] at what we actually needed to port, I [61:38] didn't factor in a few things. One, that [61:42] there were, you know, tens of uh [61:45] developers using agents [61:48] uh very quickly augmenting the codebase [61:50] with lots of different additional [61:53] functionality. Um, and [61:57] uh, you know, this codebase is one [61:58] that's it's not public open source, but [62:01] it's effectively internal open source. [62:03] So there's a bunch of folks that come in [62:04] and add the things that they need and [62:06] then the core team is helping to review [62:08] all that. Um and so there's this very [62:12] significant influx of functionality that [62:14] I hadn't accounted for. I also hadn't [62:17] accounted for the fact that the because [62:19] of how the code base came about. Um the [62:23] CLI, the Copilot CLI and the runtime [62:26] were kind of very intertwined more so [62:30] than they ideally should have been. And [62:31] so in order to port just the runtime um [62:36] and you know the CLI can be ported [62:37] separately if you wanted to but to [62:39] actually tease them apart to the point [62:40] where you don't need the CLI you don't [62:42] need to deploy the CLI as part of your [62:44] application that's using the runtime [62:46] there's this effort of teasing them [62:48] apart but then there's also an effort of [62:49] moving functionality that really doesn't [62:52] need to be in the 2e and that could be [62:54] shared in the runtime down and I failed [62:56] to account for how much of the code that [62:59] was currently marked as CLI would [63:03] actually be beneficial to move down. And [63:05] so over this effort, there was also code [63:07] that was coming down. So by the end um [63:09] we ported somewhere between four and I [63:12] think 450,000 lines of Typescript. So [63:14] about three times what I'd estimated um [63:18] which impacted both the endto-end time [63:20] that it took to do it and then also uh [63:23] sort of token counts. So after having [63:27] gone through this process, if there was [63:30] a software engineer out there on some [63:32] team considering doing some sort of [63:34] rewrite, what criteria should they look [63:37] for to decide if it's worth undertaking [63:39] it or not? It it really comes down to [63:42] what is the business value? I mean, [63:44] there's there's the ones you can do for [63:46] fun where it's your dime and you get to [63:48] decide, you know, what fun is worth. Um, [63:52] just like, you know, I I really don't [63:55] like going to casinos because I really [63:57] don't like what I consider to be losing [64:00] money gambling, but friends of mine love [64:03] it and they consider it paying for a [64:05] night's worth of entertainment. They're [64:07] not losing money, they're paying for [64:08] fun, right? And so if you're if you've [64:10] got your own personal development [64:11] project and you want to spend tokens on [64:14] porting that to something else, great. [64:17] Have fun. Like that sounds awesome. But [64:20] but the benefit you need to understand [64:22] what the benefits are. The benefits are [64:23] your enjoyment. When it comes to a [64:24] business, it's what is the business [64:27] value that this is occurring to. And [64:30] then like anything else, what is what is [64:33] it going to cost? Uh and what is the [64:36] return on that investment? And that [64:38] needs to be very clear. And there are [64:41] lots of these migration projects that [64:43] are people all over the world and [64:45] businesses all over the world are doing [64:47] even before agents where they have done [64:50] that cost assessment and they've decided [64:52] that the return on investment is very [64:55] much worthwhile. Um I saw this a lot [64:58] with net with you know people moving [65:01] from [65:02] net framework to net core. Um, and [65:06] because we invested, you know, a lot in [65:08] making .NET Core incredibly fast. And so [65:11] there would be very real performance [65:14] benefits that would come from doing that [65:16] migration. And they could figure out the [65:17] cost associated with, well, we're [65:19] spending X millions of dollars running [65:21] our services today. And we would save $3 [65:25] million a year by moving, you know, by [65:27] doing this port. And this port is going [65:29] to take a year and it's going to cost us [65:32] a developer for that year. So they do [65:34] the math and you know this is this is [65:36] absolutely worthwhile. Um even at a a [65:40] more micro level not even for ports like [65:42] what features do you invest in? when I [65:44] was um spending a lot of my time working [65:48] for example on the net um reax engine uh [65:54] you know I I could look to see how much [65:57] are we spending we as a company you know [66:00] or how much is this particular service [66:03] spending on running on regular [66:04] expressions every year if I invest two [66:08] weeks in making these changes to the net [66:12] reax engine and it makes the the [66:14] patterns that they're using four times [66:16] faster. What is the return on investment [66:18] for my two weeks and how much they save [66:20] as a service over the course of the [66:21] year, right? That these kinds of [66:23] trade-offs are everywhere. And you just [66:26] need to be very clear about why you're [66:28] doing it, what it will cost, and what [66:30] you get for it. And if you can write all [66:32] that down and you can justify it, then [66:35] it's a good project to do. If you can't [66:37] justify it, it's very suspect. [66:39] [laughter] [66:40] >> When I was doing research for this [66:41] interview, I looked into your career. [66:43] Well, on your LinkedIn there's no [66:44] information, but I found an interview [66:46] you did with [laughter] [66:48] >> Yeah, I'm very I'm I basically I I've [66:51] been at Microsoft for 25 years. I love [66:53] it. I've jumped around a lot internally, [66:56] but I've had no interest in going to [66:59] different companies and so I haven't [67:01] really invested in keeping my LinkedIn [67:02] profile up to date. [laughter] [67:05] But so when I when I did the digging [67:06] though or like listened to an interview [67:08] that you had done about the different [67:10] roles, your path to distinguished [67:14] engineer is really interesting because [67:16] you did a bunch of different roles and [67:18] it wasn't just sui on team A and then [67:23] Sui on team B. You held a variety of [67:27] ones starting as almost a like a forward [67:30] deployed engineer, consultant engineer. [67:33] Yeah. [67:33] >> And then you did technical writing at [67:35] some point and then you were an adviser [67:38] to a a technical adviser to a VP. Oh, [67:41] and you also jumped to a PM at some [67:43] point. From all those different roles [67:45] that you tried, which of them would you [67:48] say was the most um like you would [67:51] recommend someone dabble in that [67:53] direction if they wanted to try [67:54] something that wasn't the pure SU roll? [67:57] >> Yeah, every role that I did taught me [68:00] something different. And so from that [68:02] perspective like I have great respect [68:03] for people that you know are come out of [68:06] college and want to be developers and [68:08] become an engineer and that's and that's [68:09] what they do and they just stay on that [68:10] path like that's a very valid path for [68:13] me. I think I picked up a whole bunch of [68:15] skills by having more of a ciruitous [68:18] route. So like you you mentioned as a [68:21] forward deployed engineer it was more at [68:22] the time was Microsoft's consulting arm [68:25] and so I would spend anywhere from three [68:26] weeks to three months at a particular [68:29] customer site helping them solve their [68:31] problems and I would learn a whole lot [68:34] about their industry. I would learn a [68:35] whole about a bunch a whole lot about [68:38] their particular breed of problem. I'd [68:40] learn about Microsoft's offerings from [68:43] the outside and places that it was great [68:45] and places that it wasn't so great. um [68:49] and learn how our customers like their [68:53] their real struggles and and what they [68:55] were dealing with and being able to [68:56] bring that back to then later building [68:59] solutions for them was incredibly [69:02] valuable. Um, when uh I then worked on [69:07] like MSDN magazine and was doing a bunch [69:09] of technical writing and technical [69:11] editing and figuring out what I wanted [69:13] to share with people and how I wanted to [69:14] share it and um some amount of [69:17] development building things to then talk [69:19] about like I I gathered a set of skills [69:23] around how to [69:26] think through what was really [69:27] interesting, what was really relevant, [69:30] uh how to communicate it in a clear [69:32] fashion. [69:33] uh what didn't needed to be said, what [69:35] did need to be said, how to tell a story [69:37] that would really resonate. Um from a PM [69:41] perspective, it became, you know, how do [69:43] I [69:45] how do we figure out what actually needs [69:47] to be built and what are all the pieces [69:49] that need to come together to really [69:51] deliver this thing. Um obviously from a [69:54] dev perspective all the coding and the [69:56] design and the and the architectural and [69:58] all that kind of stuff. The most [70:01] interesting role for me I think which I [70:04] am incredibly thankful and glad that I [70:08] did and will never do again [70:11] uh was being um sort of as technical [70:15] advisor for a VP at Microsoft. Um it was [70:19] an amazing experience. It was a couple [70:22] years where I was there right-hand sort [70:26] of like invisible partner um shadowing [70:29] everything they were doing and ghost [70:31] writing and after every meeting having a [70:34] little you know review kind of going [70:36] through what we heard and what we think [70:37] we need to do about it and bouncing [70:40] ideas back and forth. Um it was [70:43] incredibly insightful into how things [70:46] actually work, how decisions actually [70:48] get made. um being a part of things that [70:52] I wouldn't otherwise have been a part [70:54] of, at least not for many, many years [70:56] later. Um it was awesome. It was also [71:01] incredibly draining, incredibly [71:03] selfless. Uh [71:06] and [71:08] it it took a lot out of me. Um and so I [71:13] cherish those experiences and what I [71:15] learned from them. And I would highly [71:16] encourage anyone who has the opportunity [71:18] to do so to do that kind of thing for a [71:22] period of time. [71:24] >> That role is really interesting cuz it [71:26] sounds like you're uh you're basically a [71:29] chief of staff to someone super high up [71:31] but technical. So you're not like an [71:34] executive assistant. you're like the [71:36] you're pulled into all the really I [71:39] guess privileged conversations and [71:41] you're providing that that information [71:44] that they need to make the calls. You [71:46] mentioned that you learned how things [71:48] are like really done. Um do you have [71:50] some examples of stuff that someone [71:52] might learn if they were kind of in that [71:54] that role or something that was really [71:55] helpful for you? There are [71:59] lots of times where [72:02] if you're sort of rank and file engineer [72:04] at a company and you see these decisions [72:08] that get made, an email gets sent out [72:11] about some new path we're following or [72:13] some partnership we're forming or some [72:15] investment that we're making, some [72:17] acquisition that is being done, winding [72:20] down a certain product, investing in a [72:22] new product, spinning up a new team, [72:24] like all of those things [72:27] look like they just sort of come out of [72:30] thin air and they're, you know, they're [72:33] just sort of handed down as this is the [72:35] finished decision and it's fully formed. [72:39] You don't really get insights into [72:41] everything that led up to that point and [72:44] all of the alternatives that were [72:45] considered and dismissed and why and [72:48] what the trade-offs were that were [72:50] factored in and how experienced people [72:54] in the industry are thinking through [72:56] those alternatives and those problems [72:59] and h which things they're choosing to [73:02] focus on, which things they consider to [73:04] be irrelevant. Um, [73:08] all of that is stuff that I didn't have [73:09] exposure to. And being able to be a fly [73:12] on the wall, [73:14] uh, just soaking in, not to mix [73:17] metaphors, but, you know, soaking in all [73:19] of that and then being able to have [73:21] conversations about it to, you know, I [73:24] we'd get out of a meeting and he'd [73:25] basically say, "So, what did you hear? [73:28] What do you think? Like, help, you know, [73:30] help reason through it." And sometimes I [73:32] would have valuable insights and [73:35] sometimes he would say no. That's [73:37] [laughter] [73:38] that's not how how we think about these [73:40] things. And and every one of those was a [73:41] learning experience for me. [73:43] >> Yeah. I've heard engineers say similarly [73:46] about the value of trying management [73:49] even if they ultimately decide they hate [73:50] it and they don't want to do it again is [73:52] you Yeah. You get to see how the sausage [73:55] is made a little bit more as a manager. [73:56] you see all those decisions uh in [74:00] managing the org that can be helpful in [74:02] operating uh just in your your career in [74:06] general. [74:07] >> Yeah. And you know I also I I'm I didn't [74:10] mention it before but I also did that [74:11] like I I thought that what I really [74:14] wanted to be was a manager [74:17] and um so I basically set things up [74:20] whereas a certain point I trade you I [74:23] jumped around a lot internally at [74:24] Microsoft and I changed to a role where [74:27] I had sort of handshake agreement that [74:29] six months later I would be able to sort [74:31] of shift into a lead position and have a [74:33] team of of people. And I arranged all [74:36] that and it all worked out and I became [74:38] a lead. And six months into being a [74:41] lead, I was like, [74:44] I'm not sure if this is actually what I [74:47] thought it was. And six months after [74:49] that, I was like, nope, it's definitely [74:50] not. [laughter] [74:51] This is not for me. Um, and then it took [74:55] me about a year after that further to [74:58] sort of [75:00] um, safely work my way back to uh, an IC [75:03] role where I've been since. Um, and [75:06] again to your point like super grateful [75:08] for the experience, incredibly grateful [75:10] to Microsoft for having the for having [75:13] an individual contributor career track [75:16] that allows one to sort of progress [75:18] their whole career and continue to be [75:21] promoted and whatnot, but still not have [75:22] to have direct reports. For me, what I [75:25] realized was the thing that I loved [75:27] about or that I thought the reason I [75:29] wanted to be a manager was because I [75:32] thought it was how I could help others [75:35] and how I could kind of provide [75:39] uh career guidance or help other people [75:42] solve problems or whatever it was. And [75:45] that was misguided on my part. And it [75:47] took becoming a manager to realize that [75:50] I could do all of that as an individual [75:52] contributor. [75:54] And the things that I that I loved about [75:57] being a manager were things that I [75:59] didn't have to be a manager to do. [76:01] >> Do you have any top book recommendation [76:03] that maybe impacted you? [76:05] >> Yeah, I mean it's old at this point, but [76:08] I've read it several times. Um, and I [76:11] absolutely love it. There's a book by [76:14] Charles Petzold called Code. It's [76:17] probably from what 200 [76:20] three maybe, I don't know, something in [76:22] that time frame. But I love it and it is [76:25] one of the things that I really value is [76:27] being able to explain complicated [76:30] concepts [76:32] in a way that doesn't require the person [76:34] to know anything about what you're [76:35] talking about. I think that is a [76:37] valuable skill even if you're talking to [76:40] people that do know what you're talking [76:41] about because being able to break down [76:44] complicated concepts into very clear [76:48] concise [76:50] pieces and communicate that clearly and [76:53] then at a 30,000 ft level is a is a [76:58] great way of getting everyone onto the [77:00] same page and sort of arguing your case [77:03] and pushing things in the direction that [77:04] you want them to [77:06] And code is a book where Charles [77:09] basically does that where he talks about [77:12] how computers work from the ground up [77:15] teaching people that know nothing about [77:18] computers that aren't technically [77:20] experienced about binary about logic [77:23] gates about circuits about programming [77:27] about computers about the internet and [77:29] kind of building things up from basic [77:31] principles from nothing to a reasonable [77:35] understanding of how everything works. [77:37] Um, and I was blown away by the book and [77:39] I've since reread it on multiple [77:42] occasions. Even though I know everything [77:44] that's in there, I just love the way [77:45] that it communicates things. So, if I [77:47] had one recommendation that that's it. [77:50] >> Looking back on all your experience, [77:51] you've kind of been working in the [77:53] industry for a while. What professional [77:55] advice would you give a young engineer [77:57] just entering the industry today that [78:00] you believe could be helpful? That's a [78:02] great question and it's one I've been [78:04] struggling with. Um, a couple years ago [78:07] I would have had a much more clear [78:09] answer, but with AI, [78:13] I [78:15] still don't know what the right answer [78:17] is. Um, a lot of the times that I spend [78:21] working with AI agents, I'm bringing to [78:23] bear 25 years of experience for better [78:28] or worse in being able to [78:34] very quickly sniff out things when [78:36] they're doing the right thing or they're [78:37] doing the wrong thing or how I want to [78:39] guide them. Or an agent says, I'm 100% [78:42] confident that the right answer is A. [78:43] And I'll say, "Yeah, but have you also [78:45] considered B and C and D because these [78:48] trade-offs might not be there?" Right? [78:49] And oh, of course. Like just those [78:51] little things being able to guide them [78:54] in the right way [78:56] requires you to have gone through it. [79:00] And I'm concerned that because of what [79:02] AI how AI agents are progressing that [79:04] they're taking away the opportunities to [79:07] have gone through it. [79:09] And I don't yet know what the answer [79:11] then is. Um I am a very firm believer in [79:16] um really learning kind of the basics [79:20] and building everything up from core [79:22] principles. And so like when you know [79:25] learning a language, I still think it's [79:27] valuable to learn and to start out with [79:29] a language like C because you're [79:31] understanding how computers actually [79:34] work and you understand what memory [79:36] management means and you learn about all [79:38] the various trade-offs between this [79:40] algorithm and that algorithm rather than [79:42] working at a layer where a lot of that [79:44] is obscured from you. uh where the [79:47] language and environment are providing a [79:49] ton of services to you but you don't [79:51] necessarily understand everything that's [79:53] happening and so from that perspective I [79:56] still think that there's value in [79:59] focusing on those core concepts and [80:03] really internalizing algorithms and how [80:07] machines actually work um and being able [80:11] to reason and extrapolate from that and [80:13] I suspect that that will [80:18] in whatever domain you're in. That kind [80:20] of core primitive knowledge about [80:22] distributed systems or about databases [80:25] like that foundational stuff will help [80:28] to be successful in a world where the [80:31] vast vast vast majority of the code is [80:34] actually being written by agents and [80:36] you're helping to guide them. [80:38] But I don't know that for sure. And at [80:40] the rate that these agents are learning [80:42] and evolving, I don't know what the [80:44] world's going to look like in two or [80:45] three or four years. Um, I suspect it'll [80:48] look vastly different than it does [80:50] today. I think we're in for a wild ride. [80:54] >> Awesome. Well, yeah, thank you so much [80:55] for your time today, Stephen. I really [80:57] appreciate it. [80:58] >> Pleasure. This has been a fun [80:59] conversation. Thank you. [81:00] >> Hey, thank you for watching this [81:01] podcast. If you liked it and you want to [81:03] see the show grow, please support with a [81:05] comment or a like. Also, if you have any [81:08] recommendations for people you want me [81:10] to bring on, please drop a comment. [81:12] Guests like Barbara Liskoff, Mike [81:15] Stonereaker, Mark Brooker, these were [81:17] all people that I brought on because [81:19] someone left a comment. On another note, [81:21] aside from the podcast, I'm working on [81:23] building the ergonomic keyboard that I [81:25] wish existed. Here's a glance at the [81:27] prototype. It's a split keyboard, so [81:30] there's two sides. Um, this is in the [81:32] case, but yeah, we launched on [81:33] Kickstarter and we hit our goal within [81:35] eight hours of launching. I really [81:37] appreciate it if you were one of the [81:38] people who grabbed one of the early [81:40] units. Um, we're now working on the long [81:42] journey of building the tooling now and [81:44] so if you still want to pick one up, [81:46] I've left the late pledges open on [81:48] Kickstarter, so you can grab one there. [81:50] I'll put a link in the description. [81:52] Thank you again for watching the podcast [81:54] and I'll see you in the next