Tomorrow Corp Unveils In-House Stack With Reverse Time Debugging

retrogameinternals

Unifying the language, compiler, and engine offloads determinism directly to the development environment, enabling programmers to isolate memory corruptions and crashes backwards in time without defensive code.

AI-assisted summary of Tomorrow Corporation Tech Demo — original source. Check the original for full context.

Source published: 2023-02-21

Key ideas

Tomorrow Corporation engineered a completely custom, full-stack game development environment—including its own programming language, incremental compiler, editor, debugger, and engine—inspired by Naughty Dog's PS2-era GOAL toolchain. (0:00)
The custom compiler generates native machine code on every keystroke with near-zero perceptual latency across 13,000-line files, while supporting always-on reverse execution without specialized build configurations. (1:40)
A compact 64-bit intrinsic timestamp primitive allows developers to uniquely tag entities and allocations, enabling the debugger to teleport directly back to the exact machine state where an object was created. (4:11)
Deterministic session replays recalculate simulation and rendering live across hot-reloaded assets and recompiles, and can be shared studio-wide without dirtying the recipient's local workspace. (7:33)
Time-travel debugging with reverse data breakpoints allows developers to immediately pinpoint the root cause of off-by-one buffer overflows and memory stomps without reproducing them through trial-and-error logging. (12:31)

Detailed notes

## Background Published on YouTube by retrogameinternals (Tomorrow Corporation programmer Allan Blomquist) on February 21, 2023, this video showcases internal programming and debugging tools developed in-house at indie studio Tomorrow Corporation. According to the video description, it serves as a technical demonstration of the bespoke suite the team built over many years to make their video games.

## TL;DR Tomorrow Corporation engine programmer Allan Blomquist demonstrates the studio's custom, vertically integrated tech stack, which encompasses a bespoke programming language, instant incremental native compiler, code editor, debugger, and engine. Using a port of Human Resource Machine, he demonstrates keystroke-level hot-recompilation, live mid-frame rendering during debugger stepping, and dynamic call-graph profiling linked to compact 64-bit timestamps. The presentation highlights unified deterministic session scrubbing across code and runtime states, as well as portable session recordings that allow team members to seamlessly reproduce crashes and solve dreaded memory stomps using reverse data breakpoints.

## Atmosphere & style The presentation maintains a calm, measured, and refreshingly understated tone, unfolding as an intimate, practical developer-to-developer walkthrough rather than a glossy tech keynote. Blomquist speaks with quiet humility and dry humor, casually cracking jokes about canceling evening plans over memory stomps or pointing out the frowning boss on their custom panic screen. The viewing experience feels quietly revelatory, generating a sense of technical awe as notoriously difficult engineering challenges—like instant recompilation, time-travel debugging, and bit-accurate replay—are performed effortlessly without friction.

## Bulletin - [00:00](https://www.youtube.com/watch?v=72y2EC5fkcE&t=0s) Tomorrow Corporation engineered a completely custom, full-stack game development environment—including its own programming language, incremental compiler, editor, debugger, and engine—inspired by Naughty Dog's PS2-era GOAL toolchain. - [01:40](https://www.youtube.com/watch?v=72y2EC5fkcE&t=100s) The custom compiler generates native machine code on every keystroke with near-zero perceptual latency across 13,000-line files, while supporting always-on reverse execution without specialized build configurations. - [04:11](https://www.youtube.com/watch?v=72y2EC5fkcE&t=251s) A compact 64-bit intrinsic timestamp primitive allows developers to uniquely tag entities and allocations, enabling the debugger to teleport directly back to the exact machine state where an object was created. - [07:33](https://www.youtube.com/watch?v=72y2EC5fkcE&t=453s) Deterministic session replays recalculate simulation and rendering live across hot-reloaded assets and recompiles, and can be shared studio-wide without dirtying the recipient's local workspace. - [12:31](https://www.youtube.com/watch?v=72y2EC5fkcE&t=751s) Time-travel debugging with reverse data breakpoints allows developers to immediately pinpoint the root cause of off-by-one buffer overflows and memory stomps without reproducing them through trial-and-error logging.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=72y2EC5fkcE&t=0s) Tomorrow Corporation developed a proprietary, bottom-up game creation toolchain encompassing a custom language, compiler, editor, debugger, and engine, inspired by Naughty Dog's GOAL toolchain. - [00:00](https://www.youtube.com/watch?v=72y2EC5fkcE&t=0s) The toolchain is general-purpose across Tomorrow Corporation's catalog, as evidenced by a fully functional internal port of their puzzle title Human Resource Machine. - [01:40](https://www.youtube.com/watch?v=72y2EC5fkcE&t=100s) Incremental native compilation executes on every single keystroke, allowing instant hot-reloading of executable game logic across 13,000-line source files without noticeable lag. - [01:40](https://www.youtube.com/watch?v=72y2EC5fkcE&t=100s) Deep coupling between the debugger and the engine keeps the rendering pipeline active during breakpoints, enabling incremental mid-frame draw-call inspection. - [01:40](https://www.youtube.com/watch?v=72y2EC5fkcE&t=100s) Reverse execution—including step-backward, backward breakpoints, and backward hardware data breakpoints—is an always-on feature requiring no specialized builds or recording runs. - [04:11](https://www.youtube.com/watch?v=72y2EC5fkcE&t=251s) Call graph profiling instrumentation can be dynamically injected into or removed from the compiled binary on the fly without restarting the game. - [04:11](https://www.youtube.com/watch?v=72y2EC5fkcE&t=251s) Clicking an execution timestamp directly inside the performance profiler header immediately teleports the live debugger into that exact historical invocation on that specific frame. - [04:11](https://www.youtube.com/watch?v=72y2EC5fkcE&t=251s) A cheap 64-bit runtime intrinsic timestamp enables developers to tag game entities and allocations to track their exact provenance and inspect their instantiation state retroactively. - [07:33](https://www.youtube.com/watch?v=72y2EC5fkcE&t=453s) The whole-session scrubber recalculates game simulation and graphics dynamically rather than streaming pre-recorded video, maintaining determinism across code changes and asset reloads. - [07:33](https://www.youtube.com/watch?v=72y2EC5fkcE&t=453s) The code editor maintains timeline synchronization with the engine, automatically displaying the exact historical state of the source code corresponding to the point being scrubbed. - [07:33](https://www.youtube.com/watch?v=72y2EC5fkcE&t=453s) By baking determinism into the language, compiler, engine, and debugger simultaneously, gameplay programmers are freed from writing fragile defensive code to maintain replay fidelity. - [10:07](https://www.youtube.com/watch?v=72y2EC5fkcE&t=607s) Lightweight deterministic recording sessions are generated routinely by artists, designers, and programmers alike to package reproducible issues across the studio. - [10:07](https://www.youtube.com/watch?v=72y2EC5fkcE&t=607s) The custom build pipeline virtualizes and isolates session dependencies, letting recipients replay foreign sessions without overwriting or conflicting with their local working copies. - [10:51](https://www.youtube.com/watch?v=72y2EC5fkcE&t=651s) When replaying fatal crashes, developers can scrub through the user's prior actions, jump from panic screens directly to source code, and step backward out of call stacks. - [12:31](https://www.youtube.com/watch?v=72y2EC5fkcE&t=751s) Setting reverse hardware data breakpoints on stomped memory addresses allows developers to run execution backwards directly to the off-by-one buffer write that caused the corruption. - [12:31](https://www.youtube.com/watch?v=72y2EC5fkcE&t=751s) Integrating time-travel debugging and deterministic replay directly into the development infrastructure eliminates the lengthy reproduction and print-debugging cycles common in native game programming.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=72y2EC5fkcE&t=0s) Building a Custom Toolchain from Scratch

Allan Blomquist, the engine and low-level programmer at indie studio Tomorrow Corporation, introduces an internal suite of custom-built programming and debugging tools developed over more than a decade. Rather than relying on commercial engines or off-the-shelf IDEs, Tomorrow Corporation built an entirely custom tech stack spanning their own programming language, compiler front end and back end, code editor, debugger, build system, and game engine. Blomquist traces the origin of this ambition back years to reading about GOAL (Game Oriented Assembly Lisp), the proprietary language and toolchain created by Naughty Dog for their PlayStation 2 titles: “I remember reading about that and just thinking that this is obviously like the coolest thing that I've ever heard of it didn't seem like something that you could even do.”

About four or five years prior to this demonstration, Blomquist received the opportunity to build a similar bespoke toolchain internally at Tomorrow Corporation, turning it into a dedicated passion project. To ground the showcase in a practical setting rather than an artificial toy example, Blomquist runs a port of the studio’s puzzle game *Human Resource Machine*. While *Human Resource Machine*—a game themed around assembly programming—was not originally developed with these tools, the port demonstrates that the environment is fully general and designed to support any of the company’s games.

- **Developer:** Allan Blomquist, Tomorrow Corporation. - **Scope of In-House Tech:** Custom programming language, custom compiler (front end and back end), code editor, debugger, build system, and game engine. - **Key Historical Inspiration:** Naughty Dog’s GOAL toolchain used during the PlayStation 2 era. - **Demonstration Game:** A port of *Human Resource Machine*.

---

[01:40](https://www.youtube.com/watch?v=72y2EC5fkcE&t=100s) Hot-Loading Assets, Fast Incremental Code Compilation, and Live Debugging

Blomquist begins with the standard foundation of interactive development: hot-loading data assets. Opening a game texture in an image editor, he draws on it and saves the file, showing how the sprite immediately refreshes inside the active game window without requiring a restart. From there, he extends the exact same interactive loop to executable code. In the custom code editor, he edits the logic governing the green UI boxes on the left side of the screen. Because the compiler is incremental and generates native machine code rapidly, it recompiles on every single keystroke. The game updates in real time with virtually zero perceptual latency, even when modifying a 13,000-line source file.

Moving to the rendering code for the blue floor tiles, Blomquist reveals how tightly the custom debugger is coupled with the custom engine. Setting a breakpoint at the start of a rendering function halts the game mid-frame, leaving the scene partially drawn. As Blomquist steps through draw calls one instruction at a time, each graphic element renders sequentially onto the viewport. Because the debugger controls the runtime engine, it keeps the graphics pipeline alive and interactive during stepping—functioning like a built-in graphics debugger to quickly isolate visual glitches.

Furthermore, the debugger includes full reverse execution. Blomquist steps backward using inverse variants of standard navigation: step into, step over, step out, run backward to breakpoint, and run backward to hardware data breakpoints. Blomquist highlights the massive debugging advantage of these backwards operations: “And being able to debug backwards from an assert or a crash especially with a data break point is insanely helpful in being able to figure out and fix bugs.” Crucially, reverse debugging is not a separate, high-overhead mode requiring a specialized executable; it operates natively all the time during everyday development.

```mermaid flowchart TD accTitle: Custom Toolchain Architecture accDescr: Diagram illustrating the interactive feedback loop between the editor, compiler, engine, and debugger.

subgraph DevEnvironment["Development Environment"] Editor["Custom Editor / Keypress"] Compiler["Incremental Native Compiler"] Debugger["Integrated Debugger"] end

subgraph Runtime["Live Game Runtime"] Engine["Custom Game Engine"] Viewport["Live Framebuffer Display"] end

- **Compilation Performance:** Native code generation on every keystroke with near-zero latency across 13,000-line files. - **Live Frame Stepping:** Keeps the swapchain/viewport alive during breakpoints, rendering draw calls incrementally. - **Reverse Debugging Capabilities:** Step backward (in, over, out), run backward to execution breakpoints, and run backward to hardware data breakpoints. - **Always-On Architecture:** Reverse execution requires no separate build configuration or recording pass.

---

[04:11](https://www.youtube.com/watch?v=72y2EC5fkcE&t=251s) Timestamp Profiling and 64-Bit State Intrinsic Tagging

Switching to the game window, Blomquist turns to the in-game performance profiler. By default, profiler instrumentation is omitted from builds to preserve execution speed. Clicking "Enable" triggers an immediate, seamless recompile of the game code with call graph instrumentation injected on the fly. Observing frame-time spikes in the profiler's call graph, Blomquist freezes data collection and inspects an expensive frame dominated by a single function. In a conventional profiler, clicking the function navigates to the static source code; here, Blomquist clicks an execution timestamp in the timeline header.

Because the system supports backward execution, clicking the timestamp teleports the debugger directly to that exact runtime invocation on that specific frame. Stepping into the frozen call, Blomquist inspects the local variables and discovers the delay was triggered by an unfavorable random number. Once identified, he deletes the offending code, resumes execution, re-profiles to confirm the spike is gone, and recompiles the instrumentation out of the binary without missing a beat.

Beyond automatic profiling tags, the programming language exposes an intrinsic that evaluates at runtime to return a compact 64-bit execution timestamp. Developers can attach this timestamp to game entities, allocations, and events to retain full provenance: “These time stamps are super cheap to generate and they're only 64 bits so you can use them to tag pretty much everything with what effectively is the entire state of the program and then you can go back later and inspect anything you need to if you want to find out where something came from or why it didn't get cleaned up or whatever you want to know.” In the demo, Blomquist breaks into the debugger to inspect three boxes placed on a conveyor belt. By reading the 64-bit timestamp stored in the middle box's struct, he instantly jumps the debugger back in time to the precise instant the character tossed that specific box onto the belt.

- **Dynamic Profiler Instrumentation:** On-the-fly native recompilation injects or removes call-graph profiling without restarting the game. - **Timestamp Navigation:** Clicking profiler frames transports the execution state directly to the exact call site and machine state of that frame. - **64-Bit Intrinsic Timestamps:** Cheap, low-overhead primitives that uniquely capture the complete execution state across time. - **Potential Use Cases:** Tracking memory allocations, enemy spawns, combat hit reactions, door transitions, or state leak debugging.

---

[07:33](https://www.youtube.com/watch?v=72y2EC5fkcE&t=453s) Whole-Session Deterministic Replay and Historical Code Scrubbing

Blomquist escalates the demonstration by bringing up the in-game developer console, revealing a full timeline scrubber and media playback controls that cover the entire session. The scrubber is not replaying a captured video recording; it is recalculating the game's actual simulation and rendering passes on demand. The timeline remains completely consistent across everything Blomquist did throughout the session—including hot-reloading textures, changing UI rendering code, and executing full compiler rebuilds for the profiler. Blomquist emphasizes that the system was deliberately designed to eliminate dead ends: “I specifically wanted to try not to have one-way streets in this functionality so using one feature doesn't rule out using a different one later on.”

When Blomquist scrubs the timeline back to a point prior to modifying the box-drawing routine, the code editor reflects the change. Opening the source while stopped at that past frame reveals the original code before it was edited, because the editor itself tracks timeline state alongside the runtime. This coherence enables retroactive profiling: if a hitch occurs while playtesting, the developer can pause, scrub back, inspect the performance spike, and debug without attempting to manually reproduce the bug.

Traditional deterministic replay architectures are notorious for being fragile, requiring developers to write defensive code to avoid desynchronization. Because Tomorrow Corporation built their language, compiler, engine, and debugger together, deterministic tracking is pushed into the infrastructure itself: “...in our case since we've created every level of this stack to support this we can super efficiently offload the determinism stuff to the development environment and essentially you can just write whatever code you want without having to spend any time thinking about whether or not you're breaking something.” Finally, the engine automatically outputs a lightweight session file containing this timeline on every game run. Integrated directly with their version control system, the stored session files remain tiny while allowing robust, bit-accurate replays far into the future.

- **Unified Scrubbing:** Re-simulates and re-renders full gameplay history on demand, surviving code recompilations and asset hot-loads. - **Time-Aware Editor:** The code editor displays the exact state of the source code corresponding to the point in time currently being scrubbed or debugged. - **Retroactive Hitch Profiling:** Spontaneous performance frame drops can be rewound and profiled after the fact without reproduction setups. - **Low-Overhead Session Files:** Session timelines serialize to small disk files tied into version control, ensuring long-term replay compatibility.

[10:07](https://www.youtube.com/watch?v=72y2EC5fkcE&t=607s) Portable Recording Sessions and Build System Integration

Allan Blomquist explains that recording session files are not merely produced by programmers; everyone across the entire studio team generates them routinely during development. These session packages are completely portable across machines and roles. If a level designer is tuning a puzzle and notices an unexpected behavior or encounters a random visual anomaly, they can instantly package the recorded session and send it to anyone else at the studio to be replayed live and debugged under full instrumentation.

Crucially, this system is tightly coupled with Tomorrow Corporation’s custom build pipeline and source control infrastructure. The tool guarantees that a session will replay deterministically on another developer's rig regardless of divergence in the underlying codebases:

```mermaid flowchart TD accTitle: Portable Session Debugging Architecture accDescr: Flow showing how a session recorded with local changes can be cleanly loaded on another developer machine without file conflicts.

subgraph Sender["Designer / Originating Machine"] A["Designer records session (contains uncommitted local edits)"] --> B["Portable Session Bundle (inputs, state, build artifacts)"] end

subgraph Receiver["Developer / Debugging Machine"] B --> C["Tomorrow Corp Build System"] C --> D["Isolates required assets & source revision"] D --> E["Live Deterministic Playback (Local files untouched & safe)"] end ```

The system retrieves and matches all exact dependencies, binaries, and state data required to execute the run without disturbing the recipient's working copy. Allan Blomquist underscores the safety and convenience of this design: “The system can get all the data it needs to play back the session and it doesn't touch any of my actual local files in the process so there isn't any risk of me losing work if something went wrong.”

- **Cross-discipline usage**: Designers, artists, and testers generate the identical deterministic recordings used by engine programmers. - **Revision tolerance**: Sessions remain functional even if the originating user had dirty uncommitted modifications and the recipient is sitting on a completely different source control commit with their own uncommitted edits. - **Local file isolation**: Playback environments virtualize the necessary dependencies so no active work in progress on the host machine is overwritten or dirtied.

---

[10:51](https://www.youtube.com/watch?v=72y2EC5fkcE&t=651s) Replaying a Crash and Navigating Backwards in the Call Stack

Blomquist loads a pre-packaged session recorded by a teammate whose build terminated abnormally with a fatal crash. Running the replay directly to its terminus reproduces an unhandled memory access violation. On the screen, the custom crash handler surfaces the distinctive supervisor character from *Human Resource Machine*, prompting Allan Blomquist to joke: “And you can see that the boss from Human Resource Machine lives on our Panic screen and he's not happy with me right now.”

Instead of staring at a static post-mortem crash dump, Blomquist leverages the timeline slider to scrub back through the user's run to observe the exact sequence of player actions that led to the failure. He watches the player successfully complete a level, return to the code workspace to optimize their assembly-like solution, run the level a second time, and immediately trigger the crash.

Blomquist clicks the top frame of the call stack, jumping directly from the interactive playback into the integrated source code editor right at the failing line. Recognizing that the access violation was caused by dereferencing a null pointer that had been passed into the function as a parameter, he executes a reverse step-out instruction to step execution back out of the function call to the callsite. There, he inspects the surrounding conditional block, which was supposed to assign that pointer depending on whether the game was executing in normal play mode or phone play mode.

Examining the conditional state exposes an immediate problem: the `playMode` variable holds an invalid integer value that does not correspond to any valid enumerator in the enumeration declaration. When he inspects the raw binary bit pattern of this corrupted variable, Blomquist recognizes the IEEE 754 bit layout for the floating-point value `1.0f`.

---

[12:31](https://www.youtube.com/watch?v=72y2EC5fkcE&t=751s) Reverse Data Breakpoints and Catching Memory Stomps

A corrupted variable displaying a floating-point pattern inside an integer enum field typically indicates an out-of-bounds write or memory stomp—one of the most dreaded bugs in native game programming. Allan Blomquist quips about the traditional despair such issues cause: “So at this point you're thinking this looks like a memory stomp and I better cancel my plans because who long who knows how long it's going to take me to figure this out.”

Instead of guessing or manually narrowing down writes, Blomquist drags the corrupted `playMode` variable directly into his debugger watch window and sets a reverse data breakpoint (watchpoint) directly on that memory address. He commands the execution engine to run backwards in time. The time-travel debugger instantly halts at the precise instruction that executed the offending memory write:

```mermaid flowchart LR accTitle: Buffer Overflow Memory Stomp Sequence accDescr: Sequence showing how an off-by-one bounds check caused a float write to stomp an adjacent playMode enum.

A["Buffer write index starts at 15"] --> B["Index increments to 16"] B --> C["Comparison checks: index > capacity (16)"] C -->|"Flawed check evaluates false"| D["Write float 1.0f past buffer bounds"] D --> E["Stomps memory address of playMode enum"] ```

The write instruction is an assignment writing `1.0` into a floating-point buffer. Stepping backward a few instructions before the assignment exposes an off-by-one bounds check error: - The buffer write index was initialized at `15`. - The index was incremented to `16`. - The bounds logic checked if the index was strictly greater than the buffer capacity of `16` (`index > capacity`), rather than checking if it was greater than or equal to the capacity (`index >= capacity`). - Because `16 > 16` evaluated to false, the index failed to wrap back around to zero, causing the code to write past the end of the array directly into the adjacent memory where the `playMode` enum resided.

Blomquist acknowledges that while this specific bug was contrived for the demonstration, real-world engine development is plagued by similar memory corruptions. He reflects on how painful such problems are with traditional toolchains: “And you can imagine that if I were attacking this with printfs or something instead I might be in for kind of a long night.” He notes that most everyday bugs do not even reach this level of complexity—they are simply mundane logic errors in gameplay scripts—making them trivial to isolate and resolve using deterministic time-travel replay tools. Blomquist then concludes the presentation by thanking the viewers and pointing them to Tomorrow Corporation's website for further updates.

## Notable quotes > [01:40](https://www.youtube.com/watch?v=72y2EC5fkcE&t=100s) And being able to debug backwards from an assert or a crash especially with a data break point is insanely helpful in being able to figure out and fix bugs. — Allan Blomquist

## Mentioned Allan Blomquist, Tomorrow Corporation, Human Resource Machine, GOAL (Game Oriented Assembly Lisp), Naughty Dog, PlayStation 2, IEEE 754

## Video description A demonstration of some of the internal programming and debugging tools we use at Tomorrow Corporation to make video games.

Our company website: https://tomorrowcorporation.com

My personal twitter: https://twitter.com/allan_blomquist

[Source](https://www.youtube.com/watch?v=72y2EC5fkcE)

中文简报

Tomorrow Corp 推出支持逆向时间调试的自研技术栈统一语言、编译器和引擎，将确定性直接交由开发环境处理，使程序员无需编写防御性代码即可逆向追溯时间，从而隔离内存损坏和崩溃。 Tomorrow Corporation 引擎程序员 Allan Blomquist 展示了该工作室定制的垂直整合技术栈，其中包括定制的编程语言、即时增量原生编译器、代码编辑器、调试器和引擎。通过《人力资源机器》（Human Resource Machine）的移植版本，他演示了按键级的热重编译、调试器单步执行期间的实时帧中渲染，以及与紧凑的 64 位时间戳相关联的动态调用图分析。该演示重点展示了跨代码和运行状态的统一确定性会话拖动回溯，以及便携式会话记录，使团队成员能够无缝复现崩溃，并利用逆向数据断点解决令人头疼的内存覆盖问题。 Tomorrow Corporation 开发了一套完全定制的全栈游戏开发环境——包括其自研的编程语言、增量编译器、编辑器、调试器和引擎——其灵感源自顽皮狗（Naughty Dog）PS2 时代的 GOAL 工具链。 (0:00) 该定制编译器在每次按键时都会生成原生机器码，在处理 13,000 行的文件时感知延迟几乎为零，同时支持无需特殊构建配置的常开逆向执行。 (1:40) 紧凑的 64 位内置时间戳原语允许开发人员对实体和内存分配进行唯一标记，使调试器能够直接传送回创建该对象时的精确机器状态。 (4:11) 确定性会话重放可在热重载资源和重编译过程中实时重新计算模拟和渲染，并可在整个工作室范围内共享，而不会弄脏接收者的本地工作区。 (7:33) 结合逆向数据断点的时间旅行调试使开发人员能够立即查明差一错误（off-by-one）缓冲区溢出和内存覆盖的根本原因，而无需通过反复试验的日志记录来复现它们。 (12:31)

Reading summary

Allan Blomquist demonstrates Tomorrow Corporation’s proprietary, vertically integrated development stack, arguing that unifying the programming language, compiler, engine, editor, and debugger eliminates common development dead ends through instant keystroke compilation, native time-travel debugging, and fully deterministic session replay. Full-Stack Integration and Keystroke Compilation (0:00) Tomorrow Corporation spent over a decade developing an in-house toolchain spanning a custom programming language, compiler, editor, debugger, build system, and game engine—an ambition Allan Blomquist traces to Naughty Dog’s GOAL toolchain on the PlayStation 2. Tested via a port of Human Resource Machine, the environment is engineered to support general game development rather than isolated toy examples. The stack’s tight coupling enables the incremental compiler to emit native machine code on every keystroke with near-zero latency, even in 13,000-line source files. Because the custom debugger directly drives the engine runtime, breakpoints freeze the frame mid-execution without killing the graphics pipeline. Developers can step through draw calls one instruction at a time to watch geometry and UI elements render sequentially, functioning as a real-time graphics inspector. Always-On Reverse Execution and Intrinsic Timestamps (4:11) Reverse debugging is architected directly into everyday builds without separate configurations or runtime overhead. Developers can step backward into, over, or out of functions, or run execution backward to execution and hardware data breakpoints. Blomquist highlights this backwards capability as particularly decisive when debugging crashes and assertion failures from the failure point. To trace program state, the custom language exposes a cheap 64-bit intrinsic timestamp capturing full machine state. Attaching these tags to entities or allocations allows developers to jump the debugger backward directly to the exact instruction that created an item. Dynamic profiling works on the same principle: enabling performance monitoring instantly recompiles the binary with call-graph instrumentation injected on the fly, allowing developers to click a timeline spike and teleport execution to that precise frame invocation. Whole-Session Deterministic Replay and Historical Scrubbing (7:33) An integrated timeline console allows developers to scrub across an entire play session, recalculating live simulation and rendering passes on demand rather than playing back recorded video. The timeline preserves continuity across live code edits, asset hot-reloads, and compiler rebuilds. Blomquist emphasizes that the architecture was deliberately built without 'one-way streets,' ensuring one tool's use never disables another. Because the code editor is synchronized with session time, scrubbing to an earlier moment causes the editor to display the source code as it existed at that instant. This allows retroactive profiling, where unexpected hitches during playtests can be rewound, inspected, and debugged without manual reproduction. By embedding determinism directly across the compiler, engine, and language levels, the runtime automatically serializes compact session files into version control without requiring developers to write defensive synchronization logic. Portable Sessions and Rapid Root-Cause Analysis (10:07) Session files are portable across disciplines, enabling designers and artists to record bugs and send them to programmers for bit-accurate reproduction. Tied into the custom build system, the replay runner isolates required assets and code revisions without disturbing or overwriting any uncommitted local files on the recipient’s workstation. Blomquist demonstrates this workflow on a recorded memory violation where a playMode enum variable was corrupted with the IEEE 754 bit pattern for 1.0f. Setting a reverse data breakpoint on the variable’s memory address and running execution backward immediately isolated the responsible write: an off-by-one array bounds check (evaluating index > capacity instead of index >= capacity) allowed a floating-point buffer write at index 16 to overwrite the adjacent enum. Blomquist notes that while such memory corruptions traditionally require hours of speculation, time-travel tools make both complex memory stomps and routine scripting errors straightforward to resolve. Tomorrow Corporation’s presentation demonstrates that full-stack vertical integration can resolve persistent game development frictions like slow iteration loops and nondeterministic bugs. However, Blomquist notes this environment required a multi-year engineering commitment as a dedicated passion project, highlighting the massive upfront investment needed to replace standard commercial tooling.

精炼阅读摘要

独立工作室 Tomorrow Corporation 的底层引擎程序员 Allan Blomquist 展示了团队历时十余年研发的自研技术栈，通过深度集成自定义语言、增量编译器、引擎与时间旅行调试器，实现了极低延迟的代码热重载与全确定性可逆调试。自研技术栈愿景与即时编译反馈 (0:00) Blomquist 指出，该项目的灵感源自 Naughty Dog 在 PlayStation 2 时代为《Jak and Daxter》系列开发的 GOAL 工具链。Tomorrow Corporation 并未选用现成的商业引擎或通用 IDE，而是自主搭建了一整套工具链，覆盖自研编程语言、编译器前端与后端、代码编辑器、调试器、构建系统以及游戏引擎。为了证明技术栈的通用性与工业可用性，演示基于该工具链移植的游戏《Human Resource Machine》展开。在日常迭代循环中，该工具链不仅支持基础资产的即时热重载，还将该机制扩展到可执行机器码级别。系统搭载的增量编译器速度极快，即便在包含约 13,000 行源代码的文件中进行修改，也能在每次击键时触发原生机器码的重新编译，游戏运行画面的响应近乎零感知延迟。与此同时，调试器与渲染管线保持深度协同，当开发者在绘制调用入口设置断点时，视口维持实时激活，开发者可逐条指令步进并观察每个图形元素的渲染过程，如同内置的图形管线分析工具。时间戳剖析与64位状态内建溯源 (4:11) 传统分析工具往往需要在速度与数据完整度之间妥协，且通常只能跳转至静态源码。Tomorrow Corporation 的内置性能分析器默认剥离插桩代码以保证运行效率，但当开发者点击启用时，编译器会在运行时无缝重新编译并在调用图中动态注入插桩逻辑。若遇到帧耗时异常抖动，开发者可以直接在分析器时间线中点击具体时间戳，调试器便会直接逆向跳转至该特定帧的机器执行状态与调用现场，供开发者直接检查局部变量并排查性能瓶颈。此外，自研语言提供了一个低开销的内建指令（intrinsic），该指令在运行时返回一个紧凑的 64 位执行时间戳。开发者可将此 64 位时间戳附加到任意游戏实体、内存分配或事件结构体中，以极小的内存与性能代价记录其完整的历史状态。在演示中，Blomquist 通过读取传送带上一个箱子结构体内的 64 位时间戳，调试器即刻逆向跳转至角色将该箱子抛掷到传送带上的确切历史瞬间，极大简化了实体生命周期与内存泄漏的排查链路。全会话确定性重放与历史代码时间旅行 (7:33) 该系统在游戏控制台中内置了覆盖整个运行周期的完整时间线滑块。与单纯记录视频帧的机制截然不同，该滑块通过重新计算底层的游戏模拟与渲染管线来实现历史回溯。这项确定性回放机制能够完全兼容会话期间发生的所有操作，包括纹理热更新、代码修改以及为性能分析而触发的编译器重构。Blomquist 强调，整套架构消除了技术上的“单向死胡同”，前序操作不会限制后续工具的使用。当时间线被拖拽至修改代码之前的历史帧时，代码编辑器也会联动展示对应历史时间点的代码文本状态，实现了源码与运行时状态的协同历史回溯。由于语言、编译器与引擎全链路协同设计，确定性保障被直接下沉到基础设施层，开发者无需编写防御性代码防止状态解耦。每次游戏运行都会自动生成一份极小体积的轻量化会话文件，并与版本控制系统深度绑定，以极低开销实现长期、位精确（bit-accurate）的运行重放。跨职能便携会话与内存越界逆向排查 (10:07) 录制的会话文件不仅供程序员使用，策划、美术和测试人员在日常开发中遇到异常或崩溃时，均可打包会话文件分发给其他成员。配合工作室的自研构建系统，接收方可以在不污染或覆盖本地工作目录代码的前提下，沙盒化加载会话所需的依赖、资源与源码版本，确保在代码库存在分歧或未提交修改时依然能够百分之百确定性重现崩溃现场。在演示队友提交的崩溃会话时，游戏在执行非法内存访问后触发带有《Human Resource Machine》老板角色的自定义崩溃界面。Blomquist 利用时间旅行功能逆向单步跳出调用栈，发现用于分支判断的整数枚举变量 `playMode` 被赋予了非法值，其底层二进制呈现出浮点数 1.0f 的 IEEE 754 编码特征，表明发生了原生开发中极难排查的内存覆写（memory stomp）。Blomquist 随即针对该内存地址设置逆向硬件数据断点并反向运行，系统瞬间精准定位到因数组越界检查逻辑错误（误用 `index > capacity` 而非 `>=`）导致浮点数写入相邻枚举内存的指令位置，无需依赖传统排查手段中耗时耗力的日志打印或猜测复现。 Tomorrow Corporation 展示了一套将语言、编译、引擎与全确定性可逆调试完全垂直整合的技术范式，大幅压缩了游戏开发中资产热更、性能追踪及疑难内存错误的排查耗时。不过，该演示主要基于工作室自身特定体量与技术栈的移植项目展开，此类高度定制的全栈方案在向复杂工业化管线或第三方商业引擎生态泛化时，仍面临极高的前期自研与维护成本。

Detailed notes — zh

## Background 该视频由 YouTube 频道 retrogameinternals（Tomorrow Corporation 引擎兼底层程序员 Allan Blomquist）于 2023 年 2 月 21 日发布。视频简介说明，这是一次关于 Tomorrow Corporation 内部自研游戏编程与调试工具链的深度技术演示。 ## TL;DR 视频展示了 Tomorrow Corporation 耗时数年自研的完整软件技术栈，包含专属编程语言、增量编译器、编辑器、时间旅行调试器及游戏引擎。演示以移植版《人力资源机器》为实际载体，展现了毫秒级的逐按键代码热重载、图形同步单步渲染，以及随时可用的对称反向调试和 64 位时间戳状态追踪。Allan 进一步演示了无损确定性时间轴滑块、跨团队成员且隔离本地代码修改的会话文件回放机制，并通过一个典型的崩溃与内存踩踏（memory stomp）实战案例，展现了通过反向数据断点瞬间定位越界写入 Bug 的惊人排错效率。 ## Atmosphere & style 整场演示洋溢着资深系统程序员纯粹的极客浪漫与从容笃定。Allan 的讲解沉稳且极富条理，既有回望顽皮狗 GOAL 工具链时那份不忘初心的憧憬，又带着对自己亲手打造的“心血之作”的平静自豪。观看过程像是在见证一场软件工程层面的降维打击：以往令人头疼的崩溃排查与内存破坏调试，在如同视频进度条般丝滑穿梭的确定性时间轴下被轻巧化解，带给观众极强的掌控感与大开眼界的技术震撼。 ## Bulletin - [01:40](https://www.youtube.com/watch?v=72y2EC5fkcE&t=100s) Tomorrow Corporation 展示了一套涵盖自研编程语言、编译器、IDE、调试器和引擎的全自研软件栈，其增量编译器可在数万行代码中做到“按键级”（every keystroke）原生机器码实时热重载。 - [02:51](https://www.youtube.com/watch?v=72y2EC5fkcE&t=171s) 团队将全生命周期的常态化反向调试（Reverse Debugging）与图形渲染管线结合，允许逐行单步执行时实时刷新帧缓冲，并支持利用硬件数据断点逆向溯源内存踩踏缺陷。 - [05:40](https://www.youtube.com/watch?v=72y2EC5fkcE&t=340s) 系统支持用轻量级 64 位时间戳代替庞大的调用栈来标记游戏事件，配合确定性时间轴滑块可秒级还原任意历史运行状态及当时对应的源代码版本。 - [10:07](https://www.youtube.com/watch?v=72y2EC5fkcE&t=607s) 团队跨职能共享轻量会话文件（Session File），接收方即便处于不同代码分支且有本地未提交修改，也能在零污染本地环境的前提下精准复现并反向排查崩溃。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=72y2EC5fkcE&t=0s) 技术栈开发灵感源自顽皮狗（Naughty Dog）为 PS2 打造的专用工具链 GOAL，团队历经四五年构建了从语言到底层的全自研工具体系。 - [00:00](https://www.youtube.com/watch?v=72y2EC5fkcE&t=0s) 采用完整的商业游戏《人力资源机器》（Human Resource Machine）进行技术演示，证明该工具链适用于复杂工业级项目而非玩具样例。 - [01:40](https://www.youtube.com/watch?v=72y2EC5fkcE&t=100s) 极速增量编译器使得即便面对上万行的单一源代码文件，系统也能在每次键盘敲击时直接编译生效，提供极致的热重载体验。 - [02:51](https://www.youtube.com/watch?v=72y2EC5fkcE&t=171s) 断点调试与游戏引擎画面渲染完全同步，逐行单步执行时画面元素实时上屏，可直接充当轻量级图形调试器。 - [02:51](https://www.youtube.com/watch?v=72y2EC5fkcE&t=171s) 反向调试能力（反向步入、步过、步出及反向运行至断点）作为核心机制默认常态化开启，无须进入特殊的性能损耗录制模式。 - [04:11](https://www.youtube.com/watch?v=72y2EC5fkcE&t=251s) 游戏内性能分析器可毫秒级完成全量代码插桩重编，且允许开发者直接点击性能图表尖刺的时间戳瞬间穿越回事故调用的发生点。 - [05:40](https://www.youtube.com/watch?v=72y2EC5fkcE&t=340s) 自研语言内建 64 位执行时间戳机制，能以极小开销标记实体生成、受击等全套状态，取代传统的日志与深层调用栈抓取。 - [07:33](https://www.youtube.com/watch?v=72y2EC5fkcE&t=453s) 时间轴滑块基于底层输入流与确定性（Determinism）按需实时重算渲染，并非简单的视频录制回放。 - [07:33](https://www.youtube.com/watch?v=72y2EC5fkcE&t=453s) 时间轴与编辑器具备状态感知双向同步机制，拖动时间轴回到过去时，代码编辑器亦会自动回滚至当时对应的历史源代码。 - [07:33](https://www.youtube.com/watch?v=72y2EC5fkcE&t=453s) 底层语言与编译器原生保证确定性执行，彻底免除普通确定性回放框架中对内存对齐、随机数维护等繁琐心智负担。 - [10:07](https://www.youtube.com/watch?v=72y2EC5fkcE&t=607s) 可移植会话文件（Session File）与版本控制深度结合，允许非技术人员生成复现包，并在不同分支的开发机上无损安全调试。 - [10:51](https://www.youtube.com/watch?v=72y2EC5fkcE&t=651s) 通过反向跳出函数（Reverse Step Out）可以在程序抛出崩溃（如空指针异常）时，瞬间倒退回调用处以查证传参根源。 - [12:05](https://www.youtube.com/watch?v=72y2EC5fkcE&t=725s) 结合硬件数据断点与逆向执行，能瞬时锁定导致内存踩踏（Memory Stomp）的越界写语句，将数小时甚至通宵的排错时间缩减至数秒。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=72y2EC5fkcE&t=0s) 自研工具链的缘起与演示背景 Allan Blomquist 介绍了自己的身份——独立游戏工作室 Tomorrow Corporation（明日公司）的引擎及底层程序员。团队十多年来一直在开发游戏以及制作游戏所使用的各类内部工具。如今，他们已经拥有了一套完全自主掌控的软件栈，包括自研编程语言、编译器前端与后端、代码编辑器、调试器、构建系统以及游戏引擎。谈及为何会投身于整套开发环境的研发，Allan 回忆起很多年前读过的一篇博文。那篇文章介绍了顽皮狗（Naughty Dog）为 PS2 游戏打造的专属编程工具链 GOAL（Game Oriented Assembly Lisp）。他感叹道：“I remember reading about that and just thinking that this is obviously like the coolest thing that I've ever heard of it didn't seem like something that you could even do”（我记得读到那篇介绍时，心想这显然是我听过的最酷的事情，那甚至看起来都不像是你能做到的事）。这段经历给他留下了极其深刻的印象，也让他反思自己当时的想象力匮乏。大约四五年前，他终于迎来了在自家公司亲手实现类似系统的契机，这项工作随即变成了他的心血之作（passion project）。为了展现这套环境在工业级开发中的威力，Allan 选用了一款真实完整的作品——《人力资源机器》（Human Resource Machine）的自研工具移植版。尽管原版不是在这套体系下写成的，且游戏题材恰好与编程相关，但 Allan 强调这套工具是通用的底层环境，适用于公司的任何一款游戏，选用成品游戏是为了避免陷入玩具式样例（toy example）的局限。 ```mermaid flowchart TD accTitle: Tomorrow Corporation 自研技术栈结构 accDescr: 展示明日公司完全自研的软件技术栈，包含从上层应用到编译器和调试器的完整自研链路。 subgraph Toolchain["底层工具链与语言"] PL["自研编程语言"] Comp["自研编译器 (前端与后端)"] Build["自研构建系统"] end subgraph Environment["开发与调试环境"] IDE["自研代码编辑器"] Dbg["自研时间旅行调试器"] Prof["自研内置性能分析器"] end subgraph Runtime["运行时与游戏"] Engine["自研游戏引擎"] Game["游戏项目 (如 Human Resource Machine)"] end PL --> Comp Comp --> Build Build --> IDE IDE <--> Dbg Dbg <--> Engine Prof <--> Engine Engine --> Game ``` ### [01:40](https://www.youtube.com/watch?v=72y2EC5fkcE&t=100s) 资源热重载与逐按键代码热重载演示从最基本的运行时特性开始。Allan 首先展示了资源热重载（hot load），也是整场演示中唯一涉及非代码美术操作的环节：他在外部工具中随手修改并保存了一张游戏贴图，游戏画面立刻无缝呈现改动，完全不需要停止或重启进程。随后，Allan 打开自研代码编辑器，将热加载推进到核心的代码层面。他找到了屏幕左侧绘制绿色方块的底层逻辑并直接进行改动。由于编译器采用了极高效率的增量编译设计，且生成原生机器码的速度极快，系统甚至可以在“每一次按键”（every keystroke）触发时完成编译并生效。Allan 指出，即便是面对一个长达 13,000 行源码的庞大单文件，编辑器键入字符到游戏窗口画面刷新之间也几乎感受不到任何延迟。 ### [02:51](https://www.youtube.com/watch?v=72y2EC5fkcE&t=171s) 图形同步单步调试与全功能反向调试在展示地面蓝色方块的绘制代码时，Allan 演示了自研调试器与游戏引擎紧密结合带来的独特优势。当他在绘制函数顶部打上断点时，游戏画面呈现出部分已绘制的状态；随着单步执行（step over）逐行前进，屏幕上的元素逐一绘制上屏。由于调试器与游戏引擎完全协同，系统能够在单步调试代码的同时保持屏幕画面的实时活性，宛如一个轻量级的图形调试器（graphics debugger），极易用于排查屏幕上某个杂乱像素或异常元素究竟源自哪一行代码。紧接着，Allan 展示了逆向时间轴的调试能力：调试器不仅能正向单步，还支持完全对称的反向单步调试（step backwards）。常规的步入、单步跳过、跳出都具备对应的倒退版本；开发者可以反向运行到断点，甚至能反向运行到硬件数据断点（Hardware data breakpoints）。Allan 强调，在遭遇断言失败（assert）或崩溃时，这种能力尤为强大： > “and being able to debug backwards from an assert or a crash especially with a data break point is insanely helpful in being able to figure out and fix bugs” > —— Allan Blomquist 极为关键的一点是，这种反向调试并非在某种特殊的“录制模式”或专门的调试启动项下才生效，它始终在后台无缝运转，日常编码时随时可用。 - **断点同步渲染**：逐行执行代码的同时实时刷新帧缓冲区，可视化绘制过程。 - **全套反向控制**：支持反向单步跳入（step in）、跳过（step over）、跳出（step out）。 - **硬件数据反向断点**：追踪内存被修改的历史，直接逆向定位触发越界或脏数据的根源。 - **常态化可用**：非专用 profile/debug 重启模式，全生命周期默认开启。 ### [04:11](https://www.youtube.com/watch?v=72y2EC5fkcE&t=251s) 性能分析器联动：逆流回到耗时调用的发生瞬间 Allan 切换至游戏窗口，演示了内置的性能分析工具（in-game profiler）。默认状态下，用于性能度量的插桩代码（instrumentation）并不会被编入二进制中；但只要在界面上点击启用，系统便会即时触发对游戏代码的完整重编译，毫秒间完成调用图插桩（call graph instrumentation）并采集数据。在分析器图表中出现帧耗时尖刺（spikes）后，Allan 暂停了数据收集以深入排查。在一帧耗时极高的调用中，某个特定函数占用了大部分计算时间。常规性能分析器仅支持跳转到该函数的源码文本，但这套系统借助反向调试技术，允许开发者直接点击界面上的时间戳（timestamp）。点击瞬间，调试器直接穿梭回到了那一帧中该函数被实际调用的具体时刻。单步排查后发现，耗时原因仅仅是由于随机数生成器摇出了一个倒霉的极端随机值。修改并移除了不良代码后，性能图表恢复平稳，随后他一键重新编译剔除插桩逻辑，继续流畅运行。 ### [05:40](https://www.youtube.com/watch?v=72y2EC5fkcE&t=340s) 64 位时间戳的轻量标记与时光倒流排查除了性能分析器自动打上的时间戳外，这套自研语言还为开发者提供了底层内建机制（intrinsic），可以在运行时直接求得当前执行点的 64 位执行时间戳（execution timestamp）。在传统开发中，动态分配资源时为了防范内存泄漏，往往需要通过宏或元数据记录文件名、行号甚至是调用栈；而现在，只需使用一个轻量级的 64 位整数时间戳即可取代沉重的调用栈追踪。为了验证这一概念，Allan 暂停游戏，调出了关卡传送带上用于解谜的方块数据。数组中记录了投入传送带的三个方块，每个方块都绑定了一个 64 位时间戳。Allan 选取了中间方块的时间戳并执行“回到该时刻”，调试器瞬间将整个运行环境还原至主角把那个方块抛上传送带的确切历史状态；执行撤销后，又能随时切换至另一个方块的抛出瞬间。Allan 指出，这一范式不仅限于方块解谜，它几乎能无成本地标记一切游戏事件： > “these time stamps are super cheap to generate and they're only 64 bits so you can use them to tag pretty much everything with what effectively is the entire state of the program and then you can go back later and inspect anything you need to if you want to find out where something came from or why it didn't get cleaned up or whatever you want to know” > —— Allan Blomquist 无论是敌人怪物的生成时刻、玩家最后一次受到攻击的瞬间，还是走过的每一扇门的历史轨迹，都能以 64 位的时间戳指代当时全套的程序状态。 ### [07:33](https://www.youtube.com/watch?v=72y2EC5fkcE&t=453s) 确定性时间轴滑块与无锁步协同开发在展示的最后阶段，Allan 将这套系统的确定性（determinism）机制推向极致。调出游戏内的开发者控制台后，屏幕底部出现了一条时间轴滑块（scrubber）及播放控制栏。这条时间轴完整收录了他在整场演示期间执行过的所有操作，能够自由向前、向后拖动以检查任意历史环节。Allan 强调，这绝非录像回放，而是系统根据输入流在本地按需（on demand）实时进行的游戏物理模拟和底层图形渲染。更不可思议的是，即使在演示过程中发生了彻底重新编译代码启用分析器、动态修改方块代码、热重载美术贴图等多重破坏性更改，时间轴滑块依然能在这些状态跨越中自由来回拖拽。整个工具链在架构之初就彻底规避了“单行道”（one-way streets）的设计陷阱： - **状态感知编辑器**：当滑块拖动到方块代码修改之前的节点时，编辑器中展示的代码也会立刻回退到修改前的历史源码，确保开发者调试的数据与当时的源代码绝对匹配。 - **事后性能诊断（post-facto profiling）**：如果在正常游玩中偶然察觉到画面顿挫或卡顿，开发者无须绞尽脑汁寻找复现步骤，只需把滑块拖回卡顿发生的点位，即可直接在事故现场运行性能分析。 - **零心智负担的确定性开发**：市面上基于确定性的重放框架通常极为脆弱，需要程序员小心翼翼维护内存对齐和伪随机状态。然而在这套深度整合的架构中，系统从底层语言、编译器到引擎全程接管并保证确定性，开发者可以任意编写代码，无须担心破坏确定性机制。 - **微型版本控制会话文件**：游戏每次运行都会自动保存一份伴随该时间轴的会话文件（session file）；结合自研版本控制系统的紧密集成，系统仅需存储极少量增量数据，即可保证该时间轴在未来的任何时间节点都能稳定无误地重新回放。 ### [10:07](https://www.youtube.com/watch?v=72y2EC5fkcE&t=607s) 可移植会话文件与版本控制系统的无缝集成在团队日常协作中，生成这些会话（session）录制文件的人员并不仅限于程序员，整个团队的成员都在使用这项功能。会话文件具备极高的可移植性：例如关卡设计师在设计关卡时，若在游戏实测中发现某些机制运作异常，或者随机目睹了某种奇怪的 Bug，只需将会话文件直接发送给团队中的任何其他人，接收方就能立刻进行实时回放并直接切入调试。这套体系与工作室的构建系统深度整合，从而消除了不同人员本地开发环境不一致所带来的阻碍。即便发送者在创建会话时存在尚未提交的本地代码修改，而接收者在版本控制系统（Source Control）中处于完全不同的版本分支且各自保留着未提交的本地修改，系统也能自动抓取回放所需的所有历史数据并正常复现。最关键的是，整个过程完全不会触碰或覆盖调试者本地的实际文件，彻底避免了因调试他人录像而导致自身工作进度丢失的风险。 ```mermaid flowchart TD accTitle: 会话文件共享与回放流程 accDescr: 展示团队成员之间传递会话文件并在不同本地代码状态下无损回放的机制。 subgraph Senders["生成端（如设计师）"] A["游戏测试出现异常"] --> B["生成便携会话文件 (Session File)"] B --> C["包含本地修改与特定版本数据"] end subgraph System["构建系统与版本管理"] C --> D["构建系统自动提取所需历史数据"] D --> E["隔离环境重构运行上下文"] end subgraph Receivers["接收端（如程序员）"] E --> F["直接实时回放与时间旅行调试"] F --> G["完全不影响本地未提交代码"] end ``` ### [10:51](https://www.youtube.com/watch?v=72y2EC5fkcE&t=651s) 重现崩溃现场与逆向单步回溯为了演示具体的排错流程，Allan Blomquist 加载了一个提前录制好的崩溃会话文件，模拟同事因为游戏发生崩溃而向他求助的场景。当他将录像快进运行到最末尾时，程序果然抛出了访问违规（Access Violation）错误。此时游戏画面切换到了专门的报错崩溃画面，上面赫然印着来自《人力资源机器》（Human Resource Machine）的老板头像，Allan Blomquist 幽默地调侃道：“可以看到来自《人力资源机器》的老板就住在我们的崩溃界面上，而且他现在显然对我不太满意。” 通过在时间轴上倒退查看，Allan Blomquist 梳理出了玩家的操作轨迹：测试者先通关了一次关卡，随后返回并对自己的解法逻辑进行了一番优化调整，再次运行关卡时便触发了崩溃。Allan Blomquist 点击调用栈（Call Stack）的栈顶直接跳入代码编辑器。针对报错行发生访问违规的情况，他初步推测传入的指针可能为 null。由于该指针是当前函数的输入形参，他执行了一次逆向跳出函数（reverse step out）的操作，程序瞬间倒退回进入该函数之前的调用点。现场果然证实了这一猜测：在调用该函数时，传入的就是一个 null 空指针。 - **崩溃场景复现**：玩家在首次通关后回溯并优化代码解法，二次运行导致游戏抛出访问违规崩溃。 - **逆向单步追踪**：通过反向跳出函数，直接捕获了上层调用在传参时传入空指针的明确证据。 ### [12:05](https://www.youtube.com/watch?v=72y2EC5fkcE&t=725s) 数据断点逆向追踪“内存踩踏”引发的隐蔽 Bug 顺着代码调用逻辑向上排查，该指针原本应该在一个 `if` 条件分支中根据当前处于普通游玩模式还是手机模式来完成初始化赋值。然而，当检查当前的模式变量 `playMode` 时，调试器却提示该变量所存储的值根本不是该枚举（enumeration）中的合法成员。深入观察其底层的二进制比特位模式，Allan Blomquist 敏锐地认出该数值实际上代表着浮点数 `1.0`。面对这种典型的内存被破坏现象，Allan Blomquist 感慨道：“此时你肯定会想，这看起来是一次内存踩踏（Memory Stomp），我最好把今晚的私人安排都取消掉，因为天晓得我得花多长时间才能查清楚这种问题。” 但这套调试系统让解决内存踩踏变得异常高效。Allan Blomquist 将 `playMode` 变量直接拖入监视窗口（Watch Window），并对其精准设置了一个数据断点（Data Breakpoint）。随后他下达了反向执行指令，调试器在录像的时间线上逆向搜索，瞬间锁定了最后一次向该内存地址写入数据的具体代码位置： 1. **定位非法写入点**：反向命中了一条将浮点值 `1.0` 赋值写入某缓冲区的语句，完美解释了枚举变量为何会包含浮点数 `1.0` 的位模式。 2. **审查缓冲区边界逻辑**：再往前回溯数步，发现写入该缓冲区的数组索引初始为 `15`，递增到 `16` 后与缓冲区的容量（同样是 `16`）进行校验。 3. **确认边界判断缺陷**：按照预期设计，当索引达到容量时应当环绕回绕（wrap around），但逻辑判断错误地写成了 `>` 而非 `>=`，导致越界写入并破坏了相邻内存中的枚举变量。 Allan Blomquist 总结道，虽然这是一个刻意构造的演示用例，但在实际开发中这种内存破坏 Bug 绝非罕见；如果开发者只能依赖加打印语句（`printf`）这类传统手段去排查，“那你可以想象免不了一场通宵奋战了”。好在游戏开发中绝大多数缺陷根本没有这么复杂，通常只是玩法逻辑代码中的常规失误，而借助这套时间旅行调试工具，定位和修复所有这些问题都会变得极其轻松。最后，Allan Blomquist 感谢了观众的收看，并引导大家通过简介中的链接进一步了解 Tomorrow Corporation 的技术理念。 ## Notable quotes > [02:51](https://www.youtube.com/watch?v=72y2EC5fkcE&t=171s) and being able to debug backwards from an assert or a crash especially with a data break point is insanely helpful in being able to figure out and fix bugs — Allan Blomquist ## Mentioned Allan Blomquist, Tomorrow Corporation, Naughty Dog, GOAL, Human Resource Machine, PlayStation 2 ## Video description A demonstration of some of the internal programming and debugging tools we use at Tomorrow Corporation to make video games. Our company website: https://tomorrowcorporation.com My personal twitter: https://twitter.com/allan_blomquist [Source](https://www.youtube.com/watch?v=72y2EC5fkcE)

Source quotes

[2:51] Allan Blomquist: and being able to debug backwards from an assert or a crash especially with a data break point is insanely helpful in being able to figure out and fix bugs

Source quotes

[1:40] Allan Blomquist: And being able to debug backwards from an assert or a crash especially with a data break point is insanely helpful in being able to figure out and fix bugs.

Original transcript

[0:00] hello everybody my name is Alan [0:03] Blomquist I'm the engine and sort of the [0:05] low-level programmer at indiegam [0:07] developer Tamar Corporation and today [0:10] I'm going to do a quick demo of some of [0:11] our internal tools and specifically [0:13] focus on programming and debugging [0:17] we've been developing games and a lot of [0:19] the tools that we use to make them for [0:20] over a decade now and at this point [0:22] we're using our own programming language [0:24] compiler front end and back-end editor [0:27] debugger build system and game engine [0:33] I got really interested in working on [0:34] stuff like this when I read years and [0:37] years ago it must have been a blog post [0:39] or something about this thing called [0:41] goal that they developed at naughty dog [0:43] for some of their PS2 games and it was [0:46] like a custom programming tool chain [0:49] and I remember reading about that and [0:50] just thinking that this is obviously [0:52] like the coolest thing that I've ever [0:54] heard of it didn't seem like something [0:56] that you could even do [0:58] so that really made an impression on me [1:00] and kind of showed me my own lack of [1:02] imagination at the time [1:04] but fast forward to four or five years [1:06] ago now and I got a chance to work on [1:08] something like that for our own company [1:09] and it's kind of become like a passion [1:11] project for me so I'm excited to show [1:13] off some of the cool things that I think [1:15] we've been able to come up with [1:17] for this demo I'm going to use a port [1:19] that I made of one of our games human [1:21] resource machine that wasn't originally [1:22] created using these tools but I wanted [1:24] to have a real finished game to use here [1:26] and not just like a toy example [1:29] also this game happens to be a game [1:30] about programming but this demo is about [1:33] our development environment in general [1:35] and so it would apply equally well to [1:37] any of our games [1:40] I'm going to start with just sort of the [1:42] basic version of what a lot of the rest [1:43] of this stuff is going to be based on [1:45] this is the only non-code part of the [1:47] demo where I'm going to show off my art [1:49] skills and do a simple hot load of a [1:52] data asset [1:53] so if I bring up a texture from the game [1:56] here and I make an edit and save it [1:59] then we can see that it shows up here in [2:00] the game without having to stop and [2:02] restart it so that's a nice feature to [2:04] have but now we're going to build off [2:06] that idea and see what else we can do [2:12] I'll open up our code editor here and [2:14] we'll see if we can't do something a [2:16] little more interesting [2:18] this is the code that's drawing these [2:20] green boxes over on the left side here [2:23] and since we hot loaded data it's only [2:25] fair that we Hot Load code [2:27] so if I make an edit here [2:30] we can see that that gets hot loaded too [2:32] our compiler is incremental and it [2:35] builds native code pretty fast so we can [2:37] compile on every keystroke and there's [2:39] almost no delay between me typing into [2:42] the editor and the changes showing up in [2:44] the game even in this kind of ridiculous [2:46] 13 000 line source file [2:51] if I navigate over to the code that's [2:53] drawing the floor here with these blue [2:55] boxes we can see some debugging features [2:59] if I set a breakpoint at the top of this [3:02] function you can see that the scene is [3:04] kind of partially drawn in right now [3:06] and then as I step through here I can [3:09] see the different things that I'm [3:10] drawing showing up on the screen as I go [3:13] since this is our own debugger talking [3:15] to our own game engine we can kind of [3:17] arrange to keep the screen alive while [3:19] you step through code sort of like a [3:21] graphics debugger and that can come in [3:23] handy sometimes when you're trying to [3:24] figure out where some stray little thing [3:26] on the screen is coming from [3:28] another thing I can do while I'm [3:30] stepping through my code here is [3:31] actually do a step backwards [3:34] I can do backwards versions of the usual [3:36] step in step over step out and I can run [3:39] backwards to my break point and I can [3:41] also run backwards to Hardware data [3:44] break points [3:45] and being able to debug backwards from [3:47] an assert or a crash especially with a [3:49] data break point is insanely helpful in [3:51] being able to figure out and fix bugs [3:54] and one of the keys to making this as [3:55] practically useful as possible is that [3:58] this isn't a special mode that I [4:00] launched the game into in order to do [4:01] this kind of debugging this is just the [4:03] way it works all the time so you always [4:05] have this option available while you're [4:07] working [4:11] so I'll switch over to the game window [4:12] now and we can look at our in-game [4:14] profiler [4:16] by default the instrumentation for the [4:19] profiler doesn't get compiled in but [4:21] when I click enable here we do a full [4:24] recompile of the game code and now we're [4:26] getting our call graph instrumentation [4:29] and it looks like there are some spikes [4:31] happening on some of these frames so [4:32] I'll pause this data collection so we [4:34] can take a closer look [4:37] if I look at the frame that got [4:40] expensive here there's this one function [4:41] taking it most of the time [4:43] and normally in a profiler you can click [4:46] on the name of the function and go see [4:47] the source code and I can do that here [4:50] but since we can debug backwards another [4:52] thing that I can do is click on this [4:55] time stamp up here [4:56] and when I do that I'm taken to the [4:59] actual invocation of this function on [5:01] this Frame and I can debug and see that [5:04] the reason that this took so long this [5:07] time is that I got an unlucky random [5:10] number [5:11] now obviously this is a contrived [5:14] example and this isn't a silver bullet [5:16] for every possible kind of performance [5:17] issue but it's a cool additional tool [5:20] and it can be really useful for certain [5:21] kinds of situations [5:24] so once I resume this [5:26] and get rid of this bad code here [5:29] then the profile looks good and I can [5:33] compile out the instrumentation code and [5:36] we can keep going [5:40] so the profiler can tag function [5:41] activations with these time stamps for [5:43] debugging but that's also something that [5:45] you can do yourself in code [5:47] for example a lot of the time if you [5:49] create something at one point in a [5:50] program and it needs to be cleaned up [5:52] later you tag it with some kind of debug [5:54] information so you know where it came [5:55] from [5:56] so memory allocations are a thing like [5:58] that where you normally tag those with [6:00] like a file name and a line number or if [6:02] you're lucky you can tag them with a [6:03] call stack and it makes it easier to [6:05] debug memory leaks and stuff like that [6:08] so here in our programming language we [6:10] have something like an intrinsic that [6:12] when it gets evaluated at runtime it [6:15] gives you the current execution [6:16] timestamp and you can use that to tag [6:18] stuff with instead [6:20] so I'll break into the game here and I'm [6:22] going to look at the data for the boxes [6:25] that I put on the conveyor belt to solve [6:27] this level [6:35] so I can see that there are three of [6:36] them here and if I look at the middle [6:39] one I threw on [6:40] here's the time stamp I associated with [6:42] it and then I can use that to go back [6:45] and [6:47] look at the point in time when that box [6:49] was actually thrown onto the conveyor [6:52] and if I undo that I can go back and [6:54] look at a different one [6:57] and see what was going on when I threw [6:58] that one on [7:01] so that's an example of the kind of [7:02] thing that I might tag in this sort of [7:03] game but in other games you might tag [7:05] the creation of all your bad guys or [7:08] remember the last time the player got [7:10] hit or tracked the history of every door [7:12] that you walk through or whatever else [7:13] you can think of [7:14] these time stamps are super cheap to [7:16] generate and they're only 64 bits so you [7:18] can use them to tag pretty much [7:19] everything with what effectively is the [7:22] entire state of the program and then you [7:24] can go back later and inspect anything [7:26] you need to if you want to find out [7:28] where something came from or why it [7:30] didn't get cleaned up or whatever you [7:31] want to know [7:33] so finally now we're going to turn the [7:35] determinism knob all the way up to 11. [7:39] so when I come in here and bring up our [7:41] developer console in the game we have a [7:43] scrubber and some playback controls down [7:45] here [7:46] and this timeline has everything that I [7:49] just did during this demo and I can use [7:51] it to go back and sort of inspect any [7:54] part of it that I'm interested in [7:57] and this isn't like a video that we're [7:59] looking at this is the actual simulation [8:01] and rendering of the game happening on [8:04] demand [8:08] this all still works even though I did [8:11] stuff during the demo like fully [8:13] recompile the code for the profiler I [8:17] edited the code for these boxes and I [8:19] changed this texture [8:20] but I can still scrub back and forth [8:22] through all those different changes [8:25] I specifically wanted to try not to have [8:27] one-way streets in this functionality so [8:30] using one feature doesn't rule out using [8:32] a different one later on [8:34] so if I go back to before I changed the [8:38] code for these boxes [8:45] if I come in here and I debug this now I [8:48] can see that this is the original [8:49] version of the code before I changed it [8:52] and that's because the editor is aware [8:55] of this whole system and it always lets [8:57] me work with the state of the code as it [8:58] was at the time that I'm looking at [9:01] I could also go back and profile code [9:03] after The Pact if I'm playing along and [9:05] I see a weird hitch or something like [9:07] that I don't have to start thinking [9:08] about how to reproduce it I can just go [9:10] back and see what it was [9:12] and the point of all this isn't that I'm [9:14] just dying to go back and debug my stale [9:15] code the point is that the system as a [9:17] whole is a coherent set of features that [9:19] all work together [9:21] so if you've worked with other [9:23] determinism-based replay systems one of [9:25] the issues that they tend to have is [9:27] that they feel really fragile and you [9:28] have to have constant vigilance that you [9:31] don't do something that's going to break [9:32] the determinism [9:34] but in our case since we've created [9:36] every level of this stack to support [9:38] this we can super efficiently offload [9:40] the determinism stuff to the development [9:42] environment and essentially you can just [9:44] write whatever code you want without [9:45] having to spend any time thinking about [9:47] whether or not you're breaking something [9:51] the other cool thing about this is that [9:53] we save a session file with this [9:55] timeline every single time you run the [9:57] game and since this whole thing is [9:58] integrated with our Version Control the [10:00] amount of data we actually need to save [10:02] to be able to replay a timeline stably [10:04] at any point into the future is really [10:06] small [10:07] and it's not just programmers who are [10:08] generating these session files is [10:10] everybody on the team and the files are [10:12] portable so if you're a designer working [10:14] on a level and something isn't working [10:16] quite right when you try it in the game [10:17] or you just randomly saw something weird [10:19] happen and you want somebody to look [10:21] into it you can send your session to [10:22] anyone else and they can play it back [10:24] live and debug it and this is all [10:26] integrated with our build system so that [10:29] it all just works even if you had local [10:31] changes when you created the session and [10:33] the person playing it back is on a [10:35] different revision and Source control [10:36] and they have their own local changes [10:38] the system can get all the data it needs [10:40] to play back the session and it doesn't [10:42] touch any of my actual local files in [10:44] the process so there isn't any risk of [10:46] me losing work if something went wrong [10:51] so I have a session here that I have [10:54] prepared ahead of time and we can [10:57] imagine that somebody sent this to me to [10:59] look at because their game crashed [11:02] so I'll load this up [11:05] and I'll run it to the end and sure [11:07] enough they got an access violation here [11:11] and you can see that the boss from Human [11:13] Resource machine lives on our Panic [11:15] screen and he's not happy with me right [11:17] now [11:18] so one thing I can do is just sort of go [11:21] back through this session and look at [11:22] what happened [11:25] so it looks like somebody played through [11:28] the level once [11:30] they went back and sort of optimized [11:33] their solution [11:36] played through it again [11:37] and then got this crash [11:40] so if I click on the top of the call [11:42] stack here I'm taken into the code [11:44] editor and if I got an access violation [11:47] on this line that probably means that [11:49] this pointer is null [11:52] and I know that this is a parameter to [11:56] this function so I'll do a reverse step [11:58] out of the function [11:59] and here I am back before I called into [12:01] it [12:02] and here we're passing the null pointer [12:04] in [12:05] and I can see that that pointer is [12:07] supposed to get set up in this if block [12:09] depending on if you're in normal play [12:11] mode or playing on the phone [12:15] and if I look at the current value of [12:17] this play mode variable the debugger is [12:20] saying that the value it has is not [12:21] actually a valid member of the [12:23] enumeration [12:25] and if you look at the bit pattern here [12:26] you might recognize that that's the [12:28] floating Point number one [12:31] so at this point you're thinking this [12:33] looks like a memory Stomp and I better [12:35] cancel my plans because who long who [12:38] knows how long it's going to take me to [12:39] figure this out [12:40] but [12:42] what I can do here is bring this play [12:44] mode variable into my watch window [12:47] and I can set a data breakpoint on it [12:50] and now when I run backwards I'm going [12:52] to find the most recent write that [12:55] happened into that memory [12:57] and it looks like that was right here [12:59] I'm assigning the value 1.0 to something [13:02] so that makes sense [13:04] and if I go back a couple more it looks [13:07] like the index I was using to write into [13:10] this buffer started at 15. [13:12] it incremented to 16 compared to the [13:14] capacity which is also 16. so this [13:18] should have wrapped around here but it [13:20] didn't and it looks like this is the bug [13:22] here this greater than should have been [13:24] a greater than or equal to [13:27] so obviously again this was a contrived [13:30] example but this isn't a totally unheard [13:32] of kind of bug unfortunately and you can [13:35] imagine that if I were attacking this [13:37] with printfs or something instead I [13:39] might be in for kind of a long night [13:41] luckily though most bugs aren't even [13:43] that complicated they're just like [13:45] run-of-the-mill logic errors in gameplay [13:47] code or something so being able to debug [13:49] those with this kind of system makes it [13:51] super easy [13:55] so that brings me to the end of the demo [13:58] check out the links in the description [14:00] for more great information about [14:01] tomorrow Corporation and thank you for [14:03] watching