Kelley Demonstrates Universal Linux Game Binary, Urges Valve Backing

2021: Year of the Linux Gaming Desktop - Andrew Kelley — Zig SHOWTIME

Zig SHOWTIME

Windows dominates desktop computing solely because of video games, but Linux fragmentation can be bypassed if upstream maintainers and Valve adopt hybrid ELF binaries that link graphics drivers dynamically at runtime.

AI-assisted summary of 2021: Year of the Linux Gaming Desktop - Andrew Kelley — original source. Check the original for full context.

Source published:

Key ideas

  • Andrew Kelley argues that desktop operating system market share is driven almost exclusively by video games, trapping Linux in a chicken-and-egg cycle where developers follow Windows market share and players stay where the games are. (1:36)
  • A proof-of-concept hybrid ELF executable ('static-window-9') successfully displays a Vulkan triangle across diverse Linux distributions—including glibc and musl environments like Ubuntu, NixOS, Void Linux, and Arch—without recompilation or containers. (3:45)
  • The universal binary bypasses hardcoded dynamic linker paths by starting as an uninterpreted static binary, discovering the host dynamic linker at runtime, and reinvoking itself via execve with LD_PRELOAD. (8:54)
  • Patching the official Vulkan loader to use RTLD_GLOBAL instead of RTLD_LOCAL enables process-wide symbol sharing, eliminating the need to vendor or re-implement graphics driver stacks. (25:13)

Detailed notes

## Background This presentation was delivered by Andrew Kelley on the Zig SHOWTIME channel on YouTube, published on February 19, 2021, as part of episode #21. The broadcast features a technical talk followed by an interview and community Q&A exploring whether Linux can overcome its runtime hurdles to become a viable gaming desktop.

## TL;DR Kelley argues that video games remain the primary reason Windows dominates the consumer desktop market, trapped in a self-reinforcing feedback loop that starves Linux of commercial support. To tackle the platform's notoriously brittle shared library and dynamic linker fragmentation, he demonstrates a proof-of-concept hybrid ELF executable capable of running a Vulkan window out of the box across radically different distributions like NixOS, Void Linux, and Ubuntu. The solution relies on a statically bootstrapped binary with a stripped dynamic section that inspects the host distribution at runtime, invokes execve on the system dynamic linker, and links dynamically into native host libc and GPU drivers. Ultimately, Kelley urges the Linux and graphics communities—as well as companies like Valve—to rethink dynamic linking conventions and officially support universal binaries for gamers.

## Atmosphere & style The presentation carries the pragmatic, hacker-spirited energy characteristic of the Zig community, balancing self-deprecating humor with deep systems-level tenacity. Kelley is candid about his 'clickbait' title and the sheer absurdity of the linker workarounds he had to assemble, yet his delivery remains grounded in genuine technical optimism. The atmosphere shifts naturally from a focused screen-share deep dive into assembly-level headers to a relaxed, banter-filled livestream Q&A with host Loris Cro, leaving viewers with the impression of an ambitious proof-of-concept meant to challenge entrenched industry norms.

## Bulletin - [01:36](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=96s) Andrew Kelley argues that desktop operating system market share is driven almost exclusively by video games, trapping Linux in a chicken-and-egg cycle where developers follow Windows market share and players stay where the games are. - [03:45](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=225s) A proof-of-concept hybrid ELF executable ('static-window-9') successfully displays a Vulkan triangle across diverse Linux distributions—including glibc and musl environments like Ubuntu, NixOS, Void Linux, and Arch—without recompilation or containers. - [08:54](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=534s) The universal binary bypasses hardcoded dynamic linker paths by starting as an uninterpreted static binary, discovering the host dynamic linker at runtime, and reinvoking itself via execve with LD_PRELOAD. - [25:13](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1513s) Patching the official Vulkan loader to use RTLD_GLOBAL instead of RTLD_LOCAL enables process-wide symbol sharing, eliminating the need to vendor or re-implement graphics driver stacks.

## Key takeaways - [01:36](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=96s) Windows maintains consumer desktop dominance primarily because it commands the overwhelming majority of video games, creating a self-reinforcing loop that keeps developers and gamers off Linux. - [06:01](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=361s) Standard Linux binary portability fails across distributions due to conflicting C standard libraries (glibc vs. musl) and hardcoded interpreter paths inside the ELF .interp header. - [06:01](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=361s) If a conventional Linux executable encounters a missing or mismatched interpreter path, the kernel terminates process creation before reaching main. - [12:38](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=758s) By dynamically linking against a dummy library and then stripping the DT_NEEDED dependency with patchelf, an ELF binary can retain a valid .dynamic section while omitting the .interp header. - [14:12](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=852s) Declaring dlopen as a weak external symbol enables a two-pass entry point: if dlopen is NULL, the binary executes its static bootstrapping phase; if populated, it proceeds with dynamic execution. - [14:12](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=852s) During static bootstrapping, the binary dynamically inspects host distribution metadata to locate the system dynamic linker, sets LD_PRELOAD for libdl and libpthread, and replaces its process image via execve. - [19:10](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1150s) glibc and musl require distinct initialization flows, with musl requiring explicit invocations of __libc_start_main and manual symbol resolution via dlopen. - [20:55](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1255s) Statically embedding the upstream Vulkan-Loader requires stubbing all standard C library functions as weak symbols to prevent immediate segmentation faults during the initial static bootstrap pass. - [25:13](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1513s) Upstream dynamic loading using RTLD_LOCAL isolates symbols within the loader scope; patching it to RTLD_GLOBAL allows the host graphics driver and application to share cached runtime dependencies. - [32:16](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1936s) Re-implementing the X11 network protocol purely in application code provides little practical benefit because host graphics drivers already load dynamic X11 shared libraries into memory. - [36:55](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2215s) Native Wayland support can be incorporated cleanly through fallback detection, attempting to load libwayland-client.so first before falling back to XCB and XWayland. - [29:12](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1752s) Toolchains like LLD and compiler toolchains often strip the dynamic section when an executable lacks direct dynamic dependencies, unintentionally breaking hybrid static-dynamic binary architectures. - [41:11](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2471s) Upstream collaboration across Vulkan-Loader maintainers, distribution packagers, and platform holders like Valve is required to standardize hybrid dynamic loading rather than relying on brittle build hacks.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=0s) Introduction: A Statically Linked Dream

The session opens with an intro highlighting the ultimate goal of the talk: Andrew Kelley has a dream—an "almost statically linked dream"—where video games can be built and distributed for Linux as distribution-agnostic executables. The host jokes to Gabe Newell, urging him to put down *Half-Life: Alyx Episode 2* because he will want to watch this presentation, officially titled "2021: Year of the Linux Gaming Desktop."

[01:36](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=96s) The Gaming Feedback Loop on Desktop Operating Systems

Andrew takes the stage and addresses the perennial question in the Linux community: will this year finally be the "Year of the Linux Desktop"? His blunt answer is no. Andrew argues that Windows and macOS dominate the consumer desktop market exclusively because of video games. “It's just because of games that's my premise,” he states, identifying this as the single defining advantage those platforms maintain over Linux.

Game developers target Windows and macOS because they want to run viable businesses, following the overwhelming market share visible in Steam hardware and software user surveys. If an independent or AAA developer ignores the massive blue Windows slice of the pie chart in favor of the tiny Linux slice, they simply will not make money. Gamers, in turn, choose Windows because that is where the vast catalog of games lives. This creates a self-reinforcing feedback loop:

```mermaid flowchart LR accTitle: Desktop Gaming Feedback Loop accDescr: Diagram illustrating the self-reinforcing cycle between game developers and gamers on Windows. A["Gamers choose Windows<br>(Where the games are)"] --> B["Game developers target Windows<br>(Where the money is)"] B --> A ```

Andrew concedes that he has no solution to dismantle this entrenched feedback loop, openly calling the title of his talk a "bait and switch" clickbait hook. However, he believes it is entirely possible to make the Linux gaming experience seamless for end users, which could gradually expand Linux's slice of the pie.

[03:45](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=225s) The Proof of Concept: A Universal Linux Executable

Andrew moves to a live demonstration of his proof of concept: an ELF executable named `static-window-9`, hosted at a temporary URL that he jokes he has not cleaned in years and will leave up indefinitely. The number nine in the filename, he quips, simply marks how many times he failed before getting it right.

The binary is a minimal graphical application that opens a window and renders a basic Vulkan triangle. It does not even handle window resizing or graceful exits; the user has to terminate the process from the terminal. Under the hood, it talks directly to the X11 display server protocol (or XWayland if running on Wayland) and binds into the host system's native Vulkan graphics drivers. Andrew demonstrates that this exact same ELF binary runs out-of-the-box across a wide spectrum of Linux distributions without recompilation or containers:

- Ubuntu - NixOS - Void Linux (musl libc variant) - Void Linux (glibc variant) - Clear Linux - Arch Linux - Debian Sid

He runs the binary on his local machine to verify that the triangle appears on screen as advertised, proving the concept is functional.

[06:01](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=361s) Why Linux Binary Portability Is Hard: The Dynamic Linker and Libc

To explain why running a single graphical binary everywhere is difficult, Andrew compares his executable against a standard Vulkan demo binary like `vkcube` from `vulkan-tools`. On Windows, distributing an executable that talks to graphics drivers is straightforward, making it a much friendlier platform for game developers. On standard Linux distributions, a regular binary introduces distribution-specific barriers:

1. Dynamic Dependencies: Running `ldd` against `vkcube` reveals direct shared library dependencies on `libX11`, `libc`, and the system Vulkan loader (`libvulkan`). 2. Conflicting C Standard Libraries: Host graphics drivers dynamically link against the host system's `libc`. Different distributions deploy fundamentally different C standard libraries, most notably GNU C Library (`glibc`) versus `musl`, which are not ABI-interchangeable. 3. Hardcoded Interpreter Path: Inspecting the ELF headers of a conventional binary via `readelf` reveals the `.interp` section. This section stores a hardcoded absolute file path to the system dynamic linker (such as `/lib64/ld-linux-x86-64.so.2` on glibc systems, a musl-specific path on musl distros, or a `/nix/store/...` hash path on NixOS).

If a user copies a binary compiled on one distribution to another whose interpreter path differs, the kernel fails before the program's code can execute: “If I tried to run this binary vk cube on a different Linux it would just say like error it would never execute it never even gets main because this path wouldn't exist on your computer.” This brittle coupling explains why platforms like Valve's Steam target only an Ubuntu-compatible base runtime to avoid dealing with varying system setups.

[08:54](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=534s) Evolving the Self-Execution Architecture

Because the kernel invokes the dynamic interpreter specified in `.interp` before reaching `main`, application code normally cannot intervene to resolve missing paths. Andrew credits Jonathan Marler for sparking the solution. Marler had explored an ELF binary concept that executes statically first and then invokes `execve` on itself, passing its own binary as the argument to the dynamic linker. Because dynamic linkers (such as `ld.so`) are themselves executable binaries, passing an ELF file to the loader command-line instructs it to dynamically link and run that target executable.

Marler’s initial prototype had two drawbacks: - It required linking against a dummy shared library to force the linker to emit an ELF `.dynamic` section. - It still relied on a hardcoded path to invoke the dynamic loader.

Andrew designed a more comprehensive runtime strategy to bypass these limitations:

```mermaid flowchart TD accTitle: Universal Linux Binary Execution Flow accDescr: Flowchart showing how the hybrid ELF binary bootstraps statically, discovers the dynamic linker, reinvokes itself, loads system libc and Vulkan, and draws graphics. subgraph Phase1["Static Bootstrapping"] A["Start as static binary<br>(No.interp section)"] --> B{"Running under dynamic linker?"} B -- "No" --> C["Detect system dynamic linker path at runtime"] C --> D["execve() self using discovered linker"] end subgraph Phase2["Dynamic Integration"] D --> B B -- "Yes" --> E["Detect host system libc (glibc or musl)"] E --> F["dlopen() host libc & load Vulkan driver"] F --> G["Render Vulkan graphics triangle"] end ```

[11:33](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=693s) Crafting the Hybrid ELF Binary

The foundational hurdle in this design is constructing a hybrid ELF executable that the Linux kernel will happily run statically, but which also contains the necessary ELF metadata to be processed by an external dynamic loader.

Andrew inspects his `static-window-9` binary with standard command-line tools to show its internal layout. Running `ldd` reports that the binary is "statically linked." Examining its ELF program headers with `readelf` confirms that there is no `.interp` section present, meaning the kernel directly maps the binary into memory and enters its entry point without looking for a pre-configured loader. Crucially, however, the binary still contains an ELF `.dynamic` section. This dynamic section provides the entry tables needed for dynamic linkers to bind symbols once reinvoked, bridging the gap between a standalone static binary and the host's dynamic driver ecosystem.

[12:38](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=758s) Crafting a Binary with a Dynamic Section but No Dynamic Dependencies

Andrew Kelley explains how to construct an executable that can function both statically and dynamically without being rejected or mishandled by the Linux kernel. If an executable has an `INTERP` program header, the operating system kernel immediately invokes the specified dynamic linker path; conversely, without a `.dynamic` section, dynamic linking cannot take place later. To balance both requirements, Kelley creates a dummy shared library during the build and dynamically links the executable against it. This forces the linker to emit a valid `.dynamic` section.

Immediately following compilation, Kelley uses the `patchelf` utility with the `--remove-needed` flag targeted at `libdummy`. This strips the `DT_NEEDED libdummy` tag out of the `.dynamic` section entirely. The result is a unique binary: it retains a valid `.dynamic` section structure, but it lists zero dynamic library dependencies and contains no hardcoded `INTERP` entry. Because there is no interpreter header, Linux launches the binary directly into its static entry point without attempting to resolve dynamic libraries at startup. As Kelley summarizes: “We have a executable that supports being run both dynamically and statically, so it has a dynamic section but no dynamic dependencies, and the OS or Linux will not try to run it with a dynamic linker because there's no interp entry.”

```mermaid flowchart TD accTitle: Binary Stripping and Re-execution Architecture accDescr: Diagram illustrating how the binary starts statically, detects the dynamic linker, and re-execs itself with LD_PRELOAD.

subgraph Build["Build Stage"] A["Compile with dummy shared library"] --> B["Binary has.dynamic section"] B --> C["Run patchelf --remove-needed libdummy"] C --> D["Executable has.dynamic section but NO dependencies & NO INTERP"] end

subgraph Execution["Runtime Execution"] D --> E["Kernel runs binary statically: enters main()"] E --> F{"dlopen symbol present?"} F -->|"No (First Pass)"| G["Detect native target info & dynamic linker path"] G --> H["Prepare argv: insert dynamic linker as argv[0]"] H --> I["Set LD_PRELOAD=libdl:libpthread"] I --> J["Call execve() to reload self via dynamic linker"] J --> F F -->|"Yes (Second Pass)"| K["dlopen!= null: run libc-agnostic setup & enter main2()"] end ```

[14:12](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=852s) The Two-Pass Entry Point and Self-Re-execution via execve

Reaching the `main` function introduces an immediate architectural puzzle: the binary can enter `main` under two completely different execution contexts. On the initial run, the OS kernel executed the binary directly as a static program without a dynamic linker. On the subsequent pass, the binary is invoked through the dynamic linker. Because both paths lead to the very same `main`, the startup code must detect which environment it is currently running in.

To achieve this, the program defines a weak external symbol reference for `dlopen` (`weak extern dlopen`). The `dlopen` symbol is ordinarily provided by `libc`/`libdl`. If the binary is running statically on the first pass, `dlopen` resolves to `NULL`. When `dlopen` is `NULL`, the program executes its bootstrap path: it must find the host system's dynamic linker and call `execve` on itself so it can re-enter `main` under dynamic execution. Kelley emphasizes: “Making it a weak extern symbol means that we support if it's not available, in which case um this code will run because we don't have dlopen because we're being run statically; however, if the symbol is available, then we go down here and we know that we're actually running inside the dynamic linker.”

Because different Linux distributions store their dynamic linkers at different paths, hardcoding is impossible. Kelley relies on logic already developed for the Zig standard library: - Zig's `detectNativeTargetInfo` function inspects the host distribution at runtime to extract the native target triple and the exact absolute path to the dynamic linker. - The binary constructs a new `argv` array where `argv[0]` is the detected dynamic linker path, followed by the path to the executable itself and the original command-line arguments. - Because stripping the `DT_NEEDED` tags during build removed all automatic library dependencies, the program modifies the environment variables before calling `execve`, explicitly setting `LD_PRELOAD` to pull in `libdl` and `libpthread` (necessary for dynamic loading and threading primitives like mutexes). - When `execve` executes, it wipes out the current process image and reloads the binary via the specified dynamic linker, satisfying the `LD_PRELOAD` libraries. - Upon re-entering `main`, the weak `dlopen` symbol is no longer `NULL`, allowing the binary to advance into normal dynamic execution.

[19:10](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1150s) Handling libc Diversity: glibc vs. musl

Solving the dynamic linker problem brings the program to its next hurdle: Linux distributions do not share a single C standard library implementation. While standard systems like Ubuntu use GNU C Library (glibc), lightweight distributions like Alpine Linux rely on musl. Once re-executed under the dynamic linker, the binary still has to bridge structural behavioral differences between these libraries.

On glibc-based systems, loading the dynamic linker smoothly brings in the required standard runtime environment, allowing the program to proceed straight to application logic. Musl, by contrast, is more bare-bones and requires manual intervention: - On musl, the code explicitly invokes musl's `__libc_start_main`, providing pointers to handle process initialization. - It must manually set up exit handlers (calling `exit` explicitly) and use `dlopen`/`dlsym` to resolve required standard `libc` functions. - The binary once again calls `detectNativeTargetInfo` from the Zig standard library to detect the native runtime triple (distinguishing whether the host is glibc or musl). - After distribution-specific setup finishes, execution routes into a unified second stage, `main2`, where the application can safely access the Vulkan and X11 APIs.

[20:55](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1255s) Hacking the Vulkan Loader and Weak libc Function Symbols

Even with runtime `libc` selection functional, running graphics APIs required overhauling the official Vulkan loader (`Vulkan-Loader`), a Khronos Group project that locates hardware drivers and populates function pointers for the Vulkan API. The loader smooths out system variations, but it is fundamentally designed to be built as a dynamic library linked against the host system's C library. Kelley wanted to embed and build the loader statically into the binary without tying it to a specific `libc`.

To build the loader statically, Kelley extracted only the necessary C source files from the upstream Khronos repository into the Zig project. Taking advantage of Zig's built-in C compilation pipeline (powered by `libclang`), he configured the `build.zig` build script to define the requisite preprocessor macros, compile the selected source files, and strictly enforce Position Independent Code (`-fPIC`) so the generated machine code remains valid whether run statically or dynamically.

Embedding the loader introduced a major issue: the loader code makes dozens of calls to standard C library routines—such as `memcpy`, `memset`, `putchar`, and `fclose`. During the static first pass of `main`, before the dynamic linker has run and before `libc` is loaded, calling any of these loader routines would instantly crash:

| Problem | Root Cause | Implemented Solution | |:--- |:--- |:--- | | **Premature libc calls crash** | In the initial static bootstrap pass, no `libc` is loaded in memory. Any standard C library function address is completely unresolved. | Kelley authored a custom `libc.h` header declaring every required standard function (`memcpy`, `memset`, `fclose`, etc.) with `__attribute__((weak))`. Unresolved symbols default safely to `NULL`. | | **Vulkan driver symbols hidden** | Upstream dynamic loading often uses `RTLD_LOCAL`, restricting symbol visibility to the loaded library itself. | Kelley patched the loader's calls from `RTLD_LOCAL` to `RTLD_GLOBAL`, ensuring that when the X11 and Vulkan libraries are loaded via `dlopen`, their symbols are exported globally for the underlying graphics drivers. |

As Kelley puts it regarding the missing `libc` functions on the initial pass: “If we tried to call inside here if we tried to call any of the Vulkan loader stuff it would crash because every single libc function like `memcpy`, `memset`, um you know `putchar`, `fclose`, all these, they would all just segmentation fault because they're all zero, they're all null, because they're weak.” By combining the weak-symbol header override with patched `RTLD_GLOBAL` dynamic loading flags, the executable defers all loader activity until `main2`, where the correct native `libc` and driver dependencies have finally been loaded into the global symbol space.

[25:13](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1513s) Patching the Vulkan Loader and the Static Linking Debate

Kelley explains that the overall goal was to rely on the dynamic loader's caching mechanism so that when the application requests a library, it simply reuses the exact same one the Vulkan driver already loaded. To make that guarantee hold up, Kelley had to patch the Vulkan loader directly, forcing it to perform a global dynamic load (`RTLD_GLOBAL`) rather than a local load (`RTLD_LOCAL`). By making the load global, the loaded symbols become visible across the entire process instead of remaining confined behind isolated namespaces.

```mermaid flowchart TD accTitle: Vulkan Loader Patching Flow accDescr: Diagram illustrating how patching the Vulkan loader from RTLD_LOCAL to RTLD_GLOBAL allows the application to share driver dependencies.

subgraph StandardBehavior["Unpatched Vulkan Loader"] V1["Vulkan Loader"] -->|"dlopen(..., RTLD_LOCAL)"| D1["Driver & Shared Objects"] A1["Application dlopen()"] -.->|"Cannot see symbols"| D1 end

subgraph PatchedBehavior["Patched Vulkan Loader"] V2["Vulkan Loader"] -->|"dlopen(..., RTLD_GLOBAL)"| D2["Driver & Shared Objects"] A2["Application dlopen()"] -->|"Finds cached symbols"| D2 end ```

He then touches on the politics of the library, adding a "caveat emptor" from an encounter on Discord or the Khronos group where maintainers expressed strong disapproval of static linking:

> “I was talking to one of the maintainers of this library and they were saying that they don't like it when people static link Vulkan loader because, I don't remember, there's some reason that they don't want you to static link it; but it does make the game work so, I don't know, maybe they should consider this a use case.”

Kelley argues that Linux distribution maintainers and Vulkan loader developers ought to collaborate to support static linking officially, because “this is what players want: they want to run the game.” Relying on an out-of-tree patch highlights why this demo is strictly a proof of concept. The upstream Vulkan loader repository moves fast, and maintaining a personal fork with vendor copy-pasted code introduces an ongoing maintenance burden to pull down and reconcile upstream commits.

---

[27:07](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1627s) Cross-Compiling X11 Headers and the Basic Vulkan Triangle

Once dynamic symbol caching is resolved, what remains inside the application code is invoking the X11 APIs to initialize a window and establish a Vulkan context. Kelley points out that while X11 calls ultimately just send network messages over a socket to the X server, using the exact same dynamic C runtime shared objects that the graphics driver links against remains the most dependable approach to ensure consistency across the entire address space.

Cross-compiling introduced its own friction. Because the Vulkan loader depends on X11 C header files, Kelley could not cleanly pull headers from his host operating system. He had to dig into X11's complex build system to determine how those header files were generated, extract them, and manually adjust preprocessor definitions (`#define`s) to make them cross-compilation safe. Because the underlying X11 protocol is an established network specification with a strictly frozen ABI, those constants will not break or change over time. From there, the rest of the application is minimal: Kelley notes he simply followed the basic triangle demo from `vulkan-tutorial.com`.

---

[29:12](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1752s) The Core Lesson: Raising Standards for Linux Gaming

Addressing game developers in the audience who might be eager to implement this setup in C or C++, Kelley gives a candid assessment: as things stand, it is not realistic for production use due to the sheer volume of obstacles. However, the experiment proves that bypassing the typical Linux shared library fragility is technically achievable.

> “A lot of these hoops are arbitrary and unnecessary, and I'm demonstrating that it's possible to get around them; but really I think that what we just need is an acknowledgement that this is the user experience that users, that players deserve, and we just need more people to demand this as how it works on Linux.”

Kelley wants the demo to inspire the community to raise the bar for platform standards. He mentions that the repository is public but unpolished, and in fact had recently regressed and stopped building. That failure occurred because the Zig compiler had introduced genuine improvements to how it links code and passes arguments to LLVM’s LLD linker.

```mermaid flowchart LR accTitle: Linker Assumption Conflict accDescr: Chart showing how standard linker reasoning conflicts with hybrid static and dynamic game binaries.

subgraph StandardLinker["Default Linker Logic"] L1["Inspect Dependencies"] -->|"No dynamic dependencies"| L2["Omit dynamic section entirely"] end

subgraph HybridBinary["Hybrid Portable Game"] H1["Static Game Engine"] -->|"Needs dlopen runtime calls"| H2["Requires dynamic section & loader"] end

L2 -.->|"Breaks execution"| H2 ```

In standard linker implementations, if an executable has no direct dynamic dependencies, the linker logically strips out the dynamic section entirely. For a hybrid binary—one that statically embeds its runtime yet expects to dynamic-load host driver libraries—that omission completely breaks the executable. Kelley emphasizes that tooling must explicitly recognize and formalize this hybrid approach, or else minor tooling improvements will inevitably trample the delicate list of hacks required to keep it functioning.

---

[32:16](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1936s) Q&A: Community Questions and Native X11 Protocols

Loris Cro opens the question-and-answer session, monitoring questions across Twitch, Discord, and the `#zig` IRC channel on Freenode. He reminds viewers to keep questions relevant to the talk, saving broader roadmap inquiries for later. Loris admits that despite disliking Apple, developer experience and ecosystem friction are precisely why his daily driver remains a Mac rather than a desktop Linux machine.

The first technical question comes from chat participant `gw1`, who asks whether implementing the X11 protocol purely in Zig (referencing project `juicebox`) could eliminate the need to dynamically depend on XCB or other X11 shared libraries. Kelley breaks down the runtime reality via explicit premises: 1. The game must communicate with graphics drivers like Vulkan or OpenGL. 2. The graphics drivers themselves carry hard dynamic dependencies on the X11 shared objects. 3. Therefore, those shared libraries will be loaded into the process memory space regardless of whether the main binary implements the network protocol directly. 4. Since the process already pays that memory and loading cost, the application might as well use those loaded shared objects directly rather than reinventing the client protocol stack.

---

[35:19](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2119s) Q&A: Pitching Valve and Debugging `dlopen` Scope

Viewer `zakus` jokingly asks when Valve will sponsor Zig. Kelley embraces the banter:

> “Hey Valve, you want to sponsor Zig, you should definitely do it: everyone's going to be writing games in Zig in five years.”

Loris jokes back that he will call Gabe Newell directly, remarking that he already calls him up to complain every time he misses his shots with a Tec-9 in *Counter-Strike: Global Offensive*.

A participant then asks for clarification regarding why weak symbols and `dlopen` failed before the loader patch, wondering why standard C symbols did not resolve if Vulkan was called only after loading libc. Kelley begins stepping through the exact debugging sequence on his screen (after a brief technical hiccup where Loris checks screen sharing). He clarifies that when he first attempted to `dlopen` the shared object already brought into virtual memory by the driver, the dynamic linker refused to find it. Because the unpatched Vulkan loader had originally opened the library with `RTLD_LOCAL`, its symbols remained restricted to its private loader scope, hiding them from the application until Kelley forced the loader to open them globally via `RTLD_GLOBAL`.

[36:55](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2215s) RTLD_LOCAL vs. RTLD_GLOBAL and Wayland Compatibility

Wrapping up a previous technical question from chat participant Murilo, Andrew Kelley explains the behavior of dynamic loading flags. When the Vulkan loader originally loaded its dependency using `RTLD_LOCAL`, subsequent attempts to reference or acquire that same handle failed or returned an invalid object because symbols were confined to their own local scope. Once switched to `RTLD_GLOBAL`, the dynamic linker makes those symbols globally available, ensuring that the call consistently returns a valid, shared object handle across the binary.

The host then fields an audience question asking what modifications would be required to make this dynamic linking and driver initialization setup work under Wayland. Andrew delivers a deadpan quip first: “Nothing, because Wayland, it supports the X11 protocol.” After laughing, he clarifies that while XWayland is a completely valid answer, a native implementation is also trivial to add. Explicit Wayland support would simply involve attempting to dynamically open a Wayland library object (`libwayland-client.so`) first. Because a graphics driver running in a native Wayland session depends on Wayland rather than X11, the application would check if the Wayland handle succeeds; if so, it uses it, and if not, it falls back to the X11/XCB implementation. Andrew highlights his stance on display server compatibility: “I am of the strong opinion that if Wayland is doing it right, this code should all just work fine on Wayland with almost no penalties.”

```mermaid flowchart TD accTitle: Wayland and X11 Dynamic Initialization accDescr: Application attempts to load native Wayland graphics libraries first, falling back to X11 if unavailable.

subgraph Detection ["Display Server Fallback Logic"] A["Try dlopen('libwayland-client.so')"] -->|Success| B["Use Native Wayland Pipeline"] A -->|Failure| C["Fall back to XCB / X11 Pipeline"] C --> D["Runs natively or via XWayland"] end ```

[38:33](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2313s) Absolute Paths, DT_RUNPATH, and Patching the Vulkan Loader

The Q&A turns to dynamic linking metadata, addressing an audience question about whether the ELF `DT_NEEDED` entry can simply specify the library name (e.g., `libname.so`) without an absolute path. Inspecting the standard control binary `vkcube`, Andrew demonstrates that `DT_NEEDED` already specifies library names rather than hardcoded absolute paths, listing entries like XCB, libc, and the Vulkan driver. The real friction arises from `DT_RUNPATH`:

``` Dynamic section at offset 0x... contains entries: NEEDED Shared library: [libxcb.so.0] NEEDED Shared library: [libc.so.6] RUNPATH Library runpath: [/usr/lib/x86_64-linux-gnu:...] ```

Andrew points out that hardcoding search directories inside the binary breaks portability: “And the problem is that you don't even—even with this way you just don't know what the, you know, what the run paths are until runtime, that's the problem. Like this run path would be wrong for other distros.”

Following this, viewer Johnny Martin asks whether patching Vulkan to load drivers globally would solve the duplicate shared object issue. Andrew clarifies the precise layer that would need modification. The initialization flow calls functions inside `vulkan-loader`, which in turn executes `dlopen` on the actual graphics driver. The goal is simply to have the application and loader reference that exact same loaded driver instance. Rather than altering graphics drivers directly, sending an upstream patch to `vulkan-loader` to adopt `RTLD_GLOBAL` would make this portable paradigm clean: - The driver initialization logic inside `vulkan-loader` performs the underlying `dlopen` call on the vendor driver. - Modifying `vulkan-loader` upstream avoids any requirement for GPU vendors (Nvidia, AMD, Intel) to patch their individual proprietary or open-source drivers. - While upstream maintainers might raise theoretical arguments against polluting global namespace symbols, Andrew maintains that “in this case it's the right call.”

[41:11](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2471s) The Linux Desktop Vision and Valve's Role

In closing out the session, Andrew steps back from low-level linker mechanics to assess the broader trajectory of Linux gaming. He commends Valve for the massive progress already made with Steam and Proton, observing that several modern games already perform better under Linux than they do on native Windows. He views distribution packaging conventions and rigid runtime library expectations as the final hurdles preventing seamless cross-distribution distribution for indie and commercial game developers alike.

Andrew expresses optimism that the industry can standardize around this flexible dynamic loading model: “I sincerely hope that people at Valve watch this short talk and get inspired, and use their influence and power to help the Linux distribution ecosystem support this use case better. I think that the users want it, the players want it, the game developers want it, and ultimately the Linux maintainers, the package maintainers are going to do what people want, so we just have to make people want this more.” The host agrees with this call to action, thanking Andrew for the presentation and announcing a short intermission before returning for the second half of the stream covering the upcoming Zig language roadmap.

## Notable quotes > [35:19](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2119s) Hey Valve, you want to sponsor Zig, you should definitely do it: everyone's going to be writing games in Zig in five years. — Andrew Kelley

> [25:13](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1513s) this is what players want: they want to run the game. — Andrew Kelley

## Mentioned Andrew Kelley, Loris Cro, Gabe Newell, Jonathan Marler, Zig, Linux, Windows, macOS, Ubuntu, NixOS, Void Linux, Clear Linux, Arch Linux, Debian Sid, Alpine Linux, Valve, Steam, Proton, Vulkan, Khronos Group, Vulkan-Loader, X11, XWayland, Wayland, XCB, glibc, musl, LLVM, LLD, patchelf, Half-Life: Alyx Episode 2, Counter-Strike: Global Offensive

## Video description From Zig SHOWTIME #21 Subscribe to the Zig SHOWTIME Newsletter https://zig.show

0:00 Title 1:35 Talk 32:15 Interview

[Source](https://www.youtube.com/watch?v=pq1XqP4-qOo)

中文简报

Kelley展示Linux通用游戏二进制文件,呼吁Valve提供支持 Windows主导桌面计算纯粹是因为电子游戏,但若上游维护者和Valve采用在运行时动态链接图形驱动的混合ELF二进制文件,即可绕过Linux的碎片化问题。 Kelley认为,电子游戏仍然是Windows主导消费级桌面市场的首要原因,这形成了一个自我强化的反馈循环,使Linux缺乏商业支持。为了解决该平台众所周知脆弱的共享库与动态链接器碎片化问题,他展示了一个概念验证混合ELF可执行文件,能够在NixOS、Void Linux和Ubuntu等差异极大的发行版上开箱即用地运行Vulkan窗口。该解决方案依赖于一个具有精简动态段的静态引导二进制文件,它在运行时检查宿主发行版,在系统动态链接器上调用execve,并动态链接到原生宿主libc和GPU驱动程序。最终,Kelley敦促Linux与图形社区以及Valve等公司重新思考动态链接惯例,并正式支持面向玩家的通用二进制文件。 Andrew Kelley认为,桌面操作系统的市场份额几乎完全由电子游戏驱动,使Linux陷入了“先有鸡还是先有蛋”的恶性循环中:开发者跟随Windows的市场份额,而玩家则停留在游戏所在之处。 (1:36) 一个名为“static-window-9”的概念验证混合ELF可执行文件,成功在包括Ubuntu、NixOS、Void Linux和Arch等glibc与musl环境在内的多种Linux发行版上显示Vulkan三角形,无需重新编译或使用容器。 (3:45) 该通用二进制文件通过作为未解释的静态二进制文件启动,在运行时发现宿主动态链接器,并通过带有LD_PRELOAD的execve重新调用自身,从而绕过了硬编码的动态链接器路径。 (8:54) 对官方Vulkan加载器进行补丁以使用RTLD_GLOBAL而非RTLD_LOCAL,可以实现全进程范围的符号共享,从而无需打包或重新实现图形驱动栈。 (25:13)

Reading summary

Andrew Kelley argues that desktop operating system dominance is driven primarily by video games, but Linux adoption remains hampered by brittle binary portability barriers across distributions. To resolve this, he demonstrates a proof-of-concept hybrid ELF executable that bootstraps statically, discovers host dynamic loaders, and interfaces directly with native graphics drivers. Gaming as the Desktop Market Driver (1:36) Kelley contends that Windows and macOS dominate consumer desktops largely because they host the games consumers want to play. Commercial and independent game developers target Windows to remain economically viable according to Steam user share, which reinforces gamer platform choices and creates an entrenched feedback loop. While acknowledging he cannot unilaterally break this economic dynamic, Kelley argues that eliminating packaging friction and making game binaries universally compatible across Linux distributions is a necessary first step to expanding desktop market share. Dynamic Linker and Standard Library Roadblocks (6:01) Distributing portable graphical binaries on Linux is complicated by distribution-level architecture fragmentation. Standard binaries depend on host C standard libraries, but distributions split between incompatible implementations like glibc and musl. Furthermore, standard ELF binaries include an .interp program header specifying an absolute path to a dynamic linker; if that path differs—such as on NixOS or musl-based systems—the kernel fails before the program's main function can execute. Hardcoded search paths in DT_RUNPATH entries further break cross-distribution compatibility. Two-Pass Hybrid ELF Execution Model (12:38) To bypass the interpreter restriction, Kelley implemented a two-pass execution strategy adapted from concepts by Jonathan Marler. The build dynamically links against a dummy library and immediately runs patchelf with --remove-needed, producing an executable that retains an ELF .dynamic table but has no INTERP header and no registered dynamic dependencies. The Linux kernel therefore runs the binary directly into its static entry point. Inside main, startup logic inspects a weak external dlopen reference. On the static first pass, dlopen resolves to NULL, prompting the program to use Zig's standard library functions (detectNativeTargetInfo) to identify the host platform and find the native dynamic linker. The binary configures LD_PRELOAD to include libdl and libpthread, and then invokes execve on itself with the discovered dynamic loader as argv[0]. When the binary restarts under the dynamic linker, dlopen is non-null, allowing execution to proceed into dynamic initialization. Patching the Vulkan Loader and Drivers (20:55) Because host graphics drivers depend on the host C runtime, running Vulkan requires bridging application code with native libraries. To avoid segmentation faults during the static first pass before libc is loaded, Kelley created a custom header that declares standard C functions (such as memcpy and fclose) with weak symbol attributes, defaulting them safely to NULL. Additionally, Kelley statically compiled the Khronos Vulkan loader into the binary using Zig's C compilation pipeline and patched its dlopen invocations from RTLD_LOCAL to RTLD_GLOBAL. This modification ensures that dynamically loaded driver symbols are published globally, allowing the application to reuse the driver's cached C runtime and windowing dependencies. The resulting executable (static-window-9) rendered a Vulkan triangle across Ubuntu, NixOS, Clear Linux, Arch, Debian Sid, and both glibc and musl variants of Void Linux. Kelley added that native Wayland support could be handled via a similar fallback check. Kelley emphasizes that this hybrid technique remains an experimental proof of concept rather than a production-ready standard, pointing out that modern linkers frequently optimize out empty dynamic sections and upstream Vulkan maintainers discourage static linking. He argues that Linux maintainers, Khronos, and platform stakeholders such as Valve should formally recognize and standardize hybrid dynamic loading workflows to simplify native Linux game distribution.

精炼阅读摘要

Andrew Kelley 认为,Windows 统治桌面操作系统的根源在于其游戏生态壁垒,而 Linux 难以吸引游戏开发者的技术瓶颈在于动态链接器与 C 标准库的跨发行版碎片化。他通过一个名为 static-window-9 的原型程序证明,利用混合 ELF 架构与运行时动态重载机制,开发者完全可以构建出无需容器或重新编译、即开即用的独立 Linux 游戏二进制文件。 桌面游戏反馈循环与二进制碎片化瓶颈 (1:36) Kelley 指出,玩家因庞大的游戏目录选择 Windows,游戏开发商又因商业收益优先考虑占据绝对市场份额的 Windows 用户,这一正反馈闭环锁死了 Linux 桌面游戏的发展空间。即便开发者有意适配 Linux,传统的动态链接机制也会制造巨大的分发壁垒:以常规 Vulkan 演示程序 vkcube 为例,其 ELF 头部在 .interp 段硬编码了系统动态链接器的绝对路径(如 glibc 的 /lib64/ld-linux-x86-64.so.2 或 NixOS 特定的哈希路径)。一旦将该程序复制到解释器路径不同的发行版上,Linux 内核在进入 main 函数前就会直接报错崩溃。 此外,不同发行版部署的 C 标准库互不兼容(如常见的 GNU C 库 glibc 与 Alpine 或部分 Void Linux 使用的 musl),且动态库查找路径 DT_RUNPATH 各异。由于缺少类似 Windows 的平滑驱动接口,即便是 Valve 的 Steam 也只能通过绑定特定的 Ubuntu 兼容运行时来规避底层碎片化,这进一步凸显了原生二进制难以跨发行版运行的痛点。 混合 ELF 结构与 execve 自重载机制 (8:54) 为绕过内核对硬编码解释器路径的强制校验,Kelley 借鉴 Jonathan Marler 的思路设计了一种双阶段自执行架构。在构建阶段,程序首先动态链接一个虚构的共享库以促使链接器生成 ELF .dynamic 段,随后利用 patchelf --remove-needed 剔除该依赖。由此生成的 ELF 既没有 .interp 解释器路径,也不存在显式动态依赖,使得 Linux 内核能够将其作为纯静态二进制直接执行,同时又为后续的动态链接保留了必要的元数据表结构。 在程序运行阶段,代码将 dlopen 声明为弱外部符号以区分执行环境。在内核静态启动的第一阶段,dlopen 解析为 NULL,程序随即触发自举逻辑:利用 Zig 标准库中的 detectNativeTargetInfo 在运行时探测宿主系统的动态链接器绝对路径,接着重写 argv 参数并将 LD_PRELOAD 设置为注入 libdl 和 libpthread,最终通过 execve 重新执行自身。在动态加载器接管的第二阶段,dlopen 符号解析成功,程序便能平滑切入动态运行流程。 跨 libc 兼容与 Vulkan 加载器符号改造 (19:10) 重新加载后,程序必须解决宿主环境的 C 运行库分歧。在 glibc 系统上,加载器能平滑建立运行环境;但在更为精简的 musl 系统上,程序必须显式调用 musl 的 __libc_start_main,手动配置进程退出处理函数,并借助 dlopen 与 dlsym 动态解析标准接口。完成环境探测与适配后,代码统一分流至第二阶段入口 main2,安全调用图形与系统 API。 为了在不绑定特定宿主 libc 的前提下调用底层硬件驱动,Kelley 从 Khronos 官方剥离了 Vulkan-Loader 源码并将其静态编译进二进制。为防止在第一阶段静态自举时调用未解析的 C 标准库函数导致崩溃,他编写了专用的 libc.h 头文件,将 memcpy、memset、fclose 等基础函数统一声明为弱符号。更关键的是,由于官方加载器默认采用局部符号加载标志 RTLD_LOCAL,导致驱动符号对主程序不可见,Kelley 将其打补丁改为 RTLD_GLOBAL,打破符号命名空间隔离,确保应用程序与图形驱动能够共享同一份动态库实例与缓存。 显示协议权衡与推动平台标准化的挑战 (25:13) 针对社区提出用纯 Zig 语言重写 X11 协议以剔除动态库依赖的想法,Kelley 解释称,Vulkan 或 OpenGL 图形驱动本身已对 X11 共享库存在硬性依赖,因此这些库必然会被加载到进程内存中,主程序直接复用现有共享库反而是最可靠且避免重复开销的做法。对于 Wayland 显示服务,程序只需在运行时优先尝试 dlopen 加载 libwayland-client.so,若失败再回退至 X11 或 XWayland 即可,理论上不会带来额外性能损失。 尽管该原型程序成功在 Ubuntu、NixOS、Void Linux(包含 glibc 与 musl 双版本)、Arch 和 Clear Linux 上开箱即用地渲染出三角形,Kelley 坦言此方案目前不适合直接投入生产:Khronos 维护团队反对静态链接 Vulkan-Loader,且现代编译器工具链(如 LLD)若检测到无动态依赖便会自动剔除 .dynamic 段,容易直接破坏这一黑客技巧。他强调该演示旨在证明技术可行性,并呼吁 Valve 等行业领头企业利用影响力推动 Linux 发行版与图形库维护者正视玩家体验,为灵活的混合动态加载确立正式标准。 Kelley 的概念验证证明,通过精巧的混合 ELF 构造与运行时自重载,跨 Linux 发行版的独立游戏二进制在技术上完全可行;但该方案目前仍高度依赖未经上游合并的私有补丁,且极易被工具链升级破坏。若要彻底终结 Linux 桌面游戏的分发困境,亟需 Khronos 加载器、LLVM 链接器规范以及发行版社区在系统底层对便携式混合二进制给予官方支持。

Detailed notes — zh

## Background 本视频由 YouTube 频道 Zig SHOWTIME 于 2021 年 2 月 19 日发布,内容源自线上活动 Zig SHOWTIME 第 21 期。视频简介附带了 Zig SHOWTIME 通讯的订阅链接,整场分享分为技术演讲与观众问答两大部分,由 Zig 语言创始人 Andrew Kelley 探讨 Linux 原生游戏的分发难题。 ## TL;DR 针对 Linux 平台上游戏二进制文件因硬编码动态链接器路径(.interp)与 libc 碎片化导致难以跨发行版分发的痛点,Andrew Kelley 提出了一个“近乎静态链接”的概念验证方案。该方案通过在编译期构建包含动态段却无动态依赖的混合 ELF 文件,让内核先将其作为静态程序直接执行 main 函数,随后在运行时自检探测宿主发行版的动态链接器与 libc(glibc 或 musl),并通过 execve 自我重执行完成环境劫持。结合魔改支持 PIC 编译与弱符号桩的静态 Vulkan Loader,他成功实现了同一个未经重新编译的 Vulkan 三角形程序在 Ubuntu、NixOS、Void Linux 等迥异发行版上的无缝运行,并呼吁驱动厂商、发行版维护者及 Valve 共同推动此使用场景的标准化。 ## Atmosphere & style 整场演讲兼具硬核极客的技术张力与轻松幽默的黑客文化氛围。Andrew Kelley 既敢于直面“Linux 桌面之年是个点击诱饵”的残酷现实,又在终端里抽丝剥茧地展示拆解 ELF 头部和底层装载器黑魔法的过程,言辞坦诚而犀利。主持人 Loris Cro 与观众在 Q&A 环节频繁抛出调侃 Gabe Newell(G胖)和 Valve 的梗,在严谨探讨动态链接符号作用域与 Wayland 兼容性的同时,始终流淌着一种对 Linux 生态“虽痛但爱”且渴望打破高墙的理想主义情怀。 ## Bulletin - [04:45](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=285s) Andrew Kelley 提出了名为 static-window-9 的概念验证,实现了同一个 Linux 原生 ELF 可执行文件跨 Ubuntu、NixOS、Arch 以及基于 musl 和 glibc 的 Void Linux 等异构发行版零重新编译直接运行并渲染 Vulkan 三角形。 - [12:38](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=758s) 通过“双模”ELF 黑魔法,可执行文件在构建期链接虚拟库生成动态段,随后用 patchelf 剔除 NEEDED 依赖和 PT_INTERP 解释器段,让内核将其视作静态程序启动,避免因硬编码动态链接器路径在不同系统上提前崩溃。 - [14:09](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=849s) 程序在进入 main 后通过 libc 弱符号自检运行环境,若为静态启动则动态探测宿主 ld.so 并配置 LD_PRELOAD,最后通过 execve 自我重执行,安全接入宿主系统的动态链接器和硬件驱动。 - [20:55](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1255s) 技术实现魔改了 Vulkan Loader,将其以 PIC 静态编译并打补丁将 dlopen 作用域改为 RTLD_GLOBAL,呼吁驱动厂商、发行版维护者及 Valve 正式认可并支持该静态与动态混合使用场景。 ## Key takeaways - [01:36](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=96s) 桌面操作系统市场份额被 Windows 垄断的核心根源在于游戏,这导致开发者与玩家之间形成了难以打破的自强化正反馈闭环。 - [06:20](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=380s) Linux 跨发行版分发游戏的最大障碍在于 libc 碎片化(如 glibc 与 musl)以及 ELF 二进制硬编码 .interp 解释器物理路径导致的提前崩溃。 - [08:54](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=534s) 动态链接器自身也是可执行文件的特性启发了“混合 ELF”设计:先以纯静态方式进入 main,再在运行时自主选择并启动系统动态链接器。 - [12:38](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=758s) 构建时引入哑元库并在后续抹除 NEEDED 标签,创造出了既有 .dynamic 段却无任何硬编码动态依赖的双重身份可执行文件。 - [14:09](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=849s) 程序利用 dlopen 弱符号在未解析时为 NULL 的特性,精准判断当前处于伪静态启动还是动态链接器二次唤醒状态。 - [14:09](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=849s) 复用 Zig 标准库的平台探测能力,程序在运行时自动定位宿主系统的 ld.so 路径,重组 argv 并借助 execve 覆盖自身。 - [19:10](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1150s) 针对 glibc 与 musl 初始化行为和符号暴露的巨大差异,运行时会判定目标 ABI 并对 musl 走显式调用 __libc_start_main 的定制装配路径。 - [20:55](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1255s) Vulkan Loader 被裁剪并以位置无关代码(PIC)静态编入程序,通过手写 weak 属性的 libc.h 头文件避免在静态阶段符号崩溃。 - [25:13](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1513s) 原生 Vulkan Loader 默认采用 RTLD_LOCAL 局部作用域会导致符号隔离,将其魔改为 RTLD_GLOBAL 全局加载可使主程序和图形驱动复用底层库缓存。 - [25:13](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1513s) Vulkan Loader 上游目前并不鼓励静态链接,导致该方案在缺少官方支持的情况下维护分支同步上游代码的成本极其高昂。 - [32:16](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1936s) 在 Zig 中重新实现一套纯 X11 协议并不能做到真正的纯净,因为底层的硬件图形驱动自身仍然强依赖 X11 动态链接库。 - [36:55](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2215s) 针对 Wayland 桌面协议,程序可以采用先尝试打开 Wayland 驱动对象、失败则降级回退到 X11/XWayland 的双重兼容设计。 - [38:40](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2320s) ELF 的 DT_NEEDED 虽可只填相对库名,但 RPATH/RUNPATH 的跨发行版失效问题使得依赖系统全局搜寻机制变得不可靠。 - [29:12](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1752s) 黑魔法式的构建方案容易在底层工具链(如 LLD)升级时发生功能回退,因此需要工具链与平台正式确立该模式的合法支持。 - [41:11](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2471s) 呼吁 Valve 运用在 Linux 游戏生态中的强大号召力,推动驱动厂商与各发行版维护者共同解决这一人为制造的兼容性痛点。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=0s) 开场:一个跨发行版的“静态链接梦” 活动主持人进行了一段富有幽默感的热场开场,打趣 Valve 掌门人 Gabe Newell(Gaben)说:“Gaben,我知道你在看,放下《Half-Life: Alyx》第二章吧,你该看的是这个。”主持人宣布演讲主题为《2021:Linux 游戏桌面之年》(2021: Year of the Linux Gaming Desktop),并指出主讲人 Andrew Kelley 怀揣着一个梦想——一个“近乎静态链接的梦想”(an almost statically linked dream),让视频游戏能够被打包成分发给所有 Linux 平台的单一独立可执行文件,不受具体发行版差异的束缚。随后,Andrew Kelley 登台接管讲台。 ### [01:36](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=96s) 桌面操作系统的垄断闭环与“标题党”真相 ```mermaid flowchart TD accTitle: 游戏生态与操作系统的正反馈闭环 accDescr: 展示玩家、开发者与 Windows 系统之间的自强化正反馈循环 A["Windows / macOS 拥有绝大多数玩家"] -->|"玩家群体庞大"| B["游戏开发者优先选择 Windows (为了盈利)"] B -->|"游戏库高度集中在 Windows"| C["新老玩家只能继续留在 Windows"] C -->|"不断加固市场份额"| A ``` Andrew 直入主题,调侃自己使用了极具“点击诱饵”(clickbait)属性的标题。“大家总在讨论:今年会是 Linux 桌面之年吗?不,并不是。”他抛出了核心论点:Windows 和 macOS 之所以比 Linux 受欢迎,原因只有一个,那就是游戏。根据 Steam 软硬件调查数据,操作系统市场份额的饼图中绝大部分被蓝色的 Windows 占据,而属于 Linux 的黄色小扇区微乎其微。游戏开发者不是傻子,他们的目标是赚钱,如果不瞄准蓝色的主要阵营就是跟商业规律作对。这形成了一个难以打破的自强化闭环(feedback loop):玩家去有游戏的地方,开发者去有玩家的地方。Andrew 直言自己无法打破这个闭环,但他带来了一个切实可行的概念验证(Proof of Concept),旨在消除 Linux 玩家运行游戏时的繁琐壁垒,让体验变得无比顺畅,哪怕只能让饼图里的黄色区域稍微变大一点点。 > “如果你的目标是赚钱,而你却没有瞄准饼图上的蓝色部分,那你就是做错了;如果你去瞄准黄色部分,非常感谢你,但这显然赚不到钱。” —— Andrew Kelley ### [04:45](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=285s) 概念验证:一个跑在所有主流发行版上的 Vulkan 三角形 Andrew 演示了一个名为 `static-window-9` 的 Linux ELF 可执行文件,甚至把下载 URL 挂在演示文稿上供大家验证,并打趣道名字里带个“9”纯粹是因为之前失败了整整八次。这是一个纯粹的原生 Linux 测试程序,在屏幕上绘制一个最基础的 Vulkan“Hello World”彩色三角形。它直接调用底层 X11(或在 Wayland 环境下通过 XWayland 协议)与本机的 Vulkan 显卡驱动交互,甚至还没做窗口缩放和退出处理,关掉它必须在终端手动 kill。 这个看似极简的 Demo,其核心突破在于同一个未经重新编译的二进制文件,直接在极其迥异的 Linux 发行版环境里完美运行: - Ubuntu - NixOS - 基于 musl 的 Void Linux - 基于 glibc 的 Void Linux - Clear Linux - Arch Linux - Debian Sid 针对不熟悉该领域的观众可能产生的轻视,Andrew 直言:“在 Windows 上搞出这个当然轻而易举,从游戏开发平台的角度看,Windows 在这方面确实远胜 Linux。”在 Linux 上,即便只是一个简简单单的窗口三角形,其背后横亘着操作系统架构设计强加给开发者的重重高墙。 ### [06:20](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=380s) Linux 跨发行版分发的核心症结:libc、动态链接器与硬编码路径 ```mermaid flowchart LR accTitle: 传统 Linux 二进制文件的动态加载故障 accDescr: 传统 ELF 文件硬编码动态链接器路径导致在不同 libc 或 NixOS 发行版上加载失败 subgraph ELF["标准 ELF 二进制 (如 vkcube)"] INTERP["硬编码 INTERP 路径<br/>(/lib64/ld-linux-x86-64.so.2)"] DYN["动态依赖列表:<br/>libX11.so, libc.so, libvulkan.so"] end subgraph Distros["不同 Linux 环境"] GLIBC["glibc 系统<br/>(路径通常匹配)"] MUSL["musl 系统<br/>(找不到 ld-linux 路径)"] NIX["NixOS<br/>(/nix/store/... 特殊路径)"] end INTERP -->|"路径存在"| GLIBC INTERP -.->|"路径不存在: 崩溃报错"| MUSL INTERP -.->|"路径不存在: 无法进入 main()"| NIX ``` 为了剖析问题本质,Andrew 现场通过 `ldd` 和 `readelf` 命令检查了标准的 `vkcube` 示例程序: - **驱动强依赖动态链接**:在 Linux 上调用系统的硬件 Vulkan 驱动,必须通过系统的 libc 进行动态链接。 - **libc 碎片化**:不同发行版采用完全不同的标准 C 库,最典型的分歧就是 glibc 与 musl。 - **动态链接器解释器路径写死**:在传统的动态链接 ELF 文件中,存在一个名为 `.interp` 的段(section),里面硬编码了系统动态链接器的物理路径(例如 glibc 系统的 `/lib64/ld-linux-x86-64.so.2`)。 当这个二进制文件被放到 musl 发行版或 NixOS(其库文件全部放在特殊的 `/nix/store/...` 哈希路径下)时,内核根本找不到硬编码的解释器,程序甚至连 `main` 函数都还没执行进去就会直接被系统报错终止。这正是 Steam 等平台不得不声明“仅官方支持特定版本 Ubuntu”的根本原因——为了回避碎片化,他们只能人为画地为牢。 > “在不同的 Linux 发行版上,问题在于动态链接器代码是在你的 `main` 之前运行的。我们甚至没办法在代码里写逻辑去解决这个问题,因为在我们的代码被执行之前,问题就已经发生了。” —— Andrew Kelley ### [08:54](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=534s) 自我重执行策略与“混合”ELF 文件的实现构想 在苦思冥想却没有落地的阶段,Jonathan Marler 的探索给了 Andrew 极大的启发。Jonathan 发现 Linux 的动态链接器本身其实也是一个可执行文件,如果你将动态链接器作为一个可执行命令运行,并把你的目标程序作为第一个参数传给它(例如 `/path/to/ld.so./my_program`),它就能动态加载并启动该程序。Jonathan 编写了一个概念验证:程序首先以纯静态方式启动,随后在运行时通过 `execve` 将自己传递给硬编码的动态链接器完成二次启动。 ```mermaid flowchart TD accTitle: 混合可执行文件的自启动与动态加载流程 accDescr: 展示静态/动态混合二进制从执行检测到获取驱动并渲染的全过程 S1["静态启动 (无.interp 段,内核直接交由 main)"] --> S2{"检查运行时环境:<br/>是否已在动态链接器内运行?"} S2 -->|"否"| S3["在系统中自动探测当前系统的 ld.so 路径"] S3 --> S4["调用 execve(detected_ld, argv, envp) 自我重执行"] S4 --> S2 S2 -->|"是"| S5["识别当前宿主 libc (glibc 或 musl 等) 并 dlopen"] S5 --> S6["动态查找并挂载宿主系统的 Vulkan 驱动"] S6 --> S7["进入游戏主逻辑,开始绘制渲染"] ``` 受到 Jonathan 的启发,Andrew 决定设计一套更为彻底的自包含策略: 1. **构造混合 ELF 文件**:二进制本身不包含任何动态依赖项,彻底剔除 `.interp` 段,使其在内核眼中是一个货真价实的“静态二进制”,因此可以在任何发行版上直接调用 `main`; 2. **保留 `.dynamic` 段**:虽然没有 `.interp`,但保留 ELF 规范中供动态链接器识别的元数据,使其具备“能被动态链接器当做普通动态程序加载”的能力; 3. **运行时环境自检**:在进入 `main` 函数的第一时间,检测当前进程是直接静态运行的,还是已经处于某个动态链接器内部; 4. **定位解释器并自我重启**:若处于静态模式,通过代码在运行时探测出当前宿主发行版的动态链接器实际路径,随后使用该路径调用 `execve` 自我重执行; 5. **动态挂载运行时与驱动**:自我重执行进入动态环境后,自动识别出宿主是 glibc 还是 musl,通过 `dlopen` 挂接系统原生 libc 和 Vulkan 驱动; 6. **移交游戏渲染**:完成上述所有底层管道的无缝拼装后,平滑移交到用户游戏逻辑执行绘制。 Andrew 在终端中输入 `ldd static-window-9`,终端赫然返回 `not a dynamic executable`(不是动态可执行文件);然而用 `readelf -l` 检查段表时,程序确实没有 `.interp` 解释器段,却包含了一个完整的 `.dynamic` 动态段。这种“双重身份”的混合 ELF 架构正是打破 Linux 多发行版兼容墙的关键入口。 ### [12:38](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=758s) 构造“双模”ELF:兼具动态段却无动态依赖的黑魔法 为了让编译出的可执行文件既能像静态程序一样由内核直接启动,又能具备动态链接的能力,Andrew Kelley 详细拆解了底层 ELF 结构的精妙改造过程。在常规情况下,动态链接程序会包含一个动态段(Dynamic Section),并且有一系列 `DT_NEEDED` 标签指明其依赖的具体共享库,同时包含一个 `PT_INTERP` 段指示内核调用哪个动态链接器(如 `/lib64/ld-linux-x86-64.so.2`)。为了绕过这种硬编码限制,他们采取了一套非常巧妙的构建流程: 在编译该可执行文件时,构建系统首先生成一个虚拟的哑元库(dummy library),并让可执行文件以动态共享的方式链接该库。这一步确保了链接器会在生成的二进制文件中生成完整的动态段(Dynamic Section)。紧接着,他们使用 `patchelf` 工具执行 `--remove-needed` 指令,将刚刚写入的 `libdummy` 依赖项硬生生抹除。由于原本就只依赖这一个哑元库,依赖项被删去后,ELF 文件中保留了完整的动态段,却没有任何 `NEEDED` 标签,也不包含 `PT_INTERP`。因此,Linux 操作系统在加载它时不会尝试去调用任何特定的动态链接器,而是直接将其作为静态程序执行。正如 Andrew Kelley 总结道: > “So what this means is that we have a executable that supports being run both dynamically and statically so it has a dynamic section but no dynamic dependencies and the os or linux will not try to run it with a dynamic linker because there's no interp entry.” ```mermaid flowchart TD accTitle: 构建无依赖动态ELF可执行文件的流程 accDescr: 展示如何通过dummy库生成动态段,再利用patchelf清除依赖以实现双模运行。 subgraph Build["编译构建阶段"] A["生成哑元库 libdummy"] --> B["动态链接编译可执行文件"] B --> C["生成带有 Dynamic Section 的 ELF"] end subgraph Patch["修改 ELF 阶段"] C --> D["patchelf --remove-needed libdummy"] D --> E["保留 Dynamic Section 但无 NEEDED 标签"] E --> F["去除 PT_INTERP 解释器段"] end subgraph Exec["内核加载阶段"] F --> G["Linux 内核直接作为静态二进制执行"] end ``` - 该机制的核心在于骗过系统:既提供了后续动态装载所需的结构,又避开了操作系统在 `exec` 时的依赖校验。 - 二进制文件同时支持静态直接运行与动态链接器重新接管两种状态。 --- ### [14:09](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=849s) 首次启动与二次唤醒:`dlopen` 弱符号与自我 `execve` 当控制流进入程序的 `main` 函数时,面临一个极其特殊的挑战:当前进程可能是用户第一次直接敲击命令启动的(此时没有动态链接器,处于伪静态状态),也可能是后面被动态链接器重新装载进来的。程序必须在没有任何运行时辅助的情况下判断自己的处境。 为此,代码对 `dlopen` 符号使用了一个弱外部引用(`weak extern`)。`dlopen` 是由 libc 提供的用于在运行时动态加载库的核心函数。弱符号机制的精妙之处在于:如果该符号在链接/加载时未解析,其地址会被置为 NULL 而不会引发未定义符号报错。如果进入 `main` 时 `dlopen` 指针为空,说明当前运行在静态模式下,需要先找到动态链接器并重定向自身;如果 `dlopen` 存在有效地址,则说明当前已成功运行在动态链接器环境内,可以安全地进行后续操作。 在首次启动的分支中,程序需要探测当前宿主操作系统的动态链接器位置。得益于 Zig 编译器的通用设计,这些基础设施早已封装在 Zig 的标准库中,可以直接调用 `detectNativeTargetInfo` 探测宿主发行版并获取其动态链接器的确切路径。接下来,代码重构 `argv` 参数数组:将探测到的动态链接器路径作为 `argv[0]`,把可执行文件自身的路径和原始命令行参数顺次向后偏移插入。此外,由于之前利用 `patchelf` 剥离了所有的 `NEEDED` 依赖项,即便动态链接器启动也不会自动载入基础库,因此程序还要篡改环境变量,借助 `LD_PRELOAD` 强制把 `libdl` 与 `libpthread` 注入进去。完成参数与环境准备后,程序调用 `execve` 覆盖自身。 在解释参数排列的实现细节时,Andrew 还自我纠正并澄清了代码细节: > “All we have to do is shift them over and put the dynamic linker first yeah I'm sorry for the confusion so that's for the argument arguments now we also have to mess with the environment variable a little bit because what we're going to need to use dlopen which is in libdl and we're going to need to use um some p thread stuff for mutexes.” 一旦通过 `execve` 自我替换并伴随 `LD_PRELOAD` 被动态链接器唤醒,程序第二次踏入 `main` 函数,此时 `dlopen` 已经是非空有效指针,整个动态装载通路全部打通。 --- ### [19:10](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1150s) 抹平差异:跨越 Glibc 与 Musl 的运行时适配 重新在动态链接器中苏醒之后,挑战依然没有结束。Linux 世界并非大一统,不同的发行版使用的 C 标准库(libc)大相径庭,主流包括 glibc(如 Ubuntu、Fedora、Arch)以及轻量级的 musl(如 Alpine Linux)。两者的初始化行为与符号分布存在明显差异。 在搭载 glibc 的环境(例如演讲现场演示的 `vkcube` 依赖项)下,系统动态链接器往往会一口气把几乎所有必要的全局环境与符号全部准备妥当,代码可以直接顺畅地进入核心逻辑。但在 musl 环境中,整体实现则显得更加“极简原生”(bare bones)。为了让 musl 正常工作,程序必须编写专门的适配代码:手动调用 musl 提供的 `__libc_start_main`,显式传递退出回调函数指针,并针对 musl 环境补齐一系列特定 libc 符号的 `dlopen` 查询。 为了解决这一分歧,代码再次复用了 Zig 标准库中的目标检测逻辑。当程序运行在未知 Linux 系统上时,它会主动分析本机的 ABI 和 libc 属性: > “Because the Zig compiler can detect your libc so we want to detect the native triple right there we detected that it's glibc that same code can be used to detect do we want a dynamic linking against musl or do we want a dynamic link against glibc at runtime.” 通过在运行时判定当前究竟是在与 glibc 还是 musl 打交道,程序采取对应的装配路径,最终统一步调汇入第二阶段的入口函数 `main2`,准备开始调用真正的图形与窗口系统 API。 --- ### [20:55](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1255s) 魔改 Vulkan Loader:PIC 静态编译与弱符号 libc 桩 越过了 libc 的壁垒,下一个难以逾越的障碍是 Vulkan Loader。Vulkan Loader 是 Khronos Group 维护的核心官方项目,其职责是探测系统中安装的各种物理显卡 Vulkan 驱动(ICD),并将官方 API 函数指针精确派发给实际的驱动实现。通常它是一个系统预装的动态链接库,直接静态编译它会引入灾难性的宿主依赖问题。Andrew 对 Vulkan Loader 源码进行了深度魔改(hack up),将其改造成一个内嵌的静态模块: ```mermaid flowchart TD accTitle: 魔改 Vulkan Loader 的静态与动态兼容方案 accDescr: 说明如何提取部分源码、利用Zig C编译能力生成PIC静态库,并用弱符号libc隔离静态与动态阶段。 A["提取 Vulkan Loader 核心源码"] --> B["配置编译脚本(Zig + libclang)"] B --> C["强制启用位置无关代码(PIC)"] C --> D["静态编译进游戏执行文件"] E["手写自定义 libc.h 头文件"] --> F["为所有 libc 函数附加 __attribute__((weak))"] F --> D D --> G{"首次静态运行阶段?"} G -- "是" --> H["所有 libc 符号为 NULL,严禁调用 Vulkan 代码"] G -- "否(二次动态运行)" --> I["libc 符号已由链接器填充有效地址"] I --> J["dlopen('libX11.so', RTLD_GLOBAL) 暴露符号给底层显卡驱动"] ``` 1. **按需精简与 PIC 编译**:Andrew 没有全量编译整个项目,而是挑选出真正必需的核心源码,借助 Zig 内部集成的 C 编译能力(基于 libclang)进行构建。他在构建脚本中定制了专属的 C 宏定义,并强制开启位置无关代码(Position Independent Code, PIC),从而使静态内嵌的代码在后续动态运行阶段也能合法重定位。 2. **弱符号伪装 libc 接口**:内嵌的 Vulkan Loader 源码充斥着对标准 libc 函数的调用(如 `memcpy`、`memset`、`putchar`、`fclose` 等)。如果直接编译,第一次以静态方式加载运行时,这些 libc 符号根本不存在。为了防止在首轮启动时报错崩溃,Andrew 手写了一个自定义的 `libc.h` 头文件,将该工程依赖的所有 libc 函数声明全部加上了 `__attribute__((weak))` 属性。这就意味着,在未加载动态 libc 时,这些符号全部静默解析为 NULL;一旦在动态链接阶段被重入,它们就会被宿主系统的真实符号覆盖。正如 Andrew 所强调的: > “If we tried to call inside here if we tried to call any of the vulkan loader stuff it would crash because every single libc function like memcpy memset you know putchar fclose close or all these they would all just segmentation fault because they're all zero they're all null because they're weak.” 3. **动态加载标志位的修补(`RTLD_GLOBAL`)**:除了符号适配,Vulkan Loader 与显示驱动(如 X11)之间还有深层联动。最终进入 `main2` 时,程序需要通过 `dlopen` 加载宿主系统的 `libX11` 共享库。显卡厂商的私有驱动往往隐式依赖该库中导出的符号。原本的局部加载模式会导致后续驱动找不到相关上下文,因此 Andrew 打上补丁,强制使用 `RTLD_GLOBAL` 模式载入 X11 库,使得导出的符号全局可见,确保当 Vulkan Loader 触发驱动装载时,底层的硬件驱动能够无缝完成符号绑定。 ### [25:13](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1513s) 补丁 Vulkan Loader 与静态链接的争议 为了确保动态加载器(dynamic loader)能够命中缓存并复用 Vulkan 驱动已经加载好的动态库,Andrew Kelley 解释了他为何必须修改 Vulkan Loader 的源码。如果加载器采用局部加载(local load,即 `RTLD_LOCAL`),这些库就会处于独立的命名空间中,外部无法复用;而将其修改为全局加载(global load,即 `RTLD_GLOBAL`)后,就能从机制上“保证”主程序可以拿到同一份已被驱动载入内存的库,从而避免重复加载或符号隔离的问题。 ```mermaid flowchart TD accTitle: Vulkan Loader dlopen 行为对比 accDescr: 对比原生 Vulkan Loader 的 RTLD_LOCAL 局部加载与打补丁后的 RTLD_GLOBAL 全局加载。 subgraph Default["原生行为:局部加载"] A1["Vulkan Loader dlopen"] -->|"使用 RTLD_LOCAL"| B1["独立隔离命名空间"] B1 --> C1["主程序 dlopen 无法命中缓存\n报错找不到已加载库"] end subgraph Patched["打补丁后:全局加载"] A2["修改版 Vulkan Loader"] -->|"改为 RTLD_GLOBAL"| B2["全局符号表共享"] B2 --> C2["主程序直接复用驱动已载入的动态库"] end ``` 在做出这套弱链接(weak linkage)和补丁方案的同时,Andrew 也给出了免责声明(caveat emptor)。他在 Discord 讨论组中曾与 Vulkan Loader 的维护人员交流过,对方明确表示不喜欢开发者静态链接 Vulkan Loader: > “there's some reason that it's they don't want you to static link it... but it does make the game work so um i don't know maybe they should consider this a use case” > (他们出于某种原因不希望你静态链接它……但这确实能让游戏跑起来,所以我不知道,也许他们应该把这视为一个正当的使用场景。) Andrew 指出,Linux 发行版维护者应当与 Vulkan Loader 团队通力协作,确保静态链接能够得到官方支持,因为玩家的诉求极其纯粹——他们只是想顺畅地运行游戏。但从工程现实来看,这目前仍然只是个概念验证(PoC):Vulkan Loader 上游代码库提交非常活跃,而 Andrew 采用的是直接复制代码打补丁的做法,这意味着他必须维护一个分支(fork),不断手动同步上游代码,带来了巨大的维护负担。 --- ### [27:07](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1627s) X11 交互、交叉编译陷阱与三角形 Demo 在搞定了动态加载的底层障碍后,剩下的工作就是编写实际的应用程序代码:调用 X11 API 创建窗口并初始化 Vulkan 上下文。从网络协议的本质来说,这些调用最终都只是通过 Socket 向 X11 服务器发送消息;但为了保证最大兼容性,最可靠的手段依然是调用那些图形驱动本身所依赖的 glibc 动态链接库(`.so`),确保整个进程共用同一套底层基础。 然而,在尝试交叉编译时又遇到了阻碍: - Vulkan Loader 依赖一系列 X11 头文件(`.h`),但交叉编译时绝对不能直接使用主机系统(host system)本地的文件。 - Andrew 不得不钻进 X11 那套古怪繁琐的构建系统中,研究这些头文件究竟是如何生成的。 - 他随后调整了其中的宏定义(`#define`),使它们适应交叉编译环境。 - 幸运的是,这些头文件定义的只是网络协议常量,由于 ABI 兼容性约束,这些常量不能被上游随意改动。 - 攻克这些障碍后,应用层逻辑其实非常轻量,仅仅是按照 `vulkantutorial.com` 教程编写的基础三角形渲染代码。 --- ### [29:12](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1752s) 对生态的呼吁与工具链长期支持的思考 在演示完三角形 Demo 之后,Andrew 直言不讳地指出了现状的残酷。对于那些希望用 C 或 C++ 复制这一思路的游戏开发者,他的评价是:目前这在生产中“并不现实”,只是一个证明可行性的概念验证,开发者必须跨越重重复杂的障碍。但他强调,这场演示的核心目的并非提供一个即插即用的方案,而是呼吁整个 Linux 生态反思: > “we do need driver vendors then linux distribution maintainers to explicitly acknowledge this use case” > (我们确实需要驱动厂商和 Linux 发行版维护者明确承认这一使用场景。) 许多障碍纯粹是人为制造且毫无必要的。Linux 玩家配得上开箱即用的良好体验,社区需要共同发声提高标准。此外,该项目近期在 Zig 自身的更新中发生过功能回退(regression)——因为 Zig 优化了向 LLD 传递参数的方式,导致原先的黑魔法编译失效。 Andrew 由此总结道:类似“既没有任何动态依赖、却又必须生成动态段(dynamic section)”的混合二进制,在常规链接器逻辑中会被判定为无意义并直接剔除。如果工具链和平台不将这种使用场景列为“正式认可的标准用例”,任何仅仅依靠奇技淫巧拼凑的方案,迟早会在未来的工具链迭代中被意外破坏。 --- ### [32:16](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=1936s) Q&A 环节:重写 X11 协议还是复用驱动依赖? 演示结束后,主持人 Loris Cro 接管主持并开启 Q&A 环节。Loris 表示深有同感,正是因为 Linux 生态在软件体验上的诸多折腾,即便他不喜欢苹果,他的主力机依然是一台 Mac。 在线观众 gw1 提问:社区中已有像 `juicebox` 这样的尝试,能否直接在 Zig 里重新实现一套纯 X11 协议,从而彻底摆脱对 `libxcb` 等 X11 动态库的依赖? Andrew 给出了层层递进的逻辑推导来否定该做法的实际意义: > “premise one we need to use the graphics drivers, premise two the graphics drivers use the x11 shared objects... premise three we're paying the cost therefore we're paying the cost for graphics for x11 shared objects no matter what, conclusion you should just use them since you have to” > (前提一:我们必须使用图形驱动;前提二:图形驱动本身依赖 X11 共享库;前提三:无论如何,我们都已经为此付出了代价;结论:既然你逃不掉要为它买单,不如直接使用它们。) 因为只要游戏要调用 Vulkan 或 OpenGL,驱动本身就会把 X11 动态库拉入进程的虚拟内存空间中,纯原生协议重写并不能带来内存或依赖上的绝对干净。 随后,观众 zakus 调侃询问 Valve 的赞助何时到位,Andrew 顺势开玩笑喊话 Valve 赶紧资助 Zig,并豪言“五年之内所有人都会用 Zig 来写游戏”;Loris 也幽默附和,戏称自己每次在《反恐精英:全球攻势》(CS:GO)里用 Tec-9 手枪打不中人时,都会直接给 Gabe Newell(G胖)打电话抱怨。 接着,观众提问探讨 libc 弱符号与 `RTLD_GLOBAL` 的具体成因。在经历短暂的推流屏幕排错后,Andrew 开始拆解底层执行细节:他最初在未给 Vulkan Loader 打补丁前尝试直接 `dlopen` 一个已被加载的动态库,但动态加载器直接报错提示找不到,根本原因就在于 Vulkan Loader 默认采用局部作用域,导致其对外部隔离不可见。 ### [36:55](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2215s) 动态链接符号作用域与 Wayland 兼容性 在问答环节的开头,Andrew Kelley 针对观众关于动态链接符号可见性的提问进行了补充说明。他指出,由于 Vulkan Loader 在内部使用本地作用域(`RTLD_LOCAL`)加载对象,导致随后尝试获取时因对象归属不同而失败。只有当把它们切换为全局符号可见(`RTLD_GLOBAL`)时,系统才能稳定返回有效对象。 随后,主持人转达了一个关于桌面协议的提问:要让这套技术方案在 Wayland 环境下工作,需要做哪些修改?Andrew 首先开玩笑说“什么都不用改”,因为 Wayland 原生支持 X11 协议(XWayland)。 ```mermaid flowchart TD accTitle: Wayland 与 X11 双协议支持回退流 accDescr: 描述程序在运行时优先尝试 Wayland 协议并在失败时降级回退到 X11 的逻辑流程。 subgraph Init["协议加载逻辑"] A["启动应用"] --> B{"尝试打开 Wayland 对象"} B -->|"成功"| C["使用 Wayland 图形驱动协议"] B -->|"失败"| D["回退打开 X11 (XCB) 对象"] D --> E["使用 X11 协议驱动运行"] end ``` 谈及更认真的实现方案时,Andrew 说明了如果要在程序层面显式支持 Wayland 该如何处理:代码只需要在运行时先尝试打开 Wayland 对象,若获取成功且底层图形驱动依赖 Wayland,就直接使用该对象;若失败,再回退到现有的 X11 逻辑,整个过程非常直接。Andrew 强调了自己的技术立场:“我强烈认为,如果 Wayland 做得对,这套代码在 Wayland 上应该完全能直接运行,而且几乎没有任何性能惩罚。”(“I am of the strong opinion that if Wayland is doing it right, this code should all just work fine on Wayland with almost no penalties.”) ### [38:40](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2320s) ELF DT_NEEDED、RPATH 与上游 Vulkan Loader 补丁 主持人随后念出了观众关于动态链接器与 ELF 头部信息的疑问:ELF 结构中的 `DT_NEEDED` 字段是否可以只写动态库文件名(如 `libname.so`)而省略绝对路径?为了解答该问题,Andrew 展示了 `vkcube` 作为传统链接方式的对照测试案例。在 `vkcube` 的依赖项中,`DT_NEEDED` 确实只记录了 `libxcb`、`libc` 以及 Vulkan 驱动的相对文件名,并非绝对路径;但实际的绝对路径是通过 `RPATH` / `RUNPATH` 引入的。这导致了跨发行版的核心痛点:程序在编译阶段根本无法预知运行时的具体驱动路径,写死在某个系统上的 `RUNPATH` 换到其他 Linux 发行版就会完全失效。 ```mermaid flowchart LR accTitle: Vulkan Loader 动态加载驱动调用链 accDescr: 展现应用通过 Vulkan Loader 调用 dlopen 加载显卡驱动以及上游补丁方案的关系。 subgraph App["应用代码"] A["应用二进制"] end subgraph Loader["Vulkan Loader"] B["初始化函数"] C["dlopen 显卡驱动"] end subgraph Driver["显卡驱动"] D["libvulkan_*.so"] end A -->|"调用"| B B -->|"dlopen (建议上游设为 RTLD_GLOBAL)"| C C --> D ``` 针对观众提出的“如果修改 Vulkan 使其使用 `RTLD_GLOBAL` 是否就能直接复用动态库”的疑问,Andrew 澄清这并非修改驱动本身,而是修改 Vulkan Loader: - 应用程序在初始化阶段调用 Vulkan Loader 内部的一系列初始化函数。 - 真正执行 `dlopen` 去动态加载底层显卡驱动动态库的代码位于 Vulkan Loader 中。 - 团队完全可以向上游提交补丁让 Vulkan Loader 采用全局作用域;尽管上游可能存在反对意见,但在此场景下这是最正确的技术解法。 ### [41:11](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=2471s) Valve 对 Linux 生态的影响力与中场总结 在解答完观众技术提问后,Andrew 谈到了 Linux 游戏生态的整体发展。他肯定了 Valve 和 Steam 在推动 Linux 游戏兼容性方面取得的成果,指出 Valve 已经做了大量卓有成效的工作,甚至不少游戏在 Linux 上的运行表现已经超越了 Windows 原生环境。 Andrew 表达了对整个生态系统的期许:“我真诚地希望 Valve 的团队能看到这个简短的技术演讲并受到启发,运用他们的影响力和号召力,帮助 Linux 发行版生态更好地支持这一使用场景。我认为用户想要它,玩家想要它,游戏开发者也想要它,而最终 Linux 的维护者与包管理器维护者也会顺应大家的诉求;我们只需要让更多人渴望实现这一点。”(“I sincerely hope that people at Valve watch this short talk and get inspired and use their influence and power to help the Linux distribution ecosystem support this use case better. I think that the users want it, the players want it, the game developers want it, and ultimately the Linux maintainers, the package maintainers are going to do what people want, so we just have to make people want this more.”) 最后,主持人总结并感谢了 Andrew 的分享,宣布直播进入中场休息,随后下半场将继续讨论 Zig 语言的发展路线图(Zig Roadmap)。 ## Notable quotes > [06:20](https://www.youtube.com/watch?v=pq1XqP4-qOo&t=380s) 在不同的 Linux 发行版上,问题在于动态链接器代码是在你的 main 之前运行的。我们甚至没办法在代码里写逻辑去解决这个问题,因为在我们的代码被执行之前,问题就已经发生了。 — Andrew Kelley ## Mentioned Andrew Kelley, Loris Cro, Gabe Newell, Jonathan Marler, Valve, Steam, Zig, Vulkan, Vulkan Loader, glibc, musl, X11, Wayland, XWayland, XCB, NixOS, Ubuntu, Void Linux, Clear Linux, Arch Linux, Debian Sid, Alpine Linux, patchelf, LLD, CS:GO, Half-Life: Alyx ## Video description From Zig SHOWTIME #21 Subscribe to the Zig SHOWTIME Newsletter https://zig.show 0:00 Title 1:35 Talk 32:15 Interview [Source](https://www.youtube.com/watch?v=pq1XqP4-qOo)

Source quotes

[6:20] Andrew Kelley: 在不同的 Linux 发行版上,问题在于动态链接器代码是在你的 main 之前运行的。我们甚至没办法在代码里写逻辑去解决这个问题,因为在我们的代码被执行之前,问题就已经发生了。

Source quotes

[35:19] Andrew Kelley: Hey Valve, you want to sponsor Zig, you should definitely do it: everyone's going to be writing games in Zig in five years. [25:13] Andrew Kelley: this is what players want: they want to run the game.

Original transcript

[0:00] to the title [0:17] andrew has a dream a almost [0:21] statically linked dream where video [0:22] games can be built and distributed for [0:24] linux as distro agnostic executables [0:27] gaben i know you're watching put down [0:30] half life alex episode 2 [0:32] because you might want to watch this the [0:34] talk is [0:35] 2021 year of the linux gaming desktop [0:38] andrew [0:39] this room is yours [1:36] okay i think people can hear now let's [1:38] go what's up everybody [1:40] it's andrew remember me i'm that guy [1:43] um all right let's talk about 2021 [1:48] year of the linux desktop [1:51] i'm going to tell you about how i made a [1:54] proof of concept [1:55] for portable linux game executables that [1:58] can run on any [1:59] linux distribution everyone always talks [2:03] about [2:03] year of the linux desktop is it going to [2:05] be this year [2:07] no it's not but let's explore this idea [2:10] why are windows and mac os more popular [2:13] than linux [2:14] it's just because of games that's my [2:17] premise [2:18] there's the single one reason why these [2:21] operating systems are more popular [2:24] why do game developers choose windows [2:26] and mac os [2:27] over linux well because they're not [2:30] stupid they're trying to make money [2:32] and this is based on the steam hardware [2:35] software user surveys [2:37] and if your goal is to make money and [2:39] you're not targeting the blue part of [2:41] that pie chart [2:42] you're you're doing it wrong if you're [2:44] targeting the yellow part of the pie [2:46] chart [2:47] thank you but obviously that's not going [2:49] to make you money [2:53] why do gamers choose windows why is [2:55] there so much of a big blue area there [2:58] because that's where the games are we [3:00] got here is a feedback loop [3:02] uh it's just self-reinforcing uh you [3:06] can't [3:07] you know if they if the games are on [3:10] windows then [3:10] they're gonna gamers are gonna come to [3:12] windows and then the game developers are [3:13] gonna [3:14] target windows and it's just gonna it's [3:15] gonna stay there uh [3:18] i don't have an idea to break the [3:19] feedback loop it's too hard [3:21] um so is 2021 the year of the linux [3:24] desktop [3:25] nope because i can't break the feedback [3:27] loop [3:28] uh that was just a bait and switch [3:30] clickbait title [3:31] uh but i do have a idea to make the [3:33] linux gaming experience [3:35] seamless for players which i think is [3:37] worth [3:38] exploring and maybe we can widen that [3:41] yale part of the pie chart a little bit [3:45] all right demo time uh i know that [3:49] um a large part of my audience here will [3:51] be watching this uh [3:52] later on i don't know vimeo or youtube [3:54] or whatever [3:55] and it might be months weeks [3:59] years later after this talk um tell you [4:02] what i'm going to do is i'm going to [4:03] leave this url [4:04] up i won't clean my temp directory for [4:07] years [4:07] don't worry i already haven't cleaned my [4:09] temp directory for years it's [4:11] a beautiful little url here static [4:13] window nine [4:15] um you can guess why there's a nine [4:17] there it just has to do with how many [4:18] times i failed before [4:20] uh but this is a linux uh elf file [4:23] um if you trust me i promise this is [4:25] just this [4:27] little window with the with the triangle [4:29] here [4:30] um if you want to try it if you have [4:31] linux and you want to try it [4:33] uh see if it works for you after after [4:35] all this time i'm not confident that [4:37] years later it'll still work uh but [4:39] maybe let's find out [4:41] um so [4:45] what i think should happen then is if [4:46] you're on linux and if the vk [4:48] cube demo app works for you i think this [4:51] one will work for you too [4:54] that's the proof of concept so what this [4:57] application is doing [4:58] is it's actually using so it's using x11 [5:02] or if you have weyland installed it's [5:03] going to use the [5:05] uh you know extension of whaler that [5:07] supports the x11 protocol [5:09] and it's going to actually interact with [5:11] your system vulcan [5:12] drivers and just draw this triangle it's [5:14] just a hello world triangle [5:16] doesn't even support resizing or exiting [5:17] you have to kill it manually [5:19] um but i i've tested this on ubuntu [5:23] nixos muscle-based void linux glibc [5:26] baseboard linux clear linux arch linux [5:28] debian sid and it's able to the same [5:31] binary is able to run on all these [5:33] different distributions so i think this [5:35] is a compelling [5:36] idea and i want to show you all of the [5:39] the problems i had to overcome in order [5:41] to make this proof of concept work [5:46] and uh just to show you that i mean it's [5:49] i'm showing you a slide right now but i [5:51] want to [5:52] just actually run it i mean obviously i [5:56] can [5:57] like there it is right no not fooling [6:00] yet [6:01] okay so what are some of the problems [6:05] like why is this actually hard if you're [6:06] not familiar with this problem space [6:08] uh you might just be thinking wow this [6:09] is just a hello world triangle like what [6:11] wow like whoop you do that's so easy on [6:13] windows and yeah that's one way that [6:15] windows is way better [6:16] as a game development platform than [6:18] linux [6:20] because here are some of the problems [6:21] that linux introduced into this use case [6:24] that arguably don't need to be here so [6:28] first of all if you look at a normal [6:31] a normal binary [6:34] such as bk cube uh i think i need to [6:38] install [6:40] vulkan tools [6:43] if we just look at uh so this is this is [6:46] the vk [6:47] cube program if we just look at it [6:52] and we look at the dependencies that it [6:54] has [6:55] um you can see that it has a dynamic [6:57] dependency on x11 [7:00] on lib c and vulcan loader [7:04] driver and these things are [7:08] are linux distribution dependent [7:11] um so the problem is that [7:15] the way that the vulcan drivers are [7:17] available [7:18] is dynamic linking with system libc [7:21] and that's [7:24] the problem with different linux [7:26] distributions is that they all have [7:28] they can have different kinds of lipsy [7:29] so you can have muscle or you can have [7:31] glib c or you can have other [7:33] those are the two most common ones but [7:34] you can't have other [7:36] uh also [7:39] you have to hard code the dynamic linker [7:42] path so for example on the same binary [7:44] here if i look at [7:45] lead elf oh my my face is right there [7:48] isn't it [7:50] also the font size is too small if i do [7:53] read elf [7:54] on on this binary and i do i read the [7:58] dynamic uh [7:59] all right sorry i read the um the [8:01] sections here [8:02] we can see that we have this interpreter [8:04] inside of the binary [8:06] and this path right here this is the [8:08] path to the dynamic linker it's hard [8:10] coded in the binary [8:11] so if i tried to if i tried to run this [8:14] binary vk cube [8:15] on a different linux it would just say [8:17] like error [8:18] it would never execute it never even [8:19] gets main because this path wouldn't [8:21] exist on your computer [8:23] the problem is it's different on [8:24] different uh distributions on on [8:27] muscle based linuxes it's one path on [8:30] glibc distribution is another path on [8:32] nixos it's the special path [8:34] uh and so on so that's another problem [8:37] that prevents this from working [8:39] that's one of the reasons that for [8:41] example steam only supports like [8:43] ubuntu-based linux because they're [8:44] trying to eliminate all of this [8:46] different [8:47] these different situations and just only [8:49] target one thing [8:54] so how do we solve this problem because [8:56] the dynamic leader code runs before main [8:57] so it's like we can't even put code in [8:59] there [9:00] to solve the problem because the problem [9:01] has to be solved before our code runs [9:05] well jonathan marlar [9:08] uh also shares my dream about this issue [9:12] and i've been i've been talking up a [9:14] bunch of i've been talking up a storm [9:16] i've been describing this [9:17] this vaporware solution to this problem [9:19] for a while but i hadn't done anything [9:21] but jonathan marler actually went off [9:23] and he made this [9:24] uh this uh repository and explored one [9:27] of these concepts [9:28] and what he came up with was a proof of [9:31] concept of an elf binary [9:33] that you could execute statically but [9:35] then it would it would call [9:36] xxve on itself with a hardcoded path [9:40] to the dynamic linker because the way it [9:42] works is that if you just take [9:44] uh if you just take this [9:48] this thing here this dynamic linker it's [9:50] actually an executable [9:51] um even though it's a dot so and if you [9:54] pass it the first parameter to the [9:56] in exe it will run the exe [10:00] dynamically linked with with this [10:02] dynamic linker [10:03] so he he did a proof of concept on this [10:06] and it worked and it inspired me [10:09] inspired me to [10:10] it gave me the itch to take it even [10:12] further so this and [10:13] this one had a little bit of downsides [10:15] because it required [10:17] building the binary with a dynamic [10:18] dependency on a dummy shared library [10:21] to make sure that the executable had the [10:23] dynamics section [10:25] um and then also um it just had to it [10:27] had to hard code the path to the dynamic [10:29] linker [10:30] um so i was inspired to take it even [10:32] further [10:34] and i came up with a strategy [10:37] so what we want to do is we'll have [10:40] we'll have a binary that [10:44] does not have any dynamic dependencies [10:46] so it's it's statically linked [10:48] but it has a dynamic section so it [10:50] supports being run [10:52] with the dynamic linker as the [10:53] interpreter [10:55] and then inside main it will check at [10:58] runtime if we're running in the dynamic [10:59] linker [11:00] if it's not find where the dynamic [11:03] linker path is at runtime [11:05] and then use that path it detected at [11:06] runtime to exec ve itself [11:10] then it has to detect which lib c is the [11:12] system lib c and then [11:14] it can deal open lib c and then [11:17] you can use that to find and load the [11:20] vulcan driver [11:20] at runtime and then finally we can just [11:22] do the the game code to do the triangle [11:25] so that was my idea uh but there were a [11:27] lot of problems to overcome in order to [11:30] accomplish this so the first problem [11:33] is this thing where we have a hybrid [11:36] l-file [11:37] that supports both being run statically [11:40] and also being run [11:41] dynamically linked so i'll show you an [11:43] example here [11:44] i showed you ldd on the vk cube [11:48] but now i'm going to show you on the [11:51] um the binary here [11:57] it says statically linked okay but [12:01] if we take a look with read elf [12:04] and we do the l section here [12:08] we can see that there's not an interp [12:10] section so it does not [12:12] say that there has to be an uh a [12:15] dynamic loader because otherwise we [12:18] would have to put the path in there and [12:20] we don't know what the path is [12:21] but it does have a dynamic section so [12:24] let's take a look at what the dynamics [12:25] section [12:26] says so the dynamics section has all of [12:30] these entries [12:31] this is just trash i could get rid of [12:33] this that's just garbage [12:35] um the thing that's important here is [12:37] that we do not [12:38] have any needed tags so there [12:41] there is a dynamic section but there is [12:43] not any [12:45] particular dynamic libraries that it [12:48] depends on so the way that this is [12:51] accomplished [12:52] is that when we build uh [12:57] when we build this executable we [13:00] actually do [13:01] create a dummy uh library [13:06] and we make the static uh the static [13:08] window executable [13:10] link against the dummy library [13:13] dynamically linked [13:14] this one is not dynamically linked this [13:16] one is dynamically linked it's shared [13:18] so that makes sure that there's a [13:19] dynamic section but then [13:21] we use the patch elf tool and we use [13:24] remove [13:25] needed on the the dummy library and so [13:28] all this [13:28] does is it just there was one of these [13:30] entries that just said needed [13:32] lib dummy and it just deleted it that's [13:35] all this did [13:36] so we made it dynamically linked but [13:38] then we just deleted all the [13:39] dependencies [13:40] from uh so that it actually doesn't have [13:43] any dynamic libraries [13:45] that it has to run so what this means is [13:48] that [13:48] we have a executable that supports being [13:51] run both dynamically and statically [13:53] so it has a dynamic section but no [13:55] dynamic dependencies [13:57] and the os or linux will not [14:00] try to run it with a dynamic linker [14:03] because there's no interp [14:05] entry so that brings us to [14:09] uh that gets us to maine now we can get [14:11] to maine so [14:12] what does maine have to do maine has a [14:15] really interesting [14:16] problem to solve because when we when we [14:19] get here [14:20] we get into main we don't we we [14:24] we have two entry points here on one [14:26] hand we could be being run [14:27] um in the dynamic linker or we could be [14:30] on the first run where we [14:32] have to figure out where the dynamic [14:33] linker is [14:35] so in this code [14:39] in the very beginning of main it's [14:40] actually we don't know which one it is [14:42] we have to support both [14:44] so the way that we check is we actually [14:46] have a [14:47] weak x turn on the deal open [14:51] symbol and so this is an important [14:54] symbol because [14:55] first of all this symbol is provided by [14:57] libc and it lets us dynamically load [15:00] other libraries at runtime and it's neat [15:04] because [15:04] making it a weak extern symbol means [15:07] that [15:08] we support if it's not available [15:12] in which case um this code will run [15:15] because we don't have dl open because [15:17] we're being run statically [15:19] however if the symbol is available [15:22] then we go down here and we know that [15:24] we're actually running inside the [15:25] dynamic linker [15:27] so this is the code that we have to [15:29] solve first this is the code where we [15:31] have to [15:31] find the dynamic linker and then exec ve [15:35] our self so that we can re-enter main [15:38] but this time we have a dynamic linker [15:40] but we don't know which linux [15:42] distribution we're on we're determining [15:43] that information at runtime [15:46] luckily a lot of these problems are [15:48] already solved by [15:49] the zig compiler itself and they're [15:52] put in the standard library instead of [15:54] in the compiler implementation because [15:56] they're generally useful [15:57] for example right here um so we already [16:00] have [16:00] uh this uh this function to detect [16:04] native target info and part of the [16:07] native target info that's detected is [16:09] what is the path to the dynamic linker [16:11] so we get the path [16:13] and then what we're going to do is we [16:17] create an arg v [16:18] where the first arc v0 is the dynamic [16:21] linker that [16:22] we just detected and then the first [16:24] argument [16:25] is the path to our own executable [16:29] so i'll come back to these but i'll just [16:30] show you that part [16:32] um so argv is right here [16:39] yeah so the first arg is [16:42] the dynamic linker and then oh we're [16:45] just respecting [16:46] the um other args in the function [16:49] but then we're inserting one in there so [16:53] we're gonna put [16:57] yeah right here [17:01] wait i thought we were putting the arg [17:04] to self [17:05] exe okay i got confused there for a [17:07] second [17:10] we're definitely putting the damage [17:11] linker in the first arc there [17:14] and then [17:18] oh right right so the situation is that [17:20] the the path to the self-executable is [17:22] already [17:23] in the arg things all we have to do is [17:24] shift them over and put the dynamic [17:25] linker first [17:26] yeah i'm sorry for the confusion so [17:30] that's for the argument arguments now we [17:32] also [17:33] have to mess with the environment [17:34] variable a little bit because [17:37] what we're going to need to use dl open [17:39] which is in libdl and we're going to [17:41] need to use [17:41] um some b thread stuff for mutexes [17:45] so we need next time we come into main [17:49] when we if we come in from the dynamic [17:51] linker we do need some of the lib c [17:53] stuff to be loaded [17:54] and we don't normally it's in the [17:56] dynamics section [17:57] here but remember we had to strip out [17:59] all of our needed [18:01] tags so we have to explicitly mess with [18:04] the environment variable here [18:05] so we actually take advantage of ld [18:07] preload when we invoke ourselves [18:10] and we put libdl and libp thread inside [18:13] the preload [18:14] environment variable and that way when [18:17] we call [18:17] exec ve this wipes out [18:21] the executables like kind of state and [18:24] replaces it with [18:25] reloading itself but now we're going to [18:27] have these lib c things available [18:30] and they're going to drag in whatever [18:31] dependencies they have so that's that's [18:32] going to get us what we need [18:34] so next time we come into main [18:37] we get dl open now it's not null now we [18:40] can call dl [18:41] open because now we're running in the [18:43] dynamic linker boom [18:47] now we're here so that [18:50] is how we can have a [18:56] executable that is both static and [18:59] dynamic at the same time [19:02] that's not all though that's not all the [19:04] problems we still have to solve more [19:05] problems [19:05] i'll show you [19:10] okay so next one [19:13] lipsy agnostic code now the problem is [19:17] i just showed you some of this stuff [19:18] here where um the ldd [19:20] on on bk cube this is glib c [19:25] but on other linux distributions it's [19:27] muscle and that's what this code is [19:30] so even after we load with the dynamic [19:33] linker [19:34] um not all ellipses are built the same [19:36] so on on [19:38] glib c it kind of just brings in [19:41] everything all at once and we can just [19:42] go straight to here um for muscle [19:45] it's kind of more uh bare bones so we [19:47] have to do a little bit of work to set [19:48] up we have to call [19:49] muscles lipsy start main and give it a [19:52] pointer to [19:53] um you know we have to call exit [19:55] manually and do some [19:57] some more dl opens here on the lib c [19:58] symbols but eventually we're going to [20:00] end up in main two [20:01] on the muscle case just the same and the [20:04] important part here is that we also can [20:06] call native target info detect same [20:09] thing we were doing earlier [20:10] again this is that um that code that we [20:13] have for the for the zig compiler [20:15] purposes [20:16] but now um you know because [20:19] because the zig compiler can [20:24] detect your lib c [20:27] so we want to detect the native triple [20:30] right there we detected that it's glib c [20:32] that same code can be used to detect do [20:34] we do we want a dynamic linking its [20:36] muscle [20:36] or do we want a dynamic link against gmc [20:38] at runtime [20:40] and that gets us into finally the actual [20:44] part where we can interact with the [20:46] vulcan and the x11 apis [20:49] but that's still not all the we still [20:50] didn't solve all the problems there's [20:52] still more problems to solve [20:55] so finally i had to hack up vulcan [20:58] loader [21:00] so vulcan loader is a project by chronos [21:03] group [21:04] and the purpose of it is to find [21:07] the vulcan drivers and populate the [21:11] function pointers corresponding to the [21:13] official [21:13] vulcan api [21:17] it does a lot of work for you because [21:20] it does smooth over smooth over the [21:23] differences in some systems [21:25] and i was able to hack it up to build [21:28] statically [21:30] so what i did was i i just took [21:33] the uh let's go back into this one [21:51] there so i took the um [21:55] i took the source files from vulcan [21:56] loader only the ones that were needed [21:59] and i put them in there and so you can [22:02] see that if we look at the build script [22:04] here [22:04] i had to add i used the fact that zik [22:07] can file c code [22:08] using you know the libclang abilities [22:12] i defined c macros a certain way [22:17] i had to force it to use position [22:18] independent code [22:20] and then i just listed only the files [22:22] that we actually really need [22:24] and so when you run zig build on this [22:26] project it will go ahead and build all [22:27] the vulcan loader stuff that we need [22:29] statically [22:30] um but with pick so that we can it can [22:32] also support being run dynamically [22:35] and then here's the here's the tricky [22:38] part [22:39] i had to create my own lib c h file [22:43] and put all the lib c functions [22:46] that this project depended on this i [22:48] made this file here [22:50] and i had to put attribute weak on all [22:52] of them so that way [22:54] when we go into um [22:57] main here the first one [23:00] and we are actually running statically [23:04] the thing is we still have vulcan loader [23:07] inside [23:08] all those functions that call libc are [23:10] still in there [23:11] but they're not going to work because [23:12] we're not actually loading libc right [23:14] now [23:14] so if we tried to call inside here if we [23:18] tried to call any of the vulcan loader [23:20] stuff it would crash because every [23:21] single [23:22] lib c function like mem copy m set um [23:25] you know put chair r f close closer all [23:28] these [23:28] they would all just segmentation fault [23:30] because they're all zero they're all [23:31] null [23:31] because they're weak weak means that you [23:34] want it to link [23:35] if you can but if not just make it null [23:39] so i had to i had to override all of the [23:42] ellipses [23:43] functions here um because what we had to [23:46] do is make [23:47] vulcan it's going to run [23:50] it doesn't know which lip c it's going [23:52] to run against ru it finds out at [23:54] runtime [23:54] right so i had to hack that up [23:58] and then it also had to hack a patch [24:02] into it uh rtld [24:06] global [24:10] i think it's this one [24:14] yeah so i also changed uh [24:18] i also changed this or was it the x11 [24:21] one [24:21] anyway i had to add a patch where [24:24] instead of rtld local i used rtld global [24:27] because um [24:30] when we finally get down to main two [24:32] down here [24:33] it's right here i added a comment about [24:35] it um [24:38] we need to dl open the x11 [24:41] shared library and [24:46] the the vulcan library [24:51] the vulcan uh driver will depend on this [24:53] so we know that by loading [24:56] by by um using the vulcan loader [24:59] by using this uh this is actually going [25:02] to call [25:04] i don't know it's going to call the [25:05] first vulcan loader function and when [25:07] that happens [25:08] that's going to force this to be loaded [25:10] and what we wanted to happen was that [25:13] we wanted the dynamic loader to cache [25:15] this so that when we do it [25:17] we just get the same one that the vulkan [25:19] driver already loaded [25:21] so i had to add a patch to um to the [25:25] vulcan loader in order to do a global [25:27] load instead of a local load [25:29] and that allowed that guarantees that [25:31] this will work [25:33] um instead of trying to like namespace [25:35] them or separate them by making it [25:37] global [25:38] so i had to patch it in order to do that [25:42] um so i showed you weak linkage added [25:44] patch [25:46] yeah and also caviar emptor i was on the [25:48] disco [25:50] bonus group i think and i was talking to [25:52] one of the maintainers of this library [25:54] and they were saying that they don't [25:55] like it when people static link [25:58] uh vulcan loader because i don't [26:00] remember there's some [26:01] there's some reason that it's they don't [26:03] want you to static link it [26:05] uh but it does make the game work so um [26:07] i don't know maybe they should consider [26:09] this a use case [26:10] maybe they can try and make it work uh [26:13] maybe they can [26:15] maybe linux distribution maintainers can [26:17] cooperate with vulcan loader and make [26:18] sure that you can statically link [26:20] against it because [26:21] this is what players want they want to [26:23] run the game [26:26] uh okay i think that [26:29] i think that kind of demonstrates what i [26:32] had to do with loken loader there [26:33] and it's kind of a problem right because [26:35] if i wanted to you know this is very [26:37] much a proof of concept because if i [26:39] if you go look at um if you go look at [26:42] the vulcan loader project [26:44] it it's got recent commits [26:47] and i i i i copy pasted code i have to [26:50] go pull those commits in [26:52] and and um you know it's a you have to [26:55] maintain [26:56] the fork basically it's a fork you know [26:58] so it's a it creates work for yourself [27:00] to try and keep up to date with them um [27:03] so a lot of caveats here in this proof [27:05] of concept [27:07] okay and then finally all that's left [27:10] once we solve these problems is the [27:12] application code [27:14] inside here to just do the [27:18] x11 api calls that we have to do to set [27:20] up a window [27:21] set up a vulcan context [27:24] all that stuff ultimately what these are [27:26] all doing [27:27] is um sending messages over a socket to [27:30] the [27:30] x11 server but it's [27:34] the most likely way to make it work is [27:36] by actually using the lib c [27:38] shared objects that the same ones [27:42] that the the actual graphics driver [27:44] links against [27:45] that way we just know everything's using [27:47] the same stuff [27:50] now i did also have to do some tricks [27:52] here [27:54] because we want to cross compile this [27:57] obviously [27:58] and the [28:02] these h files uh vulcan loader depends [28:05] on [28:06] and so i don't want these to be um [28:09] they can't be from my system that would [28:11] be wrong so i had to [28:13] i had to go into x11 and i had to find [28:16] out figure out their weird build system [28:17] how to produce these files and then i [28:18] had to go tweak the [28:20] pound defines to be um like [28:23] safe for cross compiling so there's a [28:25] little bit of work here to do too [28:28] but it works um these are fine i checked [28:30] them out and they're [28:31] uh you know it's it's fine it's just [28:34] constants that won't change because it's [28:35] a network protocol they can't [28:38] you know they they can't change these [28:41] things because it would break [28:43] uh it would break their abi [28:46] so that's it that's all that's left is [28:48] just the part where you do a hello world [28:50] triangle [28:51] with x11 and with uh this is just like [28:53] basic [28:54] basic vulcan um [28:58] i just followed this tutorial [28:59] vulcantutorial.com to do the triangle [29:01] that's all this is [29:04] uh let's see yeah [29:07] so that's it that's that's my demo um [29:12] that that's all the tricks that i had to [29:13] do to make this work and i know there's [29:15] probably some people out there [29:16] some uh some game devs uh who are [29:19] interested in [29:19] in this and probably i'm guessing your [29:21] projects are in c or c plus plus and you [29:23] want to [29:23] see how realistic this is uh [29:27] honestly it's not realistic it's a proof [29:30] of concept [29:32] you're gonna have to jump through a lot [29:33] of hoops to make this work [29:35] now i think this is worth pursuing um [29:38] but i really think that the [29:42] kind of kind of conclusion that i want [29:45] to make here [29:46] is that um we do need driver vendors [29:49] than linux distribution maintainers to [29:51] explicitly acknowledge this use case [29:53] a lot of these hoops are arbitrary and [29:57] unnecessary and i'm demonstrating that [30:00] it's possible to get around them [30:02] um but really i think that what we just [30:05] need is for is an acknowledgement that [30:09] this is [30:10] the user experience that users that [30:12] players deserve [30:14] and we just need more people to uh [30:17] demand [30:17] this as how it works on on linux [30:22] so i hope i hope this inspires you to [30:24] just [30:25] raise your standards for what we could [30:27] have and [30:28] if enough people believe that you know [30:31] there's a minimum bar of what's [30:32] acceptable [30:33] then the bar will raise that's kind of [30:35] what my goal is with this [30:38] um so uh if you want to check it out [30:42] the source code is here um it's very [30:44] unpolished [30:45] it also regressed recently it won't [30:47] build right now [30:48] um due to some like actual improvements [30:51] that we made into the way that zig links [30:54] code [30:55] and the arguments that we passed to uh [30:56] lld [30:58] and so my conclusion here is that this [31:00] use case is special there's a lot of [31:02] moving parts here and i think to avoid [31:06] regressing this like if we wanted to [31:08] actually support this use case i think [31:10] it would have to be [31:11] an explicitly recognized use case by by [31:14] tooling [31:15] in order to have it be reliably [31:18] supported in the long term [31:20] because if it's just a list of tricks [31:22] that you have to do to make it all work [31:25] it's going to regress like if you don't [31:27] care about this [31:28] use case it's just going to get trampled [31:30] over because it's kind of weird [31:32] like the idea of a hybrid you know like [31:34] why would you have [31:35] a dynamic section when you actually [31:37] don't have any dynamic dependencies [31:39] like a normal a a normal [31:42] if you if you just use like your [31:44] reasoning and you're writing a linker [31:45] you're like well you know [31:46] i'm going to omit the dynamics section [31:47] because there's nothing dynamic about [31:49] this [31:50] right so it's like you actually have to [31:51] think about this use case [31:53] explicitly if you want it to not regress [31:56] that's that's kind of my takeaway here [31:58] and that's just not the status quo [32:01] um so yeah i think that's uh [32:04] that's about it and uh i think we're [32:07] gonna have a qa section but i'll let uh [32:09] i'll let loris [32:10] say some words and um and take that off [32:16] okay thank you andrew uh yeah so let's [32:19] switch [32:20] to the q a okay um [32:25] let's see let's see let's see so uh [32:27] while people [32:29] uh start having questions by the way [32:31] just uh as a reminder to everybody [32:34] um try to keep it things on topic uh i'm [32:37] sure people have a ton of questions for [32:38] andrew but we're going to talk about the [32:41] the roadmap uh next so uh [32:44] let's reserve your marginal questions [32:47] for [32:48] uh for that part of the show uh so while [32:51] people start thinking and typing up [32:52] questions actually no [32:54] before that uh just as a reminder i'm [32:57] looking at twitch chat i'm looking at [32:58] discord [32:59] and i'm also looking at irc so like [33:02] hashtag zig on [33:04] uh free node so uh just make sure to tag [33:07] me [33:08] and i'll be happy to take your questions [33:11] or andrew uh wherever you're most [33:14] comfortable writing them [33:16] uh so what people think of questions [33:18] andrew [33:19] uh usually i start by asking how you [33:22] stumble but no wait um [33:26] so that was interesting thank you i [33:29] think you [33:30] you raised a great point with the with [33:32] the idea of [33:33] having expectations and and i think this [33:36] is a [33:37] this is a great point this is one of the [33:38] reasons why i [33:40] uh i'm on um my main machine is a mac [33:44] computer it's not a linux machine and [33:46] it's not like i like apple [33:48] i do not like apple but that's how [33:50] that's how it is for me [33:52] uh for other reasons and yeah i think [33:54] your point is very variable [33:56] available for um for this ecosystem [34:00] so okay let's see [34:03] let's see what we are saying okay so [34:05] let's start with gw1 [34:06] um could you implement an x11 protocol [34:09] in zig to not have [34:11] to have a dynamic dependency on xcb [34:14] juicebox [34:15] has started doing this you can [34:19] you can avoid um a dynamic dependency on [34:23] x11 [34:24] libraries just fine however if you want [34:26] to use graphics drivers such as vulkan [34:28] or opengl [34:31] the graphics drivers have a dynamic [34:33] dependency on the x11 libs [34:36] so you're already paying the cost no [34:37] matter what if you want to load the [34:39] graphics drivers [34:40] you are loading the x11 shared objects [34:44] into your memory space whether you like [34:46] it or not [34:47] so my opinion is if you have to do it [34:50] my my logic goes like this premise one [34:53] we need to use the graphics drivers [34:56] premise two the graphics drivers use the [34:58] x11 shared objects [35:00] um premise three we're paying the cost [35:03] therefore we're [35:04] paying the cost for graph for x11 shared [35:06] objects no matter what [35:08] conclusion you should just use them [35:10] since you have to [35:11] you have to [35:15] thank you um so next question [35:19] from zakus uh when is the [35:22] valve sponsorship coming the health [35:25] sponsorship [35:25] yeah hey valve you want to sponsor zig [35:28] you should definitely do it [35:29] everyone's gonna be writing games in zig [35:31] in five years [35:33] makes sense um i'll make sure to relay [35:37] these to gaben um yeah thank you [35:41] uh let's call him on the phone yeah [35:44] i always do that every time i miss my [35:46] shots with the tech 9 in cs go [35:48] i call him up and start complaining [35:52] what were the reasons why were all the [35:54] reasons you couldn't use the vulcan so [35:57] wixy symbols and rtld global [36:01] uh wouldn't the c symbols work since you [36:04] weren't since you aren't calling vulcan [36:06] until after you're loaded lipsy [36:10] uh not quite sure if i understand the [36:12] question but let me try and let me know [36:14] if i answered it correctly [36:15] um so the first the first thing i tried [36:18] here was [36:19] uh i had this and i did not have a patch [36:22] to um uh andrew you [36:26] are not sharing your screen anymore i [36:28] think let me see if it's [36:29] on my side all right should i refresh [36:31] the [36:32] place uh no it was definitely on my side [36:35] so [36:35] okay you're good to go okay so the first [36:37] thing i tried [36:38] was i tried to deal open this [36:41] shared object which i know was already [36:43] loaded into the application's [36:45] virtual memory by vulcan loader and it [36:48] just said [36:49] no it said there isn't one there [36:52] because floken loader used local and if [36:55] you [36:55] use local uh [36:59] what's the documentation for it say i [37:00] don't remember but the point is because [37:02] vulcan loader used local [37:04] and then i tried to do it later it just [37:05] said that [37:07] it didn't happen because it was a [37:09] different object or something [37:10] but when i switched them to use global [37:13] then this was started [37:14] being guaranteed to return a valid [37:18] object did that answer the question [37:21] um i hope so that was a pretty advanced [37:23] question for me uh i guess marilla will [37:25] let us know by the way it was from [37:26] from him so okay uh [37:30] let us know in the meantime i'll go [37:32] through the other questions uh [37:34] uh what would you need to change to make [37:35] this work with wayland [37:38] uh nothing because weyland it supports [37:41] the x11 protocol [37:43] how do you like that answer [37:44] [Laughter] [37:46] uh i'll give a more serious answer [37:48] though i mean that is a serious answer [37:50] but um i'll give another part two answer [37:53] uh if you wanted to explicitly support [37:55] weyland how that would work [37:57] is um here we would just [38:01] first try opening a weyland object [38:04] uh and presumably um [38:07] it works the same way where the graphics [38:09] driver depends on wayland instead of [38:10] x11 so we would just try to open a [38:13] weyland object [38:14] and then if we got it we use that one [38:16] otherwise we fall back to this one [38:18] so it'd be pretty straightforward um but [38:21] i am of the strong opinion that if [38:23] weyland is doing it right [38:24] uh this code should all just work fine [38:27] on weyland with almost no penalties [38:33] okay another question oh sorry [38:36] nothing nothing okay um another question [38:40] i'm not sure i'm not fully and i don't [38:43] fully understand it but maybe you can do [38:44] a better job than me so [38:46] can the needed section be just the leap [38:49] name dot so without the absolute path [38:52] i'm thinking this refers to the dynamic [38:54] linker [38:56] can it be just the lib name without the [38:58] dynamic path [39:00] let's take a look at uh yeah without the [39:03] absolute path [39:04] so i'm assuming it's dynamic linker but [39:07] i'm not sure maybe there's a [39:09] there's another so file mentioned [39:11] somewhere that has an absolute path [39:13] so here's um here's vk cube this is this [39:16] is kind of like our uh control case [39:19] right this is a traditional app you know [39:21] linked the normal way [39:22] um so here's the needed entries for this [39:24] one you can see it has one for [39:26] xcb these are all for libc [39:30] and then there's the vulcan driver and [39:32] it's it's not an absolute path [39:33] the absolute paths come in with the run [39:35] paths here [39:36] and the problem is that you don't even [39:40] even with this way you just don't know [39:42] what the you know what the run paths are [39:44] until run time that's the problem [39:47] like this this run path this this run [39:49] path would be wrong for [39:50] other distros [39:55] um okay maybe june will let us know if [39:58] uh follow-up is needed so [40:02] so johnny martin was saying then then or [40:04] given your answer then if we patched [40:06] vulcan to use global somehow [40:09] we could use the so [40:13] patched vulcan or vulcan loader because [40:15] the example [40:16] here what's happening is that [40:19] we call in here we end up calling a [40:22] bunch of initialization functions inside [40:24] of vulcan loader [40:25] to load the vulcan driver and [40:29] that's the code that is doing dlopen [40:32] or sorry yeah that's the code that's [40:35] doing dll open on here [40:37] and so all we're trying to do is just [40:39] get the same one [40:40] um so yeah if vulcan loader was packed [40:42] we could just send the patch upstream [40:44] see if they want it um there would be [40:46] arguments to not do it but [40:48] um in this case it's the right call [40:53] it's not a driver patch though it's a [40:56] it's a patch to [40:58] the vulcan loader okay [41:02] um i think okay people are confirming [41:04] that the answers were [41:06] satisfactory so nice [41:09] [Music] [41:11] awesome well [41:14] i mean jokes aside valve [41:18] is already doing a bunch of work to make [41:20] games work on linux right [41:22] uh steam is doing a lot of things and i [41:24] think it's being [41:25] fairly successful i think there's also [41:27] some games that actually [41:28] run better on linux than they do on [41:30] windows so [41:32] i i hope that i sincerely hope that um [41:35] people at valve [41:36] watch this short talk and get inspired [41:39] and [41:40] uh use their influence and power to [41:43] help the uh linux distribution ecosystem [41:47] support this use case better i think [41:49] that the users want it the players want [41:51] it the game developers want it [41:53] and ultimately the linux maintainers the [41:56] package maintainers are going to do what [41:58] people want [41:58] so we just have to make people want this [42:01] more [42:02] okay makes perfect sense [42:05] um okay so i think we can call it [42:09] um we can call it here so thank you [42:13] andrew for the talk that was amazing uh [42:16] we're gonna take a break and after the [42:19] break we're coming back [42:20] and we're gonna talk about the z roadmap [42:22] so um [42:23] i guess the second part is gonna be uh [42:26] even better so [42:27] um see you all soon