linux.conf.au
Rice argues that modern dynamic operating systems require unified, event-driven service supervision rather than 1970s shell scripts, urging open-source developers to adopt systemd's architectural advances instead of clinging to vintage UNIX dogma.
AI-assisted summary of The Tragedy of systemd — original source. Check the original for full context.
Source published:
Key ideas
- Traditional UNIX init conflated static system configuration with ongoing service supervision, leading to fragile shell scripts that could start daemons but lacked the ability to track or recover them on crash. (4:57)
- Lennart Poettering drew heavily on Apple's launchd for systemd's socket activation and event-driven model to address modern runtime volatility across Linux environments. (10:49)
- Rather than acting as a simple init daemon, systemd evolved into a comprehensive 'system layer' bridging user space and the kernel by absorbing udev, network state, and logging into a cohesive architecture. (13:45)
- The fierce controversy surrounding systemd stems from cultural resistance to outside disruption and an empathy deficit between maintainers navigating conflicting demands and users defending familiar workflows. (25:12)
Detailed notes
## Background Published on YouTube by linux.conf.au on January 24, 2019, this presentation features FreeBSD developer Benno Rice examining the fraught controversy surrounding systemd. Drawing on the history of operating system bootstrap architectures and contemporary UNIX alternatives, Rice explores why an integrated system manager became necessary and why its adoption provoked such deep-seated community outrage.
## TL;DR Rice traces the historical evolution of init systems from the minimalist shell scripts of early UNIX to modern service managers like Apple's launchd and Linux's systemd. He argues that modern hardware volatility, cgroups, and stateful services necessitated an intermediate 'system layer' rather than static sequential bootstrap scripts. Addressing common grievances around bloat, bugs, and POSIX portability, he separates implementation flaws from architectural concepts while highlighting genuine breakthroughs such as user units and predictable hardware naming. Ultimately, he frames the ongoing hostilities as a toxic byproduct of contempt culture and developer-user empathy deficits, urging technologists to move past reflexive resistance and adopt modern design principles.
## Atmosphere & style The presentation blends sharp technical history with disarming wit, philosophical reflection, and self-deprecating humor, adopting the five-act structure of classical tragedy to contextualize community suffering. Speaking with the grounded perspective of an experienced FreeBSD contributor, Rice maintains a measured and fair-minded posture that avoids dogmatic tribalism. Watching the talk feels like participating in an engaging, cathartic post-mortem on software culture that turns bitter engineering debates into an earnest appeal for mutual empathy and architectural ambition.
## Bulletin - [04:57](https://www.youtube.com/watch?v=o_AIw9bGogo&t=297s) Traditional UNIX init conflated static system configuration with ongoing service supervision, leading to fragile shell scripts that could start daemons but lacked the ability to track or recover them on crash. - [10:49](https://www.youtube.com/watch?v=o_AIw9bGogo&t=649s) Lennart Poettering drew heavily on Apple's launchd for systemd's socket activation and event-driven model to address modern runtime volatility across Linux environments. - [13:45](https://www.youtube.com/watch?v=o_AIw9bGogo&t=825s) Rather than acting as a simple init daemon, systemd evolved into a comprehensive 'system layer' bridging user space and the kernel by absorbing udev, network state, and logging into a cohesive architecture. - [25:12](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1512s) The fierce controversy surrounding systemd stems from cultural resistance to outside disruption and an empathy deficit between maintainers navigating conflicting demands and users defending familiar workflows.
## Key takeaways - [01:48](https://www.youtube.com/watch?v=o_AIw9bGogo&t=108s) UNIX originally succeeded via brutal simplicity, but its early init model assumed static workloads that failed under modern internet-scale service demands. - [07:57](https://www.youtube.com/watch?v=o_AIw9bGogo&t=477s) Operating systems like Windows NT and macOS implemented dedicated service management and process supervision decades ahead of classic UNIX init scripts. - [07:57](https://www.youtube.com/watch?v=o_AIw9bGogo&t=477s) Apple's launchd demonstrated that socket and Mach port activation could dramatically accelerate boot times and minimize power consumption by launching daemons purely on demand. - [10:49](https://www.youtube.com/watch?v=o_AIw9bGogo&t=649s) systemd's primary objective was not merely raw boot speed, but providing a unified supervisor capable of reacting dynamically to hardware and software state shifts. - [13:45](https://www.youtube.com/watch?v=o_AIw9bGogo&t=825s) Modern dynamic computing environments require an intermediate 'system layer' to manage hotplug devices, dynamic addressing, and time synchronization without fragile third-party glue code. - [16:12](https://www.youtube.com/watch?v=o_AIw9bGogo&t=972s) Debian's adoption of systemd triggered high-profile resignations on its Technical Committee due to debates over coupling the OS to Linux-specific kernel interfaces. - [17:18](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1038s) Criticisms claiming systemd violates the Unix philosophy mistake a modular multi-binary monorepo for a single bloated binary. - [17:18](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1038s) All software contains bugs, but systemd's catatonic failure mode prevents the entire operating system from immediately panicking during fatal internal errors. - [20:26](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1226s) Abandoning lowest-common-denominator POSIX abstractions enabled systemd to fully exploit Linux-specific primitives like cgroups and autofs. - [22:18](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1338s) The computing industry has moved from historical UNIX fragmentation to a Linux-centric monoculture, altering standard portability expectations. - [24:14](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1454s) systemd delivers superior operational capabilities through granular cgroup process tracking and native unprivileged user-level system units. - [25:12](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1512s) Technologists frequently celebrate change when driving it themselves, but resist it vehemently when external paradigms disrupt familiar workflows. - [29:04](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1744s) Incoming generations of developers conceptualize infrastructure through cloud APIs and containers rather than traditional UNIX file hierarchies and shell scripts. - [29:04](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1744s) Structured messaging transports, predictable device naming, and append-only binary logs represent durable architectural advancements over ad-hoc UNIX text parsing. - [36:17](https://www.youtube.com/watch?v=o_AIw9bGogo&t=2177s) Porting Apple's launchd to Linux was never feasible because launchd is deeply coupled to macOS-specific Darwin and Mach port architectures. - [45:14](https://www.youtube.com/watch?v=o_AIw9bGogo&t=2714s) Resolving the cultural rift in the systemd ecosystem requires mutual empathy, recognizing both the challenges of managing conflicting user demands and the disruption experienced by administrators. - [46:16](https://www.youtube.com/watch?v=o_AIw9bGogo&t=2776s) Elevating messaging and RPC transports directly into the operating system kernel could establish uniform security policies and equal-footing IPC across userland and kernel components.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=o_AIw9bGogo&t=0s) Introduction: Tragedy, Contempt Culture, and Change
Benno Rice opens by warning the audience that he has opinions. When assembling the talk, the title fell into place once he looked up Wikipedia’s definition of a tragedy: a drama based on human suffering that invokes catharsis or pleasure in the audience. Given the subject matter of systemd, Rice notes there has been more than enough suffering to go around, and he hopes to deliver the accompanying pleasure. Another major influence on the talk is an essay by Aurynn Shaw titled *Contempt Culture*, which examines how groups use contempt as a social signifier—a pattern frequently observed across programming communities where Python developers deride Perl, Perl developers mock PHP, and so on.
Rice emphasizes that the same tribal dynamics apply broadly across computing culture, particularly when confronted with change. Change inherently threatens familiarity, and while the familiar feels comfortable, that does not mean it is still good. Because traditional dramatic tragedies are structured into five acts, Rice formats his presentation into five distinct acts, starting with the origins of UNIX.
As Benno Rice observes regarding the discomfort of moving away from long-established tools: > “Change is a tricky thing because change threatens what we find familiar, and familiar is nice, familiar is comfortable, but that doesn't always mean that it is good, and it certainly doesn't mean that it is still good now if it's been around for a long time.”
---
[01:48](https://www.youtube.com/watch?v=o_AIw9bGogo&t=108s) Act I: The Brutal Simplicity of Early UNIX Init
Rice begins Act I by describing UNIX as a “happy accident.” It emerged as a direct reaction against contemporary operating systems that were seen as attempting to do far too much. UNIX took a brutally simple approach by focusing on small components and bootstrapping userspace in the simplest way possible. Looking at the manual page for `init` from Seventh Edition UNIX (1979), its documented role was straightforward: invoked as the final step of the boot procedure, it spawned a process for each terminal typewriter, and when transitioning to multi-user mode, it ran a shell script from `/etc/rc` to execute housekeeping functions such as mounting filesystems and starting daemons.
```mermaid flowchart TD accTitle: Seventh Edition UNIX Boot Flow accDescr: Simple sequence showing 1979 UNIX booting through init to run housekeeping and spawn terminal listeners. subgraph Boot ["Kernel Boot"] K["Kernel Initialized"] --> I["init (PID 1)"] end subgraph InitTasks ["Housekeeping & Terminals"] I -->|"runs script"| RC["/etc/rc (Mount fs & start daemons)"] I -->|"forks"| T1["Typewriter / TTY 1"] I -->|"forks"| T2["Typewriter / TTY 2"] end ```
Inspecting the process table of a PDP-11/70 running Seventh Edition UNIX (accessed courtesy of the Living Computers Museum and Labs) illustrates how little was actually happening: `init` forked a few times to handle terminals, alongside `cron` and `update` (whose only task was periodically flushing the filesystem superblock to disk). Very little had changed even by the time of 4.3BSD on a VAX-11/730. That system introduced `getty` for serial consoles, `routed`, `inetd`, and a couple of `telnetd` processes. Among these, `inetd` served as a service "super-daemon." Instead of leaving dozens of background daemons running persistently, `inetd` listened across network sockets and forked off an ephemeral handler process (such as `telnetd` or `ftpd`) only when an inbound connection arrived.
Describing the original philosophical design of early UNIX bootstrap routines, Benno Rice explains: > “They looked at a whole lot of the other systems that were on offer at the time and went, ‘You know what? This is just trying to do too much. We can be better than this, we just start with the small stuff and work from there,’ and it takes this brutally simple approach to bootstrapping itself and especially in bootstrapping user space.”
- `init` in Version 7 UNIX ran `/etc/rc` sequentially to perform housekeeping functions before handing control to terminal typewriters. - `update` periodically flushed filesystem metadata so unsaved disk blocks would not be lost during a crash. - `inetd` listened on behalf of multiple network services, delegating actual traffic handling to short-lived child processes on demand.
---
[04:57](https://www.youtube.com/watch?v=o_AIw9bGogo&t=297s) The Evolution from Daemons to Services
The `inetd` design functioned smoothly when network traffic was light and protocols were ephemeral. If a Telnet or FTP connection dropped, the client simply reconnected without losing significant persistent server state. However, the explosion of the World Wide Web broke this model; forking a distinct process for every individual HTTP connection became unsustainable at scale. Furthermore, complex stateful systems such as relational databases required persistent in-memory state across requests and could not be started and torn down per connection. This shift produced the modern concept of a "service"—a contiguous collection of behaviors and processes, often spanning multiple daemons or worker pools.
Traditional SysV-style `init` was never designed to manage services in this sense. While `getty` was actively supervised and restarted by `init` as terminals connected and disconnected, standard daemons were merely launched via shell scripts during boot. `init` simply stood them up and walked away; if a daemon crashed later, `init` did nothing to recover it. Even the introduction of `/etc/inittab` in System V provided only crude restart capabilities without fine-grained lifecycle awareness.
Rice argues that the historic UNIX manual conflated two fundamentally distinct tasks under the banner of "housekeeping functions": system configuration and service bootstrap. Configuration involves placing the host into an operational posture—much like a pilot adjusting flaps and slats into a landing configuration—by mounting filesystems and assigning network addresses. These tasks are set once and largely left alone. Service bootstrapping, by contrast, requires ongoing operational oversight, monitoring, and automated lifecycle management. Conflating these two concepts obscured the need for dedicated service managers, eventually paving the way for tools like Upstart and systemd.
Highlighting the fundamental limitation of traditional UNIX init scripts, Benno Rice points out: > “Old-school init can start these things, but they don't really get involved with managing them after that, they just kind of stand them up and then walk off and if it falls over they go, ‘[shrug].’”
---
[07:57](https://www.youtube.com/watch?v=o_AIw9bGogo&t=477s) Service Models in Other Operating Systems and the Birth of launchd
Looking outside the classic UNIX lineage, Rice notes that other operating systems approached background tasks with structured service models much earlier. Windows NT built a robust service framework directly into the platform from its inception. While relying heavily on shared host wrappers like `svchost.exe` introduced its own share of security challenges, the NT service manager included native capabilities for starting, stopping, dependencies, and automatic crash recovery—a design influenced by the complex minicomputer OS architectures that UNIX had originally rebelled against. Similarly, Apple developed a structured application execution model in Mac OS and iOS centered around application delegates, allowing the operating system to notify programs directly when events occur, such as application lifecycle shifts or low-memory warnings.
```mermaid flowchart LR accTitle: launchd Socket Activation Flow accDescr: launchd listens on multiple socket types and launches daemons on demand rather than at boot. subgraph Supervisor ["launchd (PID 1)"] L["launchd Socket Listener\n(UNIX sockets, TCP, Mach ports)"] end subgraph OnDemand ["Managed Daemons"] D1["System Service A\n(Spawned on access)"] D2["System Service B\n(Spawned on access)"] end L -->|"incoming connection"| D1 L -->|"Mach message"| D2 D1 -.->|"supervised / auto-restart"| L ```
In Act II, Rice turns to Apple's creation of `launchd`, introduced in Mac OS X Tiger. `launchd` consolidated almost every disparate event-handling and scheduling utility on the system, replacing `init`, `cron`, `at`, and `inetd` with a single unified daemon. Expanding the core concept of `inetd`, `launchd` listened on UNIX domain sockets, TCP ports, and Mach ports simultaneously. Instead of starting every background service during system boot, `launchd` held open their communication channels and launched the underlying daemons on demand when messages arrived, dramatically accelerating boot times. In addition, `launchd` provided continuous process supervision, monitoring service processes and automatically restarting them if they exited unexpectedly.
Summarizing how Apple’s `launchd` restructured service execution in macOS, Benno Rice states: > “So the idea of launchd was to move as many system services as possible into these daemons that could then be started as needed, because when you think about it, the actual underlying system when you're talking about a system like Mac or...”
- Windows NT shipped with native service lifecycle management and auto-restart from its earliest releases. - Apple platforms rely on application delegates to allow the host operating system to signal state transitions like memory exhaustion directly to running programs. - Introduced in Mac OS X 10.4 Tiger, `launchd` unified the duties of `init`, `cron`, `at`, and `inetd`. - `launchd` deferred service startup by holding open network sockets and Mach IPC ports, instantiating services only when traffic was directed to them.
[10:49](https://www.youtube.com/watch?v=o_AIw9bGogo&t=649s) The Legacy of launchd and the Birth of systemd
```mermaid flowchart TD accTitle: Evolution from launchd and Upstart to systemd accDescr: Diagram illustrating how Apple's launchd and Ubuntu's Upstart influenced Lennart Poettering to create systemd as an event-driven system manager.
subgraph Precursors ["Earlier Event-Driven Systems"] Launchd["Apple launchd<br/>(Demand-driven, Mach ports)"] Upstart["Ubuntu Upstart<br/>(Event-driven, shell scripts)"] end
subgraph SystemdGenesis ["systemd Architecture"] Poettering["Lennart Poettering<br/>(Blog announcement)"] SysdCore["systemd<br/>(Socket activation & dependency ordering)"] end
Launchd -->|"Inspiration: socket activation & events"| SysdCore Upstart -->|"Critiqued for lacking dependency ordering"| Poettering Poettering -->|"Implements core architecture"| SysdCore ```
Benno Rice contrasts the traditional startup model with macOS, where services start purely on demand when something requests them. This approach yields immediate advantages: it slashes boot times, curtails power consumption, and elevates security by enforcing granular access controls. Apple pulled this off because macOS relies pervasively on underlying Mach ports and messaging systems. Rice points out the irony when speaking to fellow BSD enthusiasts: while BSD adherents often dismiss systemd, they enthusiastically endorse Apple's `launchd`, seemingly oblivious to the fact that “systemd is basically a ripoff of launchd.”
Five years after Apple deployed launchd, Lennart Poettering examined the existing Linux landscape—including Ubuntu’s Upstart. While Upstart was event-driven and could handle far more event triggers than traditional SysV init, it still depended on shell scripts and, in Poettering's view, failed to handle dependency ordering effectively. Setting out his vision in a seminal blog post introducing systemd, Poettering argued: “For a fast and efficient boot up, two things are critical: to start less and to start more in parallel.” Although systemd is often associated with raw boot speed, shaving seconds off boot was merely a secondary perk; the real priority was creating a supervisor responsive to hardware and software shifts: “An init system that is responsible for maintaining services needs to listen to hardware and software changes.”
- Poettering explicitly cited Apple's launchd as a direct inspiration for systemd's socket activation and event-driven model. - Poettering's prior open-source initiatives heavily tracked Apple's ecosystem: Avahi mirrored Apple's Bonjour (ZeroConf), while PulseAudio drew direct architectural comparisons to Apple's Core Audio. - Modern operating systems, whether on laptops, bare-metal servers, or virtual machines, operate in highly volatile runtime environments that require an event-driven supervisor rather than a static sequential script runner.
---
[13:45](https://www.youtube.com/watch?v=o_AIw9bGogo&t=825s) The Emergence of the "System Layer"
Rice observes how systemd naturally broadened its scope, transitioning from a basic init daemon into a full-scale system manager. Rather than maintaining the classic, rigid division between user space (applications and daemons) and the kernel (hardware drivers and network stacks), modern operating environments demand an intermediate tier. In the early days, compiling a custom FreeBSD kernel meant manually hardcoding hardware resources—designating specific motherboard slots and IRQs. Today, almost every element of operating system state is volatile and renegotiated on the fly: hotplug devices, dynamic DHCP leases, IPv6 autoconfiguration, and network time synchronization.
Rather than forcing system administrators to glue together fifteen disparate third-party packages—each with conflicting configuration files, unique daemon behaviors, and independent failure states—systemd absorbed these chores into a cohesive tier. Rice terms this the "system layer": a unified body of software (spanning both user and kernel space, such as udev, NetworkManager, and systemd daemons) that underpins core operational facilities below the end-user application level. While this absorption broke long-established expectations and sparked considerable community angst, it answered the operational need to stop continuously reinventing baseline platform glue.
---
[16:12](https://www.youtube.com/watch?v=o_AIw9bGogo&t=972s) Adoption Across Distributions and the Debian Schism
Despite the friction, systemd gained rapid, sweeping adoption across the Linux ecosystem, ultimately running on the overwhelming majority of the Linux installed base. The transition was far from painless, and Debian's adoption process proved particularly traumatic. Because Debian historically supported alternative kernels—such as Debian GNU/kFreeBSD—switching to an init system tied exclusively to Linux-specific kernel interfaces provoked an intense, prolonged debate. Although Debian ultimately voted to adopt systemd, the fallout was severe: the rancor and personal friction of the debate drove multiple members of Debian's Technical Committee and governance leadership to resign.
``` Distribution Adoption Timeline: • Fedora 15 (May 2011) • Arch Linux (October 2012) • Red Hat Enterprise Linux 7 (June 2014) • Debian 8 "Jessie" (April 2015) • Ubuntu 15.04 (April 2015) • openSUSE 12.1 / Leap ```
---
[17:18](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1038s) Refuting Common Critiques: The Unix Philosophy and Software Bugs
Rice systematically dismantles the most common grievances leveled against systemd:
1. **"It violates the Unix philosophy."** Opponents frequently allege that systemd violates the principle of "do one thing and do it well" by acting as an all-encompassing binary. Rice dismisses this as factually incorrect: systemd is not a single executable doing everything from DHCP to NTP, but a modular suite of distinct binaries developed within a single unified source tree. Coming from FreeBSD, Rice notes that BSD developers have used central monorepos for decades to coordinate userland utilities. 2. **"It is bloated and monolithic."** While acknowledging why people feel unsettled when one upstream repository inhales so many separate utilities, Rice explains that unifying configuration schemas and service lifetimes across the operating system requires shared architectural plumbing. 3. **"It is buggy."** Addressing the claim that systemd is too unstable for core infrastructure, Rice argues: “All software has bugs, some bugs are gonna be security issues.” While critics demand that an init replacement must be virtually flawless, imposing an impossible bar simply locks systems into antiquated designs. Furthermore, systemd possesses a sensible failure mode: when it hits a fatal internal state, it goes catatonic rather than immediately panicking the entire machine, allowing administrators to access recovery tools and reboot on their own terms.
To illustrate how bug debates devolve into tribalism, Rice recounts an exchange involving developer Matthew Garrett regarding a memory corruption vulnerability in `systemd-journald` caused by allocating dynamic memory directly on the stack:
| Speaker | Argument / Context | |:--- |:--- | | **Matthew Garrett** | Commented on the journald stack vulnerability: “The moral is not don't use systemd, the moral is you know maybe don't use C for these things.” | | **OpenLDAP Advocate** | Retorted that the true takeaway was simply never to adopt or rely on systemd. | | **Matthew Garrett** | Replied silently with a link to the long historical catalog of CVEs in OpenLDAP, whose chief architect is Howard Chu, underscoring that legacy C codebases are equally prone to severe memory errors. |
Rice insists that engineers must maintain an intellectual boundary between code flaws and system architecture: “There's an important distinction to be drawn between systemd the implementation and systemd as a set of ideas.”
---
[20:26](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1226s) Personality Clashes and Non-Portability
Rice moves to the interpersonal backlash surrounding Lennart Poettering's combative persona on GitHub and mailing lists. While declining to defend Poettering's abrasive community management, Rice highlights the sheer persistence required to overhaul the Linux plumbing layer: > “You have to admire the willpower and determination of someone who at work one day goes, ‘I'm gonna write my own init system—actually no, it's going to be a full-on system layer for Linux,’ and takes it to his management and goes, ‘Hey, I'm gonna do this,’ and they go ‘No,’ and he goes ‘No way,’ and then he comes back with it and it works, and then he gets it put into pretty much every major Linux distribution.”
Lastly, Rice addresses the grievance closest to his background as a FreeBSD developer: portability. Systemd deliberately rejects POSIX-agnostic abstraction in favor of deep, aggressive integration with Linux-specific kernel primitives, most notably cgroups, autofs, and specialized process-tracking hooks. Rice acknowledges that this tight coupling risks stranding non-Linux operating systems that lack matching kernel APIs. However, systemd reaps immense functional capabilities precisely because it abandons generic compatibility. Dismissing the argument that software should forever adhere to lowest-common-denominator Unix APIs, Rice provocatively concludes: “The reality is that UNIX is dead.”
[22:18](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1338s) From Pathological Diversity to Monoculture
In the early days of UNIX and the emergence of POSIX, system design was driven by the imperative to maintain portability across an extraordinarily fragmented computing landscape. Code written on 4.3BSD running on a VAX had to be able to compile and run on SunOS, AIX, Digital UNIX, and numerous other vendor implementations. Today, however, that computing landscape has shifted dramatically. As Benno Rice remarks:
> “These days I think you've basically got Linux and some rounding errors.”
Speaking as someone who has devoted a significant part of his life and career to FreeBSD, Rice notes that FreeBSD is no longer in a position to push a proposed interface onto the broader community; if it tries, the Linux ecosystem simply moves on without caring. Conversely, Linux can introduce innovations like control groups (cgroups) on a take-it-or-leave-it basis. While this evolution marks a transition from pathological diversity to a pathological monoculture where a single platform dictates the terms, Rice argues that this break from tradition does not have to be a net negative. Freed from the constraints of maintaining strict adherence to vintage UNIX dogma, alternative operating systems like FreeBSD could similarly feel liberated to experiment and invent whatever they want without constantly worrying about what historic UNIX standards dictate.
```mermaid flowchart TD accTitle: Evolution of UNIX Diversity accDescr: Diagram contrasting historical multi-platform UNIX portability with modern Linux dominance and cgroups.
subgraph Hist["Historical UNIX Landscape"] V["4.3BSD on VAX"] S["SunOS"] A["AIX"] D["Digital UNIX"] P["POSIX / Portability Consensus"] V --- P S --- P A --- P D --- P end
subgraph Mod["Modern Landscape"] L["Linux Kernel & systemd"] CG["cgroups & native APIs"] L --> CG CG -.->|"Take it or leave it"| O["Other OSes / Rounding Errors (FreeBSD, Solaris)"] end ```
[24:14](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1454s) Features systemd Got Right: cgroups and User Units
Rice emphasizes two particular technical design decisions that make systemd compelling: its deep integration with Linux cgroups and its first-class support for user-level system units. While FreeBSD pioneered lightweight containerization through jails long ago, Rice views jails as considerably less flexible and far less granular than the resource-partitioning and process-tracking primitives afforded by modern Linux cgroups.
The other major operational victory in systemd is its native handling of user-space daemons. On systems like FreeBSD, setting up a daemon that starts and runs automatically under an unprivileged user account without requiring root intervention is notoriously awkward—often forcing administrators into convoluted wrapper shell scripts using `sudo`, `su`, or cron jobs. Under systemd, an unprivileged user can simply drop a service definition file into a configuration directory within their home folder, and the system supervisor manages its execution cleanly.
[25:12](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1512s) The Psychology of Change and the "Tired Old Blankie"
The intense controversy surrounding systemd ultimately boils down to how technologists react to disruptive change. Rice observes that technical communities have an intrinsically fraught relationship with new methodologies:
> “Nerds especially have a really complicated relationship with change: we love it to pieces, change is awesome when we're the ones doing it.”
The moment change is imposed from the outside, it is viewed with deep suspicion because it disrupts the comfortable and familiar. For decades, system administrators grew up with the paradigm of `/sbin/init` executing `/etc/rc`, which in turn stepped through a succession of shell scripts. Rice likens this deep emotional attachment to shell-scripted bootstrap systems to a “tired old blankie that we maybe should have handed off about 20 years ago.”
Because systemd represented a massive, disruptive shift championed by figures like Lennart Poettering—whose empathy toward those he was asking to change did not always appear to be his strongest suit—the pushback turned toxic. Rice unequivocally condemns the harassment directed at Poettering, stressing that no piece of software justifies sending someone death threats. At the same time, he criticizes the smug contempt displayed by parts of the BSD community who treated systemd as a punchline. He points out that ridiculing Linux users for running systemd, or adopting the attitude of *"come join FreeBSD, we'll never change,"* is actively harmful because it frames stagnation as a virtue rather than treating systemd as a valuable source of architectural ideas.
- The backlash against systemd revealed an empathy deficit on both sides: developers driving disruptive changes without sympathy for existing users, met by disproportionate venom and abuse from the community. - Contemptuous dismissal of systemd blinds alternative OS communities to genuine engineering advances and discourages constructive competition. - Rice argues that detractors should ask why systemd was created, identify the specific problems it solves, and attempt to build superior alternatives rather than clinging to 1970s shell scripts.
[29:04](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1744s) Shifting Paradigms: The Next Generation and Modern Architectural Ideas
The perspective of incoming engineers is vastly different from that of veteran "Greybeards." New system administrators and developers do not conceptualize infrastructure through the lens of traditional UNIX files and scripts. Instead, they operate natively with cloud infrastructure and application containers:
> “The next generation of people who are coming into IT, coming into software engineering, coming into systems administration, they don't necessarily think about the systems they're running in the way that we do or that I do.”
To highlight this shift, Rice reflects that twenty years ago, the concept of provisioning a UNIX instance simply by hitting an HTTP API hosted by an online bookstore would have sounded astonishing. Modern developers naturally think in terms of web APIs and containerized workloads, making API-driven system supervision a baseline expectation.
Rice details several high-level architectural ideas embodied by systemd that operating system designers should study: 1. **Unified Messaging Transports and RPC Frameworks:** systemd relies heavily on D-Bus. Although Rice is not personally fond of D-Bus itself, he strongly endorses structured messaging over unstructured text. Apple solved this similarly using Mach messaging primitives. Rice envisions an alternative, kernel-resident message transport featuring robust authentication, security policies, and access-control endpoints. Such a mechanism acts as an equalizer, allowing higher-level userland software to issue uniform service requests (such as configuring a network address) without having to care whether the backing provider resides in kernel space or user space. 2. **Proper Service Lifecycles:** Unlike classic UNIX bootstrap scripts, systemd provides an integrated state machine for service lifecycles out of the box—capabilities that Windows NT had from its inception and that macOS achieved through `launchd`, sparing administrators from having to duct-tape disparate third-party process supervisors like OpenRC, runit, or supervisord onto their systems. 3. **Appliance Automation and Declarative Configuration:** Exposing system management through formal APIs rather than ad-hoc shell scripts radically simplifies the development of hardware appliances, embedded distributions like OpenWrt, and commercial enterprise storage products. 4. **Predictable Hardware Naming Schemes:** Despite having spent years advocating the FreeBSD tradition where every Ethernet device takes the name of its specific kernel driver (and despite disliking legacy Solaris disk identifiers like `c0t0`), Rice praises systemd's predictable network interface naming for deriving consistent, reliable device names from physical hardware attributes.
[33:51](https://www.youtube.com/watch?v=o_AIw9bGogo&t=2031s) Systemd's Model: Logs, Containers, and Redefining the Application
Benno Rice continues wrapping up his talk by detailing features of systemd that he considers genuine improvements over legacy UNIX paradigms. Stable device naming is significantly better than having network interfaces or disks named arbitrarily after whatever random order the kernel happens to detect them in. He also strongly defends systemd's logging architecture, arguing that binary logs are inherently beneficial provided administrators have the proper tooling to inspect and dissect them. An append-only log file offers clear security and integrity benefits, particularly when it can be effortlessly streamed and shipped off-box to central log collectors.
```mermaid flowchart LR accTitle: Systemd Modern Application Model accDescr: Shift from traditional single-binary processes to encapsulated, unit-managed services. subgraph Legacy["Traditional UNIX Model"] direction TB B1["Single Binary"] -->|"spawns"| B2["Unmanaged Background Children"] end subgraph Modern["Systemd / Container Model"] direction TB C1["Encapsulated Unit"] -->|"manages & tracks"| C2["Grouped Processes"] C2 -->|"host feedback"| C1 end ```
Beyond logging, the rise of containers alongside systemd has enabled an entirely new conceptual model of an application. Rather than viewing software as a single isolated binary—or a master binary that blindly spawns detached child processes—an application can now be treated as an encapsulated collection of components managed uniformly as a single unit. Rice finds this approach exceptionally powerful. Looking forward, he expresses interest in seeing how Linux could adopt a macOS-style application model, where rich feedback loops between the host operating system and running services allow applications to dynamically react to host environment changes.
- Binary logs are defended on the basis of append-only durability and ease of remote transmission, assuming adequate extraction tools exist. - Applications evolve from solitary binaries into encapsulated, multi-process units managed holistically by the service manager.
[34:51](https://www.youtube.com/watch?v=o_AIw9bGogo&t=2091s) The Catharsis of Tragedy and the Call to Action
Returning to the theme introduced at the start of his presentation, Rice revisits the formal Wikipedia definition of tragedy, which hinges on evoking *catharsis* in the audience. Catharsis—classically defined as the purification or cleansing of deep-seated emotions, specifically pity and fear, through art or sudden emotional transformation—is precisely what he hopes attendees take away from his breakdown of systemd. Rather than reacting with reflexive dread or condescension, users and administrators should let go of their fear and pity regarding systemd and those who adopt it.
Rice reflects on how software engineering mirrors broader realities of life: the world evolves constantly, forcing engineers either to adapt or to waste energy fruitlessly resisting progress. He cautions that not all changes demand blind, permanent commitment upfront—many Linux distributions experimented with or transitioned through multiple intermediate init systems before landing on systemd. Emphasizing thoughtful, logical evaluation over ideological dogma, he delivers a parting challenge before closing his slides:
> “What I would challenge everyone here is look at systemd and try and find at least one thing that you like and then go see if you can implement it.”
[36:17](https://www.youtube.com/watch?v=o_AIw9bGogo&t=2177s) Q&A: Porting launchd to Linux, and Container "DLL Hell"
The audience bursts into applause, and Rice opens the floor for questions, asking attendees to keep their contributions strictly as questions rather than diatribes disguised with a question mark at the end. The first question asks whether porting Apple’s launchd over to Linux would have been a viable alternative, given that Apple had already solved many service-orchestration problems and released parts of their stack as open source. Rice explains that while systemd is aggressively Linux-specific, launchd is equally aggressively macOS-specific. It relies extensively on Mach ports and Apple-specific architectural primitives. Although Apple made exploratory gestures toward porting libdispatch and related plumbing to FreeBSD, the effort eventually petered out because Apple’s primary incentive was refining the framework for their own platform, just as the systemd project focuses entirely on Linux.
A subsequent questioner notes that while containers allow services to be bundled as clean units, isolating software this way seems to forfeit decades of progress made in dynamic library management, potentially landing developers right back into a modern variant of DLL hell. Rice pushes back on the comparison, clarifying his view of DLL hell:
> “Well I think in a... so DLL hell to me is more having five different versions of a library and always having your applications pick the wrong one.”
He points out that containers explicitly avoid that classic runtime ambiguity by encapsulating the exact required dependencies together, functioning much like the Go programming language does by statically linking binaries into self-contained operational artifacts.
- launchd depends deeply on Mach ports and Darwin internals, making cross-platform ports to Linux or FreeBSD impractical without heavy platform-level emulation. - Containers avoid traditional DLL hell by preventing mismatched system shared libraries from colliding at runtime.
[38:23](https://www.youtube.com/watch?v=o_AIw9bGogo&t=2303s) Q&A: Community Backlash, Chilling Effects, and FreeBSD Init
Following an invitation from an audience member for Rice to organize a systemd microconference at the upcoming Linux Plumbers Conference in Lisbon, Portugal, another attendee asks whether the venomous community backlash directed at systemd has created a chilling effect that discourages developers from attempting ambitious system-level innovation. Rice concedes that the intense vitriol aimed at Lennart Poettering and other systemd developers likely has discouraged people. Writing low-level bootstrap code is vastly harder than most engineers realize, but managing the social dynamics of community acceptance is harder still. Rice quips that his own struggle with navigating social consensus when advocating for modern init systems on FreeBSD eventually led him to simply yell at the project in a keynote address.
When a Linux user interested in FreeBSD asks about the current state of a next-generation init system in the FreeBSD world, Rice points to the OpenRC project, which provides service management capabilities. However, he stresses that OpenRC does not represent a holistic operating system layer:
> “There are far too many people who immediately default to this notion that systemd is only about starting and managing services, when in my opinion if you're really going to do that kind of thing you need to look at the whole thing.”
Rice emphasizes that building a modern system layer does not require packing every feature into a single repository, but it absolutely requires designing with cohesive, system-wide intent so the entire stack behaves uniformly.
[41:11](https://www.youtube.com/watch?v=o_AIw9bGogo&t=2471s) Q&A: Scope Creep, Embedded Devices, and the Broken Social Dynamic
A questioner brings up the sheer volume of previously distinct system services that systemd has gradually subsumed, asking whether consolidating all of them actually made sense. In response, Rice proposes a thought experiment: if the systemd developers had initially focused on formalizing clean, standardized management APIs for networking, time synchronization, and logging, external implementations could have remained interchangeable without needing to live in one monolithic codebase. However, he sympathizes with how project scope expands organically in practice—developers usually do not realize an abstraction boundary is needed until after they bring an external service in-house to solve an immediate integration roadblock.
Addressing a query about the merits of systemd in embedded Linux environments, Rice notes that systemd provides a robust hook for devices operating in dynamic hardware conditions, where peripherals are hotplugged and services must recover without human intervention. Standardized system APIs reduce the bespoke daemon-monitoring code embedded developers are forced to write from scratch.
Finally, a questioner asks how to bridge the toxic divide between a defensive systemd development team and an equally hostile user community. The attendee highlights that the developers often dismiss community input while users refuse to acknowledge systemd's broader architectural vision:
> “Given the interactions between the community and the systemd team in both directions have not always been the most positive to put it mildly... what needs to change both in the community and say this room and broadly and also in the systemd team in order to facilitate a better working relationship going forwards?”
Rice acknowledges that untangling this deep-seated friction is an extraordinarily difficult problem, suggesting as a first step that people watch Adam Harvey's talk on handling language version upgrades and technical transitions gracefully.
[45:14](https://www.youtube.com/watch?v=o_AIw9bGogo&t=2714s) Balancing Community Demands and Developer Empathy
Responding to an audience question about managing upgrades and encouraging user adoption, Benno Rice highlights the importance of providing constructive incentives rather than purely negative reinforcement. He points out that getting users to transition requires showing them genuine reasons to change, rather than merely beating them over the head with reprimands not to do something. At the same time, Rice acknowledges the overwhelming pressure faced by the maintainers of systemd when dealing with the wider open-source ecosystem.
Addressing the friction between the maintainers and the community, Rice admits that while he speaks from outside the project, the sheer volume of conflicting requirements creates an untenable situation for any engineering team. Users and distributions frequently arrive with mutually exclusive feature requests and design expectations, creating an environment where friction is almost inevitable. While that environment explains the defensive posture often adopted by upstream developers, Rice argues that repairing this fraught dynamic requires mutual understanding rather than dismissiveness.
> “That’s a really hard set of constraints to try and work within, and so I can see that there would be a temptation to just tell them all to get nicked; that doesn't mean that that's the right thing to do.” — Benno Rice
> “I don't really know the right way for them to fix it, but it probably involves a lot more sympathy on both sides.” — Benno Rice
- **The challenge of conflicting demands**: Developers face simultaneous, irreconcilable requests from disparate community factions (e.g., wanting $X, Y, Z$ while others demand $A, B, C$), where numerous requests directly contradict one another. - **The cultural divide**: The friction between systemd’s core authors and the broader open-source user base has led to entrenched hostility, requiring conscious empathy from both sides to resolve.
---
[46:16](https://www.youtube.com/watch?v=o_AIw9bGogo&t=2776s) Pervasive Messaging, RPC, and In-Kernel Transports
When asked by an audience member whether there are specific technical areas where systemd should push significantly further, Rice points directly toward system-wide inter-process communication (IPC) and Remote Procedure Calls (RPC). He notes that while systemd already leverages D-Bus extensively, he would like to see standardized messaging and RPC primitives become far more pervasive, deeply integrated, and thoroughly explored across the entire operating system stack.
Rice reflects on previous efforts within the Linux ecosystem to introduce a high-performance D-Bus-style transport mechanism directly into the Linux kernel (such as kdbus or BUS1). He argues that regardless of whether the implementation uses D-Bus semantics specifically or an alternative design, moving messaging primitives into the kernel layer offers major architectural advantages. By elevating IPC to a first-class kernel citizen, diverse system components are placed on an equal footing, unlocking more granular access controls, enhanced telemetry, and sophisticated security boundaries that user-space daemons cannot enforce alone.
> “I would very much like to see the messaging and RPC stuff become a lot more pervasive and a lot more thought about, if that makes sense.” — Benno Rice
> “I think that having that, even if it's not necessarily D-Bus, would be a really interesting thing to play with because it puts a lot of code on the same level and it also allows a lot more, yeah, control and interesting security features that you could do by having it in the kernel.” — Benno Rice
```mermaid flowchart TD accTitle: Potential Architecture for In-Kernel IPC accDescr: Diagram showing user-space system daemons and utilities communicating through an in-kernel messaging and RPC transport layer.
subgraph UserSpace["User Space"] SD["systemd Daemons"] App["System Services & Apps"] end
subgraph KernelSpace["Kernel Space"] IPC["In-Kernel Messaging / RPC Transport\n(D-Bus style / kdbus / BUS1)"] Sec["Security & Access Control Enforcement"] end
SD <-->|"Uniform RPC Interface"| IPC App <-->|"Uniform RPC Interface"| IPC IPC --- Sec ```
Following this response, the session chair announces that the presentation has run out of time, thanking Benno Rice as the audience concludes the session with loud applause.
## Notable quotes > [17:18](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1038s) There's an important distinction to be drawn between systemd the implementation and systemd as a set of ideas. — Benno Rice
> [25:12](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1512s) Nerds especially have a really complicated relationship with change: we love it to pieces, change is awesome when we're the ones doing it. — Benno Rice
> [10:49](https://www.youtube.com/watch?v=o_AIw9bGogo&t=649s) For a fast and efficient boot up, two things are critical: to start less and to start more in parallel. — Lennart Poettering
> [20:26](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1226s) The reality is that UNIX is dead. — Benno Rice
> [10:49](https://www.youtube.com/watch?v=o_AIw9bGogo&t=649s) An init system that is responsible for maintaining services needs to listen to hardware and software changes. — Lennart Poettering
> [22:18](https://www.youtube.com/watch?v=o_AIw9bGogo&t=1338s) These days I think you've basically got Linux and some rounding errors. — Benno Rice
## Mentioned Benno Rice, Aurynn Shaw, Lennart Poettering, Matthew Garrett, Howard Chu, Adam Harvey, systemd, FreeBSD, Debian, Fedora, Arch Linux, Red Hat Enterprise Linux, Ubuntu, openSUSE, OpenWrt, UNIX, 4.3BSD, SunOS, AIX, Solaris, Windows NT, macOS, iOS, launchd, Upstart, OpenRC, runit, supervisord, Avahi, PulseAudio, udev, NetworkManager, journald, D-Bus, kdbus, BUS1, OpenLDAP, inetd, Living Computers Museum and Labs, Linux Plumbers Conference, linux.conf.au
## Video description Benno Rice
https://2019.linux.conf.au/schedule/presentation/156/
systemd is, to put it mildly, controversial. As a FreeBSD developer I decided I wanted to know why.
I delved into the history of bootstrap systems, and even the history of UNIX and other contemporary operating systems, to try and work out why something like systemd was seem as necessary, if not desirable. I also tried to work out why so many people found it so upsetting, annoying, or otherwise rage-inducing.
Join me on a journey through the bootstrap process, the history of init, the reasons why change can be scary, and the discovery of a part of your OS you may not even know existed.
linux.conf.au is a conference about the Linux operating system, and all aspects of the thriving ecosystem of Free and Open Source Software that has grown up around it. Run since 1999, in a different Australian or New Zealand city each year, by a team of local volunteers, LCA invites more than 500 people to learn from the people who shape the future of Open Source. For more information on the conference see https://linux.conf.au/
#linux.conf.au #linux #foss #opensource
[Source](https://www.youtube.com/watch?v=o_AIw9bGogo)