1KIND Reviews
Sony's removal of OtherOS provoked researchers to attack the system hypervisor, exposing a static cryptographic nonce that allows hackers to calculate master private keys and run unsigned code permanently.
AI-assisted summary of DCEmu Reviews - 27C3 - Chaos Communication Congress 2010 - fail0verflow - FULL VIDEO — original source. Check the original for full context.
Source published:
Key ideas
- fail0verflow revealed a fatal cryptographic flaw in Sony's PlayStation 3 security: Sony reused a static nonce (m) in its ECDSA implementation, allowing researchers to calculate the master private signing keys using simple algebra from just two signatures. (35:16)
- The team demonstrated a persistent boot-time exploit against isolated SPU loader lv2ldr by triggering a buffer overflow via an oversized revocation list loaded from flash. (33:19)
- Historical analysis by Michael Steil and fail0verflow shows that Sony's removal of the official OtherOS Linux feature directly provoked the community to compromise the console within twelve months, after it had remained secure for over three years. (5:53)
- The researchers introduced AsbestOS, a custom bootloader displacing GameOS at Level 2 that successfully bootstrapped Linux on PS3 hardware and debunked Sony's claim that the PS3 Slim lacked hardware support for Linux. (27:33)
Detailed notes
## Background Published on YouTube by 1KIND Reviews on December 30, 2010, this video records the landmark presentation delivered at the 27th Chaos Communication Congress (27C3) in Berlin by hacking group fail0verflow (bushing, Sven, marcan, and Segher). The show notes outline the team's achievements and reiterate their primary objective: demonstrating that Sony's security design was fundamentally flawed after removing OtherOS, and presenting their tools—such as AsbestOS—to restore Linux across all PlayStation 3 models.
## TL;DR The presentation surveys console security across the seventh generation, emphasizing that the PS3 remained largely unexploited for years only because native Linux support satisfied hobbyists. After Sony revoked OtherOS, researchers dissected the architecture, using the PS Jailbreak USB stack overflow and their custom AsbestOS loader to turn isolated SPUs into decryption oracles. The team documents architectural oversights such as missing W^X protection, unvalidated storage encryption, and an exploitable buffer overflow in lv2ldr's revocation handling. Finally, fail0verflow unveils Sony's critical mathematical blunder—reusing a static nonce in their ECDSA signature implementation—allowing researchers to calculate master private signing keys with basic algebra and completely dismantle the platform's public-key cryptography.
## Atmosphere & style The session carries the relaxed, subversive energy of a classic Chaos Communication Congress talk, punctuated by casual audience polling, dry deadpan wit, and mutual camaraderie among the presenters. The pacing builds steadily from historical context and hardware breakdown into an intellectual triumph, moving from low-level glitching toward an explosive cryptographic punchline that leaves the audience laughing in disbelief at Sony's amateurish errors. It feels like watching a masterclass in reverse engineering, delivered not out of commercial malice, but from a genuine, uncompromising philosophical commitment to hardware ownership and running homebrew code.
## Bulletin - [35:16](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=2116s) fail0verflow revealed a fatal cryptographic flaw in Sony's PlayStation 3 security: Sony reused a static nonce (m) in its ECDSA implementation, allowing researchers to calculate the master private signing keys using simple algebra from just two signatures. - [33:19](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1999s) The team demonstrated a persistent boot-time exploit against isolated SPU loader lv2ldr by triggering a buffer overflow via an oversized revocation list loaded from flash. - [05:53](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=353s) Historical analysis by Michael Steil and fail0verflow shows that Sony's removal of the official OtherOS Linux feature directly provoked the community to compromise the console within twelve months, after it had remained secure for over three years. - [27:33](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1653s) The researchers introduced AsbestOS, a custom bootloader displacing GameOS at Level 2 that successfully bootstrapped Linux on PS3 hardware and debunked Sony's claim that the PS3 Slim lacked hardware support for Linux.
## Key takeaways - [00:58](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=58s) The Homebrew Channel on the Nintendo Wii demonstrated massive community reach, achieving over one million unique active installations across roughly 30 million vulnerable launch-era consoles. - [03:35](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=215s) The presence of official OtherOS support effectively protected the core PS3 hypervisor for over three years because security researchers had no incentive to attack it. - [05:53](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=353s) Console hacking history demonstrates that platform exploitation is overwhelmingly driven by hobbyists pursuing homebrew and Linux hardware ownership rather than commercial piracy. - [09:15](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=555s) The PS3 Cell Broadband Engine relies on Isolated SPUs executing cryptographically signed binaries with hardware-enforced memory isolation from the PowerPC core. - [11:07](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=667s) The PS3 boot sequence alternates execution and decryption between PowerPC hypervisor levels and isolated SPU loaders (metldr, lv1ldr, lv2ldr). - [12:32](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=752s) Unlike the Xbox 360, the PS3 lacks hardware-backed memory encryption, bus integrity hashing, and physical eFuses to prevent firmware downgrades. - [16:55](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1015s) PS3 internal hard drive encryption is structurally broken because static keys and initialization vectors are used without authentication tags, enabling plaintext oracle cut-and-paste attacks. - [20:05](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1205s) Sony's retroactive removal of OtherOS from original 'fat' consoles alienated the exact technical demographic capable of reverse-engineering and breaking its security stack. - [21:07](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1267s) The commercial PS Jailbreak exploit abused the USB configuration descriptor parser with an unhandled zero-length query to smash lv2 kernel C++ vtables without opening the console. - [24:15](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1455s) Sony's lv2 kernel lacked basic W^X memory mitigations, and the lv1 hypervisor acted merely as a virtualization monitor rather than cryptographically verifying executable memory mappings. - [26:20](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1580s) Extracted symmetric HMAC keys allowed the recreation of Factory Service Mode jig dongles, which paired with leaked Sony service binaries to flash arbitrary older firmware versions. - [28:41](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1721s) Because isolated SPUs decrypted SELFs upon request for any Level 2 PowerPC caller without verifying caller identity, researchers used the hardware as an indiscriminate decryption oracle. - [31:31](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1891s) Firmware revocation checks were vulnerable to flash emulation spoofing because the Level 1 hypervisor re-read loaders from flash a second time after initial execution. - [33:19](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1999s) SPU local storage memory layouts placed the revocation list buffer directly ahead of lv2ldr executable code, permitting unchecked length parameters to overwrite running loader logic. - [35:16](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=2116s) Sony's failure to generate a unique random nonce m for ECDSA signatures compromised the entire platform's cryptographic root of trust. - [44:03](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=2643s) fail0verflow clarified that their research was strictly limited to establishing low-level system control and running Linux, rejecting any interest in developing commercial game piracy tools.
## Detailed walkthrough ### [00:03](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=3s) Introduction and the Audience Survey
The session begins with an informal poll of the audience to gauge which consoles people in the room own. A show of hands reveals that some attenders own a Nintendo Wii, a group owns an Xbox 360—prompting a quick nod to Andrew "bunnie" Huang's classic book on Xbox hacking—and an overwhelming majority raises their hands for the PlayStation 3. The moderator remarks that this hardware represents an immense amount of computing power placed directly in users' hands, arguing that under any sensible definition of property and ownership, owners ought to have the freedom to do whatever they wish with their machines. With that premise established, fail0verflow takes the stage to explain how to look inside the PS3 and modern console architectures.
[00:58](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=58s) fail0verflow’s Wii Track Record and BootMii
bushing introduces the speaking group: Sven, marcan, Segher, and himself, representing fail0verflow. Because the group exceeds the presenter limit for the stage, the remaining members of their international team—spanning nearly every continent—watch from the front row. The presenters are well-known to the Chaos Communication Congress community: two years prior at 25C3, marcan and bushing presented their Wii hacks; a year before that at 24C3, bushing demonstrated the first unsigned code execution on the Wii; and in 2008, the team completely reverse-engineered and reconstructed the Wii's security architecture.
Reflecting on the Nintendo Wii, bushing notes that three years have passed since that first unsigned code execution. In that span, Nintendo issued nine separate system firmware updates. Bushing points out that eight of those updates were completely contrived, offering zero useful end-user features: only a single update provided something users actually wanted—the ability to run games from an SD card—while the remaining updates were pushed under the guise of "background improvements to improve system performance," which was simply corporate code for patching security vulnerabilities.
According to sales data, Nintendo sold around 73 million Wii consoles, making it the top-selling home console of its generation. fail0verflow estimates that roughly 30 million of those consoles were manufactured before Nintendo managed to patch the signature hash-check vulnerability the team had disclosed. Those early console owners were able to run BootMii, an alternate bootloader developed by the team. Bushing explains that reverse-engineering the Wii was dominated not by cryptographic protections, but by the extreme fragility of Nintendo's software stack:
> “The software is very, very fragile and there are very, very, very many ways to accidentally brick the device and no way to fix it, so we put time into making something that would simply let you unbrick your Wii.”
Their efforts reached widespread adoption: by September 2010, the Homebrew Channel surpassed one million unique active users, meaning more than one percent of all Wii owners worldwide had installed their homebrew platform.
- **Total Wii sales:** ~73 million units worldwide. - **Vulnerable units produced:** ~30 million consoles manufactured prior to the boot ROM/hash-check fix. - **Homebrew Channel milestone:** Over 1,000,000 unique installations logged. - **Update track record:** 9 firmware updates pushed by Nintendo, with 8 serving exclusively to patch exploits.
[03:35](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=215s) The Console Landscape: Wii, Xbox 360, and PS3
Sven steps up to survey the state of console security across the three major seventh-generation home systems: the Wii (late 2006), the Xbox 360 (late 2005), and the PlayStation 3 (late 2006).
The Wii was compromised almost immediately upon arrival. Initial drive chips enabled pirated disc backups and GameCube homebrew, and within roughly a year, unsigned native code execution was achieved. Although Nintendo repeatedly shipped patches, each attempt failed to regain control, leaving the console permanently open to homebrew.
The Xbox 360 stands in sharp contrast. Sven acknowledges that Microsoft built an exceptionally solid security architecture. Native code execution was achieved only via two isolated hardware-dependent flaws: the King Kong shader exploit and the later JTAG hack. Microsoft quickly revised hardware and patched system software to close those gaps. While Xbox 360 DVD drives were thoroughly compromised by custom drive firmware, Sven dismisses drive flashing entirely from a hacker's perspective:
> “Well, we don't care about piracy: it does not allow homebrew code, and we don't care about running code others wrote; we want to run our own code.”
The PlayStation 3 followed a completely different trajectory. Sony originally included official "OtherOS" support, allowing users to install Linux out of the box. Because Linux was officially supported, the hacker community had little incentive to attack the system's hypervisor—aside from exploring workarounds to gain hardware-accelerated 3D graphics, which Sony immediately patched out in firmware.
The turning point came when Sony launched the PS3 Slim in late 2009 without OtherOS support. Sony claimed removing Linux was necessary to cut manufacturing and engineering costs. Sven rejects this explanation as disingenuous, pointing out that Linux runs on the Slim hardware without major code alterations. Once OtherOS was stripped away, hacker interest surged. A few months later, George Hotz (geohot) released a hardware glitch exploit targeting the hypervisor, prompting Sony to retaliate by stripping OtherOS from original "fat" consoles via firmware update 3.21. That decision provoked the release of the PSJailbreak USB exploit in late summer 2010. Sven warns that by the end of their presentation, fail0verflow's newly discovered chain of software and cryptographic flaws will leave the PS3 looking just as broken as the Wii.
[05:53](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=353s) Guest Cameo: Michael Steil on "Linux Is Inevitable"
Michael Steil (of the Xbox-Linux project) joins the stage for a cameo. Steil explains that he has spoken on console hacking at the Chaos Communication Congress for eight years running, tracking broader industry patterns and statistics. His central thesis has remained constant:
> “Linux is inevitable: either you support Linux on your hardware, or it will be hacked so it'll run sooner or later.”
Steil revisits a comparative chart he first presented three years earlier, analyzing console release dates, relative security complexity, and the duration required to achieve unauthorized execution. Historically, devices with sophisticated, multi-layered security architectures took longer to fall, whereas smaller handheld devices with constrained hardware budgets succumbed faster.
Crucially, Steil notes that with the exception of the original PlayStation, every single console hack across gaming history was driven primarily by hobbyists determined to run homebrew and Linux on hardware they purchased. Piracy was merely an inevitable byproduct of broken digital rights management and leaked signing keys, not the primary engineering motivator.
Steil then updates his historical dataset to address the PlayStation 3:
```mermaid flowchart TD accTitle: Steil's Revised PS3 Vulnerability Timeline accDescr: Flowchart illustrating the timeline shift where the PS3 went unhacked for 3.5 years under OtherOS, but fell within 12 months after Sony removed Linux.
subgraph Phase1["Official Linux Era (Nov 2006 - Fall 2009)"] A["PS3 Launches with OtherOS Support"] --> B["Hackers Satisfied Running Linux"] B --> C["Core Hypervisor Unhacked for ~3.5 Years"] end
subgraph Phase2["Closure & Retaliation (Late 2009 - Fall 2010)"] D["Sony Drops OtherOS on Slim (Sept 2009)"] --> E["geohot Hypervisor Glitch (Jan 2010)"] E --> F["Sony Revokes OtherOS on Fat PS3 (Apr 2010)"] end
subgraph Phase3["Full Compromise (Late 2010)"] G["PSJailbreak USB Exploit Appears (Aug 2010)"] --> H["fail0verflow Full Crypto Breached (Dec 2010)"] end
C --> D F --> G ```
While casual observers assumed the PS3 held out unhacked for four full years due to superior security engineering, Steil demonstrates that this conclusion is flawed. As long as Sony supported Linux, the console remained secure because the homebrew community had no reason to assault the core hypervisor. Once Sony closed the platform by withholding OtherOS on the Slim and stripping it from existing consoles, it took only twelve months for the machine to be completely broken wide open. That 12-month window matches the exact time-to-exploit trajectory seen on the Xbox 360, proving that hardware openness, rather than complex cryptography, was what kept the PS3 secure.
[09:15](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=555s) The PS3 Architecture: Cell Broadband Engine and Isolated SPUs
Sven returns to delve into the PlayStation 3's underlying hardware architecture, explaining how Sony designed the Cell Broadband Engine (Cell BE) in partnership with IBM and Toshiba.
The Cell processor features a heterogeneous multi-core design: 1. **Power Processing Element (PPE):** A standard 64-bit PowerPC dual-threaded core located at the foundation of the chip. 2. **Synergistic Processing Elements (SPE / SPU):** Eight independent vector processing coprocessors flanking the PPE.
Each SPU is equipped with exactly 256 kilobytes of dedicated SRAM known as Local Storage (LS). An SPU cannot directly address main system RAM; instead, it relies entirely on Direct Memory Access (DMA) transfers to move code and data between system memory and its private Local Storage.
Under normal operating conditions, the PPE has full supervisor visibility over the SPUs. The PPE can read and write directly to each SPU's Local Storage, halt instruction execution, inspect internal vector registers, and run interactive hardware debuggers.
However, IBM and Sony engineered an advanced hardware security mechanism known as **Isolation Mode** (Isolated SPU mode):
```mermaid flowchart TD accTitle: SPU Local Storage in Normal vs Isolated Mode accDescr: Comparison of SPU Local Storage memory access permissions between standard operational mode and hardware isolated mode.
subgraph Normal["Normal Mode"] N1["PowerPC (PPE)"] -->|"Full Read / Write / Debug Access"| N2["SPU Local Storage (256 KB)"] N2 -->|"DMA Access"| N3["Main RAM"] end
subgraph Isolated["Isolated Mode"] I1["PowerPC (PPE)"] -->|"Restricted Access"| I2["Mailbox / Comm Area (Green)"] I1 -.->|"BLOCKED by Hardware"| I3["Secure Code & Data (Red Area)"] I4["Hardware Decryption Engine"] -->|"Decrypts & Verifies Authenticated Binary"| I3 end ```
When an SPU enters Isolation Mode, a cryptographically authenticated and encrypted binary is fed into the processor. The hardware itself decrypts and verifies the signature of the binary before execution begins. Once isolated: - All external PPE debugging, single-stepping, and register inspection features are physically disabled by hardware logic. - The vast majority of the 256 KB Local Storage becomes completely inaccessible to the PowerPC core. - The PPE is restricted to interacting with a tiny, designated communication buffer (the mailbox area) within Local Storage.
Sony utilized this hardware isolation boundary to execute critical cryptographic operations, DRM validations, and access controls away from the rest of the system. Sven sets up the next layer of the stack: sitting above the PPE is the PlayStation 3 hypervisor, which virtualizes the hardware into distinct logical partitions to enforce system privileges.
[11:07](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=667s) PS3 Execution Modes and Interleaved Boot Chain
On the PlayStation 3, execution is segmented across privilege levels and processor types. The primary software layer running at Level 2 is GameOS, an operating-system kernel providing system and runtime support features for games. The games themselves execute in "problem state," the standard PowerPC user mode. Level 1 (the hypervisor) and Level 2 interact directly with the Synergistic Processing Units (SPUs) operating in isolated mode. In typical operation, a game triggers an SPU routine by issuing a hypercall up to Level 1, instructing it to load an isolated loader onto an SPU to perform cryptographic operations like decryption, though games can also execute isolated SPU tasks directly.
```mermaid flowchart TD accTitle: PS3 Privilege and Boot Architecture accDescr: Flow diagram showing the interleaved execution between PowerPC levels and isolated SPUs during the boot sequence.
subgraph PPC["PowerPC Core"] L0["Level 0: Hypervisor Mode<br/>(Bringup Code)"] L1["Level 1: Hypervisor"] L2["Level 2: GameOS Kernel"] Game["Game / Application<br/>(User Mode / Problem State)"] end
subgraph SPU["Isolated SPUs"] ML1["metldr<br/>(Isolated SPU)"] L1L["lv1ldr<br/>(Isolated SPU)"] ML2["metldr<br/>(Isolated SPU)"] L2L["lv2ldr<br/>(Isolated SPU)"] end
L0 -->|"loads"| ML1 ML1 -->|"decrypts & loads"| L1L L1L -->|"decrypts & loads"| L1 L1 -->|"loads"| ML2 ML2 -->|"decrypts & loads"| L2L L2L -->|"decrypts & loads"| L2 L2 -->|"runs"| Game ```
The boot sequence relies on an interleaved handoff between the PowerPC core and isolated SPUs. At power-on, the bootloader initializes Level 0, low-level PowerPC code running in hypervisor mode whose primary purpose is to bring up the main processor and prepare an isolated SPU. That SPU loads `metldr`, an isolated binary that in turn decrypts and runs `lv1ldr`. Once decrypted, `lv1ldr` loads the Level 1 hypervisor. Level 1 continues this multi-stage relay by bringing up another isolated SPU running `metldr` to decrypt `lv2ldr`, which finally decrypts Level 2 (GameOS) into memory. Level 2 then initializes kernel mode and launches userland games. This handoff forms an alternating chain where the PowerPC and the SPUs repeatedly call and decrypt each other's subsequent stages.
[12:32](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=752s) Security Comparison: Xbox, Xbox 360, Wii, and PS3
The security architectures of the major home consoles share several core primitives, though each platform implements them with distinct strengths and flaws. A baseline feature across modern consoles is a hidden boot ROM containing privileged initialization code. While the original Xbox embedded this boot ROM inside the external Southbridge chip, every subsequent console relocated it directly inside the main CPU silicon, making it impossible to read externally or bypass via modchip bus interception. To enforce execution boundaries, every platform relies on public-key cryptography to authenticate executable binaries before launch.
| Security Feature | Xbox | Xbox 360 | Wii | PlayStation 3 | |:--- |:--- |:--- |:--- |:--- | | Hidden Boot ROM in CPU | No (in Southbridge) | Yes | Yes | Yes | | Public-Key Cryptography | Yes | Yes | Yes | Yes | | On-Die Key Storage | No | Yes | Yes | Flawed ("doesn't work very well") | | Chain of Trust from Boot | Broken | Yes | Broken (checks only at install) | Yes | | Per-Console Unique Keys | No | Yes | Yes | Yes | | Signed Executables | Yes | Yes | No (signs whole packages) | Yes | | Dedicated Security Co-Processor | No | No | Yes (Starlet) | Yes (Isolated SPUs) | | Encrypted Disc Filesystem | No | No | Yes (Full image hash tree) | No | | Encrypted Internal Storage | No | No | Yes (NAND + HMAC-SHA1) | Yes (HDD per-console key) | | Hardware Memory Encryption/Hashing | No | Yes | No | No | | Privilege Separation Layers | None | Kernel, Hypervisor | None (All Kernel on PPC) | User, Kernel, Hypervisor | | Anti-Downgrade Hardware | None | Yes (eFuses blown on update) | No | No |
Key storage and trust verification diverge sharply across designs. While the Xbox 360 and Wii securely bury their master keys in on-die storage, the PS3's implementation is compromised by design oversights. Similarly, the Wii disrupts its own cryptographic chain of trust because its operating system checks signatures exclusively during installation rather than validating binaries dynamically at every boot. Unlike the Xbox 360 and PS3, which cryptographically sign each individual executable, the Wii validates package archives.
Dedicated security processors protect internal logic: the Wii isolates all system tasks inside an ARM-based co-processor dubbed "Starlet," while the PS3 leverages isolated SPUs. For optical media, the Wii deploys an exhaustive disc encryption scheme using a complete hash-tree structure that validates data blocks on the fly as they are read, preventing file tampering.
On the storage front, both the PS3 (internal hard drive) and the Wii (NAND flash) encrypt internal media with unique per-console keys. However, Nintendo pairs its encryption with HMAC signatures to prevent offline tampering, whereas the PS3 implements bare encryption without integrity hashing.
In terms of raw memory defense, the Xbox 360 stands alone: it actively encrypts and cryptographically hashes external bus traffic to RAM. This eliminates direct memory tapping, memory-bus glitching attacks, and rogue DMA injection into protected regions. The Xbox 360 also incorporates hardware eFuses within the CPU die; firmware updates permanently blow specific fuse banks, making hardware downgrades impossible unless an attacker compromises the boot chain before the fuse-check routine runs.
[16:55](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1015s) Flaws in PS3 Storage Encryption
Despite the PS3’s hypervisor architecture, Sony made catastrophic design blunders in its storage security. The PS3's hard drive encryption uses the identical key and initialization vector (IV) across every single disk sector without applying any cryptographic hash or authentication tag. This static construction enables a straightforward cut-and-paste plaintext oracle attack that completely compromises storage privacy:
> “They pulled off this neat trick where you can copy a file into your PS3 like a movie or something then you can find those sectors by watching what changed on the drive then you take another chunk of the drive copy it on top then read the movie back and it decrypts those sectors that you copied on top for you so you can do that to decrypt any chunk of the hard drive and since they don't verify anything and everything uses the same key and the same IV it just works so that encrypted storage is pretty much screwed up to begin with.”
By watching which raw hard drive sectors change when a known user file (such as a video) is written, an attacker identifies where the ciphertext landed. Copying arbitrary encrypted system sectors directly over that file's sectors on the raw disk and reading the file back through GameOS forces the hardware disk controller to decrypt the target data into user space.
[17:40](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1060s) The Removal of OtherOS and geohot's Memory-Bus Glitch
For years, the PS3’s native "OtherOS" capability served as an effective deterrent against exploitation: because enthusiasts could readily install and run Linux natively, security researchers lacked a compelling motive to attack the core system. That dynamic changed when Sony released the PS3 Slim and stripped OtherOS from the product line without any hardware-level technical justification. Sony's decision immediately invited intense scrutiny from the hacker community.
Taking up the challenge, George Hotz ("geohot") developed an exploit inspired by Team Twiezers' earlier Nintendo Wii memory attack. While the Wii exploit relied on a pair of tweezers to short address lines on the external RAM bus, geohot devised a hardware glitch targeting the PS3's memory bus. The attack targeted the boundary between GameOS and the hypervisor:
```mermaid flowchart TD accTitle: geohot Memory Glitch Attack Sequence accDescr: Step-by-step memory allocation race condition induced by glitching the PS3 RAM bus.
A["Allocate memory buffer in Kernel"] --> B["Hypervisor creates page mapping"] B --> C["Request hypervisor to free allocation"] C --> D["Glitch RAM bus during write cycle"] D --> E["Hypervisor marks memory page as FREE"] E --> F["Hardware write corrupted: Kernel keeps valid page table mapping"] F --> G["Request hypervisor to allocate new Virtual Address Space"] G --> H["Hypervisor allocates new Page Table inside 'free' memory slot"] H --> I["Kernel directly overwrites new Page Table"] I --> J["Map Hypervisor memory into Kernel space"] J --> K["Full Hypervisor Mode Code Execution"] ```
Normally, when the kernel requests memory, the Level 1 hypervisor creates a page table mapping, and when that memory is released, the hypervisor unmaps it to revoke access. Geohot glitched the bus during the precise memory write that invalidates the mapping. While the hypervisor believed the memory block was fully freed and returned to the system pool, the corrupted write left the kernel's translation entry intact, allowing the kernel to continue reading and writing to that raw physical RAM.
By repeatedly asking the hypervisor to allocate a new virtual address space (a new page table), the system eventually assigned that table to the unmapped block. Because the kernel could still write to that memory, an attacker could rewrite the hypervisor's page table directly, map the hypervisor's own memory segments into the kernel's space, and gain full, unconstrained execution at Level 1.
[20:05](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1205s) Sony's Retaliation and the Emergence of PS Jailbreak
Geohot’s hypervisor glitch was primarily an academic proof-of-concept. It was notoriously finicky, required intrusive hardware modifications to trigger, and produced raw hypervisor dumps that few developers at the time could utilize. However, Sony responded aggressively. Instead of ignoring the niche hardware hack, Sony pushed mandatory firmware updates that retroactively ripped OtherOS out of original "fat" PlayStation 3 consoles as well.
The team emphasizes how disastrously this move backfired. Beyond alienating everyday consumers and raising legal concerns regarding European consumer protection laws, Sony targeted the exact demographic capable of dismantling their security:
> “The worst part of this is that the people who use OtherOS are the hackers so by doing this Sony pissed off the hackers that's a really really bad idea... in other words they are so getting hacked now.”
The retaliation galvanized the scene. After a brief period of quiet underground development, the console's security collapsed in rapid succession with the commercial arrival of the PS Jailbreak USB dongle, followed immediately by an explosion of open-source clones.
[21:07](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1267s) The PS Jailbreak USB Stack Glitch and Vtable Overwrite
The original PS Jailbreak shocked the console security scene by granting arbitrary code execution at the Level 2 (lv2) kernel layer without hardware modifications. Under the hood, the dongle acts as a USB hub exposing several virtual USB devices to orchestrate a precise heap-grooming and memory-corruption sequence. The process focuses on four logical devices. Device 1 first delivers the exploit payload hidden inside a standard USB descriptor directly into lv2 kernel memory space, where it sits inert in data memory. Next, Device 2 is plugged in and quickly unplugged, intentionally leaving behind residual uninitialized bytes in an internal kernel allocation buffer, specifically the sequence `04 21 b4 2f`.
Device 4 then executes the critical protocol glitch. In standard USB enumeration, an operating system solves the chicken-and-egg problem of reading variable-length configuration descriptors by reading the first eight bytes to determine total length, followed by a second read requesting the complete descriptor payload. The PS Jailbreak complies during the initial eight-byte query with a valid length, but during the follow-up request, it abruptly returns a length of zero. The lv2 USB stack panics into an unhandled state: it skips copying the data while continuing forward to parse whatever stale bytes occupy the destination buffer.
The stale memory left behind by Device 2 causes the kernel to interpret `04 21 b4 2f` as a configuration descriptor with a massive declared length of `0x2fb4` bytes. When lv2 attempts to parse subsequent descriptors, it offsets forward by `0x2fb4` bytes—well beyond the allocated buffer—and lands directly inside an array of kernel C++ objects allocated by Device 3. The rogue descriptor payload overwrites the virtual method tables (vtables) of these C++ objects with pointers redirected to the payload planted earlier by Device 1. When Device 3 is unplugged, the kernel calls its destructors, branching straight into attacker code:
> “So I put this configuration descriptor right on top of C++ objects and it overwrites the vtable of C++ objects to point to the payload this V table holds function pointers for things like the destructors and some virtual methods and things like that so when these objects get destroyed um they actually run our own code so when device number three which has a bunch of crap that created these objects gets unplugged we gain lb2 code execution.”
```mermaid flowchart TD accTitle: PS Jailbreak Exploit Flow accDescr: Sequence of USB device emulation leading to lv2 kernel code execution
subgraph MemoryGrooming ["1. Memory Grooming & Glitch"] D1["Device 1: Injects Payload into lv2 Data Buffer"] D2["Device 2: Plugs & Unplugs, Leaves '04 21 b4 2f'"] D3["Device 3: Allocates C++ Object Array"] D4["Device 4: Requests Descriptor, Glitches Length to 0"] D1 --> D2 --> D3 --> D4 end
subgraph Corruption ["2. Buffer Overread & Vtable Hijack"] Parse["lv2 Parses Uninitialized Buffer as Length 0x2fb4"] Overflow["Pointer Steps 0x2fb4 Bytes into Device 3 C++ Objects"] Overwrite["Vtable Pointers Overwritten with Device 1 Payload Address"] D4 --> Parse --> Overflow --> Overwrite end
subgraph Hijack ["3. Execution Trigger"] Unplug["Device 3 Unplugged"] Destructor["Kernel Executes Destructor via Smashed Vtable"] CodeExec["lv2 Kernel Code Execution Achieved"] Overwrite --> Unplug --> Destructor --> CodeExec end ```
[24:15](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1455s) Structural Flaws: Absent W^X and a Hypervisor Not Built for Security
Gaining execution through this exploit exposed severe architectural design flaws in Sony's operating system and hypervisor. In modern secure systems, code execution from data buffers is prevented by Write XOR Execute ($W\oplus X$), marking memory pages as either writable or executable, but never both. Sony's lv2 kernel completely lacked $W\oplus X$ protection. Worse, the underlying Level 1 (lv1) hypervisor failed to enforce hardware-backed execution boundaries. On competing platforms like the Xbox 360, the hypervisor strictly cryptographically verifies all memory pages mapped as executable before scheduling them, ensuring that unsigned code cannot run even if lower privilege layers are breached.
On the PlayStation 3, the hypervisor acts merely as an operating system virtualization layer rather than a security monitor. When lv2 requests executable memory mappings, the hypervisor grants them blindly without checking signatures or origins. This created a catastrophic imbalance: lv1 and the isolated security SPEs remained mathematically uncompromised, yet attackers controlled lv2. Because lv2 handles filesystem access, disc mounting, and storage drivers, an attacker can simply patch lv2 to redirect game disc reads to internal or external hard drives without triggering any alarms from lv1:
> “so you can break 20% of the security and copy games which is 100% of what Sony doesn't want you to do”
- The hypervisor was engineered to isolate virtualized operating systems (like OtherOS and GameOS), not to serve as an active cryptographic enforcement wall against kernel exploits. - Patching lv2 requires zero hypervisor escalations to enable game piracy, demonstrating that Sony's security boundary placement failed to protect core business interests.
[26:20](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1580s) The Downgrade Crack via Factory Service Mode
When Sony issued firmware updates to patch the USB stack vulnerability, the underground community looked to downgrade their systems back to vulnerable versions, following the historical precedent established on the PlayStation Portable (PSP). The PS Jailbreak creators discovered Sony's built-in Factory/Service Mode (FSM), a maintenance mode used by authorized repair facilities to service consoles by booting signed code directly from external USB drives. Entering Service Mode required a specialized USB hardware key—the "jig"—that conducted a cryptographic handshake with the console.
The handshake was implemented using symmetric cryptography based on HMAC. Because lv2 had already been compromised, hackers extracted the symmetric service keys directly from kernel memory and cloned the jig dongles. Soon after, an official, signed Sony service update package was leaked to the public. By inserting a clone jig, powering on the console, and supplying the leaked service application alongside any official firmware update PUP file on a USB stick, the system flashed the firmware without validating whether the version being installed was older or newer than the firmware currently running:
> “and uh there was a service app that was signed that was leaked that lets you uh reinstall the operating system on a PS3 without any kind of downgrade check so you just use this jig then put this on a USB stick put a upgrade or an an system install file on a USB drive and it just installs that without checking if it's older or newer or anything like that and since people can downgrade back to piracy”
[27:33](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1653s) AsbestOS: Linux Restoration and Remote Hypervisor Poking
Rather than pursuing game piracy, the fail0verflow team repurposed the lv2 compromise toward their primary objective: running homebrew and restoring GNU/Linux functionality. They authored **AsbestOS**, a lightweight bootloader designed to run in place of GameOS. Because GameOS and the removed OtherOS shared nearly identical hypervisor call interfaces—differentiated only by internal hypervisor permission tables—AsbestOS successfully hijacked lv2 to bootstrap a standard Linux kernel over the local network via TFTP/NFS.
The software proved that Sony's claim that the PS3 Slim could not support Linux due to hardware differences was entirely unfounded; Linux booted on the Slim hardware without any hardware modification. To accelerate reverse engineering, fail0verflow added a network RPC debugging module into AsbestOS. This turned the PS3 into a programmable "dumb slave" managed over the network from Python scripts on an external PC:
> “and uh we also added a feature to it that lets you make your PS3 into a dumb slave that you can control from your PC and you can use this to experiment with Python scripts with uh poking the hypervisor poking the SES Reading Writing memory making hyper calls basically it's a way to experiment with PS3 and especially it security system really really quickly from a PC environment”
- AsbestOS completely displaced GameOS at level 2, proving that hypervisor virtualization boundaries could host a full Linux distribution without Sony's official OtherOS hypervisor partition. - The Python netRPC bridge allowed the team to dynamically issue hypercalls, poke memory, inspect SPE register states, and map isolated subsystem behaviors in real time from comfortable development workstations.
[28:41](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1721s) Reverse Engineering SELF Binaries and the SPU Decryption Oracle
While AsbestOS enabled Linux via the dongle-and-eject power-on sequence, the team aimed for an untethered solution: turning on the PS3 and booting straight into Linux automatically. Achieving this required understanding Sony's encrypted executable binary format: the Signed ELF (SELF). A SELF file wraps standard Executable and Linkable Format (ELF) code inside complex proprietary crypto headers containing AES keys, SHA-1 digest tables, and digital signatures.
To run a binary, an isolated Synergistic Processing Unit (SPU) loader uses a hardcoded loader key to decrypt a unique SELF encryption key embedded inside the header. This unique key decrypts the header tables, validates the ECDSA digital signature across all header metadata, and verifies the individual SHA-1 section hashes before decrypting the final executable sections into memory. If a single bit in the binary or header is altered, hash verification or signature validation fails immediately, causing the loader to abort.
Sony assumed that keeping all executables encrypted inside SELFs would hide their software implementation details and prevent vulnerability discovery. However, because fail0verflow already held PowerPC code execution inside the lv2 kernel, they could bypass the encryption math altogether. The isolated SPUs act as an indiscriminate decryption oracle: any PowerPC process can submit a SELF binary to the isolated SPU, and the hardware coprocessor will decrypt and unpack the application into system RAM without verifying who made the call. Using AsbestOS and Python automation, the team rapidly decrypted all core operating system files, revealing the entire firmware attack surface:
> “Sony's idea is yeah they can't see our code so they can't find find bucks in there well as soon as we have power PC access we can just ask the spu to nicely decrypt stuff for us and it will do so it does not do any checks on what actually asks you to decry to decrypt things so with this we just use Asbestos and some this net RPC thing write some Python scripts on the PC and use it to like decrypt application s or level two or like just about every we can decrypt just about everything now by just asking it to do it so we don't even need to know how it works we just have to ask it to do it for us but so yeah the alisation is useless”
[31:31](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1891s) Revocation Failures and Downgrade Exploits
The boot process begins with loaders stored in flash memory, but Sony left architectural flaws that make true firmware revocation impossible. The initial stages—bootldr and metldr—cannot be updated because bootldr is the very first code executed upon boot, while metldr loads isolated SPU binaries and is encrypted with a per-console unique key. All subsequent stages can technically be updated by Sony, and to prevent users from downgrading to older, exploitable firmware, Sony introduced revocation lists stored directly on NOR or NAND flash. Loaders check incoming binaries against this list and refuse to boot any revoked versions. However, the earliest loaders do not support revocation lists at all.
Sony attempted to work around this architectural blind spot with a two-step verification sequence: the console first loads bootldr, lv0, metldr, lv1ldr, and lv1 from flash and executes them. Once lv1 is actively running, it re-reads all of those stages from flash a second time to verify their integrity. This implementation creates an obvious physical vulnerability. By replacing the NOR flash with a multiprocessor or FPGA emulator, an attacker can feed the PS3 an older, downgraded version on the first read, and then feed it the current, expected official version on the second read. Because lv1 checks the second read and concludes it is running current firmware, execution continues smoothly. This trick allows users to downgrade the entire system, including the revocation list itself. As the speaker emphasizes: “So their revocation is useless it's just some kind of specification thing there's no revocation there so as soon as we break one loader we break it forever and we can forever own all PS3s out there.”
```mermaid flowchart TD accTitle: Dual-Read Downgrade Bypass accDescr: Diagram illustrating how emulated flash fools lv1 verification by feeding different data on first and second reads.
subgraph EmulatedFlash["Emulated Flash Memory"] R1["First Read Request"] R2["Second Read Request"] end
subgraph ExecutionFlow["Execution & Verification"] BootExec["Load & Execute Downgraded Loaders<br/>(bootldr, lv0, metldr, lv1ldr, lv1)"] VerifyExec["lv1 Re-reads Flash to Verify System"] BootSuccess["Verification Passes:<br/>Firmware Downgraded"] end
R1 -->|"Supplies Old/Downgraded Code"| BootExec BootExec --> VerifyExec VerifyExec --> R2 R2 -->|"Supplies Real/Current Code"| BootSuccess ```
[33:19](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=1999s) Exploiting lv2ldr via Local Storage Buffer Overflow
The architecture isolates SPU local storage into protected and unprotected regions: red areas are accessible only by the isolated SPU itself, while green areas can be accessed by the host system. When supplying a revocation list, the data arrives signed and encrypted. Sony cannot decrypt or verify the signature in place within the unprotected shared memory, because a PowerPC thread could simply wait for validation to complete and overwrite the buffer with malicious code in a race condition. Consequently, the SPU loader must copy the revocation list into isolated local storage before processing it.
```mermaid flowchart LR accTitle: SPU Local Storage Buffer Overflow accDescr: Memory diagram showing oversized revocation list overflowing into lv2ldr code.
subgraph LocalStorage["Isolated Local Storage"] DestBuf["Destination Buffer<br/>(Revocation List)"] LoaderCode["lv2ldr Executable Code"] end
AttackerData["Oversized Revocation List<br/>(Controlled Length)"] -->|"Overwrites beyond boundary"| DestBuf DestBuf -.->|"Overflows into"| LoaderCode ```
The memory layout of lv2ldr reveals a critical vulnerability: the destination buffer for the revocation list sits immediately before the lv2ldr executable code in memory. Because the attacker controls both the source data and the length field of the revocation list, specifying an excessively large length value causes the copy operation to smash the stack and heap, directly overwriting the executable code of lv2ldr inside isolated SPU mode. The attacker can then inject code to dump the loader's encryption keys by pushing them across the bus to the PowerPC core. This allows full reverse engineering of lv2ldr.
Because the hypervisor blindly loads the revocation list from NOR flash without checking its length, putting this payload in flash makes lv2ldr execute attacker code at boot time without needing USB dongles or hardware glitches: “And this breaks the chain of trust pretty early.”
- SPU isolated memory restricts external PowerPC writes to unprotected regions, forcing internal copies. - Destination buffer placement prior to loader code permits linear buffer overflow. - Flash hypervisor fails to validate the length parameter of the revocation list. - Enables persistent, tetherless code execution at boot time and full extraction of isolated keys.
[35:16](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=2116s) Epic Fail: The Broken ECDSA Implementation
While breaking an isolated loader compromised the chain of trust, the team promised an "epic fail" that undermined the system's foundation entirely. The focus turned to how Sony handles cryptographic signatures across the entire platform. The PlayStation 3 relies on ECDSA (Elliptic Curve Digital Signature Algorithm) to authenticate executables, hypervisor configurations, and updates. In ECDSA, security relies on the hardness of the elliptic curve discrete logarithm problem: the signature consists of two numbers, $R$ and $S$. Computing $R$ requires scalar point multiplication on an elliptic curve, an operation that cannot be reversed. The equation for calculating $S$ incorporates the private key $K$ and a per-signature nonce $m$:
$$S = m^{-1}(H(x) + K \cdot R) \pmod N$$
Under standard cryptographic requirements, $m$ must be a cryptographically secure random value generated fresh for every single signature. In proper systems—such as fail0verflow's own implementation in the Wii Homebrew Channel network update mechanism—$m$ is sourced from `/dev/urandom` on Linux. Sony, however, hardcoded a static value for $m$, using the exact same number across signatures. Segher explains the fatal mathematical flaw: “It's supposed to be a different M every time they've got three unknowns only two equations no way to solve it but now we've got only two unknowns so it's trivial to just solve for M by just you take the two the two formulas subtract whatever just a bit of formula manipulation and then you solve M and then you've got M and just fill it in and then you solve for K.”
With $m$ known, calculating the master private key $K$ reduces to basic school algebra requiring only two distinct signatures. The extracted private keys allow fail0verflow to craft digital signatures that are fully indistinguishable from official Sony signatures. This compromises lv2, revocation lists, hypervisor configuration profiles, and installation packages. While level 1 keys remain temporarily gated by isolated hardware quirks in lv1ldr, the public key cryptography across the architecture collapsed, leaving only basic user/kernel privilege separation and on-die boot ROMs.
[41:04](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=2464s) Q&A: Origins of PSJailbreak and Tooling Roadmaps
During the transition to the question-and-answer period, the team notes that their live demonstration encountered display output bugs—mirroring the video issues suffered during their BootMii demonstration two years prior—and announces plans to present the demo during a lightning talk the following day. Attendees are invited to line up at microphones or visit the team in the Hackcenter.
An attendee asks whether fail0verflow knows the origin of the PSJailbreak USB dongle. The team answers that they have no confirmed origin or proof, though they suspect it originated from somewhere in the Southern Hemisphere. When asked about existing software implementations, they note that the open-source AsbestOS implementation already exists for TI OMAP processors, alongside clones and reverse-engineered payloads reproducing the USB exploit. Regarding their own ECDSA-based toolchain, the team explains that they relied on crude hacks to assemble their initial proof-of-concept, but plan to release cleaned-up, open-source signing tools within the following month.
Another attendee asks via IRC why the team cannot sign a software payload for commercial Blu-ray games now that they can sign code. The speakers explain that game executables are handled by apploader, which uses distinct keys. Because fail0verflow lacks a dump of that specific loader's plaintext, they cannot extract its corresponding keys: “So yeah you need the plaintext of the loader to get the public key and once you have the public key then that so we can't sign games and also we don't really care.”
- Live demo delayed due to display issues, rescheduled for a conference lightning talk. - PSJailbreak provenance remains unverified, suspected to hail from the Southern Hemisphere. - AsbestOS provides an open implementation of the USB stack exploit on TI OMAP chips. - fail0verflow commits to publishing public tools within a month. - Blu-ray game signing remains unsupported due to un-dumped apploader binaries, a vector the team dismisses as outside their interests.
[44:03](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=2643s) Focus on Low-Level Access and Closing Remarks
Addressing a final question from the audience regarding game-related capabilities and piracy, the fail0verflow team firmly draws the boundary between their research and game modding. They emphasize that their motivation lies strictly in system control, homebrew, and regaining hardware ownership rather than dealing with commercial game software. One of the speakers explains their stance on running or modifying games, noting: “About games because we care about low-level access, it's probably possible eventually, but we're not planning on working on it.” For the team, once arbitrary code execution and the ability to restore Linux are established, their core mission is complete.
With no further questions from the floor, the session chair steps in to conclude the presentation. The room is given standard conference logistics—attendees exiting the hall are asked to leave toward the right, pick up any leftover trash around their seats, and hand over any lost valuables to lost and found. The landmark presentation wraps up to warm applause and music from the Chaos Communication Congress audience, celebrating one of the most significant security disclosures in console hacking history.
## Notable quotes > [05:53](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=353s) Linux is inevitable: either you support Linux on your hardware, or it will be hacked so it'll run sooner or later. — Michael Steil
> [03:35](https://www.youtube.com/watch?v=4loZGYqaZ7I&t=215s) Well, we don't care about piracy: it does not allow homebrew code, and we don't care about running code others wrote; we want to run our own code. — Sven
## Mentioned Sony, PlayStation 3, Nintendo Wii, Xbox 360, Xbox, Nintendo DSi, fail0verflow, bushing, marcan, Segher, Sven, Michael Steil, George Hotz (geohot), Andrew 'bunnie' Huang, Chaos Communication Congress (27C3), BootMii, Homebrew Channel, AsbestOS, PSJailbreak, Cell Broadband Engine, IBM, Toshiba, PowerPC, GameOS, OtherOS, Starlet, Linux
## Video description 27th Chaos Communication Congress 2010 held in Berlin, Germany. Authors: segher, marcan, sven and bushing — fail0verflow!
More information - http://reviews.dcemu.co.uk/synopsis-of-chaos-communication-congress-sony-s-ps3-security-is-epic-fail-350530.html
FACTS: - fail0verflow is the name of our 'group'. We are a bunch of curious hackers who have been working on a bunch of things over the last 3 years. - It did not take 4 years to crack the PS3, we only started looking at the ps3 after otheros was killed. - Note: we won't be working long-term on CFW or similar. We'll release tools and a PoC, someone else can take over. The fun part is done ;) - Our current PS3 goal: AsbestOS.pup - Our goal is to have linux running on all existing PS3 consoles, whatever their firmware versions.
------------------------- Over 70 million Wiis, over 40 million Xbox 360s and over 35 million Playstation 3s have been sold in the last few years. That makes over 145 million embedded devices out there and most of them are just used to play games. But what can you do with them if you don't like playing games? You hack them to make them run your own code of course! We're going to talk about the various hacks that you can use to gain control of your hardware and make it do what you want it to do.
2010 saw the first hacks for the Playstation 3, soon after Sony removed Other OS functionality. We will detail the operation of current PS3 exploits, show a few new ones and explain where and how Sony went wrong when designing its security system, and show how these holes can be used to gain control over the system and bring Linux back to the PS3.
We will also go over hacks for the other consoles, including the JTAG hack for the Xbox 360 which made running homebrew code more convenient, and the cat-and-mouse games that Nintendo played with us to combat Wii hacks. We might also check out the security of their 'new' handheld console - the DSi.
Gamers might find this talk interesting even though it is targeted at those who hack (or design) embedded system security. A basic knowledge of crypto is therefore assumed. We will also be present in the Hackcenter before and after the presentation for those of you who are interested in learning more about the subject.
Twitter: http://www.twitter.com/fail0verflow
[Source](https://www.youtube.com/watch?v=4loZGYqaZ7I)