Bryan Cantrill from Joyent on Manta: internet-facing object storage facility that features compute

AI Council

AI-assisted summary of Bryan Cantrill from Joyent on Manta: internet-facing object storage facility that features compute — original source. Check the original for full context.

Source published: 2014-01-29

Key ideas

Joyent unveiled Manta, an internet-facing object storage system that spins up SmartOS zones directly on storage nodes to execute arbitrary compute next to data, eliminating network transfer bottlenecks. (41:04)
Manta diverges from traditional big data frameworks like Hadoop by embracing the Unix philosophy, letting developers run standard command-line tools like grep, awk, and Python pipelines directly against stored objects. (54:25)
Unlike Amazon S3's eventual consistency model, Manta implements CP writes across separate data centers, returning an HTTP 200 only after data is durably stored in both locations. (74:01)
Compute on Manta is billed at per-second granularity at roughly 40 micro-cents per CPU second, avoiding persistent hourly instance charges. (67:54)

Detailed notes

## Background Published on YouTube by AI Council on January 29, 2014, this technical talk features Bryan Cantrill, then-CTO of Joyent, presenting on Manta. According to the page description, the session was hosted as part of the Data Council community and conference series, which connects data infrastructure engineers with talks from open-source and startup innovators.

## TL;DR Tracing computing history from early disk drives and the architectural flaws of volume managers to the invention of ZFS, Bryan Cantrill outlines why traditional distributed systems struggle when compute and storage are decoupled. He introduces Manta, Joyent's internet-facing object store that solves network transfer bottlenecks by executing transient SmartOS containers directly on the storage nodes where data lives. Cantrill champions the Unix philosophy of pipes and simple composable tools over complex frameworks like Hadoop, demonstrating how massive MapReduce jobs and in-situ debugging can be executed using familiar shell commands and utilities like Awk.

## Atmosphere & style Cantrill delivers a high-velocity, irreverent, and deeply technical performance marked by biting engineering humor and vivid historical storytelling. He treats computing history not as a sterile progression of specs, but as a messy saga of hardware firmware bugs, vendor marketing deceptions, and pragmatic epiphanies. The atmosphere is that of an insider technical masterclass where profound architectural principles and live terminal demonstrations are laced with unabashed enthusiasm for Unix fundamentals.

## Bulletin - [41:04](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2464s) Joyent unveiled Manta, an internet-facing object storage system that spins up SmartOS zones directly on storage nodes to execute arbitrary compute next to data, eliminating network transfer bottlenecks. - [54:25](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3265s) Manta diverges from traditional big data frameworks like Hadoop by embracing the Unix philosophy, letting developers run standard command-line tools like grep, awk, and Python pipelines directly against stored objects. - [01:14:01](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4441s) Unlike Amazon S3's eventual consistency model, Manta implements CP writes across separate data centers, returning an HTTP 200 only after data is durably stored in both locations. - [01:07:54](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4074s) Compute on Manta is billed at per-second granularity at roughly 40 micro-cents per CPU second, avoiding persistent hourly instance charges.

## Key takeaways - [05:00](https://www.youtube.com/watch?v=79fvDDPaIoY&t=300s) The historical decoupling of file systems and volume managers created an abstraction divide that led to silent data corruption and phantom writes. - [09:31](https://www.youtube.com/watch?v=79fvDDPaIoY&t=571s) ZFS solved structural storage integrity issues by unifying the file system and volume manager into a copy-on-write pooled architecture with end-to-end checksumming. - [14:06](https://www.youtube.com/watch?v=79fvDDPaIoY&t=846s) Mechanical disk seek times have only improved fourfold since the IBM 350 RAMAC in 1956, rendering offline fsck disk repair impractical on multi-terabyte drives. - [36:41](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2201s) Hardware virtualization introduces rigid abstraction barriers that hinder dynamic resource resizing, whereas OS-level virtualization enables zero-overhead bare-metal container resizing. - [27:35](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1655s) Amazon S3 successfully redefined cloud storage by replacing POSIX semantics and partial file mutations with an immutable, write-once object model. - [41:04](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2464s) Manta co-locates compute with storage by instantiating transient SmartOS zones directly on the physical storage nodes holding the target data. - [54:25](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3265s) The modern big data ecosystem often relies on heavyweight frameworks like Hadoop and MapReduce to solve problems that can be handled far more simply by Unix text-processing pipelines. - [56:10](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3370s) Manta exposes genuine hierarchical directory structures and supports streaming uploads and downloads over standard HTTP connections. - [01:01:22](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3682s) Josh Clulow's mlogin utility provisions interactive debug zones where multi-gigabyte crash dumps are mounted locally via loopback for in-situ analysis. - [01:05:18](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3918s) Job execution latency on Manta scales independently of input file size because the computation travels to stationary data rather than moving bits across the network. - [01:14:01](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4441s) Manta provides a strongly consistent write path that demands synchronous two-datacenter replication before issuing an HTTP 200 success response. - [01:17:14](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4634s) Users can execute arbitrary runtimes and libraries including R, Go, and Python by attaching custom asset tarballs directly to their Manta job specifications. - [01:28:16](https://www.youtube.com/watch?v=79fvDDPaIoY&t=5296s) ZFS rollbacks are integrated into the high-frequency operational data path to restore compute environments to a pristine base state immediately after jobs complete.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=79fvDDPaIoY&t=0s) Engineering Team Tribulations and Honest Origin Stories

Bryan Cantrill kicks off the session with an observation about engineering culture, pointing out that throughout his entire career in Silicon Valley, he has virtually never worked with a native Californian engineer. Because engineering hubs like Silicon Valley and New York draw people from across the country and the world, teams harbor intense regional loyalties. According to Cantrill, the fastest way to tear an engineering team apart and pit colleague against colleague is simply to order pizza: “Because when you bring in a pizza to an engineering team you are actually asserting the dominance of one region over another and someone's Hometown has just been insulted and it is it is the gravest possible insult and it is serbs against fucking croats in a nanosecond.” Watching Chicagoans clash with New Yorkers over crust styles makes serving pizza a minefield, so he finds it refreshing to have non-controversial food at the meetup.

Cantrill introduces his background, noting that he has worked in operating system development for roughly twenty years. Having recently turned forty, he jokes about his youthful appearance, recalling that at twenty-two he looked like an infant. He explains that his talks frequently take a historical view, and for this presentation, he wants to present an honest origin story of the technology his team built at Joyent rather than a retrofitted myth. In software engineering, when people stumble upon an important discovery or build a breakthrough technology by getting lucky, they routinely rewrite their history to claim they had envisioned the exact end state from day one. Cantrill promises instead to explain the actual, messy problem they set out to solve and how it led to their outcome. To set up the theme of merging two distinct paradigms—compute and storage—he references the classic 1980s Reese’s Peanut Butter Cup advertising campaign (“You got your peanut butter in my chocolate! / You got your chocolate in my peanut butter!”), noting with amusement that the cultural reference dates him when few hands go up in recognition.

[03:15](https://www.youtube.com/watch?v=79fvDDPaIoY&t=195s) From the IBM RAMAC to RAID: The Limits of Single Disks

Cantrill shifts into the prehistory of storage systems, going back to the foundational eras of computing. In the 1970s and earlier, storage scaling was achieved purely by engineering physically larger disks with higher platter densities. He highlights the IBM 350 RAMAC from 1956—the first commercial hard disk drive, which featured massive platters—as an essential artifact housed at the Computer History Museum in Mountain View. Through the late 1950s, 1960s, and 1970s, storage advances consisted of shrinking drive enclosures somewhat while continually packing more bits onto single platters.

By the 1980s, the single-spindle scaling model hit physical and economic limits, mirroring the shift toward microprocessors where scaling out smaller units became mandatory. This bottleneck prompted David Patterson and his colleagues at UC Berkeley to introduce RAID: Redundant Array of Inexpensive Disks. Cantrill notes with some bite that contemporary enterprise storage vendors frequently try to rewrite history by claiming the acronym stands for "independent" rather than "inexpensive" disks, using that revisionism to justify inflated hardware margins.

- The 1956 IBM RAMAC introduced the magnetic disk storage paradigm using large, stacked physical platters. - Single-disk density scaling hit diminishing returns by the early-to-mid 1980s. - Patterson's RAID paradigm proposed aggregating arrays of small, inexpensive disks to deliver high capacity and redundancy. - Enterprise storage vendors later pushed the term "Independent Disks" to replace "Inexpensive Disks" as they charged premium pricing for storage arrays.

[05:00](https://www.youtube.com/watch?v=79fvDDPaIoY&t=300s) The "Lie" of Volume Management

Despite file systems in the 1980s being relatively young—rarely older than a decade—software was already treated as too rigid to rewrite around the new multi-disk physical reality of RAID. Rather than redesigning file systems to natively understand arrays of independent disks, the industry resorted to a classic software engineering shortcut: introducing an abstraction layer that lies. Volume management was invented to deceive the file system into believing it was still interacting with a single, massive disk, while silently distributing I/O across dozens or hundreds of underlying drives.

```mermaid flowchart TD accTitle: The Volume Management Abstraction Lie accDescr: Diagram illustrating how a traditional file system is misled by a volume manager into seeing multiple physical disks as a single block device.

subgraph FileSystemLayer ["Software Layer"] FS["File System (e.g., UFS, VxFS)"] end

subgraph AbstractionLayer ["The Lie"] VM["Volume Manager (LVM / VxVM)"] VDev["Emulated Single Block Device"] end

subgraph PhysicalDisks ["Physical Storage"] D1["Physical Disk 1"] D2["Physical Disk 2"] D3["Physical Disk..."] DN["Physical Disk N"] end

Cantrill stresses that while software abstractions often start with pragmatic intentions, fundamentally lying to an operating system subsystem eventually exacts a heavy toll: “Why actually rewrite it when we can just lie to it? It doesn't know, it's just software. We'll tell a very convincing lie, and that lie was volume management.” Because the file system had no visibility into physical disks, it assumed drive geometry was immutable. Hard drives do not spontaneously spawn extra platters on a spindle, so file systems could not easily grow, let alone shrink when capacity was removed. Features like snapshots became exceedingly brittle, and administering two decoupled systems that were entirely blind to each other turned into an operational nightmare.

[07:30](https://www.youtube.com/watch?v=79fvDDPaIoY&t=450s) The Semantic Divide and Silent Data Corruption

Throughout the 1990s, the friction between logical volume managers and file systems grew untenable. Deployments pairing Veritas File System (VxFS) with Veritas Volume Manager (VxVM), or Unix File System (UFS) with logical volume managers on Solaris or IBM AIX, subjected system administrators to catastrophic failures and unrecoverable data loss.

The crux of the dysfunction was a deep semantic divide. The protocol between a file system and a disk or volume manager is restricted to an austere, low-level block interface: read block, write block. Because the volume manager operates strictly on blocks without understanding directories, files, or metadata, and the file system has no insight into physical media topologies, neither layer can validate the correctness of the other. While a volume manager might attempt basic block-level checksumming for bit rot, it is helpless against severe device controller anomalies such as phantom writes. A phantom write occurs when the host sends a write command, the drive controller falsely acknowledges that the data has reached stable storage, but the controller never commits the bits—leaving the file system with corrupted state and no mechanism to discover the deception until reading back garbage.

- Traditional enterprise storage stacks in the 1990s routinely combined file systems and volume managers (such as VxFS on VxVM, or UFS on LVM). - The low-level block interface (read block / write block) strips away all semantic context between the storage manager and the file system. - Volume managers cannot distinguish metadata from user data, making intelligent snapshots, dynamic resizing, and targeted recovery virtually impossible. - Phantom writes—where a controller falsely acknowledges a write command as committed to stable media—completely bypass volume-level protections because the underlying layer cannot verify data provenance.

[09:31](https://www.youtube.com/watch?v=79fvDDPaIoY&t=571s) The Volume Manager Flaw and the Invention of ZFS

Bryan Cantrill illustrates the silent horror of low-level storage bugs like phantom reads. When a system asks the disk for block 63 and the drive instead serves block 36, the read is internally self-consistent. The volume management layer examines the block, detects nothing out of order, and passes it upward to the file system. The file system then finds critical system files in the middle of a database:

> “The problem with that is it's totally self-consistent so the volume management layer is like looks good to me I don't know sure pass it up to the file system file system's like whoa no no no that's like `/etc/passwd` that's not the database... it is terrifying when you go into the database and you see `/etc/passwd` or vice versa... something is very wrong in the animal brain and that and the problem is those errors can be very very deep in the stack.”

Cantrill diagnoses this architectural fracture: volume management was conceived as an emergency crutch to stitch multiple physical disks together, yet the industry adopted it as a permanent abstraction. Sun Microsystems engineers decided to abolish this artificial boundary. Jeff Bonwick and Matt Ahrens wiped the slate clean to build ZFS, designing from the ground up for systems composed of multiple disks, copious DRAM, and what they anticipated in 2001 would be omnipresent non-volatile main memory by 2014.

Comparing modern adoption to public health, Cantrill remarks on how many engineers understand the technology but remain on flawed stacks:

> “Sadly we are now in this kind of awkward scenario where many people have heard that there is a cure for polio, but they have not actually vaccinated themselves or their children and they keep being stricken with polio and they know to be sad about it.”

- ZFS demolished the rigid partition between the file system and volume manager, treating the pool as a single shared storage cloud where drives plug into the bottom and file systems attach at the top. - Cantrill notes that a file system should purely be an administrative nexus consisting of path, hierarchy, and policy—never having a direct one-to-one coupling to an underlying physical disk. - Btrfs is referred to as a later “knockoff” of ZFS; an Ars Technica retrospective had accurately categorized both as next-generation designs rather than brand-new systems, since Cantrill’s personal data had already resided exclusively on ZFS for over a decade.

```mermaid flowchart TD accTitle: ZFS Pooled Storage Model accDescr: Diagram illustrating ZFS decoupling administrative file systems from underlying physical storage pools. subgraph Filesystems["Administrative Layer (Path, Hierarchy, Policy)"] FS1["File System A"] FS2["File System B"] FS3["File System C"] end subgraph Core["ZFS Unified Storage Pool (zpool)"] SPA["Pooled Storage Engine (Copy-on-Write, Checksums)"] end subgraph Disks["Physical Media"] D1["Disk 1"] D2["Disk 2"] D3["Disk 3"] D4["Disk 4"] end FS1 --> SPA FS2 --> SPA FS3 --> SPA SPA --> D1 SPA --> D2 SPA --> D3 SPA --> D4 ```

---

[14:06](https://www.youtube.com/watch?v=79fvDDPaIoY&t=846s) Copy-on-Write, the Death of fsck, and Self-Healing Data

Because ZFS utilizes an end-to-end copy-on-write transactional architecture, it is always consistent on disk. There is no concept of an offline consistency check:

> “One of the great things about it is because it's a totally copy on write design, you don't have an on disk consistency check. There is no `fsck` for ZFS; such a thing doesn't make sense. ZFS is always consistent on disk.”

The death of `fsck` was a hard necessity driven by physics. In 1956, the IBM RAMAC spun at 1,200 RPM; fifty years later, mainstream platters spin at 7,200 RPM (with enterprise models at 10,000 or 15,000 RPM). While drive capacities exploded into terabytes, raw seek times improved by only ~4×. Scanning a multi-terabyte volume across slow mechanical spindles is computationally irrational, as demonstrated when Hacker News spent 36 continuous hours offline attempting to `fsck` a drive. Furthermore, ZFS defends against pathologically lying hardware—such as Western Digital Green drives that misreport flush completions—by offering the ability to roll pools backward transaction-by-transaction to the last known honest state.

Cantrill details how ZFS solves silent data corruption: - **Indirect Checksumming:** ZFS checksums every block, but the checksum is stored in the parent block pointer above it rather than within the data block itself. If a disk firmware bug returns the wrong block, the parent checksum will not match. - **Self-Healing Mirrors:** When bad data is detected during a read from mirrored or RAID-Z storage, ZFS fetches the valid copy from the alternate disk, repairs the corrupted sector on the faulty drive, and returns clean data to the caller. - **Hardware Bugs Unmasked:** Early deployments uncovered hardware flaws that vendors denied. In one controller model with six million units deployed worldwide, ZFS reported relentless checksum failures because a silicon bug in the controller’s DMA engine refused to transfer the final 64 bytes of a block under reproducible conditions. - **Integrity from Day One:** Cantrill rejects the software industry's tendency to excuse early data loss as acceptable growing pains: “ZFS never went through an era where it was unsafe or didn't have integrity or like people lost data.”

[18:56](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1136s) The Myth of “It Gets Better with Age” and the Birth of ZFS Storage

Bryan Cantrill criticizes the attitude that deeply flawed data-loss software will simply mature out of its architectural problems over time, comparing it to making excuses for severe delinquency: “No, no, no, this is like this is like a 12-year-old who's stealing cars... this is not like, 'Well, he's only 12'—it's like, no, he stole a car at gunpoint having just robbed a liquor store, like you've got an issue here.” You cannot simply say, “Call me when he's 18,” because you will not want to see what that system looks like at 18. Architectural data integrity cannot be postponed. In contrast, ZFS was intentionally designed from day one around provable reliability.

```mermaid flowchart TD accTitle: Evolution of ZFS Enterprise Maturation accDescr: Diagram tracing ZFS from its Sun Microsystems roots to enterprise hardening and modern adoption at Joyent.

subgraph SunEra["Sun Microsystems (2005–2008)"] A["ZFS Created (2005)"] --> B["Sun Storage 7000 / Fishworks (2006)"] B --> C["Major Customer: Oracle"] end

subgraph Hardening["Battle-Tested in the Field"] C --> D["Byzantine Firmware Bugs & Drive Resets"] D --> E["ZFS Hardened Across Hundreds of Thousands of Spindles"] end

subgraph OpenSource["Post-Sun / Joyent Era"] E --> F["illumos Community Fork (2010)"] F --> G["Joyent Cloud Infrastructure & Manta"] end ```

Cantrill outlines his own history with ZFS at Sun Microsystems. Around 2006, he and a group of coworkers decided to build an enterprise storage appliance around ZFS (the Sun Storage 7000 series, developed under the project name Fishworks). They “regrettably succeeded,” finding that one of their largest early customers was a massive database company (Oracle) that Sun hadn’t sold hardware to in years. The product showed the industry that Sun was building compelling technology again, though Cantrill jokes that he still suffers from “survivor's guilt” over how that customer relationship eventually unfolded.

[20:29](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1229s) The Gritty Reality of Enterprise Storage and Drive Firmware

Cantrill reflects on entering the enterprise storage business, remarking: “Building a storage product is like going backpacking in Afghanistan, and if you don't think you're going to a war zone, you're delusional.” Before entering the market, he assumed storage problems were well understood—phantom reads, phantom writes, bit rot, and media failure. But enterprise storage exposed him to the sheer malevolent creativity of drive firmware bugs and Byzantine drive failure modes across hundreds of thousands of active spindles.

To illustrate, Cantrill recalls a bug in the Seagate S0D hard-drive firmware where head polarity was applied improperly. At certain logical block addresses (LBAs), the firmware would mistakenly accelerate the head toward the cylinder instead of decelerating it. An internal safety mechanism would trigger to prevent physical destruction and smoke, forcing the drive to abruptly power-cycle itself. Each power cycle caused a 650-millisecond blackout before the drive began serving I/O requests again: - The system suffered bizarre 650 ms I/O latency spikes that were difficult to track down. - Because these race conditions allowed operations to eventually succeed, ZFS wouldn't immediately fault the disk, but the entire storage array would grind to an agonizing crawl. - In early ZFS implementations, massive snapshot deletions triggered heavy random reads within a transaction group context, causing operations to hang for days. Cantrill notes that while the appliance never lost customer data, “it went on some very, very, very long vacations.” - In one instance, a customer watched the slow progress of a syncing transaction group and calculated it would finish by Thursday at 8:00 PM; since they did not care about the test data, they chose to throw the setup overboard and start fresh rather than wait.

[24:58](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1498s) The Oracle Acquisition and the Infamous USENIX Disclaimer

Cantrill recounts his disillusionment following Oracle’s acquisition of Sun Microsystems in 2010. Looking back, he expresses disbelief at his own initial optimism: “If we invent a time machine, the first thing I want to do is go back to myself and just deck myself circa 2010, 'cause I was like, 'Oh, it'll be great, like Oracle understands business and Sun understands technology and it'll be like a business understands technology, man'—it's like, are you stupid?” He quickly realized Oracle was simply an aggressive monopolist.

Following his departure, Cantrill delivered a candid address on the history and fork of illumos at a USENIX conference. Because Oracle was a sponsor, the conference organizers panicked about offending the company: - Cantrill remarks that the organizers' panic was pointless because Oracle is effectively sociopathic and completely indifferent to outside criticism. - While all other session recordings were uploaded promptly, Cantrill's talk was held back for weeks. - When the video finally debuted, USENIX attached an unprecedented disclaimer reading that the speaker’s views did not represent the USENIX Association. - Rather than running the notice as a brief title card, the video editors compressed the disclaimer into a persistent black strip positioned directly above Cantrill’s head on-screen for the duration of the entire recording. Cantrill jokes that since they failed to bleep him, his next career goal is to provoke explicit censorship.

[27:35](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1655s) Joyent, ZFS Foundations, and the Philosophy of S3

After leaving Oracle, Cantrill joined Joyent alongside several Sun colleagues. Joyent had an established relationship with the technology, having been the first production user of ZFS outside of Sun Microsystems back in 2005. Joyent operated as an infrastructure-as-a-service cloud provider while also open-sourcing its software stack for private cloud deployments, with ZFS sitting at the architectural foundation of its storage.

Despite a robust compute and I/O infrastructure, Joyent was missing an HTTP-accessible object store comparable to Amazon Web Services' S3 (Simple Storage Service). Cantrill praises S3 as the first genuine cloud service and highlights a critical design decision made by the Amazon team: - Most systems engineers of the era would have mapped a traditional POSIX file-like abstraction over HTTP. - S3 deliberately rejected that model and built an immutable object store rather than a hierarchical file store. - As a foundational rule, S3 objects do not permit partial writes or in-place modifications. Cantrill stresses that abandoning in-place updates was the single most consequential insight for scaling distributed storage over the web.

[29:16](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1756s) Object Storage Constraints and Joyent's Missing S3

Bryan Cantrill explains why Amazon S3 succeeded despite ignoring traditional POSIX file-system semantics: S3 explicitly prohibited partial writes. While conventional wisdom would assume that disallowing in-place mutations of files would be completely unacceptable to users, it turned out to be an ideal trade-off. Immense volumes of real-world data—such as application logs, static web assets, and backups—are naturally write-once, read-many (WORM). Users have no desire or need to mutate bytes in the middle of an existing log file or backup archive. Relaxing that single constraint drastically simplified the architecture of large-scale distributed systems and made true object storage viable.

```mermaid flowchart TD accTitle: Trade-off between POSIX and Object Storage accDescr: Diagram illustrating how eliminating partial writes simplifies distributed storage design. subgraph POSIX["POSIX File System Constraints"] P1["In-Place Partial Writes Allowed"] P2["Complex Locking & Cache Coherency"] P1 --> P2 end subgraph S3["Object Storage Constraints"] O1["Write-Once, Read-Many (WORM)"] O2["No Middle-of-File Mutations"] O3["Simplified Distributed Architecture"] O1 --> O2 --> O3 end ```

Joyent, however, did not have an equivalent to S3. Cantrill describes this lack of an object store as a "glaring gap" and a "gaping hole" in Joyent's cloud product offering. Having recently spent intensive engineering effort wrestling with ZFS—what he jokingly refers to as his "backpacking trip to Afghanistan with ZFS"—he found himself pondering whether they could build something compelling out of ZFS or the OpenSolaris/illumos codebase to implement an object store. At the time, he concedes, he couldn't come up with a workable design, and every initial idea he had was terrible. Putting that problem on hold, he turns to the underlying architecture of cloud computing: multi-tenant virtualization.

- Object storage explicitly trades away arbitrary partial writes, catering instead to write-once, read-many (WORM) workloads like logs and media assets. - Removing in-place file mutations eliminated distributed locking bottlenecks, allowing massive horizontal scaling. - Joyent recognized its lack of an S3-like object storage offering as its single largest platform deficiency. - Cantrill’s early attempts to bend ZFS directly into an object storage substrate hit dead ends. - As Cantrill puts it: “And so that was actually extremely important, because that changed the constraints of the system, allowed them to build the distributed system.”

---

[30:35](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1835s) The Origin of Hardware Virtualization: The IBM System/360 and the 1401

Virtualization is the foundational pillar of the cloud, enabling multiple tenants to coexist securely on a single physical host while each tenant maintains the complete illusion of owning the entire system. Cantrill argues that the fundamental architectural question is *where* in the software stack one chooses to virtualize. The oldest and most enduring approach, dating back to the 1960s, is hardware-level virtualization. Its origin lies in the release of the IBM System/360 in the mid-1960s—a watershed machine that unified IBM's formerly distinct commercial and scientific computing lines into a single product family.

```mermaid flowchart LR accTitle: Emergence of Hardware Virtualization via IBM 360 accDescr: Sequence showing how customer retention forced IBM to emulate the 1401 on the System 360. subgraph MarketPressure["Market Disruption"] A["IBM 360 drops 1401 support"] --> B["Honeywell H200 clones 1401 hardware"] B --> C["Customers defect from IBM to Honeywell"] end subgraph IBMResponse["Software Emulation"] C --> D["IBM invents hardware virtualization"] D --> E["System 360 lies to legacy 1401 programs"] end ```

When IBM unveiled the System/360, it deliberately abandoned backwards compatibility with older platforms, notably the IBM 1401—a wildly popular stored-program accounting machine that had dominated early business computing. IBM's original posture was dismissive: hardware was considered the hard problem, and customers wanting to run software on the 360 were told to simply rewrite their applications from scratch. Seeing an opportunity, competitor Honeywell introduced the Honeywell 200, which was bit-for-bit compatible with the 1401. As customers defected to Honeywell in droves instead of upgrading to the System/360, IBM was forced to react. Their solution established a lasting paradigm in computer engineering: telling a deliberate lie to software. IBM constructed a pure software emulator of the 1401 that ran on the System/360, convincing legacy binaries that they were executing on bare-metal 1401 hardware.

- The IBM System/360 was introduced in the mid-1960s to consolidate commercial and scientific computing lines. - IBM initially refused to support programs written for the older IBM 1401 accounting machine, expecting customers to rewrite their codebases. - Honeywell developed the H200 to be bit-for-bit compatible with the IBM 1401, capturing market share almost overnight. - IBM answered by building a software emulation layer on the System/360, inventing the first widely deployed hardware virtualization. - As Cantrill notes about the persistent philosophy of system design: “And what IBM did, going back to that remember that earlier theme, the persistent theme of computing: it's lie time, and it's time to lie to that software and tell it that it's on an IBM 1401.”

---

[33:51](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2031s) Software Persistence and the 1401 Y2K Mystery

Cantrill shares an anecdote illustrating the staggering longevity of software. Shortly after the year 2000 rollover, while working at Sun Microsystems, he was in New York discussing the aftermath of Y2K with the Chief Information Officer of a major financial institution. At Sun, around 30% of their observed Y2K defects had actually turned out to be leap-year calculation bugs: developers remembered that years divisible by 4 are leap years and that years divisible by 100 are not, but they forgot the third rule stating that years divisible by 400 *are* leap years after all.

When Cantrill asked the bank CIO about their own post-mortem data, the CIO revealed that a plurality—roughly 45%—of their bank's Y2K bugs originated from a single hardware platform. Building dramatic suspense, the CIO finally identified the culprit: the IBM 1401, a machine last manufactured in 1963. Cantrill was stunned. Decades after the physical hardware was retired, the bank's mission-critical ancillary software was still executing inside virtualized emulator layers stacked atop successive generations of enterprise servers. Because the two-digit year representation was literally hardcoded into the original 1401 machine architecture, it was impossible to patch around the millennium turnover; the software finally had to be rewritten.

- Approximately 30% of Sun's Y2K incidents stemmed from missing the 400-year leap year rule (year 2000 was a leap year). - Roughly 45% of Y2K defects at a major New York bank originated from software written for the IBM 1401, which ceased production in the 1960s. - The 1401 code had outlived its original hardware and multiple successor platforms, preserved entirely through layered virtualization. - The bank was finally forced to rewrite the programs because the two-digit year was enshrined directly into the 1401's architectural specification. - Cantrill summarizes the revelation: “And that was an amazing moment to me, because I think it tells us about the persistence of software—that here you have software that is written in the late '50s, early '60s... and yet it has survived the death of not only that hardware, but the hardware that replaced it, and probably the hardware that replaced it.”

---

[36:39](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2199s) The Structural Limits of Hardware Virtualization

While hardware virtualization allows legacy software stacks to run without modification, Cantrill argues that the abstraction eventually breaks down under production cloud constraints. Hardware interfaces present an extremely rigid, narrow semantic contract. Operating systems expect absolute ownership over the physical resources exposed to them, which makes dynamic resource management across virtual machines clumsy and inefficient.

Memory allocation is the clearest failure point. When a hypervisor assigns 1 GB of DRAM to a guest virtual machine, it must genuinely back that allocation; an operating system kernel assumes fixed physical DIMMs and cannot tolerate memory vanishing spontaneously or appearing without hardware-level hotplug events. Overcommitting DRAM in a hardware-virtualized environment is risky and unsustainable for cloud providers. Network and storage I/O present similar bottlenecks: routing packets through an emulated device model imposes severe overhead. To recover bare-metal networking throughput, hypervisors often pass physical hardware straight into the guest (such as via SR-IOV or direct device assignment). However, Cantrill points out the irony: bypassing the emulation layer to get acceptable performance admits defeat, forfeiting the core benefits of virtualization by turning the hypervisor's "lie" back into raw hardware.

```mermaid flowchart TD accTitle: The Breakdown of Hardware Virtualization Abstractions accDescr: Diagram contrasting rigid hardware emulation against the drawbacks of direct hardware passthrough. subgraph Emulated["Hardware Virtualization (The Lie)"] E1["Hypervisor presents virtual NIC / DRAM"] E2["OS assumes fixed, physical ownership"] E3["Severe I/O overhead & cannot easily overcommit"] E1 --> E2 --> E3 end subgraph Passthrough["Direct Passthrough (Giving Up the Lie)"] P1["Pass physical NIC directly to guest"] P2["Restores raw bare-metal performance"] P3["Loses isolation, migration, and software agility"] P1 --> P2 --> P3 end ```

- Hardware-level virtualization forces hypervisors to adhere strictly to inflexible, low-level architectural interfaces. - DRAM assigned to a virtual machine cannot easily be reallocated or dynamically oversubscribed without risking severe guest instability. - Network I/O performance degrades severely under pure software hardware emulation. - Bypassing virtual devices via direct hardware passthrough restores line-rate I/O performance but strips away the management advantages of virtualization itself. - Cantrill highlights the fundamental breakdown of the approach: “The lie works great at first, but then you begin to pull things and the lie begins to snap.”

[38:41](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2321s) Virtualization Models and the Power of OS-Level Containers

Bryan Cantrill contrasts virtualization layers, noting that while Joyent Public Cloud supports hardware-based virtualization via KVM, platform-level virtualization is a dead end. Platform-as-a-Service (PaaS) offerings suffer from what he terms the "Google App Engine problem": abstracting too high in the stack prevents users from developing large applications. As soon as a system grows, it is forced to migrate elsewhere, leading Cantrill to call PaaS a terrible business model where companies fail before solving the architectural mismatch.

Joyent's core bet is OS-level virtualization. A single operating system kernel runs directly on bare-metal hardware and carves out isolated containers. To the software running inside, each container looks, feels, and smells like an independent hardware instance—even supporting reboots—while using the system call table as its boundary abstraction. Because containers run directly on the host hardware, they can inspect all physical CPUs and memory. Resource limits, such as allocating a gigabyte of DRAM, simply mean instructing the kernel to cut an instance off when it reaches that ceiling. Resizing an instance on Joyent Public Cloud requires no migration or reboot; operators can instantly adjust DRAM limits upward or downward as long as the underlying machine is not oversold.

Cantrill points to SmartOS Zones as a rock-solid implementation of this concept. Far from feeling threatened by technologies like Docker or LXC, he finds their rising popularity deeply validating because they demonstrate that the software industry is waking up to the superiority of the container model. He quips that while LXC exists, it has well-known flaws, remarking:

> “Deploy on LXC, and then if you want to have zero security problems, you can actually deploy on SmartOS.”

[41:04](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2464s) Mark Cavage's Epiphany: Co-Locating Compute with Storage

While Cantrill was intensely focused on ZFS as the foundation for an object store, Joyent brought on Mark Cavage, a senior engineer from Amazon Web Services. Cantrill remained fixated on low-level file-system mechanics, but Cavage supplied the architectural breakthrough: the true game-changer was not ZFS, but rather OS-level virtualization.

Instead of building a conventional HTTP-facing object store (similar to Amazon S3) where users issue `GET` requests, pull files across the network, and run utilities like `grep` locally, Joyent could leverage containers to bring compute to the data. If an object needs processing, the system spins up a SmartOS zone right on the physical node where that object resides, running commands directly against the local storage without moving bytes across the network. Reflecting on his own initial blindness to the concept, Cantrill remarks:

> “Sometimes, oftentimes, these big insights are—they actually, they seem simple only in hindsight.”

```mermaid flowchart LR accTitle: Traditional Object Storage vs. In-Situ Compute accDescr: Diagram contrasting traditional S3 network retrieval with Manta in-situ container execution.

subgraph TRAD["Traditional S3 Approach"] S3["Object Storage Node"] -->|"GET (Network Transfer)"| CLI["Client / Worker VM"] CLI --> RUN1["Run grep / transform"] end

subgraph MANTA["Manta / OS Virtualization Approach"] STORE["Local ZFS Storage"] -->|"Zero-Copy / Local Read"| ZONE["SmartOS Container (Spun Up on Node)"] ZONE --> RUN2["Compute directly on data"] end ```

[42:40](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2560s) The Instant Message Log of October 24, 2011

Cantrill recalls vivid memories of key technical milestones in his career, such as sitting in his poorly lit office circa 1999 when Jeff Bonwick first explained ZFS, or pitching DTrace to Bonwick during his Sun Microsystems interview. Yet neither he nor Cavage could remember where they physically stood when the idea for Manta was conceived. Cavage suspected they were having drinks at a bar, but searching through archived chat logs revealed the truth: Cavage had proposed the idea over instant messaging on October 24, 2011.

The chat log preserved the moment Cantrill's "brain detonated." Cavage floated what he labeled a crazy idea: treating MapReduce as a first-class citizen by routing JavaScript code directly to a Node.js process running inside a separate zone co-located with the stored data, asking whether it could be billable. Cantrill reacted with profanity-laced enthusiasm, an outburst he describes as the highest praise he can offer for an extraordinary technical idea. Joyent spent all of 2012 turning that concept into reality, realizing that wishing does not make it so; underneath the simplicity lay a complex distributed system that combined local ZFS storage, SmartOS zones, and clustering primitives.

- **October 24, 2011:** Mark Cavage pitches co-locating MapReduce jobs with raw storage inside zones over IM. - **2012:** Joyent engineers spend the full calendar year architecting and building the production distributed system that became Manta.

[44:50](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2690s) Unix Lineage and Doug McIlroy's Pipe

As Manta took shape, the team realized the platform was resurrecting the timeless Unix toolset for distributed, ad-hoc data processing. Cantrill considers himself an "acolyte in the Church of Unix" and frequently reminds modern engineers that core systems concepts originated long before Linux. Unix emerged in the 1970s as a pun on Multics, the sprawling Bell Labs operating system that introduced foundational elements like the forward slash (`/`) in pathnames.

When Bell Labs lost access to its GE 635 mainframe running Multics, Ken Thompson and Dennis Ritchie sought to escape the "big system mentality" and build something lightweight and direct. The very first Unix utilities were throwaway test programs written simply to exercise the new file system. As Cantrill points out, developers executing basic Unix commands today are using test-driven development conceived before most of them were born.

The foundational design of Unix belongs equally to Doug McIlroy, whom Cantrill places alongside Thompson and Ritchie in the Unix pantheon:

> “If you think of Ken Thompson and Dennis Ritchie as being the inventors of Unix, you really need to put Doug McIlroy in that pantheon.”

McIlroy championed the concept of piping—connecting the output of one process directly into the input of another—as early as 1969, two years before Unix debuted. He originally envisioned these streams as "hoses." Thompson resisted the idea for three years despite working in an adjoining office at Bell Labs. The concept finally gained traction once McIlroy devised an ergonomic syntax for the shell, initially using the caret (`^`) character before evolving into the modern vertical pipe (`|`).

[48:11](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2891s) The Origin of the Pipe and the Unix Philosophy

Bryan Cantrill recounts the legendary genesis of the Unix pipe. Doug McIlroy had spent roughly three years lobbying Ken Thompson to introduce pipes into the system, only to be repeatedly told that it was technically impossible. But when Thompson finally agreed to try, he implemented it in a single night. Cantrill jokes about this classic dynamic in engineering: “Technically impossible means I really, really, really don't feel like it.” The next morning brought what McIlroy called an “orgy of one-liners,” establishing a tradition where immense syntactic and processing power is packed into a single, understandable command.

```mermaid flowchart LR accTitle: The Unix Philosophy Data Flow accDescr: Simple single-purpose tools connected together over a universal text stream interface. subgraph UnixTools["Unix Philosophy: Simple Tools"] direction LR P1["Tool 1\n(Do one thing well)"] -->|"Text stream\n(Universal interface)"| P2["Tool 2\n(Work together)"] P2 -->|"Text stream\n(Universal interface)"| P3["Tool 3\n(Work together)"] end ```

Cantrill describes this milestone in mythic terms: “This is like the Book of Genesis... talk about like 'let there be light' and the next morning we had this orgy of one-liners.” That transformative morning gave rise to the foundational tenets of the Unix philosophy as formulated verbatim by McIlroy: 1. Write programs that do one thing and do it well. 2. Write programs to work together. 3. Write programs to handle text streams, because that is a universal interface.

Cantrill considers this the single most critical philosophy in systems development: “The way we are able to develop complicated systems is by building out of simple ones, taking simple systems and actually connecting them in interesting and new ways.”

[50:08](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3008s) The Holy War: Libraries vs. Frameworks

As Cantrill has grown older, he views system design as an ongoing “holy war between the forces of complexity and the forces of simplicity.” He frames this divide as a conflict between a library-centric view and a framework-centric view:

- **The Library-Centric View:** Provides toolboxes filled with discrete utilities that developers can pick up and combine however they choose, preserving transparency and user agency. - **The Framework-Centric View:** Drops the developer into an all-encompassing, opaque world. Cantrill compares this to falling through a black hole: “The framework point of view says, okay, you pop into existence in this crazy-ass context where, you know, here are the things you can kind of do... it's like, no, no, you fell through a black hole, you just kind of wake up here.”

Cantrill acknowledges that frameworks have their place, but he warns that being suspended in an artificial universe makes systems remarkably difficult to reason about. His verdict is unyielding: “When you can build things in terms of the Unix philosophy, you always should.”

[51:14](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3074s) The Epic Rap Battle: Knuth vs. McIlroy

Cantrill shifts to what he dubs “the epic rap battle of computer science history.” While sidelined for four weeks with a knee injury, he watched vast amounts of online content, including the *Epic Rap Battles of History* series, which inspired him to view a classic computing showdown in the same light. The contest originated in John Bentley’s *Programming Pearls* column in 1986. Bentley posed a text-processing problem: read a file of text, determine the *n* most frequently used words, and print out a sorted list of those words alongside their frequencies. Today, Cantrill points out, this is a basic intern interview question or something covered by projects like GitHub's CLI mystery.

Donald Knuth responded to Bentley's challenge with an elaborate solution written in his literate programming system, WEB. Knuth produced seven dense pages of Pascal-like code and sophisticated algorithms. But like Indiana Jones pulling a revolver on a sword-wielding opponent, Doug McIlroy delivered a counter-solution using a short pipeline of standard Unix commands:

```sh tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q ```

Cantrill highlights several fascinating facets of McIlroy’s response: - **Performance parity:** In 1986, McIlroy's pipeline performed almost identically to Knuth's hand-crafted code, and modern versions remain extremely competitive because simple, optimized tools compose efficiently. - **The missing command:** McIlroy terminated his pipeline with `sed ${1}q` rather than `head -n`. Investigating this, Cantrill was stunned to discover that AT&T Unix in 1986 had `tail`, but `head` had not yet been integrated. - **Literate programming critique:** While expressing deep respect for Knuth, Cantrill relegates literate programming to the “stinker pile,” arguing that seven pages of dense code for a basic text pipeline demonstrates the trap of over-engineering.

[54:25](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3265s) Big Data, MapReduce, and the Betrayal of Unix

Cantrill argues that the modern Big Data ecosystem has repeated the exact mistake Knuth made. He quotes verbatim the canonical URL access frequency example from Jeffrey Dean and Sanjay Ghemawat's Google MapReduce paper:

> “The count URL access frequency function processes logs of web page requests and outputs a tuple `(URL, 1)`. The reduce function adds together all values of the same URL and emits a `(URL, total count)` pair.”

Cantrill notes that this problem is identical to Bentley’s 1986 challenge, yet the modern big data solutions abandon simplicity: “The solutions give the Unix philosophy a night that never forgets... the solutions are really are a desecration of the Unix philosophy.” The paper’s appendix features roughly 150 lines of dense, compressed C++ just to perform URL counts, alongside a 70-line implementation of word count. Cantrill scoffs at requiring an entire compilation and runtime stack for an operation so fundamental that Unix solved it with a two-character command:

> “Word count: an idea that's so old that the command to implement it in Unix is two characters: `wc`. If you want to do a word count of a document, your first instinct is like, 'Okay, I need a C++ compiler'—it's like, what? Wait, are you one of the framework people? How did you get in here?”

Cantrill laments the bizarre state of enterprise computing: “Why is it that when I do a word count on my laptop it's `wc`, but when I do a word count against my logs in the cloud, all of a sudden it's C++, and it's Java, and it's Hadoop? It's just wrong.”

[56:10](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3370s) Manta's Model: Hierarchical Storage Meets In-Situ Compute

Joyent designed Manta to bring Unix tools directly to object storage rather than forcing users into heavyweight frameworks. Cantrill switches to a live terminal demo to show how Manta operates. Unlike Amazon S3, which operates as a flat key-value store that permits slashes in key names without providing true folder semantics, Manta incorporates a real hierarchical directory structure borrowed from Unix and Multics. Cantrill mocks S3's confusing behavior where creating keys like `foo/bar/baz` leads to unexpected behavior when an engineer queries `foo/bar` expecting a directory listing: “It's like, 'Oh no, no, no, that's not the way I work.' Really? It's like, 'Well, why did you take the forward slash?' 'I don't know, you wanted a forward slash.'”

Cantrill runs an `mls` listing on his public directory (`/bcantrill/public`), displaying typical cluttered storage while demonstrating its speed. An audience member asks whether the commands are executing live over the network against remote servers while he types. Cantrill confirms they are: Joyent's primary Manta deployment is located in the US-East region in Ashburn, Virginia, meaning the audience on the East Coast experiences lightning-fast round-trip latency. He observes that while it performs well from the West Coast, proximity to the data center makes it feel like “a rocket,” validating that “that whole speed of light thing—there's a lot of truth to it.”

[58:18](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3498s) Browsing Manta Objects and the Unix V7 Time Machine

While poking through his home directory on Manta, Bryan Cantrill notices a curious directory name and jokes that he knows better than to open it in front of a live audience. He demonstrates that directory operations are blisteringly fast, allowing users to run standard Unix directory commands such as `ls` and `mfind` directly against remote object hierarchies. To provide a realistic directory tree with varied files, Cantrill previously uploaded the entire file tree of Research Unix Seventh Edition (V7) into `/bcantrill/public/v7`. Running `mfind` across that public directory immediately returns a comprehensive list of objects structured just like a classic filesystem.

```mermaid flowchart LR accTitle: Interactive Object Fetch and Pipe accDescr: Streaming an object from Manta to the local macOS terminal via mget and piping to nroff. subgraph Manta ["Manta Storage"] Obj["V7 Man Page Object (/bcantrill/public/v7/...)"] end subgraph Client ["Local Mac Environment"] MGet["mget"] Nroff["nroff -man"] Stdout["Terminal Display"] end Obj -->|"HTTP stream"| MGet MGet -->|"pipe stdout"| Nroff Nroff --> Stdout ```

Browsing through `usr/man`, he discovers an obscure man page entry for `pkon`, an obsolete packet protocol driver from ancient Unix. Curious to see if local tooling can still parse historical markup, he pipes the object directly from `mget` into his Mac's local `nroff` pipeline. The terminal instantly renders the formatted Seventh Edition documentation cleanly. As Cantrill marvels at the round trip:

> “I literally extracted a man page from like the mid 70s and rendered it on my local nroff on my local Mac.”

- Directory listing tools in Manta present object storage as hierarchical Unix filesystems with standard slash-delimited paths. - Objects can be fetched directly with `mget` and piped straight into standard local Unix utilities without saving intermediate files to disk.

---

[01:00:01](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3601s) Object Streaming vs. S3 Multi-Part Uploads

Cantrill contrasts Manta's object ingestion model with Amazon S3's rigid operational constraints. In S3, working with large files forces developers into the cumbersome machinery of multi-part uploads, slicing files into rigid chunk sizes and orchestrating multi-step commit handshakes. Under Manta, objects have no arbitrary size ceilings: an object can be a few bytes or an entire terabyte. Clients simply stream raw bytes into an open HTTP connection. The only operational requirement for files over 5 gigabytes is an upfront `Content-Length` notification, which allows Manta's internal placement system to schedule the incoming stream directly onto a physical storage node with sufficient continuous drive capacity.

> “If you don't have to do big goes S3 you don't care about this if you do your life is a living hell because of the multi-part upload business and the way the way you have to chunk these things in for us you just stream so you just open and and just pound us with data.”

To illustrate distributed compute execution, Cantrill triggers a multi-object job with `mjob create`. Rather than using a complex stream editor substitution, he pipes the input list through `head -10` to process a slice of files, leaving the job running asynchronously in the background while he walks through the underlying runtime mechanics.

- Manta accepts streaming uploads directly without requiring clients to chunk files locally. - Transfers exceeding 5 GB require an upfront size declaration purely so Manta can select a storage node with adequate remaining free capacity. - Workflows kick off asynchronously via `mjob create`, taking streaming object paths as job inputs.

---

[01:01:22](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3682s) Interactive In-Situ Debugging and the Genesis of `mlogin`

While batch map-reduce jobs excel at churning through structured metrics, Cantrill explains that Joyent originally built Manta's compute capabilities to tackle a thorny engineering headache: analyzing massive kernel crash dumps and user-space core files. Transferring multi-gigabyte or terabyte system dumps over wide-area networks to developer laptops is painfully slow. Keeping the dump stationary and computing in-situ solves the transport bottleneck, but post-mortem debugging cannot be easily automated via batch commands like `grep`. Forensic debugging is inherently exploratory and interactive.

Frustrated by the inability to interact directly with crash dumps where they rested, Cantrill agitated internally for a solution until engineer Josh Clulow created `mlogin`. Running `mlogin` provisions an interactive, transient compute zone scheduled on the exact physical storage node holding the target object, immediately dropping the developer into a live interactive shell with the object mounted locally.

```mermaid flowchart TD accTitle: Manta mlogin Architecture accDescr: Josh Clulow's mlogin mechanism provisioning an interactive container directly adjacent to an object on storage. Client["Engineer Terminal (mlogin object_path)"] subgraph StorageNode ["Target Storage Node"] Disk[("ZFS Disk Pool Stored Object / Core Dump")] Zone["Transient SmartOS Zone $MANTA_INPUT_FILE"] end Client -->|"Spawns interactive session"| Zone Disk -->|"Loopback / Read-only mount"| Zone ```

> “I complained about this just loudly enough that Josh Clulow one of my colleagues at Joyent invented this thing called M login.”

Logging into an object live on stage, Cantrill verifies the session environment: the target object is exposed as `$MANTA_INPUT_FILE`. Running the `file` utility against the path confirms it as ancient `troff` preprocessor text. The engineer is constrained entirely within a sandboxed zone: they are completely local to the physical node's storage layer yet fully isolated from all neighboring tenant data.

- `mlogin` creates a real-time, interactive shell inside an ephemeral container colocated on the physical disk node holding the object. - The environment variable `$MANTA_INPUT_FILE` points directly to the mounted file system payload, eliminating data transfer over network boundaries.

---

[01:03:18](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3798s) The Illusion of Linux: SmartOS Zones and GNU Tooling

Cantrill recounts an amusing interaction with a developer who praised Manta by saying Joyent had finally come to its senses, abandoned SmartOS, and built an environment that was "soup to nuts Linux." The user reached this conclusion entirely because logging in with `mlogin` felt instantly identical to modern Linux distributions. Joyent intentionally configured the default execution path to prioritize modern GNU utilities: typing `grep` invokes GNU grep, `tar` resolves to GNU tar in `/opt/local/bin`, and `awk` runs GNU awk.

Beneath that familiar layer, however, the operating system remains illumos-based SmartOS. Legacy Unix utilities still reside under `/usr/bin` for backward compatibility. Cantrill explores `/usr/bin` in the container, laughing as he browses obscure three-letter utilities from the 1970s and 1980s:

``` bfs atq ```

He tests `atq` to view the queue of `at` batch jobs, reminiscing that `at` was famously implicated in historical leap-year bugs. While legacy binaries remain tucked away in `/usr/bin`, placing GNU tooling first in `$PATH` ensures that developer scripts, build systems, and operational habits function without cognitive friction.

> “You actually have all the ancient Unix software but it's actually out of the way in terms of a path perspective and the the software that you actually use is all the all the GNU software.”

- Manta compute containers default to GNU coreutils in `/opt/local/bin`, minimizing friction for users accustomed to Linux environments. - Legacy illumos and Solaris system binaries remain present in `/usr/bin` without interfering with standard user pipelines.

---

[01:05:18](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3918s) Bentley Challenge Results and Zero-Data-Movement Scaling

While Cantrill explores the shell environment, the background Manta job completes. It has executed the John Bentley word-frequency benchmark across 37 separate input objects, compiling and sorting the aggregate top words across the corpus. Cantrill highlights a key operational reality of distributed in-situ computation: the vast majority of the elapsed wall-clock time was consumed by container provisioning bureaucracy—spinning up secure SmartOS zones and establishing runtime sandboxes—rather than raw data computation.

Because compute executes directly on the physical storage nodes holding each object, the pipeline does not stream bulk inputs over a network backplane. Cantrill explains that the overall job latency would remain virtually identical whether each input file measured a few kilobytes, hundreds of megabytes, or multiple gigabytes. The underlying data never moves; only the transient compute workload is dispatched to the storage layer.

> “This would not run in a materially different amount of time if these things were hundreds of megabytes or gigabyte a piece because the data is not moving.”

- Distributed execution latency in Manta is dominated by container lifecycle management and setup bureaucracy rather than disk I/O. - Because data is processed in place, processing time scales minimally between small sample files and multi-gigabyte production datasets.

---

[01:05:56](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3956s) Project Thoth: Cloud-Scale Core Dump Forensics and Pricing

Building on Manta's storage and compute capabilities, Joyent developed **Project Thoth**, an internal diagnostics system designed to capture and archive every core dump and operating system crash dump generated across their public cloud infrastructure in perpetuity. Describing himself as a "software disaster fetishist," Cantrill initially planned to capture every single process crash across the entire fleet.

However, opening the floodgates to tenant workloads produced an unexpected operational crisis: PHP applications across the public cloud generated an unrelenting firehose of process crashes that effectively mounted a distributed denial-of-service attack against Thoth's ingestion pipeline.

> “I hadn't really spent much time in the general prison population before I did turn that on and all of a sudden like PHP was basically dossing my system because that thing core dumps 24/7 365 that is like a disc stress test for core dumps.”

Joyent subsequently implemented strict whitelisting. Today, Thoth captures and catalogs system crash dumps alongside crashes from critical language runtimes like Java, Ruby, and Erlang—though Cantrill jokes that trying to make sense of an Erlang core dump is obtuse enough to make one contemplate ritual suicide. Thoth enables Joyent engineers to analyze crash patterns across thousands of production machines from a single centralized interface, running map-reduce jobs to identify shared stack traces and regressions.

Cantrill closes this section by discussing Manta's billing economics. Storage pricing is pinned directly to standard Amazon S3 rates, which have driven storage costs down to commodity levels. Compute execution is billed at rates equivalent to Amazon EC2, with one crucial commercial distinction: users are billed purely for the exact CPU seconds consumed by their tasks rather than paying continuous hourly instance rents.

[01:07:54](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4074s) Pricing Granularity and the Absurdity of Cloud Billing

Manta’s compute pricing is structured around actual runtime execution: if a job takes 10 seconds, the customer pays strictly for 10 seconds of compute. The rate comes down to roughly 40 micro-cents per second. While that sounds negligible, Bryan Cantrill points out that at scale—such as processing a million objects where each task runs for half an hour—the aggregate compute time rapidly mounts into years.

When an audience member asks about the billing resolution, Cantrill clarifies that while Joyent’s traditional Infrastructure-as-a-Service (IaaS) rounds up to the nearest hour (or roughly 14 cents), Manta is billed strictly by the second due to the transient nature of jobs. He pokes fun at Google’s marketing blitz around tight billing granularity, comparing a cloud provider boasting about per-minute or per-second billing to a hotel advertising that it rents rooms by the hour:

> “Google making a big deal out of it I thought was a bit strange; it's a bit like a hotel making a big deal by the fact that they charge by the hour... like, out of curiosity, like what kind of clientele is attracted to the by-the-hour pricing?”

Cantrill notes that Joyent could easily switch its core cloud service to per-second billing if customers truly demanded it, but he suspects that summing the rounding differences across every customer in the Joyent Public Cloud would amount to pocket change.

[01:09:32](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4172s) Implementing Manta Billing with Manta and Awk Puppet Theater

Handling billing is an inherently frustrating engineering chore. In industry-standard setups like Amazon S3, calculating usage traditionally triggers the deployment of an enormous Apache Hadoop MapReduce cluster. Cantrill admits that when Joyent first confronted the billing problem, he dreadfully wondered if they would need to stand up a Hadoop cluster of their own. Lead architect Mark Cavage shut that down immediately, asking if he was stupid and insisting: “No, you fool, we're going to use Manta for it.”

Manta dogfoods its own infrastructure: every customer request generates audit records that are stored as JSON payloads directly in Manta. Customers can inspect their own usage reports and bills via CLI commands like `mls`. During the talk, Cantrill jokingly navigates into his own invoice directory in Manta, displaying the raw JSON payload and teasing an audience member who asks whether customers can delete their bills. Cantrill notes that deleting your invoice file in Manta is the digital equivalent of burning your paper phone bill—Joyent maintains its own copies anyway.

To aggregate billing data, Manta runs batch jobs across these JSON logs using `json`, a CLI stream tool written by Joyent engineer Trent Mick. While the engineer who originally built the billing pipeline wrote the reduction phase in Node.js, Cantrill looked at the bloated script and realized it could be condensed into five lines of `awk`:

> “When I finally go around the bend, I probably will have my own public-access channel where I have like the Christian puppets for introducing awk: ‘No, baby, use awk.’”

After what Cantrill calls "Sunday school for awk" (or "awk Puppet Time Theater"), the pipeline was rebuilt using pure Unix utilities. This self-hosting milestone—where Manta's head started eating its tail—proved how fast ad-hoc analytics can be developed. An engineer can iterate live inside an interactive container via `mlogin`, test pipelines against a single object, step up to 10 objects, and then feed a million objects straight into `mjob` without provisioning any external infrastructure.

```mermaid flowchart TD accTitle: Manta Self-Hosted Billing Pipeline accDescr: Diagram showing how Manta processes its own billing records using JSON logs and awk reduction pipelines.

subgraph DataGeneration["Audit & Storage"] Req["User Requests"] -->|"generate logs"| Logs["JSON Audit Logs in Manta"] Logs --> Inv["User Invoices (JSON)"] end

subgraph BatchProcessing["Manta Batch Processing"] Logs -->|"mjob map"| JsonTool["json CLI tool (Trent Mick)"] JsonTool -->|"filter fields"| AwkStage["awk 5-line Reducer"] AwkStage --> FinalBill["Aggregated Usage & Revenue"] end ```

[01:14:01](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4441s) CAP Theorem Trade-offs: Split-Brain Truths and S3's "Just Kidding 200"

Cantrill touches on the distributed systems design of Manta, referencing an in-depth blog post by lead engineer Dave Pacheco on CAP theorem trade-offs. While distributed systems designers jokingly try to pick Consistency and Availability, network partitions are an unavoidable physical reality, forcing an architectural choice between CP (consistent and partition-tolerant) and AP (available and partition-tolerant). Manta resolves this by splitting its behavior cleanly between reads and writes: - **Writes are CP:** By default, writes specify two copies that are dispatched across separate data centers situated in Ashburn, Virginia (Cantrill quips that whenever a storm tracks through Ashburn, the entire cloud computing industry gets nervous). A write is only acknowledged with an HTTP `200 OK` once the object has been durably stored in both fault domains. If one facility is unreachable, Manta immediately returns an HTTP `500 Server Error`. - **Reads are AP:** Reads require only a single surviving data center to serve the requested object.

Cantrill contrasts Manta’s strict CP write policy with Amazon S3's eventual consistency model, which he criticizes for delivering what amounts to a "just kidding 200":

> “This is in contrast to S3: S3 will 200 you back when it's written it to enough and then things will catch up—except when they don't... You do a put: 200; get: 404... You're not going to get—if we can't write to both sides, we know that one data center is down, you'll get a 500 back, because with a 500 we think you know what to do with a 500: like, we don't have it, retry later, as opposed to like a 200 that's like a ‘just kidding 200’, which is like, how the hell are you supposed to do anything with that?”

[01:16:13](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4573s) Bringing Compute to Data as the Future of Big Data

Rounding out Manta's feature set, Cantrill highlights its built-in directory sorting via `munt` (powered by Moray and PostgreSQL), CORS headers, and "SnapLinks" (an architectural capability allowing zero-copy hardlinks for instant object versioning). The entire client toolchain is distributed as a Node.js SDK and CLI utilities available via `npm install manta`.

Cantrill reflects on the foundational philosophy of Manta: moving compute to data rather than hauling massive datasets across networks to ephemeral compute instances. While dataset sizes continue to expand exponentially, compute definitions—like a single line of `awk` or classic Unix shell pipelines—remain compact and lightweight, retaining the same power they possessed in Douglas McIlroy’s era:

> “We believe that the the idea that Manta represents—namely bringing compute to data—is absolutely the path forward for big data.”

- **Manta CLI package:** Distributed through npm (`npm install manta`). - **Data storage replication:** 2 copies across separate Ashburn, VA data centers by default. - **Write failure behavior:** Returns HTTP 500 immediately instead of returning an HTTP 200 before replication finishes. - **SnapLinks:** Hardlink primitive in Manta allowing instantaneous, space-efficient object versioning.

[01:17:14](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4634s) ZFS as the Foundation and Running Non-Java Tooling on Big Data

Bryan Cantrill reflects on the architecture underpinning Manta, joking that there might be some “Btrfs-LXC knockoff in like 50 years,” but reiterates that he would welcome competitors adopting the same computing model: “I would welcome other systems that have the same model, because it would verify that we're not entirely crazy.” In Joyent's view, ZFS remains the only sane storage underpinning capable of making this architecture work reliably. Having been publicly available for six months, Manta has already transformed how Joyent develops software and how users approach data processing.

```mermaid flowchart TD accTitle: Deploying Arbitrary Runtimes to Manta accDescr: Diagram showing how user tarballs create custom execution environments inside compute zones alongside data.

subgraph JobConfig["Job Configuration"] Assets["Asset Payload (Tarball)"] ResReq["Resource Spec (RAM & Storage)"] end

subgraph ExecutionZone["Manta Compute Zone"] Deploy["Unpack Tarball"] DataStream["Stream Local Object Data"] Runtime["Non-Java Engines (R, Python, Go)"] end

JobConfig --> Deploy Deploy --> Runtime DataStream --> Runtime ```

An audience member asks about the physical and environment constraints on jobs, observing that the ability to log in means one can execute arbitrary programs. Cantrill confirms that users do not even need an interactive session to deploy custom software. Instead, a job specification can include an asset payload—a tarball that Manta automatically lays down in the compute zone. This design liberates big data engineering from the traditional Java ecosystem, allowing engineers to run R, Go, or Python directly against data at full hardware speed:

- In data science, existing big-data R frameworks are often half-baked, but Manta enables high-performance R directly over large datasets. - Communities like natural language processing (NLP) have standardized heavily on Python libraries; Cantrill remarks, “If you want to do big data natural language, it's a nonstarter to do it in any other language besides Python.” - Jobs can define exact memory and storage constraints, which adjust billing accordingly.

[01:19:16](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4756s) Memory Footprints, Streaming Mappers, and Academic Pushback

When asked about specific environment defaults, Cantrill checks the system parameters: jobs receive 1 GB of RAM and 8 GB of local disk space by default. A scientist in the audience immediately objects, arguing that an 8 GB disk limit cannot handle real big data because they routinely work with individual objects and files of four terabytes. Cantrill clarifies the distinction between total data volume and worker memory: in a streaming MapReduce architecture, a 1 GB mapper can effortlessly stream four terabytes of data through its pipeline. For heavy aggregation in reducers, users can scale their memory allocation up to 64 GB of DRAM.

The audience member remains skeptical, complaining that toy examples undermine credibility: “I just feel a little nervous because, like, previously your example of big data was a bag of words, and, like, I'm sorry, I'm a scientist and I work in high-throughput sequencing, and it's not a bag anymore.” The exchange becomes tense yet spirited as the audience member adds, “If you want me to take you seriously, you can't use that as an example.” Cantrill defends the pedagogical choice by referencing Jeffrey Dean's classic Google MapReduce paper:

> “I actually would recommend, I mean, if you haven't read the Dean paper you should, because the Dean paper uses the same examples.”

[01:21:19](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4879s) Genomic Workloads, Object Sizing, and Infrastructure Economics

Cantrill emphasizes that high-throughput sequencing and genomic data (A, T, C, and G reads) represent an ideal sweet spot for Manta. He points to Christopher Hogg, a former scientist who joined Joyent after migrating his proteomics pipelines and R workflows to Manta. Turning to the concept of scale, Cantrill admits that in his own systems-debugging realm, “big data” usually tops out around 100 GB crash dumps, but stresses that storing single multi-terabyte files in an object store is often poor design because it limits parallelism. In systems like HDFS, terabyte-scale datasets are chunked into 100 MB blocks, and Manta users similarly benefit from deliberate object framing:

> “You don't have an internet pipe big enough to fill Manta faster than we can accommodate you, I can guarantee you.”

The debate shifts to workloads requiring massive compute running for weeks or years. The scientist notes their sequencing data already resides entirely in Amazon S3 and questions pricing. Cantrill clarifies that workloads needing permanent long-running compute nodes are fundamentally Infrastructure as a Service (IaaS) rather than serverless MapReduce. While Manta's baseline list rate is comparable to standard EC2 compute (roughly 14 cents an hour billed down to second-level granularity), Joyent negotiates custom pricing structures for users committing massive datasets and continuous compute footprints.

[01:25:31](https://www.youtube.com/watch?v=79fvDDPaIoY&t=5131s) Broad Domains: From Log Processing to Video Analytics

Cantrill wraps up his presentation by encouraging practitioners across diverse domains to explore how compute-on-storage alters their workflows. While scientific computing is gaining traction, everyday enterprise workloads like log processing remain Manta's most immediate and natural fit. Because compute runs local to storage without requiring network file transfers across VPCs, log analysis jobs execute with dramatic efficiency:

- Log processing remains the primary canonical workload, functioning exactly as the platform was engineered to do. - Video analytics pipelines use Manta to run distributed encoding, chunking, and computer vision jobs without provisioning dedicated rendering farms. - The Joyent engineering team routinely watches unexpected, highly creative pipelines ignite across the cluster as users exploit arbitrary POSIX binaries directly against object stores.

[01:26:38](https://www.youtube.com/watch?v=79fvDDPaIoY&t=5198s) Manta's General-Purpose Model and Pipeline Flexibility

Bryan Cantrill jokes that one unintended benefit of Manta’s architecture is that Bitcoin mining is not profitable on it: “Bitcoin mining is actually not profitable on this, thank God.” Moving beyond crypto, he emphasizes that the system has an enormous variety of real-world use cases. The most exciting aspect from Joyent's perspective is how Manta appeals directly to the creativity of engineers: “People have been and will continue to surprise us with what they're doing with this because it is a general purpose compute platform, and you can do anything you can do with general purpose compute you can do on Manta.”

An audience member asks about the workflow execution model, inquiring whether users always initiate remote operations as jobs from their local terminal, starting with commands like `mfind` and piping them into subsequent phases. Cantrill confirms this, referencing a live demo where `mfind` generated a list of objects locally and piped them into `mjob create`. He explains that although users might question hauling an object list out of the cloud only to pipe it back in, Manta allows multiple architectural approaches. A user can easily configure a job whose initial phase runs `mfind` directly inside the cloud as part of a multi-stage MapReduce workflow, keeping the entire pipeline remote. Cantrill points out that, just like the Unix philosophy it mirrors, Manta offers deep flexibility depending on what a developer wants to accomplish.

```mermaid flowchart LR accTitle: Pipeline Construction Options in Manta accDescr: Diagram contrasting running mfind locally versus executing mfind directly inside a cloud compute job. subgraph LocalPiped ["Standard Local Invocation"] A["mfind (local)"] -->|"Object names"| B["mjob create (cloud)"] end subgraph CloudNative ["Fully Cloud-Native Pipeline"] C["Phase 1: Cloud mfind"] -->|"Direct stream"| D["Phase 2: Map"] D -->|"Piped output"| E["Phase 3: Reduce"] end ```

[01:28:16](https://www.youtube.com/watch?v=79fvDDPaIoY&t=5296s) Stressing ZFS Rollbacks and the Hardware Firmware Crisis

An audience member asks Cantrill about his biggest current issue with ZFS and whether he is ever tempted to work directly on hardware firmware. Cantrill quips that if he enters the firmware space, “it's going to be with a suicide vest,” drawing laughter from the room. Turning to ZFS, he explains that Joyent is pushing the filesystem into previously untested dimensions, specifically around rapid and continuous rollbacks. Whenever an interactive `mlogin` session terminates, a ZFS rollback immediately restores the underlying SmartOS zone to its pristine base state.

Historically, system administrators used rollbacks infrequently—mainly to recover from system errors or play around with snapshots—meaning rollbacks were never positioned in the high-frequency operational data path. Manta puts rollbacks directly into that critical path: “We're hitting rollback hard, and that's been great, and we found, you know, found and fixed a couple of issues there, but, you know, by and large, the only real way to find ZFS issues these days is to push it into some wholly new dimension like that.” Despite these edge cases, ZFS remains rock-solid for Joyent. When it comes to storage drive firmware, Cantrill expresses resignation, noting that while disk drive firmware has slightly improved, the overall hardware firmware crisis will likely never end, leaving systems engineers to continuously defend against unreliable components: “I would never buy the first rev of a firmware of anything for sure.”

- **ZFS Rollback in Data Path:** Zones are rolled back to their clean snapshot state immediately after an interactive session (`mlogin`) ends, transforming an administrative tool into an automated, continuous operational task. - **Hardware Strategy:** Cantrill notes that disk drive firmware has matured modestly, but advises never running the initial revision (rev 1) of any firmware in production.

[01:30:14](https://www.youtube.com/watch?v=79fvDDPaIoY&t=5414s) Parallel Job Execution and Zone Allocation

An audience member asks for clarification on execution concurrency, questioning whether Manta jobs can operate on only one object at a time. Cantrill clarifies that Manta is inherently massively parallel. In the live demonstration using 300 input files, a single `mjob` invocation instantly instantiates 300 independent zones to process each object simultaneously in the map phase.

He highlights the pipeline where commands like `tr` piped to `uniq` run across 300 zones in parallel, humorously admitting with an asterisk that testing against 300 manual pages is “not true Big Data” and joking that he should generate gigabyte- or terabyte-sized man pages to make it look heavier. Following the distributed map phase, the data flows directly into the reduce stage, which in this demonstration consolidates the intermediate streams into a single reducer zone.

## Notable quotes > [20:29](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1229s) Building a storage product is like going backpacking in Afghanistan, and if you don't think you're going to a war zone, you're delusional. — Bryan Cantrill

> [50:08](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3008s) When you can build things in terms of the Unix philosophy, you always should. — Bryan Cantrill

> [14:06](https://www.youtube.com/watch?v=79fvDDPaIoY&t=846s) There is no fsck for ZFS; such a thing doesn't make sense. ZFS is always consistent on disk. — Bryan Cantrill

> [01:16:13](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4573s) We believe that the the idea that Manta represents—namely bringing compute to data—is absolutely the path forward for big data. — Bryan Cantrill

> [01:17:14](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4634s) If you want to do big data natural language, it's a nonstarter to do it in any other language besides Python. — Bryan Cantrill

> [48:11](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2891s) The way we are able to develop complicated systems is by building out of simple ones, taking simple systems and actually connecting them in interesting and new ways. — Bryan Cantrill

## Mentioned Amazon EC2, Amazon S3, Amazon Web Services (AWS), Apache Hadoop, Ars Technica, Awk, Bell Labs, Btrfs, Bryan Cantrill, Christopher Hogg, David Patterson, Dave Pacheco, Dennis Ritchie, Donald Knuth, Doug McIlroy, DTrace, GE 635, Google, Google App Engine, Hacker News, HDFS, Honeywell 200, IBM 1401, IBM 350 RAMAC, IBM AIX, IBM System/360, illumos, Jeff Bonwick, Jeffrey Dean, John Bentley, Josh Clulow, Joyent, Joyent Public Cloud, KVM, LXC, Manta, Mark Cavage, Matt Ahrens, Moray, Multics, Node.js, Oracle, PostgreSQL, Programming Pearls, Project Thoth, RAID, Research Unix Seventh Edition (V7), Sanjay Ghemawat, Seagate S0D, SmartOS, Solaris, Sun Microsystems, Sun Storage 7000 (Fishworks), Trent Mick, Unix, Unix File System (UFS), USENIX, Veritas File System (VxFS), Veritas Volume Manager (VxVM), WEB, Western Digital Green, ZFS

## Video description ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.

FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai Facebook: https://www.facebook.com/datacouncilai Eventbrite: https://www.eventbrite.com/o/data-council-30357384520 -

🎟️ GET YOUR TICKET TO AI COUNCIL 2026 🎟️ Meet the world's top AI infrastructure minds where architects of AI share what works. Three days of high-quality technical talks and meaningful interactions.

→ https://aicouncil.com/sf-2026

⚡ FIND US: X: https://x.com/AICouncilConf LinkedIn: https://www.linkedin.com/company/aicouncilconf/ Website: https://aicouncil.com/

[Source](https://www.youtube.com/watch?v=79fvDDPaIoY)

中文简报

Joyent 的 Bryan Cantrill 详解 Manta：兼具计算能力的面向互联网对象存储 Bryan Cantrill 回顾了从早期磁盘驱动器、卷管理器的架构缺陷到 ZFS 发明的计算发展史，阐述了当计算与存储解耦时传统分布式系统为何举步维艰。他介绍了 Joyent 的面向互联网对象存储系统 Manta，该系统通过直接在数据所在的存储节点上运行临时 SmartOS 容器，解决了网络传输瓶颈。Cantrill 提倡管道和简单可组合工具的 Unix 哲学，而非 Hadoop 等复杂框架，并展示了如何使用熟悉的 Shell 命令以及 Awk 等实用工具来执行海量 MapReduce 任务并进行就地调试。 Joyent 推出面向互联网的对象存储系统 Manta，可直接在存储节点上启动 SmartOS zone 并在紧邻数据处执行任意计算，从而消除了网络传输瓶颈。 (41:04) Manta 秉承 Unix 哲学，有别于 Hadoop 等传统大数据框架，允许开发者直接针对存储对象运行 grep、awk 及 Python 管道等标准命令行工具。 (54:25) 与 Amazon S3 的最终一致性模型不同，Manta 跨独立数据中心实现 CP 写入，仅在数据于两处均持久化存储后才返回 HTTP 200。 (74:01) Manta 上的计算以秒为粒度计费，每 CPU 秒约 40 微美分，避免了持续的按小时实例收费。 (67:54)

Reading summary

Joyent developed Manta, an internet-facing object storage platform that colocates compute jobs directly on the storage nodes holding target data, aiming to eliminate the network overhead of transferring massive datasets. Storage Abstractions, Data Corruption, and ZFS (3:15) Bryan Cantrill traces storage evolution from the 1956 IBM RAMAC to RAID, arguing that the volume management layers introduced in the 1980s operated as an architectural crutch. By presenting multiple disks to file systems as a single block device without semantic awareness, systems suffered from phantom writes and silent data corruption. These failures led Sun Microsystems engineers Jeff Bonwick and Matt Ahrens to develop ZFS, which integrated storage management with a copy-on-write architecture, eliminate offline filesystem checks like fsck, and introduced parent block pointer indirect checksums to catch firmware deceptions. Container Virtualization Over Hardware Emulation (38:41) Cantrill distinguishes between hardware virtualization and operating-system-level virtualization, arguing that hardware hypervisors impose rigid resource constraints, particularly around overcommitting dynamic memory. Joyent built its public infrastructure around SmartOS zones—OS-level containers sharing a single illumos kernel. Containers can dynamically resize memory limits upward or downward without requiring reboots or virtual machine migrations, providing the low-latency isolation needed to run transient software sandboxes directly atop bare-metal storage pools. Colocating In-Situ Compute With Storage (41:04) Rather than replicating Amazon S3's paradigm of fetching objects across the network to external compute nodes, Joyent architect Mark Cavage proposed executing jobs locally on the physical nodes storing the data. When an analysis or post-mortem debugging task like mlogin is invoked, Manta spins up a transient SmartOS zone directly adjacent to the object. Because the input data is read locally via ZFS, overall job processing times scale minimally regardless of whether files are small samples or multi-gigabyte core dumps. Unix Pipelines Versus MapReduce Frameworks Cantrill critiques modern Big Data frameworks, arguing that complex C++ MapReduce setups overcomplicate elementary problems solved decades earlier by Doug McIlroy’s Unix pipes. Manta implements this pipeline philosophy by permitting standard Unix utilities—such as grep, sed, and awk—to operate directly in-situ across objects. Jobs receive a default baseline of 1 GB RAM and 8 GB local scratch space, supporting user-supplied binaries, asset tarballs, and parallel streaming mappers across hundreds of objects simultaneously. Distributed Trade-offs and Consistency Model (74:01) Under the CAP theorem, Manta divides its guarantees between operations: writes are CP, requiring durable writes to two distinct data center fault domains in Ashburn, Virginia before returning an HTTP 200, or immediately failing with an HTTP 500 error during partitions. Reads operate under an AP model requiring only one reachable copy. For accounting and system tracking, Joyent dogfoods Manta by streaming audit JSON logs directly into the storage cluster and aggregating billing metrics with command-line utilities. While Cantrill maintains that in-situ execution represents the future of big data processing, he notes unresolved engineering friction with underlying hard-drive firmware anomalies and the operational stress placed on rapid ZFS rollbacks when recycling transient compute zones.

精炼阅读摘要

Joyent 首席技术官 Bryan Cantrill 指出，现代大数据处理长期受制于跨网络搬运数据的低效架构与沉重的计算框架；为此，Joyent 开发了将计算直接下沉至存储节点的分布式对象存储系统 Manta，依托 ZFS 的数据完整性保障与操作系统级容器虚拟化，在云端重新激活了轻量、高效的 Unix 流式处理哲学。存储抽象的历史漏洞与 ZFS 完整性 (9:31) 回顾数据存储演进，单盘存储密度在 1980 年代遭遇物理极限后，加州大学伯克利分校提出了 RAID 架构。然而，软件工业当时并未重构文件系统以适配多盘物理现实，而是引入逻辑卷管理（Volume Management）这一抽象层来向文件系统隐瞒底层物理拓扑。Cantrill 直言卷管理是“临时拼凑的谎言”，在 1990 年代引发了灾难性的语义断层：卷管理器仅处理底层数据块，文件系统仅理解目录与文件元数据，两者通信局限于极度简陋的块读写接口。这种脱节使得幽灵写（phantom write）与幽灵读等硬件固件故障无法被有效拦截，造成严重的静默数据损坏。为彻底解决这一结构性缺陷，Sun Microsystems 的工程师 Jeff Bonwick 与 Matt Ahrens 摒弃了卷管理层，自底向上研发出完全事务性的写时复制（Copy-on-Write）文件系统 ZFS。ZFS 废除了在海量机械磁盘上面对物理寻道瓶颈时极度低效的脱机一致性检查工具（fsck）。通过在父级块指针中存放间接校验和（indirect checksumming），ZFS 能够有效捕获静默损坏并进行自愈，同时利用事务回滚机制防御虚报刷盘完成的劣质硬件固件，为现代大规模分布式存储奠定了高可靠的基础底座。虚拟化范式抉择与计算向数据靠拢 (41:04) 云计算的基础在于多租户隔离虚拟化，但虚拟化处于系统架构的哪一层直接决定了系统的弹性上限。Cantrill 指出，源自 1960 年代 IBM System/360 兼容 1401 旧机型的硬件级虚拟化，在现代云环境中面临严格的资源调度刚性：虚拟机管理程序（Hypervisor）必须提供真实的物理 DRAM 支撑，无法灵活超卖，且 I/O 虚拟化存在显著开销；而平台即服务（PaaS）又过早收窄了抽象边界。相比之下，Joyent 押注基于 SmartOS 的操作系统级容器虚拟化（Zones），单个内核直接运行在裸机硬件上，以系统调用为边界，实现毫秒级拉起、轻量隔离以及无需重启即可动态调整 DRAM 限制的高效资源调度。在构建具备互联网访问能力的对象存储时，Joyent 工程师 Mark Cavage 于 2011 年 10 月提出了关键的架构突破：不再模仿类似 Amazon S3 仅允许客户端通过 HTTP GET 跨网络拉取大文件到本地处理的模式，而是利用轻量容器直接将计算带到数据存储的位置。当存储对象需要处理时，系统直接在该对象所存放的物理存储节点上即时拉起 SmartOS Zone 容器，使任意计算程序直接访问本地磁盘数据，从而彻底消除了海量数据跨网络传输的带宽与时间瓶颈。 Unix 哲学复兴与对重型框架的批判 (51:14) Cantrill 将软件架构设计视作“简单性力量与复杂性力量之间的圣战”，坚决反对让开发者深陷黑盒与复杂抽象的重型框架体系。他援引 1986 年计算机科学史上经典的文本词频统计挑战：Donald Knuth 耗费 7 页复杂的 WEB 语言代码实现算法，而 Doug McIlroy 仅用 6 行由管道连接的基础 Unix 命令（tr、sort、uniq、sed）便达成了极具竞争力的性能与优雅实现。 Cantrill 批评现代大数据生态重蹈了 Knuth 的覆辙。Google MapReduce 论文在解决相同的词频与 URL 访问计数问题时，附录却给出了 150 行密集的 C++ 代码。他认为要求开发者为了基础的计数与文本提取去引入臃肿的编译工具链与庞大的计算框架，是对以单一职责和通用文本流为核心的 Unix 哲学的严重背离。Manta 将对象存储设计为原生分层目录体系，使用户能够通过标准命令行工具组合，直接在云端对海量存储对象发起分布式流式处理。 Manta 工程实现、原位调试与一致性权衡 (61:22) 在工程实现上，Manta 规避了传统对象存储繁琐的分块上传握手机制，对大文件支持直接以流式 HTTP 方式写入，仅需预先声明 Content-Length 以便调度器在物理节点上分配合适连续空间。针对分布式环境中最棘手的大型内核转储（crash dump）与核心转储分析，工程师 Josh Clulow 研发了 `mlogin` 工具，能够直接在目标转储文件所在的物理存储节点瞬时生成交互式容器，让工程师免受数小时下载数百 GB 镜像之苦，直接在搭载完整 GNU 工具链的环境中现场调试系统；Joyent 据此建立了长期归档与分析云端崩溃的“Thoth 项目”。针对 CAP 定理在分布式系统中的取舍，Manta 实施了非对称的一致性策略：写操作采用严格的 CP 模式，默认必须同步写入位于弗吉尼亚州阿什本的两个独立数据中心后才返回 HTTP 200，若分区出现故障则立即返回 HTTP 500，杜绝了 S3 最终一致性模型下偶发的“假成功”隐患；读操作则遵循 AP 模式以保障服务的高可用。其按实际执行秒数计费的机制不仅消除了虚高账单，Manta 甚至直接利用自身的流式处理能力与 awk 脚本实现了对平台计费日志的大规模并行聚合。尽管有生物信息学研究者质疑小内存映射器在应对单体数 TB 基因测序数据时的实用性，Cantrill 强调流式 MapReduce 架构天然支持小内存处理大流，且系统已被证实能高效支撑日志挖掘、基因组学与视频分析等多种异构工作负载。他坦言底层存储驱动固件的偶发性严重缺陷依然是系统工程难以根除的痛点，而 Manta 高频调用 ZFS 事务回滚来重置交互式调试容器的做法，不仅证明了底层文件系统与原生轻量容器在原位计算中的敏捷性，也为探索脱离重型计算框架束缚的通用云基础设施提供了实践样本。

Detailed notes — zh

## Background 该视频由 YouTube 频道“AI Council”（原 Data Council）于 2014 年 1 月 29 日发布。视频记录了 Joyent 前首席技术官 Bryan Cantrill 的技术演讲，页面描述介绍该频道是一个面向数据专业人士的学习与交流社区，本期内容深入探讨了 Joyent 推出的集成计算能力的面向互联网对象存储服务 Manta。 ## TL;DR Bryan Cantrill 从单盘容量瓶颈、卷管理抽象的欺骗性以及 ZFS 打破层级边界的历史讲起，深入剖析了传统存储架构面临的固件故障与数据静默损坏挑战。随后他对比了硬件虚拟化与操作系统级虚拟化，阐述了将计算移向存储核心的设计哲学，并详细介绍了 Joyent Manta 的诞生背景及其如何将经典 Unix 管道与对象存储深度结合。最后，他通过现场演示展示了 Manta 的秒级计费、交互式调试工具 mlogin、流式原地计算能力，并回应了现场关于科学计算与高通量基因测序等极端负载的质疑与讨论。 ## Atmosphere & style 整场演讲洋溢着硬核黑客的犀利幽默与直言不讳的战斗气息，Bryan Cantrill 用大量辛辣的比喻和亲身工程历险故事将原本晦涩的底层操作系统概念讲得妙趣横生。台下观众反应热烈，在关于科研大数据负载的质询环节中更激发出火花四溅但务实透彻的工程交锋。现场节奏极具沉浸感与感染力，听众仿佛置身于一场充满自嘲、反思与工程骄傲的系统技术奥德赛之旅。 ## Bulletin - [41:04](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2464s) Joyent 基于 ZFS 和 SmartOS Zones 推出了面向互联网的对象存储系统 Manta，开创了在数据存储节点原地拉起容器直接执行计算的“就地计算”模式，避免海量数据跨网络回传。 - [54:25](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3265s) Manta 的分布式批处理深度践行 Unix 哲学，允许开发者将 awk、tr、grep 等原生命令行管道直接打包分发至云端并发运行，全面替代笨重的 Hadoop 和 Java/C++ MapReduce 框架。 - [01:14:01](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4441s) 在 CAP 定理取舍上，Manta 采用读写分离策略：写入端为 CP 系统，必须在两个独立数据中心均落盘才返回 HTTP 200，杜绝了 S3 常见的虚假 200 与随后读出 404 的问题；读取端则为 AP 系统保证高可用。 - [01:07:54](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4074s) Manta 的算力按秒精确计费（单价约 40 微美分/秒），同时内置的 mlogin 工具允许开发者直接进入存储节点上的隔离容器对超大崩溃转储文件进行交互式调试。 ## Key takeaways - [05:00](https://www.youtube.com/watch?v=79fvDDPaIoY&t=300s) 传统存储通过逻辑卷管理（LVM）向文件系统隐藏多盘拓扑，这种“友善的谎言”导致接口语义极度贫乏，无法防范幽灵写入（Phantom Write）与硬件错位读等静默数据损坏。 - [10:47](https://www.youtube.com/watch?v=79fvDDPaIoY&t=647s) ZFS 彻底打破了文件系统与卷管理的边界，通过统一存储池、写时复制（Copy-on-Write）以及将校验和存放在父级元数据指针中的“间接校验和”技术，从根本上实现了静默数据损坏自愈并淘汰了离线 fsck。 - [21:08](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1268s) 底层磁盘控制器与硬件固件充满病态的非死即活拜占庭故障，例如希捷 s0d 固件的磁头极性反冲导致 650 毫秒断电重启，这迫使成熟存储系统必须从架构上容忍灰度故障与事务回滚。 - [28:15](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1695s) Amazon S3 的核心远见在于将其设计为不支持局部写入（Partial Writes）的对象存储，利用 WORM（单写多读）特性放宽分布式系统约束，而非直接封装传统文件系统的 POSIX 抽象。 - [30:27](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1827s) 从 IBM 360 模拟 1401 开始的硬件虚拟化虽然兼容性好，但因狭窄的物理抽象面临 DRAM 与 I/O 无法灵活超卖的瓶颈；而基于系统调用表的操作系统级虚拟化（如 SmartOS Zones）能实现毫秒级动态调容与裸机性能。 - [41:04](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2464s) Manta 的核心架构灵感源于 Mark Cavage 的洞见：利用操作系统级虚拟化在存储节点上直接拉起容器，把计算带到数据面前，彻底改变大数据处理的数据传输成本。 - [54:25](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3265s) 现代大数据技术栈充斥着对简单 Unix 哲学的背叛，本该用简单单行命令解决的词频统计在 Hadoop 生态中演变成了数百行繁重的 Java 和 C++ 编译代码。 - [58:18](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3498s) Manta 采用真正的树状层级存储，并支持纯流式上传海量单对象，规避了 S3 复杂且繁琐的分块上传（Multi-part Upload）机制。 - [01:09:32](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4172s) Joyent 利用 Manta 内部的自举能力构建了线上崩溃分析平台 Thoth，并仅用 5 行 awk 脚本替代原本复杂的 Node.js 代码来实现自身的计费日志聚合。 - [01:19:54](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4794s) 面对高通量基因测序等 PB 级科学计算场景，流式计算模型（Mapper 流式读、Reducer 配置大内存）能有效避免过大单文件对并行度的破坏，验证了存储计算一体化的通用扩展能力。 - [01:28:16](https://www.youtube.com/watch?v=79fvDDPaIoY&t=5296s) 在工程实践中，Manta 将 ZFS 的 zfs rollback 特性推入核心数据路径，在每次交互会话退出时高频恢复干净容器环境，借此检验并压榨了文件系统的稳定性极限。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=79fvDDPaIoY&t=0s) 开场：硅谷工程师的地域文化与“披萨战争” Bryan Cantrill 登台开场。他首先调侃了自己在加州硅谷度过的整个职业生涯，直言自己在这段漫长的时间里“从未共事过任何一位加利福尼亚本地出生的工程师”。硅谷的工程师团队本质上是从全美乃至全世界汇聚而来，纽约虽然在某种程度上也是如此，但在加州表现得尤为极致。而这种高度多元的背景，带来了一个出人意料的团队管理难题： > “因为加州、因为任何工程团队本质上都是从全美和全世界吸收人才，这意味着分裂一支工程团队、让兄弟反目成仇最稳妥的办法，就是点外卖披萨送进来。” Cantrill 戏谑道，给工程师团队点披萨绝不仅是一顿饭，而是一种“地域霸权的宣示”——某位工程师的家乡风味势必会遭遇严重侮辱，场面瞬间演变成“极度血腥的冲突”，就像“纳秒之间爆发的塞族人对克族人战争”。芝加哥深盘派和纽约薄底派之间的争斗惨烈至极，以至于在线下聚会活动中“我们根本不配拥有美好的事物”，甚至无法安稳享用一份地道的纽约风格披萨。不过，作为非交战方的 Cantrill 很高兴今晚的聚会现场终于有了一份没有引发纷争的披萨。 ### [01:30](https://www.youtube.com/watch?v=79fvDDPaIoY&t=90s) 技术史诗与坦诚的技术起源真相 Cantrill 将本场演讲定义为一场“技术史诗之旅（Odyssey）”。他谈到自己从事操作系统底层开发已有约 20 年，刚满 40 岁，自嘲说如果大家觉得他现在看起来像 12 岁，那应该看看他 22 岁刚入行时的模样，简直就跟婴儿没两样。Cantrill 极度推崇以历史的视角来审视技术，他强调今晚要讲述 Joyent 开发的一项核心技术（即 Manta）的真实诞生故事： > “当你偶然发现某个伟大的事物，或者说你实际上撞了大运，做出了某项重要发现、开发出某种重要技术时，你总是会在事后重构你的思路，粉饰成‘啊对，这就是我一开始就打算解决的问题’；但现实很少是这样的。” 为了证明技术组合的奇妙碰撞，Cantrill 引用了 80 年代 Reese's 花生酱巧克力的经典广告梗：“你的电脑掉进了我的数据里 / 你的花生酱掉进了我的巧克力里，两种绝妙风味融合在一起”。台下观众反应平平，Cantrill 笑着恳求大家看在同情的份上举举手，自嘲这个引用纯粹暴露了自己的年龄，随后正式引出本场演讲的技术主轴——文件系统的演进史。 ### [03:15](https://www.youtube.com/watch?v=79fvDDPaIoY&t=195s) 单盘容量撞墙与 RAID 的诞生：低成本磁盘阵列 Cantrill 带领观众回顾 1970 至 1980 年代文件系统的“史前时代”。在早期，存储行业信奉的法则是将单块磁盘做得越来越大。从 1956 年 IBM 制造出人类第一台硬盘 RAMAC（305 RAMAC）开始，磁盘长着巨大的盘片，行业在随后的 70 年代至 80 年代初所做的事情，无非是不断提高单盘的记录密度、稍微缩小尺寸，但核心依然是靠单盘容量扩张。到了 1980 年代，这种依靠单盘无限制变大的路径走到了尽头。正如微处理器的发展逻辑一样，物理定律决定了单一部件无法无限做大，工程师必须将更小的部件连接串联起来。于是，加州大学伯克利分校的 David Patterson 等人提出了 RAID（Redundant Array of Inexpensive Disks，廉价磁盘冗余阵列）。 Cantrill 辛辣地讽刺了存储硬件厂商的贪婪：如今很多靠买硬盘谋生的运维人员早已忘记 RAID 中的“I”代表的是“Inexpensive（廉价）”，因为现在的存储厂商恨不得向客户坚称“那是代表 Independent（独立），我不知道你从哪听来‘廉价’的”。Cantrill 痛骂：“老子是从历史里学来的，混蛋，你那高价不过是想给你的下一艘游艇付首付款！” ### [05:00](https://www.youtube.com/watch?v=79fvDDPaIoY&t=300s) 计算机科学经典骗局：逻辑卷管理（LVM）的谎言与代价在 1980 年代，尽管传统文件系统本身才诞生不过一二十年，但在软件领域，这已经老旧而僵化到让当时的工程师难以承受重写的代价。面对底层硬件从“单盘”变成“磁盘阵列”的剧变，计算机科学界做出了它最习惯的妥协： > “既然能直接对它撒谎，为什么还要大费周折重写它呢？它只是一段软件，根本分不清真相，我们会编造一个天衣无缝的谎言。” 这个谎言正是“卷管理（Volume Management / LVM）”。计算机系统在文件系统和一堆物理磁盘之间插了一层逻辑卷管理器，把几十上百块物理磁盘伪装成“一块磁盘”，骗上面的文件系统说它依然在和一个单一磁盘愉快对话。 ```mermaid flowchart TD accTitle: The Lie of Logical Volume Management accDescr: An architectural breakdown showing the traditional disconnect between the file system and multi-disk physical hardware bridged by logical volume managers. subgraph UserSpace["软件认知层 (The Lie)"] FS["传统文件系统 (UFS / VxFS 等) 『认为自己只挂载在单一磁盘上』"] end subgraph VolumeLayer["伪装与欺骗层"] LVM["逻辑卷管理器 (LVM / VxVM) 『将多盘拼凑并伪装成单一大块设备』"] end subgraph PhysicalDisks["物理硬件层 (Reality)"] D1["磁盘 1"] D2["磁盘 2"] D3["磁盘..."] DN["磁盘 N (10 ~ 100+ 块)"] end FS -->|"块接口 (Read / Write Block)"| LVM LVM --> D1 LVM --> D2 LVM --> D3 LVM --> DN ``` Cantrill 指出，谎言往往始于善意，但谎言早晚会反噬人类： - **无法动态扩容**：文件系统的逻辑认知中“磁盘是不会凭空变大的”，盘片不可能在主轴上“互生情愫度过春宵然后惊喜生出两张新盘片”，因此文件系统在很长时间内几乎完全是静态的，扩容极其困难，缩容更是绝无可能。 - **缺乏语义信息（Semantic Void）**：卷管理层并不知道文件的任何语义结构，实现诸如快照（Snapshot）等功能痛苦万分，存储管理变成了灾难。 - **致命的系统崩溃**：到了 1990 年代，部署在 VxVM 上的 VxFS、部署在 LVM 上的 UFS 等系统频繁出现灾难性故障，甚至导致整个卷的数据彻底灰飞烟灭，“让经历过那个时代的系统管理员至今都需要心理治疗”。 ### [08:25](https://www.youtube.com/watch?v=79fvDDPaIoY&t=505s) 语义断层与静默数据损坏：幽灵写入（Phantom Write） Cantrill 深入剖析了文件系统与磁盘之间由极度狭隘的接口引发的“语义断层（Semantic Divide）”。文件系统和块设备之间唯一的沟通语言就是极其简陋的块操作指令：`write block`、`read block`。这种极其狭窄的接口导致双方根本无法相互校验和自我修复。在数据完整性保护方面，虽然文件系统能够通过校验和（Checksum）来捕捉静默位翻转（Bit rot），但底层卷管理器完全无法识别更高阶的逻辑错误，比如最致命的“幽灵写入（Phantom Write）”： - 操作系统向磁盘控制器发出写入某一特定数据块的指令。 - 控制器向操作系统汇报已成功写入非易失性稳定存储介质（Stable Storage）。 - 然而在物理层面，该写入操作由于硬件固件缺陷或电气异常根本没有发生。 - 当系统后续发起读取请求（例如需要第 65 号块）时，控制器却一脸茫然，仿佛从没听说过这次写入，直接导致数据逻辑产生不可逆的撕裂与损坏。 ### [09:31](https://www.youtube.com/watch?v=79fvDDPaIoY&t=571s) 幻读灾难与卷管理抽象的历史包袱在传统的分层存储架构中，底层硬件与固件缺陷引发的静默数据损坏堪称噩梦。Bryan Cantrill 指出了一种极为真实的“幻读”（phantom read）现象：操作系统本想读取 63 号扇区，底层固件却阴错阳差地返回了 36 号扇区。由于读出来的数据本身在结构上完全自洽且校验无误，下层的卷管理层（Volume Management Layer）会若无其事地将其放行并透传给上层文件系统。然而，文件系统接盘后才会惊恐地发现：“等等，这里本该是数据库的数据，怎么读出来的竟然是 `/etc/passwd`？”经历过那个时代的人对这种事故记忆犹新，当你在数据库表空间里赫然看到系统密码文件时，那种深入脑髓的惊悚感足以让人生理不适。这类隐蔽的错误深埋在整个技术栈的底层，暴露出系统在可管理性、性能以及复杂度上的全方位崩溃。 Cantrill 尖锐地批评道，卷管理抽象本应只是计算机存储演化过程中的临时权宜之计，却不幸演化成了一种长期苟延残喘的行业解决方案。直到今天，许多系统依然被困在文件系统与卷管理割裂的架构深渊中。早在 Sun Microsystems 工作的 14 年间，他们就深刻意识到这种割裂带来的恶果，并下定决心实施一场彻头彻尾的底层革命。 > “The problem with that is it's totally self-consistent so the volume management layer is like looks good to me I don't know sure pass it up to the file system file system's like whoa no no no that's like Etsy password that's not the database.” —— Bryan Cantrill --- ### [10:47](https://www.youtube.com/watch?v=79fvDDPaIoY&t=647s) 脊髓灰质炎疫苗隐喻与 ZFS 的重构之道作为对旧有存储体制的颠覆，Sun Microsystems 开发了 ZFS。Cantrill 在现场做了一个互动，欣慰于全场绝大多数人都听说过 ZFS，同时也理解为何实际在生产环境中部署它的人并不多。他戏谑地将这种现状比作医疗悲剧：很多人明明知道世界上早已存在能根治脊髓灰质炎的疫苗，却因为各种借口（例如“主管不批准我接种”、“疫苗有软件许可证风险”）始终没有接种，任由自己和下一代不断遭受小儿麻痹症的折磨，并在灾难降临时感叹“要是当年打了疫苗该多好”。针对 Linux 社区后来模仿 ZFS 开发的 Btrfs（Cantrill 调侃其为“better-FS”或“butter-FS”），他提到 Ars Technica 当周发表的一篇深度历史报道，文中非常客观地将其定义为“下一代系统”而非“全新系统”，因为 ZFS 的设计理念其实早在十多年前就已成熟，Cantrill 本人所有的数据在 ZFS 上已经稳健运行了超过十年。 ZFS 的核心是彻底打破卷管理与文件系统之间的边界。早在 20 世纪 90 年代末，Sun 的顶尖工程师 Jeff Bonwick 就对传统架构积怨已久，立志彻底摧毁旧范式。随后，Jeff Bonwick 与 Matt Ahrens 等工程师拿出一张白纸，从零开始重新构想存储系统。他们基于几大现实物理定律与技术趋势进行架构设计： 1. **多磁盘并存是不可逆转的客观现实**； 2. **系统拥有大容量 DRAM**，必须极致、高效地发挥其缓存作用； 3. **预设非易失性主内存（NVM）的普及**：这是设计初期唯一打偏的预测。Cantrill 吐槽道，在 2001 年他们深信到 2014 年计算机必定全面普及非易失性内存，结果现实残酷地证明这一设想依然遥不可及。 ```mermaid flowchart TD accTitle: ZFS 存储池架构与传统分层模型对比 accDescr: 对比传统架构中文件系统与卷管理的严格隔离，以及 ZFS 打通边界后由统一存储池对接虚拟文件系统的结构。 subgraph Legacy["传统架构 (繁琐且割裂)"] direction TB L_FS["POSIX 文件系统 (ext3 / UFS 等)"] L_VM["卷管理层 (LVM / 软 RAID)"] L_Disk["物理磁盘 (块设备)"] L_Disk --> L_VM --> L_FS end subgraph ZFS_Arch["ZFS 架构 (存储池模型)"] direction TB subgraph Top["管理中枢 (轻量级文件系统)"] Z_FS1["/export/home"] Z_FS2["/var/data"] Z_FS3["/db (独立配额/快照策略)"] end Z_Pool["ZFS 存储池 (zpool 统一资源分配)"] subgraph Bottom["物理设备拓扑"] Z_D1["磁盘 1"] Z_D2["磁盘 2"] Z_D3["磁盘 N..."] end Z_D1 & Z_D2 & Z_D3 --> Z_Pool Z_Pool --> Top end ``` 在 ZFS 的模型中，底层磁盘被组合成统一的存储池（Storage Pool），而顶层的文件系统被剥离掉了任何与物理磁盘的硬性绑定，退化为一个纯粹的“管理纽带”（administrative nexus）。一个文件系统在本质上仅仅代表了命名路径、目录层级和管理策略（如配额、压缩、快照），执行 `zfs create` 创建文件系统瞬间即可完成，底层则根据需求从存储池中透明地按需分配空间，并支持灵活配置镜像（Mirroring）或 RAID-Z（如双校验盘的 RAID-Z2）等冗余策略。 > “As long as I had known Jeff he wanted to completely annihilate this and he wanted to go completely rethink this problem and so what he and Matt Ahrens and some other engineers did is they literally went back to a clean sheet of paper and how would we design this given that we have multiple discs are a fact of life.” —— Bryan Cantrill --- ### [14:06](https://www.youtube.com/watch?v=79fvDDPaIoY&t=846s) 写时复制、谎话连篇的硬件与 fsck 的消亡由于 ZFS 采用了纯粹的写时复制（Copy-on-Write）设计，数据在磁盘上的状态在任何时刻都是事务性自洽的。这意味着 ZFS 体系中彻底废黜了传统文件系统不可或缺的 `fsck`（文件系统一致性检查工具）——在 ZFS 的设计哲学中，离线扫描磁盘以修复一致性的做法完全没有存在的必要。Cantrill 强调，在大数据时代，强行对数 TB 级别的存储介质执行 `fsck` 纯属非理性的自杀行为。他顺带讽刺了早前 Hacker News 遭遇故障后整整宕机 36 小时来跑 `fsck` 的惨剧。之所以绝不能用 `fsck` 暴力扫盘，根源在于磁盘机械转速在半个世纪里停滞不前的物理硬伤： - 1956 年问世的 IBM RAMAC 机械转速为 1200 RPM； - 历经近六十年演进，主流机械硬盘的转速仅徘徊在 7200 RPM，哪怕是企业级的高速盘也仅达 10,000 RPM 或 15,000 RPM； - 存储容量膨胀了数百万倍，但随机读取的 I/O 速度仅仅提升了约 4 倍。面对海量数据，系统根本没有吞吐预算去把整个磁盘读一遍来验证元数据。但现实往往更加残酷：底层的存储硬件常常是“病态的谎言制造者”（pathologically lying devices）。由于软件的一致性最终依赖于硬件宣称的写入成功信号，一旦硬盘控制器阳奉阴违，系统就会万劫不复。为此，ZFS 在后续演进中加入了按事务回滚存储池（Pool Rollback）的机制：如果遇到某些固件严重故障、平时表现得像“敌特破坏分子”但偶尔还能存对数据的劣质硬盘（Cantrill 此处点名调侃了西数绿盘 Western Digital Green），ZFS 能够自动一步步倒退事务，直到回溯到硬件尚未撒谎的历史安全状态。 > “IBM RAMAC is a 1200 RPM device most spindles today are 7200 RPM devices... it's like 4X faster than it was in 1956 that is that should be terrifying to you and that what that means is you can't go back and just read all your data to check the consistency of your file system that should be something that does not exist it doesn't exist in ZFS.” —— Bryan Cantrill --- ### [16:11](https://www.youtube.com/watch?v=79fvDDPaIoY&t=971s) 间接校验和、静默损坏自愈与不可妥协的工程完整性由于 ZFS 统管从文件系统到磁盘块的整个协议栈，它实现了一项跨层架构绝对无法达成的杀手锏特性：**间接块级校验和（Indirect Block Checksumming）**。ZFS 不会在数据块内部保存其本身的校验和，而是将该数据块的校验和存放在指向它的父级元数据指针块中（Parent Block Pointer）。这种设计直接粉碎了底层固件读取错位所造成的“自洽假象”。一旦底层硬件因为固件 Bug 读错了扇区，ZFS 拿父节点预期的校验和与读出的块进行比对，便能当场抓包：“嘿，固件！你给错块了，休想用假数据糊弄我！” ```mermaid flowchart TD accTitle: ZFS 间接校验和树形验证机制 accDescr: 展示父级指针块中存储子块校验和的结构，以此规避硬件错位读取导致的静默损坏。 subgraph Parent["父级元数据节点 (间接块 / Inode)"] P_Block["数据块指针 A [物理地址: 块 #63] [预期校验和: 0x9AF4...]"] end subgraph Storage["底层物理磁盘介质"] D_63["块 #63 (真实数据, 哈希值 0x9AF4...)"] D_36["块 #36 (固件误读的幻读块, 哈希值 0x11B2...)"] end Parent -->|"定向读取"| D_63 Parent -.->|"固件故障误读"| D_36 subgraph Logic["ZFS 验证逻辑"] V{"读入内容校验和 == 预期校验和 (0x9AF4...)?"} Fix["从镜像镜像副本读取正确数据并自动修复底层损坏块"] Panic["阻断错误扩散并触发报警"] end D_36 --> Logic V -- "不匹配 (发现固件幻读)" --> Panic Panic --> Fix ``` 在配置了镜像或 RAID-Z 的环境中，ZFS 的自愈能力（Self-healing）会立即介入：它直接从完好的镜像副本中读取正确数据，并在静默修复损坏副本的同时无缝返回正确数据给上层调用者。Cantrill 回忆起 ZFS 早期推广时的一段戏剧性往事：系统频繁报出底层校验和异常，但同时骄傲地宣称“别担心，数据已被我成功纠正”。彼时工程师和客户的第一反应都是质疑 ZFS：“系统运行在出货量高达 600 万套的顶级控制器或昂贵 SAN 阵列上，硬件怎么可能出错？一定是 ZFS 算错了！” 然而事实往往无情地击碎盲目迷信。经过深度排查，他们发现那款出货数百万的工业级磁盘控制器芯片中存在一个致命的 DMA 硅片硬件缺陷（Silicon Bug）：在特定的可复现边界条件下，该 DMA 引擎在传输数据块时，会毫无理由地丢弃最后的 64 个字节。Cantrill 感慨，天知道在 ZFS 揭露这一真相之前，世界上有多少医疗生命维持系统因为关键数据末尾丢失 64 字节而莫名停摆。许多客户砸重金购买企业级存储阵列，认为“花了大价钱就不可能出错”，但残酷的底层现实是：分布式系统与硬件极难写对。在结语部分，Cantrill 强调了严肃系统工程的核心操守：任何以安全性与完整性为立身之本的系统，在其生命周期内决不能经历一段“可以容忍丢数据”的幼年期。他犀利地抨击了开源软件界某种将严重缺陷轻描淡写为成长阵痛的不良风气——有些系统动辄以“虽然早期版本会丢数据，但只要有足够多的眼球盯着，代码终究会变好”作为推脱责任的借口。对 ZFS 而言，数据完整性从第一行代码开始，就是不可妥协的底线。 > “One thing I would say by the way and this is true this should be true for any system that is designed around safety and integrity: ZFS never went through an era where it was unsafe or didn't have integrity or like people lost data.” —— Bryan Cantrill ### [18:56](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1136s) 存储系统的可靠性与 Fishworks 的“苦难历程” 布莱恩·坎特里尔（Bryan Cantrill）以极具讽刺意味的生动比喻驳斥了某些文件系统“虽会丢失数据但一直在变好”的论调。他指出，这就像面对一个年仅 12 岁却刚刚持枪抢劫了酒行、接着又在拔枪威胁下偷车的小孩，你绝不能轻描淡写地说“哎呀，他才 12 岁，等他 18 岁懂事了再联系我吧”，因为你很清楚这种根子上的问题到了 18 岁只会变成灾难。像 Btrfs 这种经常丢数据、甚至在架构层面上就存在可靠性隐患的系统，绝非靠修修补补就能解决问题，而 ZFS 在架构设计之初便把数据绝对可靠放在首位。回溯个人经历，布莱恩谈到了 ZFS 在 Sun Microsystems 的演进史。2006 年，他和一群同事决定基于 ZFS 打造一款真正面向企业的统一存储产品（内部代号 Fishworks，即后来的 Sun Storage 7000 系列）。他半开玩笑地自嘲道：“我们不幸地成功了”，其最早、最重要的大客户之一恰恰是那家 Sun 已经很久没能卖进硬件的知名数据库巨头——甲骨文（Oracle）。甚至在某种程度上，正是这款优秀产品的诞生，让外界突然意识到“Sun 又能做出有趣的东西了”，间接为后续的收购埋下了伏笔，也让他时常产生某种荒诞的“幸存者负罪感”。然而，踏入企业级存储领域的艰难程度完全超出了最初的预估。坎特里尔感叹道，在 2006 年自以为做好了充分准备就去研发存储硬件，无异于直接跑去阿富汗徒步旅行，“如果你不觉得自己是在踏入战区，那你纯粹是痴人说梦”。如今人们用到的 ZFS 之所以坚不可摧，是因为它在企业存储的残酷战场上历经了多次九死一生的洗礼。 > “在 2006 年我们着手做这件事时，打造一款存储产品就像去阿富汗背包旅行，如果你不觉得自己是踏入了一个交火战区，那你纯粹是痴人说梦。”（“Building a storage product is like going backpacking in Afghanistan, and if you don't think you're going to a war zone, you're delusional.” —— Bryan Cantrill） --- ### [21:08](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1268s) 拜占庭固件故障与 ZFS 的“漫长假期” 在研发存储产品的过程中，工程团队深刻领教了底层硬件损坏方式的多样与诡异。原本大家以为只要防御虚假读写（phantom reads/writes）或静默数据损坏（bit rot）就足够了，但硬盘固件展现出的“创造力”和拜占庭式故障（Byzantine failures）彻底粉碎了这种天真想法。坎特里尔现场回忆起一款至今让他心有余悸的希捷（Seagate）s0d 固件 Bug：该固件在磁头极性控制上出现代码缺陷，当寻道到达特定逻辑块地址（LBA）时，磁头不但不减速，反而加速撞向柱面；硬盘主板上的 JTAG 监测电路检测到异常后立刻执行紧急保护直接断电重启。整个重启上电重新提供 IO 服务的过程恰好耗时 650 毫秒，导致系统监控中不断冒出 650 毫秒的怪异 IO 离群值。这种偶发性能衰退对当时的文件系统构成了致命打击。在真实生产环境中，支撑数十万块主轴硬盘运行的成熟系统与普通系统的核心分水岭，就在于能否优雅处理这类非死即活的灰度硬件故障。由于极性竞争条件只是偶发，每次 IO 最终依然能够成功完成，ZFS 便不会将盘剔除，而是被拖入漫长卡顿；再叠加当时 ZFS 在大快照删除时会在事务上下文内发起海量随机读，整个存储阵列直接陷入瘫痪。坎特里尔笑称这就像“灵犬莱西（Lassie）离家出走度长假”：系统虽未丢掉任何字节，但事务组（txg）刷盘可能要等到几天之后。曾有客户用 DTrace 算完进度后无奈表示“等它刷完得周四晚八点，反正这里面的数据也不要了，直接扔掉重来吧”。 ```mermaid flowchart TD accTitle: Seagate Firmware Bug IO Stalling accDescr: 希捷固件寻道反冲导致磁盘不断重启与 ZFS 卡顿的处理流程 subgraph Hardware["磁盘硬件异常"] A["寻道至特定 LBA"] --> B["磁头极性错误: 异常加速冲撞"] B --> C["硬件过载保护触发: 硬盘断电重启"] C --> D["重启耗时约 650ms 重新上线"] end subgraph ZFS["ZFS 层面反应"] D --> E["出现 650ms IO 离群高延迟"] E --> F{"IO 是否最终成功?"} F -->|"是 (竞争条件偶尔通过)"| G["系统不标记故障盘"] G --> H["大快照删除/事务组同步陷入死锁般漫长等待"] end ``` > “当时在 Fishworks，我们常说我们从来没有丢失过任何人的数据，但它确实经常跑去度一些非常、非常、非常漫长的长假。”（“The as I said at Fishworks, we never lost anyone's data, but it went on some very, very, very long vacations.” —— Bryan Cantrill） --- ### [24:58](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1498s) 甲骨文收购、被审查的演讲与转战 Joyent 伴随甲骨文闪电式收购 Sun Microsystems，坎特里尔经历了极大的心理落差。2010 年入职甲骨文之初，他还天真地幻想“甲骨文懂商业，Sun 懂技术，两者结合定能成就一番伟业”；但现实迅速将他打醒，甲骨文骨子里是一家不折不扣的垄断商业公司。他自嘲如今如果有时间机器，第一件事就是穿梭回 2010 年狠狠给自己一记耳光。最终他选择离职，并在开源社区掀起了名为“Fork illumos”的行动。在一次由甲骨文赞助的 USENIX 大会上，坎特里尔发表了一场火药味十足的无审查演讲。尽管他清楚“甲骨文是个反社会人格者（sociopath），根本不在乎任何人对它的评价”，但惊慌失措的会议组织方却在视频发布时将他的专场雪藏了数周。最后视频虽被放出，但主办方在上方硬生生压上了一条硕大的黑底免责声明：“以下观点不代表 USENIX 协会立场”，并在整个演讲过程中全程悬挂在坎特里尔的头顶上方，造就了开源界一段广为流传的奇景。离开 Sun 体系后，坎特里尔与多位核心工程师加盟了 Joyent。早在 2005 年，Joyent 便是 Sun 之外全网第一家在生产环境中重度使用 ZFS 的公有云先驱。作为一家以 ZFS 和容器技术为核心竞争力的基础设施即服务（IaaS）云厂商，Joyent 既运营公有云，也向客户输出私有云整套软件栈。 > “如果真发明了时间机器，我想做的第一件事就是穿梭回去，狠狠给 2010 年左右的自己一拳。”（“If we invent a time machine, the first thing I want to do is go back to myself and just deck myself circa 2010.” —— Bryan Cantrill） - **illumos 创立背景**：反抗甲骨文封闭 Solaris 源码，全面分叉并建立独立维护的开源操作系统及 ZFS 上游。 - **USENIX 视频事件**：所有演讲视频同期上线，唯独坎特里尔的演讲被扣留两周并被打上全屏常驻的免责黑条。 - **Joyent 与 ZFS 的渊源**：2005 年即成为 Sun 外部首个 ZFS 商业生产环境采纳者，双方创始人团队很早便保持深度协作。 --- ### [28:15](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1695s) 为什么对象存储（S3）必须摒弃传统文件系统抽象在完善了以 ZFS 为基础的计算与虚拟化基础设施后，Joyent 意识到自身产品矩阵中唯一缺失的关键拼图是类似于 AWS S3 的对象存储能力。提及 AWS，坎特里尔给予了高度评价，指出由 Amazon 推出的 S3（Simple Storage Service）是云计算历史上真正意义上的第一个云服务，其本质是一个基于 HTTP 协议的存储系统。更重要的是，S3 在底层架构上做出了一个极具洞见的关键决策。坎特里尔坦承，换作当时包括他自己在内的大多数传统存储工程师，很可能会图省事直接拿一套既有的文件系统抽象（如 POSIX 层次目录与块语义），简单封装上一层 HTTP 接口对外提供服务。然而亚马逊没有掉入这个陷阱：S3 坚决定义为了“对象存储”（Object Store）而非“文件存储”（File Store）。在 S3 的语义规范中，彻底剔除了对对象的“部分写入”（Partial Writes）支持——对象要么整体创建写入，要么整体覆盖替换。这一对传统文件系统语义的彻底割裂与简化，成为了构建高伸缩性、分布式强一致性海量存储系统的真正基石。 > “S3 并没有像坦白讲我以及当时很多其他人可能会做的那样，把现有的类文件系统抽象直接套上一层 HTTP 对外提供；他们根本没有这么干。”（“S3 did not do what frankly I would have done and a lot of other people would have done at the time, which would be to take an existing file-like abstraction and make it available over HTTP; that's not what they did.” —— Bryan Cantrill） ### [29:16](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1756s) 对象存储的取舍与 Joyent 面临的“S3 空白” Bryan Cantrill 回顾了对象存储为何能够成立的核心原因。传统的文件系统直觉通常认为“不允许局部写入（partial writes）是绝对无法被用户接受的”，但现实证明这种限制完全行得通。互联网上存在海量的“单写多读”（Write Once, Read Many，简称 WORM）数据，无论是日志文件、静态 Web 资产还是系统备份，用户绝不会去修改一个日志中间的几行，也不会去篡改一个备份归档的中间字节，甚至极其排斥这类操作。因此，对象存储将数据视为不可变的整体，不仅没有降低价值，反而彻底改变了分布式系统的设计约束，让构建高可扩展的系统（如 Amazon S3）成为可能。这种设计约束的解除衬托出 Joyent 当时的窘境：Joyent 旗下一直缺乏类似 S3 的对象存储服务。Bryan 直言，这是他们整个产品体系中最刺眼、最无法回避的巨大缺口。当时他刚从阿富汗背包徒步归来，脑子里满是 ZFS，极度渴望用 ZFS 或者 C 语言打造出一套真正有趣的分布式对象存储底座。然而，他苦思冥想后发现自己构思出的所有方案要么行不通，要么“显而易见地糟糕透顶”，不得不暂时搁置这个方向。 > “The fact that it was object storage was actually that was valuable, and so that was actually extremely important because that changed the constraints of the system, allowed them to build the distributed system.” —— Bryan Cantrill --- ### [30:27](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1827s) 云计算的基石与硬件虚拟化的起源：IBM 360 与 1401 在暂时搁下 ZFS 之后，Bryan 将话题转向了云计算的灵魂支柱——虚拟化。虚拟化的本质是实现多租户隔离（Multi-tenancy），在单台物理机上为多个用户营造出“独占整台机器”的幻觉，并确保各自的安全边界。技术选型的关键在于：在整个软件/硬件栈的哪一层制造这种抽象？追溯历史，硬件级虚拟化（Hardware-based virtualization）并非现代产物，它早在 20 世纪 60 年代伴随 IBM System/360 的诞生便已成型。 ```mermaid flowchart TD accTitle: 硬件虚拟化的诞生历史 accDescr: 展示从 IBM 1401 到 Honeywell 兼容机竞争，再到 IBM 360 软件模拟虚拟化的演进逻辑 subgraph Legacy["传统商业计算时代"] A["IBM 1401 记账计算机 (1959-1963，硬件固定双位年份)"] end subgraph Transition["市场竞争冲突"] B["IBM System/360 发布 (统合商业与科研架构，放弃直接兼容)"] C["Honeywell H-200 推出 (逐比特兼容 1401，热销抢占市场)"] end subgraph Virtualization["虚拟化谎言的诞生"] D["IBM 纯软件模拟器 (在 360 上骗过 1401 软件，硬件虚拟化成型)"] end A -->|"IBM 要求客户重写代码"| B A -->|"市场拒绝重写"| C C -->|"威胁 IBM 核心业务"| D B -.->|"在 360 之上构建"| D ``` IBM 360 是计算机历史上最重要的里程碑之一，它首次将原本割裂的商业计算与科学计算产品线统一到同一套系统架构中。然而，这种革新抛弃了早期的 IBM 1401。1401 本质上是一台处于早期计算蒙昧时期的记账机，采用存储程序结构，完全不适合科学计算。60 年代 IBM 的工程师高傲地认为“硬件才是攻坚难关，软件随时可以找人重写”，因此 360 完全不支持运行 1401 的程序。但这遭到了客户的强烈抵制，老客户根本不愿承担重写海量软件的巨大成本。IBM 的老对手 Honeywell 敏锐捕捉到战机，顺势推出了与 IBM 1401 逐比特兼容（bit-for-bit compatible）的 Honeywell H-200，并以此引发销售狂潮，直接蚕食了 1401 和 1410 的替换市场。面对市场份额的流失，IBM 迅速改变了策略，再次启用了计算机行业经久不衰的核心伎俩——“撒谎”。IBM 没有去改动 360 的芯片，而是在其上用纯软件构建了一整套 1401 的虚拟硬件层。这套软件构造如此逼真，以至于跑在上面的 1401 原生程序完全不知道自己运行在 System/360 上。Bryan 指出，这种“向操作系统内核撒谎、让它深信自己独占物理硬件”的模式，从 60 年代一直延续到了今天的 AWS EC2 虚拟机中。 > “We are going to create an IBM 1401 strictly out of software, it's going to be a pure software construct, and it will be so persuasive and convincing that the 1401 software running on top of it will have no idea that it is in fact not running on a 1401 but a 360.” —— Bryan Cantrill - **IBM 1401 的定位**：50 年代末至 60 年代初的商业记账计算机，软件深度依赖硬件特性。 - **Honeywell H-200**：因实现对 1401 的二进制兼容，在市场上极速抢夺 IBM 客户。 - **虚拟化谎言的雏形**：通过纯软件模拟硬件指令集与外设行为，瞒过旧系统代码。 --- ### [33:52](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2032s) 永生不灭的软件与千年虫轶事 Bryan 分享了一段发生于 2000 年前后的亲身经历，以此论证软件惊人的生命力。当时他在 Sun Microsystems 供职，千年虫（Y2K）危机刚刚平稳度过。事实证明，Sun 内部大约 30% 所谓的 Y2K 问题实际上是闰年计算错误——许多程序员虽然记住了“四年一闰”和“百年不闰”，却遗漏了“四百年又闰”的第三条规则，导致无法被 100 整除但能被 400 整除的 2000 年在软件中判定出错。在纽约与一家大型银行的首席信息官（CIO）闲聊时，该 CIO 透露了一个震撼事实：他们银行遭遇的 Y2K 问题中，竟然有高达 45% 的绝大多数故障源自同一个硬件平台。当 Bryan 急切追问是哪款设备时，对方戏剧性地揭晓了答案——正是那台早在 1963 年就已经停产的 IBM 1401。 | 历史计算平台 | 运行状态与生命周期 | Y2K 表现及核心成因 | |:--- |:--- |:--- | | **IBM 1401** | 1963 年停产，通过虚拟机套虚拟机层层转译，在 2000 年仍支撑银行辅助系统运转 | 贡献了该银行 45% 的故障；双位年份被直接硬编码固化在原始硬件设计中，无法通过兼容层消除 | | **现代虚拟化实例 (如 AWS)** | 运行虚拟化内核，假定底层硬件独占 | 沿用了同一设计哲学：向客户机操作系统掩盖底层的真实虚拟环境 | 尽管这些 1401 软件并未运行在核心交易结算流程，只支撑着边缘辅助功能，但这一事实依旧让 Bryan 目瞪口呆。软件的寿命远远超过了最初设计它的硬件，甚至熬死了替代硬件以及替代硬件的下一代硬件。历经多重虚拟化包装，这些诞生于 50 年代末或 60 年代的代码依然活跃在生产环境。然而，由于 1401 的硬件体系把两位年份数字直接固化在了硬件逻辑中，虚拟化层也无能为力，面对 Y2K 最终的解决手段只能是彻底重写。 > “Here you have software that is written in the late 50s, early 60s... and yet it has survived the death of not only that hardware, but the hardware that replaced it, and probably the hardware that replaced it.” —— Bryan Cantrill --- ### [36:39](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2199s) 硬件虚拟化的破局与资源瓶颈硬件虚拟化虽然能够近乎完美地兼容并运行历史技术栈，但当应用场景拓展到现代云计算的高密度、高性能诉求时，“谎言”便开始出现裂痕。硬件虚拟化的抽象语义极其窄小死板。以内存为例，如果向虚拟机承诺了 1GB 的 DRAM，就必须切切实实分配出 1GB 的物理内存。操作系统内核拥有绝对的控制预期，既不可能突然在运行时冒出“一根之前没见过的内存条”，也不可能容忍内存凭空被削减。因此，一台宿主机能够承载的虚拟机数量，在工程上几乎严格受限于“物理 DRAM 总量除以单个虚机规格”。虽然不合规的服务商可以通过内存超卖（overselling）来空手套白狼，甚至赌在崩盘前将业务转手套现，但这在正规工程实践中根本无法长久。类似的问题在 CPU 和网络 I/O 调度中同样严峻。在网络 I/O 方面，纯粹的硬件模拟几乎无法提供高吞吐、低延迟，迫使业界不得不采用网卡直通等穿透手段将硬件直接暴露给客户机。然而，一旦引入直通技术，虚拟机便重新与具体物理硬件绑定，原本为了多租户隔离与动态调度而构筑的“虚拟化谎言”也就荡然无存。 > “The lie works great at first, but then you begin to pull things and the lie begins to snap... When you are virtualizing the hardware, the semantics that you have are very narrow.” —— Bryan Cantrill - **DRAM 的强语义约束**：物理内存无法像无状态数据那样随意超售，内核对分配的物理地址空间有严格假设。 - **虚拟机密度瓶颈**：单台宿主机的并发承载上限被硬件资源（DRAM/CPU）硬性除法锁死。 - **I/O 虚拟化的悖论**：追求原生性能就必须绕过模拟层（如硬件直通），而一旦放弃虚拟化层模拟，虚拟化原本带来的抽象隔离与灵活性优势便随之丧失。 ### [038:41] 操作系统级虚拟化与容器的真正优势在 Joyent 公共云中，除了通过 KVM 提供基于硬件的虚拟化之外，还深度采用了操作系统级虚拟化（OS-level virtualization）。Bryan Cantrill 首先反驳了平台级虚拟化（PaaS），直言其“行不通”。他指出，Google App Engine 等平台将抽象层做得过高，导致开发者无法在其上构建大型应用；一旦业务规模变大，就不得不整体搬迁，而绝大多数 PaaS 企业在用户长大前自己就已倒闭，这是一种极其糟糕的商业模式。 ```mermaid flowchart TD accTitle: 硬件虚拟化与操作系统级虚拟化的抽象层对比 accDescr: 对比KVM硬件虚拟化与SmartOS容器在抽象层级和资源访问上的不同路径。 subgraph Hardware_Virtualization["传统硬件虚拟化 (KVM)"] HW1["裸机硬件"] --> KVM["KVM 虚拟机监控器"] KVM --> GUEST_OS["完整 Guest OS 实例"] GUEST_OS --> APP1["应用程序"] end subgraph OS_Virtualization["操作系统级虚拟化 (SmartOS Zones)"] HW2["裸机硬件"] --> KERNEL["单一 OS 内核"] KERNEL -->|"系统调用表 (Syscall Table)"| C1["Zone / 容器 1 (DRAM 上限控制)"] KERNEL -->|"系统调用表 (Syscall Table)"| C2["Zone / 容器 2 (可动态调容)"] C1 --> APP2["应用程序"] C2 --> APP3["应用程序"] end ``` 相比之下，操作系统级虚拟化由单个操作系统内核直接承载在硬件上，并在其上隔离出多个容器。这些容器在外观、体验和操作上与独立的操作系统无异，甚至能够单独重启，但向上传递的抽象接口并非模拟的硬件设备，而是底层的“系统调用表”（system call table）。容器进程直接运行在物理硬件上，能直接看到宿主机的所有 CPU 和全部 DRAM。对容器内存的限制，仅仅是在内核层记下一笔配额并在超限时切断分配；若用户需要动态扩缩容，管理员无需关机迁移，只要未超卖物理机资源，几秒内便能将 DRAM 配额随意调大或调小。对于市场上热度攀升的 Docker 和 LXC，Bryan Cantrill 表现出了认可与调侃并存的态度。他认为这印证了操作系统级虚拟化路线的正确性： > “People ask like, 'How do you guys feel about like Docker and LXC and so on?' I actually... to me it is very validating to see that, because I think what I see other people seeing in Docker and LXC is the power of the model, and I think the model is the right model.” - SmartOS 拥有经过多年严苛检验的 Zones 实现，具备极高的稳定性和安全性。 - 硬件抽象与系统调用抽象的区别，决定了容器能够以极低的开销获得极致的原生性能与动态资源调配能力。 - Bryan 戏谑地表示，开发者当然可以先在 LXC 上部署，但如果想要“零安全漏洞”，最终还是应当迁移到 SmartOS Zones 上。 ### [041:04] Mark Cavage 的关键灵感：计算向存储靠拢与 Manta 的诞生在筹备构建新型对象存储系统时，Bryan Cantrill 曾一度执迷于 ZFS 本身的各种存储特性。直到他招募了前 AWS 资深工程师 Mark Cavage，后者的一个突破性洞察打破了僵局：系统的核心并不是 ZFS，而是操作系统级虚拟化。传统以 S3 为代表的 HTTP 对象存储只能进行简单的 `PUT` 与 `GET`，若要对数据进行诸如 `grep` 的过滤或处理，用户必须先将海量数据通过网络拉取回本地，再执行计算；而操作系统级虚拟化允许平台在对象数据实际存放的物理节点上，原地瞬间启动一个轻量级的 Zone 容器，在数据落盘的位置直接发起计算，彻底免去庞大对象在网络间的来回搬运。 ```mermaid flowchart LR accTitle: 传统对象存储计算模式与 Manta 原地计算模式对比 accDescr: 展示传统拉取数据模式与Manta在数据本地拉起Zone容器就地计算的区别。 subgraph Traditional["传统方式 (如 S3)"] STORE1[("存储节点\nObject A")] -->|"GET 跨网传输"| COMP1["计算实例\n(grep / 过滤)"] end subgraph Manta_Way["Manta 就地计算"] STORE2[("存储节点\n本地 ZFS 存储")] ZONE["本地瞬间拉起 Zone 容器\n就地执行管道命令"] STORE2 <-->|"零网络开销\n直接读写"| ZONE end ``` 这个看似简单却威力无穷的想法让 Bryan 大为震撼。他感叹许多重大的架构突破在事后看来都显得无比简明，但在被提出之前却难以参透。他清楚记得自己 1999 年在昏暗的办公室里第一次听 Jeff Bonwick 阐述 ZFS，也记得在 Sun 面试时向 Jeff 首次描述 DTrace 的场景，却一度想不起 Mark 是在哪里向他提出 Manta 原型的。两人甚至一度误以为是在某家酒吧里的闲谈，直到翻查内部即时通讯（IM）日志，才找到了脑力爆发的确切时刻——2011 年 10 月 24 日。 - IM 记录证明，Mark Cavage 提出了通过将 MapReduce 降级为一等公民支持、将 JS 代码路由到与数据同节点的独立 Zone 中执行的狂想。 - Bryan 当时在聊天窗口中直接用带有粗口但极具赞赏意味的黑客黑话回应对方，感叹自己为何没能早点想到这一点。 - 架构构想确定后，Joyent 团队在整个 2012 年全力投入工程落地，因为“光靠愿望无法让系统自动实现”，在底层 ZFS 和 Zones 之上，仍需构建一套极其复杂的分布式调度与元数据系统。 ### [044:50] Unix 管道哲学与 Doug McIlroy 的遗产在推进 Manta 项目的过程中，团队意识到他们实际上是在分布式规模上深度复用经典的 Unix 工具链来执行任意的即席数据处理（ad hoc data processing）。作为自称的“Unix 教会追随者”（acolyte in the Church of Unix），Bryan Cantrill 梳理了这段技术谱系，提醒听众 Linux 并没有发明这些核心抽象，它们早在 1970 年代便已随着 Unix 确立，而 Unix 本身则是对 MULTICS 系统的反思与精简改良（甚至连路径分隔符正斜杠 `/` 也是由 MULTICS 引入的）。当贝尔实验室的 GE 635 机器被收回、失去 MULTICS 环境后，Ken Thompson 和 Dennis Ritchie 等人由于对大系统思想感到压抑，转而在极为有限的小型硬件上从头构建简洁系统。Unix 最初编写的很多程序（如众多今天仍被开发者日常键入的基础命令），最初仅仅是用来验证文件系统是否可用的测试代码，现代开发者每天都在享受着出生前便已写就的测试驱动开发成果。 Bryan 强调，世人在提及 Unix 创造者时往往只推崇 Ken Thompson 和 Dennis Ritchie，但必须将 Doug McIlroy 同样置于万神殿之中。正是 McIlroy 提出了现代 Unix 最核心的灵魂——将一个命令的输出通过管道（pipe）直接输送给另一个命令的输入： > “Talk about like the simplicity sometimes not being obvious: this was obvious to McIlroy in 1969, this is two years before Unix... It took him three years to persuade Thompson to implement it.” - 管道机制最早在 1969 年被 Doug McIlroy 构思出来，当时他将其形象地称为“软管”（hoses）。 - 尽管两人的办公室紧挨着，McIlroy 几乎每天都在游说，但让 Thompson 接受并实现这一特性花费了整整三年时间。 - 最终的突破在于 McIlroy 设计出了优雅的语法形式；而最初用于表示管道重定向的字符并非现代标准的竖线 `|`，而是插入符（caret `^`）。 ### [48:11](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2891s) 管道的诞生与 Unix 哲学管道符号（从早期的抑扬符 `^` 发展而来）出现后，Ken Thompson 曾表示“我这就去实现它”。果不其然，他只用了一个晚上就写了出来。这正体现了软件行业最经典的一幕：某人花了三年时间告诉你某件事在技术上完全不可能，然后他用一天时间就搞定了。Bryan Cantrill 调侃道：“‘技术上不可能’的意思其实是‘我真的、真的、真的不想干’。” 随后发生的事情被 Bryan Cantrill 比作软件界的《创世纪》——犹如神说“要有光”，第二天清晨 Unix 世界就迎来了单行命令（one-liner）的狂欢。单行命令之所以极为关键，是因为它能将庞大的语法威力浓缩在一行之中，让人一目了然；任何能激发出海量单行命令的系统，必然抓住了某种核心本质。就在那个清晨，Unix 真正诞生了，并催生了由 Doug McIlroy 总结的 Unix 哲学：编写只做一件事并做好它的程序；编写能够协同工作的程序；编写处理文本流的程序，因为那是通用的接口。 ```mermaid flowchart LR accTitle: Unix 哲学的核心链路 accDescr: 展现小工具通过标准文本流组装成复杂系统的 Unix 核心设计理念。 subgraph Tools["单一职责工具"] T1["工具 A 只做好一件事"] T2["工具 B 只做好一件事"] end subgraph Interface["通用协议"] Stream["文本流 (Text Stream)"] end subgraph Pipeline["复合能力"] Result["高内聚单行命令 (One-liner)"] end T1 --> Stream Stream --> T2 T2 --> Result ``` Bryan Cantrill 强调，这是系统开发中最重要的哲学，构建复杂系统的唯一正道就是用简单系统相互连接。 > “The way we are able to develop complicated systems is by building out of simple ones, taking simple systems and actually connecting them in interesting and new ways, and the way you're able to do that is by having programs that do one thing and they do it well.” —— Bryan Cantrill - **单行命令的力量**：单行命令不仅是语法的紧凑表达，更是系统模块化与可组合性的直接证明。 - **Doug McIlroy 的三原则**：单一职责（Do one thing and do it well）、协同工作（Work together）、面向文本流（Handle text streams）。 --- ### [50:08](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3008s) 库与框架的“圣战”：计算机史上的 Epic Rap Battle 随着年龄增长，Bryan Cantrill 愈发将软件设计看作一场“复杂势力与简单势力”之间的圣战。这场冲突具体体现为“以框架为中心（Framework-Centric）”与“以库为中心（Library-Centric）”的世界观差异。库的世界观提供工具箱，让开发者自由取用组合；而框架的世界观则将开发者强行置入一个诡异的上下文，让人仿佛掉进黑洞醒来、悬浮在虚空中，完全无法直观推演系统的运行逻辑。虽然框架有其存在价值，但只要能按 Unix 哲学去构建，就永远应当选择简单组合的方式。为了阐述这种哲学对抗，Bryan Cantrill 提到了 1986 年 Jon Bentley 在《编程珠玑》（Programming Pearls）中提出的经典挑战：读取文本文件，统计出现频率最高的 N 个词，并按频率递降排序打印。这个在今天看来如同实习生面试题的任务，在当时引发了计算机科学史上堪比“Epic Rap Battle”的巅峰对决： 1. **Donald Knuth 的解答**：伟大的图灵奖得主 Donald Knuth 采用其推崇的“文学编程（Literate Programming）”范式，在自创的 WEB 系统（基于 Pascal）中写了整整 7 页极其晦涩密集的算法代码。Cantrill 戏称文学编程属于 Knuth 极少数成果中的“臭弹（stinker）”。 2. **Doug McIlroy 的反击**：Doug McIlroy 就像电影《夺宝奇兵》里拔枪击毙挥刀刺客的印第安纳·琼斯一样，仅仅用了一条由 6 个命令组成的 Unix 管道单行命令，就利落地解决了整个问题。 ```shell tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q ``` 在这条命令中，`tr` 用于分词和大小写转换，`sort` 与 `uniq -c` 用于聚合统计，随后的 `sort -rn` 进行逆序排序，最后的 `sed ${1}q` 打印前 N 行（在退出前退出）。令人吃惊的是，在 1986 年两者的性能几乎不相上下，而且 Doug McIlroy 当时之所以用 `sed` 而不是 `head`，仅仅是因为 1986 年 AT&T 的 Unix 源码里压根还没有收录 `head` 命令（当时只有 `tail`）。 > “And then, like Indiana Jones pulling a pistol, Doug McIlroy has this solution... and the amazing thing is that in 1986, the performance was nearly identical.” —— Bryan Cantrill --- ### [54:25](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3265s) 大数据时代的倒退与 Manta 的破局历史在当下的“大数据”潮流中惊人地重演。Bryan Cantrill 指出，Jeffrey Dean 等人关于 MapReduce 的经典论文中，统计 URL 访问频率的示例与 1986 年 Bentley 的挑战如出一辙——Map 端处理请求日志输出 `(URL, 1)`，Reduce 端汇聚并输出 `(URL, total_count)`。然而，现代分布式计算的解决方案彻底背弃了 Unix 哲学，堪称对后者的亵渎：为了做词频统计，附录里塞满了 150 行高度压缩的 C++ 代码；一个经典的 `wc`（word count）功能，在当代程序员的直觉中竟然变成了“我需要一个 C++ 编译器”、“我需要 Java 和 Hadoop”。 Cantrill 质疑道：“为什么在笔记本上统计字数用 `wc` 就够了，到了云端海量日志面前，却突然变成了 C++、Java 和 Hadoop？”为了打破这一荒谬现状，Joyent 开发了 Manta。Manta 允许用户将 Bentley 挑战中的原生 Unix 单行命令，直接复制粘贴到云端海量对象上并发运行。 ```mermaid flowchart TD accTitle: 本地计算与云端大数据的工具链断层对比 accDescr: 对比传统 Unix 的轻量 CLI 方案与当代大数据的重型框架方案。 subgraph Local["本地单机分析"] L1["文本文件"] --> L2["标准 CLI 工具 (tr, sort, uniq, wc)"] L2 --> L3["毫秒级得出统计结果"] end subgraph BigData["现代大数据堆栈 (反 Unix 哲学)"] B1["分布式日志"] --> B2["繁重中间件 (Hadoop / C++ / Java MapReduce)"] B2 --> B3["数百行编译型代码与框架约束"] end subgraph JoyentManta["Manta 原生模式"] M1["云端对象存储"] --> M2["直接将 Unix 管道 推送到数据节点就地执行"] M2 --> M3["极简、可组合的流式结果"] end ``` Cantrill 现场演示了 Manta 的基本操作，并重点阐述了它与 Amazon S3 的根本差异： - **真树状层级存储（True Hierarchical Storage）**：S3 本质上是一个伪装成文件系统的扁平 Key-Value 存储，它接受斜杠 `/` 却不提供真正的目录层级，导致列出父目录内容等操作极度别扭；而 Manta 从 Unix/Multics 继承了严格的树状目录结构。 - **网络交互与就近延迟**：Cantrill 在现场执行 `mls /bcantrill/public` 列出公有目录。针对观众关于这是否正在向远端真实发送请求的疑问，Cantrill 给予了肯定答复。Manta 的集群部署在美国东部（弗吉尼亚州阿什本 Ashburn），在美东现场敲下命令时，由于物理距离极短，请求往返快如火箭。 > “Why is it that when I do a word count on my laptop it's `wc`, but when I do a word count against my logs in the cloud it's all of a sudden it's C++ and it's Java and it's Hadoop? It's just wrong.” —— Bryan Cantrill ### [58:18](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3498s) 命令行交互体验与流式大对象存储 Bryan Cantrill 展示了在 Manta 中通过命令行浏览与检索对象的高速体验。在列出自己的主目录时，他打趣说看到了名为 `man-prawn` 的目录，并调侃“那绝对不能打开”。通过类似 Unix 目录结构的指令，用户可以对远程存储执行 `mls` 或 `mfind`。为了进行演示，Cantrill 之前将 Unix V7（Version 7 Unix）的历史归档全部上传到了公开目录中。他当场执行 `mfind ~~/public/v7`，瞬间列出了目录下的所有对象。 ``` Bryan 的本地 Mac 终端 Manta 分布式对象存储 | | | 1. mget.../man/man4/pe.4 | |----------------------------------------->| | 2. 返回 1970 年代 roff 排版源文件 | |<-----------------------------------------| | | |--[ 管道流入本地 nroff 排版引擎 ] | v 终端直接渲染 1970 年代 PE-11 驱动手册页 ``` 为了从存储中提取特定对象，Cantrill 顺手查找了一个名为 `pe.4`（PE-11 驱动）的古老手册页。他直接在本地终端将 `mget` 的流通过管道接入本地系统的 `nroff` 格式化程序，当场在屏幕上渲染出了这篇来自 1970 年代中期的 Unix 手册页。他惊叹道这简直就是“手册页时间旅行”（man page time machine）。关于对象存储本身，Cantrill 强调了 Manta 区别于 Amazon S3 的核心优势： - **无对象大小硬性限制**：虽然该手册页对象体积很小，但对象也可以是 1 TB 大小。 - **纯流式上传（Streaming）**：避免了 S3 繁琐复杂的 Multi-part Upload 分块逻辑。Cantrill 直言：“If you have to deal with big objects in S3, your life is a living hell because of the multi-part upload business and the way you have to chunk these things in.” - **大对象预分配调度**：用户只需持续向系统灌入字节流；如果单对象大于 5 GB，仅需在请求头中提前告知系统预期大小，以便调度器直接将数据流接入拥有足够连续存储空间的存储节点。 ### [01:00:46](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3646s) 原地计算、交互式调试与 `mlogin` 的诞生 Cantrill 随后演示了如何向 Manta 提交计算作业（Job）。他通过管道将一组检索到的对象路径输入给 `mjob create`，在后台启动针对这些对象的处理流水线。他省去了冗长的 `sed` 清洗命令，直接配合 `head -n 10` 截取输入来提交处理任务。由此引出的一个关键问题是：对于超大文件（例如巨型的 core dump 核心转储文件或操作系统崩溃快照 crash dump），用户并不总是希望只运行一个离线批处理脚本返回 `grep` 结果，而是需要**交互式调试**。Cantrill 抱怨自己既不想通过网络搬运几十上百吉字节的 Dump 文件，又无法忍受黑盒批处理。为此，Joyent 的同事 Josh Clulow 提出了并实现了 `mlogin` 工具。 ```mermaid flowchart TD accTitle: mlogin Interactive Session Architecture accDescr: Diagram illustrating how mlogin places the user shell directly onto the storage node inside an isolated SmartOS container where the target object is mounted locally. subgraph Client["客户端环境"] CLI["用户终端 (mlogin 命令)"] end subgraph StorageNode["Manta 存储节点物理机"] subgraph Zone["隔离的计算 Zone (SmartOS Container)"] Shell["交互式 Shell 会话"] EnvVar["$MANTA_INPUT_FILE 环境变量"] LocFile["本地挂载的目标对象 (只读)"] end ZFS["底座 ZFS 存储池"] end CLI -->|"建立 SSH / 终端连接"| Shell ZFS -->|"本地直接绑定 / 挂载"| LocFile Shell -->|"检查与读写"| LocFile EnvVar -.->|"指向"| LocFile ``` 通过执行 `mlogin` 命令，Manta 会在存有该对象的具体物理存储节点上即时创建一个隔离的轻量级 Zone（容器），并将对象以本地文件形式挂载进该容器。Cantrill 现场登录进目标对象所在的容器，并打印出环境变量 `$MANTA_INPUT_FILE`。在该容器内，他直接运行 `file $MANTA_INPUT_FILE`，工具准确辨认出该对象为原始的 roff/troff 预处理器宏文本。容器内部具有严格的租户隔离，用户只能看到分配给自己的目标对象和专用文件系统，无法窥探存储服务器上的其他数据。 ### [01:03:19](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3799s) SmartOS 用户空间工具链与 John Bentley 词频作业完成在交互式容器内部，Cantrill 分享了一个用户常常产生的有趣误解：有开发者曾兴奋地对他说，感谢 Joyent 终于抛弃了 SmartOS 那套东西，构建了一个彻头彻尾的 Linux 环境。Cantrill 当时困惑不解，后来才意识到，这是因为 Joyent 在 SmartOS 容器的默认 `$PATH` 中把 GNU 工具链放在了最前列。用户在容器里敲 `grep`、`tar`、`awk` 时，调用的全是 GNU grep、GNU tar 和 gawk（位于 `/opt/local/bin`），使得习惯 Linux 的工程师感觉格外自然。而传统的 Unix 系统原生二进制工具则完整保留在 `/usr/bin` 中。Cantrill 在终端顺手列出各种古老的三字母命令（如 `bfs`、`atq`），还回忆起 `at` 守护进程曾经引发的历史闰年 Bug。在此期间，后台运行的 Manta 作业顺利完成。该任务分布式处理了 37 个输入对象，完整运行了图灵奖得主 Donald Knuth 与 John Bentley 探讨过的“经典词频统计挑战”，并输出了词频最高的结果列表。Cantrill 重点指出了分布式原地计算的性能本质： - **开销在调度调度而不在 I/O**：执行耗时绝大部分花费在创建容器和分发任务的系统流程上。 - **数据不动计算动**：“This would not run in a materially different amount of time if these things were hundreds of megabytes or gigabyte a piece because the data is not moving.” 即使输入文件从几千字节扩大到数百兆或数吉字节，运行耗时也不会有实质性差异，因为海量数据本身不需要在网络中传输。 ### [01:05:53](https://www.youtube.com/watch?v=79fvDDPaIoY&t=3953s) 生产级崩溃分析系统 Thoth 与 Manta 计费模式 Cantrill 自嘲为一个“软件灾难狂热者”（software disaster fetishist），他阐述了 Joyent 内部如何借助 Manta 从根本上重塑软件交付与缺陷排查方式。他们构建了名为 **Thoth** 的崩溃转储分析平台。Thoth 会持续抓取公有云基础设施中发生的所有 core dump 和 crash dump，并集中存入 Manta。借助 `mlogin` 和自动化脚本，工程师能够长期、永久地对线上崩溃现场进行回溯分析。在此过程中发生了一段有趣的插曲：Cantrill 最初计划无差别捕获公有云多租户环境中的所有崩溃，他将公有云环境比喻为“通用监狱犯人区”（general prison population）。但该机制刚一上线，系统就几乎被 PHP 进程的 core dump 冲垮了。Cantrill 吐槽道：“PHP was basically DoSing my system, because that thing core dumps 24/7 365. That is like a disk stress test for core dumps.” 随后他们配置了捕获白名单，重点监控公有云租户环境中的 Ruby、Java 以及 Erlang 运行时崩溃，同时全量保留操作系统内核崩溃转储。在演讲尾声，Cantrill 介绍了 Manta 的计费模型。虽然他半开玩笑地表示如果由他决定他宁愿免费开放，但最终制定的价格策略极具竞争力： - **存储定价**：直接对齐 Amazon S3 的价格标准，保持极其低廉的单位存储费率。 - **计算定价**：在计算单价上与 Amazon EC2 保持同等水平，但计费粒度精确到**秒**，用户只需为真正消耗 CPU 周期进行数据处理的实际秒数付费，无需为闲置虚拟机承担长期租赁开销。 ### [01:07:54](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4074s) 秒级计费与云计算的“按小时收费”之怪相 Bryan Cantrill 详细拆解了 Manta 的计费粒度与底层商业逻辑。在 Manta 中，计算是严格按秒计费的：如果一个任务运行 10 秒，用户就只支付 10 秒的算力费用，单价大约是每秒 40 微美分（40 micro-cents per second）。虽然单价看似微乎其微，但规模效应非常惊人——如果用户有 100 万个对象，每个对象处理半小时，累计的计算时间瞬间就会飙升到以“年”为单位。当被现场听众问及这种计费精度是专属于 Manta 还是整个 Joyent 时，Cantrill 解释道，Joyent 的传统 IaaS（基础设施即服务）通常向上取整到最近的小时（大约折合每小时 14 美分）。 Cantrill 顺势调侃了 Google 当时大肆宣传的更细粒度计费。他认为对 Manta 这种具有瞬态特性的服务来说，按秒计费非常自然且必须；但如果传统的云虚拟机厂商以此为核心卖点，逻辑就显得颇为荒诞。Cantrill 打趣道：“这有点像一家酒店在大肆宣传他们是按小时收费的——出于好奇，究竟什么样的客户会被这种按小时收费的定价吸引呢？”虽然台下有人打趣接话，但 Cantrill 指出，Joyent 在 IaaS 层面完全有技术能力做按秒计费，只是没看到真正的市场需求；他甚至开玩笑说，把 Joyent 公有云上所有客户四舍五入抹掉的差价全加起来，那点钱他自己口袋里大概就能掏出来。 - Manta 计算费用低至每秒 40 微美分（40 micro-cents），严格根据作业实际运行的秒数结算。 - 任务规模呈爆发式增长：100 万个对象各运行 30 分钟，总计算时长即可达数个计算年。 - Joyent 的 IaaS 云主机采用按小时向上取整（约 14 美分），未推行秒级计费纯粹是由于市场需求不足，而非技术限制。 --- ### [01:09:32](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4172s) 用 Manta 解决 Manta 的计费问题：告别 Hadoop，拥抱 Awk 谈及计费系统的具体实现，Cantrill 坦言计费是软件工程中最令人厌烦的脏活：“计费真是一个该死且烦人的问题……我们居然不得不为了提供商品和服务去实际收钱，这实在太令人遗憾了。”在业界的标准做法中，比如 Amazon 的 S3 或其他类似的大型云服务，处理大规模计费日志的“标配”往往是直接拉起一套笨重的 Hadoop 集群，依赖 MapReduce 框架的大锤来砸这颗坚果。Cantrill 尴尬地回忆道，当团队最初讨论计费系统搭建时，他自己也曾下意识地犯怵：“天啊，难道我们也得搭一套 Hadoop 集群吗？”结果另一位核心工程师 Mark 当场痛骂把他骂醒：“你脑子进水了吗？不，你这个混蛋，我们直接用 Manta 来做啊！” 这成了 Manta 项目的“吃狗粮”转折点——系统开始用自己来解决自己的计费。用户在 Manta 上的每一次操作都会生成日志并沉淀在 Manta 内部（例如在用户的目录中执行 `mls` 就能直接看到包含账单在内的使用日志）。为了演示，Cantrill 现场调出了自己的账单 JSON 文件，并幽默地对听众表示如果有人看账单能感到兴奋，那就尽管看。台下有人问能不能直接删掉账单，Cantrill 笑着说：“这就像 Verizon 的账单寄到家里时你直接把它烧了一样……你完全可以删掉这份发票副本，但很不幸，我们内部留了底，我们永远领先你一步。” 在具体的技术流水线上，Manta 对所有的 JSON 日志负载运行 Manta MapReduce 任务。最初，团队利用工程师 Trent Mick 编写的 `json` 命令行工具过滤字段，然后在 Node.js 中实现 Reduce 阶段。但 Cantrill 审查代码时发现逻辑相当繁琐，便对负责的年轻工程师提议：“这其实挺复杂的，你大概用五行 awk 就能把它全替换掉。”对方好奇地问“什么是 awk？”，Cantrill 瞬间化身“布道师”，甚至调侃自己老了发疯后要开一个公共电视频道，拿着基督教布袋木偶去演《awk 基础入门》。最终，团队把复杂的 Node.js 代码精简成了极为精炼的 Unix 原生命令行管道，这一转折极大地解放了生产力。 ```mermaid flowchart LR accTitle: Manta 内部计费流水线 accDescr: 展现从原始请求日志到 Awk Reduce 阶段的数据聚合流程 subgraph Storage["存储层"] A["API 请求 / 操作"] --> B["原始 JSON 日志 (存储于 Manta)"] end subgraph Pipeline["Manta MapReduce 处理流水线"] B --> C["Map 阶段: json 命令行工具筛选"] C --> D["Reduce 阶段: 精简的 awk 单行脚本"] end subgraph Output["生成结果"] D --> E["最终客户账单 (以 JSON 形式存回)"] end ``` - 彻底抛弃笨重的外部 Hadoop / MapReduce 集群，Manta 完全自举运行自身的计费分析。 - 采用由 Joyent 工程师 Trent Mick 编写的 `json` CLI 工具解析对象元数据与日志。 - 将原本冗长复杂的 Node.js Reduce 代码重构成仅仅 5 行的 Unix `awk` 脚本，借助 `mlogin` 交互式调试管道并横向平滑扩展到上万甚至百万级对象。 --- ### [01:14:01](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4441s) CAP 定理的抉择：写为 CP、读为 AP，打破 S3 伪 200 的幻象在探讨系统架构的可靠性时，Cantrill 引用了 Manta 核心工程师 Dave Pacheco 撰写的经典博文，深入剖析了 Manta 在 CAP 定理上的权衡。众所周知，分布式系统无法同时兼顾一致性（C）、可用性（A）与分区容错性（P），必须在 CP 与 AP 之间做出选择。针对这一点，Cantrill 强调 Manta 的架构设计采取了极具创造性的策略：**将读操作与写操作彻底解耦，针对写是 CP 系统，针对读则是 AP 系统**。 Cantrill 尖锐地对比了 Manta 与 Amazon S3 在写入一致性上的处理方式。默认情况下，Manta 写入时要求在位于弗吉尼亚州阿什本（Ashburn）的两个不同故障域的数据中心内各保留一份副本（双副本）。只有当两个数据中心都成功落盘后，系统才会向客户端返回 HTTP 200 OK；如果其中一个数据中心发生故障导致无法写入双副本，Manta 会直接向调用方返回 HTTP 500 错误。Cantrill 指出，HTTP 500 明确告诉客户端“写入未就绪，请稍后重试”；反观 S3，在数据仅写入部分节点时就会盲目返回 200，导致开发者经常遇到刚收到“PUT 200”接着“GET”却拿到“404 Not Found”的荒谬现象。Cantrill 犀利地讽刺这种行为是“开玩笑式的 200”（just kidding 200），让调用端根本无法可靠处理。而在读取端，Manta 表现为 AP 系统：只要两个数据中心中有任意一个在线，读取请求就能成功响应。 | 架构特性 | Joyent Manta | Amazon S3 | |:--- |:--- |:--- | | **CAP 归属** | **写为 CP，读为 AP**（读写分离治理） | 典型 AP 模型（弱一致性 / 最终一致性） | | **写确认语义** | 双数据中心均成功落盘才返回 HTTP 200 | 写入达到部分法定数量即返回 HTTP 200 | | **故障反馈** | 节点不可用时明确返回 **HTTP 500**，驱动调用端重试 | 容易出现 **PUT 200 后立即 GET 却抛出 404** 的中间态 | | **读可用性** | 任意一个数据中心存活即可完成读取 | 保证高可用，但可能读到旧数据或短暂不可见 | - 双副本机制跨越阿什本（Ashburn）的两个独立数据中心与故障域。 - 依托系统底层的 msplit 以及基于 ZFS 的 SnapLink 技术，原生实现了廉价、秒级的多版本控制（Versioning）与 CORS 支持。 - 客户端生态直接集成 Node.js，用户只需执行 `npm install -g manta` 即可获得完备的 CLI 套件。 --- ### [01:16:35](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4595s) 大数据的归宿：“计算靠拢数据”而非跨网回传在演讲的尾声，Bryan Cantrill 总结了 Manta 所代表的核心技术哲学：在大数据时代，“将计算推向数据”（Bringing compute to data）是不可逆转的唯一出路。Cantrill 解释道，如今全球的数据规模正以极其夸张的指数级膨胀，动辄数 TB 或 PB；然而与之形成鲜明对比的是，用于处理这些海量数据的计算程序文本本身（例如一条简洁的 `awk` 单行脚本、几行分析管道）却极其轻量，其紧凑程度与当年 Doug McIlroy 发明 Unix 管道的时代别无二致。 Cantrill 严肃地论证，在物理现实面前，从远程对象存储系统中把浩瀚的数据通过网络长途“回传”（back-haul）到临时的计算节点上，不仅在经济成本上极度昂贵，在架构延迟上也是完全不可接受的。关于这一根本方向，Bryan Cantrill 坚定地表示：“从我们的角度来看，我们坚信 Manta 所代表的理念——即将计算带到数据面前——绝对是大数据向前发展的必然路径。”他预测，尽管 Manta 是这一领域的开创者，但未来的云计算世界中，必然会有越来越多的现代系统沿着“存储即计算”这条技术演进路线跟进。 - 数据规模（Payload）正经历爆炸式扩张，但执行逻辑的程序体积（Program Text）依然极度小巧稳定。 - 彻底终结了将 TB 级对象数据经由内网或公网长途回传至瞬态虚拟机计算的低效模式。 - “计算向数据靠拢”被确立为现代分布式云存储和批处理系统的核心演进标准。 ### [01:17:14](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4634s) Manta 对大数据的颠覆与运行环境约束 Bryan Cantrill 开门见山地指出，在未来 50 年内，业界或许会出现基于 Btrfs 和 LXC 的某种拙劣模仿版（knockoff），但他们坚信基于 ZFS 的系统才是唯一可靠且理性的底层支撑架构。Cantrill 甚至坦言欢迎其他系统采用相同的架构模型，因为“这至少能证明我们还没有彻底疯掉”。Joyent 内部不仅全面切换至 Manta，而且软件开发工作流也因此发生了深刻改变；在 Manta 上线仅仅六个月的时间里，它就已经彻底刷新了很多人对大数据基础设施的认知。现场听众随即提问，想知道在 Manta 环境中执行任务到底有哪些具体的资源或物理限制，尤其是用户意识到自己可以直接登录进对象并执行任意程序时，这个系统的边界究竟在哪里。Cantrill 解释说，除了直接登录，系统还支持在提交 Job 时直接携带一个所谓的 Asset Payload（本质上是一个 Tar 包）。平台在启动容器环境时会自动解压并部署该 Payload。这一机制彻底解开了语言束缚：用户不再被迫使用 Java，而可以直接在裸机级别的硬件速度下执行 R 语言、Go 或者 Python 程序。特别是针对自然语言处理（NLP）领域的科研与工程人员，社区积累的大量工具几乎全在 Python 生态中，“如果想在其他语言上搞大数据自然语言处理根本就是行不通的（nonstarter）”，而 Manta 终于让这类任务能够开箱即用。关于单实例的默认计算配额，Cantrill 现场查询并确认了系统分配的资源： - 默认提供 1 GB 的 DRAM 内存。 - 默认配备大约 8 GB 的临时存储空间。 - 资源规格可以根据作业需求动态向上调节（例如为 Reducer 阶段分配高达 64 GB 的内存）。 - 费用根据作业声明的资源大小进行相应调整。 ```mermaid flowchart LR accTitle: Manta Job Execution Model accDescr: Diagram showing how user jobs deploy assets and process data across mapper and reducer containers. subgraph Client["客户端任务提交"] A["提交 Job 请求"] B["携带 Asset Payload (Tarball)"] end subgraph Compute["Manta 瞬态执行环境"] C["Mapper 容器 (默认 1GB RAM / 8GB 磁盘)"] D["Reducer 容器 (最高支持 64GB RAM)"] end subgraph Storage["底层存储"] E["ZFS 对象存储 (流式读取)"] end A --> C B --> C E -->|"流式处理 TB 级数据"| C C -->|"中间结果汇总"| D ``` ### [01:19:54](https://www.youtube.com/watch?v=79fvDDPaIoY&t=4794s) 科学计算的质疑与基因测序工作负载讨论当听到系统默认分配 1 GB 内存和 8 GB 存储时，现场一位来自高通量基因测序领域的科学家发出了尖锐的质疑。该科学家直言自己的单文件对象动辄达到 4 TB，8 GB 的存储根本算不上什么大数据；甚至对 Cantrill 在演示中拿简单的“词频统计（bag of words）”来解释大数据感到不满，讽刺道：“我是个搞高通量测序的科学家，现实问题早就不是词频统计这种小儿科了，如果你们想让我认真对待这项技术，就不能拿这种玩具例子当幌子。” 面对听众的质疑，Cantrill 迅速进行了解释与辩护。他首先指出，早先展示的 1 GB 内存只是用于流式读取数据的 Mapper 容器，面对一个 4 TB 大小的文件，以流式传输（stream）穿过 1 GB 物理内存来处理是完全绰绰有余的；至于需要汇聚大量状态的 Reducer，系统支持配置 64 GB 的内存。Cantrill 笑着回应，经典的 MapReduce 论文（Jeffrey Dean 等人的论文）同样拿词频统计作为核心示例，因为这就像编程世界里的“Hello World”，只是为了讲清模型抽象，并不代表系统只能承载这点容量。 Cantrill 进一步以高通量基因测序为例展开反驳： - A、T、C、G 序列的流式匹配与计算恰恰是 Manta 的绝对优势场景（Sweet Spot）。 - 原本从事蛋白质组学（proteomics）的科学家 Christopher Hogg 在六个月前因为对传统科研体制感到厌倦而加入 Joyent，正是他主导了 Manta 与 R 语言的深度集成。 - 单个对象的大小在工程设计上应当合理切分：即使系统支持 TB 级单对象，把单文件做成几 TB 往往是过度设计（Overkill），反而会人为扼杀并发度。类似于 HDFS 按照 100 MB 边界对数据切片，Manta 提倡根据并发度主动设计对象的粒度。 - Cantrill 强调 Manta 的扩展性不受数据体量限制，并放出豪言：“我向你保证，你根本拉不出一条足够粗的公网管道，能快到塞满 Manta 的吞吐瓶颈。” ### [01:23:51](https://www.youtube.com/watch?v=79fvDDPaIoY&t=5031s) 计费模式、云迁移与长尾应用生态随后现场就科学计算场景下的巨额算力成本展开了讨论。针对科研场景下动辄连续运行数周的大规模计算，Cantrill 承认这种极端长时间占用的计算可能更适合传统的 IaaS 虚拟机实例。在确认了该科学家的原始数据目前全部存储在 AWS S3 上之后，Cantrill 表示价格永远可以商量，Joyent 愿意提供专属条款，绝不会让定价模型成为阻碍。 “因为如果这真的是你的核心难题，我们会坐下来好好谈谈。”Cantrill 解释道，Manta 针对算力的定价基准直接参照了 EC2 的标准实例价格（每小时约 14 美分），但核心差异在于 Manta 采用秒级粒度进行计费。对于突发性、高并发的流式分析，秒级计费能大幅削减那些常驻集群产生的冗余开销。如果确实有海量数据与持续的算力需求，双方完全可以针对性定制方案，毕竟这正是系统被创造出来的初衷。 Cantrill 在收尾阶段总结了 Manta 在不同业务领域的渗透现状： - **日志处理（Log Processing）**：这是 Manta 最天然、契合度最高的初始杀手级场景（First natural fit），处理效率极高。 - **科学计算与基因组学**：为科研机构提供原生的语言支持与极高的并行吞吐能力。 - **视频分析与非结构化处理**：越来越多客户利用其管道处理海量多媒体数据。 - **未知领域的创造性用法**：Cantrill 透露现在几乎每天都能在后台看到前所未见的奇怪 Job 跑起来，团队常常会惊呼“这到底是个什么玩意儿”，用户对计算存储一体化的创造力正在不断超出设计者的预期。 ### [01:26:38](https://www.youtube.com/watch?v=79fvDDPaIoY&t=5198s) 通用算力平台与云端任务构建模式在谈到 Manta 平台承载的实际负载时，Bryan Cantrill 开玩笑地指出，用户虽然可以用它做任何通用计算，但“谢天谢地，在上面挖比特币其实无利可图”（Bitcoin mining is actually not profitable on this, thank God）。排除掉挖矿后，Manta 展现出了极强的适应能力。Cantrill 强调，Manta 最令人兴奋的地方在于它能充分激发用户的创造力：作为一个通用的计算平台，只要能在通用算力上运行的逻辑，都可以在 Manta 上实现，因此用户不断用全新的用法带来惊喜。正如 Cantrill 所总结的那样：“People have been and will continue to surprise us with what they're doing with this because it is a general purpose compute platform, and you can do anything you can do with general purpose compute you can do on Manta.” 现场观众提问，是否通常所有操作都以远程作业（Jobs）的形式组织，即用户在本地控制台输入命令（如 `mpipe` 或某些指令），然后后续的一切都拆解分发出去。Cantrill 确认了这种模式，并以演示中的任务流程为例进行剖析：在刚才的命令中，本地的 `mfind` 先列出对象路径，然后通过本地 Unix 管道传给 `mjob create`。他指出，用户可能会质疑为什么要把对象列表先拉回本地再发回云端；实际上完全可以设计一个由 Map-Map-Reduce 组成的多阶段作业，让第一阶段（Map）直接在云端内部执行 `mfind`，从而实现全链路云端化。这正是 Unix 设计哲学的体现——系统提供了极大的灵活性，用户可以根据具体需求自由组合控制流。 ```mermaid flowchart LR accTitle: Manta 任务提交流程对比 accDescr: 对比了本地协调查找与完全在云端执行 mfind 的两种任务构建方式。 subgraph Mode1 ["模式一：本地管道协调"] A1["本地 mfind （获取对象列表）"] -->|"stdout 管道"| B1["本地 mjob create"] B1 -->|"提交作业规范"| C1["Manta 云端集群执行"] end subgraph Mode2 ["模式二：完全云端化流水线"] A2["第 1 阶段 (Map): 云端执行 mfind"] -->|"内部输出传递"| B2["第 2 阶段 (Map): 数据处理任务"] B2 -->|"聚合汇总"| C2["第 3 阶段 (Reduce): 输出最终结果"] end ``` ### [01:28:16](https://www.youtube.com/watch?v=79fvDDPaIoY&t=5296s) ZFS 的新极限与永无休止的固件噩梦现场观众随后转向底层存储架构，提问 Joyent 目前在 ZFS 上遇到的最大痛点是什么，以及 Cantrill 是否曾动念深入硬件固件（firmware）层面去排查问题。Cantrill 幽默地自嘲回应，如果他要介入固件层，恐怕得“绑着自杀式炸弹背心去”（it's going to be with a suicide vest）。谈及 ZFS，Cantrill 解释道，Manta 正在将 ZFS 推向从未涉足的全新维度：系统频繁执行 `zfs rollback`。在 Manta 中，当交互式登录会话（如 `mlogin`）退出时，后台必须立刻对所在的 SmartOS Zone 执行 ZFS 回滚，将其恢复至干净的初始状态。过去在运维中，回滚通常只是管理员手动用于恢复灾难或测试玩票的操作，从来没有被置于高吞吐的核心数据路径（Data Path）之上；但在 Manta 中，系统在数据链路里高强度调用回滚机制。Cantrill 指出，通过这种极限压榨，团队确实发现并修复了几个深层缺陷：“The only real way to find ZFS issues these days is to push it into some wholly new dimension like that, to take some aspect of the system that like exists, but has not been just leaned on, and go actually lean on it.”尽管如此，ZFS 依旧保持着坚如磐石的稳定性，Joyent 也有足够的工程实力继续在此基础上深耕演进。至于存储硬件的固件生态，Cantrill 的态度则充满了无可奈何与幻灭感。他坦言自己对固件质量已经彻底放弃幻想，团队不得不长期与极其不可靠的底层硬件组件共存。虽然磁盘固件的状况比早期稍有改善，但他总结出了一条血泪教训：“I would never buy the first rev of a firmware of anything for sure.” ### [01:30:14](https://www.youtube.com/watch?v=79fvDDPaIoY&t=5414s) 大规模 Map-Reduce 任务的并行实例化机制针对观众关于任务是否只能逐个处理单个对象的疑问（是否一个作业只能对应一个输入），Cantrill 进行了直接纠正。他解释了作业的并发并行原理：在刚才演示的 `mjob` 任务中，系统是同时对检索出的全部 300 个对象进行并行处理的。针对这 300 个对象，Manta 在云端集群中瞬时拉起了 300 个独立的 Zones（容器）。 Cantrill 详细拆解了该任务的阶段执行架构： - **Map 阶段**：任务中由 `tr` 管道连接至 `uniq` 的清洗逻辑，会直接分发到 300 个计算 Zone 中并行运行。Cantrill 在此处风趣地自嘲打了个免责声明：“星号，我知道这算不上真正的大数据，它们只是一堆 Unix man pages 文本而已；也许回头我得造几个 TB 级别的 man pages 才能称得上大数据。” - **Reduce 阶段**：在该示例中配置了单个 Reducer，因此系统在汇集前面 300 个 Zone 的处理结果后，会将数据拉取并汇流到一个专门的 Zone 中执行最终汇总运算。这种纯粹利用标准输入输出（stdin/stdout）和轻量级容器隔离的模式，展示了 Manta 将 Unix 工具链与大规模并行数据处理相结合的简洁威力。 ## Notable quotes > [18:56](https://www.youtube.com/watch?v=79fvDDPaIoY&t=1136s) Building a storage product is like going backpacking in Afghanistan, and if you don't think you're going to a war zone, you're delusional. — Bryan Cantrill > [48:11](https://www.youtube.com/watch?v=79fvDDPaIoY&t=2891s) The way we are able to develop complicated systems is by building out of simple ones, taking simple systems and actually connecting them in interesting and new ways, and the way you're able to do that is by having programs that do one thing and they do it well. — Bryan Cantrill > [16:11](https://www.youtube.com/watch?v=79fvDDPaIoY&t=971s) One thing I would say by the way and this is true this should be true for any system that is designed around safety and integrity: ZFS never went through an era where it was unsafe or didn't have integrity or like people lost data. — Bryan Cantrill > [01:28:16](https://www.youtube.com/watch?v=79fvDDPaIoY&t=5296s) I would never buy the first rev of a firmware of anything for sure. — Bryan Cantrill ## Mentioned Bryan Cantrill, Joyent, Manta, ZFS, Amazon S3, AWS EC2, Sun Microsystems, Oracle, SmartOS, illumos, Jeff Bonwick, Matt Ahrens, Mark Cavage, Doug McIlroy, Ken Thompson, Dennis Ritchie, Donald Knuth, John Bentley, Dave Pacheco, Trent Mick, Christopher Hogg, IBM 1401, IBM 360, Honeywell H-200, Hadoop, Docker, LXC, KVM, Btrfs, Thoth, DTrace, Seagate ## Video description ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups. FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai Facebook: https://www.facebook.com/datacouncilai Eventbrite: https://www.eventbrite.com/o/data-council-30357384520 - 🎟️ GET YOUR TICKET TO AI COUNCIL 2026 🎟️ Meet the world's top AI infrastructure minds where architects of AI share what works. Three days of high-quality technical talks and meaningful interactions. → https://aicouncil.com/sf-2026 ⚡ FIND US: X: https://x.com/AICouncilConf LinkedIn: https://www.linkedin.com/company/aicouncilconf/ Website: https://aicouncil.com/ [Source](https://www.youtube.com/watch?v=79fvDDPaIoY)

Source quotes

[18:56] Bryan Cantrill: Building a storage product is like going backpacking in Afghanistan, and if you don't think you're going to a war zone, you're delusional. [48:11] Bryan Cantrill: The way we are able to develop complicated systems is by building out of simple ones, taking simple systems and actually connecting them in interesting and new ways, and the way you're able to do that is by having programs that do one thing and they do it well. [16:11] Bryan Cantrill: One thing I would say by the way and this is true this should be true for any system that is designed around safety and integrity: ZFS never went through an era where it was unsafe or didn't have integrity or like people lost data. [88:16] Bryan Cantrill: I would never buy the first rev of a firmware of anything for sure.

Source quotes

[20:29] Bryan Cantrill: Building a storage product is like going backpacking in Afghanistan, and if you don't think you're going to a war zone, you're delusional. [50:08] Bryan Cantrill: When you can build things in terms of the Unix philosophy, you always should. [14:06] Bryan Cantrill: There is no fsck for ZFS; such a thing doesn't make sense. ZFS is always consistent on disk. [76:13] Bryan Cantrill: We believe that the the idea that Manta represents—namely bringing compute to data—is absolutely the path forward for big data. [77:14] Bryan Cantrill: If you want to do big data natural language, it's a nonstarter to do it in any other language besides Python. [48:11] Bryan Cantrill: The way we are able to develop complicated systems is by building out of simple ones, taking simple systems and actually connecting them in interesting and new ways.

Original transcript

[0:00] [Music] [0:08] [Applause] [0:09] [Music] [0:14] uh in in California I've spent my entire [0:17] career in Silicon Valley um and I have [0:20] never worked with a single native [0:23] Californian in terms of Engineers [0:25] everybody from California from somewhere [0:26] else this is true of New York too to a [0:28] degree but it's even more true in [0:29] California [0:30] um and so what that means because [0:33] California because any engineering team [0:34] is essentially drawing people from all [0:36] over the country in the world that means [0:37] that the shest way to divide an [0:40] engineering team the surest way to set [0:43] brother against brother and brother [0:45] against sister it's sister against s the [0:46] the fastest way to do that is to bring [0:48] in a [0:50] pizza because when you bring in a pizza [0:53] to an engineering team you are actually [0:55] asserting the dominance of one region [0:58] over another and someone's Hometown has [1:01] just been insulted and it is it is the [1:04] gravest possible insult and it is serbs [1:07] against [ __ ] croats in a nanc so like [1:10] we can't have nice things at our meetups [1:13] we can't have this is a a fine New York [1:16] style pizza that we are not permitted to [1:19] have and and I'm a non-combatant in this [1:21] by and large um but the watching the [1:24] Chicagoans against the New Yorkers and [1:25] so on anyway it's ugly it's ugly it's [1:27] great to have like non-controversial [1:28] pizza out here [1:30] um so um I um I um we're going to take [1:35] kind of an odyssey here um those of you [1:38] um who have have have never seen me [1:40] present before done I know me um I um [1:44] have been in OS development um for about [1:46] 20 years um amazingly I actually just [1:48] turned 40 which is really troubling um [1:50] the um if you think I look 12 now you [1:54] should have seen what I looked like when [1:55] I was actually 22 I mean I actually did [1:57] look like an infant effectively um but [2:00] um I'm not 40 I've been around for a [2:01] while but I've I always enjoy um looking [2:04] at technology historically so we're [2:05] going to go on a couple of historical [2:07] Odyssey um and I am going to tell you [2:10] the the story of of a technology that [2:13] that we have built and I'm going to give [2:14] you the honest story The Honest origin [2:17] story for a change you know whenever and [2:19] I I have done this myself I'm guilty of [2:21] this myself but when you stumble across [2:24] something great where you effectively [2:26] get lucky um and you make some important [2:29] disc Discovery or develop some important [2:31] technology you always kind of retrofit [2:33] your thinking to be like oh yes of [2:34] course this is exactly the problem I was [2:36] seeking to solve um but that's rarely [2:38] the case so I'm actually tell you the [2:39] problem you're actually seeking to solve [2:41] and how we ended up with this technology [2:43] um and we're going to start off with a [2:45] show up hands how do people get this [2:46] reference your computer fell into my [2:47] data your peanut butter fell into my [2:50] chocolate oh my God that is can please [2:52] just can someone raise their hands out [2:54] of sympathy your peanut butter F my [2:55] chocolate you like are two great taste [2:57] to taste great together it's like okay [3:00] I I I'm glad that at least some of you [3:02] are are are uh playing along but so this [3:04] is um this is an ad from the80s I'm [3:08] really dating myself more than anything [3:09] else so it's not your fault if you don't [3:10] get this so anyway we we'll let's just [3:12] move right along um so I want to start [3:15] with the with the prehistory of the file [3:17] system and actually we're going to we're [3:19] going to do the prehistory now and then [3:20] we'll do the pre- prehistory actually in [3:22] uh in a little bit um but so the um the [3:26] file system when it was originally [3:27] developed um in um in the 70s um you had [3:31] bigger and bigger discs and there was [3:34] this idea that you would just make discs [3:36] bigger I if you've ever seen like the [3:38] IBM ramac um if you ever get a chance to [3:41] out to Silicon Valley you definitely [3:42] want to go to the Computer History [3:43] Museum in Mountain View there's very [3:45] very little to see in Mountain View [3:47] Mountain View basically needs a nuclear [3:48] weapon with the exception of the [3:49] Computer History Museum um which is an [3:53] amazing amazing artifacts um and in [3:56] they've got the IBM Ram act which is the [3:58] the the first hard drive 19 56 um and [4:01] you they got these huge platters and [4:03] what they did from 1956 through the 70s [4:06] and early ' 80s they were just [4:07] increasing the density um and decreasing [4:10] the size a little bit of of these drives [4:12] but basically getting single drives that [4:13] were denser and denser and denser and [4:14] that ran out of gas in the ' 80s um and [4:16] in the 80s it was clear that like okay [4:19] just as having microprocessors as well [4:21] right where it's like you just can't [4:22] make a single thing that is arbitrarily [4:24] big you actually need to take some of [4:25] these smaller things and we need to [4:27] start to string them together so that's [4:29] when what happened in the 1980s um and [4:31] that was Patterson developing raid R in [4:33] Array of inexpensive discs um if you buy [4:35] discs or if you're in a position where [4:37] you buy discs for a living you may well [4:39] forget that the I stands for inexpensive [4:41] um because your storage vendor [4:42] definitely seems to have forgotten that [4:44] um and actually there are storage [4:45] vendors that will tell you no it's it's [4:46] independent discs not inexpensive discs [4:49] I don't know where you got that um it's [4:51] it's like I got that from history you [4:53] [ __ ] um paying for another down [4:55] payment on your next boat um but the uh [4:58] it was R AR expensive discs and the [5:00] thing that's interesting to me and this [5:02] is very telling about software to me is [5:03] that the file systems in the 1980s were [5:06] very young right the oldest file system [5:10] was no older than than a couple of [5:12] decades and there's no file system that [5:13] was way around that long most of the [5:15] file systems in use in the 80s were 10 [5:17] years old or less and yet and yet and [5:20] yet and yet that was so old for software [5:23] and so intractable that the idea of [5:26] rewriting our software around this [5:29] changed Hardware notion was just [5:31] impossible like we can't rewrite the [5:33] file systems um we need to do what we've [5:36] done many times over in computer science [5:38] and we will continue to many times over [5:40] why actually rewrite it when we can just [5:42] lie to [5:44] it um it doesn't know it's just software [5:48] we'll tell a very convincing lie and [5:50] that lie was volume management volume [5:51] management was born and what we're going [5:52] to do is we're going to take a bunch of [5:54] discs and we're going to make it look [5:55] like sh sh look like one disc it doesn't [5:59] no so the file system is sitting on here [6:02] and it thinks it's happily talking to [6:03] one dis and meanwhile you're talking to [6:05] many discs below here and there are good [6:08] reasons for doing this um there's often [6:11] Lies start off with good reasons um and [6:14] if you can get away with it maybe that's [6:15] the end of it but you know the the [6:17] reason we don't go around lying aside [6:19] from the fact that it's wrong is that it [6:20] also doesn't work very well um and these [6:23] things tend to catch up with us and this [6:25] lie also caught up with us this lie kind [6:26] of caught up with Humanity because what [6:28] happened was you had the the entire the [6:31] the software that could see all of the [6:33] structure of the data was just convinced [6:35] it was sitting on one disc but you're [6:37] actually sitting on 10 20 30 40 50 100 [6:40] discs well what happens when you want to [6:42] go make this larger well the software is [6:46] like discs don't get larger that doesn't [6:48] happen it's not like it grows a platter [6:50] it's not like two platters in the [6:52] spindle I start feeling affectionately [6:54] towards one another spend a couple [6:55] nights together and all of a sudden you [6:56] have like surprise two new platters um [7:00] but no it's like that doesn't happen [7:01] that the discs don't don't grow so you [7:03] couldn't actually grow a file system [7:05] right file systems were effectively [7:06] static you definitely couldn't shrink a [7:08] file system parts of discs don't [7:10] disappear uh and you had all these other [7:12] problems where the this volume [7:14] management layer was having to do more [7:16] and more and more and more and more and [7:18] didn't have any semantic information so [7:19] things like snapshots are really hard [7:21] things that you Administration is a [7:23] nightmare because you have to go [7:24] administer these two things one of which [7:26] gly does not know about the other at all [7:28] um so this became really really really [7:30] burdensome um and the um by the time we [7:33] got to the um I where I think we'll fast [7:37] forward to what the excuse me there we [7:41] go um so by the time we got to this is [7:45] over the 1990s um and uh just spelling [7:48] out some of some of of of the the [7:51] deficiencies um this was getting [7:53] obviously really really bad by the late [7:55] 90s um so if you were if you've been [7:58] around as long as I have or you remember [8:00] some of these systems and you deployed [8:01] on things like vxfs on top of vxvm or [8:04] ufs on top of lvm or IBM's lvm you have [8:07] these these logical volume managers that [8:09] um you probably are still in therapy [8:10] over at some level um because these um [8:13] these things had really nightmarish [8:16] failures where your data would be gone [8:18] at the end of it which is not that's not [8:19] funny um that that's really really [8:21] troubling um and the you know a um a [8:25] major major problem between this divide [8:27] it's kind of semantic divide [8:30] um is that when you are writing the the [8:32] interface between a file system and a [8:34] disk is just a block write this block [8:37] write this block write this block read [8:38] this block read this block read this [8:39] block well that's a very narrow kind of [8:42] of semantic interface and there's no [8:46] real one does not have a real way of of [8:48] checking the other so a file system you [8:51] can uh if if you want to actually [8:53] protect against kind of physical errors [8:55] in the in the volume management layer so [8:56] you want to protect against bit rot [8:59] right well you can do that relatively [9:01] easily because you can take a block and [9:02] you can check on it but what if you at [9:04] the volume management layer want to [9:06] detect again detect say a phantom right [9:10] so a phantom right is a right that [9:11] actually didn't happen you did you did [9:13] the right you told the controller to [9:14] write it told you I wrote it and you [9:16] said seriously like it's on stable [9:18] storage and the controller is like [9:20] seriously would I lie to you it is on [9:21] stable storage and then you go back like [9:24] okay hey I need block 65 they're like I [9:25] don't know what the [ __ ] you're talking [9:27] about man what are you who told you that [9:29] I tell you that like where did you it's [9:31] like oh firmware um but the um these are [9:34] real Phantom reads you read block 63 I [9:37] how about block [9:38] 36 um the problem with that is it's [9:41] totally self-consistent so the volume [9:43] management layer is like looks good to [9:44] me I don't know sure pass it up to the [9:46] file system file system's like whoa no [9:48] no no that's like Etsy password that's [9:49] not the database um the and and if you [9:53] remember you know from this era you you [9:55] know I probably had you had at least one [9:58] of these but it is terrifying when you [10:00] go into the database and you see Etsy [10:02] password or vice versa um it's like no [10:04] no no no no something is very wrong in [10:07] the animal brain um and that and the [10:09] problem is those errors can be very very [10:10] deep in the stack so um the problem here [10:14] the problem with the manageability the [10:15] problem with there was the performance [10:17] problems um the the complexity problems [10:19] the problem the problems were in the [10:21] abstractions this volume management [10:23] abstraction which really should have [10:24] been invented as a short-term crutch had [10:27] become a long-term solution and the [10:28] tragedy is actually that still today [10:30] people are still living um with this [10:33] divide and it's a tragedy because we at [10:36] Sun as Sun micros for 14 years um saw [10:39] this problem uh and wanted to go do [10:41] something um quite radical actually [10:43] we'll I can go past that and we [10:45] developed this thing called ZFS so um [10:47] how many people heard of [10:48] ZFS oh thank God how many you actually [10:50] deploy [10:52] ZFS right don't be embarrassed it's fine [10:54] I know I know that's that's actually [10:56] that's an impressive number but no I I [10:58] sadly we are now in this uh we are in [11:01] this kind of awkward scenario where many [11:03] people have heard that there is a cure [11:04] for polio um but they have not actually [11:07] vaccinated themselves or their children [11:09] and they keep being stricken with polio [11:11] and they know to be sad about it and [11:13] they know to be oh God if we'd only had [11:14] the vaccine oh well um it's like [11:16] unfortunately like my manager won't let [11:18] me have the vaccine um or there's I hear [11:21] there's like a licensing issue with the [11:22] vaccine or something like that I don't [11:24] even know [11:26] um so sadly um we are we're not [11:29] vaccinated um and uh and actually [11:31] there's a very good um AR technika [11:34] article you may have seen earlier this [11:35] week um going into the history of ZFS um [11:39] and the knockoff that was created of ZFS [11:41] called betters bits um if you're B about [11:45] it um and um and actually it was they it [11:49] the article was actually very well [11:51] written and went out of its way to call [11:52] them next Generation systems not new [11:54] systems the article was very up front [11:56] about pointing out that these systems [11:58] have been around for for a long time ZF [12:00] my data has been exclusively on ZFS for [12:02] well over a decade at this point um so [12:05] exactly um the um so um ZFS um was a [12:11] total rethink um of this divide and as [12:15] long as I had known I came out to Sun to [12:17] work with this engineer Jeff bonwick as [12:19] long as I had known Jeff he wanted to [12:20] completely annihilate this and he wanted [12:21] to go completely rethink this problem [12:23] and so what he and Matt aens and some [12:25] other Engineers did is they literally [12:27] went back to a clean sheet of paper and [12:29] how would we Design This given that we [12:31] have multiple we multiple discs are a [12:34] fact of life um also by the way the [12:37] other kind of fact of life that's [12:38] important is that we see um dram is [12:40] actually there's there's a lot of dram [12:41] we want to use that effectively um we [12:43] also see and this is where okay so those [12:46] two things were right this is where [12:47] missed the mark slightly um one of the [12:50] things that they were really designing [12:51] for um was the the the presence of [12:53] non-volatile main memory um which back [12:55] in 2001 I would have thought surely by [12:58] 2014 like you just came back from 2014 [13:00] wow how is the nonvolatile main memory [13:02] it's like what are you [ __ ] talking [13:03] about no no it's like you're so pathetic [13:05] it's never gonna happen um I would love [13:07] to see nonb made memory I don't know if [13:08] where going to see it but the ZFS was [13:10] also kind of designed with that in mind [13:12] um but so there these kind of these big [13:14] trends that ZFS was designed around um [13:16] Blew Up the Volume management divide and [13:18] now youve got one Cloud effectively [13:20] where you plug discs into the bottom and [13:23] you have file systems plugging in at the [13:25] top because a file system really is only [13:27] an administrative Nexus it shouldn't be [13:29] anything more than that so you should be [13:31] able there should be no real I mean a [13:34] file system is just it is it is a path [13:37] um it is hierarchy uh and it is policy [13:39] that's it that's all file system should [13:41] be it a file system shouldn't have any [13:44] direct one toone association with a dis [13:46] lie or not so in the ZFS model you just [13:49] ZFS Crea a file system I mean there's no [13:51] um you don't have to have a [13:53] corresponding disc similarly the the uh [13:56] we've got multiple uh raid policies you [13:58] can have mirroring ry2 Del Perry raid [14:00] and so on and that's been added to over [14:02] time um and that allows you to solve a [14:04] whole bunch of manageability problems um [14:06] and you can deal because ZFS one of the [14:09] great things about it is because um it's [14:11] a it's a totally copy onri design um you [14:14] don't have an on disk consistency check [14:16] there is no fsck for ZFS such a thing [14:19] doesn't make sense um ZFS is always [14:21] consistent on disk um and actually one [14:23] of the things that they've U kind of [14:25] interesting is dealing with pathological [14:28] pathologically lying devices um because [14:31] you are only as consistent on disk as [14:33] the ultimately the device tells you you [14:35] are um and so one of the things that CFS [14:37] has added over the years is the actually [14:38] the ability to roll pools back [14:40] transaction by transaction until you get [14:42] to one that the disc was actually not [14:43] lying about I mean the disc lies all the [14:45] time if the dis is the actual enemy you [14:48] actually are [ __ ] I mean there's [14:49] nothing that any body of software can do [14:50] about it um but you can at least you [14:52] know get for the the those discs that [14:54] are in that kind of that gray area that [14:56] appear to be enemy action but sometimes [14:59] store the correct data say the Western [15:00] Digital green drives um um you could ZFS [15:04] has actually evolved to be able to to [15:06] always be consistent on on dis not [15:07] require fsck because the thing that they [15:09] saw is that with with data data sizes [15:12] climbing it's like you C you can't or uh [15:15] I guess Hacker News did try for the 36 [15:18] hours that were down you shouldn't try [15:20] to fsck a terabyte um that that's not a [15:24] rational thing to go do because the io [15:26] speeds of spindles have not [15:28] fundamentally changed IBM ramac is a [15:30] 1200 RPM device most spindles today are [15:34] 7200 RPM devices you have some 10K some [15:37] 15K then you got ssds which are their [15:39] own topic of conversation but that's [15:41] amazing to me that you've gone from 1200 [15:42] RPM to to 7200 RPM um in 50 years um so [15:47] that's not the performance is not [15:49] substantively different um so the a [15:52] terabyte is is a lot more storage than [15:54] we had in 1956 but actually doing random [15:57] reads from it is actually not materially [15:59] faster it's like 4X faster than it was [16:00] in 1956 um that is that should be [16:03] terrifying to you uh and that what that [16:04] means is you can't go back and just read [16:06] all your data to check the consistency [16:08] of your file system that should be [16:09] something that does not exist um it [16:11] doesn't exist in CFS um ZFS also by [16:14] nature of the fact that it kind of owns [16:16] the whole stack ZFS check sums every [16:18] block but it checksums every block [16:20] indirectly which is something you can't [16:22] do when you've got this this volume [16:23] management file system divide so it [16:25] knows when it pulls a block off of the [16:28] dis it knows what the check sum for that [16:30] block is supposed to be not because the [16:32] check sum is stored in the block but [16:34] because the check sum is stored in the [16:35] block that points to that block so it [16:37] knows when it's got the wrong data I be [16:39] like hey oh you firmware that's the [16:42] wrong one I C you no no don't give me [16:46] that one [ __ ] um so the and if you [16:49] mirror ZFS will actually heal your data [16:51] so ZFS if you get the if you get the [16:53] wrong data off of one half of the mirror [16:54] CFS will write the good half the the [16:56] good copy over it and you got these [16:58] errors um early in cfs's life um we [17:02] would see these check sum errors come [17:04] out of ZFS and it would tell you don't [17:06] worry I they had some tck some errors [17:08] but it's cool like I I got the right [17:09] data and you're like okay that's [17:11] obviously wrong because we are running [17:14] on and you kind of you can pick your [17:15] vendor you can pick your controller um [17:18] we had one where it's like we were [17:19] getting really persistent check some [17:21] errors from every instance of a [17:24] particular controller that was like that [17:27] had had there was like 6 million of in [17:29] existence and you're just like this [17:30] control I'm sorry this controller can't [17:32] be this broken it's just not possible as [17:34] it turns out it was that broken um and [17:37] the the there was a bug a a a silicon [17:39] bug um in the the dma engine um and it [17:43] just like under certain conditions that [17:45] were actually relatively reproducible it [17:46] just wouldn't dma the last 64 bytes of a [17:48] block just didn't feel like it um and [17:51] like God only knows how many like life [17:53] support systems suddenly clicked off [17:55] because the last 64 bytes was was [17:57] missing from these payloads and we still [17:59] and we would see this across the board [18:00] what was great is and this again [18:01] happened early in cfs's life it's not as [18:02] amusing now because people now trust CFS [18:04] as opposed to whatever they're deployed [18:06] on but you get the No No ZFS has to be [18:09] wrong because I spent so much money on [18:12] San X it can't be wrong it's like well [18:15] funny story it actually is wrong so you [18:16] know you and these guys get it wrong um [18:19] it's because it's hard um is the reason [18:21] they get it wrong but um ZFS has been [18:24] you know again over the years um we've [18:25] gotten to the point we've got Absolute [18:27] Data reliability one thing I would say [18:29] by the way and this is true this should [18:30] be true for any system that is designed [18:32] around safety and integrity ZFS never [18:35] went through an era where it was unsafe [18:38] or didn't have integrity or like people [18:41] lost data um and it's kind of amazing to [18:44] me how much in software people are [18:47] willing to write off software that that [18:50] that does this as kind of pubescence [18:53] it's like yeah I lost my data but you [18:54] know a million eyeballs like it's [18:56] getting better all the time it's like no [18:58] no no this is like this is like a [18:59] 12-year-old who's stealing cars like [19:01] this is an issue this is not like well [19:03] he's only 12 it's like no he like he [19:05] stole a car at gunpoint having just [19:09] robbed a liquor store like youve got an [19:11] issue here this is not you can't just be [19:13] like well call me when he's 18 it's like [19:15] yeah you want to see what this guy's [19:16] like when he's 18 um and so this whole [19:19] idea of like oh you know yeah better Us [19:20] lost my data but yeah I'll come back [19:22] tomorrow uh it's like no no no that [19:24] that's there are things that are wrong [19:26] by architecture you should not and CFS [19:28] was was architected to be reliable in [19:30] terms of data so and that's kind of the [19:32] uh and we go into lots and lots of depth [19:34] on CFS um the in terms of my own [19:37] personal history um at Sun um we had [19:40] developed ZFS uh in 2006 I and a bunch [19:43] of co-workers wanted to go and actually [19:44] build an Enterprise storage product uh [19:46] around ZFS um so that's what we did um [19:49] and we um we regrettably succeeded um [19:53] and unfortunately um we um uh one of our [19:57] biggest early customers um was actually [19:59] a customer that we hadn't sold a lot of [20:01] gear to in a long time this database [20:02] company you might have heard of um and [20:06] um and I I sadly I I I think that there [20:10] is there is uh at least to a certain [20:12] degree they're like oh wow like you guys [20:13] are making interesting things [20:15] again you um so um we we were actually [20:19] thank you you know I just I I I have [20:22] that daily affirmation I still feel like [20:24] I've got survivors guilt I just feel [20:25] like there's more I could have done um [20:27] was all [ __ ] up [20:29] um but so we built this storage product [20:31] um based on CFS um and uh building a [20:34] storage product by the way and I say [20:36] this having gone into this without what [20:38] I thought were kind of eyes open in in [20:40] 2006 when we started doing this uh [20:42] building a storage product is like going [20:43] backpacking in Afghanistan um and if you [20:47] don't think you're going to a war zone [20:49] you're delusional um the and so we were [20:53] like oh I think this will be like we'll [20:54] make an Enterprise storage product then [20:55] it'll be great um and uh and we learned [20:58] a lot and we improved ZFS a lot actually [20:59] as a result the ZFS that you have today [21:02] um is a ZFS that's done a couple of [21:04] gritty tours of Duty um in in the [21:06] Enterprise Storage Wars because what you [21:08] learn in doing that is the number of [21:11] ways that storage can fail because going [21:14] into it I was like okay I got it like [21:15] Phantom reads Phantom wrs bit rot like [21:18] all good sounds good oh boy no no no the [21:21] the firmware is so many steps ahead of [21:23] you in terms of its creativity um and [21:26] the level of Byzantine failures cuz fir [21:28] is like really you think that's all I [21:29] got um we had a firmware bug um this is [21:34] this is where I can start rattling on [21:35] this is where actually I need to Rattle [21:36] off my therapist says I need to to talk [21:39] about the firmware revision numbers just [21:40] to to let it out it'll it'll corrode me [21:43] um so the seate s0d firmware had a [21:46] beautiful bug in it where they uh there [21:49] was a software bug on the head where it [21:52] misapplied uh the polarity was wrong um [21:55] on the on the head it's like yeah like [21:57] good polarity [21:59] seems like that's important it's like it [22:00] is important yes uh and they when you [22:04] got to certain lbas got to certain [22:06] logical block addresses instead of [22:08] decelerating the head it would [22:10] accelerate the head um into the cylinder [22:13] and something on on the disc some J tag [22:15] or something was like oh no no no you [22:17] don't mean that you don't mean that no [22:19] no this will actually there'll be smoke [22:20] coming out of that if you do that so it [22:21] would actually power cycle the device it [22:23] would power cycle the drive so and it [22:25] would bounce the drive well how long [22:26] does it take the drive to bounce come [22:28] back up and start serving ios's about [22:31] 650 [22:32] milliseconds so you start seeing these [22:34] IO outliers at 650 millisecs what the [22:37] [ __ ] is that and ZFS is not designed at [22:40] the time I should say I think this the [22:41] major major difference between a file [22:44] system that works that you trust and one [22:47] that has actually runs hundreds of [22:49] thousands of spindles in Enterprise [22:51] storage is the ability to sort out these [22:53] kinds of failures because ZFS would go [22:56] on a long vacation when it would hit [22:58] these kind spendal um and it would be [23:00] like because the problem is the io would [23:01] Al would because it was effec a race [23:03] condition at the head the io could [23:05] succeed and because the iio could [23:06] succeed ZFS would kind of drag on well [23:08] the whole array would kind of would [23:10] would grind down um things doing doing [23:12] things like big snapshot deletion doing [23:14] things like I mean if you ever if you're [23:15] using ZFS back in the day if you had a [23:18] gigantic snapshot and deleted it we'd be [23:19] doing a bunch of random read from the [23:21] context of of a transaction which would [23:23] the as I said at Fisher works we never [23:25] lost anyone's data but it went on some [23:27] very very very long vacations um and it [23:30] was like oh Lassie come home and the [23:31] thing is like you could be on the [23:32] machine and you could see like you can [23:34] actually kind of like you're using DS to [23:35] like watch its progress and you're just [23:37] like I actually know one customer just [23:39] like okay like I'm figuring out the [23:40] progress like okay I know when this is [23:43] going to be done when this transaction [23:45] group will be synced out let's see today [23:46] it is Monday um and it literally was be [23:49] like by Thursday at 8:00 P.M it'll be [23:51] done and he's like yeah fortunately I [23:53] don't care about the data on the system [23:54] so let's just throw this one overboard [23:55] and start over um but but we [24:10] [Music] [24:58] was bad bad it was a blitz cre license [25:01] anyway um so the um and after that whole [25:06] thing uh I left um and actually I went I [25:09] went in or stupid I went into Oracle [25:11] with this like stupidly I just want to [25:13] go back with the time machine and punch [25:15] myself in the mouth if we invent a time [25:17] machine the first thing I want to do is [25:18] go back to myself and just deck myself [25:20] Circa 2010 cuz I was like oh it'll be [25:23] great like Oracle understands business [25:25] and Sun understands technology and it'll [25:27] be like a business understands [25:28] technology man it's like are you stupid [25:31] um it wasn't that at all um and that's [25:34] not what or was at all they're they're a [25:35] monopolist they're not anyway whatever [25:37] um you can go see a full uncensored [25:39] history if you Google Fork lumos um the [25:41] uh the funny thing actually about this [25:43] talk is that uh I gave this talk which [25:46] was at at yck um at a conference it [25:48] turns out Oracle sponsored um and and [25:52] the the kind of the the the the the [25:55] conference was a little concerned about [25:56] some my caner and I by the way did not [25:57] mention the Nazis in that talk I not [25:59] like did not say that at all um but [26:01] there were there were a couple other [26:02] things I did say um and the uh but I [26:05] knew that Oracle doesn't care because [26:06] the think is Oracle a sociopath they [26:08] don't care what I trust me Oracle does [26:10] not care what I think about them or what [26:11] anybody thinks about them um the uh so [26:14] but the the the conference organizers [26:17] were very concerned what Oracle would [26:18] think and so when the videos for this [26:20] conference were all made available every [26:22] video came up except for mine and you [26:25] know I believe in incompetence over ma [26:28] um almost always I'm just like all right [26:29] well someone just like lost the file or [26:30] whatever I don't know they gotten R but [26:32] like a week passes two week passes to [26:34] you know and finally it's like where [26:35] wait where is the video why is every [26:37] other video for every other talk up [26:38] except for mine then finally my video [26:40] appears you're like oh okay and you go [26:42] watch and like oh this is a big [26:43] disclaimer that has never been on any [26:45] other usix video the following views do [26:47] not represent the views of the yix [26:48] association da da da da da and you're [26:50] like all right all right you wieners uh [26:53] fine fine fine and then you go into and [26:55] the the format of the talk is like [26:57] you've got the slide and you got like my [26:59] little talking head down here and the [27:01] slide here and there's like this black [27:03] strip above my head on the video and [27:05] they took this disclaimer and like [27:07] mashed it up and it's above my head the [27:10] entire time I'm [27:14] speaking they they did not bleep me out [27:17] so that is obviously I'm glad you asked [27:20] because that is clearly the the next [27:22] Peak that I need to to conquer I [27:24] actually need to try to get actually [27:26] censored um I didn't think they should [27:28] start like flashing it when I started [27:30] getting really colorful about Oracle but [27:31] anyway you you can go watch that talk [27:33] and you all of its uncensored Glory um [27:35] but so you know moved on um I came to [27:38] joyant um and a lot of us um went to a [27:41] bunch of different places but a bunch of [27:43] folks came to joyant um and um for us [27:46] for joyant and joyant was the actually [27:49] long before I was at joyant joyant was [27:51] the first user of ZFS outside of Sun so [27:54] Z the first production user of ZFS was [27:56] joyan um in 2005 um and so we worked [27:59] very closely with the founders of the [28:00] company um and I was excited to go work [28:02] there um and so for us um ZFS is really [28:05] the core of what we do we are an [28:07] infrastructure of service cloud provider [28:09] um that also makes available our [28:10] software stack um to run private [28:13] clouds okay so we had great [28:15] infrastructure as of service but the [28:17] thing that we were missing was optic [28:19] storage S3 right so people use S3 you [28:22] know about S3 right so this is um do [28:25] actually have more people really heard [28:27] of ZFS and S3 I assume you're just like [28:28] not raising your hand because like [28:30] obviously everyone knows about S3 I'm [28:31] not we had a Shan the Shane there you go [28:34] that's right that's right very good so [28:36] S3 for those you who don't know is the [28:38] simple storage service um this is the [28:40] really the first cloud service to get to [28:42] give Amazon their proper this is the [28:44] first cloud service is a storage service [28:46] an HTTP based storage service um and S3 [28:49] did something very very very important [28:52] um S3 did not do what frankly I would [28:55] have done and a lot of other people [28:56] would have done at the time which would [28:58] be to take an existing fil like [29:01] abstraction and make it available over [29:03] HTTP that's not what they did is an [29:05] object store not a file store so in [29:09] particular you cannot do partial rights [29:11] to an S3 object and that is really [29:14] really important because when you say I [29:16] would have guessed that like no no you [29:17] can't just not allow partial rights like [29:20] won't that be won't that not be [29:22] acceptable but it turns out that's [29:24] actually acceptable because there's a [29:26] lot of data out there that is write once [29:28] read many um and you've got these [29:30] objects that are logs or whatever you're [29:32] not mutating the middle of your log [29:33] you're not mutating the middle of your [29:35] of your web assets you're not mutating [29:36] the middle of your backup in fact you [29:38] don't want to do that um so the fact [29:40] that it was object storage was actually [29:42] that was valuable um and so that was [29:44] actually extremely important um because [29:46] that changed the constraints of the [29:48] system allowed them to build the [29:49] distributed system um and we [29:51] historically have not had an S3 so we [29:53] don't have S3 um and it has been the [29:56] kind of the the glaring Gap on the [29:58] gaping hole in our offering from our [30:00] perspective that we didn't have this [30:01] object storage and so the question that [30:03] we had is can we build um a a something [30:08] interesting out of ZFS so I just kind of [30:10] come through this backpacking trip to [30:12] Afghanistan with ZFS and I'm thinking [30:14] what can we go do with ZFS what can do [30:15] ZFS or CCE what can we go do to make it [30:18] it easier to imp object storage and I [30:20] just yeah I couldn't come up with [30:22] anything um or the stuff I came up with [30:24] was just terrible obviously um so all [30:27] right let's put that aside um and I want [30:29] to talk about virtualization in the [30:30] cloud for a little bit and and I promise [30:32] we we'll come back to to ZFS in a second [30:35] so um virtualization is uh is absolutely [30:39] essential to the cloud virtualization is [30:41] all about allowing multiple Tendencies [30:44] allowing multiple tenants on a single [30:45] system giving multiple people the [30:47] illusion that they have an entire system [30:49] and allowing them to be on their secur [30:50] way uh you need virtualization to be [30:52] able to do that um and the question is [30:54] where do you virtualize in the stack [30:56] because you actually have a couple of [30:57] choices [30:58] um and the historical answer since the [31:01] 60s is Hardware based virtualization so [31:03] we started in the ' 80s now going back [31:04] to the 60s um actually I can't go much [31:06] further back than than I guess the ram [31:08] ma is as far back as we're going to go [31:10] but um the um and actually it's kind of [31:13] interesting how this was born um the um [31:17] Harbor virtualization was born with the [31:19] introduction of the IBM 360 so the IBM [31:22] 360 uh introduced in the the mid-60s [31:25] probably the most important single [31:27] computer that was ever [31:28] um ever introduced because it combined [31:31] IBM's commercial and scientific lines [31:33] into a single um microprocessor into [31:36] into into a single system the 360 um one [31:39] of the systems that they left behind was [31:40] the IBM 1401 the IBM 1401 was a an [31:44] accounting machine it was one of these [31:46] things that people were using as a [31:47] computer but it was kind of it existed [31:49] in this kind of this gray of Twilight of [31:52] early Computing um and you could and [31:55] certainly you had programs on it I mean [31:56] it was a store programed computer um but [31:58] it was not appropriate for scientific [32:00] calculation the 360 um didn't run any [32:03] 1401 programs because [ __ ] that um and [32:06] like if you want to if you want software [32:07] for this thing like write some um that [32:09] was the that was literally the logic in [32:11] the 60s that Hardware is the tough stuff [32:14] and so like if you want some software [32:15] like yeah like get writing I don't know [32:17] like go find some people and there's the [32:18] instructors at architecture go to town [32:21] well the thing that's amazing to me is [32:22] that even by the mid-60s that was an [32:25] unacceptable answer and the because [32:27] people had IBM 1401s and that they were [32:30] running software on and they wanted to [32:33] go deployed on the 360 but they didn't [32:34] want didn't want to Rite that software [32:36] IBM classic in in kind of classic you [32:39] know tale of the industry fashion IBM is [32:41] like well no [ __ ] that we're not just [32:42] not going to do it so Honeywell who was [32:45] a competitor of IBM said actually why [32:47] don't we go to that and so they did the [32:48] Honeywell h200 which was which was bit [32:50] forbit compatible with the 1401 um all [32:54] of a sudden that thing starts selling [32:55] like hot cakes um because everyone is [32:57] replaced their 1401 and their 1410s with [32:59] Honeywell h20s not with the IBM 360 well [33:02] suddenly you got IBM's undivided [33:04] attention um and what IBM did going back [33:08] to that remember that that earlier theme [33:10] the persistent theme of computing it's [33:12] livetime um and it's time to lie to that [33:15] software and tell it that it's on an [33:17] ibm1 1401 we are going to create an IBM [33:21] 1401 strictly out of software it's going [33:23] to be a pure software construct and it [33:26] will be so persuasive and convincing [33:29] that the 1401 software running on top of [33:31] it will have no idea that it is in fact [33:34] not running on a 1401 but a 360 and this [33:37] is and we have this this theme has [33:39] carried through the day when you run an [33:41] Amazon your operating system kernel that [33:43] you are running on AWS is absolutely [33:45] convinced that it's running on Hardware [33:47] a statement that is somewhat true um but [33:51] it's not completely true uh the thing [33:52] that's actually an amazing little uh [33:54] little footnote on this one actually is [33:56] I was actually out here in New York um a [33:58] after the Y2K rollover and I was talking [34:01] to a uh CIO from a very big Bank out [34:04] here and we're just kind of shoting the [34:05] [ __ ] about Y2K which is obviously a big [34:07] fizzle uh for us at Sun um about 30% of [34:10] our Y2K problems were actually leap year [34:12] problems I don't if anyone else had [34:14] these so recall that every four years is [34:16] a leap year except for every 100 except [34:20] for every 400 and there's a stunning [34:23] amount of software that managed to [34:24] encode rules one and two and not rule [34:26] three um because 2,000 of course 400 [34:29] divides 2,000 evenly 2,000 is a leap [34:32] here despite the fact that it divides [34:34] 100 that 100 divides it evenly so anyway [34:36] whatever so we had a bunch of those bugs [34:38] you know who cares um and so we and he [34:40] said you know what's funny um in terms [34:42] of Y2K problems um all of our Y2K [34:45] problems or the the plurality 45% [34:47] something have that came from a single [34:50] Hardware platform I was just like yes [34:53] what what is it and he had good kind of [34:55] like good dramatic pausing and pacing I [34:56] me I was going to pee my pain [34:58] like what is it what is it what is it [34:59] what is it uh the IBM [35:02] 1401 and it was [35:06] like I mean I was like the the the who's [35:09] he what he the what the the same IBM [35:12] 1401 that was last made in 1963 he's [35:14] like the very same it's like he's like [35:18] oh yeah yeah and apparently like [35:20] apparently this kind of caught Everybody [35:22] by surprise everyone was like a wow okay [35:25] we and clearly like this is not like [35:27] running the exchange or anything I mean [35:29] this is clearly running in in in very [35:33] not that's my knowledge uh [35:35] the this is clearly running kind of [35:37] ancillary components but it was it was [35:40] there and it was real and and like like [35:42] the Y2K solution for this is where like [35:44] Honeywell h200 may have bought the 1401 [35:47] some life but ultimately like the Y2K [35:49] solution for the IBM 1401 actually is [35:51] write some new software um they're [35:53] actually the two digits of the date are [35:55] actually enshrined in the hardware so [35:56] you're kind of boned um [35:58] but but to me and that was that was an [36:00] amazing moment to me because I think it [36:02] it tells us about the Persistence of [36:04] software that here you have software [36:07] that is written in in the late 50s early [36:10] 60s maybe as late as the late 60s but [36:12] not the 70s by the 70s this is an [36:14] antique this this machine um and yet it [36:18] has survived the death of not only that [36:20] Hardware but the hardware that replaced [36:22] it and probably the hardware that [36:23] replaced it it is running virtualized [36:25] and virtualized and virtualized and [36:26] running on it was running on machine um [36:28] it like amazing I mean I it's totally I [36:32] I I think I was like staggering around [36:34] for an hour after that um but it it it [36:37] it's it's amazing so that is certainly [36:39] the oldest answer to this question and [36:41] it's still a valid answer is to [36:42] virtualize the hardware um but the [36:45] problem is that you know it's it's this [36:47] theme again the LIE works great at first [36:50] but then you begin to pull things and a [36:53] lie begins to snap um and the in [36:56] particular um when you are virtualizing [36:59] the hardware similarly to when you are [37:01] virtualizing the the the device the dis [37:04] when you're virtualizing the hardware [37:06] the the semantics that you have are very [37:08] narrow in particular like take a dam if [37:11] you have virtual hardware and I'm going [37:13] to give you virtual Hardware I give you [37:15] a certain amount of dam I give you a gig [37:16] of dam I have to really give you that [37:19] gig of dam I can't really lie about that [37:22] um I mean I can try to lie about that [37:24] but the operating system really thinks [37:25] that it has a gig of D RAM and it's got [37:27] the right to think that um and I can't [37:29] make Dam show up later the operating [37:31] system is not like what is uh there's [37:33] another dim over here that I didn't see [37:35] like what's this thing um you defin you [37:37] can't take Dam away so you when you have [37:40] in terms of the number of VMS that you [37:42] can reasonably put on a machine take [37:44] your dram and divide and yes you could [37:46] you can if you're a crooked service [37:48] provider you can oversell your dram um [37:51] and you know may you know maybe that [37:52] will you know last enough for you to [37:54] sell the business and and you know move [37:55] to a remote island um but that basically [37:58] doesn't work you can't really overell [37:59] the dam because the the operating system [38:01] very much believes it has Dam there and [38:03] this is true of CPU it's true especially [38:05] of IO networking IO you're basically [38:07] just boned on networking IO there's [38:09] there's not a way to make that efficient [38:11] because you're going to have to get out [38:12] and talk to the and you know there are [38:15] tricks you can pull but the tricks you [38:16] that are pulled that are effective are [38:18] actually taking the hardware and and [38:20] taking it straight into the guest so [38:22] it's like well it's not really harder [38:23] virtualization anymore is it it's like [38:25] oh we won't tell the lie like this lie [38:27] will happen to be the truth um and now [38:29] now it performs it's like yeah but now [38:31] we're getting all the advantages of the [38:32] LIE we're losing because we're not [38:33] actually virtualized so there are lots [38:35] of problems with hard race viralization [38:36] the lots of advantages too namely I can [38:38] run my my own my old stack completely so [38:41] we have in the joint public Cloud we've [38:43] got hard based virtualization via KVM [38:45] but we also have an uh well I brief [38:48] segue to platform level [38:50] virtualization uh doesn't work uh so the [38:54] platform you can virtualize very high in [38:55] the stack the problem with doing that is [38:57] you like what they call the Google app [38:58] engine problem where you are you're so [39:00] high in the stack that you can't [39:02] actually develop big apps that when [39:04] something gets big it has to move [39:05] somewhere else um when something and and [39:08] All Passes have this problem but by then [39:10] they've usually gone out of business [39:11] because this is a terrible terrible [39:12] terrible business model um so you know [39:15] our solution um is OS level [39:18] virtualization so OS level [39:19] virtualization is where I've got one OS [39:20] kernel sitting on the hardware and I'm [39:22] going to punch out multiple containers [39:25] and these containers are going to look [39:26] feel and smell like their own operating [39:29] system like their own Hardware even I [39:30] can reboot them but they're actually not [39:32] their own Hardware the level of the [39:34] abstraction going up to this thing is [39:35] not the hardware the abstraction is the [39:37] system call table and as a result it's [39:40] sitting on the hardware it is running on [39:42] the hardware it sees all the CPUs it [39:44] sees all the dam now you may only allow [39:47] it to say to only allocate say a gig of [39:50] dam but when you give a virtual Osa gig [39:52] of D Ram all you're doing is writing [39:54] down somewhere hey like by the way like [39:57] this guy hits a gig of dam cut them off [39:59] that's all you're doing and if you want [40:01] to so so if you are deployed in the [40:03] joint public cloud and you want to [40:04] resize an instance you just go resize it [40:06] and we just say oh okay all right make [40:08] that two gigs I guess okay make it four [40:10] gigs mean the provided that we're not [40:12] overselling the Box we can always resize [40:14] Dam up and down it's very nice and there [40:15] are lots of other great attributes to [40:17] this uh it performs really well and so [40:19] on uh we've got a really we've got zones [40:22] in in smartos which is a really [40:24] bulletproof implementation of this um [40:26] and I think it's it's been great to see [40:28] I have to say this model I've been so [40:30] convinced about this model that um [40:32] people ask like how do you guys feel [40:33] about like Docker and lxc and so on I [40:35] actually I to me it is very validating [40:38] to see that because I think what I see [40:41] other people seeing in Docker and alxe [40:43] is the power of the model and I think [40:45] the model is the right model so I [40:46] actually want I'm I'm glad that those [40:48] Technologies exist um and then I'm [40:50] especially glad that at least in Alex's [40:52] case they exist but they're flawed um so [40:54] um so definitely you know uh deploy an [40:56] Alex and then if you want to have zero [40:59] security problems you can actually [40:59] Deploy on S um sorry that was um so the [41:04] and the I was really kind of fixated on [41:07] that kind of the the ZFS um side of this [41:10] when when we go back to object storage [41:12] um and I hired this guy from from Amazon [41:14] Mark cavage who's a senior guy at AWS um [41:18] and you know he and I were kind of [41:20] mulling about this problem about how we [41:21] develop this kind of optic store and I [41:23] just could not stop thinking about ZFS [41:26] and he had the the key Insight which is [41:28] actually it's not about ZFS Dumbo it's [41:32] about OS level virtualization because [41:34] what OS level virtualization allows us [41:35] to do is build an object store all law [41:38] S3 HTP facing Object Store put put put [41:41] get get get but if I want to actually do [41:43] anything to those objects I want to [41:45] compute on those objects if I want do [41:47] something besides just retrieve them [41:49] instead of having to get it and then run [41:51] something on instead of having to get it [41:52] and GP say I can actually just spin up a [41:56] Zone spin up a container [41:57] where that object lives andp right where [42:00] the object [42:02] lives without actually moving the object [42:05] it's like [42:07] oh like [ __ ] why did I not think of [42:12] that uh and in fact it was funny is when [42:14] I was kind of going back and thinking [42:16] about this this is such a like kind of a [42:19] moment where you're just like wow [42:20] especially for me like I've been you [42:21] know living these Technologies for [42:23] decades it's like how could I not see [42:25] this but sometimes [42:27] often times um these big insights are [42:31] they actually um they seem simple only [42:34] in hindsight um that these big insights [42:37] actually require really a great Insight [42:40] um and the thing that was funny is I [42:42] remember vividly where I was when Jeff [42:45] first described ZFS to me um I was in my [42:48] poorly lit office Circ 1999 so I I [42:50] remember that vividly I remember vividly [42:53] where I was when I first described dra [42:55] trce to Jeff when I interviewed it son I [42:58] remember but I could not remember for [43:00] the life of me where I was physically [43:03] when Mark told me about the the idea [43:07] that would become Manta and he couldn't [43:09] remember it either well no no actually [43:11] he remembered it incorrectly like I'm [43:12] pretty sure like wasn't we weren't we [43:14] together at a bar like no no no that's [43:15] wrong I would remember that and so we [43:17] went back through IM logs trying to [43:19] figure out where the maybe we can just [43:20] like find the first time we were [43:21] referring to it and then go backwards [43:23] and then the I realized the reason I [43:24] couldn't remember where I was he told me [43:26] over I am so this is the actual sound of [43:32] my brain detonating um we we have [43:35] recorded for posterity on on the 24th of [43:38] October 2011 okay so he's got the crazy [43:40] idea we escalate M produced for first [43:42] class s by way routing your JS code some [43:44] no thing running on separate Zone on the [43:46] same place as your data um which is not [43:48] exactly what I said but it was kind of [43:49] obvious what the next step is not sure [43:52] if this would be bable or not and I'm [43:54] like [ __ ] that's a good idea and then I [43:56] just you know I really was not raised [43:58] this way but I I do kind of talk like a [43:59] sailor over here um the um apparently [44:03] this is like the highest praise I can [44:04] offer is to call someone a [ __ ] for [44:06] coming up with a idea which is really an [44:08] inappropriate thing to say to someone [44:09] who's just like had a really [44:10] breakthrough idea it's like you know [44:12] this is earlier You' been asking about [44:14] about AAL cross Zone and CFS um and then [44:17] of course you just get like why did not [44:18] think of that um so it's a great idea [44:22] it's a really really really good idea um [44:24] and so we built it um that was in [44:26] obviously that was October 24th of 2011 [44:28] uh and we spent all of 2012 actually [44:30] building this thing because you can't [44:32] it's unfortunately um wishing doesn't [44:34] make it so um so there's actually a [44:37] pretty sophisticated distributed system [44:38] you have to build on top of this thing I [44:39] mean yes you we use ZFS as as the [44:42] fundamental local store gender pitting [44:44] we use zones fundamentally um but we had [44:47] to go build a big distributor system on [44:48] top of it um and as we were building it [44:50] what was becoming clear is that like wow [44:53] what we've actually done here is is we [44:55] are actually leveraging [44:57] the Unix Tool Set uh to be able to go do [45:01] this kind of ad hoc data processing um [45:03] and I'm you know I I would like to think [45:06] of myself as an acolyte in the Church of [45:07] Unix so um I this was I I think um we be [45:11] kind you begin to see the kind of [45:12] historical context of that and for that [45:14] we're go back in time just quickly again [45:17] um the I assum you all know about Unix [45:20] um terms out sometimes I have to remind [45:23] people that Linux did not invent some of [45:24] these abstractions they actually [45:25] predated it was it's a very it's a good [45:28] Unix system but um Unix first first [45:31] appeared in the 1970s um and Unix was [45:34] actually a play on mulix um mulix was [45:36] the system that actually predated Unix [45:38] um mulix by the way created the forward [45:39] slash if you're curious among other [45:41] things um and the you one of the the the [45:44] the best things that happened in history [45:46] is the multic system that was running at [45:47] Bell Labs running the [45:49] g635 um that was taken away from them um [45:52] and they now were they had used multic [45:55] and they enjoyed it although they [45:56] thought it was a big [ __ ] some system and [45:58] now it was gone um and so they wanted to [46:00] go build something of their own um and [46:02] so what they actually did the funny [46:04] thing is uh and as as Dennis Richie say [46:05] we were oppressed a bit we were a bit [46:07] oppressed by the big system mentality of [46:09] course it was the 70s too and they were [46:11] all kind of hippie so there's that too [46:12] um and Ken wanted to do something simple [46:14] the thing that's kind of interesting [46:15] actually is that the um the the first [46:19] mention it there that the the first [46:21] programs actually were designed to test [46:23] the file system that's actually what [46:24] they actually implemented was a file [46:26] system and these commands were basically [46:28] throwaway commands to just verify that [46:30] the file system was working um commands [46:33] that you many of you have typed today um [46:36] are you you are you are using a you [46:38] using test driven development from [46:40] before you were born for most of you um [46:42] before I was born and or when you were [46:44] very very young um so the uh and Unix [46:49] the the thing that's kind of interesting [46:50] is that the the Unix philosophy did not [46:53] only start with Unix um the Unix philos [46:57] philosophy really started with Doug [46:59] millroy so I don't know if people know [47:00] who Doug millroy is uh Doug millroy is [47:03] the uh and Doug Mill Roy you think of [47:05] Ken Thompson and Dennis Richie as being [47:06] the avenders of Unix you really need to [47:08] put Dennis [47:09] Richie Doug Mill Roy rather um in that [47:13] Pantheon and the reason for that is [47:15] because it was it was mckill Roy's idea [47:19] that is the thing that we most associate [47:20] with Unix um and that is the actual [47:22] ability to pipe the output of one [47:25] command into the input of another um and [47:28] the thing that is funny talk about like [47:29] the I you know talk about like the the [47:31] Simplicity um sometimes not being [47:33] obvious this was obvious to to millroy [47:36] in 1969 this is two years before Unix [47:39] and he's he's got the uh he was he [47:41] called them I think hoses um it took him [47:44] three years to persuade Thompson [47:47] Implement he's literally like in the [47:48] next office um and you to kind of read [47:50] the history on this it sounds like [47:51] there's like a daily kind of beating um [47:54] and uh um and finally the the big [47:57] breakthrough was uh mckel Roy comes up [47:59] with a a syntax um that made it clean in [48:03] terms of piping it that syntax by was [48:05] not the pipe character I don't know if [48:06] people know what the syntax was it was [48:08] the carrot carrot yeah it was it was the [48:11] the circumflex very good the true old [48:13] school um the um but so the pipe came [48:17] very shortly thereafter um and then Ken [48:19] said I'm going to do it and of course it [48:21] took him a night which is like the [48:22] classic thing in software it's like you [48:23] spend 3 years telling me this is [48:25] technically impossible and you do it in [48:27] a day it's like um you know technically [48:31] impossible means I really really really [48:33] don't feel like it um and then I I I [48:36] love this [48:39] um this is like the Book of Genesis I I [48:42] I mean this [48:44] is to me I mean talk about like let [48:46] there be light and the next morning we [48:48] had this org of oneliner unfortunately I [48:50] think I don't actually know that we know [48:52] what morning that was um but um and [48:56] actually the thing that that's funny is [48:57] that I think what a one onliners are [49:00] have always been kind of very important [49:02] to me because they are the you're you're [49:05] got all the syntactic power that you're [49:06] putting in one line that you can kind of [49:08] understand um and I think any system [49:10] that has that results in an orgy of [49:12] onliners is on to something um and and [49:15] certainly and this is kind of the the ER [49:16] example of that um but truly uh the kind [49:20] of Unix was born I think that morning um [49:23] and and that gave rise to the unix's [49:25] philosophy and this is at verbatim the [49:28] way mckelroy uh talks about the the Unix [49:31] philosophy write programs that do one [49:33] thing and do it well write programs to [49:36] work together write programs that handle [49:38] text streams because that is a universal [49:40] interface [49:42] so that is I believe the most important [49:45] philosophy in systems development I [49:47] think that we should all apply this to [49:51] everything that we do the way we are [49:53] able to develop complicated systems is [49:56] by building out of simple ones taking [49:58] simple systems and actually and [50:00] connecting them in interesting and new [50:02] ways and the way you're able to do that [50:03] is by having programs that do one thing [50:05] and they do it well um and I think that [50:08] by the way not everyone believes this so [50:10] there's a bit of a tension I think [50:11] between and I as the older I get the [50:13] more I kind of view it as a holy war [50:15] between the forces of complexity and the [50:17] forces of Simplicity um the forces uh [50:20] making these big complicated for lack of [50:22] a better word and I'm sorry if I offend [50:24] anybody but I think it it is a frame [50:26] work Centric view versus a library [50:29] Centric view the library Centric view [50:31] says hey I'm going to come up and I got [50:32] I got a bunch of toolboxes for you got a [50:34] bunch of tools you can use them however [50:36] you want to use them framework the [50:38] framework point of view says okay you [50:40] pop into existence in this crazy ass [50:42] context where you know here are the [50:44] things you can kind do it's like where [50:45] am I it's like no no you fell through a [50:46] black hole you just kind of wake up here [50:48] um you're like okay all right well and [50:50] you're kind of in this you're kind of [50:51] like suspended in space um and I think [50:54] it makes it very hard to reason about [50:56] this system when you are implementing in [50:59] terms of of a framework and I don't want [51:01] to imply that Frameworks don't have [51:02] their place they do but I think when you [51:05] can build from in things in terms of the [51:07] unix's philosophy you always should I [51:09] and I think that this is the most really [51:10] the most important revolution in in [51:11] software thinking this gives rise to [51:14] what I call the epic rap battle of [51:16] Computer Science History so I I I had [51:19] this this knee injury earlier in the [51:20] year uh and I ended up on I ended up on [51:24] my my back for basically four weeks um [51:27] and I watched I think I watched the [51:28] entire [ __ ] internet um and the uh [51:31] but the If You Ever Seen The Epic Rap [51:33] Battles are really very funny um so the [51:36] uh and like Darth Vader versus Hitler [51:37] was particular good one um the um I [51:41] think it's like Tom SEO versus Larry [51:43] Ellison or whatever um but the uh the [51:46] one for for computer science is John so [51:48] John Bentley programming pearls and he [51:50] poses this problem read aile text [51:52] determine the end most frequently used [51:54] words and print out a sorted list of [51:56] those words words along with their [51:57] frequencies the amazing thing about this [52:01] challenge in [52:02] 1986 is that this is probably an [52:05] interview question for an intern [52:08] right I mean this is a question of like [52:11] you know have you played around with [52:12] Unix for a couple of months is basically [52:15] that is what that question is um people [52:17] saw like that I I thought kind of neat [52:19] that um the the CLI mystery on GitHub [52:23] where you you solve this little Mystery [52:24] by using basic CLI tools I think which [52:26] is cool you know it introduces people to [52:28] to the power of this but you would [52:29] expect someone who's able to do that to [52:31] be able to do this Don can's solution [52:33] the great Don can uh who you know love [52:37] obviously Don k a a one of the fathers [52:40] of computer science um but his solution [52:42] to this was an elaborate program in this [52:45] new system of course called Web um and [52:49] when seems you haven't heard of it for a [52:50] reason um this a a pascal like this is [52:53] the whole literate programming thing [52:55] which is like okay Don K like a lot of [52:56] very good ideas some stinkers literate [52:59] programming I'm going to put in the [53:00] stinker pile what have you um seven [53:02] pages of dense dense algorithm for this [53:05] problem and then like like Indiana Jones [53:09] pulling a [53:11] pistol Doug [53:13] millroy has this [53:16] solution um and the amazing thing is [53:20] that the the in [53:21] 1986 the performance was nearly [53:24] identical the the the performance [53:26] difference was almost the same and now I [53:29] mean you're very pressed to beat this [53:30] this is this this performs really really [53:32] well this performs really well cuz these [53:34] are simple tools doing one thing one [53:35] thing that you might so this is um you [53:38] know TR translates characters from this [53:40] is going from upper case to lower case [53:42] um and you can kind of see all the [53:44] things that it's doing here sorting [53:45] uning my C sorting minus RN and then I [53:48] think this is kind of funny um and then [53:50] the set over here to give you if if [53:53] assume that the top end the end that you [53:55] want is stored in do one we're doing a [53:58] said dollar1 Q which is the equivalent [54:02] of head they're just doing like a head [54:05] minus 10 or whatever why the hell didn't [54:07] he use head it's [54:09] 1986 head was not in AT&T Unix in 1986 [54:13] is that amazing uh I was I was actually [54:16] stunned by I was it took I I had to go [54:19] back and and check but sure enough head [54:20] was actually the head tail but not head [54:22] so go figure it anyway so the um what [54:25] we're seeing with big data [54:27] is that history is repeating itself so [54:30] remember that Bentley challenge here is [54:32] the challenge verbatim from the V map [54:36] reduce paper from the the dean paper the [54:39] so the count URL access frequency front [54:41] processes logs of web page requests and [54:43] outputs at uh Tuple URL comma 1 the [54:46] reduce function adds together all values [54:48] of the same URL emits a URL total count [54:51] pair wow that sounds a lot like [54:52] Bentley's challenge but the [54:55] solutions do not adhere to the Unix [54:58] philosophy the solutions give the Unix [55:00] philosophy a night that never forgets uh [55:02] the solutions are are are are are really [55:05] are a desecration of the Unix philosophy [55:09] uh and in particular they got a an [55:11] appendix that has in its like 150 lines [55:15] of C++ the kind of C++ has been like [55:18] really heavy heavily edited down to just [55:21] jam into 150 lines to do just this uh [55:25] then then right there's a 7 line word [55:27] count word count an idea that's so old [55:31] that the command to implement in Unix is [55:33] two characters [55:36] WC if you want to do a word count of if [55:41] you want to do a word count of a [55:42] document your first instinct is like [55:44] okay I need a C++ compiler it's [55:48] like what wait are you one of the [55:50] framework people how did you get in here [55:53] um it's like no you have WC well why is [55:56] it that when I do a word count on my [55:59] laptop it's WC but when I do a word [56:02] count against my logs in the cloud it's [56:05] all of a sudden it's C++ and it's Java [56:07] and it's Hadoop it's just wrong so what [56:10] what Manta allows us to do is and this [56:13] is actually [56:15] verbatim Bentley's challenge you can cut [56:17] and paste that in fact we'll do that in [56:19] just a second um and run this in terms [56:22] of man so what this is doing let me [56:23] actually go kind of run these some of [56:25] these things give you just a flavor for [56:26] what what Manta looks like [56:29] here so let's see people see that I hope [56:32] so um all of the uh the Manta commands [56:35] um of course because we are uh borrowing [56:38] from from Unix and multics before it uh [56:41] one of the differences between Manta um [56:43] which is the system that we develop it's [56:45] both object store that has computers of [56:46] first class operation the difference [56:48] between Manta and S3 is that we have [56:49] true hierarchical storage so S3 is not [56:52] hierarchical if you've use S3 it's just [56:55] it's a key Value Store it's one that [56:57] violates a bit of a gentleman's [56:58] agreement in that it accepts the forward [57:00] slash as a character so you you create [57:03] oh I'm going to create food barbaz and [57:05] it's like okay um and it creates that [57:08] you're like okay now I want to create [57:09] you know foodbar Bop and now I want to [57:11] like let me go see what's under food SL [57:13] bar it's like oh no no no that's not the [57:15] way I work really it's like well why did [57:17] you take the forward slash like I don't [57:18] know you wanted a forward slash um so [57:22] that's you know that I think that they [57:24] were I think if they were redoing us [57:26] today they would make it hierarchical [57:27] because it's too difficult otherwise um [57:29] but uh man is hierarchical and I'm just [57:30] going to list my public directory so be [57:33] Cal public um so what that did and you [57:35] can see that that's well you can see two [57:37] things one uh I have a bunch of crap in [57:39] my home directory because some things [57:40] never actually change um but um two you [57:44] can see that that is actually yes while [57:46] you're typing stuff is it actually going [57:49] yes yes yes yes and yes and actually [57:52] it's great I love being on the East [57:54] Coast I'm so close to ashur burn like [57:57] when I'm when I'm running this down I'm [57:58] like wow that's a rocket and I mean it's [58:00] it's fast on the west coast too but it [58:01] is really fast here um so yeah I mean [58:04] you're right to ask and I actually was [58:06] kind of like wow they said actually like [58:07] that's going over the internet right um [58:09] because we are so close to ashor here so [58:11] man is all in Us East um and that whole [58:13] speed of light thing there's a lot of [58:14] truth to it um so as it turns out um now [58:18] see this is why I shouldn't do a list of [58:19] my home what is Mana prawn that that [58:21] that we're not going to open [58:23] um I smart enough not to open that but [58:26] so um the so you can see that this is [58:29] this is it is super fast and I can [58:31] actually go do um I I can actually LS [58:35] these things as directories um and in [58:37] fact we people have written this is I'm [58:39] going into the I think these are all the [58:40] V this is the V7 took all the the V7 [58:44] stuff and uploaded to my public just to [58:45] give you kind of a bunch of files um and [58:47] we got the kind of things you would [58:48] expect so I can do like an M find of um [58:52] of bant Republic V7 um and that did a [58:55] find of that directory that so these are [58:57] all a bunch of objects that are um that [59:00] are under that directory so if I want to [59:02] actually get one of these I could like I [59:04] don't know let's um let's actually let's [59:06] go to um all right user is it user man [59:11] um have let's see we'll get one of these [59:13] Let's uh mget um what's what the hell is [59:17] what is peon why Unix why do you are [59:21] there still peon established packet [59:24] protocol all right [59:26] okay this was a thing apparently um yeah [59:29] so that's actually and actually should [59:31] be able to this might work would that [59:33] work oh my God so that talk about like [59:37] that was like a man page time machine I [59:39] just did I literally extracted a man [59:42] page from like the mid 70s and rendered [59:45] it on my local enroth on my local Mac um [59:48] so wow okay that that it all it all [59:50] worked um but the um so um and they even [59:54] have a l stand sign in like utf8 or [59:56] whatever man they are on it okay well [59:59] whatever um so that's an object I use [60:01] mget to get it um obviously this is a [60:03] small object but it could be you know it [60:05] could be a terabyte um we don't limit [60:07] you on object size um you can stream if [60:09] you have to do with big objects in S3 if [60:12] you don't have to do big goes S3 you [60:13] don't care about this if you do your [60:15] life is a living hell because of the [60:17] multi-part upload business and the way [60:19] the way you have to chunk these things [60:21] in for us you just stream so you just [60:23] open and and just pound us with dat [60:26] um and you if it's more than 5 gigs you [60:29] just need to tell us in advance how much [60:31] we should expect so we can go stream you [60:33] right onto a machine that's got enough [60:34] storage for for you that's it um but [60:36] other than that it just works so there's [60:38] there's an input um and you can input a [60:40] file you can pipe to input M get and so [60:42] on um and we can kick off an M job a [60:46] mantha job and so let's actually go do [60:47] that um so and then I can kind of show [60:50] you what this thing does so I'm going to [60:52] bring up a new window over here and I'm [60:54] just going to literally cut and paste [60:55] that so uh and I we will and I'm [60:58] actually but I'm not going to do the set [60:59] I can't force myself to type the stupid [61:01] said thing we're just going to do head [61:02] minus 10 for that um so um what that's [61:05] going to do is find a bunch of objects [61:08] and it ticked off a job and I'm going to [61:09] let that actually run in the background [61:11] uh and I'll show you what some of those [61:13] parts look like um so if we do you saw [61:15] the M find um and the M job create is [61:19] running that where all those objects [61:21] live and to give you kind of a flavor [61:22] for that actually um let me let's go [61:25] look at some of these object objects and [61:26] to kind of give you a flavor what you [61:27] can do one of the things that we want [61:29] that we actually invented um is so one [61:32] of the early things that we started [61:33] using this for is the ability to [61:35] actually go debug core and crash dumps [61:38] so the when you debug a core or crash [61:40] dump the let's see if I got I don't [61:41] think I got an example coming up later [61:43] the when you go to debug a core or crash [61:45] dump the the the dump is this huge kind [61:48] of thing and you want to be able to go [61:49] debug it interactively it's a great [61:51] thing for insitu compute because I don't [61:52] want to move this big dump around well [61:55] the problem is that I want to debug it [61:57] interactively and so I don't want to [61:59] just have a job that just goes off and [62:00] Reps this and returns I want to actually [62:03] go I I I want to log in interactively to [62:06] the Zone that's sitting on this object [62:08] and fortunately I complained about this [62:10] just loudly enough um that Josh PL one [62:12] of my colleagues at joyant invented this [62:14] thing called M login so what M login [62:16] does is it actually logs in to so it's [62:20] creating a job I have now logged into [62:24] the [62:24] object [62:29] yeah so if I look at my [62:33] environment what I have is Manta input [62:37] input file is this object so if I I can [62:40] do a file of manta input [62:45] file and it will say that it's wow file [62:48] knows that it's t pre-processor output [62:51] ancient history detective um so um the [62:56] that file was just here and if I wanted [62:58] to operate on [63:00] it not that there it is but I'm actually [63:04] sitting on the machine I'm I'm local to [63:06] the machine I can't see anything else so [63:09] if I look at at what's mounted here what [63:11] we have done is we we can only see [63:14] obviously only my object and I'm in a um [63:18] actually one of the things that's been [63:19] kind of aming I actually had someone [63:20] tell me it's like you know I love Manta [63:22] because you guys finally stopped trying [63:25] to to you know the your whole like smart [63:27] OS thing you finally built something [63:29] that is like Soup To Nuts Linux and I'm [63:31] like uh what what are you talk what are [63:33] you talking about it's like well I did [63:35] an M log in there and I'm sitting in [63:36] Linux which is great I'm like right [63:40] right yeah exactly well you know what [63:41] you know we just yeah we we finally you [63:44] were right after all these years um the [63:46] reason they thought that is because if [63:49] you do if you look at grep you're [63:51] getting the gnu grep if you use tar [63:54] you're using the we get the ganu tar [63:56] when you we we give we don't give you I [63:58] mean we still and so if you do a witch [64:00] tar actually it's it's op local bin tar [64:03] if you still want like the ancient tar [64:05] it's actually still sitting in user bin [64:07] um if if you want to go back in time um [64:10] and in fact actually all the crazy [ __ ] [64:12] sitting here so um you know you've got [64:15] all sorts of ancient garbage sitting in [64:17] user bin that you don't use lot actually [64:19] this is always an entertaining thing to [64:20] go do [64:22] um there the uh these are all the three [64:25] letter commands of which I probably know [64:26] a third of those and I'm sure there are [64:29] at least like 10 commands that that no [64:31] one ever like BFS there's a classic um [64:34] atq what the hell is [64:38] that [64:40] atq oh look at that I can actually [64:42] display my ad [64:44] jobs thank you [64:46] 80s um all right actually what was funny [64:50] is speak at was the at was the the uh [64:54] one of the pieces of software that had [64:55] the leape bug actually um that good old [64:58] at that's the last time at and I cross [65:00] paths um there you go people have used [65:02] atq that's good um but so you actually [65:03] you have all the ancient Unix software [65:05] but it's actually out of the way in [65:06] terms of a path perspective and the the [65:08] software that you actually use is all [65:09] the all the G software that if you use a [65:11] you're getting a new o and so on so it [65:13] it it feels much more familiar and what [65:15] you're going to find is that you just [65:17] that you're able and actually it's [65:18] amazing when you are and here you can [65:20] see my job finished over here um that [65:23] this is the answer across those 37 [65:25] inputs these are the top words that is [65:28] that is the the John Bentley challenge [65:30] running on Manta the thing is that most [65:32] of the time that was spent executing [65:34] this was actually doing all the [65:35] bureaucracy of getting into those zones [65:37] and actually getting those jobs running [65:39] this would not run in a materially [65:41] different amount of time if these things [65:43] were hundreds of megabytes or gigabyte a [65:45] piece because the data is not moving the [65:48] the data is is sitting on on those [65:49] machines um so what we found is that [65:53] this opens up a lot of possibility [65:56] um and we are actually busily using man [65:59] to change the way we develop and deliver [66:00] software so I mean the speaking for [66:03] myself um as a software disaster [66:06] fetishist um the uh the ability to [66:09] actually keep cord dumps in perpetuity [66:12] keep crash jumps in perpetuity and debug [66:14] them um so we developed a system called [66:16] Thoth um and tho is a A system that [66:19] allows us to that that using mlogin and [66:21] so on where we can actually store every [66:23] corn crash jump that happens in the the [66:25] the public cloud and then and constantly [66:28] analyze them for and actually I I say [66:31] every corner crash in the in the public [66:33] Cloud which actually was my intention um [66:35] I hadn't really spent much time in the [66:37] general prison population before um I [66:40] did turn that on and all of a sudden [66:41] like PHP was basically dossing my system [66:45] um because that thing C dumps 24/7 [66:48] 365 that is like a disc stress test for [66:51] cord dumps uh and cord dumps in all [66:53] sorts of wild ways um so so we we do [66:56] actually limit it I've got a white list [66:58] um but it's actually very important to [66:59] me that if you are a a public CL [67:01] customer if Ruby cord dumps if Java cord [67:03] dumps if Airline cord dumps if Airline [67:06] [ __ ] I actually just contate ritual [67:07] suicide you actually looking at Air [67:09] Airline cordum but uh it's airline [67:11] cordum are a little bit a little bit [67:12] obtuse um but if those things cor up I [67:15] want to understand why um and certainly [67:17] if you have a if we have a crash and [67:20] oper some crash we want to understand [67:21] why now we've actually got we've got one [67:24] place where we go to debug all that [67:25] we're doing that with logs we do that [67:26] instrumentation in fact the the Rubicon [67:28] for Manta was we have to bill for Manta [67:31] sorry um if it were up to me it would [67:32] just be free but the uh we we bill um we [67:36] just to get this out of the way um we [67:38] charge S3 prices for storage so it's S3 [67:42] has set the tone for for price for [67:45] storage and it's dirt dirt dirt dirt [67:47] cheap so it's S3 prices for storage and [67:49] effectively ec2 prices for your compute [67:52] but you only pay for the seconds um that [67:54] you actually run so if you run a job [67:56] that takes 10 seconds um then you pay [67:59] for 10 seconds of compute and it is like [68:01] it's 40 micro cents per second I mean it [68:05] turns out to be like you have to do that [68:07] actually a decent number of times um I [68:09] mean you still can do a surprising [68:10] amount of work though I mean if you have [68:11] a million objects and you run for half [68:13] an hour on each of those objects um you [68:15] can all of a sudden the years can go by [68:18] is that building resolution for man or [68:22] Jo um the right now it's it's uh well [68:25] joyant compute is I mean we do that by I [68:27] mean I think we round up to the nearest [68:28] hour for infrastructures of service um [68:31] and if if there were um Google made a [68:33] big deal of of getting down to a tighter [68:36] granularity we could do it I don't [68:38] really and I found it peculiar for for [68:41] mans that makes a lot of sense you only [68:42] want to charge per the second because [68:43] you create these jobs so [68:45] transiently in Google making a big deal [68:48] out of it I thought was a bit strange [68:49] it's a bit like a hotel making a big [68:50] deal by the fact that they charge by the [68:52] hour um it's like well out of curiosity [68:55] like what kind of clientele is attracted [68:57] to the buy the hour pricing [68:59] butes yeah I yeah I guess but the um the [69:04] I mean for us it's like I mean but an [69:05] hour of I mean it is it is like so we do [69:07] round up to like the nearest 14 cents in [69:09] the infrastructur for sur side but [69:11] actually I mean we could chargy the [69:12] second actually we just don't do it [69:13] because we don't think there's demand [69:15] for it but if you want it actually we [69:16] should go do that um we should go I mean [69:19] I think we we should probably add up [69:20] with the differences for all customers [69:22] in the The Joint public cloud my guess [69:24] is that I've got that number like in my [69:26] pocket [69:27] um um but anyway um so the um so but in [69:32] order to build for Manta um you billing [69:36] is such a [ __ ] annoying problem um [69:39] and the um right exactly it's really [69:41] regrettable that we have to actually [69:43] take money in exchange for goods and [69:44] services um and actually the way this is [69:48] done for S3 at Amazon and the way this [69:50] is done in a lot of places for this kind [69:51] of a service is hey it's Hadoop time [69:54] it's time for a big map ruce cluster [69:57] from the mother of all Frameworks um and [70:01] embarrassingly when we were thinking [70:03] about we start bilding I'm like oh God [70:05] are we going have to stand up like a [70:06] Hadoop cluster for this and Mark was [70:07] like are you stupid like no you [ __ ] [70:10] we're going to use mantha for it and so [70:12] we actually use mantha uses itself to [70:15] implement billing so what mantha does it [70:17] is it when you do any requests what have [70:20] you records are stored in Manta you can [70:23] actually go to your your if I if I do an [70:25] MLs of just be Canal um I will actually [70:29] get um reports and in those reports I [70:31] can actually get usage [70:35] logs you can log into your bill you [70:39] can [70:40] yeah yeah let's go yeah if you want yeah [70:42] I mean if you if if logging into a bill [70:45] is something that like you get your [70:46] rocks off on let's just do it right now [70:49] um like here we go it's Bill time feel [70:53] the wind in your [70:54] hair [70:57] uh it's right here the bill is right [70:58] here and you can like print that out in [71:00] beautiful Json how leing uh there ah [71:04] there you actually you can delete it [71:05] because this this actually build is this [71:08] is only this is like the analog here the [71:12] true analog here is uh Verizon I don't [71:15] know what you're talking about because [71:16] when your bill showed up I burned [71:19] it okay well okay that's okay that's [71:22] that's fine then you probably don't know [71:23] how much you owe us uh because you so [71:25] you can tort your bill if you want and [71:27] the this is actually only effectively [71:29] this is a copy of your invoice and you [71:30] can actually delete this um As You Wish [71:34] um it's only there for your purposes we [71:36] got another copy unfortunately it's like [71:38] yeah you know one step ahead on that one [71:41] um the [71:43] um the um but what what we do do is we [71:48] because this actually the records look [71:49] the same for us so we do these big [71:51] mantha jobs that run on all these Json [71:54] payloads and then we use Json which is a [71:56] tool that joint engineer TR mck wrote um [71:59] that we put user Ben but does not back3 [72:03] um the uh we use a Json tool to kind of [72:06] pick out the the particular things we're [72:07] interested in and then we actually the [72:09] engineer did this originally did that [72:12] that um reduce phase in node which is [72:14] fine node does it fine um but we're [72:17] looking at at the code like this is [72:18] actually pretty complicated you could [72:20] probably replace this with like five [72:22] lines of a he he was like a tell me more [72:26] um and it's like yes child so we were we [72:29] um we had like Sunday school for a um [72:32] where we read like [72:34] actually when I finally go around the [72:36] bend I probably will have my own Public [72:37] Access Channel where I have like the the [72:39] Christian puppets for like introducing a [72:41] to no baby use AR um that's actually a [72:45] very good idea I think I actually why am [72:48] I here instead of doing that um so a [72:51] puppet time theater I love o Puppet Time [72:53] theater I'm I definitely the Puppeteer [72:55] um and so we introduced a to to this [72:57] young engineer like oh W this is great [72:58] and now it's time terms a so now it's [73:00] it's just basically Unix command line um [73:03] to do this stuff um and what we're so [73:07] that was kind of crossing a Rubicon for [73:08] Manta when we used when that the head [73:11] started eating the tail we started [73:12] actually using man that Implement itself [73:14] um and we realized that the the power of [73:16] doing this stuff because you get you can [73:18] build these these ad hoc uh you don't [73:21] have to build a map ruce cluster anymore [73:23] you just store your data and you can do [73:25] all this ad hoc iteration just the way [73:27] you do on the Unix command line um so [73:29] it's it's extraordinar powerful it's [73:31] very interactive you've got M login um [73:33] the way we develop these things is we M [73:35] log in and you get your pipeline working [73:37] just show and then you come back and [73:38] then you create an M job out of it [73:40] create on kind of one object okay that [73:41] works good then then you kind of pipe in [73:43] 10 objects look like it's working well [73:45] and then you pipe in a thousand or [73:46] 10,000 or a million or what have you um [73:49] so it's uh it's it for us um it's been a [73:52] real Revolution um and I just to that [73:55] that's my email we could do that as well [73:57] but that'd be a bit awkward um the [74:01] um in terms of the the the cap tradeoffs [74:03] with a lot more details and I'd be happy [74:05] to um to I want let us get to the bar at [74:08] some point but I'd be um happy to go [74:10] into details on this um and but Dave [74:12] Dave P one of the lead Engineers on [74:13] Manta he's got a terrific blog entry [74:15] talking about the cap trade-offs um [74:17] recall cap is consistency availability [74:20] partition tolerance um you get to only [74:22] pick two of those um of course [74:24] everyone's like okay great I pick [74:26] consistency and availability it's like [74:28] no no no no no you have to pick [74:29] partition tolerance no so you are either [74:33] a CP system or an AP system for us we [74:37] actually split it in terms of reads [74:38] versus rights so for rights um we are a [74:41] CP system for reads we are an AP system [74:44] what that means is when you do a write [74:46] you do a write by default you can [74:47] specify the number of copies by default [74:49] you specify two copies they go to mantis [74:51] in two different data centers sitting in [74:53] Asher Virginia every time the start [74:55] tracking through Ashburn I guarantee you [74:57] all of cloud computing gets nervous um [74:59] because like those are the failures that [75:00] actually aren't supposed to happen so um [75:02] provided Ashburn does not actually get [75:04] wiped off the map um you've got two data [75:07] centers that are in two different fault [75:08] domains um and the the you will be Act [75:12] Your Right will be act back when that [75:15] object is in both of them if the object [75:18] is not in both of them you will not get [75:20] act back you will not get a 200 this is [75:22] in contrast to S3 S3 will 200 you back [75:26] when it's written it to enough and then [75:28] the then things will catch up except [75:31] when they don't so if you use S3 a lot [75:34] one of the things you may notice is you [75:36] do a put 200 get 404 like wait a minute [75:41] you just told me you put it there why [75:44] when I get it are you telling me you [75:45] can't find it it's like you do have it [75:47] right and it does show up eventually um [75:49] and you know usually seconds sometimes [75:52] minutes depends on sometimes tens of [75:54] minutes if AWS is having a bad day um we [75:57] don't do it that way we you're not going [75:59] to get if we can't write to both sides [76:01] we know that when data center is down [76:02] you'll get a 500 back because a 500 we [76:04] think you know what to do with a 500 [76:05] like we don't have it retry later um as [76:08] opposed to like a 200 that's like a just [76:10] kidding 200 which is like how the hell [76:11] you supposed to do anything with that um [76:13] on reads however if you only need one of [76:15] those data centers to be up to be able [76:16] to do those reads um we've got a we've [76:19] got H sword you saw that I mean thanks [76:21] to molx um we've also got a this snap [76:24] link phenomenon that allow that allows [76:26] you to affect versioning very easily [76:28] cores headers and so on um the SDK is a [76:31] node you can npm install Manta for the [76:33] command line interface that I was using [76:35] um I think I think from our perspective [76:37] um we believe that the the idea that [76:40] Manta represents namely bringing compute [76:43] to data is absolutely the path forward [76:47] for Big Data um the the size of data is [76:51] growing so dramatically and the size of [76:54] compute the the actual program text the [76:57] a oneliner it's like those onliners are [77:00] just as true now as they were in mckill [77:02] Roy's day right those are not growing it [77:05] it does not make sense to back haul data [77:08] from optic storage in the transient [77:10] compute so we believe that I I don't [77:12] think that man is going to be the only [77:13] system that does this um I'm sure [77:14] there's going to be some better FS lxc [77:17] knockoff in like 50 years but the um [77:20] would actually be great I actually you [77:21] know honestly I I I would I would [77:23] welcome other systems that have the same [77:24] model um because um it would verify that [77:26] we're not entirely crazy um but the um [77:29] we personally believe that CFS is the [77:31] only SAN storage under pending for that [77:32] system um we think that manth is just [77:34] the first of these systems we think [77:35] we're going to see more U and we're [77:37] actively using mantha mantha is changing [77:38] the way we develop software um so um I [77:41] think that man is going to change we [77:43] think um the way a lot of people think [77:44] about Big Data it already has in the six [77:46] months that it's been out yes what are [77:49] the current constraints around the [77:51] environment that you [77:53] canun [77:56] uh in terms of resource constraints [77:58] physical constraints just Environ in [78:00] general like I mean there's a pretty [78:03] sweet conclusion that you can come to [78:04] when you can log into your because you [78:07] can start running arbitrary programs [78:09] yeah yeah absolutely so one thing I [78:11] actually didn't show here but you can [78:12] run arbitrary programs what you can do [78:14] is you can actually specify that with a [78:15] job you don't even need to log into your [78:16] object to do that when you create a job [78:18] you can give it an asset payload which [78:20] is basically a tarball that'll be laid [78:21] down and that'll be laid down for you [78:23] automatically and so now I mean and [78:25] actually you know if you want to do uh r [78:28] on big data right there is no real R [78:31] system they're kind of some half-ass [78:33] kinds of things out there but you can go [78:34] do r on Big Data running at Hardware [78:36] speed you if you wanted to go do go on [78:39] Big Data if you wanted to go I mean if [78:41] you wanted to go use these things that [78:43] aren't Java frankly on Big Data you can [78:46] now you you go use Python for big python [78:50] you look at all these communities there [78:51] are you know the python in particular [78:54] you've got the you know the natural [78:55] language people have all gravitated to [78:57] python right and they're all kind of [78:59] building tooling for one another in [79:01] Python if you want to do big data [79:03] natural language it's not it's a [79:06] nonstarter to do it in any other [79:07] language besides python now you can do [79:08] all that with this you to an object how [79:14] much right that's a great question so [79:16] the um when you um when you create your [79:19] job you give it some resource [79:21] constraints and as you can imagine like [79:23] that changes your bill um so um by [79:26] default uh if actually this is why by [79:29] default I think we give you one gig but [79:31] the easiest way to um so yes we give you [79:35] one gig by default um so by default [79:38] you've got one gig um by default you [79:41] have um I I think we give you eight uh [79:45] gigs or more I can't remember of storage [79:47] um I guess I could [79:49] do will that tell me yeah 8 gigs there [79:51] you go that's not really big data well [79:54] no [79:55] I work with like I pull up like objects [79:57] and data files that are like four terab [79:59] no no no sorry these are this is only in [80:02] in effectively uh this is your [80:05] mapper right you you've got four [80:07] terabytes you're going to stream that [80:09] four terabytes through a giga physical [80:11] is more than enough to to stream through [80:13] a now if you where where the dam now [80:16] just a couple things one your your dam [80:18] is you you can size your Dam to be [80:20] larger right so if you want 64 gigs of [80:23] dam especially in your reducers right [80:25] not in your mappers but where this [80:26] becomes an issue is when you have you [80:28] got a big reducer we allow you all with [80:32] Hado um we allow you to have and just [80:34] just show you the way this is specified [80:37] I just feel a little nervous because [80:38] like previously your example of B data [80:41] was a bag of words and like I'm sorry [80:43] I'm a scientist and I work in High group [80:46] of sequencing and uh it's not a bag [80:49] anymore hone oh yeah no no I know like a [80:51] really funny example like if like any [80:53] other I don't know who else is a [80:55] scientist here well well we just like [80:58] hello world is not a real program right [80:59] so yeah but if you want me to take you [81:02] seriously you can't use that as an [81:04] example okay so the um so you I actually [81:07] would recommend I mean if you haven't [81:09] read the dean paper you should because [81:10] the dean paper uses the same [81:15] examples okay so yeah I mean sorry if [81:17] the examples are too small for you I'm [81:19] happy to take your arbitrary size data [81:20] we can go to town because the um the um [81:24] no actually I mean we we can do demos on [81:26] terabyte siiz objects um they in terms [81:29] of uh you know they um well actually DNA [81:33] in particular if you're actually looking [81:34] at sequencing um a and TS and C's and [81:36] G's what's [81:37] that yeah so um that is a total sweet [81:41] spot for us um because in fact actually [81:43] and we we've got folks that are are are [81:45] currently uh actually storing in fact [81:47] you should go look at uh Christopher hog [81:49] who is um currently work joyant but was [81:51] actually a scientist as of like six [81:53] months ago um and started using mantha [81:55] for his proteomics um and he was was [81:59] enjoying what he was doing so much and [82:00] he was kind of sick of science for a lot [82:02] of good reasons um he's now working at [82:05] joyant and so he's did a lot of the r [82:07] integration um so if you go look at [82:09] Chris's stuff and I can assure and you [82:10] and CR have a conversation but um I can [82:12] assure you that we we know what big data [82:14] is so when I'm looking at at at was like [82:17] a funny example you know sure so but [82:19] when I mean big data for me admittedly [82:22] big data for me tops out at probably [82:23] about 100 gigs which is about the the [82:25] biggest kind of Crash jump you get no [82:27] but no no 100 gigs for a single object [82:29] no but remember this that's only for a [82:30] single object so when you and I mean [82:33] even for you you wouldn't want your your [82:35] your a single object you can have a [82:37] terabyte single object but but that's [82:39] that's probably Overkill you're limiting [82:40] the amount of parallelism you can [82:42] actually get on that thing um and you [82:44] probably want to frame that object [82:45] differently certainly if you're in hdfs [82:47] you're framing that object 100 Meg [82:48] boundaries right I mean you think of it [82:50] as as a terabyte object but it's [82:52] actually framed at much smaller [82:53] granularity so for in Manta the object [82:56] framing is much more deliberate design [82:57] decision um but no the the like with [83:00] with you know Hadoop certainly other [83:02] systems there these things scale [83:04] arbitrarily with data size so as much [83:06] data as you've got um you you don't have [83:09] an internet pipe big enough to to fill [83:12] man faster than we can accommodate you I [83:14] can guarantee you [83:16] um what's that example just for the [83:19] [ __ ] scientist in the back you know [83:21] just like so if there's an [ __ ] [83:22] scientist in the back just like be [83:26] be assur you can still your [83:28] [Music] [83:36] [Music] [83:51] yes yeah so I mean that if you're do [83:54] well if you're standing up that much [83:56] compute so I mean if you're standing up [83:58] compute for weeks at a time um you want [84:01] to be I mean that's a better fit for [84:03] infrastructures of service workloads [84:05] anyway I mean to be honest with you like [84:07] if that were your problem we would well [84:09] a couple things one we would talk about [84:12] um I mean is your data in the cloud [84:13] today or is it is it all3 all it's all [84:16] it's an all S3 yeah so I mean if if you [84:18] wanted to do that on joint like if don't [84:20] let pricing get in your way we would [84:21] figure out a way to give you I should we [84:24] if you wanted to have like a little [84:25] asterisk call on there um which is I [84:27] mean if you're going to be running that [84:28] much compute for that for that longer [84:30] period of time um but I mean it's [84:32] basically it's E2 price it right that [84:34] pricing is E2 [84:36] pricing it's it's 14 cents an [84:40] hour 14 cents an hour is a for I mean [84:43] that runs for years well no but that [84:46] that I mean maybe it's it doesn't [84:48] compare to you know Reserve business [84:49] prices but that is basically ec2 [84:52] pricing I I compete against ec2 you have [84:54] to trust me on the E2 price sh and um so [84:57] I I mean that and and it's you know we [84:59] can argue about you know how expensive [85:02] or or or cheap that is but that is it's [85:04] kind of it's going right for compute but [85:06] we charge you on on kind of second gr [85:08] orative um so we we should ad yeah if [85:11] like said don't get hung up on that [85:12] though um if you want if you want to [85:15] give us a lot of data and a lot of [85:16] compute let's figure out how to do it um [85:18] because that's what the system is [85:19] designed to do and is doing right now so [85:21] I mean there are lots of folks that are [85:22] actually using this lot for lots of data [85:26] um [85:28] are okay I'm happy to accommodate ask [85:31] all questions I I God knows I've asked [85:33] plenty of my own life so um so I mean [85:37] just in terms of [85:39] um of wrapping up wherever my slide went [85:43] off to um I think it's very unhelpfully [85:46] no where did it well all right forget it [85:48] Google Manta there it is uh well you can [85:50] Google Manta and that's probably the [85:51] best way to do it but um uh lots more [85:53] information and would say that you know [85:55] if you got particular domains interested [85:56] in um like scientific Computing um we [86:00] should um go into more detail there and [86:03] I'll Point some stuff that that that [86:04] we've done there specifically for that [86:06] but this is what we're doing now [86:07] effectively is going around the [86:08] different domains um the one that's a [86:10] really natural fit uh first kind of [86:12] natural fit is log processing for sure [86:14] uh if you got log processing this is [86:15] just that's what this was made to do um [86:18] it does it very very well but we've got [86:20] um we've got folks doing video analytics [86:23] on it we've got folks do I mean it's you [86:25] know not a day passes where we don't see [86:27] some strange job light up man you're [86:30] like wow what the hell is that and where [86:32] somebody's think has come up with some [86:34] creative thing to go do um fortunately [86:36] that that creativity can't really extend [86:38] the Bitcoin mining because Bitcoin [86:39] mining is actually not profitable on [86:40] this thank god um the um but um the lots [86:46] and lots and lots of of of applications [86:48] this I think one of the exciting things [86:49] about this Mar perspective is that man [86:51] is one of those technologies that um [86:53] appeals to the creat ity of its users um [86:55] and so people have been and will [86:57] continue to surprise us with what [86:58] they're doing with this because it is a [86:59] general purpose compute platform um and [87:01] you can do anything you can do with [87:02] general purpose compute you can do on [87:06] Mana yes questions do I understand [87:09] correctly that um you usually always do [87:13] everything in in jobs like remote jobs [87:15] like everything everything you type on [87:17] your own console you basically start [87:19] with an ampy or in something and then [87:22] everything from that point on into [87:24] different that's right that's right so [87:28] in in the example that we ran over here [87:30] where we we launched a job um the or [87:34] we'll go back over here um so what this [87:36] is doing is that M find is it it is just [87:39] creating that list of objects um and [87:41] then it's piping that through locally to [87:43] M job create now if that were a you may [87:46] say well wait a minute why am I taking [87:48] that list of objects hauling it out of [87:51] the cloud and then sending the list back [87:53] so you could actually have an initial [87:55] phase of this that did that M find so if [87:58] you want to you could have a map map [88:00] ruce where the the first phase of that [88:02] job does does an M find and then the M [88:04] find then the whole thing would happen [88:06] on on the cloud it's just a matter of [88:08] you know uh and there's all as with Unix [88:10] there's a lot of kind of flexibility [88:12] about how you do what and depends on [88:14] what you're trying to [88:16] do yes what's your biggest issue with [88:19] ZFS now and and sort of tangentially uh [88:23] did you ever tempted like try to get in [88:25] the [88:27] firmware uh if I get in the firmware [88:30] it's going to be with a suicide v um the [88:32] um that's like that's like your thing [88:34] though [88:36] right uh yeah I guess Jihad maybe um the [88:41] um um our the the biggest issue with ZFS [88:45] for us [88:48] um I think that we are um we're pushing [88:51] ZFS in the New Dimensions with we do [88:53] roll backs all the time because when you [88:56] land in a Zone when I when I exit from [88:59] this m login session there's going to be [89:01] a ZFS roll back behind me that's going [89:02] to roll that zone back to what it was [89:04] previously um so we're doing um so [89:07] previously you do a roll back kind of [89:09] like to play around you do a roll back [89:10] to kind of go restore from things you [89:12] don't really have like roll back in the [89:13] data path um we do now so we're hitting [89:16] roll back hard um and that's been great [89:19] and we found you know found and fixed a [89:21] couple of issues there um but you know [89:23] by and large um the only real way to [89:25] find ZFS issues these days is to push it [89:29] into some wholly new dimension like that [89:30] to take some aspect of the system that [89:32] like exists but but but has not been [89:35] just leaned on and go actually lean on [89:37] it um but CFS continues to be Rock Solid [89:39] for us um the um and we you know [89:44] fortunately we we've got the expertise [89:46] to um to continue to go develop that um [89:49] in terms of of firmware um I I think the [89:52] firmware crisis is never going to end I [89:53] think that we're going to have just i' [89:55] given up I mean I think that the um we [89:58] have to deal with very unreliable [90:00] Hardware components I mean I think that [90:01] the disc firware situation actually got [90:02] a little bit better um I would say that [90:05] if you are I would never buy the first D [90:07] of a firmware of anything for sure um [90:09] that's I think what I learned there um I [90:13] don't know if that's a satisfying answer [90:14] or not yes I was wondering can only work [90:17] at one object time so if I would like to [90:19] have I can have one M block and or M job [90:22] that works on [90:24] no no that's how it works so this m job [90:27] is actually operating on all 300 of [90:28] those objects creating 300 zones 300 [90:31] zones okay and so so this so I explain [90:35] this so this map phase this TR pipe TR [90:39] pipe to Unique that is landing on 300 [90:41] zones in [90:43] parallel okay asterisk not true Big Data [90:47] I know are just Man pages um maybe I'll [90:50] create some terab size Man pages just so [90:52] we can um [90:55] the [90:56] um uh and then the reduce phase I've got [90:59] one reducer here so this is running in [91:02] one zone SP