AWS Divides Cloud Fleets Into Cells to Stop Cascading System Crashes

Amazon Web Services

Distributed failures are inevitable under physical and logical constraints, requiring operators to compartmentalize workloads through cell architectures and shuffle-sharding so toxic requests cannot trigger region-wide collapse.

AI-assisted summary of AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures (ARC338) — original source. Check the original for full context.

Source published: 2018-11-27

Key ideas

AWS structures fault isolation across four core tiers—Region Isolation, Availability Zone Independence, Cell-Based Architectures, and Shuffle-Sharding—to contain both physical and logical disruptions. (4:23)
Cell-based architectures impose strict ceilings on cluster growth, capping the catastrophic blast radius of a software failure or poison pill to 1/N, where N represents the number of cells. (22:16)
Shuffle-sharding leverages combinatorial mathematics to assign customers overlapping virtual subsets of nodes; scaling from 8 nodes to a 100-node cluster with 5-node shards produces over 75 million combinations, ensuring 77% of tenants experience zero impact from a toxic tenant. (34:46)
Global services like AWS IAM maintain static stability by strictly decoupling the centralized control plane from regionally cached data planes, allowing runtime authentication to persist uninterrupted during global outages or network partitions. (51:51)

Detailed notes

## Background Published by Amazon Web Services on YouTube on November 27, 2018, this presentation from AWS re:Invent 2018 explores AWS's operational practices and system availability philosophies. The session breaks down core blast radius reduction design techniques—including region isolation, availability zone independence, cell-based architectures, and shuffle-sharding—informed by decades of operating Amazon.com and AWS cloud infrastructure.

## TL;DR AWS Distinguished Engineer Peter Vosshall explains how AWS architectures are engineered around fundamental constraints like the speed of light, the CAP theorem, and Murphy's Law by actively containing the blast radius of inevitable failures. The talk outlines four containment tiers: fully isolated regional stacks, Availability Zone Independence (AZI) with decoupled control planes, cell-based bulkheading that caps catastrophic failure impact to 1/N of a fleet, and shuffle-sharding combinatorics that virtualize single-tenant resilience on multi-tenant hardware. Vosshall pairs these system designs with strict operational rigor, detailing one-zone deployment pipelines, automated canary rollbacks, and AWS's hands-on 'builder-operator' culture before reviewing real-world edge cases like the 2008 Amazon S3 bit-flip incident.

## Atmosphere & style Peter Vosshall presents with the calm, authoritative candor of a veteran systems architect who has navigated high-severity outages firsthand. The narrative balances high-level naval bulkhead metaphors and combinatorial mathematics with understated dry humor regarding 'pesky speed of light issues' and customer poison-pill requests. Watching the session feels like sitting in on an executive engineering post-mortem where catastrophic worst-case scenarios are systematically dismantled into elegant, repeatable operational doctrine.

## Bulletin - [04:23](https://www.youtube.com/watch?v=swQbA4zub20&t=263s) AWS structures fault isolation across four core tiers—Region Isolation, Availability Zone Independence, Cell-Based Architectures, and Shuffle-Sharding—to contain both physical and logical disruptions. - [22:16](https://www.youtube.com/watch?v=swQbA4zub20&t=1336s) Cell-based architectures impose strict ceilings on cluster growth, capping the catastrophic blast radius of a software failure or poison pill to 1/N, where N represents the number of cells. - [34:46](https://www.youtube.com/watch?v=swQbA4zub20&t=2086s) Shuffle-sharding leverages combinatorial mathematics to assign customers overlapping virtual subsets of nodes; scaling from 8 nodes to a 100-node cluster with 5-node shards produces over 75 million combinations, ensuring 77% of tenants experience zero impact from a toxic tenant. - [51:51](https://www.youtube.com/watch?v=swQbA4zub20&t=3111s) Global services like AWS IAM maintain static stability by strictly decoupling the centralized control plane from regionally cached data planes, allowing runtime authentication to persist uninterrupted during global outages or network partitions.

## Key takeaways - [00:01](https://www.youtube.com/watch?v=swQbA4zub20&t=1s) Distributed systems engineering is fundamentally constrained by three unavoidable properties: the speed of light, the CAP theorem, and Murphy's Law. - [00:01](https://www.youtube.com/watch?v=swQbA4zub20&t=1s) Blast radius is quantified across three dimensions: the number of customers affected, the fraction of functionality impaired, and the geographic scope of the impact. - [06:50](https://www.youtube.com/watch?v=swQbA4zub20&t=410s) AWS regions operate under a shared-nothing model with independent control and data planes, preventing outages from propagating across regional boundaries. - [06:50](https://www.youtube.com/watch?v=swQbA4zub20&t=410s) Cross-region features like VPC Peering rely on an isolated Cross-Region Orchestrator that interacts with standard regional front-door APIs rather than letting control planes link directly. - [11:23](https://www.youtube.com/watch?v=swQbA4zub20&t=683s) Availability Zones are spaced miles apart to eliminate correlated physical risks like power drops and severed fiber, while staying close enough to support low-latency synchronous replication. - [15:36](https://www.youtube.com/watch?v=swQbA4zub20&t=936s) Under Availability Zone Independence (AZI), zonal resources maintain autonomous zone-local control and data planes so that failure in one zone leaves neighboring zones fully operational. - [18:30](https://www.youtube.com/watch?v=swQbA4zub20&t=1110s) Cell-based architecture borrows the naval bulkhead concept, deploying multiple independent instantiations of a service behind an ultra-thin routing layer to cap software failures at 1/N. - [24:21](https://www.youtube.com/watch?v=swQbA4zub20&t=1461s) Enforcing maximum size caps on individual cells enables exhaustive pre-production stress testing and simplifies fleet management by scaling out via new cells rather than scaling up. - [27:46](https://www.youtube.com/watch?v=swQbA4zub20&t=1666s) Effective cell routing requires 'cutting with the grain' along natural workload boundaries like customer IDs, VPCs, or physical data centers to minimize expensive scatter-gather and batch operations. - [34:46](https://www.youtube.com/watch?v=swQbA4zub20&t=2086s) Shuffle-sharding uses deterministic hashing to map tenants to unique combinations of nodes, dramatically lowering failure impact without requiring fully isolated physical infrastructure. - [34:46](https://www.youtube.com/watch?v=swQbA4zub20&t=2086s) To prevent cascading failures in a shuffle-sharded cluster, node assignments must remain strictly immutable, prohibiting automated dynamic failover to unassigned healthy hosts. - [39:42](https://www.youtube.com/watch?v=swQbA4zub20&t=2382s) AWS strictly enforces a rule never to mutate more than one Availability Zone at a time, deploying software through one-box canaries, soak times, and automated metric rollbacks. - [42:35](https://www.youtube.com/watch?v=swQbA4zub20&t=2555s) Under the 'builder-operator' culture, software engineers who build services are responsible for operating them on-call, driving continuous post-incident improvements through the Correction of Errors process. - [46:01](https://www.youtube.com/watch?v=swQbA4zub20&t=2761s) The 2008 Amazon S3 gossip protocol outage demonstrated how a single flipped bit on one server NIC bypassed an unchecksummed layer and triggered a fleet-wide cascade. - [48:49](https://www.youtube.com/watch?v=swQbA4zub20&t=2929s) While regions and AZs are customer-configurable primitives, AWS deliberately abstracts physical data center buildings within an AZ and treats cells and shuffle-sharding as platform-managed safety mechanisms. - [50:46](https://www.youtube.com/watch?v=swQbA4zub20&t=3046s) Phased, multi-stage deployments across cells and regions can span days or weeks, making backward and forward wire compatibility a critical engineering requirement during prolonged version skew. - [53:48](https://www.youtube.com/watch?v=swQbA4zub20&t=3228s) Cell architectures are best suited for hard state, whereas shuffle-sharding thrives in stateless or soft-state services, with hybrid patterns increasingly layered together in systems like DynamoDB.

## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=swQbA4zub20&t=1s) Welcome, Laws of the Universe, and the Meaning of Blast Radius

Peter Vosshall, a Distinguished Engineer at AWS, opens by welcoming attendees to AWS re:Invent 2018. Reflecting on his career designing highly available, large-scale distributed systems, he explains that engineers must constantly contend with three unyielding properties of the physical and computational universe:

1. **The Speed of Light:** Light travels at approximately 186 miles per millisecond. While fast, distance imposes unavoidable latency limits on communication. He mentions that earlier that morning, the NASA Jet Propulsion Laboratory (JPL)—an AWS customer—successfully touched down their Mars lander, a feat complicated by interplanetary light-travel delays. 2. **The CAP Theorem:** Postulated by Eric Brewer in 1998 and later formally proven by Seth Gilbert and Nancy Lynch at MIT, the theorem shows that a distributed system cannot simultaneously provide consistency and availability under network partitions. 3. **Murphy's Law:** Vosshall observes, “Inevitably things will fail in a variety of interesting ways.”

Because security, availability, and durability are the top priorities at AWS, systems must be built to assume Murphy’s Law will strike. This introduces the concept of the *blast radius*. Failure in distributed systems is rarely binary; it is not simply working or failing, but rather a question of scope and degree of impact. Blast radius measures impact across three main dimensions: how many customers or workloads are affected, what fraction of functionality is broken, and across what geographic scope the disruption takes place (a rack, an entire data center, or a whole region).

AWS relentlessly works to shrink this blast radius. Vosshall highlights the internal *Correction of Errors* (COE) post-mortem process, which service teams undergo following any operational event to identify root causes and preventive actions. A core requirement in the COE template is a mandatory thought exercise: “How could you cut the blast radius for a similar event in half?”

---

[04:23](https://www.youtube.com/watch?v=swQbA4zub20&t=263s) Forms of Failure: Physical, Environmental, and Logical

Failures manifest in both physical and non-physical ways, and resilient architectures must anticipate both:

- **Hardware and Network Faults:** Servers crash, storage disks fail either outright or through silent, random I/O errors, and network switches or routers can introduce random bit flips in transit. - **Environmental Disruptions:** Outside the data centers, municipal or construction utility workers accidentally sever fiber cables, severe electrical storms trigger utility power drops, and catastrophic events like fires or storms physically compromise facilities. - **Traffic Overloads:** Sudden massive demand spikes or Distributed Denial of Service (DDoS) attacks can overwhelm system capacities. - **Poison Pills (Black Swans):** Problematic requests that are unusually resource-heavy or trigger latent edge-case software bugs. These are especially hazardous because failed client applications automatically retry, causing a single "poison pill" to cascade through internal dependencies and infect an entire service fleet. - **Deployments and Bugs:** Flawed software deployments, bad configuration pushes, or unintended software bugs can inadvertently compromise production stability.

To protect against these fault modes, Vosshall outlines four architectural pillars—**Region Isolation**, **Availability Zone Independence**, **Cell-Based Architecture**, and **Shuffle-Sharding**—complemented by disciplined operational practices.

---

[06:50](https://www.youtube.com/watch?v=swQbA4zub20&t=410s) Region Isolation and Cross-Region Orchestration

Vosshall begins with the highest containment boundary: region isolation. At the time of the presentation, AWS operates across 19 geographical regions globally, with 5 additional regions announced. Customers choose regions to optimize latency or comply with data residency regulations.

Each AWS region is constructed as a completely separate, independent stack of AWS services operating under a shared-nothing architecture. Services have their own dedicated regional API endpoints (for example, `ec2.us-west-1.amazonaws.com` versus an endpoint in `us-east-2`). There is no single centralized global Amazon EC2 control plane or data plane; Amazon SQS, Amazon SageMaker, and Amazon EC2 instances in one region possess zero runtime awareness of instances in another. Because the stacks do not communicate, a failure in one region cannot directly trigger a failure in another, providing the ultimate blast radius boundary.

```mermaid flowchart TD accTitle: AWS Inter-Region VPC Peering Architecture accDescr: The Cross-Region Orchestrator coordinates isolated EC2 control planes across distinct regions via external APIs.

subgraph CRO["Management Layer"] Orch["Cross-Region Orchestrator (Safety Ring-Fencing & Workflow Engine)"] end

subgraph RegionA["AWS Region: us-west-1"] APIA["EC2 Regional API Endpoint"] VPCA["Customer VPC A"] APIA --> VPCA end

subgraph RegionB["AWS Region: us-east-2"] APIB["EC2 Regional API Endpoint"] VPCB["Customer VPC B"] APIB --> VPCB end

Vosshall explains how AWS maintains this isolation even when offering inter-region features such as Amazon S3 Cross-Region Replication, Amazon DynamoDB Global Tables, and Inter-Region Amazon VPC Peering. Taking VPC Peering as an example, establishing a virtual network link between VPCs in different regions involves coordinating customer approvals and updating routing tables in both control planes. Rather than permitting the EC2 control plane in `us-west-1` to speak directly to the EC2 control plane in `us-east-2`, AWS deploys a dedicated higher-level service: the **Cross-Region Orchestrator**.

The Orchestrator manages the workflow and sends requests into each region through its standard public-facing API front door, behaving like any other authorized caller. It incorporates safety mechanisms and strict "ring-fencing" to ensure that an operational error or bug cannot synchronously destabilize both regions at once. Even during physical disasters—such as earthquakes in Japan—the failure blast radius is guaranteed to stop at the regional boundary. However, because a region-wide outage is still unacceptable for customers hosting single-region workloads, AWS relies on Availability Zone independence to partition failures within a region.

[11:23](https://www.youtube.com/watch?v=swQbA4zub20&t=683s) Availability Zones, Physical Isolation, and Regional Topology

At the macro scale, an AWS region—such as Northern Virginia, Dublin, or Frankfurt—is not a single data center. Rather, it is composed of multiple distinct data centers dispersed across the metropolitan area, termed Availability Zones (AZs). These AZs are interconnected via high-speed, private fiber links. The physical placement follows a careful balance: they are spaced miles apart to drive the probability of a correlated failure down to near zero—barring widespread catastrophes like major regional earthquakes. By placing them miles apart, localized hazards like a passing tornado or an isolated municipal utility outage will not knock out multiple facilities at once, especially since AWS provisions them across independent utility suppliers within the region.

Simultaneously, the zones are kept close enough together to avoid latency penalties and circumvent “pesky speed of light issues.” Logically, systems can treat them as if they are co-located, easily supporting synchronous replication protocols. Each AZ is not restricted to a solitary building; because AWS enforces a fixed maximum physical size for its data centers, larger AZs may comprise several buildings. Across regions, AWS constructs at least three Availability Zones per region (a baseline guaranteed for all future region builds), which provides the quorum required for distributed consensus protocols to function reliably.

```mermaid flowchart TD accTitle: Regional Multi-AZ Topology accDescr: An AWS Region containing three distinct Availability Zones separated by miles yet interconnected by private fiber. subgraph Region["AWS Region (e.g., Northern Virginia)"] subgraph AZ1["Availability Zone A"] DC1["Data Center Building(s)"] P1["Utility Supplier 1"] end subgraph AZ2["Availability Zone B"] DC2["Data Center Building(s)"] P2["Utility Supplier 2"] end subgraph AZ3["Availability Zone C"] DC3["Data Center Building(s)"] P3["Utility Supplier 3"] end AZ1 <-->|"High-Speed Low-Latency Private Fiber"| AZ2 AZ2 <-->|"High-Speed Low-Latency Private Fiber"| AZ3 AZ1 <-->|"High-Speed Low-Latency Private Fiber"| AZ3 end ```

- Globally at the time of this presentation, AWS operated 19 regions containing 57 total AZs, with an additional 5 regions and 15 AZs planned for deployment. - Spreading data centers ensures that common disruptions—such as a construction worker severing a fiber line or localized power loss—do not cross AZ boundaries. - Minimum three-AZ designs satisfy consensus agreement requirements for distributed systems while avoiding single points of failure.

[13:40](https://www.youtube.com/watch?v=swQbA4zub20&t=820s) Multi-AZ Applications and Zero Blast Radius for Infrastructure Events

Because correlated failure across a region is minimized by design, application architectures can achieve fault tolerance by mirroring AWS's internal operational practices. In a standard multi-AZ deployment, compute instances run behind an Elastic Load Balancer (ELB) distributed across zones, backed by a relational database configured in a primary-standby pair across AZs.

If an entire Availability Zone drops offline due to a power outage or network severance, the ELB detects the failure and immediately halts traffic to the impaired zone. Meanwhile, Amazon RDS automatically initiates a database failover to the healthy standby located in the unaffected zone. Because failure detection and failover happen rapidly, the event registers as little more than a transient hiccup to end customers, effectively achieving “almost like a non-event that you lost, you know, one of the data centers in your application.” AWS relies on this exact model for durable, highly available regional services like Amazon S3, which utilizes multi-AZ distribution to reach its 11 nines of durability.

As the speaker emphasizes regarding this architectural model: > “And so with this architecture you can basically get to zero blast radius when you have a data center and get hit by tornado or a utility worker hitting a cutting out fiber connection.”

[15:36](https://www.youtube.com/watch?v=swQbA4zub20&t=936s) Zonal Services and Availability Zone Independence (AZI)

Certain AWS services are not purely regional abstractions; they offer zonal resources such as Amazon EC2 instances or Amazon EBS volumes. For these, customers explicitly choose the AZ where resources reside. To enforce minimum blast radius boundaries, AWS implements a design principle known as Availability Zone Independence (AZI), pairing physical isolation with independent control planes at the zone tier.

To illustrate this, the speaker contrasts control planes and data planes: - The **control plane** provides the administrative interface used to configure and manage resources (e.g., running `ec2:RunInstances`, which ingests an API call and orchestrates VM provisioning). - The **data plane** carries out the actual day-to-day workload (e.g., the running EC2 instance itself, SSH connections, or the network fabric that bridges instances within a VPC).

```mermaid flowchart TD accTitle: Availability Zone Independence Architecture accDescr: Regional Entry Point routing to independent zonal control and data planes across AZs. subgraph RegionalLayer["Regional Tier"] RCP["Regional Control Plane (Multi-AZ Resilient)"] Agg["Aggregation (e.g., DescribeInstances) & Non-Zonal Config (Security Groups)"] RCP --- Agg end subgraph AZ_A["Availability Zone A"] ZCP_A["Zone-Local Control Plane"] DP_A["Zonal Data Plane (EC2, EBS)"] ZCP_A --> DP_A end subgraph AZ_B["Availability Zone B"] ZCP_B["Zone-Local Control Plane"] DP_B["Zonal Data Plane (EC2, EBS)"] ZCP_B --> DP_B end RegionalLayer --> ZCP_A RegionalLayer --> ZCP_B ```

Under AZI, each zone runs its own fully autonomous control and data planes that do not share state with neighboring zones. When customers interact with EC2 via a regional endpoint, a multi-AZ fault-tolerant regional control plane serves as the routing facade. It handles zone-agnostic artifacts (such as security groups) and aggregates cross-zone queries (such as `ec2:DescribeInstances`). If a failure strikes an AZ, the zonal data plane and zone-local control plane within that zone may be lost. However, neighboring AZs remain unaffected, and the regional control plane continues routing API calls to the healthy zones without interruption.

[18:30](https://www.youtube.com/watch?v=swQbA4zub20&t=1110s) The Theoretical Blast Radius Dilemma and Bulkhead Compartmentalization

While multi-AZ isolation eliminates correlated infrastructure failures, it introduces a separate vulnerability when looking at the worst-case scenario. If an infrastructure event takes down an AZ, a regional service automatically shifts traffic away, remaining healthy, whereas a zonal service sustains impact strictly within the damaged zone.

However, considering theoretical blast radius—the "Black Swan event" or catastrophic non-infrastructure software bug—the profile flips: - For a purely zonal service, the worst-case failure boundary is still restricted to that single zone. - For a regional service, a critical software fault, poison-pill payload, or control-plane bug can bring down the entire service region-wide.

The speaker highlights this acute operational anxiety: > “For the regional service it is the entire service, and that’s the thing that we lose sleep at night every night thinking about.”

To resolve this asymmetry, AWS turned to an engineering metaphor from naval architecture. For centuries, ocean-going ships have been constructed with watertight compartments separated by transverse bulkheads. When a ship's hull is breached, flooding is isolated to a single bulkhead compartment, preserving buoyancy and overall structural integrity.

```mermaid flowchart LR accTitle: Cell-Based Architecture Compartmentalization accDescr: Traffic partition from entry router into isolated cells containing independent instances of the full application stack. Traffic["Customer Ingress Traffic"] --> Router["Regional Entry Point / Partitioning Router"] subgraph Cell0["Cell 0"] ELB0["ELB"] --> App0["App Stack"] --> DB0["Storage"] end subgraph Cell1["Cell 1"] ELB1["ELB"] --> App1["App Stack"] --> DB1["Storage"] end subgraph CellN["Cell N"] ELBN["ELB"] --> AppN["App Stack"] --> DBN["Storage"] end Router -->|"Partition: Customer Group 0"| Cell0 Router -->|"Partition: Customer Group 1"| Cell1 Router -->|"Partition: Customer Group N"| CellN ```

Translating this bulkhead concept into cloud services yielded *cellular architecture* (or *cell-based architecture*). Rather than maintaining one monolithic regional service stack spanning the AZs, AWS deploys multiple, completely independent instantiations of the entire stack—each known as a "cell."

Incoming regional traffic passes through an entry-point aggregation and routing layer that partitions workloads across these cells, such as grouping requests by customer identifier. The cells operate with zero knowledge of one another. The speaker notes with a grin that while the audience members “all look nice,” if a single “naughty” user introduces a malicious request or triggers a fatal edge-case bug, the ensuing failure remains quarantined within their assigned cell, leaving all other customer workloads unharmed.

[22:16](https://www.youtube.com/watch?v=swQbA4zub20&t=1336s) Principles of Cell-Based Architectures and Failure Containment

To preserve the appearance of a single, unified endpoint while partitioning underlying systems, AWS places a thin routing layer—called a cell router—in front of the fleet. The cell router makes incoming routing decisions to direct traffic to distinct, self-contained sub-units known as cells. These internal structures remain completely transparent to customers while delivering high resilience and fault tolerance. Conceptually, this mirrors the pattern of Availability Zone Independence (AZI), but operates along an orthogonal dimension.

```mermaid flowchart TD accTitle: Regional Cell-Based Architecture accDescr: Incoming traffic passes through a cell router to discrete cells distributed across Availability Zones.

Client["Client Traffic"] --> Router["Cell Router (Thinnest Possible Layer)"]

subgraph Cells["Cell Fleet (1/N Blast Radius)"] Router --> Cell1["Cell 1 (AZ-a, AZ-b, AZ-c)"] Router --> Cell2["Cell 2 (AZ-a, AZ-b, AZ-c)"] Router --> Cell3["Cell 3 (AZ-a, AZ-b, AZ-c)"] end ```

In a traditional regional service, resilience is achieved by distributing components across multiple Availability Zones (AZs) so that the loss of an entire AZ does not drop the service. A cell-based architecture retains that cross-AZ resilience inside each cell, but adds a second layer of containment for internal software bugs, poison pills, or systemic issues. As the speaker highlights: “The other examples of failure, the ones that we lose sleep at night over, the failure in the cell-based service, it's contained to the cell, so the impact is 1 over N, where N is the number of cells, rather than the whole service.” While diagrams frequently illustrate three cells for simplicity, real production systems scale to much higher values of N, driving the overall impact of a failure down to a tiny percentage of the total customer workload.

This technique is applied not only to regional services, but also to zonal services—most notably within the Amazon EC2 control plane: - Dividing zonal services into sub-cells preserves expected resilience during an AZ failure while providing additional blast-radius reduction. - Because physical AZs often consist of multiple distinct data centers, aligning zonal cells directly with the underlying physical data center infrastructure bounds hardware-correlated failures to an even smaller blast radius. - Internal software bugs within a single zone remain constrained to that specific zonal cell rather than impacting the zone in its entirety.

[24:21](https://www.youtube.com/watch?v=swQbA4zub20&t=1461s) System Properties, Scaling Limits, and Cell Sizing Trade-offs

Beyond catastrophic failure containment, cell-based architectures provide critical day-to-day operational benefits, such as workload isolation against noisy neighbors. A fundamental advantage lies in how these systems scale. Rather than relying on traditional vertical scaling or endlessly growing a massive monolithic cluster, AWS imposes strict architectural ceilings: “Cells, like our data centers, have a maximum size; we won't let them grow past a certain point, and so if we need to continue to grow a system, rather than growing the cells past that point, we'll add another cell.”

Capacitating cells to a fixed maximum ceiling fundamentally alters system predictability: - **Exhaustive Testing at Cap:** Because cells have an absolute upper bound on size, teams can deploy realistic test harnesses to stress-test cells to the breaking point, proving exactly how each unit behaves under peak load. - **Operational Manageability:** Inspecting logs, troubleshooting node degradations, and querying fleet states is substantially easier on a smaller subset of nodes than across an unbounded fleet spanning millions of resources. - **Scale-Out Growth:** System expansion is achieved through adding more standardized cells, not by running uncharted configurations at unprecedented scale.

Engineering a cell-based system involves balancing the trade-offs of cell sizing:

Despite these operational tensions, AWS maintains a clear guiding principle: “There's no right answer here, except that, all things being equal, we will always prefer the lower blast radius to these other considerations.”

[27:46](https://www.youtube.com/watch?v=swQbA4zub20&t=1666s) The Routing Layer, Partitioning Dimensions, and Cross-Cell Operations

Because the cell router sits in front of the entire cell fleet, it represents a single shared ingress point. If the router fails, the system immediately regresses to a regional outage. To prevent this, AWS treats the router with extreme discipline. Teams invest heavily in stress-testing and battle-hardening the component to survive extreme "Black Swan" events. The guiding design rule is radical simplicity: “When we talk to teams about adopting a cell-based architecture, we call this component the 'thinnest possible layer' to kind of reinforce it should be really simple.”

Determining how to partition traffic across cells requires evaluating how workloads flow. While partitioning by AWS account or customer ID is common, the EC2 control plane partitions cells based on physical data center infrastructure. In other architectures, VPC boundaries are selected—especially where cross-account VPC interactions exist. The overarching design advice is to always "cut with the grain":

```mermaid flowchart TD accTitle: Cutting With the Grain in System Partitioning accDescr: Visual analogy contrasting splitting wood along its natural grain versus splitting against it.

subgraph Good["Cut With the Grain (Natural Boundaries)"] Nat["Customer ID / VPC / AZ Hardware"] Nat --> Clean["Clean, uncoordinated routing Low shared state"] end

subgraph Bad["Cut Across the Grain (Unnatural Boundaries)"] Cross["Arbitrary cross-cutting state"] Cross --> Entangled["Frequent cross-cell dependencies Complex coordinated logic"] end ```

Inevitably, some workloads cannot be cleanly isolated and require cross-cell operations, which must be strictly minimized to protect the router's simplicity: - **Scatter-Gather Queries:** Read APIs that interrogate multiple cells simultaneously and consolidate replies, such as the Amazon EC2 `DescribeInstances` API. - **Batch Operations:** Mutation APIs that target resources residing across disparate cells in a single call, such as an EC2 `TerminateInstances` request with multiple instance IDs. - **Coordinated Writes & Cell Migration:** The most complex scenarios involve moving customer workloads from one cell to another to balance hot cells or backfill newly launched capacity.

To execute cell migrations safely, AWS mirrors the mechanics of live virtual machine migrations: 1. An invisible clone of the workload state is instantiated in the target cell. 2. The clone continuously syncs updates from the actively mutating source cell. 3. At the final moment, traffic to both is briefly frozen to capture the terminal synchronization delta. 4. An atomic routing flip shifts all traffic to the target cell, managed cleanly at the router layer.

While the catastrophic events that cause widespread outages are exceedingly rare, the investment in cell-based architectures ensures that when Black Swan scenarios do materialize, the blast radius is reliably constrained to $1/N$.

[32:56](https://www.youtube.com/watch?v=swQbA4zub20&t=1976s) Transition to Shuffle-Sharding

While cell-based architectures provide robust isolation for stateful systems, AWS employs an even more powerful technique for stateless or soft-state services: shuffle-sharding. Like cell architectures, it focuses on blast radius containment, but leverages combinatorial mathematics to dramatically minimize the impact of poisoned requests or noisy neighbors without requiring an overwhelming multiplication of physical infrastructure.

[33:20](https://www.youtube.com/watch?v=swQbA4zub20&t=2000s) From Monolithic Pools to Cell-Based Architecture

To ground the discussion of blast radius reduction, the speaker presents a hypothetical setup: eight backend nodes serving requests distributed by a load balancer across eight distinct customers. Playing into the Las Vegas setting of re:Invent, the customers are represented with gambling and card-suit icons. The danger begins when one customer—designated as the "diamond" customer—introduces a toxic workload. Whether it is an unusually expensive query or a "poison pill" request that triggers an edge-case software bug, the result is fatal: diamond sends a request, and a server crashes.

```mermaid flowchart LR accTitle: Blast Radius Comparison accDescr: Diagram comparing monolithic architecture against cell-based architecture when a toxic workload hits.

subgraph Shared["Monolithic Shared Fleet (100% Impact)"] direction TB C1["Customer Diamond (Toxic)"] --> LB1["Load Balancer"] LB1 --> N1["Node 1 (Crashes)"] LB1 --> N2["Nodes 2-8 (Cascade Failures)"] N2 -.->|"All Customers Impacted"| Imp1["100% Blast Radius"] end

subgraph Cells["Cell-Based Architecture (25% Impact)"] direction TB C2["Customer Diamond (Toxic)"] --> Cell1["Cell 1 (Nodes 1-2)"] Cell1 -.->|"Only Cell 1 Down"| Imp2["25% Blast Radius"] OtherC["Other Customers"] --> Cell2["Cells 2-4 (Nodes 3-8)"] Cell2 --> Healthy["Healthy & Isolated"] end ```

In a traditional shared pool, the remaining seven servers take on the load, but diamond does not stop. The customer retries, systematically knocking down node after node until the entire fleet collapses. In this shared environment, the blast radius is 100% of the customer base—the ultimate worst-case scenario. To prevent this complete outage, systems evolve into a cell-based architecture where capacity is divided into discrete, independent partitions. By assigning a specific subset of customers to each cell, diamond's toxic traffic is trapped strictly within its assigned boundary. If eight nodes are split into four cells of two nodes each, diamond can only take down its own cell, instantly shrinking the blast radius from 100% down to 25%—a fourfold improvement.

- Blast radius in a standard cell partition is calculated simply as: $\text{Blast Radius} = \frac{\text{Number of Customers}}{\text{Number of Cells}}$. - Increasing resilience in this model requires either adding more cells or migrating to a more advanced partitioning scheme.

---

[34:46](https://www.youtube.com/watch?v=swQbA4zub20&t=2086s) The Mathematics and Mechanics of Shuffle-Sharding

When a system does not require pinning specific customers to dedicated physical infrastructure, it can leverage shuffle-sharding. Instead of grouping users into static physical silos, shuffle-sharding assigns each customer to an overlapping virtual shard consisting of a small subset of nodes chosen pseudo-randomly using deterministic hash functions. In an eight-node cluster where each customer is assigned a unique pair of two nodes, diamond might be assigned to nodes 1 and 4, while other customers are scattered across other two-node pairs.

```mermaid flowchart TD accTitle: Shuffle-Sharding Node Overlap accDescr: Flowchart illustrating how eight nodes are shared via combinations so non-toxic customers survive node outages.

subgraph Cluster["8-Node Cluster"] N1["Node 1 (Down)"] N2["Node 2 (Up)"] N3["Node 3 (Up)"] N4["Node 4 (Down)"] N5["Node 5 (Up)"] N6["Node 6 (Up)"] N7["Node 7 (Up)"] N8["Node 8 (Up)"] end

D["Diamond (Assigned 1 & 4)"] -.->|Crashes Both| N1 D -.->|Crashes Both| N4

H["Hearts (Assigned 1 & 2)"] -->|Fails| N1 H -->|Succeeds via Retry| N2

C["Clubs (Assigned 4 & 5)"] -->|Fails| N4 C -->|Succeeds via Retry| N5 ```

When diamond crashes nodes 1 and 4, other tenants sharing those machines are not completely knocked offline. For instance, customer "hearts" might be mapped to nodes 1 and 2; while node 1 is dead, node 2 remains completely healthy. Provided the client is fault-tolerant and retries failed requests, hearts survives the outage. Across the eight-node fleet, the total number of two-node combinations is governed by the binomial coefficient:

$$\binom{8}{2} = \frac{8!}{2!(8 - 2)!} = 28\text{ unique combinations}$$

Because there are 28 distinct combinations, a toxic tenant taking down its two assigned nodes affects only $\frac{1}{28}$ of the virtual shards, reducing the catastrophic failure blast radius to just 3.6%. While roughly half the fleet might experience a transient hiccup from sharing a single degraded node, only identical combinations face total service loss.

When scaled up to production proportions, the combinatorial power becomes staggering: - Expanding the cluster from 8 nodes to 100 nodes and assigning each customer a virtual shard of 5 nodes yields: $$\binom{100}{5} = \frac{100!}{5!(95!)} \approx 75{,}287{,}520\text{ combinations}$$ - When a bad customer knocks out all 5 of its assigned nodes out of 100, 77% of all other customers share zero nodes with that bad actor and experience zero impact. - 99.8% of customers retain a healthy majority (at least 3 out of 5) of their nodes, meaning they can route around the failure seamlessly without customer-visible downtime. - Only an astronomically tiny fraction of users—those dealt the exact same five-card hand out of over 75 million combinations—suffer a complete outage.

As the speaker emphasizes: “What's magical about this and it's all in the math is we've created a multi-tenant system and then used the shuffle sharding to create a single basically a single tenant experience which is obviously what AWS aspires to do.”

Executing shuffle-sharding in practice requires strict structural rules: - **Fault-tolerant clients:** Clients must implement automatic retries with backoff so that when requests to an impacted node fail, they retry against their remaining healthy nodes. - **Fixed assignments without failover:** Node assignments must remain immutable; dynamic failover to alternative healthy nodes must never be permitted. As the speaker warns, allowing a poisoned workload to fail over to new nodes breaks isolation and allows cascades to infect the entire system. - **Resource versatility:** Shuffle-sharding applies beyond compute instances to worker queues, database partitions, and storage endpoints. - **Routing layer:** Traffic can be steered using an intelligent shuffle-sharding proxy or through customer-specific DNS hostnames that resolve directly to the client's assigned set of IPs, delivering routing at virtually zero operational cost.

---

[39:42](https://www.youtube.com/watch?v=swQbA4zub20&t=2382s) Operational Rigor: One-Zone Deployment Rules and Staggered Waves

Architectural boundaries mean nothing if operational mechanisms do not actively respect them. Layered on top of cell-based isolation and shuffle-sharding is a disciplined operational philosophy regarding deployments and changes. The speaker quotes Werner Vogels' writing on compartmentalization: “I'd wager that every new AWS engineer knows within their first week if not their first day that we never want to touch more than one zone at a time.” If teams mutate multiple Availability Zones simultaneously, the foundational promise of independent fault domains collapses under the weight of correlated, human-induced failure.

```mermaid flowchart TD accTitle: Staggered Software Deployment Pipeline accDescr: Sequence diagram illustrating AWS's progressive canary and staggered deployment methodology.

subgraph Stage1["Phase 1: Canary Verification"] OB["One-Box Test / 5-10 Hosts"] --> Syn["Synthetic Canaries & Integration Tests"] end

subgraph Stage3["Phase 3: Broad Regional Rollout"] AZ3 --> Reg2["Next Region"] Reg2 --> RegN["Global Fleet"] end

subgraph Rollback["Automated Safety Interlock"] Alarms{"Alarm Breached?"} -.->|Yes| AutoRB["Automatic Instant Rollback"] end

AZ1 -.-> Alarms AZ2 -.-> Alarms Syn -.-> Alarms ```

To maintain these blast radius barriers, software deployments proceed in strictly gated, staggered waves: - **Fractional / One-Box Deployments:** Changes launch first to a single server (a "one-box" deployment) or, in the case of Amazon EC2 hardware fleet updates, across an initial batch of five to ten physical machines. - **Synthetic Canaries and Continuous Testing:** During rollouts, automated test suites and continuous Canaries—software harnesses running end-to-end user workflows that invoke actual public APIs—monitor system health. - **Automated Rollbacks:** If metric thresholds degrade or Canaries register anomalies, deployment pipelines halt immediately and roll back without manual intervention while engineers investigate. - **Staggered Blast Boundaries:** Releases move progressively through cells, across one Availability Zone at a time, and then across regions over extended bake periods, balancing delivery velocity with rigorous failure containment.

---

[42:35](https://www.youtube.com/watch?v=swQbA4zub20&t=2555s) Automation and the Builder-Operator Culture

The discipline applied to deployments extends across every layer of cloud infrastructure management. Manual configuration of network devices or out-of-band credential rotation introduces unpredictable human error. By encoding these operational changes in software automation, routines are subjected to code reviews, pre-production testing, and deterministic, repeatable execution.

This rigorous automation is sustained by Amazon's end-to-end ownership model. AWS eschews the traditional organizational split between product developers and external operations teams. Instead, service teams consist of engineers acting as "builder-operators": they design, write, test, deploy, and support their own services in production 24/7.

This creates an immediate operational feedback loop. When a failure occurs, the engineers who wrote the software are the ones responding to the alerts and conducting the post-incident analysis. They author the Correction of Errors (COE) documents, examining root causes and identifying architectural flaws. Because the engineers who run the post-mortems are the same ones who write the code, they are uniquely empowered to alter service topologies and ensure that the blast radius of any future recurrence is cut in half or eliminated entirely.

[44:31](https://www.youtube.com/watch?v=swQbA4zub20&t=2671s) Summary: Compartmentalization, Shuffle-Sharding, and Operational Practices

The speaker concludes the presentation by summarizing the tiered compartmentalization mechanisms AWS uses to contain failures and restrict blast radius. The containment model begins at the highest level with independent AWS regions and the strict isolation enforced between them. Inside each region, the architecture subdivides into Availability Zones, followed by cell-based architectures that slice services into self-contained operational units. On top of that physical partitioning lies shuffle-sharding, which leverages mathematical combinatorics to map customers across subsets of resources, generating virtual cells that isolate customer impacts from one another.

```mermaid flowchart TD accTitle: AWS Blast Radius Containment Hierarchy accDescr: Diagram showing the layered containment boundaries from regions down to shuffle-sharding, protected by operational practices. subgraph IsolationLayers["Compartmentalization Layers"] R["Regions (Strong Isolation)"] AZ["Availability Zones (Fault Isolation)"] CELL["Cells (Zonal & Regional Partitioning)"] SS["Shuffle-Sharding (Virtual Cells & Combinatorics)"] R --> AZ --> CELL --> SS end subgraph OperationalShield["Operational Protection"] Deploy["Automated Phased Deployments"] Ops["Builder-Operators On-Call"] end OperationalShield -.->|"Protects & Enforces"| IsolationLayers ```

To prevent failures from jumping across these boundaries, AWS reinforces this compartmentalization with rigorous operational disciplines. Systems rely on automated, progressive, multi-stage deployments rather than broad, simultaneous updates. Furthermore, service teams operate under a "builder-operator" model where the software engineers who write the code run it in production. Because they are directly on the frontlines, engineers clearly see how technical and architectural decisions impact live system availability. The speaker wraps up the formal presentation by emphasizing that all of these designs share a unified objective: “all with the goal of reducing blast radius.”

- **Layered Isolation Hierarchy:** Regions $\rightarrow$ Availability Zones $\rightarrow$ Cells $\rightarrow$ Shuffle-sharding (virtual cells). - **Phased Automation:** Production changes roll out step-by-step through automated pipelines to catch regressions before they reach broader infrastructure. - **Builder-Operator Model:** Developers manage their own operational footprint, keeping feedback loops tight between code authoring and runtime failure modes.

---

[46:01](https://www.youtube.com/watch?v=swQbA4zub20&t=2761s) Q&A: The 2008 S3 Bit Flip and Human Error at the CLI

Opening the floor to questions, an attendee asks whether AWS has encountered an extreme Murphy's Law scenario where the team believed every single failure mode was accounted for, only for an unexpected issue to strike. The speaker immediately points to a historic incident from 2008 in Amazon S3 involving a silent network bit flip. S3 was engineered from the beginning with an obsession over data integrity, instrumented with multiple layers of checksumming throughout the software stack to detect data corruption. However, an issue arose on a single network interface card (NIC) on one server that occasionally flipped a single bit in passing network packets.

```mermaid flowchart LR accTitle: 2008 S3 Bit Flip Failure Propagation accDescr: Flow diagram showing how a single bit flip in a server NIC bypassed missing checksums and destabilized Amazon S3. subgraph Origin["Fault Inception"] NIC["Flaky Server NIC"] -->|"Flips single bit"| Packet["Corrupted Packet"] end subgraph Protocol["Gossip Protocol Layer"] Packet -->|"Bypasses unchecked layer"| Gossip["Gossip State Protocol"] Gossip -->|"Unknown host detected"| Recon["Trigger Expensive Reconciliation"] end subgraph Impact["Service Degradation"] Recon --> Storm["Resource Exhaustion & Cascade"] Storm --> Outage["Full Regional S3 Outage"] end ```

The corrupted packet hit S3's internal group communication layer, which relied on a gossip protocol to track node health across the fleet. Because of the flipped bit, the gossip protocol parsed a corrupted server hostname that it had never encountered before. Assuming an untracked node had joined the cluster, the protocol initiated an expensive reconciliation routine to synchronize state. As that bad packet propagated, it triggered cascading reconciliations across the fleet that ultimately overwhelmed the service. Reflecting on the disproportionate blast radius of that bug, the speaker notes: “So one server, one NIC, one bit took down our regional service, and that was the only layer we had checksumming all the way up and down the stack—that was the only layer that didn't have the checksumming, someone we'd forgotten to add it there.”

An attendee follows up by asking about a more recent incident where human error impacted S3. The speaker acknowledges the event, explaining that it illustrates the ongoing challenge of operational discipline. Even when an engineer thoroughly understands AWS fault isolation philosophies, a manual typo or miscalculated input on the command line can break through architectural boundaries if safety checks do not catch it. The lesson reinforced the necessity of removing manual CLI interventions in favor of strict automation, input validation, and canary testing.

- **The 2008 S3 Gossip Protocol Outage:** Caused by a single faulty network card flipping one bit in a hostname packet, exposing the sole unchecksummed software layer in the system. - **The Cascade Mechanism:** The unrecognized hostname forced nodes into intensive cluster reconciliation loops, exhausting system resources across the region. - **Public Post-Mortem:** Details of the outage remain published on the AWS Service Health Dashboard history. - **CLI Human Error:** Demonstrates that human fallibility can bypass logical isolation boundaries, underscoring why automated, guarded deployment tooling is mandatory.

---

[48:49](https://www.youtube.com/watch?v=swQbA4zub20&t=2929s) Q&A: Customer Visibility of Cells and Data Center Affinities

An audience member asks whether the compartmentalization primitives described in the presentation—specifically regions, Availability Zones, cells, and shuffle-sharding—are exposed directly to customers as purchasable services or if they are built into the platform automatically. The speaker clarifies the divide between customer-facing primitives and internal architectural mechanics:

> “Regions are given to you, these are given to you; cells and shuffle-sharding are sort of different things in that they are they're like the watertight compartments inside the ship. So you could ask the the purser on the boat to give you a tour of the lower decks and maybe they'd show you the compartments, but otherwise you know you don't really care about that and you get them for free—it's just part of the safety of the ship.”

The attendee presses further, asking if an engineer building their own service can explicitly align a cell with a specific underlying data center facility within an Availability Zone. The speaker explains that AWS allows customers to establish strict zonal affinity for resources such as Amazon EC2 instances, AWS CloudHSM modules, and Amazon EBS volumes. However, an Availability Zone is not always a single building; an individual AZ may consist of one or multiple discrete data centers engineered with low-latency links. AWS intentionally abstracts those specific physical data center facilities away from the customer API. While customers have full control over placing workloads into specific Availability Zones, they cannot target individual physical data center buildings within an AZ.

- **Customer-Controlled Boundaries:** Regions and Availability Zones can be targeted directly in AWS APIs. - **Underlying Topologies:** An Availability Zone comprises one or more discrete physical data centers, but physical building-level placement within that AZ is deliberately hidden from customers. - **Internal Safety Constructs:** Cells and shuffle-sharding function largely as invisible, built-in infrastructural safeguards for managed services.

---

[50:46](https://www.youtube.com/watch?v=swQbA4zub20&t=3046s) Q&A: Managing Version Skew Across Multi-Week Phased Rollouts

Another attendee asks about the operational realities of deployment pipelines as architectures transition from broad regional models down to fine-grained cell-based deployments. Specifically, the attendee asks how AWS tracks version management and operational insight during rollout windows when different cells or shards are running different versions of software. The speaker confirms that rolling out updates across a large fleet introduces prolonged periods of version skew:

> “That's definitely consideration, and a deployment may take, depending on the system, it may take several days to several weeks. In some operating structure, we're particularly careful about the pace of deployment. In fact, in some of the networking components for VPC, it's a particularly delicate affair to roll out changes, and it might take a while.”

Because a change might take days or even weeks to clear all staging environments, one-box canaries, individual cells, and regions, teams must treat version skew as an active design constraint. Systems must be engineered to support backward and forward wire compatibility so that nodes running newer revisions can communicate seamlessly with nodes running older builds without corrupting data or triggering protocol mismatches.

- **Deployment Velocity vs. Safety:** Rollouts span several days to multiple weeks depending on the criticality and risk of the service. - **VPC Networking Sensitivity:** Changes to underlying VPC virtual networking components are handled with exceptional caution and extended soak times. - **Operational Complexity:** Operators must monitor divergent software versions coexisting across cells and maintain strict API and wire-format backward compatibility.

---

[51:51](https://www.youtube.com/watch?v=swQbA4zub20&t=3111s) Q&A: Global Services and IAM Data Plane Independence

An attendee asks about the resilience architecture of global AWS services, specifically how control planes, data planes, and edge nodes are organized to maintain regional availability during broader failures. Using AWS Identity and Access Management (IAM) as the primary example, the attendee asks how a globally defined service avoids becoming a single point of failure.

The speaker explains that IAM is treated as a specialized architecture. IAM account credentials, policies, and configuration metadata are managed globally, meaning its control plane—where customers create users or update permissions—is centralized. However, AWS strictly decouples the control plane from the runtime data plane:

```mermaid flowchart TD accTitle: Decoupled IAM Control Plane and Regional Data Planes accDescr: Architecture diagram showing the global IAM control plane replicating asynchronously to isolated regional data planes that operate independently. subgraph GlobalControl["Global Scope"] CP["IAM Global Control Plane (Write / Admin)"] end subgraph Region1["AWS Region A"] DP1["Local IAM Data Plane Cache (Read / Auth)"] Workload1["Regional Services (EC2, S3, etc.)"] DP1 --> Workload1 end subgraph Region2["AWS Region B"] DP2["Local IAM Data Plane Cache (Read / Auth)"] Workload2["Regional Services (EC2, S3, etc.)"] DP2 --> Workload2 end CP -.->|"Asynchronous Metadata Sync"| DP1 CP -.->|"Asynchronous Metadata Sync"| DP2 ```

Every individual AWS region hosts its own independent IAM data plane containing a local cache of identity and credential information. When workloads authenticate requests or validate credentials within an AWS region, the regional services query their local data plane replica. As the speaker highlights regarding this core architectural rule:

> “There is another philosophy that we have, which is the separation between control plane and data plane, and ensuring that the data plane can continue to function even if there is a control plane issue.”

Even if an inter-region network disruption completely isolates a region from the global IAM control plane, or if the centralized source of truth experiences an outage, workloads in that disconnected region continue validating credentials without interruption.

- **Decoupled Architecture:** The global IAM control plane handles administrative state and updates, while isolated regional data planes evaluate authentication and authorization checks. - **Static Stability During Partitions:** If the global control plane goes down or connectivity drops, each regional data plane continues operating independently using its local state. - **Blast Radius Prevention:** Outages affecting IAM administration cannot propagate to disrupt runtime request authentication within individual regions.

---

[53:48](https://www.youtube.com/watch?v=swQbA4zub20&t=3228s) Q&A: Balancing State in Cells vs. Stateless Shuffle-Sharding

The final audience question explores the relationship between cell-based architectures and shuffle-sharding. The attendee asks whether shuffle-sharding is implemented at the same routing and control layers that establish cell boundaries, noting that shuffle-sharding might conflict with cell designs that partition workloads along specific service characteristics.

The speaker clarifies that the main distinction lies in state management. Traditional cell architectures are often adopted specifically because a service holds state that cannot easily be scattered across arbitrary instances; requests must route to the specific cell holding that customer's data. Conversely, shuffle-sharding delivers its greatest benefits in stateless systems or services that use "soft state" (where instances can pull down and locally cache data on demand):

> “So I guess maybe another way to answer that is shuffle-sharding is not an improvement on cell-based in all cases. It's really a system-specific on when you might be able to apply it.”

The speaker shares an advanced pattern AWS is actively implementing: combining both techniques into a hybrid design. By taking a cell-based architecture and applying shuffle-sharding *within* or across those cells, stateless request layers can route traffic across overlapping sub-clusters. The speaker notes that the team is actively implementing this combined approach within Amazon DynamoDB to deliver an even higher tier of fault tolerance.

Wrapping up the session, the speaker thanks the attendees, expresses hope that the session provided actionable insights, and wishes the audience an enjoyable week at re:Invent as the audience applauds.

- **Statefulness Constraints:** Services with hard state constraints rely on deterministic cell routing, whereas stateless services or soft-state caching layers are ideal candidates for shuffle-sharding. - **Context-Specific Application:** Shuffle-sharding is not an outright replacement for cell architectures, but a complementary pattern chosen based on system requirements. - **Hybrid Architectures in Practice:** AWS is layering shuffle-sharding over cell-based topologies in critical engines like DynamoDB to maximize both horizontal scalability and fault containment.

## Notable quotes > [39:42](https://www.youtube.com/watch?v=swQbA4zub20&t=2382s) I'd wager that every new AWS engineer knows within their first week if not their first day that we never want to touch more than one zone at a time. — Werner Vogels

> [24:21](https://www.youtube.com/watch?v=swQbA4zub20&t=1461s) Cells, like our data centers, have a maximum size; we won't let them grow past a certain point, and so if we need to continue to grow a system, rather than growing the cells past that point, we'll add another cell. — Peter Vosshall

## Mentioned Peter Vosshall, Werner Vogels, Eric Brewer, Seth Gilbert, Nancy Lynch, NASA Jet Propulsion Laboratory (JPL), Amazon Web Services (AWS), Amazon.com, Amazon EC2, Amazon S3, Amazon DynamoDB, Amazon RDS, Amazon EBS, Amazon SQS, Amazon SageMaker, AWS Identity and Access Management (IAM), AWS CloudHSM, Amazon Virtual Private Cloud (VPC), Elastic Load Balancing (ELB), CAP Theorem, Murphy's Law

## Video description At AWS, we obsess over operational excellence. We have a deep understanding of system availability, informed by over a decade of experience operating the cloud and our roots of operating Amazon.com for nearly a quarter-century. One thing we've learned is that failures come in many forms, some expected, and some unexpected. It's vital to build from the ground up and embrace failure. A core consideration is how to minimize the "blast radius" of any failures. In this talk, we discuss a range of blast radius reduction design techniques that we employ, including cell-based architecture, shuffle-sharding, availability zone independence, and region isolation. We also discuss how blast radius reduction infuses our operational practices.

[Source](https://www.youtube.com/watch?v=swQbA4zub20)

中文简报

AWS采用单元架构阻断系统级联崩溃在物理与逻辑约束下，分布式故障在所难免，这要求运营人员通过单元架构和混洗分片隔离工作负载，防止恶意请求引发全区域瘫痪。 AWS杰出工程师Peter Vosshall阐述了AWS架构如何围绕光速、CAP定理及墨菲定律等基础约束进行工程设计，通过主动控制不可避免之故障的爆炸半径来保障系统可靠性。演讲概述了四个故障隔离层级：完全隔离的区域技术栈、控制平面解耦的可用区独立性（AZI）、将灾难性故障影响限制在集群1/N规模的单元化舱壁隔离，以及在多租户硬件上虚拟化单租户弹性的混洗分片组合技术。Vosshall还将这些系统设计与严苛的运维纪律相结合，详细介绍了单可用区部署流水线、自动化金丝雀回滚机制，以及AWS务实的“构建者即运维者”文化，并回顾了2008年Amazon S3单比特翻转等真实边缘案例。 AWS通过四个核心层级——区域隔离、可用区独立性、单元架构和混洗分片——构建故障隔离体系，以遏制物理和逻辑层面的服务中断。 (4:23) 单元架构为集群规模设定了严格上限，将软件故障或“毒丸”请求引发的灾难性爆炸半径限制在1/N（N代表单元总数）。 (22:16) 混洗分片利用组合数学为客户分配相互重叠的虚拟节点子集；从8个节点扩展到拥有5节点分片的100节点集群可产生超过7500万种组合，确保77%的租户完全不受恶意租户的影响。 (34:46) AWS IAM等全球服务通过将中心化控制平面与区域缓存数据平面严格解耦来保持静态稳定性，确保在发生全球性宕机或网络分区时，运行时身份验证仍可不间断执行。 (51:51)

Reading summary

AWS Distinguished Engineer Peter Vosshall argues that large-scale distributed systems must be engineered around the inevitability of failure by systematically shrinking the blast radius of disruptions. To contain both environmental disasters and latent software flaws, AWS implements a layered isolation model combining region isolation, Availability Zone independence, cellular partitioning, and combinatorial shuffle-sharding. Physical Isolation across Regions and Availability Zones (6:50) AWS constructs regions as independent stacks operating on a shared-nothing architecture, ensuring an outage in one region cannot directly compromise another. When features require cross-region coordination, such as Inter-Region VPC Peering, a dedicated Cross-Region Orchestrator interacts solely through external API endpoints with ring-fencing safety checks. Within regions, AWS deploys at least three Availability Zones (AZs) spaced miles apart across distinct utility providers to prevent correlated environmental disruptions like storms or fiber cuts, yet close enough to support low-latency synchronous replication. To prevent zonal failures from cascading, AWS applies Availability Zone Independence (AZI). In services like Amazon EC2, zonal resources maintain autonomous control and data planes that do not share state with neighboring zones. While regional control planes aggregate metadata across zones, individual zone failures leave adjacent zones operating unaffected. Vosshall notes that while customers can target regions and AZs directly, AWS abstracts away the specific underlying physical data center buildings that compose an AZ. Cellular Bulkheads and Finite Scaling Caps (18:30) While multi-AZ architectures reliably mitigate physical infrastructure faults, Vosshall observes that shared regional services remain vulnerable to catastrophic "Black Swan" software bugs or poison-pill queries that can destabilize an entire region. To counter this asymmetry, AWS borrows the bulkhead concept from naval architecture, deploying services as multiple independent, self-contained stacks known as cells. A thin, hardened routing layer partitions ingress traffic by customer ID or natural workload boundaries, capping failure impact to 1/N of the customer base, where N represents the number of cells. Critically, AWS imposes strict maximum capacity ceilings on individual cells instead of allowing them to grow indefinitely. Enforcing finite cell sizes allows engineering teams to exhaustively test units up to their breaking point under realistic test harnesses. System growth is achieved by horizontally provisioning additional standardized cells rather than running unverified, massive monolithic clusters. Combinatorial Fault Isolation via Shuffle-Sharding (34:46) For stateless or soft-state services, AWS applies shuffle-sharding to achieve near single-tenant isolation within multi-tenant infrastructure using combinatorial mathematics. Instead of assigning a customer to a dedicated pool, customers are assigned to an overlapping virtual shard of nodes selected via deterministic hashing. In a hypothetical 8-node fleet assigned in pairs, 28 unique combinations exist; a toxic payload knocking out two nodes affects only 3.6% of customer combinations, leaving the rest able to process requests through healthy secondary nodes. When scaled to an enterprise footprint—such as assigning 5 nodes out of a 100-node fleet—the model yields over 75 million combinations. If a toxic workload disables all 5 assigned nodes, 77% of other customers share zero nodes with the bad actor, 99.8% retain a working majority of at least 3 nodes, and only customers sharing the exact same combination experience downtime. Vosshall emphasizes that shuffle-sharding requires fault-tolerant client retries and strictly immutable node assignments, warning that allowing dynamic failover to fresh nodes would cause poison pills to cascade across the entire fleet. Decoupled Planes, Automation, and Progressive Deployment (39:42) Architectural boundaries depend on operational discipline to prevent human-induced common-mode failures. AWS enforces an organization-wide rule never to touch more than one Availability Zone at a time. Deployments advance through guarded canary stages—from single-box tests to individual AZs separated by extended bake periods—backed by automated metric monitors that trigger instant rollbacks without human intervention. To mitigate operational risk, AWS replaces manual command-line interventions with tested software automation managed by builder-operators who maintain and support the software they author. This operational containment is reinforced by decoupling administrative control planes from local runtime data planes. For global services like AWS Identity and Access Management (IAM), policy administration is centralized globally, but each region hosts an autonomous data plane cache. Regional services evaluate authentication locally, ensuring request processing remains statically stable even during global control-plane outages or network partitions. Vosshall emphasizes that reducing blast radius is an ongoing operational commitment, formalized in AWS's Correction of Errors post-mortem reviews by a mandatory thought exercise: identifying how to cut the impact of any recurrence in half. However, implementing fine-grained cell rollouts across global fleets introduces prolonged deployment windows lasting days or weeks, forcing engineers to rigorously maintain backward and forward wire compatibility across coexisting software versions.

精炼阅读摘要

AWS杰出工程师Peter Vosshall在AWS re:Invent 2018上指出，分布式系统必须预设墨菲定律必然生效，其核心防御策略是通过物理隔离、单元化架构、混洗分片与分阶段运维，将单点或级联故障的影响范围（爆炸半径）缩减至最小。爆炸半径维度与物理隔离边界 (0:01) Vosshall指出，分布式系统的故障很少是非黑即白的彻底瘫痪，而是表现为不同程度的影响范围。爆炸半径可从受影响的工作负载数量、受损功能比例以及地理覆盖范围（机架、数据中心乃至整个区域）三个维度来衡量。为将影响降至最低，AWS实施了无共享架构的区域隔离（Region Isolation），全球各区域的服务运行栈与API端点完全独立，即便发生跨区域VPC对等连接等协作，也由独立的跨区域编排器通过公网API标准入口执行，防止级联故障穿透区域边界。在区域内部，AWS通过相距数英里且具备独立公用供电设施的可用区（AZ）构建物理容灾屏障，各区之间距离适中，既避免因单一市政施工挖断光缆或龙卷风等灾害导致共模故障，又能通过低延迟光纤支持分布式一致性协议所需的仲裁（要求至少3个可用区）。此外，AWS在区域控制面之下对EC2等服务实施可用区独立性（AZI）设计，使每个可用区拥有自洽的本地控制面与数据面，将硬件和基础设施停机的爆炸半径收敛在特定可用区内部。单元化架构防范逻辑黑天鹅与毒丸蔓延 (18:30) 尽管多可用区架构能有效防御物理基础设施故障，但面对软件缺陷、恶意攻击或触发极端边缘逻辑的“毒丸请求”（Poison Pills），传统区域级单体服务依然面临全局瘫痪的风险。为解决这一理论爆炸半径问题，AWS引入了类似于造船业水密舱壁的单元化架构（Cell-Based Architecture）。该架构将原本覆盖整个区域的大型单体系统，横向拆分为N个功能完备且彼此完全无感知的独立“单元”（Cell），并在最前端配置逻辑极简、经深度压力测试的精简路由层（Cell Router）。通过按照客户ID、VPC或物理数据中心基础设施进行“顺纹理”切分，恶意请求或由缺陷引发的崩溃被严格限制在单一单元内，将整体系统的理论故障损失限制在总量的1/N。此外，单元被赋予严格的容量上限，系统扩张完全依赖横向增加单元数量而非垂直无限制扩容，这不仅让工程团队能够以极高保真度对单元进行极限破坏性测试，也大幅降低了日常运维与状态排查的复杂度。混洗分片机制以数学组合抑制租户冲击 (32:56) 对于无状态或弱状态服务，AWS进一步采用了基于组合数学的混洗分片（Shuffle-Sharding）技术。在传统多租户共享资源池中，单个异常租户可能通过重试机制引发级联崩溃，造成100%的系统冲击；即便采用固定单元划分，故障依然会波及该单元内的所有客户。混洗分片则利用确定性哈希，为每个客户分配由全量节点池中少数节点组合而成的虚拟分片。以100个节点的集群为例，若为每个租户分配5个节点，理论上可衍生出多达75,287,520种独特组合。当某租户发送毒丸请求导致其对应的5个节点全部下线时，全集群中多达77%的其他客户因完全没有重合节点而丝毫不受影响，99.8%的客户仍能保留至少3个健康节点并通过容错重试正常运作，仅有被分配到完全相同五节点组合的极小概率租户才会遭遇完全服务中断。Vosshall强调，该方案必须要求客户端具备重试与退避能力，且严禁异常工作负载动态故障转移至其他节点，以杜绝故障扩散。阶段化发布纪律与构建者兼运营者文化 (39:42) 严密的架构隔离如果缺乏严格的操作规程，极易被人为误操作破坏。AWS奉行“同一时间绝不动碰超过一个可用区”的铁律，所有软件部署均执行严格的分阶段流水线：从单个实例（One-Box）或极少数宿主机金丝雀测试开始，由集成测试套件和自动化金丝雀调用真实API验证指标，出现异常立即自动回滚，随后才依次按单元、单可用区逐步推进，最终耗时数天乃至数周跨越不同区域。这要求系统在网络与API协议上具备严苛的前后兼容性，以承载发布期间长期存在的版本差异。在组织机制层面，AWS推行“构建者即运营者”（Builder-Operator）文化，彻底打破开发与运维的职能壁垒。编写代码的工程师必须全天候负责生产环境运维。在发生运行故障后，服务团队必须执行“错误纠正”（Correction of Errors, COE）复盘流程，其中一项核心强制考量即为“若同类事件再次发生，如何将其爆炸半径削减一半”，从而促使一线工程师持续优化系统拓扑与防护边界。现场案例复盘与底层边界的技术权衡 (46:01) Vosshall在问答环节结合历史案例复盘了爆炸半径失控的教训。在2008年Amazon S3的大规模故障中，全系统唯独在节点间状态同步的Gossip协议层缺少了校验和，导致单台服务器网卡上的单个位翻转（Bit Flip）生成了畸变的主机名，进而引发全区域节点的密集状态对齐风暴，最终致使区域级服务停摆；该案例与后续由命令行输入失误引发的事故共同表明，必须依赖输入校验和自动化流水线杜绝人为干预。针对IAM等全局定义服务，AWS则通过严格解耦全局控制面与区域本地数据面，使各区域依赖本地凭证缓存实现“静态稳定性”，即便跨区域网络中断也能独立鉴权。关于边界暴露与状态管理的权衡，Vosshall澄清可用区在底层通常由多栋物理数据中心建筑组成，AWS API仅向客户暴露区域和可用区维度，内部单元划分与建筑细节对用户透明。在架构选型上，强状态服务通常必须采用确定性单元路由，而无状态或弱状态缓存层则适合混洗分片，AWS目前正将两者融合（如在DynamoDB中应用混洗分片与单元嵌套），以在更大规模下兼顾横向可扩展性与故障隔离。 AWS的弹性体系表明，高可用并非源于消除故障的发生，而是基于物理分区、单元化设计与组合数学将故障影响降至极低比例。然而，将爆炸半径层层压缩的代价是发布周期拉长、多版本兼容负担加重以及底层路由复杂度提升，要求团队必须在强隔离与系统状态管理之间持续进行架构权衡。

Detailed notes — zh

## Background 该视频由 Amazon Web Services 官方 YouTube 频道于 2018 年 11 月 27 日发布，时长约 55 分钟。视频记录了 AWS 杰出工程师 Peter Vosshall 在 re:Invent 2018 大会上的演讲，围绕 AWS 运维卓越性与高可用性经验，深入探讨如何利用单元化架构、洗牌分片、可用区独立性和区域隔离等手段，在底层设计中全面最小化系统故障的“爆炸半径”（Blast Radius）。 ## TL;DR 演讲直面分布式系统必经的物理与软件故障现实，强调架构设计的核心在于抑制不可避免的墨菲定律破坏力，而非空谈零故障。AWS 建立了从物理到逻辑的纵深防御体系：外部依托无共享的区域隔离与多可用区（AZ）独立性，内部推行具备固定容量上限的单元架构（Cell-Based Architecture），并在无状态与软状态场景下应用数学组合极高的洗牌分片（Shuffle Sharding）技术，将故障影响面压缩至极低比例。同时，结合渐进式金丝雀发布与“构建即运维（Builder-Operator）”的全生命周期所有权文化，确保运维操作严格敬畏故障隔离边界，实现系统的高韧性与自愈能力。 ## Atmosphere & style 整场演讲严谨专业又不失风趣幽默，主讲人以深入浅出的工程叙事将冷酷的计算机科学公理娓娓道来。台上既有对“黑天鹅”、“毒药丸”和历史翻车惨痛教训的自嘲与敬畏，也有展示水密隔舱类比与组合数学魔力时的胸有成竹。现场技术沉浸感极强，讲者从容笃定地拆解复杂工业架构，提问环节坦诚复盘事故根因，让听众在浓厚的技术极客氛围中深刻感受到顶尖云厂商对高可用与运维细节的极致执着。 ## Bulletin - [02:35](https://www.youtube.com/watch?v=swQbA4zub20&t=155s) AWS 应对分布式系统故障的核心原则是全面拥抱故障并控制“故障半径（Blast Radius）”，并在 COE 复盘中强制要求团队思考如何将下一次同类故障的影响面直接削减 50%。 - [21:08](https://www.youtube.com/watch?v=swQbA4zub20&t=1268s) 通过单元化架构（Cell-Based Architecture），系统被拆分为多个设有固定规格上限且互不依赖的独立单元，将区域级灾难性故障的影响上限由 100% 压缩至 1/N。 - [37:00](https://www.youtube.com/watch?v=swQbA4zub20&t=2220s) 洗牌分片（Shuffle Sharding）利用二项式组合将租户映射到节点子集（如 100 节点中取 5 节点产生逾 7500 万种组合），在多租户池中实现近乎单租户的隔离体验，使毒丸请求完全拖垮其他客户的概率微乎其微。 - [39:42](https://www.youtube.com/watch?v=swQbA4zub20&t=2382s) 在运维层面，AWS 将绝不同时变更超过一个可用区作为工程红线，依托严格错峰的渐进式金丝雀发布与自动化回滚机制来守护架构隔离边界。 ## Key takeaways - [00:01](https://www.youtube.com/watch?v=swQbA4zub20&t=1s) 分布式系统受制于光速上限、CAP 定理以及不可避免的墨菲定律，系统设计的终极目标并非彻底杜绝故障，而是构建严格的隔离防线来限制影响范围。 - [02:35](https://www.youtube.com/watch?v=swQbA4zub20&t=155s) 故障绝非“可用与不可用”的非黑即白状态，评估爆炸半径必须从受影响客户数、受损功能面以及物理/逻辑地理位置三个维度综合度量。 - [04:23](https://www.youtube.com/watch?v=swQbA4zub20&t=263s) 除物理灾害与硬件故障外，能精准触发系统 Bug 或消耗巨额资源的“毒药丸（Poison Pills）”会经由客户端重试引发全集群级联崩溃，是危害最严重的逻辑故障。 - [06:50](https://www.youtube.com/watch?v=swQbA4zub20&t=410s) AWS 各 Region 之间采用完全独立的无共享架构（Shared-Nothing Architecture），跨 Region 间没有共享计算与存储，构筑了宏观层面的最强硬隔离墙。 - [08:24](https://www.youtube.com/watch?v=swQbA4zub20&t=504s) 针对跨区域 VPC 对等连接等业务，AWS 设立独立的 Cross-Region Orchestrator 悬浮在各 Region 之上，仅通过标准对外 API 前门下发指令，绝不破坏底层隔离边界。 - [11:23](https://www.youtube.com/watch?v=swQbA4zub20&t=683s) 可用区（AZ）之间相距数英里以阻断市政电网或龙卷风等关联失效，同时又保持微秒级延迟以支持同步复制协议，现代 Region 标配至少 3 个 AZ 支撑分布式共识仲裁。 - [15:36](https://www.youtube.com/watch?v=swQbA4zub20&t=936s) AWS 严格贯彻可用区独立性（AZI）并解耦控制面与数据面：各 AZ 本地控制面与数据面完全独立，即便全局控制面或某 AZ 瘫痪，健康 AZ 的运行时与数据面仍能持续运转。 - [18:30](https://www.youtube.com/watch?v=swQbA4zub20&t=1110s) 汲取造船工业水密舱壁（Bulkhead）的隔离哲学，单元化架构将完整的跨 AZ 服务栈实例化为多个彼此隔离的 Cell，由极简的 Cell Router 实施租户分片分发。 - [24:21](https://www.youtube.com/watch?v=swQbA4zub20&t=1461s) 单元化架构推翻了无限制向上扩容（Scale Up）的模式，规定每个 Cell 必须设有最大规格上限，系统扩容仅通过追加新单元（Scale Out）实现，确保单单元可被打断点式深度压测。 - [26:03](https://www.youtube.com/watch?v=swQbA4zub20&t=1563s) 在单元尺寸的权衡中，AWS 坚守在同等条件下始终优先选择更小爆炸半径的底线；单元切分应顺应数据流动的自然纹理（Cut with the grain），严控 Scatter-Gather 等跨单元复杂交互。 - [37:00](https://www.youtube.com/watch?v=swQbA4zub20&t=2220s) 洗牌分片（Shuffle Sharding）基于组合数学将客户映射到节点池的一个子集中，要求客户端具备重试能力并严禁对失效节点做跨分片自动转移，以防止故障级联传染。 - [42:35](https://www.youtube.com/watch?v=swQbA4zub20&t=2555s) AWS 全面推行“谁构建谁运维（Builder-Operator）”模型与全流程自动化脚本，通过错峰部署、单机盒验证与全天候合成金丝雀测试，杜绝人为操作突破可用区隔离边界。 - [46:01](https://www.youtube.com/watch?v=swQbA4zub20&t=2761s) 2008 年 S3 因单网卡单比特翻转篡改机器名触发 Gossip 协议对账雪崩导致区域级瘫痪的事故，证明即便拥有全栈校验和，单点疏漏依然可能借助级联效应击垮系统。 - [52:00](https://www.youtube.com/watch?v=swQbA4zub20&t=3120s) 以 IAM 为代表的全局服务在每个 Region 部署完全独立的数据面，即使全局控制面与单点真相源（Source of Truth）断开，本地数据面仍能无阻断完成 API 凭据鉴权与请求授权。 - [53:48](https://www.youtube.com/watch?v=swQbA4zub20&t=3228s) 洗牌分片更契合无状态或弱状态场景，而强状态系统更适合确定性的单元架构；前沿演进（如 DynamoDB）正在探索在无状态 Cell 之间叠加洗牌分片的多层复合架构。 ## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=swQbA4zub20&t=1s) 宇宙法则与墨菲定律：分布式系统的现实挑战在会议开场，AWS 杰出工程师（Distinguished Engineer）Peter Vosshall 热烈欢迎现场首次参加 re:Invent 的与会者，并预祝大家度过充实且充满派对与技术分享的一周。Vosshall 的整个职业生涯都专注于构建大规模、高可用的分布式系统。他指出，投身这一领域的人很快就会意识到，物理世界和计算机科学中有几条必须直面的基本法则。首先是光速限制（约每毫秒 186 英里），光速看似极快，但当天上午喷气推进实验室（JPL）尝试让火星探测器着陆时，超远距离通信带来的延迟就成了巨大阻碍（Vosshall 笑着恭喜同为 AWS 客户的 JPL 团队成功着陆探测器）。其次是计算机科学中棘手的 CAP 定理，由 Eric Brewer 于 1998 年提出并由 MIT 的 Seth Gilbert 与 Nancy Lynch 证明，它指出任何网络分区容错的分布式系统都无法同时保证强一致性与可用性。最让人头疼的莫过于墨菲定律（Murphy's Law）。Vosshall 幽默地提到，虽然不知道那位名叫墨菲的人提出的定律是否经过了数学证明，但工程实践已经无数次证实了它的正确性——“任何可能出错的事情终究会出错”，系统迟早会以各种离奇的方式崩溃。对于 AWS 而言，安全性、可用性与持久性是绝对的最高优先级，然而无论工程水平多高，系统都必须在墨菲定律的阴影下运行。因此，本次演讲的核心并不是空谈如何消灭故障，而是分享 AWS 如何通过架构与运维手段将墨菲定律生效时的破坏程度控制在最小范围，从而让客户也能将这些技术应用到自身的系统构建中。正如 Peter Vosshall 所言：“Perhaps most annoying though is this observation by some guy named Murphy which is basically stuff breaks.” - 光速通信延迟约为每毫秒 186 英里（186 miles per millisecond），在远距离分布式通信中构成了物理上限。 - CAP 定理限制了分布式系统在分区容错（Partition Tolerance）前提下无法兼得一致性（Consistency）与可用性（Availability）。 - 演讲核心理念：在承认故障必然发生的前提下，系统设计必须拥抱故障，专注于抑制影响程度。 ### [02:35](https://www.youtube.com/watch?v=swQbA4zub20&t=155s) 什么是“故障半径”（Blast Radius）与 COE 减半思维在 AWS 的日常工程语言中，“爆炸半径”（Blast Radius，即故障影响范围）是一个极其核心的概念。Vosshall 解释说，系统的故障从来不是非黑即白的二进制状态（不是单纯的“挂了”或“没挂”），而是存在明确的影响范围与程度层级。当故障发生时，AWS 习惯从三个维度来审视爆炸半径：受到影响的客户或工作负载数量是多少？受损的功能范围有多大（是整体不可用，还是仅有某一局部特性降级）？以及受影响的地理位置与基础设施范围在哪里（是单个机架、单个数据中心，还是整个区域）？显然，机架级故障的影响要远远优于整个区域级瘫痪。尽管 AWS 投入了巨大的工程精力来预防故障并保持极高的系统可用性，但团队同样执着于在极少数不可避免的故障发生时缩小爆炸半径。这种理念深度融入了 AWS 著名的“错误校正”（Correction of Errors，简称 COE）复盘机制中。每当发生运维事件，对应的服务团队都会深入挖掘根本原因并制定防范措施。在 COE 的分析模板中，有一道必答的思想实验题：“如果是类似的故障再次发生，你如何将爆炸半径削减一半？”这种自省机制迫使工程团队在系统设计的每一处细节上，都将故障隔离作为核心指标。Peter Vosshall 强调：“Failure isn't binary; it's not 'it's failing or it's not'—there is a degree of impact.” - 评估爆炸半径的三个关键维度：客户与负载数量（Who）、受影响的功能范围（What）、受影响的物理/逻辑位置（Where）。 - COE（Correction of Errors）复盘制度中的核心思考题：如何通过架构或流程改进，使下一次同类故障的影响半径直接减少 50%。 ### [04:23](https://www.youtube.com/watch?v=swQbA4zub20&t=263s) 现实世界中的故障百态：从物理灾害到“毒药请求” 在深入讨论技术方案之前，Vosshall 梳理了分布式系统必须应对的各种故障类型，涵盖物理损坏与软件层面的隐式失效。在物理世界中，服务器可能会发生意外宕机，磁盘会发生完全下线或产生随机的 I/O 错误，网络硬件设备故障时甚至会引发偶发的“比特翻转”（bit flips，AWS 曾数次在生产环境中遭遇此类问题）。走出数据中心之外，市政施工人员可能会挖断主干光纤，雷暴天气可能造成市政电网瘫痪，更极端的暴风雨或火灾甚至能彻底摧毁整座数据中心建筑。然而，非物理层面的逻辑故障往往更加阴险棘手。服务可能会遭遇恶意 DDoS 攻击或业务激增导致的过载。更为致命的是所谓的“黑天鹅请求”，AWS 内部将其形象地称为“毒药丸”（poison pills）——这类请求或者需要消耗极其昂贵的计算资源，或者精准触发了系统深处的隐藏 Bug。更为可怕的是，客户端在请求失败后往往会启动重试机制，导致这颗“毒药”在整个集群中不断被转发执行，单点故障迅速演变成蔓延整个系统的级联崩溃。此外，软件发布失误、错误的配置变更以及各种未测出的 Bug，也是随时可能打破生产稳定的隐形炸弹。针对这种级联效应，Peter Vosshall 发出警告：“A single failure can cascade and cause an entire system to get infected.” ```mermaid flowchart TD accTitle: 典型分布式系统故障类型分类 accDescr: 图表展示了导致系统故障的物理硬件/环境因素与非物理软件/流量因素 subgraph Physical["物理层基础设施故障"] P1["服务器宕机 / 磁盘随机 I/O 错误"] P2["网络设备比特翻转 / 光纤被挖断"] P3["市政电网雷暴停电 / 数据中心火灾风暴"] end subgraph Logical["逻辑层与软件环境故障"] L1["DDoS 攻击 / 突发流量过载"] L2["毒药丸 (Poison Pills) 与级联重试"] L3["代码 Bug / 危险配置变更与错误发布"] end Physical --> Impact["产生不同范围的系统影响"] Logical --> Impact ``` ### [06:50](https://www.youtube.com/watch?v=swQbA4zub20&t=410s) 区域隔离：无共享架构与单 Region 爆炸边界为了抵御前述的各类破坏，AWS 确立了一整套由大到小的隔离防线：Region 隔离、可用区独立性（AZ Independence）、基于单元的架构（Cell-based Architecture）、乱序分片（Shuffle Sharding）以及配套的运维实践。防线的第一道基石是“区域隔离”（Region Isolation）。当时 AWS 在全球部署了 19 个独立 Region（并宣布即将上线另外 5 个）。客户可以根据延迟要求和数据合规政策自由选择目标 Region。从底层设计上讲，每一个 Region 都是一套完全独立的 AWS 服务软件栈，每个 Region 拥有自己独立的 API 服务端点（Endpoint）。例如在 EC2 中，访问 `us-west-1` 和访问 `us-east-2` 走的是完全不同的域名和控制栈，后端不存在所谓“全局单一的 EC2”。EC2、SQS、SageMaker 等所有服务在不同 Region 之间是互不知晓、互不相通的。这种经典的“无共享架构”（Shared-Nothing Architecture）在宏观层面构筑了最强大的隔离墙，即使某个国家发生大地震导致该 Region 受创，故障的影响范围也绝不会蔓延至其他 Region。Peter Vosshall 总结道：“This is a shared-nothing architecture which gives us the ultimate in blast radius protection.” - 区域特性：独立的技术栈、独立的 API Endpoint，跨 Region 间不共享计算或状态存储。 - 架构收益：任何区域性灾难或配置事故，其最大爆炸半径严格被圈定在单个 Region 内部。 ### [08:24](https://www.youtube.com/watch?v=swQbA4zub20&t=504s) 跨区域功能的挑战与 Cross-Region Orchestrator 既然 Region 之间完全互不通信，那类似 S3 跨区域复制（Cross-Region Replication）、DynamoDB 全局表（Global Tables）以及跨区域 VPC 对等连接（Inter-Region VPC Peering）等跨 Region 功能又是如何实现的？Vosshall 以跨区域 VPC Peering 为例进行了解析。VPC 允许客户在 AWS 云中构建逻辑隔离的虚拟私有网络并自定义网段、路由表和网关。跨区域 VPC Peering 则允许客户在不走公网的前提下打通两个不同 Region 的 VPC，甚至连接不同客户拥有的 VPC。创建 VPC Peering 需要跨越两个 Region 的控制平面执行配置更改，并严格管理双方客户的审批工作流。然而，两边的 EC2 控制平面本身是严禁直接互通的。为了解决这一矛盾，AWS 设计了一个专门的独立服务——跨区域编排器（Cross-Region Orchestrator）。该编排器悬浮在各 Region 栈之上，专门负责状态机工作流和安全审批，随后它像普通外部客户一样，通过各自 Region 的标准对外 API“前门”下发配置变更。同时，编排器内置了严格的安全围栏（Ring-Fencing）机制，限制了双向交互的行为范围，确保在发生任何单边异常时都不会连带污染另一个 Region，从而捍卫单 Region 故障半径的红线。Peter Vosshall 解释该服务的作用时指出：“The answer: we have a dedicated service called the Cross-Region Orchestrator that sits on top of these systems and implements the workflow, manages through the approval process, and comes through sort of the front door of EC2 just like any other API would.” ```mermaid flowchart LR accTitle: 跨区域 VPC 对等连接与编排器架构 accDescr: 展示了 Cross-Region Orchestrator 如何在不打破 Region 隔离的前提下通过公开 API 协调两个区域 subgraph Orchestration["协调层"] CRO["Cross-Region Orchestrator\n(状态机 / 审批工作流 / 安全围栏)"] end subgraph RegionA["AWS 区域 A (如 us-west-1)"] APIA["EC2 API 前门"] VPCA["VPC A 控制平面与网络"] end subgraph RegionB["AWS 区域 B (如 us-east-2)"] APIB["EC2 API 前门"] VPCB["VPC B 控制平面与网络"] end CRO -->|"调用标准 API"| APIA CRO -->|"调用标准 API"| APIB APIA --> VPCA APIB --> VPCB VPCA <-.->|"跨区域专网对等通道\n(数据面直连)"| VPCB ``` ### [10:29](https://www.youtube.com/watch?v=swQbA4zub20&t=629s) 迈向更细粒度：引出可用区独立性（AZ Independence）虽然将故障限制在单 Region 已经是非常强悍的安全边界，但这对于那些业务仅部署在单个 Region 内的客户来说依然是不可承受之重。一旦某个 Region 发生重大故障，单 Region 内客户感知到的影响依然是 100%。为了追求更极致的韧性，必须在 Region 内部进一步拆解系统结构，寻找更细颗粒度的隔离单元。 Vosshall 借此自然地引出了下一项核心机制——可用区独立性（Availability Zone Independence）。每个 Region 内部并不是铁板一块，而是由多个在物理和逻辑上高度解耦的可用区（AZ）所组成。如何通过 AZ 的独立性设计来将爆炸半径进一步从“区域级”压缩至“可用区级”，成为了构建高可用云原生服务的下一个关键命题。正如 Peter Vosshall 坦言：“In the worst case we have a single region blast radius in case of a failure; that's still pretty big, right? That's a lot of impact, especially if you're a customer that sits only in that region.” - Region 级别隔离是最高层级的安全保障，但粒度仍显过粗。 - 架构设计目标：在 Region 内部通过可用区（AZ）构建物理与逻辑边界，将故障抑制在可用区内部，让多 AZ 部署的业务实现自愈与持续服务。 ### [11:23](https://www.youtube.com/watch?v=swQbA4zub20&t=683s) 区域与可用区架构：物理隔离与极低关联失效在宏观尺度上，AWS 区域（Region）表现为例如美国北部弗吉尼亚（Northern Virginia）、都柏林（Dublin）或法兰克福（Frankfurt）等地理位置。然而，一个区域绝非单一的数据中心，而是由分布在整个大都会区（Metropolitan area）的多个数据中心组成，这些独立的物理位置被称为可用区（Availability Zones，简称 AZ）。各个 AZ 之间通过高速私有光纤链路（High-speed private fiber links）进行互联。 ```mermaid flowchart TD accTitle: AWS 区域内多可用区架构拓扑 accDescr: 展示一个区域内多个 AZ 通过高速私有光纤连接，同时实现物理与公用设施隔离 subgraph Region["AWS Region (例如: Northern Virginia)"] subgraph AZ1["Availability Zone 1"] DC1A["Data Center 1"] DC1B["Data Center 2"] end subgraph AZ2["Availability Zone 2"] DC2A["Data Center 1"] DC2B["Data Center 2"] end subgraph AZ3["Availability Zone 3"] DC3A["Data Center 1"] end AZ1 <-->|"高速私有光纤"| AZ2 AZ2 <-->|"高速私有光纤"| AZ3 AZ1 <-->|"高速私有光纤"| AZ3 end ``` AZ 之间的物理距离把控得十分精密：一方面，它们彼此相隔数英里，“相距足够远，除了地震这种极端情况外，出现关联失效（Correlated failure）的概率极低。”这意味着如果龙卷风袭击某一区域，不太可能同时波及多个设施；即使市政电力供应发生故障，由于区域内的各 AZ 接入了不同的公用事业单位（Utility suppliers）电网，单一断电事故绝不会同时瘫痪多个可用区。另一方面，它们又“足够靠近，完全不会受到恼人的光速延迟问题困扰”，在逻辑上可被视作处于同一地点，系统能够在没有显著延迟惩罚（Latency penalty）的情况下平稳运行同步复制协议（Synchronous replication protocols）。每个 AZ 内部并非仅有一栋建筑，而是包含 $n$ 个数据中心。AWS 会将单体数据中心限制在固定的最大规模（Fixed maximum size），因此在规模较大的 AZ 中，往往由多栋独立的建筑组合而成。每个区域通常拥有 3 个或更多的 AZ，而且未来所有新建区域都将至少包含 3 个 AZ。这种设计特别契合分布式系统中的共识协议（Consensus protocols，例如 Paxos/Raft），因为至少需要 3 个独立站点才能有效达成跨节点的一致性共识仲裁。截至演讲当时，AWS 全球已构建 19 个区域、57 个可用区，另有 5 个新区域和 15 个可用区正在建设中。这种跨 AZ 架构使得整个区域发生关联失效的概率大幅压低，从而显著缩小了爆炸半径。 - **物理距离平衡点**：相隔数英里，既阻断了龙卷风、电网中断等自然与基础设施关联故障，又保证了微秒级低延迟，满足同步复制需求。 - **电网隔离**：不同可用区连接独立的市政电力供应线路，防止大面积串联跳闸。 - **三可用区原则**：现代与未来所有区域均标配至少 3 个 AZ，为分布式法定人数（Quorum）及共识协议提供原生底层支撑。 --- ### [13:49](https://www.youtube.com/watch?v=swQbA4zub20&t=829s) 多可用区（Multi-AZ）应用弹性与区域级服务范式客户构建系统时，可以像 AWS 构建底层服务一样，直接复用 Multi-AZ 架构模式来缩小故障影响范围。以一个典型的三层架构为例：前端部署 Elastic Load Balancer（ELB），将流量分发到横跨多个 AZ 的 EC2 实例集群；底层数据持久化层则采用多 AZ 部署的关系型数据库（如 Amazon RDS 主备实例对，Primary-Standby pair）。 ```mermaid flowchart TD accTitle: 多 AZ 应用容灾及自动故障转移流程 accDescr: 展示 ELB 路由流量及 RDS 跨可用区主备切换以实现零停机爆炸半径控制 Client["客户端请求"] --> ELB["Elastic Load Balancer"] subgraph AZ_A["可用区 A (故障中)"] EC2_A["EC2 计算实例 (下线)"] DB_Pri["RDS 主库 Primary (不可用)"] end subgraph AZ_B["可用区 B (健康)"] EC2_B["EC2 计算实例 (健康运行)"] DB_Stby["RDS 备库 Standby (自动升主)"] end ELB -.->|"停止转发"| EC2_A ELB -->|"正常转发"| EC2_B EC2_B -->|"接管数据库连接"| DB_Stby DB_Pri -.->|"自动故障转移 (Failover)"| DB_Stby ``` 当某一 AZ 突发断电或网络中断故障时，系统的恢复链条完全自动化：ELB 迅速检测到底层健康检查异常，即刻停止向该 AZ 内的实例分发流量；同时，原本连向主数据库的连接虽因网络断开而中断，但 RDS 会立刻触发跨可用区自动故障转移（Failover），将健康 AZ 中的备用数据库提升为主库，应用程序重新建立连接即可恢复正常。“这些故障检测事件可以非常迅速地完成，因此您的应用程序操作可能只会产生微小的卡顿，但除此之外，哪怕应用程序损失了一座数据中心，也几乎像什么都没发生过一样。” 演讲者指出，AWS 旗下所有按区域级（Regional）运行的服务均依循这种理念进行设计与运维。该模式赋予系统卓越的容错能力，意味着业务能够承受单个 AZ 的全盘崩溃而不会对最终客户造成任何实质冲击。不仅高可用服务如此，持久性存储亦是如此：Amazon S3 正是依托多可用区底层复制模型，实现了高达 11 个 9（99.999999999%）的数据持久性标准。通过 Multi-AZ 布局，当某座数据中心遭遇龙卷风袭击或施工挖掘机切断光纤时，故障对宏观业务的实际爆炸半径能够真正趋近于零。 --- ### [15:36](https://www.youtube.com/watch?v=swQbA4zub20&t=936s) 区分控制面与数据面：可用区独立性（AZI） AWS 的某些核心产品允许客户直接分配基于单可用区的专用资源（Zonal resources），例如单一的 EC2 实例或 EBS 卷，由用户自行决策将其放置在哪一个 AZ 中以构建整体弹性系统。为了将此类基于可用区资源的爆炸半径缩至极小，AWS 确立了“可用区独立性”（Availability Zone Independence，简称 AZI）原则，并在每个 AZ 内部构建完全独立的本地控制面。此处详细区分了“控制面”（Control Plane）与“数据面”（Data Plane）的技术边界： 1. **控制面**：用户发起管理动作与执行资源调配的交互层（例如 EC2 的 `RunInstances` API 调用）。它负责接收解析 API 请求，并调度执行创建及拉起虚拟机所必需的一切后台事务。 2. **数据面**：实际承担工作负载与业务流量的运行时环境。在 EC2 场景下，数据面即虚拟机实例本身、用户登录的 SSH 会话、运行在实例上的应用程序代码，以及负责在 VPC 内部打通实例间互联的底层虚拟化网络。在 AZI 原则下，各 AZ 的数据面在物理与逻辑上高度隔离：它们虽然能够通过内部网络互相通信，但在系统感知层面上彼此对立、互不依赖；各 AZ 内部的本地控制面亦是如此。然而，用户访问 EC2 等服务时使用的是统一的区域级终端节点（Regional Endpoint），这必然要求系统顶层存在一层抽象。 ```mermaid flowchart TD accTitle: 区域控制面与各可用区独立控制面架构 accDescr: 展示顶层多可用区容错的区域控制面如何路由和聚合各 AZ 本地控制面与数据面 subgraph TopLevel["区域统一接入"] RegCP["区域控制面 Regional Control Plane\n(多 AZ 高可用 / 处理全区资源如安全组)"] end subgraph AZ_A["可用区 A (AZI 独立隔离)"] CtrlA["AZ 本地控制面"] DataA["EC2 实例数据面"] CtrlA --> DataA end subgraph AZ_B["可用区 B (AZI 独立隔离)"] CtrlB["AZ 本地控制面"] DataB["EC2 实例数据面"] CtrlB --> DataB end RegCP -->|"下发单区资源指令"| CtrlA RegCP -->|"下发单区资源指令"| CtrlB ``` 为此，AWS 在各可用区之上构建了“区域控制面”（Regional Control Plane）。它作为对外暴露的统一接入网关，统一承接不绑定特定 AZ 的全局配置（例如安全组 Security Groups），并承担全区 API 聚合功能（例如调用 `DescribeInstances` 查询全区所有虚拟机实例清单时，区域控制面会并发向每个 AZ 的控制面发起轮询并归纳结果）。这套组合架构确保了极高的容错性：一旦某个 AZ 彻底失效，虽然该特定 AZ 内的局部资源无法运行，但其他 AZ 的数据面与本地控制面完全不受干扰。区域控制面本身依托多可用区容错构建，在局部灾难时仍旧维持运转；除了无法处理针对已下线 AZ 的操作请求外，凡是发往健康可用区创建实例等 API 请求，均能自动绕开故障区域完成执行。 --- ### [18:30](https://www.youtube.com/watch?v=swQbA4zub20&t=1110s) 理论爆炸半径的隐患与舱壁隔离思想尽管多可用区架构大幅降低了物理故障的关联度，但演讲者指出了一个深层次的架构焦虑——“理论爆炸半径”（Theoretical blast radius）与实际表现并不完全一致。所谓理论爆炸半径，是指系统在遭遇不可预知的极端极端情况（即黑天鹅事件，Black Swan event）或严重非基础设施故障（如全局代码缺陷、突发毒药请求 Poison Pill）时的最坏崩溃边界。对于典型的区域级服务而言，如果发生最坏情况的未知异常，“其理论爆炸半径是整个服务，而这正是我们每晚辗转反侧、难以入眠所思考的事情。”（“for the regional service it is the entire service and that's the thing that we lose sleep at night every night thinking about.”）反观单可用区资源，其最坏情况的物理上限被严格锁定在该特定 AZ 之内，绝不会波及全区。这一现象引发了核心思考：能否在不牺牲区域级抽象便利性的前提下，打破传统的物理 AZ 维度，通过其他切分方式使区域级服务同样具备严格的隔离上限？ ```mermaid flowchart LR accTitle: 船舶水密舱壁与单元化架构的隔离哲学映射 accDescr: 展示船舶利用水密舱壁抵御船体破损，与单元化架构利用单元隔离抵御系统故障的类比 subgraph Ship["传统造船工程 (Shipbuilding)"] H1["舱室 Compartment 1"] --- B1["水密舱壁 Bulkhead"] B1 --- H2["舱室 Compartment 2 (受损破水)"] H2 --- B2["水密舱壁 Bulkhead"] B2 --- H3["舱室 Compartment 3"] end subgraph Cellular["单元化系统架构 (Cell-Based)"] C1["Cell 0 (客户集群 A)"] C2["Cell 1 (恶意/毒药流量受限单元)"] Cn["Cell N (客户集群 N)"] end ``` 这种架构模式的灵感来源于历史悠久的造船工业。数百年来，大型船舶均采用水密舱壁（Bulkheads）将船体内部划分为数个彼此完全封闭的水密隔舱（Watertight compartments）。一旦外壳受损破裂导致海水灌入，涌水会被牢牢限制在破损的单一舱室内，其余舱段依然完好无损，从而确保船舶维持浮力不致沉没；与此同时，舱壁本身还构成了船体坚固的结构支撑骨架。造船业这种“通过物理隔断最小化故障影响，并借此强化整体结构稳固性”的哲学，被 AWS 深度吸纳并移植到分布式软件工程中，演化出了关键的“单元架构”（Cellular Architecture，或称 Cell-Based Architecture）。 --- ### [21:08](https://www.youtube.com/watch?v=swQbA4zub20&t=1268s) 单元化架构（Cell-Based Architecture）的核心实践在单元化架构体系中，原先由 ELB、多 AZ 计算集群和后端持久化存储构成的常规服务技术栈（Service stack），不再作为一套单体庞然大物支撑整个区域的所有负载。取而代之的是，工程团队将这一整套原本就具备跨 AZ 容灾特性的完整系统架构进行多份独立实例化部署（Multiple instantiations）。每一个被独立复制出来的服务技术栈被称为一个“单元”（Cell）。各个 Cell 之间物理与逻辑全链路彻底断开，互不知晓对方的存在，不存在跨 Cell 的共享资源或内部级联调用。 ```mermaid flowchart TD accTitle: 单元化架构流量路由与故障隔离 accDescr: 展示网关路由层将不同租户负载分片导入独立 Cell，限制单一故障爆炸半径 Traffic["全量业务请求接入"] --> Router["区域路由与分发层 (Router/Aggregator)"] subgraph Cell_0["Cell 0"] ELB0["ELB 0"] --> Compute0["Multi-AZ 计算集群"] Compute0 --> DB0["数据存储"] end subgraph Cell_1["Cell 1 (故障隔离域)"] ELB1["ELB 1"] --> Compute1["Multi-AZ 计算集群"] Compute1 --> DB1["数据存储"] end subgraph Cell_N["Cell N"] ELBn["ELB N"] --> Computen["Multi-AZ 计算集群"] Computen --> DBn["数据存储"] end Router -->|"租户分区 0"| Cell_0 Router -->|"异常/恶意租户分区 1"| Cell_1 Router -->|"租户分区 N"| Cell_N ``` 随后，系统在前端引入精简的路由与聚合层，将区域内的全部工作负载按照特定维度（Partitioning）进行切分并分发给不同单元。最常见的切分维度是基于客户（Customer）标识： - 将第一批客户分配给 Cell 0； - 将第二批客户分配给 Cell 1； - 依次类推，直至分配至 Cell $n$。这种分区机制直接将黑天鹅事件与潜在恶意行为的破坏面锁死。演讲者幽默地打了个比方：“台下的各位看起来都很友善，但也许你们当中某个人很调皮，会给我们制造麻烦；他们如果真的捅了篓子，也仅仅会在其所在的那个单一 Cell 里引发故障，其他所有的 Cell 依然安然无恙。”通过多单元隔离，即使遭遇能击垮整套应用栈的软件致命缺陷或针对特定数据边界的毒药请求，系统在区域维度的受损面也被严格限制在 $\frac{1}{n}$，其余绝大部分租户和业务将完全免受牵连。 ### [22:16](https://www.youtube.com/watch?v=swQbA4zub20&t=1336s) 基于单元的架构（Cell-Based Architecture）与故障半径收敛为了让拆分后的多个单元对外仍表现为一个统一的单一服务，AWS 在所有单元之上引入了一层单元路由器（Cell Router）来做出路由决策。整体结构被称为基于单元的服务（Cell-Based Service）。单元是完全对客户透明的内部架构，但它提供了强大的弹性和容错能力。这种设计在形式上与可用区独立性（AZ Independence）类似，只是在不同的维度上对系统进行了切分。在传统的区域级服务（Regional Service）中，系统通过跨可用区部署来获得高可用，当单个 AZ 发生物理级故障时服务能够正常容错；但在面对“那些让工程师夜不能寐的未知软件故障或配置毒丸”时，传统架构往往会承受整区域瘫痪的代价。而引入单元化架构后，不仅依然具备跨 AZ 的容错能力，更关键的是能将灾难性故障严格限制在单个单元内部。正如演讲者所总结：“the failure in the cell-based service it's contained to the cell so the impact is 1 over N or n is the number of cells rather than the whole service”。虽然演讲中只示意了 3 个单元，但在生产环境中 $N$ 的取值可以大得多，因此 $1/N$ 的影响面所代表的受影响工作负载比例非常小。这种单元化思想不仅能用于区域服务，同样可以深度应用于单可用区服务（Zonal Services）。例如在 Amazon EC2 控制平面的设计中，每个可用区内部的服务也被进一步划分成了多个单元。某些可用区实际上由多个物理数据中心组成，如果将可用区内部的单元与底层物理基础设施严格对齐，那么即便发生单机房甚至硬件级的故障，受影响的爆炸半径也能进一步缩小，为可用区服务带来更极致的韧性。 ```mermaid flowchart TD accTitle: 基于单元的架构路由拓扑 accDescr: 展示外部流量经过单元路由器分发至互相隔离的多个服务单元内部 Client["客户端请求"] --> Router["Cell Router (单元路由器)"] subgraph CellSystem ["单元化服务 (Cell-Based Service)"] Router -->|"路由划分"| C1["Cell 1 (1/N 爆炸半径)"] Router -->|"路由划分"| C2["Cell 2 (1/N 爆炸半径)"] Router -->|"路由划分"| CN["Cell N (1/N 爆炸半径)"] end C1 -.->|"跨 AZ 部署"| AZ1["AZ-a / AZ-b / AZ-c"] C2 -.->|"跨 AZ 部署"| AZ1 CN -.->|"跨 AZ 部署"| AZ1 ``` - 爆炸半径收缩数学公式：单次严重故障影响比例由传统区域服务的 $100\%$ 下降至 $1/N$（$N$ 为单元总数）。 - EC2 控制平面应用：在单可用区服务内部继续推行单元化，并与物理数据中心设施对齐，实现物理级故障半径控制。 --- ### [24:21](https://www.youtube.com/watch?v=swQbA4zub20&t=1461s) 单元化架构的系统特性与扩展哲学基于单元的架构不仅能隔离故障，还能有效解决“吵闹邻居”（Noisy Neighbor）问题，实现严格的工作负载隔离。此外，它还彻底改变了 AWS 扩展系统的工程哲学。传统系统在面对业务增长时往往采用向上扩展（Scale Up），即不断向单一体系内堆砌容量。而在单元架构下，虽然单个单元也可以扩容，但 AWS 制定了一项极其关键的原则：单元与数据中心一样，必须设有最大容量上限（Maximum Size），绝不允许无限制膨胀。一旦单单元达到设计容量上限，系统不再允许该单元继续变大，而是通过横向新建并上线一个新单元来进行 Scale Out。拥有明确的上限规模带来了一个巨大的工程优势：工程师可以使用合乎现实的测试资源与配置，在单元的最大尺寸下进行全方位的压力测试与破坏性测试（Test to failure），从而对该单元在业务极限负荷下的运行表现拥有百分之百的把握。此外，较小的单元使日常运维和故障排查更加得心应手，工程师无需面对海量的全量日志和节点群，只需针对出问题的单个单元进行快速排查定位。 - 隔离优势：天然具备工作负载隔离与吵闹邻居抑制能力。 - 规模上限：单元设有硬性尺寸天花板，系统扩展完全依靠追加单元数量（Scale Out）。 - 可测试性：由于规模固定可控，可直接以最大规格实施打满乃至打崩的压力测试。 - 运维可控性：审查日志或排查异常节点时，作用域被圈定在单单元极小的数据与机器范围内。 --- ### [26:03](https://www.youtube.com/watch?v=swQbA4zub20&t=1563s) 单元大小权衡与极其轻薄的单元路由器在构建单元化架构时，首要的权衡取舍就是单元的尺寸：是采用“大量的小型单元”，还是“少量的大型单元”？小型单元的爆炸半径更小，极易进行打断点式的极限测试，运维排查也轻便；但其劣势在于固定成本更高（例如每个单元可能都需要专属负载均衡器），且当面对拥有海量工作负载的大型客户时，小单元可能容纳不下，迫使系统必须处理跨单元拆分客户的复杂逻辑。反之，大型单元具有更好的成本效益，易于容纳大客户，管理仪表盘也更为精简直观。对于这种两难取舍，演讲者明确给出了 AWS 的底线原则：“all things being equal we will always prefer the lower blast radius do these other considerations”。另一个系统级的核心焦点在于单元路由器（Cell Router）。由于所有流量都必须经过它分发给各单元，它是整套架构中仅存的全局共享组件。一旦单元路由器挂掉，爆炸半径将瞬间打回原形，重新扩散为整个区域。演讲者严肃指出，AWS 倾注了大量工程资源对该组件进行严苛的压力测试与实战淬火（Battle-hardened），以确保即使面临极端罕见的黑天鹅事件它也能屹立不倒。为了做到万无一失，保护该组件的核心手段就是竭尽全力降低其复杂度。正如演讲者向团队灌输的理念：“so when we talk to teams about adopting a cell-based architecture we call this component the thinnest possible layer to kind of reinforce should be really simple”。越薄、逻辑越纯粹，路由层产生 Bug 和系统崩溃的几率就越低。 - 架构权衡抉择：在同等条件下，AWS 永远优先选择更小的爆炸半径，而非单纯迎合运维便利或成本。 - 共享单点风险：Cell Router 若瘫痪会导致区域级故障重现，因此必须保持极高鲁棒性。 - 极简原则：路由层必须作为“可能的最薄一层（thinnest possible layer）”来设计与构建。 --- ### [28:42](https://www.youtube.com/watch?v=swQbA4zub20&t=1722s) 划分维度、“顺应纹理”与跨单元交互处理系统如何切分单元是架构设计的另一道考题。可以按客户维度划分，但在某些 EC2 控制平面场景下，单元是按照底层数据中心物理基础设施划分的；在涉及跨租户的复杂网络架构中，单元也可能以 VPC 为维度进行划分。面对纷繁复杂的业务场景，演讲者给出了经典建议：“the recommendation I always use is cut with the grain”。他类比木工原理进行解释：木材具有天然的纹理走向，顺着木纹极易劈开，横着逆纹则寸步难行；软件系统同样有其天然的业务“纹理”，切分必须顺应系统内在的数据流动与关联关系。在单元架构中，必须极力避免但有时又不得不面对的是跨单元用例（Cross-Cell Use Cases）。这种操作会污染轻薄路由层的纯粹性，并重新扩大故障半径。主要的跨单元挑战包括： 1. **分散-聚合查询（Scatter-Gather Queries）**：如 EC2 的 `DescribeInstances` API，单个请求需要分发至多个单元打探数据，再聚合为单条响应返回。 2. **批处理操作（Batch Operations）**：如包含多个实例 ID 的 EC2 `TerminateInstances` API，一次调用需要跨单元协调执行。 3. **跨单元协同写入（Coordinated Writes）与单元迁移（Cell Migration）**：这是最复杂的一环。 ```mermaid flowchart LR accTitle: 跨单元工作负载热迁移时序 accDescr: 展示从源单元到目标单元的工作负载克隆、增量同步与原子切换过程 subgraph Step1 ["第一阶段: 后台克隆"] S1["源单元 (Cell 1)"] -->|"持续提供读写服务"| S1 S1 -.->|"后台数据镜像同步"| T1["目标单元 (Cell 2) 隐形副本"] end subgraph Step2 ["第二阶段: 追齐收敛"] S2["源单元 (Cell 1)"] -->|"增量数据追赶"| T2["目标单元 (Cell 2)"] end subgraph Step3 ["第三阶段: 原子切换"] Freeze["短暂冻结双端写入"] --> Cutover["Cell Router 修改路由指向"] Cutover --> Active["目标单元 (Cell 2) 正式接管"] end Step1 --> Step2 --> Step3 ``` 单元迁移类似于虚拟机热迁移（VM Migration）：当需要均衡单元负载、解决热点，或者扩容了新单元需要重平衡数据时，系统会在目标单元建立隐形克隆，并在源单元持续承接变更的同时后台拉取增量追齐同步；在数据无限逼近一致的最后一刻，短暂冻结两端并由单元路由器完成原子切换（Atomic Flip）。这类跨单元写入极具挑战，必须在路由层予以严密谨慎的协调。 - 划分哲学：识别系统的核心实体走向，切分维度绝不与天然数据流向对抗（Cut with the grain）。 - 跨单元调用痛点：Scatter-Gather 查询与批量写入会强制路由层承载额外编排逻辑，必须极度克制。 - 迁移机制：基于无感克隆、增量追齐、末尾瞬时冻结与原子切流，将客户无缝平移至新单元。 --- ### [32:09](https://www.youtube.com/watch?v=swQbA4zub20&t=1929s) 迈向无状态与软状态场景：混洗分片（Shuffle Sharding）的引入尽管引发全区域瘫痪的灾难性事件极其罕见，AWS 在日常高可用建设上已投入了大量心血，但出于对极致韧性的偏执追求，AWS 仍然不惜耗费巨额工程资源推行单元化，以期在百年一遇的黑天鹅事件袭来时将损失死死限制在 $1/N$。然而在单元化之外，还有一种在无状态（Stateless）或软状态（Soft-State）服务中表现更加惊艳、更具扩展性的前沿设计技术正在被广泛采用，那就是混洗分片（Shuffle Sharding）。演讲者情绪激昂地指出，这项技术甚至比单元架构“更加令人印象深刻和激动人心”，它将在纯无状态及弱状态体系中带来革命性的故障隔离效果，成为抵御毒丸和突发宕机的高维防御工具。 - 韧性投资哲学：即便黑天鹅概率无限接近于零，AWS 也宁愿投入额外工程代价确保极端情况下的半径收敛。 - 技术进阶：从单元化架构顺畅过渡至数学表现更为卓越的 Shuffle Sharding。 ### [03:20](https://www.youtube.com/watch?v=swQbA4zub20&t=200s) 从单体池到单元化架构与洗牌分片（Shuffle Sharding）演讲者在现场借用了拉斯维加斯赌场相关的扑克牌花色图标，来生动比喻多租户系统中的各种客户流量。假设系统拥有 8 个处理节点，由一台负载均衡器负责分发请求，同时有 8 个不同的客户正在调用系统。如果其中一位被称为“方块（Diamond）”的客户因请求过于昂贵，或是触发了系统潜在的 Bug，导致其发送的恶意工作负载击垮了它所连接的第 1 台服务器。由于该客户遭遇失败后不会离开，而是持续不断地重试，最终会把负载均衡器背后的全部 8 个节点逐一打崩。在没有做隔离保护的情况下，“爆炸半径（Blast Radius）”将覆盖 100% 的客户。正如演讲者所警告的：“This is like the worst case scenario we really want to avoid, this is where we go to cell-based architecture.” 为了打破这种连锁故障，AWS 采用了“单元化架构（Cell-Based Architecture）”，即将节点划分为不同的单元（Cell），每个单元仅分配一部分客户。若将 8 个节点分为 4 个单元，方块客户制造的崩溃将被严格限制在它所在的那个单元内，系统的故障爆炸半径直接从 100% 骤降至 25%（客户数除以单元数），获得了 4 倍的可靠性改善。然而，更强大的手段是“洗牌分片（Shuffle Sharding）”。该方案不再把客户固化在某个固定的独立单元内，而是利用哈希函数计算，将每位客户随机且确定性地映射到容量池中的 2 个节点上（例如方块分配在第 1 和第 4 节点，黑桃分配在另一对节点）。 ```mermaid flowchart TD accTitle: 洗牌分片故障隔离机制 accDescr: 展示方块租户击垮两台节点后，由于洗牌分片机制，红心租户仍有一台节点存活可继续提供服务。 subgraph Cluster["8 节点集群"] N1["节点 1 (宕机 ❌)"] N2["节点 2 (正常)"] N3["节点 3 (正常)"] N4["节点 4 (宕机 ❌)"] N5["节点 5 (正常)"] N6["节点 6 (正常)"] N7["节点 7 (正常)"] N8["节点 8 (正常)"] end TenantD["租户：方块 (有缺陷的请求)"] -.-> N1 TenantD -.-> N4 TenantH["租户：红心 (正常业务)"] -.-> N1 TenantH ==>|"客户端重试成功"| N2 ``` 此时如果方块客户再次打崩了分配给它的两台节点，真正巧妙的数学奇迹便显现出来了： - 与方块共享其中一台节点的“红心（Hearts）”客户，其另一半请求被分配到了完全未受影响的正常节点上。只要红心客户端具备自动重试机制，就能无缝绕过故障，维持系统的高可用运行。 - 在 8 个节点中任选 2 个节点的数学组合（二项式系数）为 $\binom{8}{2} = 28$ 种。 - 只有同样完全落在与方块相同两个节点上的客户才会彻底瘫痪，完全受损的理论比例仅为 $1/28 \approx 3.6\%$。 - 系统爆炸半径从最初的 100%，经由单元化降到 25%，再通过洗牌分片进一步暴降至仅 3.6%。 --- ### [37:00](https://www.youtube.com/watch?v=swQbA4zub20&t=2220s) 洗牌分片的数学原理与实现红线洗牌分片背后的组合数学（高中所学的二项式系数）随着容量池规模 $N$ 的增长会呈现爆炸式倍增。演讲者给出了一个工业级大规模系统的真实推演：假设后端运行着 100 个节点，系统为每位客户分配 5 个节点组合。从 100 张牌的牌组中任取 5 张，总共可以产生超过 7500 万种（约 75,287,520 种）互不相同的分片组合。在这样的规模下，即使某一位异常客户彻底瘫痪了它的全部 5 个节点： 1. 约 77% 的客户根本没有与它重叠任何节点，完全零感知； 2. 99.8% 的客户（在前几层组合中）依然保有绝大多数（过半数）节点可用，无需复杂故障转移即可轻松通过多数健康节点处理流量； 3. 与故障源完全重叠全部 5 个节点的概率低到微乎其微。演讲者高度赞誉了这种设计理念：“What's magical about this and it's all in the math is we've created a multi-tenant system and then used the shuffle sharding to create a single basically a single tenant experience.” 要使洗牌分片在生产中发挥这种单租户般的物理隔离体验，系统必须满足三项严苛要求： - **容错客户端（Fault-Tolerant Client）：** 客户端必须在遇到瞬时失败时具备重试逻辑，这是绕过局部损坏节点的基石。 - **不可打破的静态分配（Fixed Assignments）：** 必须严格遵循“发什么牌就打什么牌”。如果因为某客户的 5 个节点宕机，调度机制自作聪明地做自动故障转移（Failover）将它挪到另外 5 个节点，该问题就会演变成级联传染病，击垮整套基础设施。 - **智能路由层（Routing Mechanism）：** 必须具备感知洗牌分片的路由器，或直接借助 DNS 解析——AWS 经常为不同客户提供专有的 DNS 域名，该域名直接解析到该客户对应的洗牌节点 IP 集，以极低成本实现内置路由。除服务器外，洗牌分片还广泛用于工作队列（Queues）与存储资源。 --- ### [39:42](https://www.youtube.com/watch?v=swQbA4zub20&t=2382s) 运行实践：分区意识与渐进式金丝雀发布再完美的架构隔离，如果在日常运维中粗暴地全局操作，也会瞬间毁于一旦。AWS 将运维防线的核心提炼为一种“工程宗教”般的文化意识——分区隔离（Compartmentalization）。演讲者引用了 AWS CTO Werner Vogels 当年的一篇博文强调：“I'd wager that every new AWS engineer knows within their first week if not their first day that we never want to touch more than one zone at a time.” 无论是可用区独立还是区域隔离，一旦因人为疏忽或自动化任务同时跨多个故障域执行，一切故障半径控制都会彻底失效。为了践行这一原则，AWS 的软件发布机制采取了极其审慎的“错峰推进（Staggered Deployments）”： - **发布节奏的张弛平衡：** 部署既要足够快以满足客户对新功能的需求，又要足够慢以保证每次扩大影响面时都有充足的信心验证稳定性。 - **层层推进的故障域：** 软件遵循 Cell $\rightarrow$ Availability Zone $\rightarrow$ Region 的顺序推进，绝不并行跨域部署。 - **域内的微粒度测试：** 即使在单个可用区或单元内，也会首先进行单机盒测试（One-box test）；在 EC2 硬件层面的部署中，通常会从仅 5 到 10 台机器起步，确认平稳后才逐步放量。 - **金丝雀流量与自动回滚：** 部署过程中并行运行自动化集成测试与合成金丝雀（Canaries），这些测试应用全天候模拟真实客户调用系统 API。只要监控指标出现异常扰动，部署流程会立即触发自动回滚，待团队复盘调查并修复缺陷后才会重启部署。 --- ### [42:35](https://www.youtube.com/watch?v=swQbA4zub20&t=2555s) 端到端自动化与“谁构建谁运维”的全生命周期闭环除发布流程外，网络设备配置变更、服务安全凭据（Credentials）轮换下发等所有底层基建操作都已完全剔除人工干预，全面实现自动化。因为人类是极易犯错的，而通过代码固化的自动化脚本支持严格的同行代码审查（Code Review）、自动化测试，具备可预测性且能千百次高精度复现。在组织文化层面，AWS 与亚马逊坚持“端到端所有权（End-to-End Ownership）”与“构建者即运维者（Builder-Operator）”模型： - **双重角色闭环：** 设计和编写软件的工程师，正是负责将代码发布上线、并在生产发生告警时值班（On-call）响应的人。 - **正向工程反馈：** 运维实践中暴露的系统脆弱点，能无损转化为下一轮架构设计的关键输入，促使开发人员从源头上规避架构反模式。 - **COE（错误纠正报告）实践：** 当严重故障发生后，负责填报 COE（Correction of Errors）模板的正是这批工程师。他们深入审视系统的爆炸半径，并亲自编写工程整改代码，确保当同类风险再次降临时，爆炸半径至少被削减一半（Cut in half or more）。 ### [44:31](https://www.youtube.com/watch?v=swQbA4zub20&t=2671s) 故障爆炸半径缩减的总结与 Q&A 开始讲师在此处对整场演讲的核心逻辑进行了全面收尾。AWS 采用多种控制与隔离机制来缩减故障的影响范围（Blast Radius）：最外层是 Region（区域）及其强隔离特性，向内推进到 Availability Zone（可用区，AZ），再到可用区之上的另一维度——Cell（单元架构），接着通过洗牌分片（Shuffle Sharding）的组合数学魔力构建“虚拟单元”。讲师强调，这些架构隔离边界必须辅以严格的运维实践来保护：包括分步骤、自动化的分阶段部署（Phased Deployments），以及“Builder-Operator”（构建即运维）的团队文化，让工程师贴近生产一线，深切理解自身决策如何影响系统可用性。 ```mermaid flowchart TD accTitle: AWS Blast Radius Isolation Layers accDescr: Diagram showing the nested isolation mechanisms from Regions down to Shuffle Sharding protected by operational practices. subgraph ArchLayers["架构隔离层级"] R["Region (强隔离)"] --> AZ["Availability Zone (可用区)"] AZ --> C["Cell (单元架构)"] C --> SS["Shuffle Sharding (虚拟单元/组合分片)"] end subgraph OpsGuard["运维实践保护"] PD["自动化分阶段部署 (Phased Deployments)"] BO["构建即运维团队 (Builder-Operators)"] end OpsGuard -.->|"守护隔离边界"| ArchLayers ``` 总结完毕后，讲师宣布进入现场问答环节，并引导观众移步走廊正中央设立的立式麦克风提问。 - 讲师总结的核心主线：“variety of containment compartmentalization mechanisms that we use to reduce blast radius” - 运维机制保障：“step-by-step phased deployments that are automated and then service teams that are builder operators” --- ### [46:01](https://www.youtube.com/watch?v=swQbA4zub20&t=2761s) 墨菲定律与历史教训：S3 单比特翻转引发的区域性宕机与人为失误现场第一位提问者询问讲师：AWS 是否经历过自认为万无一失、覆盖了所有故障模式，却依然撞上“墨菲定律”（Murphy's Law）的极端案例？讲师毫不犹豫地给出了 2008 年 Amazon S3 发生的一次惨痛历史事件。S3 团队对数据完整性极其重视，系统自上而下拥有密集的校验和（Checksum）层级，以防止任何网络比特翻转导致数据损坏。然而在 2008 年，某台服务器网卡（NIC）出现故障，偶尔会翻转一个比特。S3 底层使用 Gossip 协议检测整个集群服务器的健康状态，该协议在处理数据包时发现了一个“从未见过的奇怪服务器名称”——因为机器名里正好有一个比特被翻转了。这直接触发了开销极大的状态对账协议，引发系统级雪崩，最终导致整个区域的 S3 服务彻底瘫痪。事后复盘发现，整个技术栈从上到下都做了校验和，唯独 Gossip 协议通信这一层遗漏了。正如讲师所总结的：“So one server, one NIC, one bit took down our regional service, and that was the only layer we had checksumming all the way up and down the stack—that was the only layer that didn't have the checksumming, someone we'd forgotten to add it there.” 紧接着观众追问了另一起更近期的 S3 人为操作失误事件。讲师确认了该事件的存在，并指出这恰恰印证了演讲中强调的原则：如果操作没有敬畏故障隔离边界，再完美的架构设计也会崩溃。当时涉事工程师深知架构哲学，但依然在命令行输入时犯下了失误，这再次凸显了全面自动化与严格测试的核心价值。 - 2008 年 S3 事故根因：单台物理机网卡（NIC）间歇性单比特翻转（Bit Flip）。 - 传播链条：单比特翻转篡改机器名 $\rightarrow$ Gossip 协议触发高开销对账 $\rightarrow$ 区域级 S3 全面中断。 - 人为故障防范：依赖命令行极易产生人为失误，必须推行软件自动化防护。 --- ### [48:49](https://www.youtube.com/watch?v=swQbA4zub20&t=2929s) 架构隔离层级的收费与数据中心底层亲和性一位名为 Peter 的观众提问：演讲中讲到的 Region、AZ、Cell 到 Shuffle Sharding 等层级，是直接向客户开放还是需要付费购买的服务？讲师澄清，Region 和 AZ 是直接暴露给客户的基础构件；而 Cell 与 Shuffle Sharding 属于 AWS 内部的系统架构机制，讲师形象地打比方道：“they are like the watertight compartments inside the ship... you don't really care about that and you get them for free, it's just part of the safety of the ship.”（它们就像轮船内部的水密舱，乘客无须操心，它们作为船体安全的一部分完全免费提供）。观众随后深入追问：如果客户想为自己的服务构建 Cell，并且希望将 Cell 精确对齐到特定的物理数据中心（Data Center），是否可行？讲师解释，客户可以通过可用区（AZ）获得计算资源的亲和性（如 EC2、CloudHSM、EBS），因为 AZ 与底层物理设施逻辑绑定；但是一个 AZ 在物理上通常由一个或多个数据中心构成。AWS 将多数据中心聚合抽象为 AZ，并不向客户透传或开放对底层具体物理机房的选择控制权。 - Region 与 AZ：对用户显式可见、由用户自主配置。 - Cell 与 Shuffle Sharding：AWS 内部基础设施实现，对用户透明、免费提供。 - AZ 与物理机房对应关系：一个 AZ 对应一个或多个物理数据中心（Data Center），但多机房细节对外部黑盒化。 --- ### [50:46](https://www.youtube.com/watch?v=swQbA4zub20&t=3046s) 跨 Cell 部署时的版本漂移与变更节奏管理现场观众提出疑问：当部署单元从 Region 细化到 Cell 级别时，版本管理与运维复杂度会受到什么影响？跨多个位置进行渐进式部署时，如何处理版本不一致的时间窗口？讲师对此表示认同，指出跨区域、跨 Cell 推进部署时，版本在不同位置存在不一致是必然的权衡考量。部署推进周期的长短取决于系统本身，短则几天，长则几周。对于极为核心和敏感的基础网络组件（例如 Amazon VPC），变更风险极高，整个团队对部署节奏把控极其严谨缓慢。因此团队必须在运维层面高度警惕不同节点上运行的具体版本，并做好向前/向后兼容的严格设计。 - 渐进部署周期：根据系统复杂度与安全要求，耗时从数天至数周不等。 - 极高风险系统示例：Amazon VPC 等底层核心网络组件的变更推进尤为审慎缓慢。 --- ### [52:00](https://www.youtube.com/watch?v=swQbA4zub20&t=3120s) 全局服务（以 IAM 为例）的控制面与数据面解耦观众提问有关全球性服务（Global Services）的架构设计：当涉及跨 Region、跨可用区或 Edge 节点时，这些全局服务的控制面（Control Plane）与数据面（Data Plane）是如何保证弹性的？例如 AWS IAM（Identity and Access Management）。讲师指出 IAM 是一个非常经典的案例。从控制面来看，账户信息、用户凭据等元数据在全球范围内同步，属于全局服务。但在架构设计上，每个 Region 都部署了独立的 IAM 数据面。该本地数据面完全能够与 IAM 的全局真相源（Source of Truth）断开连接并独立运行。讲师特别强调了一条核心设计哲学：“there is another philosophy that we have, which is the separation between control plane and data plane, and ensuring that the data plane can continue to function even if there is a control plane issue.”（我们拥有另一项核心哲学，那就是控制面与数据面的严格分离，并确保即使控制面发生故障，数据面也能持续运转）。即使全球控制面不可用或区域网络隔离，每个 Region 依然能利用本地缓存的数据面正常校验 API 凭据并授权请求，从而阻断了全局连锁故障。 ```mermaid flowchart TD accTitle: IAM Control Plane vs Data Plane Separation accDescr: Shows the global IAM control plane replicating credentials to independent regional data planes that run disconnected. subgraph GlobalCP["全局控制面 (Global Control Plane)"] ST["IAM Source of Truth (账户/凭证管理)"] end subgraph RegA["Region A"] DPA["IAM 区域数据面 (独立运行)"] API_A["本地服务 API 鉴权"] --> DPA end subgraph RegB["Region B"] DPB["IAM 区域数据面 (独立运行)"] API_B["本地服务 API 鉴权"] --> DPB end ST -.->|"异步同步状态"| DPA ST -.->|"异步同步状态"| DPB ``` - IAM 架构：全局控制面管理 Source of Truth；各 Region 拥有完全独立的数据面。 - 容灾表现：控制面不可用或发生网络分区时，本地鉴权与核心业务流程不受影响。 --- ### [53:48](https://www.youtube.com/watch?v=swQbA4zub20&t=3228s) 单元架构（Cell-based）与洗牌分片（Shuffle Sharding）的适用边界及结合最后一位观众提问：在单元化架构中，控制层路由往往需要顺应服务的特性（Cut with the grain）来划分；而切换到洗牌分片时，是否也是在同一层面实现？洗牌分片是否会导致失去基于服务状态特性的控制？讲师分析道，两者在有状态与无状态场景下的适应性截然不同。对于典型的单元化架构，系统通常需要严格管理状态，不能随意将状态散落在整个集群中；而洗牌分片则天然更适合于无状态服务，或者是节点能通过本地缓存应对请求的“软状态”（Soft State）场景，此时请求对特定节点没有强亲和性要求。讲师进而透露了一项前沿演进：两者并非非此即彼，团队正在探索将单元架构与洗牌分片进行多层叠加——即在无状态系统中构建 Cell，再在各个 Cell 之间对请求进行 Shuffle Sharding，以获得更高的弹性组合特性，目前 DynamoDB 团队正在积极推进此项架构实践。最后，讲师总结了其权衡本质：“shuffle sharding is not an improvement on cell-based in all cases, it's really a system-specific on when you might be able to apply it.”（洗牌分片并非在任何情况下都是单元架构的升级版，它完全取决于具体的系统属性）。问答结束后，讲师在全场掌声中致谢并宣布演讲结束。 - 无状态 / 软状态：非常适合应用洗牌分片，无强节点亲和性（Node Affinity）。 - 强状态系统：必须采用 Cell 等确定性路由以保障状态约束。 - 架构进阶演进：在 Cell-based 架构之上叠加 Shuffle Sharding 请求路由（如 DynamoDB 的工程演进）。 ## Notable quotes > [02:35](https://www.youtube.com/watch?v=swQbA4zub20&t=155s) Failure isn't binary; it's not 'it's failing or it's not'—there is a degree of impact. — Peter Vosshall > [39:42](https://www.youtube.com/watch?v=swQbA4zub20&t=2382s) I'd wager that every new AWS engineer knows within their first week if not their first day that we never want to touch more than one zone at a time. — Peter Vosshall > [04:23](https://www.youtube.com/watch?v=swQbA4zub20&t=263s) A single failure can cascade and cause an entire system to get infected. — Peter Vosshall > [53:48](https://www.youtube.com/watch?v=swQbA4zub20&t=3228s) shuffle sharding is not an improvement on cell-based in all cases, it's really a system-specific on when you might be able to apply it. — Peter Vosshall ## Mentioned AWS, Peter Vosshall, Eric Brewer, Seth Gilbert, Nancy Lynch, Werner Vogels, JPL, Amazon EC2, Amazon S3, Amazon RDS, Amazon EBS, Amazon VPC, Amazon DynamoDB, AWS IAM, Amazon SQS, Amazon SageMaker, AWS CloudHSM, Elastic Load Balancer ## Video description At AWS, we obsess over operational excellence. We have a deep understanding of system availability, informed by over a decade of experience operating the cloud and our roots of operating Amazon.com for nearly a quarter-century. One thing we've learned is that failures come in many forms, some expected, and some unexpected. It's vital to build from the ground up and embrace failure. A core consideration is how to minimize the "blast radius" of any failures. In this talk, we discuss a range of blast radius reduction design techniques that we employ, including cell-based architecture, shuffle-sharding, availability zone independence, and region isolation. We also discuss how blast radius reduction infuses our operational practices. [Source](https://www.youtube.com/watch?v=swQbA4zub20)

Source quotes

[2:35] Peter Vosshall: Failure isn't binary; it's not 'it's failing or it's not'—there is a degree of impact. [39:42] Peter Vosshall: I'd wager that every new AWS engineer knows within their first week if not their first day that we never want to touch more than one zone at a time. [4:23] Peter Vosshall: A single failure can cascade and cause an entire system to get infected. [53:48] Peter Vosshall: shuffle sharding is not an improvement on cell-based in all cases, it's really a system-specific on when you might be able to apply it.

Source quotes

[39:42] Werner Vogels: I'd wager that every new AWS engineer knows within their first week if not their first day that we never want to touch more than one zone at a time. [24:21] Peter Vosshall: Cells, like our data centers, have a maximum size; we won't let them grow past a certain point, and so if we need to continue to grow a system, rather than growing the cells past that point, we'll add another cell.

Original transcript

[0:01] good afternoon thanks for coming to this [0:05] session who's here at reinvent for the [0:07] first time awesome welcome should be an [0:12] exciting week lots of sessions lots of [0:15] parties lots of fun so I am Peter Voss [0:19] all I am a distinguished engineer at AWS [0:22] and today I'm going to talk about how [0:25] AWS minimizes the blast radius of [0:27] failures so I spent my career focused on [0:31] building highly available large-scale [0:33] distributed systems and one of the [0:36] things I've learned and something you [0:38] learn quickly in this field is that [0:40] there are some fundamental properties of [0:41] the universe that you basically have to [0:43] contend with the first is the speed of [0:46] light so 186 miles per millisecond is [0:51] pretty darn fast but as the folks at JPL [0:54] this morning trying to land their their [0:57] Mars rover learned it really gets in the [0:59] way when you travel any long distance [1:01] communications by the way they did land [1:03] the rover successfully so [1:05] congratulations to our our friends at [1:07] JPL a happy Amazon customer a tip us [1:10] customer the next thing is sort of this [1:14] pesky property universe known as the cap [1:17] theorem so this is a an observation you [1:21] can't Simas simultaneously have [1:22] consistency and availability in a [1:25] distributed system that is built to be [1:26] partitioned tolerant so this was first [1:28] postulated by Eric Brewer in 98 and then [1:31] proven by Seth Gilbert and Nancy Lynch [1:34] at MIT [1:36] perhaps most annoying though is this [1:38] observation by some guy named Murphy [1:40] which is basically stuff breaks [1:43] inevitably things will fail in a variety [1:46] of interesting ways I don't know if this [1:48] has been proven but in my experience [1:49] that seems to have been borne out to be [1:51] the truth now security availability [1:56] durability these are all super important [1:59] to AWS in fact they're our top priority [2:01] and yet we have to contend with with [2:04] Murphy's Law so today I'm going to talk [2:08] about the techniques that Amazon uses [2:11] AWS uses [2:13] to contain the blast radius or the the [2:16] degree of impact when Murphy's Law does [2:19] in fact strike and I hope you'll walk [2:21] away from this talk getting a better [2:23] understanding of how a DPS builds [2:25] resilience into our systems and also [2:28] some techniques that you can use to [2:30] reduce the blast radius in your own [2:32] systems so this term blast radius is a [2:36] it's a really useful one because failure [2:40] is in binary it's not a it's failing or [2:43] it's not there is a degree of impact [2:45] it's a really useful term it's part of [2:48] our common language at AWS and basically [2:52] it's a way to describe the degree of [2:54] impact so a table us if we have a [2:57] failure one way to talk about blast [2:59] radius as well how many customers did it [3:01] impact or how many workloads or what [3:04] functionality maybe it was just a [3:05] portion of the functionality that was [3:07] impacted and then finally in what [3:09] locations was it a racket a data center [3:11] was it a whole data center was it an [3:13] entire region obviously we would always [3:16] prefer a smaller blast we as a single [3:18] rack failing to a larger scope failure [3:23] so while we do relentlessly focus on [3:26] preventing failures and and I would say [3:28] that we've gotten very good at keeping [3:30] our systems extremely highly available [3:32] we also relentlessly focus on reducing [3:35] blast radius for those very rare cases [3:38] where we do have a failure and try to [3:40] contain it [3:41] and make it as small as possible one way [3:46] to do that we do this is in our [3:48] correction of errors process so that's [3:51] our post mortem process that we use [3:53] whenever there's an event the service [3:55] team will go through analyze what were [3:58] the root causes of the failure and then [4:01] identify a set of actions to take to [4:03] prevent recurrence one of the questions [4:06] in the template for doing this [4:09] correction of errors is about customer [4:10] impact and we asked as a thought [4:12] exercise [4:12] how could you cut the blast radius for a [4:14] similar event in half so we're always [4:16] thinking about even when we do have an [4:18] event how can we make it even smaller [4:20] blast radius the next time [4:23] so before talking about how to do that [4:25] let's talk about how things can fail so [4:26] one way things can fail as well servers [4:28] crash maybe not like this but servers do [4:31] crash disks fail in a variety of [4:33] interesting ways they might go [4:35] completely offline they might have [4:36] random i/o errors network devices can [4:39] fail and if they don't fail they could [4:41] introduce random bit flips we've [4:43] definitely seen this happen a few times [4:46] meanwhile outside the data center [4:49] utility workers can accidentally cause [4:52] fiber cuts you can have electrical [4:55] storms that can cause utility power [4:58] failures in a more extreme scenario you [5:03] could have data centers actually get [5:05] physically damaged by storms or fires so [5:10] far I've talked about physical failures [5:11] right but there's also non-physical [5:14] affairs we have to worry about one is [5:16] just a surge of traffic whether it's a [5:18] DDoS attack or a extreme surge in demand [5:23] on one of our services which can cause [5:26] overload conditions we have to worry [5:29] about black swan' requests we actually [5:31] call them these poison pills internally [5:33] I've done a little research and that [5:35] seems to be just an AWS term but you can [5:37] think of these as these particularly [5:39] problematic requests that are either [5:41] really expensive or there's something [5:43] about them that trigger a bug in in the [5:46] system and they can be particularly [5:48] pernicious because a client will retry [5:51] after a failure and so a single failure [5:54] can cascade and cause an entire system [5:57] to get infected [5:58] so we worried a lot about these poison [6:00] pills or black swans we also have to be [6:04] mindful the fact that sometimes a [6:05] software deployment or configuration [6:06] change could introduce the problem and [6:10] then of course most generally there are [6:12] bugs and obviously none of us want to [6:16] have bugs get into production and we [6:18] have a very good success rate and not [6:20] letting bugs get into production but [6:21] there is still that that possible [6:23] eventualities and so now that I've [6:28] talked about failures and what we mean [6:30] by blast radius I'll better talk about [6:32] the various techniques we use to contain [6:34] it [6:35] so region isolation availability zone [6:38] independence cell based architecture [6:40] shuffle sharding and then finally in [6:43] tandem with all these techniques the [6:44] operational practices that we use to get [6:47] the desired outcome so let's start with [6:50] region isolation so as you probably know [6:53] AWS is deployed globally in 19 separate [6:57] locations that we call regions and we [7:02] actually have five additional regions [7:03] that we've announced that will be coming [7:04] on online soon and as a customer you [7:08] choose the region that you want to run [7:10] your workloads in based on factors like [7:12] like latency or maybe data residency [7:17] each one of these regions is a separate [7:20] distinct stack of AWS services and each [7:26] region has a separate set of end points [7:28] for the api's you use to interact with [7:30] the services so for example in ec2 if [7:32] you want to interact with ec2 in uswest [7:35] one you would use the ec2 that uswest [7:38] one that either Sam it's not that common [7:40] endpoint meanwhile if you wanted to use [7:43] easy to than us used you wussies - then [7:46] you would use separate endpoint and the [7:49] reason is there's no single global ec2 [7:53] there's a separate stack in every region [7:56] and these are isolated in San Shi ations [7:58] and they don't know about each other [8:01] so this is a shared nothing architecture [8:03] which gives us the ultimate in blast [8:05] radius protection so there's our ec2 you [8:10] know Amazon Elastic Compute cloud [8:11] service in u.s. west wall and US east - [8:14] they don't talk to each other they don't [8:16] know about each other same is true for a [8:18] service like Amazon FQs Amazon sage [8:20] maker and and so on now you might be [8:24] wondering well but there are multi [8:27] region or global features you know what [8:30] about those features things like s3 [8:34] cross region replication or dynamos [8:36] global tables or ec2 virtual private [8:40] cloud peering so let me talk about how [8:43] we address those by going through an [8:45] example so there [8:46] is an example of inter-regional private [8:49] cloud peering so virtual private cloud [8:51] or VPC basically lets you provision the [8:54] logically isolated section of the AWS [8:57] cloud where you can launch resources in [9:00] a virtual network that you define you [9:02] define the IP addresses in your virtual [9:03] network configure route tables gateways [9:06] and so forth and with inter region DPC [9:10] peering you can actually take two v pcs [9:12] in two different regions and virtually [9:15] connect them so they can communicate [9:17] without having to go over the the public [9:20] internet so here's our V PC peering [9:24] connection now setting one of these [9:26] things up involves a workflow and a [9:28] setup approvals particularly because [9:30] they can be v pcs owned by different [9:32] customers so you want both customers to [9:34] agree to establishing this connection [9:37] and that involves configuration changes [9:39] in the V PC configuration on each of [9:44] these in each of these regions but these [9:47] easy to control planes don't talk to [9:49] each other so how do we accomplish this [9:51] the answer we have a dedicated service [9:55] called the cross region Orchestrator [9:56] that sits on top of these systems and [10:01] implements the workflow manages through [10:03] the approval process and comes through [10:05] the sort of the front door of BC - just [10:08] like any other API would and it also has [10:11] a certain number of safety features it [10:13] ring fences some of the interactions to [10:16] ensure that there is no possibility for [10:18] multiple regions to be impacted by you [10:20] know whatever issue there could be at [10:22] the same time so we can preserve that [10:24] single region blast radius so kind of a [10:29] recap here we've got all these regions [10:32] and we're deeply committed to this [10:34] principle of region isolation so in the [10:37] event of say an earthquake in Japan [10:39] which we did experience a few years ago [10:42] there could be impacted in that region [10:44] but it's going to be limited to that [10:45] region so in the worst case we have a [10:48] single region blast radius in case of a [10:50] failure that's still pretty big right [10:54] that's that's a lot of impact especially [10:56] if you're [10:57] a customer that sits only in that region [11:00] so obviously we want to do better so how [11:04] do we do better now I'm gonna talk about [11:06] availability zone independence so let's [11:09] drill into the design of a region kind [11:11] of double click on it and see how we [11:13] limit blast radius within a region so [11:17] there's our region a region is actually [11:18] composed of well before I talk about its [11:21] proposed over a region sits in a in a [11:23] location right so at the macro scale its [11:25] Northern Virginia or Dublin or Frankfurt [11:28] and but it's not a single data center [11:31] it's it's actually a it's composed of [11:35] multiple data centers that are spread [11:38] across the metropolitan area of the [11:41] region and we call these different [11:43] locations availability zones and their [11:44] and their cross connected with [11:47] high-speed private fiber links now these [11:53] are far enough apart from each other [11:54] that there's a very very low possibility [11:58] for correlated failure except maybe in [12:00] the in the earthquake case so you can [12:02] think of them as miles apart and that [12:06] means if a tornado comes through it's [12:07] unlikely that it would hit multiple [12:09] facilities if there's a utility issue we [12:13] actually run off of different utility [12:15] suppliers in a region so that won't [12:17] affect multiple availability zones at [12:18] the same time and at the same time [12:21] they're close enough to each other so [12:23] again think about it as miles away they [12:26] don't have that pesky speed of light [12:27] issues you can think of them logically [12:28] being in the same place run things like [12:31] synchronous replication protocols and so [12:32] forth without any latency penalty so [12:37] each availability zone is basically n [12:40] data centers it's not just one data [12:41] center in some so in some cases because [12:44] we keep the size of the data center set [12:48] to a fixed maximum size in larger AZ's [12:51] we might have multiple buildings for na [12:53] Z and then we have n of those [12:56] availability zones per region usually [12:58] three or more in fact in all future [13:01] region builds will have at least three [13:03] and that's useful first for a certain [13:06] should be the system's consensus [13:08] protocols that [13:10] best suited when you have had these [13:11] three locations so you can get to [13:14] consensus agreement across them globally [13:17] we have 19 regions with 57 total [13:21] availability zones and five more regions [13:23] coming online with 15 more AZ's so with [13:28] this architecture of AZ's within a [13:30] region we now have the possibility to [13:32] reduce the blast radius because we [13:35] reduce the possibility of a correlated [13:37] failure across the entire region and you [13:40] can take advantage of this multi a-z [13:42] architecture in old applications just [13:44] like we do in a DBMS by using a multi [13:46] a-z architecture for your application so [13:49] let me go through a quick example here [13:51] super simple so you have your [13:54] application it runs on a set of [13:55] instances that you've deployed across [13:57] multiple AZ's and then run an elastic [14:00] load balancer to low balance the traffic [14:02] across them and then behind the scenes [14:04] are using you know a relational database [14:06] maybe for your persistence and that set [14:09] up in a multi a Z primary standby pair [14:12] so if you have a failure in one of the [14:14] availability zones the elastic load [14:16] balancer will detect that and stop [14:18] sending traffic to the field a Z or the [14:20] instances in the failed a Z [14:23] meanwhile the application is connected [14:26] to the master database that has gone [14:28] away me because there's a power failure [14:29] or a network failure but you could fail [14:33] that database over to the to the healthy [14:38] AZ which happens automatically with [14:40] Amazon RDS and then your application is [14:43] up and running again [14:43] and these failure detection events could [14:47] happen fairly quickly so there might be [14:49] a slight hiccup the operation your [14:51] application but otherwise it's almost [14:53] like a non-event that you lost you know [14:55] one of the data centers in your [14:56] application so this is how all of our [14:58] services that run regionally in AWS are [15:03] designed and operated it's a really [15:07] powerful model cuz it gives you this [15:08] fault tolerance and it basically means [15:11] that you can withstand an AZ failure and [15:14] not have any impact on your customers [15:16] it's also a powerful design for [15:18] durability so s3 uses the multi AZ model [15:21] to get its 11 [15:22] of durability and so with this [15:25] architecture you can basically get to [15:26] zero blast radius when you have a data [15:28] center and get hit by tornado or a [15:31] utility worker hitting a cutting out [15:33] fiber connection so what about the [15:36] services in a WMS that are zonal so some [15:39] of our applications are some of our [15:41] services give you the opportunities and [15:43] application to have zonal resources like [15:46] an ec2 instance or an EBS volume you [15:50] decide which AZ these live in and that's [15:52] part of the ear story for creating a [15:54] resilient application [15:56] these are zone specific resources and [16:00] for minimum blast radius considerations [16:04] we actually have a zone local control [16:06] plane for the resources in each of our [16:09] Easy's with a principle of availability [16:12] zone independence so we talk about [16:15] control planes real quickly if you're [16:16] not familiar with the term so control [16:18] plane is the thing that you interact [16:19] with to administer these resources [16:21] whether they're zonal or regional and [16:25] then the data plane is the thing you [16:26] interact with to actually do the work [16:28] you want to get accomplished so some [16:30] examples are here for Amazon ec2 the [16:32] control plane is what handles run [16:34] instances so it takes your API request [16:36] and then does all the work necessary to [16:39] launch the VM as per your instructions [16:41] in your API and then the data plane for [16:44] you to do is basically the instance [16:46] right [16:47] the thing you SSH into or the run your [16:49] application on it's also the network [16:51] that attaches your you should do it [16:54] since to other instances in your V PC so [16:59] let's talk about azi her availability [17:00] zone independence for these things so [17:02] the data plane runs in each of these [17:04] AZ's and these are isolated from each [17:07] other they can obviously communicate [17:08] over a network but otherwise the data [17:09] planes don't really know about each [17:11] other and the same is true of the [17:13] control planes for these for these [17:16] resources now I mentioned earlier [17:20] there's a regional endpoint to access [17:22] ec2 so there must be some layer that you [17:25] connect to that sits on top of these and [17:27] there is there's a regional control [17:28] plane that acts as the entry point and [17:30] also handles things that are [17:33] zone specific so for easy to things like [17:36] security groups or not zone specific it [17:40] also aggregates API is like describe [17:42] instances so if you want to find out [17:44] about all your instances in a region it [17:46] will need to interrogate the control [17:48] planes across all the easies so in this [17:52] model again if you lose an AZ you're [17:55] gonna lose your zonal resources but [17:56] you've expected that anyway but the data [17:59] planes and the other ATS are fine the [18:01] control planes for those zones are fine [18:03] and the regional control control plane [18:06] is built to be multi AZ fault tolerant [18:08] so it will also continue to operate fine [18:12] except for the fact that it won't be [18:14] able to service requests that target the [18:15] zone that's down so if you have API [18:18] calls in to one of the healthy as ease [18:20] to launch an instance there it should be [18:23] able to route around this particular [18:25] type of AZ failure [18:30] okay so let's review the black phase [18:32] improvements that we get from [18:33] availability zones so here's a regional [18:36] service spread across three AZ's if zone [18:41] a fails regional service is fine because [18:43] it's able to fail away from from the [18:45] failed AZ meanwhile our zonal service [18:49] has an impact adjust in that zone but [18:51] not the other zones the other zones are [18:53] isolated and they won't have any impact [18:55] these are all these are all good things [18:59] the theoretical blast radius is a [19:01] different story so by a theoretical [19:02] blast rates I mean in the worst case [19:05] scenario the thing that you know the [19:07] Black Swan event like what's the worst [19:09] case that could happen here for the [19:11] regional service it is the entire [19:13] service and that's the thing that we [19:15] lose sleep at night every night thinking [19:17] about for the zonal service it's it's [19:20] still the zone so there's something nice [19:22] about this property of these [19:23] availability zones we still don't like [19:25] that there's a non infrastructure event [19:27] that could take out a service in his own [19:29] but it is still nice that it's contained [19:31] to the zone can we get the same kind of [19:33] resilience in our regional service [19:36] without having to sort of target it at a [19:38] in a zone local kind of way [19:41] so let's take a step back and look at [19:42] this abstracted architecture so there's [19:46] an entry point a regional entry point [19:49] into the service it has an aggregation [19:52] layer that might do a few things but [19:54] mostly it's a routing layer into a set [19:56] of compartmentalize resources and then [19:59] there's failure isolation between them [20:01] so in the availability zone case these [20:03] are AZ's down here and then a regional [20:06] control plane that accesses them but [20:09] more generally there's this [20:10] compartmentalization that is giving us [20:13] this nice fault isolation can we use [20:16] that in a in a different way in a [20:18] different dimension than AZ's and get [20:21] some smaller blast radius there's [20:24] another way to think about this [20:25] abstracted architecture which is how [20:28] they build chips so for centuries now [20:31] ships have been built with these [20:32] watertight compartments that are [20:35] separated by bulkheads and the reason [20:38] that this is that if there's a there's [20:39] damage to the hull and it causes [20:40] flooding the flooding is contained into [20:43] one of those compartments and the rest [20:46] of the ship is still intact and the ship [20:48] stays afloat those bulkheads also [20:51] provide structural integrity these are [20:53] both nice properties right you have this [20:56] fault Tower it's minimized impact of [20:59] failure and higher structural integrity [21:03] and so we've taken these ideas and [21:06] applied them to our regional services [21:08] and what we call cellular architecture [21:10] or cell based architecture so let me go [21:15] through our simple example again of a [21:18] application with a load balancer compute [21:20] and some storage and it's not shown here [21:24] but this is an application that is [21:25] running in multiple lazy's and has the [21:27] the failover as it described before so [21:31] in cell based architecture we take this [21:34] service stack configuration and we [21:37] create multiple instantiations of it and [21:43] these are fully iced I didn't know about [21:44] each other and each one of these stacks [21:48] is what we call a cell [21:52] and then what we'll take our workload [21:53] and basically low balance it partition [21:57] it over these these cells one way to do [21:59] that might be by customer so we'll put [22:01] this section into cell zero the section [22:03] into cell one this section into cell n [22:07] now you guys all look nice but maybe [22:10] there's someone naughty in here it's [22:11] gonna cause us a problem they're gonna [22:12] only gonna cause up the problem in that [22:14] one cell the other cells are gonna be [22:16] fine now we need some way to contain [22:19] this thing or make this thing look like [22:21] a single service still so we put a cell [22:23] router on top of it that makes those [22:24] routing decisions and that whole thing [22:28] is what we call a cell based service so [22:32] the cells are an internal structure [22:33] that's invisible to you as a customer [22:35] but provide resilience and fault [22:38] tolerance and this looks just like the [22:41] picture for azi or availability zone [22:44] independence but on a different [22:46] dimension so let's talk about what that [22:49] looks like [22:49] so here's the regional service getting [22:52] resilience from availability zones [22:54] here's that same service divided into [22:56] cells and you can see now that with an [23:00] availability zone failure both services [23:02] are resilient to that because their [23:03] fault tolerant across AZ's and the other [23:08] examples of failure the ones that we [23:10] lose sleep at at night over the failure [23:13] in their in the cell based service it's [23:15] contained to the cell so the impact is 1 [23:17] over N or n is the number of cells [23:19] rather than the whole service and I've [23:22] shown three here for the purposes of [23:25] presentation but the number of cells can [23:27] actually be much higher so one over n [23:29] could be a fairly small percentage of [23:30] the overall set of workloads that you're [23:33] supporting now that this approach can [23:38] also be applied to zonal services and we [23:40] do this in our ec2 control plane where [23:44] you divide each of the zonal services [23:45] into cells as well so you still have a [23:48] failure as you'd expect if an [23:50] availability zone goes down what's [23:52] interesting though is as I mentioned [23:53] some availability zones or multiple data [23:55] centers and so you can actually have a [23:57] smaller blast radius [23:58] in certain cases if your zonal cells are [24:01] aligned with the physical infrastructure [24:03] which is the case with our zonal [24:05] easy to control plane services that's [24:08] actually a nice improvement and then [24:11] again for the other failure cases that [24:13] we worry about there is also a smaller [24:15] ablation Shrek blast radius even for the [24:17] zonal services so let's look at the [24:21] system properties of a cell based [24:23] architecture we've I talked about some [24:26] of them the first one is workload [24:28] isolation and this is useful not just [24:30] for failures but also just noisy [24:31] neighbor problems and then of course [24:35] there's the the failure containment so [24:37] if we lose a cell the other cells are [24:38] fine there's also this nice property [24:43] that is really powerful really important [24:45] to us at AWS which is how we scale these [24:47] things [24:48] so rather than scaling up a service [24:52] which is sort of the traditional way [24:53] just add you know more and more capacity [24:55] to it in a cell based architecture you [24:59] could also add more capacity to cells [25:00] but one of the things that we include in [25:04] our goals for a sublet are characters [25:06] that cells like our data centers have a [25:09] maximum size we won't let them grow past [25:10] a certain point and so if we need to [25:13] continue to grow a system rather than [25:15] growing the cells past that point we'll [25:18] add another cell so you grow the system [25:20] by scaling it out with more and more [25:22] cells the fact that the cells have a [25:26] maximum size means you can test them at [25:28] that maximum size with a reasonable test [25:30] configuration so you can test them to [25:33] failure you can do all sorts of stress [25:34] testing and get confidence that you [25:36] understand how that piece of the system [25:38] is going to operate as you you know as [25:41] you get more and more demand on your [25:43] system these cells are also more [25:46] manageable because they're smaller so if [25:47] there's some issue you need to look [25:49] through logs or otherwise inspect the [25:52] nodes in the cell it's gonna be smaller [25:55] it's just a piece of your system rather [25:56] than the whole system so it's gonna be [25:58] easier to work through so let's now talk [26:03] about some of the core considerations in [26:05] a cell based architecture so the first [26:09] is cell size [26:14] sort of the trade-off here is you can [26:16] have a large number of smaller cells or [26:19] a smaller number of large cells in the [26:23] the case where you have smaller cells [26:25] that's nice because now your blast [26:26] radius is you know that much smaller and [26:29] those smaller things are easy to test [26:31] easier to break to understand what their [26:33] breaking points are and 30 to operate in [26:36] terms of figuring out if there's an [26:37] issue you know smaller number of notes [26:39] to go in and take a peek at on the other [26:43] end of the spectrum though larger cells [26:45] have some good properties which is first [26:47] if there's a fixed cost to each of these [26:49] which often there is you know maybe [26:51] there's a separate low bounce or each [26:53] one of them then you get cost efficiency [26:55] by having fewer of them [26:57] you also get reduced splits so that this [27:00] is an important consideration if if [27:03] we're dividing our workload by a [27:06] customer some of our customers might be [27:07] large they have a large number of [27:09] workloads and that may be too large to [27:12] fit in a smaller cell if we're using [27:16] larger cells we may be able to fit that [27:18] larger customer into a single cell right [27:20] and not have to worry about deal with [27:21] the complexity of splitting across [27:23] multiple cells and finally as a whole [27:26] this system that has fewer cells is [27:29] easier to operate because you it's [27:30] easier to think about easier to look at [27:31] dashboards and so forth there's no right [27:34] answer here except that all things being [27:36] equal we will always prefer the lower [27:39] about blast radius do these other [27:41] considerations another which I'm sure [27:46] you've been thinking about looking at [27:49] this diagram is well what about the [27:50] router the cell router is this remaining [27:53] shared component across the entire [27:56] system and so it's really important that [27:58] that thing not fail because now you're [28:00] back to the regional blast radius and so [28:04] we spent a lot of effort making sure [28:06] that that component is stress tested and [28:11] battle-hardened so that we know when it [28:13] we have high confidence that even in the [28:16] Black Swan scenarios that it's going to [28:17] stay resilient and stay up and and one [28:21] of the ways to accomplish that is to [28:22] keep it as simple as possible [28:25] so when we talk to teams about adopting [28:27] a sublation architecture we call this [28:29] component the thinnest possible layer to [28:30] kind of reinforce should be really [28:33] simple and yeah that's all I'll say [28:38] about that another consideration is [28:42] partitioning dimension so I've talked [28:44] about how we might divide cells along [28:48] lines of customers but then in the ec2 [28:51] control plane case there's an aspect of [28:54] the control plane that actually is cell [28:56] based based on physical infrastructure [28:58] in our data centers which makes sense [29:00] for that application another scenario we [29:06] may divide not by customer by VPC [29:08] especially because sometimes V pcs may [29:11] have crossed customer scenarios and so [29:14] it takes some analysis to decide what's [29:16] the right way to carve this thing up and [29:19] the recommendation I always use is cut [29:21] with the grain and if you don't know [29:23] what that means then think about it this [29:25] way that you know wood has a certain [29:26] grain and it's easy to split along one [29:28] dimension and really hard to split cross [29:31] across the grain and every system has [29:34] has a natural grain to it another [29:40] consideration is what I call cross cell [29:42] use cases these may be unavoidable the [29:47] goal is to keep them to a minimum [29:48] because that adds complexity to the [29:51] thinnest possible layer and also [29:53] increases the blast radius for those [29:56] those operations one example to scatter [29:59] gatherer queries so what this means is [30:01] there may be an API that comes in that [30:03] needs to interrogate multiple cells so [30:07] if scatter requests out and then gather [30:09] responses are sent out a single reply so [30:11] an example in ec2 is the describe [30:13] instance this case I mentioned earlier [30:15] now there is batch operations so if you [30:18] need to execute work on multiple cells [30:20] in a single operation so again an AC - [30:23] maybe the terminate instances API where [30:25] you can spend send multiple instance IDs [30:27] that can be a cross sale use case the [30:33] last in the probably hardest is [30:34] coordinated writes where you're actually [30:36] doing trying sending atomically but a [30:38] car [30:39] multiple cells those require careful [30:42] consideration and one example of that is [30:44] cell migration so cell migration is when [30:47] you relocate a workload from one cell to [30:50] another it's maybe we decided this [30:52] customer we're gonna move that customer [30:53] into cell two from cell 1 and you may [30:58] choose do this because you want to [31:00] manage the amount of load or heat that's [31:02] on each cell or maybe just want to load [31:05] bounce the sizes of them or maybe you've [31:07] added a salad you and you need to and [31:09] your approach for adding cells involves [31:12] you know moving existing workloads over [31:14] into the new cell and the process that [31:17] you used to do the migration is not [31:20] unlike a VM migration so if you're [31:22] familiar with how VM migration works [31:24] they say there's a invisible clone that [31:27] gets created in the target location and [31:29] it gets brought up to date and [31:32] synchronized with the source of course [31:35] the source is still changing so this [31:37] could take a while for it to get close [31:40] to begin sync and at the last possible [31:42] moment [31:42] both are frozen for the final completion [31:46] of the syncing and then an atomic flip [31:48] over to the to the target location that [31:52] works for VMs and that's the same [31:53] approach that we use for migrating [31:55] workloads across cells it requires a [31:58] careful coordination best-managed at the [32:02] router level we have a few approaches [32:05] that we've been using to accomplish this [32:09] so again with the cell Bay structure [32:13] we're able to reduce the blast radius [32:14] from 100 percent down to one over n [32:17] where n is the number of cells [32:18] I should reinforce that the events that [32:23] cause these types of additives are [32:25] exceedingly rare we could spend a long [32:32] time not even having to worry about the [32:34] kind of failure happening because of all [32:35] the other things that abus does but [32:38] we're so focused on resilience that [32:40] we're investing additional engineering [32:41] work to get to that picture on the right [32:44] which is a smaller blast radius even [32:46] when those Black Swan events occur [32:50] so cells are great but there's another [32:56] technique that we've been using that is [32:57] even more impressive and more exciting I [33:01] think which is called shuffle charting [33:04] and shuffle charting is a technique that [33:06] is like cell based architectures and [33:09] it's particularly useful in stateless or [33:12] soft state services so we'll be walk [33:16] through what shuffle charting looks like [33:17] so here's another simple service we've [33:20] got eight nodes and these ain't nodes or [33:23] handling requests that are sent by a [33:26] load balancer and then we have eight [33:29] different customers that are sending [33:30] requests so we're in Vegas I use some [33:33] some gambling relevant to high cons here [33:36] there were ten are different customers [33:39] and let's imagine one of them Dimond [33:42] here is is introducing a bad workload [33:45] for whatever reason maybe it's expensive [33:46] request maybe it's one of these requests [33:50] that triggers a bug in the system so [33:53] Dimon sends a request in and that [33:56] request causes one of our servers to [33:57] crash okay that's all right we've got [34:00] seven others maybe Dimon will go away or [34:03] change what it's doing probably not [34:06] it'll probably keep retrying and [34:09] eventually take out the whole system so [34:11] here our blast radius is basically all [34:14] the customers this is like the worst [34:16] case scenario we really want to avoid [34:18] this this is where we go to sell based [34:21] architecture so we divide our our [34:23] customers assign a subset of customers [34:25] to each sell [34:28] now when diamond comes along and causes [34:31] problems that problems contained adjust [34:34] to the sell this is a 4x improvement [34:38] right we've gone from 100% down to 25% [34:40] the blast radius is the number of [34:41] customers divided by the number of cells [34:44] and again we could approve that further [34:46] by adding more cells or in a system [34:50] where we don't need to really worry [34:52] about which customers land on which [34:54] nodes we could shuffle shard them which [34:58] is a little bit different and it's [34:59] nuanced but you'll see shortly how [35:01] powerful this is [35:03] so we'll take each customer and assign [35:05] them to two nodes effectively at random [35:08] not really a random own will use hash [35:10] functions to be predictable that where [35:15] are these customers land on these nodes [35:16] but basically we've assigned them at [35:18] random so diamond gets assigned to the [35:20] first and fourth nodes will put spades [35:23] on those two our role of two on the dice [35:27] goes to those two nodes and so on [35:29] Ceaser's basically shuffled randomly [35:33] across our set of capacity so now again [35:37] diamond comes along takes out the two [35:39] nodes that are assigned to it but here's [35:41] where it gets interesting look at who's [35:44] sharing those nodes with diamond one of [35:47] them is hearts hearts however has a [35:52] second know that it's assigned to that's [35:55] not impacted to buy the average so as [35:56] long as that customer retries it's [35:59] fault-tolerant even though one of its [36:02] nodes is down one of its noses up it's [36:04] able to continue operation the same is [36:06] true on the other node where clubs has [36:09] that same property so in this case our [36:14] blast radius is actually the number of [36:16] customers divided by the number of [36:17] combinations of of two pairs out of [36:21] eight which turns out there are 28 of [36:25] them which is 3.6 percent so if we had a [36:29] much larger number of customers we'd [36:30] expect these are well distributed [36:33] randomly you would have three poor 63.6% [36:35] of customers impacted by the failure at [36:38] that I showed meanwhile less than half [36:42] of the customers would be in that [36:43] scenario where they're sharing at least [36:45] one of the nodes so they may see a [36:47] little bit of impact a little bit of [36:48] hiccup but they're fine so we went from [36:51] 25 percent down to 3.6 percent going [36:55] from the cell-based down to shuffle [36:56] shark now this is a small system oh I [36:59] should show you the math so the math [37:00] here is probably as you remember from [37:01] high school the binomial coefficient as [37:06] you look at this math to realize as n [37:08] grows our number of combinations grows [37:12] really quickly so let's say we go from 8 [37:14] nodes up to 100 [37:15] larged like a huge number of notes it's [37:18] a reasonable number to run in a [37:19] large-scale system so say we have a [37:22] hundred nodes and then we give each [37:23] customer five combinations we're sorry [37:26] if five nodes to represent their [37:28] combination the math tells us that's [37:31] going to be 75 million different [37:33] combinations I think of it basically as [37:35] you know a deck couple hundred cards [37:36] there's fire you know 75 million [37:39] different combinations of cards you can [37:40] get from by picking randomly from that [37:42] deck which is amazing because now you [37:45] can see all right 77 percent of [37:47] customers are not going to see any [37:48] impact when diamond comes along and [37:50] takes out it's five nodes but more [37:52] interestingly ninety-nine point eight [37:55] percent so those first three rows are [37:58] still gonna have a majority of their [37:59] nodes available so they're gonna have a [38:01] better chance than not to completely you [38:05] know route around that problem without [38:06] even having to try and meanwhile that [38:10] very very very low percentage of [38:12] customers is basically the percentage of [38:16] customers that are going to be sharing [38:17] completely those same five notes what's [38:21] magical about this and it's all in the [38:23] math is we've created a multi-tenant [38:27] system and then used the shuffle [38:30] charting to create a single basically a [38:32] single tenant experience which is [38:34] obviously what a POS aspires to do now [38:38] this needs a fault our client as I [38:40] mentioned so one that when it gets a [38:42] failure will retry but that's that's not [38:44] hard that's that's pretty common what's [38:47] interesting also is this not only works [38:48] for servers it can work for queues they [38:51] can work for other resources it's also [38:54] critically dependent on fixed [38:56] assignments so you're you're stuck with [39:00] the hand that we deal you if there's any [39:04] sort of failover like oh well your five [39:05] nodes are down I'll give you these five [39:06] then you get back into that old world [39:10] where now a problem can infect and [39:13] cascade across an entire system so it [39:15] really depends on those fixed [39:16] assignments and if I need some sort of [39:18] routing mechanism so either a shuffle [39:20] sharding aware router or dns can be [39:25] another so in some of our servers both [39:27] will hand a customer's [39:28] if ik DNS name and that will resolve to [39:32] the customer specific shuffle sharted [39:34] set for that customer which basically [39:37] gives them the routing for free cool so [39:42] let's talk about the operational [39:43] practices that we now layer on top of [39:46] these architectural techniques to [39:48] achieve the lowest possible blast radius [39:53] the first is not even the practice but [39:56] really ever a mindset and maybe even a [39:59] religion at a table us which is probably [40:02] best captured by Vernors blog from [40:04] earlier this year about [40:06] compartmentalization [40:07] now that's this back shall just read it [40:09] so I won't do as accident I have nothing [40:12] to see in burger talk but I'd wager that [40:15] every new a TBS engineer knows within [40:16] their first week it's not their first [40:18] day that we never want to touch more [40:20] than one zone at a time this is so [40:24] important because if we have [40:26] availability zone fault isolation or we [40:29] have region isolation as a core tenet of [40:31] our blast radius reduction that's gonna [40:34] go out the window if we have you know [40:36] some correlated failure introduced by [40:38] some manual action or automated action [40:40] on multiple of these at the same time [40:43] the most common and most I guess obvious [40:47] example of this is with software [40:48] deployments so our software deployments [40:51] are done in a staggered way across zones [40:55] across regions over time you know [41:00] quickly enough that we can get features [41:01] app to customers that you know because [41:03] we like to launch features but slowly [41:06] enough that we have confidence that as [41:08] we're pushing this change broader and [41:10] wider that it's not gonna cause an issue [41:13] so we'll start slow observe test and [41:17] then maybe speed it up as as it goes out [41:20] broader and broader and that will that's [41:25] the case with cells it's the case with [41:26] availability zones casement regions and [41:29] then within which each of those [41:30] deployment units will do a fractional [41:33] deployment to their to so a one box test [41:35] has a very first step for a service for [41:39] our ec2 Harbor deployments [41:42] we'll start with maybe five or ten [41:45] machines at first verify that things are [41:49] working and then gradually speed it up [41:52] as it expands across the infrastructure [41:55] in tandem with that we have a bunch of [41:57] automated tests that run that's part of [41:59] the deployment as well as Canaries or [42:02] these test applications that are [42:03] mimicking you know a real-world customer [42:06] in booking api's and will monitor the [42:09] results of those if there's any problem [42:10] the role the deployment get [42:12] automatically rolled back we'll look at [42:14] what happened and either decide it was [42:17] not an issue that we need to worry about [42:18] or fix the problem before we start the [42:20] deployment again so this is really [42:23] important this is this is what we need [42:25] to do to make sure that we've not [42:27] compromised the boundaries that we've [42:29] put between cells and AZ's and regions [42:35] all that automation sorry all that [42:37] deployment machinery is automated [42:41] including the rules about the tests that [42:45] have to succeed including the timing and [42:47] the windows when we progress to the next [42:49] stage but automation is in key in [42:52] general right it's not just employments [42:53] there are other things that we do with [42:54] our infrastructure to manage it whether [42:56] it's configuring network devices or you [43:00] know propagating new credentials to the [43:02] stacks that could be done by hand [43:05] but humans are prone to error it's much [43:08] better to automate it but it could be [43:10] because you can review whatever it is [43:13] that you're building in your automation [43:15] you can test it it's predictable and how [43:18] it's gonna operate and you can repeat it [43:19] over and over again and know that it's [43:21] gonna work the same way every time and [43:25] then finally as you may have heard AWS [43:28] and Amazon in general is has a very [43:31] strong philosophy around end-to-end [43:32] ownership so our service teams are [43:36] composed of engineers who are builder [43:38] operators so as engineers we build this [43:40] up we design the software we test it we [43:43] also are the ones that deploy it and [43:46] we're the ones that operate it and [43:47] respond to issues in production what [43:49] this does is it gives us this wonderful [43:51] opportunity to have a feedback loop in [43:53] terms of design choices what [43:56] impact is operationally and [43:58] understanding what changes we need to [43:59] make to avoid future problems when there [44:03] is a failure that we need to worry about [44:05] and so I mentioned earlier the [44:07] correction of errors template that we [44:08] use that's usually filled out by [44:11] engineers or in partnership with [44:12] engineers where they think about the [44:15] blast radius and so they're in a great [44:17] position to then go and implement the [44:19] changes to make sure the next time if [44:21] that event occurs the blast radius is [44:23] cut in half [44:23] or more so to wrap up we've got a [44:31] variety of containment [44:33] compartmentalization mechanisms that we [44:35] use to reduce blast radius [44:37] it starts with regions and the strong [44:38] isolation between regions then it goes [44:41] into availability zones and then this [44:44] alternate dimension of availability [44:46] zones compartmentalization which [44:47] ourselves and then the magic of shuffle [44:51] charting to get sort of virtual cells [44:54] and taking advantage of the [44:56] combinatorics of the shuffle charting [44:59] and then with those we protect that [45:02] compartmentalization with operation [45:04] operation operational practices so step [45:08] by step phase deployments that are [45:10] automated and then service teams that [45:13] are builder operators so they are close [45:16] to the the frontlines understand how [45:18] their decisions are impacting the [45:20] availability of their systems and all [45:24] with the goal of reducing blast radius [45:28] so that's my talk I hope you learned a [45:32] few things and I'm happy to take [45:34] questions now if anyone has them and I [45:36] think there are yeah there are live [45:38] microphones down the center aisles here [46:01] yeah so the question was is there an [46:02] example of Murphy's Law where we thought [46:06] we had everything nailed down everything [46:08] sorted out that we answered although the [46:12] possible failure modes its it goes back [46:16] to my slide of the the bit flips on the [46:21] networking side so this is a example for [46:23] many many years ago [46:24] so in s/3 s/3 cares deeply about data [46:29] care deeply about the integrity of data [46:31] and there are many layers in s3 of [46:35] checksumming [46:37] so if there is an arrant but bit flip [46:41] introduced we make sure it doesn't you [46:43] know we detected it's not an issue in [46:47] 2008 we had an event in s3 where there [46:52] was one network card on one server that [46:57] every now and then was flipping one bit [47:01] and there was a layer of the system that [47:04] handles basically the group [47:06] communication across the system uses [47:09] gossip protocols so can detect whether I [47:13] can understand what the state is of all [47:14] the servers in the system whether [47:15] they're healthy or not [47:16] well the gossip protocol noticed there's [47:20] a funny server name in this packet [47:23] because one of the bits was flipped in [47:25] the server name I never heard of that [47:27] hos before and it triggered a much more [47:29] expensive sort of reconciliation [47:31] protocol long story short you could read [47:35] the long story there's a post mortem [47:37] that Stoll published on the somewhere in [47:40] the dashboard status dashboard pages [47:42] talks about the outage but it took down [47:43] s3 completely so one server one NIC one [47:49] bit took down our regional service and [47:52] that was the only layer we had checks [47:55] coming all the way up and down the stack [47:56] that was the only layer that didn't have [47:58] the check something someone we'd [47:59] forgotten to add it there [48:09] yeah so the follow question here is [48:12] wasn't there a more recent event where [48:14] there was a human error involved in an [48:16] s3 patent yes there was and this comes [48:19] back to my operation back to slides I [48:21] was talking about which is all your best [48:25] plans can can fall apart if whatever [48:30] you're doing doesn't respect those fault [48:32] isolation boundaries in this case the [48:36] engineer very certainly knew our [48:38] philosophy and it was more of a just a [48:42] human error mistake on the command line [48:45] which goes back to the importance of [48:46] automation and importance of testing of [48:48] those those things [48:49] Peter yes with the everything that you [48:54] talked about in the different layers [48:55] from regions availability zones to cells [48:58] to the shuffle shorty to your customers [49:01] are are they given to them or are they [49:05] for purchase as services good question [49:09] so regions are given to you these are [49:13] given to you cells and shuffle charting [49:16] are sort of different things in that [49:18] they are they're like the watertight [49:22] compartments inside the ship so you [49:24] could ask the the purser on the boat to [49:27] give you a tour of the lower decks and [49:28] maybe they'd show you the compartments [49:29] but otherwise you know you don't really [49:32] care about that and you get them for [49:33] free it's just part of the safety of the [49:35] ship so they're visible and they're [49:39] they're free I wanted to have a cell for [49:51] my own service that I'm making and I [49:53] actually care about aligning that cell [49:55] with a particular data center so you can [50:05] get that kind of affinity with your [50:07] zonal resources right so you can you can [50:09] get affinity with your ec2 instances and [50:11] your Cloud HSM instance and your EBS [50:15] volumes because we give you full control [50:16] over the place well and availability [50:20] zone you can think of as a lot [50:22] whole data center there they're one in [50:23] the same so a data center will always be [50:28] in one easy right an AZ will always be [50:33] one or more data centers you're saying [50:36] for the multiple data center or a Z's [50:39] exactly got it yeah no that that is if [50:42] that is not visible to you as a customer [50:43] you don't have control over that Thanks [50:46] yep so as you move from regions to cells [50:50] how does the version management of stuff [50:53] that you're deploying is there any [50:55] operational insight into that of being [50:58] deployed there I mean how do we how do [51:01] we keep does it have any impact on you [51:05] know managing operations deploying in a [51:07] region versus deploying at a shard level [51:10] I said that any complexities there yeah [51:12] so I think what you're saying is there's [51:14] this window of time when we're going [51:17] through the progression of the [51:18] deployment and the versions may not be [51:20] in sync across multiple locations is [51:23] that the core of your question yes yeah [51:25] that's definitely consideration and a [51:28] deployment may take depending on the [51:29] system it may take several days to [51:33] several weeks in some operating [51:35] structure were particularly careful [51:39] about the pace of deployment in fact in [51:42] some of the networking components for [51:44] VPC it's a particularly delicate affair [51:47] to roll out changes now it might take a [51:49] while so yeah that is a complication [51:51] that we have to be mindful of of well [51:54] which version is running here yeah some [52:00] questions regarding the global services [52:03] we talked a lot about regions zones how [52:05] hard the global services regarding [52:08] control playing day to play in [52:10] resiliency in respect to the regions or [52:12] edge notes and organize so is your [52:15] question about regional services that [52:18] are have global features or truly global [52:20] services for example I am I am okay yeah [52:23] that's a great question so I am as a [52:26] special case because [52:28] your account information and your [52:32] credentials and so forth are available [52:34] globally so it is in a sense of global [52:36] service however each region has a [52:39] separate control plane sorry a separate [52:43] data plate that can operate completely [52:46] disconnected from the source of truth [52:49] for I am the control plane however is [52:54] global and so there are particular [52:56] considerations that we take with a [52:58] service like that to make sure that it [53:00] continues to be available even in the [53:02] face of some of the failures that I [53:03] talked about but I should I had it on [53:07] the site and I forgot to mention that [53:08] there is another philosophy that we have [53:11] which is the separation between control [53:13] plane and data plane and ensuring that [53:16] the data plane can continue to function [53:17] even if there is a control plane issue [53:20] and that's exactly the scenario and I am [53:23] that's critically important because a [53:25] regent can be disconnected from the [53:26] source of truth or it can there could be [53:29] other issues but the control plane we [53:30] wanna make sure that you can still you [53:33] know validate your credentials and so [53:34] forth you know in each region and not [53:36] have global impact can you go to the mic [53:48] so when you're talking about the cell [53:51] structure and the shuffle shouting I [53:56] mean when you're talking about the cell [53:58] stuff you were talking about in the [54:00] control layer setting out the routing to [54:02] cut with the grain of the service ways [54:05] to break it up when you move to shuffle [54:07] shouting is that done at the same level [54:08] because obviously you lose that sort of [54:11] control based on the characteristics of [54:14] the service yeah it's a good question [54:16] and for for cell based services usually [54:24] that prevents you from shuffle charting [54:26] is that you have you have to have some [54:29] control over state so you can't a sort [54:32] of willy-nilly a scatter of state across [54:34] the entire fleet some servers don't have [54:37] state or the earthís off state that they [54:39] can they can cache locally and then [54:41] operate fine the shop charting works [54:45] well for for the latter scenario whether [54:47] it doesn't need to be any particular [54:48] node affinity the there's another sort [54:55] of advanced topic here which is kind of [54:57] interesting which is you can actually [54:58] layer both of these things together you [55:01] could take a cell based architecture and [55:02] then shuffle shard requests across them [55:05] in a stateless system to get a different [55:08] kind of resilience property that's [55:09] something that we're working on now with [55:11] with with dynamodb so I don't know if I [55:15] answered your question okay [55:18] so I guess maybe never another way to [55:20] answer to that as shuffle charting and [55:22] is not an improvement on cell based in [55:24] all cases it's really a system specific [55:28] on when you might be able to apply it [55:30] cool okay look that there's no more [55:36] questions thanks again for coming [55:38] hope you learn something and enjoy your [55:40] week at reinvent [55:40] [Applause]