AWS Engineer Brooker Warns Caches Trigger Tech's Worst Outages

Ryan Peterman

Uncached traffic spikes lock backends into self-sustaining metastable failure loops, driving what Marc Brooker estimates to be a majority of catastrophic industry outages.

AI-assisted summary of AWS Distinguished Eng: Why Caching Is Bad System Design | Marc Brooker — original source. Check the original for full context.

Source published: 2026-10-16

Key ideas

Traditional distributed caching introduces a binary modality between warm and cold states that frequently triggers self-sustaining metastable failures. (0:00)
AWS Distinguished Engineer Marc Brooker estimates that metastable failures serve as an underlying cause in a majority of the industry's largest postmortems. (4:18)
Engineers should favor complete caches, proactive pre-warming, local in-memory materialized views, and scalable datastores like Amazon DynamoDB and Aurora DSQL over ad-hoc distributed caching layers. (2:25)
Recovering from metastable failure states often requires drastic, high-risk interventions such as taking entire service fleets offline for a cold reboot. (4:18)

Detailed notes

## Background Published on YouTube by Ryan Peterman on October 16, 2026, this clip features an interview with AWS Distinguished Engineer Marc Brooker. According to the description, the discussion draws from technical insights across more than 3,000 cloud system postmortems to examine why default caching patterns can result in hazardous system design.

## TL;DR While caching is widely treated as a universal reflex in system design, Marc Brooker explains that distributed caches introduce a severe operational modality between warm and cold states. When uncached traffic hits an unscaled backend, systems frequently descend into self-sustaining metastable failures where resource contention and network saturation prevent the cache from ever repopulating. To mitigate this risk, Brooker advocates for complete caches, proactive replica warming, local materialized views, or relying directly on horizontally scalable datastores like DynamoDB and Aurora DSQL. Although individual developers may rarely experience them in daily work, metastable failures remain the underlying cause behind a majority of the tech industry's most catastrophic and prolonged outages.

## Atmosphere & style The conversation carries a measured, deeply technical, and contrarian tone that dismantles standard software engineering dogma with veteran authority. Peterman plays the role of an engaged interlocutor challenging textbook interview reflexes, giving Brooker room to articulate nuanced distributed systems theory in accessible, vivid terms. Listening to the exchange feels like sitting in on a high-level postmortem post-analysis, balancing intellectual curiosity with the grim operational reality of having to reboot entire server fleets during catastrophic outages.

## Bulletin - [00:00](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=0s) Traditional distributed caching introduces a binary modality between warm and cold states that frequently triggers self-sustaining metastable failures. - [04:18](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=258s) AWS Distinguished Engineer Marc Brooker estimates that metastable failures serve as an underlying cause in a majority of the industry's largest postmortems. - [02:25](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=145s) Engineers should favor complete caches, proactive pre-warming, local in-memory materialized views, and scalable datastores like Amazon DynamoDB and Aurora DSQL over ad-hoc distributed caching layers. - [04:18](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=258s) Recovering from metastable failure states often requires drastic, high-risk interventions such as taking entire service fleets offline for a cold reboot.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=0s) In standard system design prep and discussions, engineers almost universally default to adding a cache to solve scaling problems. - [00:00](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=0s) Caching delivers theoretical performance gains through spatial and temporal locality, but introduces severe operational risks in distributed architectures. - [00:00](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=0s) Distributed caches create a hazardous two-state modality: a healthy warm state, and a degraded cold or corrupted state where underlying backends cannot absorb full traffic. - [00:00](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=0s) A cold or corrupted cache can trigger a metastable failure, where backend resource contention and network saturation permanently prevent the cache from refilling. - [02:25](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=145s) Amazon Aurora DSQL avoids cache miss storms by implementing a complete cache tier that contains every single row in the database. - [02:25](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=145s) Amazon Aurora sidesteps failover traffic shock by continuously streaming cache updates from the primary leader to standby replicas ahead of time. - [02:25](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=145s) For datasets that change infrequently, copying the entire dataset into a local host's memory as a materialized view is preferable to maintaining a distributed cache. - [02:25](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=145s) Engineering teams should push horizontally scalable databases like Amazon DynamoDB to meet latency requirements natively rather than papering over performance gaps with caching. - [04:18](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=258s) An individual engineering team may work for years without seeing a metastable failure, creating a false sense of safety around fragile cache architectures. - [04:18](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=258s) Despite their day-to-day rarity, metastable failures represent an underlying mechanism in a majority of the tech industry's most catastrophic outages. - [04:18](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=258s) Outages driven by metastability exhibit significantly longer mean time to recover (MTTR) compared to standard transient service failures. - [04:18](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=258s) Breaking out of a metastable failure state often leaves operators with no viable option other than risky full-fleet cold reboots.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=0s) The Modality Trap and Metastable Failures in Distributed Caching

Ryan Peterman opens by noting that Marc Brooker has worked at AWS for nearly two decades building distributed systems. Peterman observes that in standard system design discussions and interview prep, the universal reflex is almost invariably to add a cache: “I almost hear almost 100% of the time people will say, 'Just throw a cache on it.' Or you have a system design, you say, 'How do you make it better? Let's put a cache here. Let's put a cache there.'” Peterman references a tweet by Brooker arguing that caching can actually be bad system design despite being promoted as industry best practice, asking him to unpack that stance.

Brooker acknowledges the fundamental appeal of caching: it leverages core computer science principles like temporal and spatial locality to make systems dramatically faster and scale better. However, Brooker explains that in distributed architectures, caches introduce a dangerous binary modality. In state one, the cache is populated with the correct data in time and space, leaving the system performing smoothly. In state two, the cache is either cold, empty, or poisoned with the wrong data. Because the underlying backend is rarely provisioned or scaled to absorb the full brunt of uncached traffic, the system slows to a crawl or falls over completely, leaving customers stranded.

Worse still, this failure state is frequently self-sustaining. Brooker frames this as a classic metastable failure: the system abruptly shifts from a healthy operating state into a degraded state, and once in that degraded state, it remains permanently stuck there because it lacks the internal capacity to recover on its own. The incoming tidal wave of traffic overwhelms the database with contention or saturates network links, which in turn prevents the cache from repopulating with the data needed to escape the crash. As Brooker describes this trap: “And this is this kind of idea of meta-stable failures, where the system has has switched from state one to state two, and in state two it's still stable, right? Like it's still it's down, but it's not going to come back up under its own energy because, for example, all of this traffic is causing a huge amount of contention in my database or is saturating the network and so I can't even refill the cache.”

```mermaid flowchart TD accTitle: Metastable Failure Cycle accDescr: Diagram showing how a warm cache degrades into an unrecoverable metastable failure state.

subgraph HealthyState["Stable Healthy Mode"] Warm["Warm Cache"] -->|"Absorbs bulk traffic"| Fast["System Fast & Scaled"] end

HealthyState -.->|"Cache eviction or cold start"| MetastableState ```

- **Temporal and spatial locality:** The theoretical justification for caching, optimizing retrieval by assuming data accessed recently or nearby will be needed again. - **System modality:** The operational gap between a warm state (where traffic hits memory) and an empty or invalidated state (where backend services absorb full client volume). - **Metastable failure dynamic:** A degraded equilibrium where backend load and resource contention physically prevent the cache from warming back up, making recovery impossible without external intervention.

---

[02:25](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=145s) Architectural Alternatives: Complete Caches, Proactive Warming, and Scalable Datastores

To circumvent the hazards of cache modality, Brooker points to specific architectural strategies that eliminate the cold-cache edge case entirely. Within AWS, he highlights Amazon Aurora DSQL, where the storage tier acts conceptually like a cache, but with one critical distinction: it is a complete cache holding every single row in the database. Because the cache is exhaustive, the system never encounters an empty or incomplete state from which it must desperately recover under load. Brooker also points to classical relational database architectures such as Amazon Aurora, where the primary leader actively and continuously feeds cache updates to prospective failover targets. When an unexpected failover occurs, the replica’s cache is already pre-warmed, completely sidestepping sudden traffic shock.

Beyond database engines, Brooker notes that while he would not enforce an absolute rule for every single scenario, he actively encourages teams around him to avoid conventional caching layers whenever possible. One preferred alternative is using complete materialized views for read paths, particularly when the underlying dataset changes infrequently. If an internal dataset updates only once a week, engineers should simply copy the entire dataset down to the local host and hold it directly in memory rather than introducing an external distributed cache.

Another primary recommendation is relying directly on horizontally scalable database technologies like Amazon Aurora DSQL or Amazon DynamoDB. Brooker stresses that engineering teams should push database providers to meet strict scale and latency requirements natively rather than papering over performance gaps with an ad-hoc cache. Summarizing the trade-off, Brooker emphasizes: “So, caching isn't a bad pattern, but it is a pattern with some significant downsides that are, you know, really, uh, best avoided.”

- **Complete caches (Aurora DSQL):** Eliminates cache miss recovery storms by ensuring the caching tier holds 100% of the database rows. - **Proactive replica pre-warming (Aurora):** Primary instances stream cache-worthy data to failover replicas ahead of time to ensure zero cache warmup delay upon failover. - **Local materialized views:** For slow-moving reference data (such as weekly updates), pulling entire copies into application memory removes distributed network dependencies entirely. - **Holding datastores accountable:** Relying on elastic storage backends (e.g., DynamoDB) to handle real scale directly instead of masking performance ceilings with front-facing caches.

---

[04:18](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=258s) The Frequency Paradox and Impact of Metastable Failures Across the Industry

Peterman asks Brooker how often metastable failures actually happen in real-world systems. Brooker explains that there is a curious paradox around how frequently they occur. On an ordinary team, an engineer might run production systems for several years without ever encountering a metastable failure firsthand. However, when looking across the broader tech industry and examining its most catastrophic outages, metastable dynamics appear as a root driver with striking regularity.

Brooker highlights the gravity of these incidents: “But if you look across the biggest, most impactful, uh, you know, system postmortems across the industry, I would say that these kinds of metastable failures have been an underlying cause in probably a majority of them.” Because of this disproportionate impact, Brooker argues it is critical for the engineering community to study and understand metastability deeply.

When a metastable failure strikes, it does not behave like a standard transient bug. These failures represent massive, widespread outages with significantly longer times to recover (MTTR) and complex remediation paths. Systems stuck in a metastable trap cannot simply bleed off traffic or wait for background tasks to catch up; resolving the deadlock frequently forces operators into high-risk, painful recovery procedures like taking the entire fleet offline and rebooting it all at once to break the contention cycle. For this reason, Brooker describes metastable failures as both exceptionally rare in routine daily operations, yet simultaneously common among the most devastating incidents the industry faces.

- **Postmortem prevalence:** Metastable failures serve as the underlying failure mechanism in what Brooker estimates to be a majority of high-severity industry postmortems. - **Prolonged recovery times:** Outages triggered by cache collapses and metastable states take far longer to mitigate than ordinary transient service degradations. - **Severe mitigation steps:** Operators often have no alternative other than full cold reboots (“turning it off and turning it back on again”), which carries immense operational risk during major incidents.

## Notable quotes > [02:25](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=145s) Caching isn't a bad pattern, but it is a pattern with some significant downsides that are, you know, really, uh, best avoided. — Marc Brooker

> [04:18](https://www.youtube.com/watch?v=LDUPPlb8JJI&t=258s) If you look across the biggest, most impactful, uh, you know, system postmortems across the industry, I would say that these kinds of metastable failures have been an underlying cause in probably a majority of them. — Marc Brooker

## Mentioned AWS, Marc Brooker, Ryan Peterman, Amazon Aurora, Amazon Aurora DSQL, Amazon DynamoDB

## Video description In this episode, I talked to Marc Brooker, a distinguished engineer at AWS who started there as a new grad and rose through the ranks. We discussed technical learnings from 3,000+ cloud system postmortems, how software engineering is changing with AI, how to find impactful problems and much more.

Watch the full conversation: https://www.youtube.com/watch?v=u3GjIXP9N0s

[Source](https://www.youtube.com/watch?v=LDUPPlb8JJI)

中文简报

AWS工程师Brooker警告：缓存是引发科技界最严重宕机事故的推手未命中缓存的流量激增会使后端陷入持续自我维系的亚稳态故障循环，Marc Brooker估计科技行业大多数灾难性宕机皆源于此。尽管缓存在系统设计中普遍被视作一种本能反应，但Marc Brooker指出，分布式缓存在热态与冷态之间引入了严重的运行模态差异。当未经缓存的流量冲击未经扩容的后端时，系统往往会陷入自我维系的亚稳态故障，此时资源争用和网络饱和会使缓存完全无法重新填充。为了降低此类风险，Brooker主张采用完整缓存、主动预热副本、本地物化视图，或直接依赖DynamoDB和Aurora DSQL等可水平扩展的数据存储。尽管普通开发者在日常工作中可能鲜少遇到，但亚稳态故障仍然是科技行业大多数最严重、最持久宕机事故背后的根本原因。传统的分布式缓存会在热态与冷态之间引入二元模态，从而频繁引发自我维系的亚稳态故障。 (0:00) AWS杰出工程师Marc Brooker估计，亚稳态故障是该行业大多数重大事故复盘报告背后的根本原因。 (4:18) 工程师应优先考虑完整缓存、主动预热、本地内存物化视图，以及像Amazon DynamoDB和Aurora DSQL这样可扩展的数据存储，而非临时搭建的分布式缓存层。 (2:25) 从亚稳态故障中恢复通常需要采取极端且高风险的干预措施，例如让整个服务集群下线进行冷重启。 (4:18)

Reading summary

AWS Distinguished Engineer Marc Brooker argues that reflexively adding caches to distributed systems introduces dangerous operational modality and risks self-sustaining metastable failures. Rather than relying on distributed caches to mask backend limits, he advises engineers to adopt complete caches, proactive pre-warming, local materialized views, or natively scalable datastores. The Modality Trap and Metastable Failures (0:00) While caching relies on spatial and temporal locality to improve performance, Brooker warns that it forces distributed architectures into a binary modality. In the healthy state, a warm cache absorbs client traffic and allows backend systems to operate smoothly. However, if the cache becomes cold, empty, or poisoned, the underlying datastore is suddenly forced to absorb the full, uncached traffic volume—a load it is rarely provisioned to handle. This shift frequently triggers what Brooker defines as a metastable failure. Once the system enters this degraded state, it cannot recover on its own energy. The massive influx of uncached requests creates severe database contention and saturates network links, which in turn physically blocks the cache from refilling. As a result, the system remains stable in its failed state until external operators intervene. Architectural Alternatives to Distributed Caches (2:25) To avoid cold-cache traps, Brooker points to designs that remove incomplete cache states entirely. In Amazon Aurora DSQL, the storage tier operates as a complete cache containing every row in the database, ensuring the system never faces a partial or cold state under load. Similarly, standard Amazon Aurora architectures have the primary instance continuously push updates to prospective failover replicas, ensuring standby caches are fully pre-warmed before taking over production traffic. At the application level, Brooker encourages avoiding external distributed caches whenever feasible. For data that updates infrequently, such as weekly reference data, services should pull complete materialized views directly into local host memory. For higher-throughput systems, Brooker advises teams to rely on horizontally scalable datastores like Amazon DynamoDB or Aurora DSQL to natively meet latency and scale requirements, rather than papering over backend capacity ceilings with ad-hoc caches. The Frequency Paradox in Major Outages (4:18) Brooker observes a paradox in how metastable failures appear in production: an individual engineering team can manage services for years without ever witnessing one, yet he estimates that metastable dynamics serve as the root cause in a majority of the tech industry's most severe postmortems. Unlike transient errors that clear up as traffic subsides, metastable incidents involve extended recovery times and complex mitigation paths. Because the vicious cycle of backend contention prevents self-healing, operators often have to resort to high-risk recovery procedures, such as shutting down and cold-rebooting entire server fleets at once to break the deadlock. Brooker clarifies that caching is not an inherently flawed concept, but its systemic failure modes introduce operational risks that are best avoided. Designing architectures around natively scalable storage and complete, pre-warmed data sets provides far greater resilience against unrecoverable system collapses.

精炼阅读摘要

AWS 杰出工程师 Marc Brooker 指出，业界普遍视作最佳实践的分布式缓存实际上引入了脆弱的双模态机制，极易引发系统无法自愈的亚稳态故障；系统设计应尽量避免盲目添加缓存层，优先采用全量缓存、主动预热或具备原生水平扩展能力的数据存储。缓存双模态与亚稳态故障死锁 (0:00) Brooker 承认缓存利用时间与空间局部性原理，能显著提升响应速度与系统规模，但指出分布式架构中的缓存会产生危险的二元模态（Modality）。在状态一下，缓存命中率良好且数据正确，系统平稳运行；但在状态二下，若缓存出现冷启动、清空或数据污染，未命中流量将全数涌向后端。由于底层数据库与存储通常不会按承受 100% 原始流量的标准预留冗余容量，突发访问会迅速将后端资源耗尽，导致系统严重降级或彻底瘫痪。更为危险的是，该故障模式具备自我维持的“亚稳态”（Metastable Failure）特征。Brooker 解释称，系统一旦从健康状态跌入降级状态，就会停留在该故障模态下无法依靠自身恢复。外部源源不断的请求会在数据库层引发严重冲突并占满网络带宽，这种持续的高压争用反过来彻底阻断了系统重新读取数据、填充缓存的可能，使系统陷入不可自拔的死锁平衡。消除冷启动边缘条件的替代架构 (2:25) 为避开缓存模态带来的风险，Brooker 提出了能够从根本上消除冷缓存边缘场景的架构设计。在 AWS 内部，Amazon Aurora DSQL 采用概念类似于缓存的存储层设计，但该层包含数据库中的每一行数据，即“全量缓存”（Complete Cache），从而排除了因数据缺失而必须在重载下恢复的脆弱状态；而在 Amazon Aurora 等关系型数据库中，主节点会持续将缓存内容流式推送到故障转移备机，确保副本在接管流量前即完成预热，避免倒换瞬时的流量冲击。在应用层设计上，Brooker 鼓励团队在非必要场景下规避常规分布式缓存。对于低频变动数据（如每周仅更新一次的数据集），工程师应直接将全量数据复制并驻留于应用服务器的本地内存中，以物化视图形式提供只读访问，完全摆脱分布式网络依赖。此外，系统应尽可能直接依赖 Amazon DynamoDB 或 Aurora DSQL 这类高弹性水平扩展存储，倒逼底层数据引擎原生满足严苛的吞吐与延迟指标，而非用分布式缓存粉饰存储性能短板。发生率悖论与极端故障恢复代价 (4:18) 针对亚稳态故障在生产环境中的实际表现，Brooker 提出了一个行业悖论：在单个工程团队的日常运维中，工程师可能连续数年都不会亲身经历一次亚稳态失效；然而在全行业范围审视最具破坏力的大型宕机事故复盘报告（Postmortems）时，亚稳态故障往往作为底层驱动因素出现在绝大多数严重案例中。此类故障不同于常规的短暂软硬件偶发错误，其平均恢复时间（MTTR）极长，且无法通过自然排队泄流自行缓解。由于资源争用已彻底封堵了自愈通道，运维人员往往被迫采取极具操作风险的高危手段——例如将整个服务集群全量下线并执行强制冷重启（Cold Reboot），以此硬性切断争用循环。因此，Brooker 强调工程界必须深入研究并警惕此类低频却具毁灭性的失效动态。 Brooker 总结认为，缓存本身并非完全不可用的错误模式，但其伴随的系统性可用性隐患十分显著，属于架构设计中应尽力规避的复杂妥协。系统设计者需在局部性带来的性能收益与系统陷入亚稳态灾难的风险之间审慎权衡，优先推动底层存储实现应对全量真实负载的原生扩展能力。

Source quotes

[2:25] Marc Brooker: Caching isn't a bad pattern, but it is a pattern with some significant downsides that are, you know, really, uh, best avoided. [4:18] Marc Brooker: If you look across the biggest, most impactful, uh, you know, system postmortems across the industry, I would say that these kinds of metastable failures have been an underlying cause in probably a majority of them.

Original transcript

[0:00] I mean, since you have worked on AWS for [0:04] almost two decades, [0:06] I'm sure you have a lot of experience [0:08] building distributed systems. And I [0:10] think one of the most common advice that [0:12] you hear, I guess this is maybe in the [0:14] context of system design, is I I almost [0:17] hear almost 100% of the time people will [0:20] say, "Just throw a cache on it." Or you [0:23] have a system design, you say, "How do [0:24] you make it better? Let's put a cache [0:26] here. Let's put a cache there." And I [0:28] saw you had a tweet that said that there [0:30] are cases where caches are are bad [0:33] despite people saying it's best [0:34] practice. I was curious if you could [0:36] explain that. [0:37] >> Yeah, so caching's good, right? Like [0:39] it's hey, I'm I'm going to take the [0:41] these core ideas from computer science [0:43] of of temporal and spatial locality, and [0:46] I'm going to exploit those to make my [0:48] system faster, scale better, etc. And [0:52] so, you know, obviously very attractive. [0:54] But, [0:56] the downside of caches, especially in [0:58] distributed systems, is they have this [1:00] mode, right? Like they have this [1:02] you know, the there's a mode where the [1:04] cache is full, and the cache is full of [1:06] the right data in time and space to [1:08] perform very well. [1:10] And there's a mode where the cache is [1:11] empty or contains the wrong data. [1:14] And [1:16] in the first mode, the system is fast [1:18] and happy and healthy. [1:21] In the second mode, the system is slow, [1:24] often down because now the back end [1:26] isn't scaled to deal with [1:28] all of this un-cached traffic. Customers [1:31] are very disappointed. Um [1:34] and often it is down in a stable way. [1:37] And this is this kind of idea of [1:39] meta-stable failures, where the system [1:41] has has um switched from state one to [1:44] state two, and in state two it's still [1:46] stable, right? Like it's still it's [1:48] down, but it's not going to come back up [1:50] under its own energy because, for [1:53] example, all of this traffic is causing [1:55] a huge amount of contention in my [1:57] database or is saturating the network [1:59] and so I can't even refill the cache. [2:01] It's not even getting the right kind of [2:03] data in. [2:05] And so, you know, when I talk about the [2:06] downsides of caches, it's really about, [2:08] you know, how do we avoid that modality [2:12] between [2:14] you know, fast and, you know, [2:17] uh the that that value of caches and [2:21] the, you know, how do we avoid the stale [2:22] read where we're down? [2:23] Um [2:25] And so, if I go back to [2:27] to DSQL, like our answer there is DSQL, [2:31] what we call the storage tier, is [2:32] essentially a cache, but it is a [2:34] complete cache. It contains every row in [2:37] the database. [2:38] Um and so, it doesn't have this mode [2:40] where how do I recover from it being [2:42] empty or containing the wrong data? It [2:45] contains all of the data. [2:47] Um [2:49] Similarly, if you look at a a more, [2:51] let's say, classical relational database [2:53] design like Aurora, [2:55] the Aurora leader is constantly telling [2:57] the potential failover targets, "Here's [2:59] something you should cache. Here's [3:00] something you should cache. Here's [3:01] something you should cache." So, when a [3:03] failover happens, the cache is warm on, [3:07] you know, on on the failover target. Um [3:10] and so, those are the kinds of things [3:11] that you can do to avoid those [3:13] modalities. [3:14] But in general, [3:16] um [3:17] you know, and I I I I wouldn't extract [3:19] this as a rule or or or or or say that, [3:22] you know, this applies 100% of the time. [3:25] But in general, I prefer to see the [3:28] teams around me avoiding caching where [3:30] possible. [3:31] I prefer patterns where you have a, [3:35] let's say, complete materialized view of [3:37] the data if you need very fast access to [3:39] it. Especially if it's slow moving, just [3:41] pull it down on your local machine and [3:43] work with it in memory. [3:44] You know, if it's only being updated [3:46] once a week, who cares? Like, just make [3:47] lots of copies of it. [3:49] Um [3:50] Uh so, that's that's one pattern or, you [3:53] know, use a scalable back end, you know, [3:55] D sequel or Dynamo DB or whatever your [3:58] favorite scalable database is [4:00] and keep your database vendor honest [4:02] about getting to the the scale and [4:04] performance you need rather than putting [4:06] a cash in front of things. [4:08] So, caching isn't a bad pattern, but it [4:10] is a pattern with some [4:12] significant downsides that are, you [4:15] know, really, [4:16] uh, best avoided. [4:18] >> In practice, how how often do you see [4:21] that metastable failure though? [4:23] >> Yeah, you know, this is, uh, it's not [4:26] it's not super common, right? Like you [4:27] might go years without seeing, you know, [4:29] something like that. But if you look [4:31] across [4:33] the biggest, most impactful, [4:36] uh, you know, system postmortems across [4:37] the industry, I would say that these [4:39] kinds of metastable failures have been [4:43] an underlying cause in probably a [4:45] majority of them. And [4:48] it's super important that, you know, as [4:50] an industry and as a community of [4:52] practice, we understand those things [4:53] deeply because [4:55] the also those cases where these do [4:58] happen, [4:59] you know, tend to be [5:01] larger-scale issues, longer recovery [5:04] time issues, and and and more complex to [5:07] fix issues, right? Where you have to [5:09] often, you know, turn it off and turn it [5:11] back on again, which is this very very [5:14] painful thing for a a a team or an [5:16] organization to do. [5:18] Um, [5:19] and you know, so again, like you you [5:22] might go years operating a system with [5:23] seeing nothing like this and and but if [5:26] you look at the most impactful issues, [5:29] it's actually fairly common as an [5:31] underlying cause for those issues. And [5:33] so, you know, it's kind of both of these [5:35] things of being quite uncommon and being [5:37] being rather common.