Zach Tellman - Everything Will Flow
ClojureTV
AI-assisted summary of Zach Tellman - Everything Will Flow — original source. Check the original for full context.
Source published:
Key ideas
- Unbounded in-memory queues are fundamentally broken because production systems are open systems where arrival rates cannot be controlled, inevitably leading to catastrophic out-of-memory crashes. (12:34)
- Scaling out worker capacity and arrival rates proportionally does not prevent system failure; it merely tightens response-time variance and masks instability right up until catastrophic collapse. (9:46)
- Buffers do not decouple execution time—an unbuffered rendezvous channel already does that; their true purpose is serving as shock absorbers that mask the latency of pausing and resuming backpressure. (15:13)
- Architectures must centralize drop or reject policies at application edges while preserving upstream backpressure throughout internal pipelines to prevent silent data loss or cascading outages. (17:42)
Detailed notes
## Background Published on YouTube by ClojureTV on April 22, 2015, this talk features Zach Tellman examining the operational complexities of queue-driven architectures. According to the video description, queues unlock problem spaces unreachable by pure immutable primitives, but naive implementations in libraries like core.async risk building brittle systems, making an understanding of queueing theory essential for practical Clojure engineering.
## TL;DR Tellman demonstrates that modern software is layered with queues coordinating execution across thread pools, operating systems, and network boundaries. Through mathematical simulations, he proves that open systems face abrupt, rocket-like latency collapses when traffic spikes, and that adding consumers merely masks this variance until catastrophic failure occurs. To prevent memory exhaustion, architectures must replace unbounded queues with backpressure, explicit rejection, or dropping policies at application boundaries. Using case studies from high-throughput services and real-time chat servers, he shows how to protect systems using durable disk queues, token-bucket throttling, instrumented thread pools, and stream topology inspection.
## Atmosphere & style The presentation feels like a sobering, battle-hardened masterclass delivered with deadpan pragmatism and intellectual clarity. Tellman builds an engaging narrative arc that transitions from theoretical probability distributions to live simulation spectrograms and painful real-world post-mortems of cascading cloud outages. His delivery is steady, unpretentious, and urgent, creating an atmosphere that strips away architectural wishful thinking to urge developers out of willful ignorance and into rigorous systems engineering.
## Bulletin - [12:34](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=754s) Unbounded in-memory queues are fundamentally broken because production systems are open systems where arrival rates cannot be controlled, inevitably leading to catastrophic out-of-memory crashes. - [09:46](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=586s) Scaling out worker capacity and arrival rates proportionally does not prevent system failure; it merely tightens response-time variance and masks instability right up until catastrophic collapse. - [15:13](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=913s) Buffers do not decouple execution time—an unbuffered rendezvous channel already does that; their true purpose is serving as shock absorbers that mask the latency of pausing and resuming backpressure. - [17:42](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1062s) Architectures must centralize drop or reject policies at application edges while preserving upstream backpressure throughout internal pipelines to prevent silent data loss or cascading outages.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=0s) Queues fundamentally orchestrate side effects by separating what needs to happen from when it happens, moving beyond pure, immutable functional transformations. - [01:04](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=64s) Modern execution stacks nest roughly half a dozen queues between application channels, thread pool blocking queues, OS run queues, and hardware CPU cores. - [03:32](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=212s) Open systems are inherently unstable compared to closed systems because producers generate requests independently without waiting for consumer availability. - [08:50](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=530s) Queues under growing load experience a rocket-like latency spike where queuing delay completely dominates total execution time. - [09:46](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=586s) Adding consumers narrows latency variance and creates an illusion of safety, but multi-consumer systems still fail at the exact same load threshold as single consumers. - [12:34](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=754s) When arrival rates outstrip processing capacity, systems have only three fundamental options: drop, reject, or apply backpressure. - [15:13](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=913s) Buffers are unnecessary for temporal decoupling; an unbuffered channel safely coordinates handoffs, while buffers exist primarily to amortize backpressure signaling delays. - [17:42](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1062s) Chaining buffered queues in series compounds and magnifies downstream latency, making end-to-end performance unpredictable. - [22:10](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1330s) Relying on implicit in-memory buffering in third-party clients during downstream outages cascades quickly into JVM out-of-memory crashes across entire clusters. - [23:55](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1435s) Decoupling network persistence by appending directly to local disk queues isolates external storage outages and keeps ingestion endpoints available. - [26:49](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1609s) Instrumenting thread pools with quantile distributions rather than scalar averages is essential to expose tail latency and task rejection spikes under load. - [31:25](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1885s) Chat rooms and pub-sub systems require size-bounded output buffers with disconnection timeouts to prevent slow subscribers from stalling shared distribution streams. - [35:37](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=2137s) Inbound flooding should be countered via transport-level TCP backpressure combined with token-bucket burst limits rather than complex disconnection logic. - [37:18](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=2238s) Dynamic asynchronous architectures require stream-level introspection to track shifting consumer topologies and avoid willful operational blindness. - [39:20](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=2360s) All queues are physically bounded by host memory; robust software requires explicit edge policies, universal backpressure in middleware, and deep instrumentation.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=0s) Queues, Side Effects, and the Clojure Ecosystem
Queues appear deceptively elementary on the surface: a producer enqueues messages, a consumer dequeues them—usually, though not always, in FIFO order—and that is essentially the whole mechanism. However, an often-overlooked property of queues is their inherently side-effecting nature. When an enqueue operation completes, the caller receives only an acknowledgment that the item was accepted rather than a pure transformation of the input data. The data placed in a queue exists to describe an action, making queues fundamentally a mechanism for orchestrating side effects.
Because of this focus on side effects and state transitions, queues were long relegated to the sidelines of the Clojure community. Developers historically relied on standard Java concurrency utilities, wrapping or abstracting over them whenever possible rather than treating them as first-class primitives. That changed dramatically with the introduction of `core.async`, which brought queueing and asynchronous coordination straight into mainstream Clojure application architecture. Zach Tellman argues that embracing queues is essential for real-world software engineering:
> “we of course are in the business of making computers do things right we don't just raise the ambient temperature of the room”
[01:04](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=64s) Queues All the Way Down: Core.async, Thread Pools, and Hardware
Examining the machinery beneath modern concurrency primitives reveals layers of nested queues. A single `core.async` channel does not contain just one queue; it manages three separate ones: a buffer that stores the payload messages, a puts queue where producers wait when attempting to insert into a full buffer, and a takes queue where consumers wait when attempting to extract from an empty buffer. Even after a producer successfully delivers a message to a consumer, the handoff is still incomplete. The consumer attaches a callback to execute work against that message, and running that callback requires dispatching it to a thread pool.
A Java thread pool inserts another queue—a `BlockingQueue`—between tasks requesting execution and the worker threads servicing them. By default, thread pools scale dynamically: when a caller attempts to place work on a full queue, the pool instantiates a new worker thread until reaching its configured ceiling. If that limit is exceeded, the pool abruptly throws a `RejectedExecutionException`. Internally, the `BlockingQueue` mirrors `core.async`: it holds a message buffer alongside two condition queues named `notFull` and `notEmpty`. These conditions do not transmit data; they simply provide parking locations where blocked threads release their locks and sleep until a single thread is signaled to wake up. Further down the operating system stack, software threads queue on the kernel scheduler before finally waiting to execute on physical hardware threads and CPU cores.
```mermaid flowchart TD accTitle: Layers of Nested Queues accDescr: Diagram showing how a message travels across channel queues, thread pools, and OS hardware queues.
subgraph Channel["core.async Channel"] Puts["Puts Queue"] Buf["Message Buffer"] Takes["Takes Queue"] end
subgraph ThreadPool["Java Thread Pool"] BQ["BlockingQueue\n(notFull / notEmpty)"] Workers["Worker Threads"] end
subgraph OS["Operating System & Hardware"] Sched["OS Run Queue"] Cores["CPU Cores"] end
Puts --> Buf Buf --> Takes Takes --> BQ BQ --> Workers Workers --> Sched Sched --> Cores ```
Between high-level application logic and the underlying silicon, roughly half a dozen distinct queues coordinate execution. Tellman emphasizes that this omnipresence stems from a core operational benefit:
> “and the reason that cues are so ubiquitous is because they separate what we want to have happen uh from when it happens and this is really valuable because most of the time when we write code we don't know what else is going on”
- `core.async` channels rely on three internal queues: a message buffer, a waiting-producers queue (`puts`), and a waiting-consumers queue (`takes`). - Java thread pools expand when their underlying `BlockingQueue` fills up, raising a `RejectedExecutionException` once thread limits are breached. - Java condition queues (`notFull` and `notEmpty`) coordinate thread parking and single-thread wakeups without passing data payloads. - Software threads are queued once more by the OS scheduler before mapping onto physical processor cores.
[03:32](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=212s) Queuing Theory and Closed vs. Open Systems
Because queues mediate nearly every layer of modern software execution, understanding how they behave, degrade, and fail under heavy load is critical. Queuing theory provides the formal mathematical foundation for these questions, though it belongs primarily to the domain of statistics rather than everyday software engineering. Approaching the discipline requires filtering theoretical statistical proofs down to practical mental models that inform operational intuition. For developers seeking to explore the field further, Tellman recommends the academic literature—noting that while the best available textbooks are roughly 90% pure probability theory with only sporadic mentions of computing systems, the insights gained can profoundly shape architectural instincts.
A foundational architectural distinction within queuing theory is the boundary separating closed systems from open systems:
- **Closed Systems**: A producer must wait for the consumer to finish processing the current unit of work before producing another. Classic examples include a REPL—which must read input, evaluate it, and print output before prompting the user again—or an individual browsing a website, where the user must receive the rendered page before clicking another link. Because control yields cooperatively back and forth, closed systems are inherently stable and straightforward to reason about. - **Open Systems**: Producers emit requests without any central coordination or awareness of consumer availability. Incoming tasks arrive continuously regardless of whether the processing workers are occupied.
Tellman explains that open systems are the primary source of failure in real-world infrastructure:
> “and closed systems are uh certainly a thing that we deal with and they are as it turns out fairly easy to reason about but many of the things that give us trouble and many of the systems that we build are open systems where there is no coordination where when a request comes in we're processing it it's almost a given that a request is to come in before we're ready”
[06:01](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=361s) Simulating Open Systems: Arrival Patterns and Latency Spectrograms
Production metrics such as Amazon Web Services dashboard graphs contain too many confounding variables to illustrate the core mathematical dynamics of open systems. To isolate these behaviors, Tellman constructs a software simulation. In this model, request arrival times follow an exponential distribution, where the interval between successive arrivals varies stochastically. Empirically, exponential distributions accurately reflect uncoordinated human behaviors, such as people lining up at bank tellers, visiting food trucks, or competing for shared public resources. Even if an endpoint sustains an average rate of 1,000 queries per second, requests do not arrive at neat, uniform 1-millisecond intervals; instead, they arrive in unpredictable bursts that pile up on one another, separated by intermittent idle periods.
While arrival intervals match an exponential distribution, task execution complexity follows a Pareto distribution. This power-law distribution models varying tail lengths (comparable to the familiar 80/20 rule), allowing the simulation to modulate the proportion of unusually complex, expensive operations entering the pipeline. To mirror operational degradation—such as increasing disk saturation or neighbor contention on a multi-tenant cloud virtual machine—the simulation slowly widens the tail over time, shifting simple tasks toward heavier computation.
To capture the resulting system dynamics across time, Tellman uses a spectrogram:
- **Vertical Axis**: Displays end-to-end task latency—the complete duration spanning arrival, queue dwell time, processing, and final completion—plotted on a logarithmic scale to keep extreme tail outliers visible alongside baseline throughput as instability sets in. - **Horizontal Axis**: Tracks the progressive widening of the execution-time distribution, moving smoothly from a thin tail where almost all tasks are trivial to a fat tail where complex outliers become common. - **Heat Map Color Density**: Visualizes probability density, starting on the left with a sharp, intensely colored red band indicating that the vast majority of tasks complete within a tight, predictable latency window before degradations begin to disrupt the system.
[08:50](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=530s) The Rocket-Like Failure Mode of a Growing Queue
Zach Tellman explains how a queue in an open system destabilizes as load creeps up. Looking at the simulation data, about two-thirds of the way through the timeline, the system suddenly jumps to an entirely new equilibrium. On the chart, the red dots scatter several orders of magnitude above where they hovered previously. While this new state is technically an equilibrium and not yet completely out of control, everything operates significantly slower. Eventually, as load continues to pile on, the response times shoot upward like a rocket.
```mermaid flowchart LR accTitle: Unbounded Queue Degradation accDescr: Three progressive stages of an unbounded queue moving from low latency to memory exhaustion. subgraph Stable["Normal Load"] A["Low Latency<br/>Tight Distribution"] end subgraph Saturated["Intermediate Shift"] B["High Latency Equilibrium<br/>Queue Wait Dominates"] end subgraph Collapse["Runaway Failure"] C["Rocketing Latency<br/>OOM Crash"] end A --> B --> C ```
Tellman underscores that this exponential jump is the hallmark of a queue in crisis: “This is an equilibrium we're not out of control yet but everything is significantly significantly slower than it was before and then as it continues we see that it gets more and more out of control until it just sort of takes off like a rocket and this is what happens when a queue grows out of control right this is a system in crisis this is what will cause our system to run out of memory and crash.” Additionally, the distribution of latencies flattens from a tight spread into a very thin, uniform band. Because requests spend almost all their time waiting behind other backlog items rather than being processed, queuing delay dominates total execution time, averaging everything out into a consistent yet painfully slow response profile.
[09:46](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=586s) Scaling Consumers, Variance, and the Illusion of Safety
When systems run out of capacity, the instinctive engineering response is simply to add more machines or worker threads. Adding a second consumer initially allows the system to keep up, delaying the flare-up until near the end of the run. Adding two more consumers makes the system appear virtually unfazed, remaining well within safe operating bounds. However, Tellman shifts to a far less obvious experiment: what happens when consumer capacity and incoming throughput are scaled up together by the exact same proportion? Comparing a 1x arrival rate with 1 consumer, a 4x rate with 4 consumers, and a 16x rate with 16 consumers reveals a startling reality: every single configuration fails at the exact same point in time.
The visual shape of the failure leading up to that cliff edge, however, is radically different. In a 1-consumer setup, an unusually expensive task completely ties up the single worker, causing all subsequent work to pile up immediately behind it. With 4 consumers, an expensive outlier ties up only 25% of the capacity, allowing the remaining three consumers to continue draining the queue and making forward progress. In a 16-consumer setup, it requires an extraordinarily rare confluence of multiple concurrent expensive events to saturate all workers at once. Consequently, the 16-consumer system maintains clean, sub-millisecond response times almost right up to the exact moment it falls off a cliff. Tellman warns that this deceptive dynamic perfectly mirrors modern architectures:
> “And I will point out that this is the best sort of analog for the systems that we build we have multiple machines we have multiple threads and so we can absolutely have a system which is responding in submillisecond time right up to the point where it becomes completely unstable right it happens pretty much all at once.”
The key takeaways from this behavior include: - Production systems are almost always open systems, making them difficult to reason about during traffic spikes. - Long-tail, complex tasks—not the mean or median complexity—are what cause queues to saturate and trigger cascade failures. - Engineers must closely track response time outliers because downstream variance directly dictates whether upstream consumers remain stable or collapse. - Adding consumers does not prevent an open system from being overwhelmed; it merely narrows variance and masks instability until sudden, catastrophic failure occurs.
[12:34](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=754s) Why Unbounded Queues Are Broken: Drop, Reject, or Pause
Because arrival rates will inevitably outstrip capacity at some point, Tellman delivers the central warning of his talk:
> “And so the real lesson here and the one thing that I hope that you take away from this is that unbounded queues are fundamentally broken because it puts the correctness of your system in somebody else's hands right you have no control over how fast the data is showing up and so that means that someone can intentionally or unintentionally cause your system to break and so we cannot have our strategy for dealing with too much data to hold on to it and hope for the best we have to you know do something a little bit more proactive.”
When excess volume overwhelms capacity, systems have only three fundamental options:
| Strategy | Definition | Typical Applications & Drawbacks | |:--- |:--- |:--- | | **Drop** | Discard the incoming data silently without informing the sender. | Valid in telemetry or sensor systems where newer samples render older values obsolete, or in best-effort protocols like StatsD over UDP. Drawback: network crises trigger drops precisely when accurate monitoring is needed most. | | **Reject** | Discard the incoming data and notify the sender that it was refused. | The standard approach at the web layer (e.g., HTTP `503 Service Unavailable`, Twitter's classic "fail whale"). Necessary because servers rarely possess the application context or identity required to process the user later. | | **Pause (Backpressure)** | Signal upstream producers to stop sending until capacity frees up. | The ideal strategy for libraries, frameworks, and internal pipelines. Avoids making application-specific drop/reject guesses by delegating timing control to upstream callers. |
Tellman points out that developers often rely on dropping data far too readily. In monitoring infrastructure using StatsD over UDP, metrics packets are fired blindly with a prayer that they arrive. Yet under heavy network distress, packet loss spikes—meaning accurate telemetry vanishes precisely when operators need it most. Conversely, while rejecting data via HTTP 503 errors or fail whales is unpleasant, application servers lack the state to do anything more graceful; an overloaded e-commerce store cannot tell an anonymous shopper that it will email them once the checkout cart becomes available.
[15:13](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=913s) Backpressure in core.async and the Real Purpose of Buffers
When building lower-level libraries or general-purpose frameworks rather than edge web applications, developers cannot predict whether dropping or rejecting tasks is acceptable. Instead, they must exert backpressure to pause incoming work, deferring operational choices to upstream callers who understand the broader business domain. This architectural necessity explains why Clojure's `core.async` employs three distinct queues internally rather than just a single storage line: it maintains a distinct puts buffer to track operations on which backpressure has been exerted. When a producer attempts to write to a saturated channel, the runtime registers the pending put, holds the execution, and notifies the producer only once consumer capacity frees up.
```mermaid flowchart LR accTitle: Core.async Channel Structure accDescr: Architecture of an unbuffered core.async channel showing puts and takes synchronization. subgraph Channel["Unbuffered core.async Channel"] P["Puts Buffer<br/>(Suspended Producers)"] T["Takes Buffer<br/>(Waiting Consumers)"] end Prod["Upstream Producer"] -->|"put! / >!"| P P -.->|"Direct Handoff<br/>(Rendezvous)"| T T -->|"take! / <!"| Cons["Downstream Consumer"] ```
Queues decouple *what* needs to be done from *when* it is done, allowing bounded hardware resources to handle potentially unbounded external demand. However, Tellman emphasizes that a buffer—the storage container holding pending items—is completely unnecessary to achieve this decoupling:
> “And these are all sort of saying the same thing but the crucial point I want to make is that the buffer right the thing that holds on to the messages is unnecessary for any of these properties right in fact we could just as easily have a channel which has just the puts and the takes just waits to sort of pass the message off once it's available and in fact in core.async this is the default channel configuration it has no buffer.”
An unbuffered channel acts as a closed system. Because an unbuffered channel forces an explicit rendezvous where the producer yields control until a consumer physically accepts the value, it prevents runaway queue growth.
If unbuffered channels provide safety and buffers inherently introduce latency, why use buffers at all? Tellman explains that asserting and releasing backpressure carries a non-trivial physical cost: - Restarting a paused thread requires copying stack frames back into working memory. - Resuming paused TCP connections requires round-trip network signaling before window sizes ramp traffic back up. - Re-engaging paused disk or database readers incurs storage seek and lookup latencies.
The idle time spent waiting between a consumer signaling readiness and the upstream producer actually delivering the next task harms aggregate throughput. Buffers exist not as limitless holding pens for excessive demand, but as shock absorbers that hide backpressure signaling delays and keep workers continuously supplied with work.
[17:42](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1062s) Planning for Excess Data, Backpressure, and the Limits of Buffers
Zach Tellman explains that smoothing out throughput into a higher, stable rate always comes at a direct cost to system latency. Because bursts and surges of volume cannot be avoided in real-world software, engineering for overload must be deliberate rather than accidental. When an influx occurs, a system must possess a clear strategy for coping with excess traffic. Wherever possible, backpressure represents the cleanest, most neutral recourse available: it allows a component simply to signal to its caller that it cannot accept further data, leaving the responsibility of how to handle the rejection to the caller. That signal can travel back up the call chain all the way to the boundary of the application, where edge-level logic can uniformly decide whether to drop packets, shed load, or reject requests, ensuring policy remains centralized rather than scattered indiscriminately through internal components.
While buffering can smooth throughput, its hidden cost remains inflated latency. Consequently, buffers should be introduced sparingly and only when strictly required. Tellman cautions strongly against composing buffered queues in series, as chained buffers tend to compound and magnify downstream delays and unpredictability. Looking at the wider Clojure ecosystem, he observes a troubling disconnect between these queueing principles and actual implementations. While libraries such as `core.async` implement backpressure internally with great care, trouble surfaces when they are connected to underlying networking transports or server interfaces where traffic volume cannot be controlled:
> “And wherever possible we should use backpressure, because that's the most neutral choice that we can make, right? That just kind of tells someone else, ‘I can't handle this data, you figure out what that means.’”
```mermaid flowchart LR accTitle: The Golden Chain of Backpressure accDescr: Incoming network data flows forward while backpressure signals flow in reverse to edge rejection logic. subgraph Edge["Periphery / Edge"] Client["Client / Network Traffic"] Reject["Centralized Rejection / Dropping Logic"] end subgraph App["Application Core"] Q1["Ingest / Unbuffered Pipeline"] Worker["Worker / Downstream Processor"] end
Client --> Q1 Q1 --> Worker Worker -.->|"Backpressure (can't handle data)"| Q1 Q1 -.->|"Propagates back"| Reject Reject -.->|"Reject / Shed Load"| Client ```
- In some technologies, omitting backpressure is a protocol failure: for example, the Java EE WebSocket specification completely lacks a mechanism for backpressure, offering developers inside servlet containers no standard means to slow upstream data rates. - In other instances, authors fail to link queues to transports due to oversight, never having suffered the cascading failures that occur when the "golden chain of backpressure" is broken. - Consumers of Clojure networking and async libraries must critically audit dependencies to ensure foundational plumbing will withstand production spikes rather than forcing complete architectural rewrites under load.
---
[20:16](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1216s) Designing "Maw": The High-Throughput Ingestion Endpoint
To demonstrate these challenges in practice, Tellman presents "Maw," an internal service designed as an omnivorous endpoint built to ingest arbitrary payloads delivered by external partners and customers. Maw’s operational model is conceptually direct: capture enormous volumes of incoming data, perform light processing, and persist the records for future batch workloads. To minimize operational friction on complex infrastructure, Maw was architected on top of Amazon Web Services (AWS), using an Elastic Load Balancer (ELB) to fan incoming network traffic across multiple nodes, and targeting Amazon S3 as a reliable, highly available, replicated storage destination.
The application’s constraints dictated a very specific architecture. First, it had to scale horizontally to meet massive traffic expansion: while Maw originally handled roughly 40,000 queries per second, its peak volume reached approximately 300,000 queries per second within 18 months. Second, it had to remain exceptionally low-maintenance; Tellman notes that if Maw requires more than a single day of maintenance per month, something has gone terribly wrong. Third, ingestion did not need to operate in real-time—it simply required that all collected records landed securely on S3 by the end of the day. Finally, Maw benefited from a forgiving reliability metric: data utility scaled strictly linearly with volume:
> “And the most important property is that the loss of the data is more or less directly proportional to loss utility. If we lose 1% of the data, we lose about 1% of the utility.”
- Unlike transactional domains like payment processing—where a 1% data drop destroys the system's baseline usefulness—losing 1% of Maw's telemetry input simply reduces downstream analytical utility by roughly 1%, greatly simplifying trade-offs. - The service was designed so traffic increases would not force continuous ground-up redesigns of the application tier.
---
[22:10](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1330s) The S3 Outage and the Cascading Failure of Unbounded Buffers
Before Maw, Tellman built an ingestion system consuming the Twitter "garden hose" (the official stream providing 10% of the Twitter fire hose), which generated a modest 600 tweets per second. That earlier pipeline passed an HTTP client stream into a standard Hadoop S3 client to write records to Amazon S3 without incident. When bootstrapping Maw's prototype for an anticipated 20,000 entries per second, he reused that exact architecture, merely placing an HTTP web server in front of the Hadoop S3 client.
This design concealed a fatal vulnerability: Amazon S3 occasionally experiences transient blips during which it temporarily stops accepting write operations. When S3 stalled, the Hadoop S3 client attempted to survive the outage by silently buffering all incoming records in memory, expecting the connection to clear so it could flush the backlog. At Maw's incoming volume, this unconstrained in-memory buffering quickly consumed all available JVM heap space:
> “And so we ran out of memory, and the system fell over, and then all of the traffic that that machine was handling fell over to the other machines, which themselves fell over, and then the entire system was dead.”
- In-memory buffering had likely triggered during the Twitter garden hose integration as well, but because 600 tweets per second generated a minimal footprint during brief S3 pauses, the queue flushed silently before memory ran out. - The failure exposed the danger of invisible unbounded buffers: the Hadoop client provided no status metrics, making it impossible to diagnose backlogs until the JVM crashed from an out-of-memory error.
---
[23:55](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1435s) Decoupling Ingestion with Durable Disks and Explicit Metrics
Reflecting on the outage, Tellman realized he had burdened the intake layer with an unnecessary guarantee: forcing an synchronous round-trip write over the network into S3’s replicated storage before acknowledging the payload. All that ingestion actually required was moving data out of volatile process memory into a durable, local destination. Writing records to the machine’s local disk offered far greater immediate availability than S3, despite the operational trade-off that a sudden hardware loss would destroy the un-flushed data on that host.
To decouple ingestion from network persistence, Tellman authored two open-source libraries: `durable-queue` (a simple, disk-backed queue that has processed roughly 3 to 4 trillion events) and `s3-journal` (a buffer built on `durable-queue` that commits items to disk before asynchronously uploading them to S3). Furthermore, to eliminate the blind spots of the earlier client, both libraries expose fine-grained metrics for health monitoring, reporting enqueue counts, successful uploads, in-memory file slabs, pending records, and retry attempts:
> “These are all things that are absolutely critical because we cannot prove properties about our system, right? The system is too complicated. We can only look and see what happens in as much as the metrics that we have created for ourselves allow us to do that.”
```mermaid flowchart LR accTitle: Maw Ingestion Pipeline with Durable Disk Queue accDescr: HTTP requests hit the Aleph/Netty server, write directly to a local durable queue, and are uploaded asynchronously to S3. subgraph Host["Single Maw Node"] HTTP["Aleph / Netty HTTP Server"] DQ["Local Disk (durable-queue)"] Journal["Upload Loop (s3-journal)"] end Remote["Amazon S3"]
HTTP -->|"1. Fast append"| DQ DQ -->|"2. Read batches"| Journal Journal -->|"3. Asynchronous flush"| Remote ```
- Writing to local storage eliminated the downstream latency of S3 replication hops from the critical ingest path. - Splitting the ingest path into an HTTP-to-disk write step and an independent disk-to-S3 upload loop isolated S3 downtime from request acceptance. - On the ingestion boundary, the Aleph HTTP server (backed by Netty) faced an additional constraint: it must return fast HTTP acknowledgments to upstream senders, because clients throttle their transmission rates if upstream latency spikes.
[26:49](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1609s) Netty's Threading Model and Instrumented Thread Pools
Zach Tellman explains that Netty has a very particular threading model: whenever a connection opens, it is assigned to a thread, and multiple connections are assigned to that same thread. Consequently, if any single connection blocks on that thread, it implicitly blocks every other connection sharing it, dragging along requests that never chose to be affected. The standard remedy is placing another thread pool behind Netty to distribute work and prevent one misbehaving connection from causing others to back up. However, metrics gathered at the Ring handler level often diverge significantly from actual response times—the true time from packet entry to packet exit. Tellman points out that thread pools themselves are notoriously under-instrumented: they act as an invisible boundary where a queue exists, tasks wait an unknown amount of time, and work completes at an unspecified moment.
```mermaid flowchart TD accTitle: Netty Threading Model and Backing Thread Pool accDescr: Incoming network connections map to shared Netty I/O worker threads, which forward tasks through an instrumented thread pool queue to isolated worker threads. subgraph NettyLayer["Netty I/O Layer"] C1["Connection 1"] --> T1["Shared Netty Thread"] C2["Connection 2 (slow)"] --> T1 end subgraph PoolLayer["Instrumented Execution Pool"] T1 --> Q["Thread Pool Queue"] Q --> W1["Worker Thread A"] Q --> W2["Worker Thread B"] end ```
To combat this blindness, Tellman wrote a library that includes an instrumented thread pool tracking core operational statistics. Showing a JSON endpoint from live Factual servers, he examines task rejection, completion, and arrival rates. Crucially, none of these values are simple scalars; they are quantile distributions because uncoordinated client requests spike and drop dynamically over time. When capacity planning, these tails matter: the difference between the median and the 99.9th percentile for arrival and completion rates is roughly a 3x spread. Even though task rejections happen infrequently, the metrics show that the system does drop data under peak capacity constraints, making rejection tracking an absolute necessity.
- **Quantiles over scalars:** Task arrival, completion, and rejection must be measured across quantiles because uneven arrival bursts produce a 3x spread between the median and the 99.9th percentile. - **Queue vs. task latency:** Task latency is a superset measuring the entire lifecycle (queue wait plus execution time), whereas queue latency isolates time spent waiting in line. - **Sampled metrics artifact:** In the sampled statistics, the 99.9th percentile of queue latency occasionally reads higher than that of task latency because the samples capture slightly different operational windows. - **Unbuffered queue delays:** Even though the monitored queue has an unbuffered length of zero, traversing it can still take up to 2 milliseconds due to lower-level buffered queues underlying the Java execution runtime.
[29:44](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1784s) Measurement Limits and System Tooling
Having visibility into queues and thread pools gives operators a clearer picture, yet Tellman admits it remains an approximation. The instrumented thread pool does not inspect the deep internals of Netty, meaning that if a massive divergence ever appeared between internal metrics and external measurements, engineers would still have to dig deeper. But having partial visibility beats willful ignorance:
> “There’s always something more to measure, but the idea is that we just don’t want to be sort of willfully blind and close our eyes and pretend that everything’s okay.”
Understanding what constitutes a completed task is vital when architecting systems, Tellman argues, because an accumulation of incomplete tasks forces an architecture to apply backpressure, drop packets, or reject input outright. Moving the goalposts to complete work earlier provides enormous leverage. When selecting engineering tools, raw performance benchmarks only show that software can run fast under artificial conditions; operators should always prioritize tools that actively report their own runtime speed in production. Tellman reminds the audience that certain system variables will always escape direct control—such as a mobile user driving into a tunnel and dropping their connection—yet from the user's perspective, that hiatus represents the service's latency.
[31:25](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1885s) Building a Robust Chat Server with Manifold
To demonstrate these dynamics in practice, Tellman introduces a classic scenario: a multi-user chat server where participants enter a room and broadcast messages to everyone else. Using his library `manifold`—chosen because of its familiarity and useful stream operators, though `core.async` or alternative stream libraries could serve the same role—he defines an event bus implementing a publish-subscribe model. Publishing to a topic sends the message to all active subscribers; if no one is subscribed, the message becomes a no-op that evaporates into the ether. A basic chat handler extracts the requested room from an incoming connection, publishes that connection's messages to the room stream, and pipes the room's stream back out to the connection.
```mermaid flowchart LR accTitle: Multi-user Chat Room Pipeline with Backpressure Timeout accDescr: Incoming user messages are published to an event bus room, buffered, and piped into each client connection with a drop timeout. U1["Client 1 (Publisher)"] -->|"publish"| EB["Manifold Event Bus Room"] EB -->|"pipe"| B["Aggregate Size Buffer (limit 1000)"] B -->|"connect (10s timeout)"| U2["Client 2 (Subscriber)"] ```
This naive implementation breaks down quickly under real-world conditions. Because publishing requires delivering the message to every subscriber in the room, a single slow consumer delays message distribution for all other users. To prevent one bad connection from destabilizing the chat room, Tellman explains how to add a buffer between the room's stream and the client's output connection. Rather than bounding the buffer by message count—which risks instability since a message could range from a single word to a novel—the buffer limits aggregate message byte size (such as 1,000 units). However, a buffer merely postpones failure instead of preventing it. Ultimately, the system must drop unresponsive consumers:
> “The only thing that we really can do is once someone is slow to receive our messages, is unable to receive them, we kick them out.”
Using Manifold's `connect` function, the server pipes the stream into the network sink with an explicit timeout. If a subscriber's buffer remains full and fails to accept a message for 10 seconds, the connection terminates automatically, forcing them to reconnect.
[34:42](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=2082s) Protecting the System from Bad Actors and Floods
Solving the slow-consumer problem immediately exposes a reciprocal failure mode: if a publisher sends messages at an extreme rate, they can fill everyone else's buffers simultaneously and trigger mass disconnections across the room. Whether caused by malicious intent or naive client code, the underlying problem remains identical, and Tellman emphasizes that engineers cannot afford to care about the user's motive.
> “It’s your job to make sure that you don’t get knocked over, that, you know, the many don’t suffer because of the actions of the few and so you really need to think about this carefully.”
To defend against floods and prevent a few active clients from taking down the entire room, Tellman explains that incoming connections must be strictly throttled—for example, limiting a client's stream to a rate of only one message per second.
[35:37](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=2137s) Boundary Conditions: Throttling, Credits, and Robustness
Zach Tellman explains how inbound throttling differs from outbound disconnect logic. On the inbound side, a system does not need complex disconnection logic when dealing with a peer that sends traffic too aggressively. Instead, the server can rely on transport-level backpressure: it uses TCP backpressure to simply refuse additional data, telling the client that it will not accept the message. If refusing the incoming packets causes the client to fail, Zach notes, “that's their problem.”
```mermaid flowchart LR accTitle: Inbound backpressure and token-bucket throttling accDescr: Incoming TCP messages accumulate up to a 10-second burst credit or face backpressure. subgraph Inbound["Client Connection"] Client["Inbound Client"] -->|"Sends messages"| TCP["TCP Backpressure"] end subgraph Throttling["Rate Limiting Policy"] TCP -->|"Permitted flow"| Credits{"Credits Available?\n(Max 10s burst)"} Credits -->|"Yes"| Accept["Process Message\n(1 msg/sec rate)"] Credits -->|"No credits"| Stalled["Push back via TCP"] end ```
Moving into the specifics of rate limiting, Zach addresses the nuances of throttling rules. A naive rule enforcing exactly one message per second can feel irritating and unnatural to human users. For instance, if a participant sits quietly in a chat room for ten minutes, it is reasonable to ask whether they must still be limited to one message a second or if they should be allowed to send ten minutes' worth of messages in a burst. Both behaviors are valid design choices in throttling schemes. In his example, he configures a token-bucket approach where users accrue credits while idle: they can accumulate up to 10 seconds’ worth of credit. Thus, a silent user can send an immediate burst of 10 messages, but no more.
Defining these concrete rules strips away the illusion of an idealized, generic architecture. Zach points out that the example becomes “a little bit less platonic than it was before” because real-world resilience demands explicit trade-offs. Engineers must define the maximum memory allocated to each connection, determine acceptable client latency timeouts, and fix the inbound message rates. By moving beyond hand-waving abstractions where everyone projects their own ideal chat room onto the design, the architecture becomes precise and grounded. As Zach emphasizes:
> “This is what you need to do, right? These are the steps that you need to take. And I want to emphasize also that it's not hard to do this, right? You have to think a little bit about what these boundary conditions should be, but this is what's required for us to be able to make these robust systems, for us to be able to ride out the behavior of the outside world.”
[37:18](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=2238s) Topology Visibility and Dynamic Queue Systems
Zach shifts focus to a major architectural hazard in asynchronous systems: managing dynamic queue cardinality. In simpler architectures, an application might rely on one or two static queues. A fixed cardinality makes operational observability straightforward because engineers can attach permanent graphs and monitor queue depth directly. By contrast, an application like a chat room generates an untold number of queues that dynamically appear, disconnect, and reconfigure as users join and leave. Standard queue monitoring tools fail in this environment because the underlying system is constantly changing.
```mermaid flowchart TD accTitle: Manifold topology tracking between streams accDescr: Downstream inspection enables tracing flow across transient queue topologies. subgraph Topology["Dynamic Manifold Streams"] Source["Source Stream"] -->|"connect / consume"| DownstreamA["Downstream Stream A"] Source -->|"connect / consume"| DownstreamB["Downstream Stream B"] end subgraph Inspection["Observability"] Source -.->|"inspect topology"| Meta["Topology Markers\n('What streams are downstream?')"] end ```
Zach highlights that this dynamic churn presents a serious visibility problem in libraries like `core.async`:
> “If you think about what this would look like for core.async, you have all these go routines that are sort of like shoving messages in every direction, but you can't look at it and sort of apprehend it as a whole.”
To tackle this weakness in his library, Manifold, Zach introduced stream-level introspection. Operations such as `connect` or `consume` register internal markers that record a stream’s connections. Any stream can be inspected at runtime to reveal its immediate downstream consumers, creating an inspectable graph of the running pipeline. Zach acknowledges that visualizing these graphs remains an unsolved problem: early attempts using Graphviz quickly turned into unreadable, unscalable hairballs even for moderately complex topologies. Determining whether the solution is a better visualization engine or offline event logging to disk remains an open area for community exploration, but he stresses that Clojure developers cannot simply cross their fingers and hope transient asynchronous architectures behave.
[39:20](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=2360s) Core Principles of Queues and Closing Thoughts
Zach summarizes the core lessons of the talk into three operational principles for building resilient queue-driven software:
1. **Unbounded queues do not exist.** Any seemingly unbounded queue is physically bounded by host memory or another shared resource, and crossing that hidden boundary crashes the entire application. 2. **Handle overload at the edges, preserve backpressure in the middle.** Application boundaries must have explicit shedding, throttling, or backpressure policies for handling excess traffic. Conversely, internal, reusable components must never arbitrarily drop data; they should propagate backpressure upstream to allow higher-level callers to make informed trade-offs. 3. **Demand metrics everywhere.** Developers must insist on deep instrumentation across all stages of execution, continuously seeking visibility into internal queues, even when components attempt to obscure their operational state.
As Zach finishes, he reflects on the responsibility developers bear when designing systems:
> “And when we build an application it should always account for what happens when it receives too much data, but where we're not at the sort of boundaries of application, we're trying to write something which is more general or less opinionated or can be reused elsewhere, we should always use back pressure, right? We should always defer to someone else who knows better than us what to do.”
Concluding right as his allotted speaking time expires at 40 minutes, Zach invites the audience to continue the discussion and ask follow-up questions out in the hallway.
## Notable quotes > [17:42](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1062s) And wherever possible we should use backpressure, because that's the most neutral choice that we can make, right? That just kind of tells someone else, ‘I can't handle this data, you figure out what that means. — Zach Tellman
> [29:44](https://www.youtube.com/watch?v=1bNOO3xxMc0&t=1784s) There’s always something more to measure, but the idea is that we just don’t want to be sort of willfully blind and close our eyes and pretend that everything’s okay. — Zach Tellman
## Mentioned Zach Tellman, Clojure, core.async, Java, Amazon Web Services, Amazon S3, Elastic Load Balancer, Twitter, StatsD, UDP, HTTP, TCP, Hadoop, Maw, durable-queue, s3-journal, Aleph, Netty, Ring, Factual, Manifold, Graphviz
## Video description Queues, it seems, are back in fashion. This is exciting, as queues let us explore problem spaces that were unreachable by pure, immutable primitives.
But queues, and the execution models that drive them, are very different than what we're used to. Naive use of libraries like core.async will create brittle, unpredictable systems that fall over when we least expect.
This talk will cover the basics of queueing theory, and its practical application in Clojure systems large and small.
[Source](https://www.youtube.com/watch?v=1bNOO3xxMc0)