Rails 8's Solid Cable 4.1 Matches Redis Speed, Drops SQLite Latency

Ruby on Rails

By batching database writes and bypassing Action Cable's default worker pool, Solid Cable 4.1 allows relational databases to match Redis throughput under heavy load—and surpass it on SQLite—eliminating the need to maintain a separate Redis cluster for WebSockets.

AI-assisted summary of A deep dive into Solid Cable - Nick Pezza — original source. Check the original for full context.

Source published: 2026-10-01

Key ideas

Solid Cable eliminates the need for dedicated Redis clusters in Rails 8 by backing Action Cable with an Active Record polling architecture that works across SQLite, PostgreSQL, and MySQL. (1:49)
Solid Cable 4.1 introduces write batching via SQL insert_all and memoized configuration, allowing SQLite to hit 1,048 messages per second at 1,000 concurrent users and outperform Redis. (17:06)
An upstream Rails optimization bypassing Action Cable's worker pool thread queue for default stream handlers expanded broadcast handling capacity by approximately 35% across all adapters. (15:30)
Nick Pezza introduced ChronoCable, an experimental gem leveraging database persistence to guarantee strictly ordered, gap-recoverable Action Cable delivery for sequential streams like LLM tokens. (22:16)

Detailed notes

## Background Published on YouTube by the Ruby on Rails channel on October 1, 2026, this 26-minute talk features Nick Pezza, the author and maintainer of Solid Cable. According to the video description, Pezza breaks down how the database-backed Action Cable adapter operates under the hood, benchmarks it against Redis and PostgreSQL across SQLite, PostgreSQL, and MySQL, and shares the optimizations in Solid Cable 4.1 alongside a preview of an experimental gem called ChronoCable.

## TL;DR Nick Pezza explains how he built Solid Cable—originally in just 112 lines of code—to replace Redis with a database-neutral, Active Record polling adapter that ultimately became the Rails 8 default. After initial Grafana k6 benchmarks revealed high-concurrency bottlenecks, Pezza profiled the system and introduced major optimizations in Solid Cable 4.1, including memoized configurations, pre-hashed channel keys, batched database writes, and an upstream Rails patch that bypasses the Action Cable worker pool for default stream handlers. These improvements drastically flattened tail latency and nearly doubled MySQL and PostgreSQL throughput at 1,000 concurrent users, while pushing SQLite past Redis to reach up to 1,071 messages per second. Finally, Pezza previews ChronoCable, an experimental extension that leverages database persistence and client-side sequence tracking to guarantee strict in-order message delivery and automatic gap recovery for drops or out-of-order WebSocket frames.

## Atmosphere & style The talk carries a pragmatic, humble, and deeply engineering-focused tone, blending origin-story humor with rigorous performance analysis. Pezza guides the audience from the relatable frustration of managing extra infrastructure to a candid examination of benchmark failures—complete with a lighthearted nod to Dr. Ian Malcolm—before delivering a satisfying arc of profiling and architectural redemption. Watching the presentation feels like peering over the shoulder of a meticulous systems engineer who favors low-tech simplicity and transparent, data-driven problem solving.

## Bulletin - [01:49](https://www.youtube.com/watch?v=9WEABvHEKws&t=109s) Solid Cable eliminates the need for dedicated Redis clusters in Rails 8 by backing Action Cable with an Active Record polling architecture that works across SQLite, PostgreSQL, and MySQL. - [17:06](https://www.youtube.com/watch?v=9WEABvHEKws&t=1026s) Solid Cable 4.1 introduces write batching via SQL insert_all and memoized configuration, allowing SQLite to hit 1,048 messages per second at 1,000 concurrent users and outperform Redis. - [15:30](https://www.youtube.com/watch?v=9WEABvHEKws&t=930s) An upstream Rails optimization bypassing Action Cable's worker pool thread queue for default stream handlers expanded broadcast handling capacity by approximately 35% across all adapters. - [22:16](https://www.youtube.com/watch?v=9WEABvHEKws&t=1336s) Nick Pezza introduced ChronoCable, an experimental gem leveraging database persistence to guarantee strictly ordered, gap-recoverable Action Cable delivery for sequential streams like LLM tokens.

## Key takeaways - [00:11](https://www.youtube.com/watch?v=9WEABvHEKws&t=11s) Before Rails 8, Action Cable forced trade-offs between single-process isolation (Async), secondary operational dependencies (Redis), or an 8 KB payload ceiling (PostgreSQL LISTEN/NOTIFY). - [01:49](https://www.youtube.com/watch?v=9WEABvHEKws&t=109s) Solid Cable replaces push-based pub/sub with low-tech relational database polling, allowing applications to remove Redis without engine-specific primitives. - [03:00](https://www.youtube.com/watch?v=9WEABvHEKws&t=180s) The adapter achieves high broadcast throughput by bypassing Active Record model instantiation, callbacks, and transactions in favor of direct database inserts. - [06:27](https://www.youtube.com/watch?v=9WEABvHEKws&t=387s) To prevent table bloat without dedicated cron jobs, Solid Cable uses probabilistic auto-trimming on writes, mathematically calibrated to prune roughly twice the incoming write volume. - [06:27](https://www.youtube.com/watch?v=9WEABvHEKws&t=387s) Channel names are hashed into 64-bit integers with SHA-256 to drastically shrink index size on disk and accelerate polling scans. - [07:40](https://www.youtube.com/watch?v=9WEABvHEKws&t=460s) Subscribers track isolated starting lines and cursors via a thread-safe Concurrent::Map, preventing race conditions where new listeners received historical broadcasts. - [09:21](https://www.youtube.com/watch?v=9WEABvHEKws&t=561s) In burst-reconnect benchmarks ('Thundering Herd'), Solid Cable achieved parity with native push architectures across all major database engines. - [11:38](https://www.youtube.com/watch?v=9WEABvHEKws&t=698s) Profiling revealed that per-broadcast YAML re-parsing and per-poll channel re-hashing caused severe garbage collection spikes and CPU contention under high concurrency. - [14:08](https://www.youtube.com/watch?v=9WEABvHEKws&t=848s) Solid Cable 4.1 implements write batching using a dedicated ingestion thread that coalesces outgoing broadcasts into bulk insert_all statements based on size or timeout thresholds. - [15:30](https://www.youtube.com/watch?v=9WEABvHEKws&t=930s) Bypassing Action Cable's connection worker pool queue when streams use default handlers eliminated unnecessary thread context switches, flattening latency curves across all adapters. - [17:06](https://www.youtube.com/watch?v=9WEABvHEKws&t=1026s) At 1,000 concurrent users, Solid Cable 4.1 enabled SQLite to reach 1,048 msg/sec, while PostgreSQL reached 1,000 msg/sec and MySQL doubled its throughput to 961 msg/sec. - [18:57](https://www.youtube.com/watch?v=9WEABvHEKws&t=1137s) Decreasing the polling interval to 25 ms significantly slashes latency under moderate traffic, but aggressive polling hits diminishing returns on network-bound databases under saturation. - [20:45](https://www.youtube.com/watch?v=9WEABvHEKws&t=1245s) Auto-trimming expired records incurs minimal throughput penalties even under heavy loads, confirming it is an operationally safe default. - [22:16](https://www.youtube.com/watch?v=9WEABvHEKws&t=1336s) Action Cable does not guarantee sequential delivery across worker threads, creating chaotic UI rendering for ordered data streams like LLM token output. - [24:25](https://www.youtube.com/watch?v=9WEABvHEKws&t=1465s) ChronoCable enforces ordering by assigning channel-scoped sequence numbers on the server and buffering out-of-order payloads in the browser until sequence gaps are resolved. - [25:52](https://www.youtube.com/watch?v=9WEABvHEKws&t=1552s) Upon network disconnect and reconnect, ChronoCable detects missed sequence IDs via subscription confirmation payloads and fetches missing historical messages from the database.

## Detailed walkthrough ### [00:11](https://www.youtube.com/watch?v=9WEABvHEKws&t=11s) The Landscape of Action Cable Adapters Before Rails 8

Prior to Rails 8, Action Cable relied on three primary adapters to solve the core challenge of real-time communication: Async, Redis, and PostgreSQL. When a Rails application broadcasts a message, the adapter is responsible for delivering that payload across application processes so that WebSockets can transmit it down to connected browsers. The defining distinction between these adapters comes down to where messages can travel. The Async adapter is by far the simplest, requiring zero external dependencies, but it only functions within a single Rails process. While that works well in local development, production Rails applications are distributed across multiple processes and machines. Nick Pezza compares Async to "talking inside of a single room. Everyone in the room can hear you, but that message can't leave the room."

To break out of that single room, Redis and PostgreSQL offer shared communication mediums where one process publishes a message, the data store fans it out, and any listening process catches it. However, both present distinct operational trade-offs:

- **Redis**: Deploying Redis means provisioning, monitoring, and maintaining an entirely separate data store alongside the primary relational database. - **PostgreSQL**: Works out of the box if the application already runs on Postgres, but its native `LISTEN`/`NOTIFY` pub/sub mechanism enforces a payload ceiling of roughly 8 kilobytes. This limit creates real problems when broadcasting large Turbo HTML morphs or fragments. - **SQLite and MySQL**: Running an auxiliary PostgreSQL instance purely to provide Action Cable pub/sub makes little operational sense, leaving Redis as the begrudging default choice for non-Postgres stacks.

---

[01:49](https://www.youtube.com/watch?v=9WEABvHEKws&t=109s) The Spark: Killing Redis with Active Record

At his day job, Nick Pezza found himself unexpectedly tasked with managing his company's entire infrastructure. Eager to simplify operations, he audited their tech stack where Redis was being used for only two things: caching and Action Cable. He quickly migrated application caching over to Solid Cache, leaving Action Cable as the sole remaining dependency anchoring Redis to their servers.

Around that time, David Heinemeier Hansson (DHH) filed an issue in Rails regarding the extraction of an Active Record-backed Action Cable adapter from 37signals' Campfire. Nick assumed he could simply wait for Campfire's launch, drop in the extracted adapter, and decommission Redis. That hope vanished when a follow-up comment revealed Campfire had chosen to stick with Redis for Action Cable after all. Undeterred, Nick asked himself: "How hard can it be to implement an adapter that only uses Active Record?"

```mermaid flowchart TD accTitle: Push vs Poll in Action Cable Adapters accDescr: Comparison between push-based adapters like Redis/Postgres and pull-based polling in Solid Cable.

Unlike Redis and PostgreSQL, which maintain persistent open connections to push notifications immediately to Rails workers, an Active Record adapter aiming for full database neutrality cannot depend on engine-specific primitives like Postgres `LISTEN`/`NOTIFY`. Following the philosophy of Solid Queue, Nick embraced low-tech simplicity: polling. The system runs an infinite loop querying the relational database for newly arrived records, sleeps for a handful of milliseconds, and checks again.

---

[03:00](https://www.youtube.com/watch?v=9WEABvHEKws&t=180s) Architecture and Core Methods: Adapter and Listener

At the foundation of Solid Cable is the `Messages` Active Record model, which stores a `channel` string and a `payload`, with an index placed directly on `channel` to facilitate polling lookups. The adapter architecture is split into two classes:

1. **`Adapter`**: Inherits from Action Cable's base adapter, managing high-level adapter state and defining the public contract (`broadcast`, `subscribe`, `unsubscribe`, and `listener`). 2. **`Listener`**: Inherits from Action Cable's `SubscriberMap`, managing channel subscriptions, maintaining the list of watched channels, and running the background polling thread.

```mermaid flowchart LR accTitle: Solid Cable Class Hierarchy and Responsibilities accDescr: Class relationships and responsibilities between Adapter, Listener, and Action Cable base classes.

subgraph BaseClasses["Action Cable Base"] AC_Base["ActionCable::SubscriptionAdapter::Base"] AC_SubMap["ActionCable::SubscriptionAdapter::SubscriberMap"] end

subgraph SolidCableClasses["Solid Cable Implementation"] SC_Adapter["SolidCable::Adapter"] SC_Listener["SolidCable::Listener"] end

AC_Base --> SC_Adapter AC_SubMap --> SC_Listener SC_Adapter -->|"delegates to"| SC_Listener ```

The `Adapter` methods work through direct, lightweight mechanisms: - **`broadcast(channel, payload)`**: Bypasses full model instantiation, validations, callbacks, and automatic transaction wrapping by performing a direct Active Record `insert`. This ensures broadcast writes are as cheap and fast as possible. - **`subscribe(channel, subscriber)`**: Calls `listener.add_subscriber(channel, subscriber)`. That method is implemented on the `SubscriberMap` superclass, which stores subscribers in an internal hash guarded by a mutex to prevent race conditions. If the incoming subscriber is the first for that channel, it invokes `add_channel`, which simply appends the channel to an internal watched list. In Redis or Postgres, this is the exact spot where a `SUBSCRIBE` or `LISTEN` command would be executed. - **`unsubscribe(channel, subscriber)`**: Delegates to `remove_subscriber` on the `SubscriberMap`. When the last subscriber departs, it calls `remove_channel` to take the channel off the watched list, mirroring where Redis calls `UNSUBSCRIBE` and Postgres issues `UNLISTEN`.

---

[05:23](https://www.youtube.com/watch?v=9WEABvHEKws&t=323s) The Polling Loop and Version One Simplicity

When the `Listener` object initializes, it spins up a background thread that executes the main `listen` method. The loop runs continuously until shutdown:

```mermaid flowchart TD accTitle: Listener Polling Loop accDescr: Flow diagram showing how the listener thread checks for messages, updates IDs, broadcasts, and sleeps.

Start(["Start Listener Loop"]) --> Query["Query DB for messages WHERE channel IN (watched) AND id > last_seen_id"] Query --> Check{"New Messages Found?"} Check -- Yes --> Fanout["Broadcast each message to connected WebSockets"] Fanout --> UpdateID["Update last_seen_id to highest processed ID"] UpdateID --> Sleep["Sleep poll_interval (Default: 100ms)"] Check -- No --> Sleep Sleep --> Start ```

On every iteration, the thread queries the messages table for any entries matching watched channels whose IDs are greater than the last processed ID. It immediately broadcasts each match down to the registered subscribers and advances `last_id`. To prevent replaying historical data on boot, `last_id` was initially lazily initialized to the newest message ID present in the database. After processing, the thread sleeps for a configurable duration—defaulting to 100 milliseconds—before cycling again.

Nick notes that Version 1 of Solid Cable consisted of only about 112 lines of code across the adapter and listener, alongside the `Message` model and minor configuration parsing. Deployed into production at his company, it immediately permitted the retirement of their Redis cluster without incident.

---

[06:27](https://www.youtube.com/watch?v=9WEABvHEKws&t=387s) The Road to Rails 8: Auto-Trimming and Channel Hashing

A few months after Solid Cable 1.0 shipped, Nick received an unexpected email from DHH proposing that Solid Cable become the default Action Cable adapter in Rails 8. However, making it a framework default required solving a major operational hurdle: table bloat. Because broadcast messages become obsolete once delivered, keeping them forever degrades query performance over time.

To resolve this without introducing background cron workers, Solid Cable adopted probabilistic auto-trimming tied directly to write traffic: - Every time a broadcast occurs, if `auto_trimming` is active, the system runs a probabilistic check. - Writes have a small chance of triggering a trim query that deletes a batch of records exceeding the configurable retention period. - The probability is mathematically calibrated so that cleanup capacity averages roughly twice the volume of incoming writes, exerting consistent downward pressure on table growth as traffic surges.

Additionally, engineers at 37signals suggested avoiding large text indexes on the `channel` column. Solid Cable was updated to run channels through SHA-256 and store the result as a fixed-size integer (`channel_hash`). Indexing and querying this 64-bit integer drastically reduced index size on disk and accelerated database index scans during high-frequency polling.

---

[07:40](https://www.youtube.com/watch?v=9WEABvHEKws&t=460s) Fixing Subscriber Race Conditions and Channel Starting Lines

Real-world adoption surfaced an edge-case bug in message targeting. Originally, Solid Cable maintained a single global `last_id` across the entire listener. If a client subscribed to a brand-new channel while the listener was already running, historical messages for that channel might still reside in the database with IDs greater than the listener's historical cursor, causing the new subscriber to erroneously receive stale broadcasts.

To eliminate this bug: - **Per-channel cursors**: Solid Cable tracks an independent `last_seen_id` for every active channel subscription. When a channel is registered via `add_channel`, it records the current maximum message ID, establishing that subscription's unique starting line. - **Concurrency handling**: Because channels are added and removed by request threads while the listener polling loop runs simultaneously on a background thread, the channel state was migrated into a `Concurrent::Map` using atomic operations to prevent race conditions. - **Eager startup**: Solid Cable retains a global cursor to scan forward through the table efficiently, but eagerly initializes it at startup to guarantee no messages slipping between listener boot and the initial query pass. Before dispatching any message, the listener verifies the payload ID against the specific channel's `last_seen_id`.

[08:38](https://www.youtube.com/watch?v=9WEABvHEKws&t=518s) Benchmarking Setup: Solid Cable 4.0 vs. Redis and Native Postgres

The global message ID keeps polling queries lean and guarantees that startup remains safe, while the per-channel message ID ensures subscribers strictly receive messages broadcast after their specific channel was registered. To determine how Solid Cable stacks up against traditional push-based solutions, a comprehensive benchmarking suite was executed against Solid Cable 4.0 and Rails `main` (pinned to a July 31st commit). All benchmarks ran on a single machine provisioned with two virtual CPUs and 8 GB of RAM. Hosting Redis, MySQL, and PostgreSQL locally on the same box eliminated external network variance. Solid Cable operated with a default polling interval of 100 milliseconds, and automatic message trimming was explicitly disabled to measure raw adapter performance without maintenance interference. Load generation and metric collection were handled via Grafana k6, with all benchmark applications and test scripts made available in the `bench` directory of the Solid Cable repository.

- **Hardware & Environment**: 2 vCPUs, 8 GB RAM; Redis, MySQL, and PostgreSQL colocated on the test box. - **Software Tested**: Solid Cable 4.0, Rails `main` (commit dated July 31). - **Solid Cable Tuning**: 100 ms polling interval; auto-trimming disabled to avoid background prune overhead. - **Tooling**: Grafana k6 for scenario simulation and metric capture.

---

[09:21](https://www.youtube.com/watch?v=9WEABvHEKws&t=561s) Constant Load and Thundering Herd Benchmarks

The first evaluation simulates constant load by sustaining a fixed population of active users over a 5-minute run. Each user connects, sends a ping, waits for the corresponding polling response message, repeats that cycle 20 times, and disconnects, at which point a replacement user immediately spawns into the pool. Throughput represents the number of full-trip round-trip messages per second. At 250 active users, performance across adapters is tightly grouped near 280 messages per second. When concurrency climbs toward saturation, Redis and the native PostgreSQL adapter scale up linearly. SQLite paired with Solid Cable holds up admirably, but Solid Cable running on PostgreSQL—and especially MySQL—plateaus around 500 users.

Latency exhibits the inverse relationship: round-trip times remain solid up to 500 users, but at 750 and 1,000 users, response times balloon across all adapters. Under heavy saturation, Solid Cable's latency degrades significantly faster (particularly against MySQL), while Redis and the native PostgreSQL adapter restrain their latency far longer.

The "Thundering Herd" test tests burst concurrency: a fixed block of users connects simultaneously, subscribes, transmits a single ping, verifies the receipt of the pong, and terminates the connection. This emulates a post-deployment restart when every connected client must reconnect concurrently. Completion speed is evaluated via round-trip latency and total elapsed duration.

| User Concurrency | Redis | Native Postgres | Solid Cable (Postgres) | Solid Cable (SQLite) | Solid Cable (MySQL) | |:--- |:--- |:--- |:--- |:--- |:--- | | **250 users** | ~2.0s | ~2.0s | ~2.0s | ~2.0s | ~2.0s | | **500 users** | ~3.0s | ~3.0s | ~3.0s | ~3.0s | ~3.0s | | **1,000 users** | 5.2s | ~5.0s | 5.7s | 5.2s | ~6.0s |

Even under an aggressive reconnect spike, Solid Cable completes the herd storm in essentially the same ballpark as native push architectures.

---

[11:38](https://www.youtube.com/watch?v=9WEABvHEKws&t=698s) The Lecture Benchmark: Fanout Limits and Hotspots

The "Lecture" scenario evaluates broadcast fan-out efficiency: a swarm of clients subscribes to a single shared channel while a producer broadcasts 7 messages per second continuously for 5 minutes. At 250 subscribers, Redis and native PostgreSQL report an average latency of approximately 70 milliseconds, whereas Solid Cable averages around 120 milliseconds. The 50 ms difference closely matches Solid Cable's 100 ms polling window. At 500 subscribers, Redis and PostgreSQL average 90 milliseconds, while Solid Cable floats near 145 milliseconds.

By 750 subscribers, the system hits extreme friction. Redis and native PostgreSQL retain a 95th-percentile (P95) latency below 200 milliseconds, whereas Solid Cable surges to 400–500 milliseconds. Pushing the delivery frequency to 10 messages per second at 750 subscribers caused *every* adapter—including Redis and native PostgreSQL—to fail the benchmark entirely.

Looking at these degradation curves and searching for a positive interpretation brought to mind "the immortal words of the great Dr. Ian Malcolm: We can do better." Detailed profiling uncovered several distinct performance sinks:

1. **YAML Re-parsing & Config Allocations**: On every broadcast, Solid Cable checked whether auto-trimming was active; on every polling pass, it inspected the polling interval and logging silence settings. Each check was reading the `cable.yml` file, re-parsing YAML structures, and allocating throwaway objects, causing garbage collection spikes. The fix memoized the configuration into a dedicated configuration object parsed once on boot. 2. **Channel Key Re-hashing**: Previously, the watcher stored the raw string channel name in its tracking map and re-hashed every channel key on every polling iteration. The optimization hashes the channel string once upon subscription and keys the map by hash, permitting the polling thread to inject the hashes directly into the database query without repeating work.

---

[14:08](https://www.youtube.com/watch?v=9WEABvHEKws&t=848s) Write Batching in Solid Cable

Under high broadcast rates, inserting every single broadcast individually generated intense write contention on the database engine. Because Solid Cable already batched *reads* via poll cycles, the write path was refactored to apply that same batching strategy. The new broadcaster batches outgoing messages based on two boundaries: a maximum batch size threshold or a maximum elapsed delay interval, whichever boundary is hit first.

```mermaid flowchart TD accTitle: Solid Cable Batched Broadcaster Flow accDescr: Incoming broadcasts enter a thread-safe queue where a worker thread drains them into batches and inserts them via insert_all.

subgraph Ingestion ["Ingestion"] A["broadcast(channel, payload)"] --> B["Wrap in Message Object"] B --> C["Push to Ingestion Queue"] end

subgraph BatchWorker ["Broadcaster Thread"] C --> D["Queue.pop (Blocks until first message)"] D --> E["Drain pending items from queue"] E --> F{"Batch size met OR timeout reached?"} F -- "Yes" --> G["SolidCable::Message.insert_all(...)"] F -- "No" --> E end ```

The broadcaster launches a dedicated background thread monitoring the ingestion queue. The thread blocks on `pop` until a message arrives, avoiding wasteful CPU spinning. Once awakened, it drains all accumulated queue items, accumulates messages up to the batch limit or delay deadline, and issues a single SQL `insert_all` command.

---

[15:30](https://www.youtube.com/watch?v=9WEABvHEKws&t=930s) Action Cable Worker Pool Bypass and Benchmarking 4.1

When the lecture test crumbled at 10 messages per second across all adapters, the common point of failure pointed to Rails core internals rather than database drivers. When Action Cable registers a stream via `stream_from`, it builds a stream handler. In the absence of a custom application callback block, Action Cable defaults to a standard handler that merely relays the raw broadcast buffer over the WebSocket connection.

Historically, this basic default handler was wrapped inside a `WorkerPoolStreamHandler`. Every broadcast had to hop onto the connection worker thread pool before hitting the socket. While this worker queue prevents custom Ruby callbacks from blocking Action Cable's event loop executor, it added a pointless thread context switch and queue traversal for the default case. The upstream patch circumvents the worker pool entirely when no user callback block is supplied, dispatching messages straight to the socket.

```mermaid flowchart LR accTitle: Stream Handler Routing Optimization accDescr: Action Cable avoids routing default broadcasts through the worker pool thread queue.

subgraph Legacy ["Pre-Optimization"] L1["Broadcast Received"] --> L2["Worker Pool Thread Queue"] L2 --> L3["WebSocket Frame Sent"] end

subgraph Optimized ["Rails Upstream Fix"] O1["Broadcast Received"] --> O2{"Has Custom Callback?"} O2 -- "Yes" --> O3["Worker Pool Queue"] O2 -- "No (Default)" --> O4["Direct WebSocket Send"] end ```

This optimization, merged into Rails core, cleared the broadcast congestion bottleneck. Solid Cable was re-evaluated under these updates with the batch broadcaster tuned to a batch size of 4 and a 1-millisecond delay. (The native PostgreSQL adapter was omitted from the subsequent charts, as its behavior tracked Redis near-identically, and Thundering Herd was omitted due to its brief, bursty nature showing negligible change.) At 250 users under constant load, throughput numbers revealed significant improvements.

[17:06](https://www.youtube.com/watch?v=9WEABvHEKws&t=1026s) Benchmark Gains Across Adapters and Action Cable Fixes

At the lower end of the spectrum with modest user counts, the recent optimizations leave performance basically unchanged because traffic levels simply are not high enough for the changes to show a noticeable difference. Once the test scales to 500 and 750 users, however, the performance gains become readily apparent. Both Postgres and MySQL close much of their historical performance gap with SQLite, and all three database adapters advance significantly closer to Redis. At the 1,000-user mark, the leap is even more dramatic: SQLite actually edges ahead of Redis, delivering 1,048 messages per second. PostgreSQL surges from 609 to 1,000 messages per second, while MySQL nearly doubles its throughput, jumping from 491 to 961 messages per second.

The latency improvements tell an even more compelling story. Prior to these architectural optimizations, Solid Cable's latency rapidly deteriorated as user counts climbed into higher tiers. With the new changes in place, the latency curve remains dramatically flatter between 500 and 1,000 concurrent users. MySQL’s P95 latency drops from 412 milliseconds down to 174 milliseconds at 500 users, and tumbles from 953 milliseconds to 228 milliseconds at 750 users. PostgreSQL follows an identical trajectory, falling from 668 milliseconds down to 231 milliseconds at 750 users. At 1,000 users, SQLite reduces its average latency from 390 milliseconds to 140 milliseconds, PostgreSQL drops from 477 milliseconds to 179 milliseconds, and MySQL plummets from well over a second to just 210 milliseconds. Tail latency emerges as the standout metric: SQLite posts a substantially better P95 than Redis at both 750 and 1,000 users, translating into superior throughput, while PostgreSQL and MySQL move from trailing far behind to essentially matching Redis at 1,000 users.

The "lecture" benchmark—simulating one publisher broadcasting to an expanding audience of listeners—demonstrates solid improvements across the board as subscriber counts rise. Instead of latency running away uncontrollably at the higher end, the response curve stays remarkably flat. Crucially, all adapters now successfully survive the stress test of 750 users at 10 messages per second, a benchmark run that previously crashed and failed. This represents an approximate 35% increase in total message handling capacity. Because these specific gains stem primarily from stream handler refactoring inside Action Cable itself rather than adapter-specific logic, the benefits propagate across every supported adapter.

- **Throughput at 1,000 users**: SQLite reached 1,048 msg/sec (surpassing Redis); PostgreSQL rose from 609 to 1,000 msg/sec; MySQL jumped from 491 to 961 msg/sec. - **P95 Latency at 750 users**: MySQL dropped from 953 ms to 228 ms; PostgreSQL dropped from 668 ms to 231 ms. - **Latency at 1,000 users**: MySQL dropped from >1,000 ms to 210 ms (P95); PostgreSQL dropped from 477 ms to 179 ms (P95); SQLite dropped from 390 ms to 140 ms (average). - **Lecture benchmark**: All adapters passed the previously failing 750-user / 10 msg/sec tier (~35% throughput increase).

---

[18:57](https://www.youtube.com/watch?v=9WEABvHEKws&t=1137s) Tuning Knobs: Polling Interval and Latency Trade-Offs

Solid Cable provides several configuration settings to fine-tune operational behavior. Evaluating these settings using the constant load test—which offers the cleanest composite view of throughput versus latency—reveals how database polling impacts delivery speeds. By default, Solid Cable uses a polling interval of 100 milliseconds. Slashing that interval down to 25 milliseconds increases database query frequency by a factor of four, but provides the system with opportunities to discover and process newly broadcast messages significantly earlier.

The benchmark data confirms an immediate latency reduction through 500 users and stretching close to 750 users. At 250 users, Solid Cable’s P95 latency drops from roughly 160 milliseconds down to approximately 90 milliseconds, and at 500 users it hovers around 115 milliseconds—not quite matching Redis’s instantaneous delivery, but closing the distance. However, once the test reaches saturation at 750 and 1,000 users, issuing reads four times as often creates diminishing returns for network-bound databases like PostgreSQL and MySQL. SQLite, by contrast, avoids network overhead and "keeps stretching its legs," widening its lead. Pushing SQLite down to an aggressive 10-millisecond polling interval at 250 users yields a 65-millisecond average latency and a 78-millisecond P95.

Throughput mirrors this pattern: decreasing the polling interval supplies a noticeable uplift across most Solid Cable adapters. SQLite benefits the most, accelerating up to 1,071 messages per second at 1,000 users. PostgreSQL and MySQL demonstrate modest throughput gains through 750 users, though MySQL surrenders a bit of throughput at the absolute top tier due to query saturation. The takeaway for production environments is clear: when traffic is light to moderate and delivery latency is critical, lowering the polling interval yields tangible gains. However, when a system operates near saturation, hyper-aggressive polling works against network databases. There is no single universal setting; tuning must balance specific latency requirements against the database engine's headroom.

- **25 ms Polling Interval Latency**: At 250 users, P95 dropped from ~160 ms to ~90 ms; at 500 users, P95 hit ~115 ms. - **SQLite at 10 ms Polling Interval (250 users)**: Achieved a 65 ms average latency and a 78 ms P95. - **SQLite Maximum Throughput**: Reached 1,071 msg/sec at 1,000 users under a 25 ms polling interval.

---

[20:45](https://www.youtube.com/watch?v=9WEABvHEKws&t=1245s) The Cost of Auto-Trimming and Solid Cable 4.1

Another core configuration knob is auto-trimming, which purges expired records from the messages table. Testing the overhead of background pruning involves configuring a trim batch size of 100 messages and a retention period of 30 seconds across a five-minute constant load test, keeping the default 100-millisecond polling interval so only one variable changes at a time. Under this setup, background worker threads actively prune expired records for the bulk of the test duration.

At 250 and 500 concurrent users, auto-trimming imposes zero noticeable penalty—it is effectively free. Even under a heavy load of 1,000 users, the throughput penalty remains negligible. SQLite throughput is virtually identical with or without trimming enabled. PostgreSQL sees a slight dip from 1,000 down to 956 messages per second, while MySQL moves from 961 to 942 messages per second. Latency absorbs a somewhat larger relative hit on PostgreSQL and MySQL, but the overall cost of keeping the database table lean remains modest. Unless an application is already running on the ragged edge of database hardware capacity, auto-trimming represents a sensible operational trade-off, which is why it is enabled by default.

Solid Cable delivers a simpler operational model by utilizing an existing relational database, eliminating the need to maintain a separate Redis cluster while remaining surprisingly close in performance under heavy load—or outperforming it in the case of SQLite. The only real architectural compromise appears at very low traffic levels: idle or lightly loaded Redis and native PostgreSQL adapters push messages instantly over long-lived sockets, beating Solid Cable’s polling pause. But for the vast majority of Rails applications, the primary database is more than capable of supporting Action Cable traffic directly. All performance improvements, batched writes, and latency fixes discussed are packaged and available in the Solid Cable 4.1 release.

- **Auto-trimming settings**: Tested with a 100-message batch size and a 30-second retention window. - **Throughput cost at 1,000 users**: PostgreSQL fell slightly from 1,000 to 956 msg/sec; MySQL shifted from 961 to 942 msg/sec; SQLite saw virtually no change. - **Solid Cable 4.1**: Released with all highlighted throughput optimizations and Action Cable stream-handling updates.

---

[22:16](https://www.youtube.com/watch?v=9WEABvHEKws&t=1336s) Solving Out-of-Order Messages: Introducing ChronoCable

Beyond performance tuning, Action Cable exhibits a structural limitation: it does not guarantee in-order message delivery. Action Cable hands incoming broadcast payloads off to a worker thread pool. Depending on operating system thread scheduling and execution timing, two messages broadcast in sequence `A` then `B` can easily be delivered over the WebSocket connection in reverse sequence `B` then `A`. For casual UI updates or notifications, transient misordering is harmless. But for modern reactive patterns—such as streaming tokens from a Large Language Model (LLM)—delivery order is vital. If text chunks arrive out of sequence, the resulting text renders incorrectly. A live demonstration firing a Turbo Stream that rapidly appends incrementing integer values exposes the failure mode vividly: numbers scramble on screen in "absolute chaos."

```mermaid flowchart TD accTitle: ChronoCable Message Sequencing Flow accDescr: Sequence assignment on broadcast and sequential verification on the client.

subgraph Server["Server Broadcast"] A["Broadcast Triggered"] --> B{"Ordered Stream?"} B -- "No" --> C["Normal Queue Storage"] B -- "Yes" --> D["Lock Channel Record"] D --> E["Increment Sequence ID"] E --> F["Attach ID to Message & Queue"] end

subgraph Client["Client Stream Processing"] G["Receive Message"] --> H{"Matches Expected ID?"} H -- "Yes" --> I["Consume Message & Increment Expected"] H -- "No (Ahead)" --> J["Buffer Message in Waiting Queue"] J --> K["Request History from Server"] end

F -.->|"WebSocket Delivery"| G ```

This ordering problem plays directly into Solid Cable’s strengths. Because Solid Cable persists every broadcast in relational database tables, sequence information can be saved alongside the payload. By assigning each message an auto-incrementing sequence number scoped to its specific channel, the system captures sufficient state to ensure clients consume payloads in the exact order they were broadcast.

To implement this model, Nick Pezza introduced ChronoCable, an experimental gem consisting of monkey patches for Action Cable and Solid Cable designed to enforce strict channel ordering and recover lost messages over flaky connections. ChronoCable introduces a `Channel` model that stores the channel hash, tracks the current integer sequence number, and marks whether ordered delivery is active. Developers can opt into ordered streaming via a new `deliver_in_order` parameter:

```ruby # Action Cable Channel stream_from "chat_room_1", deliver_in_order: true ```

When integrating with Hotwire and Turbo, the same flag can be passed down as a data attribute on a standard Turbo Stream tag:

```erb <%= turbo_stream_from "chat_room_1", data: { deliver_in_order: true } %> ```

Under the hood, ChronoCable intercepts every broadcast to check if the destination channel requires strict ordering. If enabled, it locks the corresponding channel row, increments the sequence counter, assigns that channel-scoped identifier to the message, and queues it for insertion into the database table, which now includes a `channel_id` sequence column. The architectural trade-off is throughput: row locking and individual sequence assignment cut into the write-batching gains introduced in Solid Cable 4.1.

---

[24:25](https://www.youtube.com/watch?v=9WEABvHEKws&t=1465s) ChronoCable Client Protocol and Gap Recovery

Because strict ordering cannot be solved on the backend alone, ChronoCable offloads sequencing logic to the browser, requiring the client-side JavaScript runtime to track local state for every active ordered stream. The connection handshake proceeds normally: the client transmits a standard subscription request containing the requested channel name. Upon confirmation, the server appends the stream's current sequence ID to the confirmation payload. If the server indicates that the latest sequence ID on the stream was 1, the browser initializes its internal state to expect sequence ID 2 as the next inbound message.

When messages arrive over the WebSocket, they carry their stream identifier and assigned sequence ID. If the sequence matches the client's expected integer counter, the browser immediately consumes the payload (e.g., rendering the Turbo Stream append) and advances its internal expected sequence counter by one.

If network jitter or thread scheduling causes a message to arrive ahead of sequence—such as message 5 arriving when the browser is waiting for message 3—the client pauses processing. It stores message 5 in an in-memory buffer (the waiting queue) without rendering it, and instantly dispatches a message back to the server requesting the stream’s history, asking for all missing messages starting from the last successfully rendered ID.

While awaiting the server's history response, any additional out-of-order broadcasts that arrive are placed into the waiting queue. Once the server returns the requested history payload, the client catches up through the following sequence:

1. It consumes the missing historical messages in strict order (e.g., sequence 3, then sequence 4), advancing its expected counter as it goes. 2. It processes the buffered messages in the waiting queue. 3. If an item in the waiting queue was already provided by the history response (such as sequence 4), the client discards the duplicate. 4. If a buffered message represents the next valid unseen message in sequence (such as sequence 5), it is consumed and the expected sequence advances. 5. Once the waiting queue is drained, normal real-time stream listening resumes seamlessly.

[25:52](https://www.youtube.com/watch?v=9WEABvHEKws&t=1552s) Recovering Missed Messages After a Connection Drop in ChronoCable

Nick walks through how ChronoCable handles network interruptions and ensures ordered, recoverable delivery when a client's internet connection drops. When the browser comes back online, reconnects, and resubscribes to the channel, the server's subscription confirmation includes the current state of the stream. In Nick's example, the confirmation tells the client that the stream is currently on message 7, whereas the browser was still expecting message 6. That discrepancy immediately alerts the client that messages were missed while the connection was down.

To recover, the browser requests the stream history from the server starting from the last message it successfully saw. Using that historical payload to catch back up, the client consumes message 6 and then message 7 strictly in order, advances its internal pointer for the next expected message, and seamlessly resumes normal real-time streaming.

```mermaid flowchart LR accTitle: ChronoCable Reconnection and Message Recovery Flow accDescr: Shows how a browser detects a gap between expected message 6 and current stream message 7 upon reconnecting, fetches stream history, and resumes normal streaming.

subgraph Reconnect["1. Reconnect & Detect Gap"] A["Internet drops & browser reconnects"] --> B["Subscription confirmation: stream at msg 7"] B --> C["Browser expects msg 6 (gap detected)"] end

subgraph CatchUp["2. Fetch History & Resume"] D["Request stream history from last seen msg"] --> E["Consume msg 6 then msg 7 in order"] E --> F["Advance expected pointer & resume streaming"] end

C --> D ```

[26:18](https://www.youtube.com/watch?v=9WEABvHEKws&t=1578s) Closing Thoughts and Repository Links

Wrapping up his preview of ChronoCable, Nick encourages anyone working with Action Cable channels where strict message ordering really matters to "give it a whirl and let me know." He concludes the presentation by sharing the repository links on screen and inviting the community to check out the projects, open issues, and share feedback.

## Notable quotes > [00:11](https://www.youtube.com/watch?v=9WEABvHEKws&t=11s) talking inside of a single room. Everyone in the room can hear you, but that message can't leave the room. — Nick Pezza

> [01:49](https://www.youtube.com/watch?v=9WEABvHEKws&t=109s) How hard can it be to implement an adapter that only uses Active Record? — Nick Pezza

> [11:38](https://www.youtube.com/watch?v=9WEABvHEKws&t=698s) the immortal words of the great Dr. Ian Malcolm: We can do better. — Nick Pezza

> [18:57](https://www.youtube.com/watch?v=9WEABvHEKws&t=1137s) keeps stretching its legs — Nick Pezza

> [18:57](https://www.youtube.com/watch?v=9WEABvHEKws&t=1137s) There is no single universal setting; tuning must balance specific latency requirements against the database engine's headroom. — Nick Pezza

> [20:45](https://www.youtube.com/watch?v=9WEABvHEKws&t=1245s) Unless an application is already running on the ragged edge of database hardware capacity, auto-trimming represents a sensible operational trade-off, which is why it is enabled by default. — Nick Pezza

> [22:16](https://www.youtube.com/watch?v=9WEABvHEKws&t=1336s) Action Cable exhibits a structural limitation: it does not guarantee in-order message delivery. — Nick Pezza

> [22:16](https://www.youtube.com/watch?v=9WEABvHEKws&t=1336s) absolute chaos — Nick Pezza

> [26:18](https://www.youtube.com/watch?v=9WEABvHEKws&t=1578s) give it a whirl and let me know. — Nick Pezza

## Mentioned Nick Pezza, David Heinemeier Hansson (DHH), Dr. Ian Malcolm, Ruby on Rails, Shopify, 37signals, Campfire, Solid Cable, Action Cable, ChronoCable, Solid Cache, Solid Queue, Active Record, Redis, PostgreSQL, SQLite, MySQL, Grafana k6, Hotwire, Turbo, WebSockets

## Video description Nick Pezza, who wrote and maintains Solid Cable, walks through how the database-backed Action Cable adapter works under the hood. He then benchmarks it against Redis and Postgres on SQLite, Postgres and MySQL.

Nick shares the optimizations in Solid Cable 4.1, including batched writes and a fix inside Action Cable itself, that bring it close to Redis and sometimes past it. He also previews ChronoCable, an experimental gem for ordered, recoverable message delivery.

***

Thank you Shopify for sponsoring the editing and post-production of these videos. Check out insights from the Engineering team at: https://shopify.engineering/

#RailsWorld #RubyOnRails #SolidCable #ActionCable #WebSockets #Performance

[Source](https://www.youtube.com/watch?v=9WEABvHEKws)

中文简报

Rails 8 的 Solid Cable 4.1 速度比肩 Redis，并降低 SQLite 延迟通过批量化数据库写入并绕过 Action Cable 的默认工作线程池，Solid Cable 4.1 让关系型数据库在高负载下的吞吐量能够匹敌 Redis——在 SQLite 上甚至实现超越——从而无需为 WebSocket 维护单独的 Redis 集群。 Nick Pezza 解释了他最初如何仅用 112 行代码构建 Solid Cable，以数据库中立的 Active Record 轮询适配器替代 Redis，并最终使其成为 Rails 8 的默认选项。在最初的 Grafana k6 基准测试暴露出高并发瓶颈后，Pezza 对系统进行了性能分析，并在 Solid Cable 4.1 中引入了多项重大优化，包括记忆化配置、预哈希通道键、批量数据库写入，以及一项针对默认流处理器绕过 Action Cable 工作线程池的上游 Rails 补丁。这些改进大幅平抑了尾部延迟，并在 1,000 个并发用户下使 MySQL 和 PostgreSQL 的吞吐量几乎翻倍，同时推动 SQLite 超越 Redis，达到每秒最高 1,071 条消息。最后，Pezza 预告了 ChronoCable，这是一个实验性扩展，利用数据库持久化和客户端序列追踪，为丢弃或乱序的 WebSocket 数据帧提供严格的按顺序消息传递保证及自动缺口恢复。 Solid Cable 通过支持 SQLite、PostgreSQL 和 MySQL 的 Active Record 轮询架构为 Action Cable 提供支持，消除了 Rails 8 对专用 Redis 集群的需求。 (1:49) Solid Cable 4.1 引入了基于 SQL insert_all 的批量写入以及记忆化配置，使 SQLite 在 1,000 个并发用户下达到每秒 1,048 条消息，性能超越 Redis。 (17:06) 一项针对默认流处理器绕过 Action Cable 工作线程池线程队列的上游 Rails 优化，使所有适配器的广播处理能力提升了约 35%。 (15:30) Nick Pezza 介绍了 ChronoCable，这是一个利用数据库持久化为 LLM token 等时序流保证 Action Cable 严格有序且可恢复缺口传递的实验性 gem。 (22:16)

Reading summary

Nick Pezza demonstrates that Solid Cable can eliminate external Redis dependencies in Rails applications by implementing database polling via Active Record, matching or surpassing traditional push adapters under heavy workloads when paired with core framework optimizations. Furthermore, persisting messages in relational tables enables extensions like ChronoCable to resolve Action Cable's longstanding lack of guaranteed message ordering. Database Polling and Table Maintenance Architecture (3:00) Unlike Redis or PostgreSQL LISTEN/NOTIFY, which rely on persistent push connections, Solid Cable achieves database engine neutrality through low-tech polling. Its listener thread continuously queries the messages table for records where channel hashes match registered subscriptions and primary keys exceed the last seen identifier. To maximize throughput, broadcast operations bypass Active Record validations, callbacks, and automatic transaction blocks by executing direct database inserts. Channel subscriptions are maintained in a thread-safe Concurrent::Map with per-channel cursor tracking to prevent race conditions and ensure new subscribers do not receive historical messages. Bottleneck Mitigation and Throughput Benchmarks (8:38) Initial benchmarks evaluated on a 2-vCPU, 8 GB RAM instance revealed that while Solid Cable matched Redis at low concurrency, high-frequency fan-out tests (such as 10 messages per second across 750 subscribers) caused latency spikes and adapter failures across all adapters. Profiling traced these bottlenecks to YAML configuration re-parsing, repeated channel key hashing during polling passes, write contention, and thread context switching inside Rails' default WorkerPoolStreamHandler. Optimizations implemented in Solid Cable 4.1 and Rails core resolved these bottlenecks by memoizing configuration, pre-hashing channel names into 64-bit integers, batching broadcast writes via a blocking queue with SQL insert_all, and bypassing the worker pool when stream callbacks are absent. Under constant load at 1,000 concurrent users, PostgreSQL throughput rose from 609 to 1,000 messages per second, MySQL climbed from 491 to 961 messages per second, and SQLite reached 1,048 messages per second—surpassing Redis. Latency curves flattened substantially, with SQLite and PostgreSQL achieving P95 tail latencies of 140 ms and 179 ms, respectively. Trade-offs in Polling Intervals and Pruning (18:57) Solid Cable defaults to a 100-millisecond polling interval, but reducing this window to 25 milliseconds lowers P95 latency at 250 users from approximately 160 ms to 90 ms. However, Pezza cautions that hyper-aggressive polling yields diminishing returns for network-bound databases like PostgreSQL and MySQL near hardware saturation. SQLite avoids network overhead entirely, achieving a 78 ms P95 latency at a 10 ms interval and peaking at 1,071 messages per second at 1,000 users. To prevent table bloat without background cron jobs, Solid Cable uses probabilistic auto-trimming linked to write operations, mathematically calibrated so cleanup capacity averages roughly twice the rate of incoming writes. In benchmark tests with a 100-message batch size and 30-second retention, auto-trimming imposed negligible throughput overhead: PostgreSQL experienced a minor drop from 1,000 to 956 messages per second at 1,000 users, MySQL slipped from 961 to 942 messages per second, and SQLite exhibited virtually identical throughput. Enforcing Message Ordering with ChronoCable (22:16) Standard Action Cable dispatches broadcasts through thread pools, meaning operating system scheduling can deliver payloads out of sequence—a major defect for streaming formats such as LLM text chunks. Pezza introduced ChronoCable, an experimental extension leveraging Solid Cable's relational storage to enforce message sequencing. When strict ordering is enabled, ChronoCable locks the channel database row during broadcast to increment and attach an auto-incrementing sequence integer to the payload, trading away write-batching throughput in exchange for ordering guarantees. The sequencing logic is enforced by client-side JavaScript. The browser compares incoming message sequence numbers against its internal counter; if a gap is detected due to network latency or disconnection, the client buffers subsequent messages in memory and requests historical messages from the server starting from the last verified ID. Once the server supplies the missing sequence, the browser renders the missed messages in order, drains its buffer, and seamlessly resumes real-time processing. Pezza concludes that Solid Cable 4.1 makes standard relational databases fully capable of handling Action Cable workloads for most Rails applications, trading sub-millisecond idle delivery for reduced operational infrastructure. While ChronoCable demonstrates that persistent database storage can solve message ordering issues, its reliance on channel row locking introduces throughput trade-offs that developers must balance against their sequencing requirements.

Source quotes

[0:11] Nick Pezza: talking inside of a single room. Everyone in the room can hear you, but that message can't leave the room. [1:49] Nick Pezza: How hard can it be to implement an adapter that only uses Active Record? [11:38] Nick Pezza: the immortal words of the great Dr. Ian Malcolm: We can do better. [18:57] Nick Pezza: keeps stretching its legs [18:57] Nick Pezza: There is no single universal setting; tuning must balance specific latency requirements against the database engine's headroom. [20:45] Nick Pezza: Unless an application is already running on the ragged edge of database hardware capacity, auto-trimming represents a sensible operational trade-off, which is why it is enabled by default. [22:16] Nick Pezza: Action Cable exhibits a structural limitation: it does not guarantee in-order message delivery. [22:16] Nick Pezza: absolute chaos [26:18] Nick Pezza: give it a whirl and let me know.

Original transcript

[0:06] [music] [0:11] >> Hey there, I'm Nick Pezza. I wrote and [0:12] maintain solid cable and today we're [0:14] going to look at how it works and how it [0:16] compares to the other actual cable [0:17] adapters in rails. [0:20] Prior to rails eight, action cable [0:22] relied on one of three adapters, Redis, [0:24] Postgres and async. [0:26] They all solve the same basic problem. [0:27] When your application broadcasts a [0:28] message, that adapter's responsible for [0:30] getting that message to the rails [0:32] processes that need to know about it. [0:34] From there, those processes can send the [0:36] message down to the connected browsers [0:37] over their website connections. [0:39] The important difference between these [0:40] adapters is really about where those [0:42] messages are able to travel. [0:44] The async adapter is the simplest and [0:46] has no dependencies, but it only works [0:48] within a single rails process. Which [0:50] works great in development, but [0:51] production rails apps usually run across [0:53] multiple processes, often spread across [0:55] multiple machines. [0:57] Think of it like talking inside of a [0:58] single room. Everyone in the room can [1:00] hear you, but you that message can't [1:01] leave the room. [1:03] >> [snorts] [1:04] >> Redis and Postgres solve this by giving [1:05] your rails process a shared place to [1:07] communicate through. [1:08] One rails process publishes a message, [1:10] Redis or Postgres distributes it, and [1:12] every process that needs that message [1:14] can receive it. But they come with their [1:16] own trade-offs. [1:18] If you choose Redis, deploying your [1:19] rails app now meant learning how to [1:20] manage yet another data store alongside [1:23] your primary database. [1:25] If your app already used Postgres, the [1:27] Postgres adapter mostly just worked. [1:29] The trade-off is that its pub/sub [1:31] mechanism limits payloads to roughly 8 [1:33] kilobytes. [1:34] That's usually fine, but if you're using [1:35] turbo to broadcast larger chunks of [1:37] HTML, you could run into that limit. [1:40] And if your app used SQLite or MySQL, [1:42] standing up Postgres just for action [1:44] cable didn't make much sense. [1:46] So Redis tends to become the default [1:47] choice. [1:49] At work we're using Redis for both [1:50] action cable and caching. [1:52] I unexpectedly found myself responsible [1:54] for managing all of our infrastructure. [1:56] So I started looking for ways to [1:57] simplify things and make my life a lot [1:59] easier. [2:01] I moved our caching over to Solid Cache, [2:03] which just left Action Cable as the last [2:05] thing keeping Redis around. [2:06] It was right around this time that I saw [2:08] this issue that David had filed in Rails [2:10] about extracting an Active Record base [2:12] adapter out of Campfire. So, I figured I [2:14] just had to wait for Campfire to land, [2:16] drop the new adapter in, and finally [2:17] remove Redis from our stack. [2:20] But then a little while later I saw this [2:21] comment come through where Campfire [2:22] ended up using Redis for Action Cable [2:24] after all. [2:26] So, I figured how hard can it be to [2:27] implement an adapter that only uses [2:28] Active Record? [2:31] Redis and Postgres both keep a [2:32] persistent connection open to each Rails [2:34] process. [2:35] So, when a new message arrives, they can [2:37] send that to the connected Rails process [2:39] immediately. [2:40] To build an adapter on top of Active [2:42] Record that's database agnostic, we [2:44] can't depend on Redis's pub/sub or [2:46] Postgres's listen/notify mechanisms. [2:48] So, just like Solid Queue, we can lean [2:49] on a little bit of low-tech. [2:52] Polling. [2:53] We can set up an infinite loop, ask the [2:55] database if it has any new messages, and [2:57] then sleep for a few milliseconds before [2:58] checking again. [3:00] Before looking at the code for the [3:01] adapter, at the core of Solid Cable is [3:03] our messages model. [3:05] Which has a channel and a payload along [3:06] with an index on channel, since that's [3:08] what we'll be querying by as it polls [3:09] for new messages. [3:11] The adapter is really made up of two [3:12] classes. [3:14] One inherits from Action Cable's base [3:15] adapter class, which sets up some shared [3:17] state and defines the methods every [3:19] adapter needs to implement. [3:21] The other's the listener, which in Solid [3:23] Cable's case handles polling. [3:25] It inherits from Action Cable's [3:26] subscriber map, which keeps track of [3:28] who's subscribed to each channel and [3:30] delivers and handles delivering [3:31] broadcast out to them. [3:35] Our adapter class implements four main [3:36] methods: broadcast, subscribe, [3:38] unsubscribe, and listener. Let's start [3:40] with broadcast. [3:42] It just inserts a row into the messages [3:43] table. Using insert lets us skip Active [3:46] Record callbacks, validations, and the [3:47] usual transaction wrapping. None of [3:49] which we need here, so we can make the [3:51] write as lightweight as possible. [3:53] Next up is subscribe. [3:55] It calls a listener and then calls add [3:57] subscriber on listener with a channel [3:59] and the subscriber. So, let's look at [4:00] what that listener is actually doing. [4:03] The listener method is responsible for [4:04] instantiating the second half of the [4:05] adapter, the listener object. [4:07] But, the listener itself doesn't define [4:09] the add subscriber method. That comes [4:10] from the subscriber map superclass. [4:14] Inside subscriber map, initialization [4:16] sets up a hash that tracks subscribers [4:17] by channel. [4:19] When we add a subscriber, we append it [4:21] to the list for that channel and all [4:22] that happens inside of a mutex to avoid [4:24] those pesky race conditions. [4:27] And this is the first subscriber for the [4:29] channel, we call the add channel method, [4:31] which is implemented by our listener [4:32] class. [4:34] Add channel just adds the channel name [4:35] to a list of watch channels that we'll [4:37] use to gather messages to be sent out [4:38] during polling. [4:40] This is where Redis we call it subscribe [4:41] method or Postgres we call it listen [4:43] command inside of their adapters. [4:47] So, that's subscribe. Let's look [4:48] Let's look at unsubscribe. [4:51] As you'd expect, unsubscribe does the [4:52] inverse of subscribe. It calls the [4:54] remove subscriber on the listener object [4:56] and just like add subscriber, that [4:58] method actually lives on subscriber map. [5:01] We call the We remove the subscriber [5:03] from the channel's list and if there are [5:04] no subscribers left, we call remove [5:06] channel, which is implemented by our [5:07] listener class. [5:10] It just removes the channel name from [5:11] our list of watch channels. This is [5:13] where Redis we call unsubscribe and [5:15] Postgres we call unlisten inside their [5:16] adapters. [5:19] And with that, we've covered the main [5:20] methods the adapter class defines. [5:21] Pretty simple stuff. [5:23] Now, let's go look at the listeners [5:24] class initializ- [5:25] listeners class initialization. [5:28] At [snorts] the end of initialization, [5:30] the listener starts a new thread and [5:31] calls listen inside it. Let's go look at [5:33] what that listen method is actually [5:35] doing. [5:37] Listen starts a loop that keeps running [5:38] until the listener gets shut down. [5:40] On each pass through that loop, we query [5:43] for any messages for the channel [5:45] with an ID greater than the one we've [5:47] seen and then broadcast each of those [5:48] messages out to their subscribers. [5:51] After we broadcast each message, we [5:52] update last ID to to that message's ID, [5:55] so the next poll knows where to continue [5:57] from. [5:58] And when the listener first starts up, [5:59] we lazily initialize last ID to the most [6:01] recent message [6:03] already in the table, so we don't replay [6:04] old broadcasts. [6:07] And finally, we sleep for a configured [6:08] amount of time before starting the loop [6:09] again. By default, it's 100 [6:11] milliseconds. [6:14] And that's it. In version one, the [6:15] adapter and the listener together were [6:16] only about 112 lines of code, plus the [6:19] message model and a little configuration [6:20] parsing. [6:22] That's the version I ended up deploying [6:23] at work, and it let me remove Redis from [6:24] our infrastructure without a hitch. [6:27] And then a few months after the release [6:28] of version one, to my surprise, I got [6:30] this email from DHH about adding Solid [6:32] Cable to Rails as the default adapter. [6:35] But, one of the blockers to doing so was [6:37] adding auto trimming. [6:38] Once a message has been already [6:39] broadcast, we don't need to keep it [6:40] around forever. [6:42] Auto trimming periodically deletes older [6:44] messages, so the table stays small and [6:46] the query polling queries stay fast. [6:49] Now, when we broadcast out a message, we [6:50] check if the config has auto trimming [6:52] enabled, and if so, we run the trim job. [6:55] Rather than trimming on a fixed [6:56] schedule, Solid Cable ties cleanup to [6:58] write activity. [6:59] With the default settings, each write [7:01] has a small chance of triggering a trim [7:02] of up to the batch size of old messages. [7:05] The probability is tuned so that over [7:06] time trimming is roughly twice the [7:08] capacity of incoming writes. [7:10] That gives the table consistent downward [7:11] pressure as traffic increases, while the [7:13] configurable retention period determines [7:15] which messages are old enough to delete. [7:18] The other change the folks at 37signals [7:19] suggested was adding a hashed version of [7:21] the channel name to each message. [7:23] We use SHA-256 to turn that turn the [7:25] name into a fixed-size integer, store [7:27] that alongside the the payload, and [7:29] index that column. [7:32] That gives us a much smaller index and [7:34] cheaper lookups, so when the listener [7:35] checks for new messages, it's querying [7:36] by the channel hash rather than the [7:38] channel name. [7:40] One of the more notable bugs that was [7:41] found after release was around how we [7:43] determine which messages should be [7:44] broadcast. [7:46] Originally, Solid Cable tracked a single [7:47] last seen ID for a whole listener. [7:50] Problem is that if you if you subscribe [7:51] to a new channel while listeners are [7:52] already running, [7:54] there may be older messages for that [7:55] channel still sitting in the table that [7:57] this new subscriber should never [7:58] receive. [8:00] >> [snorts] [8:00] >> So, we started tracking a separate last [8:02] seen ID for each subscribed channel. [8:04] When a channel is added, we record the [8:06] newest ID for at that point, which gives [8:08] that subscription its own starting line. [8:11] Because the per channel last seen IDs [8:13] can be added, updated, and removed from [8:14] one thread while the listener is pulling [8:16] on another, we move that channel state [8:18] into a concurrent map and use atomic [8:20] updates to avoid races. [8:23] We still keep a global last ID to [8:24] efficiently scan forward through the [8:25] table, and we eagerly initialize it now [8:28] when the listener starts, so we don't [8:29] miss any messages between initialization [8:31] and the first poll. [8:33] Then, before broadcasting a message, we [8:35] also check the last seen ID for that [8:37] specific channel. [8:38] So, the global ID keeps polling [8:39] efficient and makes startup safe, while [8:41] the per channel ID makes sure [8:42] subscribers only receive messages that [8:44] were broadcast after that channel was [8:46] added. [8:48] All right, let's look at some numbers [8:49] and see how these adapters compare. [8:51] All these benchmarks are run using Solid [8:53] Cable version 4.0 and Rails main from a [8:55] July 31st commit on a box with two CPUs [8:58] and 8 gigs of RAM. [9:00] Redis, MySQL, and Postgres were all [9:01] running on that same machine to avoid [9:02] any network hops, and Solid Cable was [9:04] configured with a 100 ms polling [9:06] interval, and I also disabled auto [9:08] trimming to isolate the adapter [9:09] performance. [9:11] I use Grafana's k6 to generate the load [9:13] and capture the metrics, and all of the [9:15] benchmarking app and load testing [9:16] scripts are available in the bench [9:17] directory of the Solid Cable repo. [9:21] So, the first benchmark up is constant [9:23] load. It keeps a fixed number of users [9:24] active at all times. [9:26] Each user connects, sends a ping, waits [9:28] for the polling message, and repeats [9:30] that 20 times. Once they finish, they [9:32] disconnect, and a new user immediately [9:34] takes their place. [9:36] That cycle runs continuously for 5 [9:38] minutes. So, we're simulating a steady [9:39] stream of users connecting, exchanging [9:41] messages, and leaving. [9:44] And these are the throughputs for each [9:45] adapter at different user counts. [9:47] We're measuring how many messages per [9:48] second make the full trip through the [9:49] system and back. [9:51] At 250 users, all the adapters are [9:53] pretty close around 280 messages per [9:54] second. [9:56] But once you really start saturating the [9:57] system, Redis and the native Postgres [9:59] adapter keep scaling. [10:00] Solid Cable and SQLite hold up pretty [10:02] well, while Postgres and especially [10:03] MySQL start to level off around the 500 [10:05] user mark. [10:07] The other thing we care about in this [10:08] benchmark is latency. How long it takes [10:10] a a ping to go out and the corresponding [10:12] pong to come back. [10:14] As latency increases, clients spend more [10:16] time waiting, which eventually limits [10:17] how much throughput the adapter can [10:19] sustain. [10:21] Up through about 500 users, things are [10:23] still pretty healthy. [10:24] Around 750 and 1,000 users though, [10:26] latency starts climbing across the [10:27] board. [10:29] Solid Cable degrades more aggressively [10:30] under that load, especially on MySQL, [10:32] while Redis and the native Postgres [10:33] adapter hold their latency down for [10:35] longer. [10:37] Next benchmark up is Thundering Herd. [10:39] It connects a fixed number of users [10:41] simultaneously, and each user [10:42] subscribes, sends one ping, validates [10:45] the pong, and then disconnects. [10:47] It's testing how an adapter handles a [10:48] sudden burst in connections, for [10:50] example, after a deployment forces every [10:52] client to reconnect at once. [10:54] The main thing we're looking at is how [10:55] quickly each adapter works through the [10:57] resulting backlog, which we measure [10:58] through message round trip latency and [11:00] total completion time. [11:03] At 250 users, the total completion times [11:05] are all very close, right around 2 [11:06] seconds. At 500 users, we start to see a [11:08] little separation, but the adapter still [11:10] clears the storm in roughly 3 seconds. [11:15] By 1,000 users though, the differences [11:16] are a little more obvious. Redis [11:17] finishes in 5.2 seconds, and the native [11:19] Postgres adapter in about five. [11:22] Solid Cable and Postgres is still close [11:23] to 5.7 seconds, while SQLite is even [11:26] closer at 5.2, with MySQL bringing up [11:28] the rear in around six. [11:30] So, even under a pretty aggressive [11:31] reconnect storm, Solid Cable is still in [11:32] the same general range as the Push based [11:34] adapters. [11:38] And last up is the lecture benchmark. [11:40] It subscribes a number of users to the [11:41] same channel, then publishes seven [11:43] messages per second for five minutes. [11:45] It's designed to show how efficiently a [11:47] single broadcast can fan out to a large [11:49] number of subscribers at the same time. [11:53] These are the latency numbers. At 250 [11:55] subscribers, Redis and the native [11:57] Postgres adapter stay around 70 [11:58] milliseconds on average, while Solid [12:00] Cable is closer to 120. Since our [12:02] polling interval is set to 100 [12:03] milliseconds, that difference is mostly [12:05] the cost of polling showing up. [12:08] At 500 subscribers, everything is still [12:10] everything is still holding together [12:11] pretty well. Redis and the native [12:12] Postgres adapter are around 90 [12:14] millisecond average, and Solid Cable is [12:15] around 145. [12:18] At 750 subscribers though, you start to [12:20] see the system come under some real [12:21] pressure. [12:22] Redis and the native Postgres adapter [12:23] are still under 200 millisecond P95, [12:26] while Solid Cable climbs to the 4 to 500 [12:28] range. [12:30] I'll also note at 10 messages per second [12:32] instead of the seven we used at 750 [12:34] subscribers, every adapter failed the [12:36] benchmark. We'll come back to why that [12:38] happened in a bit. [12:40] So, in aggregate, Solid Cable stayed [12:42] pretty competitive at the lower end of [12:43] the the load test even with a 100 [12:45] millisecond polling interval. But as we [12:47] push push the system harder and added [12:48] more users, its weaknesses started to [12:50] show. [12:51] And after staring at these higher and [12:52] lower numbers for a while and trying to [12:54] find some positive way to put a spin on [12:55] them, the immortal words of the great [12:57] Dr. Ian Malcolm kept coming to mind. [13:00] We can do better. [13:02] So, I instrumented a bunch of different [13:03] parts of Solid Cable to see where the [13:04] time was actually going. And pretty [13:06] quickly a few obvious hotspots showed up [13:07] that we could speed up. [13:09] First up was configuration handling. [13:12] Every broadcast checks whether auto [13:13] trimming is enabled, and every pass [13:15] through the polling loop checks the [13:16] polling interval and whether polling [13:17] logs should be silenced. [13:20] Problem was that each of those checks [13:21] went back through reading the cable [13:22] config, re-parsing YAML and all the [13:24] values, and allocating objects over and [13:26] over again. Not super great. Under load, [13:29] that turned into a surprising amount of [13:30] redundant work and GC pressure. [13:33] So, instead, we move all that parsing [13:34] into a Memcached configuration object. [13:36] We read the cable config once, memorize [13:37] all the values, and reuse that object [13:39] everywhere. [13:41] Next up was how we tracked watch [13:42] channels. [13:45] When channel's added, we added to added [13:47] to our watch list, we stored the raw [13:48] channel name as the key in our map. Then [13:50] on every polling pass, we take all those [13:52] channel names, hash them again, and use [13:54] those hashes in the polling query. [13:57] Now, we hash the channel once when it's [13:58] added and store that hash as the key in [14:00] our channel map. Then when our listener [14:02] polls, those hashes are ready to go [14:03] straight into the query. No rehashing [14:05] every channel on every pass through the [14:06] loop. [14:08] Next up is batching rights. [14:11] Right now, every broadcast results in [14:13] its own insert into the messages table. [14:15] That's simple, but under heavy load, it [14:17] means a lot of individual rights hitting [14:18] the database. [14:20] We're already batching reads by pulling [14:22] from multiple messages at once, so we [14:23] can apply the same idea on the right [14:25] side. Collect broadcasts together and [14:27] insert them in batches to reduce write [14:28] pressure. [14:30] The new broadcaster groups messages into [14:32] batches based on either a maximum batch [14:33] size or maximum delay, whichever happens [14:35] first. [14:37] When it starts up, it creates a queue [14:38] for incoming broadcasts and spins up a [14:40] dedicated thread to watch that queue. [14:43] Each call to broadcast just wraps the [14:44] channel and the payload in a small [14:45] message object and pushes it onto that [14:47] queue. [14:49] The thread that watches the queue starts [14:51] by waiting for the first message to show [14:52] up in the queue. [14:54] Pop blocks the thread here until one [14:56] arrives, so we're not spinning there [14:57] waiting for the [14:58] not spinning while there's nothing to [14:59] do. [15:01] Once that first message comes in, we [15:03] drain anything else that may already be [15:04] waiting in the queue into our batch in [15:06] case we've gotten behind. [15:10] Then we keep collecting messages until [15:11] either the batch reaches its target size [15:13] or the delay expires, whichever happens [15:14] first. [15:16] And then we stop collecting and flush [15:17] the batch. Instead of doing one insert [15:19] per broadcast, we use insert all to [15:21] write the entire batch in a single SQL [15:22] statement. [15:23] That cuts down on the number of writes [15:24] in the database and reduces a lot of the [15:26] overhead we were seeing under heavier [15:27] load. [15:30] Lastly, like I said, while running the [15:31] lecture benchmark at 750 subscribers and [15:33] 10 messages per second instead of seven, [15:36] Redis, Postgres, and Solid Cable all [15:37] crapped out. [15:39] Since all the adapters failed around the [15:40] same load, that led me to believe this [15:41] is an issue inside Action Cable. [15:43] And sure enough, it was. [15:45] When we call a stream from, Action Cable [15:47] builds a handler for that stream. [15:49] If you don't provide a custom callback, [15:51] it uses the default handler whose job is [15:52] basically just to take the broadcast and [15:54] send it down the web socket. [15:57] Before this change, even that simple [15:58] default handler was wrapped with a [16:00] worker pool stream handler. So, every [16:02] incoming message took an extra trip [16:04] through the connection worker thread [16:05] pool before being sent to the client. [16:08] That extra pool is useful for custom [16:09] callbacks because they can run arbitrary [16:11] application code, and you don't want [16:13] that work blocking the executor. [16:15] But for the default handler, that extra [16:16] hop just meant bouncing every message [16:18] through another thread pool for no real [16:20] benefit. [16:22] After this change, we only use the [16:23] worker pool when there's actually [16:24] user-supplied callback. For the normal [16:26] case, the default stream handler runs [16:28] directly, removing that extra queue hop [16:30] from every broadcast. [16:31] This change will be included in the [16:33] upcoming Rails release. [16:36] So, let's rerun the benchmarks and see [16:37] where we landed after all of these [16:38] changes. [16:39] I've removed the Postgres adapter from [16:41] the upcoming charts to give us a little [16:42] more room in the charts since its [16:43] results were generally very close to [16:45] Redis. [16:46] Also, I'll skip the Thundering Herd [16:48] benchmark here since it didn't really [16:49] change after these updates, mainly [16:50] because it had so little sustained [16:52] traffic. [16:53] For Solid Cable, I configured the batch [16:55] broadcaster with a batch size of four [16:56] and a 1-ms delay. [16:59] So, the first benchmark up is constant [17:01] load, and we get [17:04] these are throughput numbers. At 250 [17:06] users, things are basically unchanged. [17:08] The traffic just isn't high enough for [17:09] these changes to make much difference. [17:11] At 500 and 750 users, though, we start [17:13] seeing the gains. [17:14] Postgres and MySQL close a lot of the [17:16] gap with SQLite and all three move much [17:18] closer to Redis. [17:20] At 1,000 users, the improvement is much [17:22] bigger. SQLite actually edges past Redis [17:24] at 1,048 messages per second. PostgreSQL [17:27] jumps from 609 to 1,000 and MySQL nearly [17:29] doubles from 491 to 961. [17:33] It's even better story with latency. [17:35] Before these changes, Solid Cable's [17:36] latency started falling apart as we [17:37] pushed into the higher user counts. Now [17:39] the curve is dramatically flatter from [17:41] 500 to 1,000 users. [17:43] MySQL's P95 drops from 400 from 412 [17:46] milliseconds to 174 at 500 users. [17:50] And from 953 to 228 milliseconds at 750. [17:55] PostgreSQL shows the same pattern [17:56] dropping from 668 milliseconds to to 231 [17:59] at 750 users. [18:01] At 1,000 users, SQLite drops from 390 [18:04] millisecond average to 140. [18:07] PostgreSQL drops from 477 to 179 and [18:09] MySQL goes from taking over a second to [18:11] just 210 milliseconds. [18:13] And the tail latency is the real winner [18:15] here. SQLite now has a dramatically [18:17] better P95 than Redis at 750 and 1,000 [18:19] users, which results in more throughput. [18:22] While PostgreSQL and MySQL have gone [18:23] from being way behind to basically [18:24] matching Redis at 1,000 users. [18:27] And finally, the lecture benchmark. [18:30] Here we see solid improvements across [18:31] the board as the subscriber count [18:32] climbs. Latency stays much flatter than [18:34] before instead of starting to run away [18:36] at the higher end. [18:38] And more importantly, all the adapters [18:39] now make it through the 750 user 10 [18:41] messages per second test, which failed [18:43] before. [18:44] That works out to roughly a 35% uplift [18:46] in messages being handled. [18:49] And these improvements mostly come from [18:50] the stream handler changes inside Action [18:51] Cable, so the benefit actually applies [18:53] across adapters. [18:57] Solid Cable gives us a handful of knobs [18:58] and dials to tune how it behaves. So, [19:00] what do those actually do to [19:01] performance? [19:02] For the next few benchmarks, I'm going [19:03] to stick with just the constant load [19:04] test since it gives us the most balanced [19:06] view of throughput and latency. [19:09] Solid Cable's default polling interval [19:10] is 100 milliseconds. So, what happens if [19:12] we drop that to 25? [19:14] We'll be pulling the database four times [19:16] as often, but in exchange we should be [19:17] able to pick up new messages sooner. [19:19] Let's see what that does to throughput [19:20] and latency. [19:23] And sure enough, it gives us a pretty [19:24] big latency improvement through 500 [19:26] users and even near 750. [19:28] At 250 users, SolidCable drops from [19:30] roughly 160 millisecond P95 to around [19:32] 90. [19:33] At And at 500 users, we're down around [19:35] 100 115 milliseconds. Not quite down to [19:38] Redis, but much closer. [19:41] Once we start saturating the system at [19:43] 750 and 1,000 users, those extra reads [19:45] start to show diminishing returns for [19:46] Postgres and MySQL. SQLite, on the other [19:49] hand, keeps stretching its legs pulling [19:50] further ahead. [19:53] Just to see how far we could push it, I [19:54] also ran SQLite at a 10 millisecond [19:56] pulling interval with 250 users. That [19:59] came in at 65 millisecond average and 78 [20:01] millisecond P95. [20:03] Closer still, but Redis still keeps the [20:04] lead. [20:07] Throughput tells a similar story. [20:09] Dropping the pulling interval gives us a [20:10] nice bump across most of the SolidCable [20:12] adapters. SQLite benefits the most, [20:14] climbing to 1,071 messages per second at [20:16] 1,000 users. [20:19] Postgres and MySQL show modest gains [20:21] through 750 users, with MySQL giving a [20:23] little bit back at the very top end. [20:26] So, if your traffic is relatively low [20:28] and delivery latency matters, it may be [20:30] worth lowering your pulling interval. [20:31] But as your system gets more saturated, [20:33] pulling too aggressively can start to [20:34] work against you, depending on the [20:36] database you're using. [20:38] There isn't one perfect value, so tune [20:39] it for what your infrastructure can [20:40] handle and what your latency [20:41] requirements are. [20:45] Now, let's look at the cost of auto [20:46] trimming. I'll be using a trim batch [20:47] size of 100 messages and a retention [20:49] period of 30 seconds. So, for most of [20:51] the 5-minute benchmark, we'll be [20:53] actively trimming old messages in the [20:54] background. [20:56] I'm also using the back to using the [20:57] default 100 millisecond pulling [20:59] interval, so we're only changing one [21:00] variable at a time. [21:03] At 250 and 500 users, auto trimming is [21:05] basically free. So, I've dropped this [21:06] from the chart and zoomed in at the [21:07] higher end. [21:09] Even here, the throughput impact is [21:10] tiny. At 1,000 users, SQLite is [21:12] essentially unchanged. Postgres drops [21:15] from 1,000 to 956 messages per second, [21:17] and MySQL from 961 to 942. [21:21] Latency takes a somewhat bigger hit on [21:22] Postgres and MySQL, but overall, the [21:24] cost of continuously cleaning up the [21:25] table is pretty modest. [21:28] So, unless you're already operating [21:29] right at the edge of capacity, auto [21:31] trimming is probably a pretty reasonable [21:32] trade-off, which is why it's enabled by [21:33] default. [21:36] So, that's Solid Cable. It offers a [21:37] simpler infrastructure alternative while [21:39] still delivering performance that can [21:40] now stay surprisingly close to Redis, [21:42] even under fairly heavy load, and in [21:44] some cases, like SQLite, exceed it. [21:46] The biggest trade-off shows up at the [21:47] low end. When the system isn't under [21:49] much load, Redis and the native Postgres [21:51] adapter usually have a lower delivery [21:52] latency because they can push messages [21:54] immediately instead of waiting for the [21:56] next poll. [21:58] If that low low latency really matters, [21:59] Solid Cable gives you room to tune the [22:01] polling interval and close some of that [22:02] gap. [22:03] But for a lot of Rails apps, though, the [22:05] database you already have is probably [22:06] more than enough that more than capable [22:08] of handling Action Cable, too. [22:11] All the improvements I've shown today [22:12] were are included in the 4.1 release of [22:14] Solid Cable, which is out now. [22:17] I did bring one more gift, though. [22:20] Action Cable currently doesn't guarantee [22:22] message delivery order. [22:24] Because messages are handed off to a [22:25] thread pool, two broadcasts can be [22:26] received in one order, but actually [22:28] delivered in another depending on how [22:29] those threads get scheduled. [22:32] Most of the time, that's not a big deal. [22:33] If two notifications arrive slightly out [22:35] of order, nobody notices. [22:37] But for something like streaming an LLM [22:38] response, order matters a lot. If chunks [22:41] of the response are delivered out of [22:42] sequence, the text can be assembled [22:43] incorrectly. [22:45] Here, I have an example of that. [22:47] I'm firing off a turbo stream that just [22:48] appends incrementing numbers as fast as [22:50] possible, and they appear out of order. [22:52] Absolute chaos. [22:53] This is where Solid Cable has an [22:54] advantage. [22:56] Because every broad every message is [22:57] stored in the database, we can persist [22:59] ordering information right alongside it. [23:01] We we give each message a sequence [23:03] within its channel, then we have enough [23:04] information to make sure that those [23:06] messages are consumed in the same order [23:08] that they were broadcast. [23:11] So, I've been working on a new gem [23:12] called ChronoCable, which is basically a [23:13] collection of monkey patches for [23:15] ActionCable and SolidCable that allows a [23:17] channel to keep its messages in order [23:19] and can recover dropped messages. This [23:21] is very much a work in progress, so lots [23:22] may still change. [23:25] ChronoCable has a new channel model that [23:27] stores the channel hash, tracks the [23:28] current sequence number for that [23:29] channel, and records whether messages on [23:31] that channel should be delivered in [23:32] order. [23:34] You can opt to stream in order delivery [23:36] with a new deliver in order option on [23:38] stream from. [23:39] When the stream is set up, ChronoCable [23:41] creates the corresponding channel record [23:43] and marks it for order delivery. [23:46] If you're using Turbo, you can pass [23:47] deliver in order through the data [23:48] attributes on Turbo stream from to opt [23:50] that stream into order delivery. [23:53] When we broadcast, ChronoCable first [23:54] checks whether that channel uses order [23:56] delivery. If it does, we lock the [23:58] channel record, increment its sequence, [24:00] and attach that channel specific ID to [24:02] the message before it gets queued for [24:03] storage. [24:04] ChronoCable also adds a new channel ID [24:06] column on the messages table [24:08] that stores the sequence ID. [24:12] The downside of this is it cuts into a [24:13] lot some of the benefit we were getting [24:14] from batching [24:16] because order ordered channels now need [24:18] extra database work on every broadcast. [24:20] Still experimenting with ways to reduce [24:21] that cost, but for now, let's look at [24:23] how the front end works. [24:26] This is where a lot of the new work [24:26] happens because the browser now has to [24:28] keep a little bit of state for each [24:29] ordered stream. [24:31] The subscription starts like normal. The [24:33] browser sends a subscribe message to the [24:34] server with the channel it wants to [24:36] listen to. [24:37] When the subscription's confirmed, the [24:39] server now sends back to us the current [24:41] sequence ID for each stream in that [24:43] subscription. [24:45] That tells the browser the last message [24:47] on chat was one, so the next message it [24:48] should expect is two. [24:52] When a message is broadcast, we now [24:53] include the stream name along with that [24:55] message's a ID. [24:57] We check that the message ID is the next [24:58] one we're expecting. If it is, we [25:00] consume the message and advance the [25:02] expected ID to the next one in the [25:03] sequence. [25:05] But if a message arrives ahead of the [25:06] one we're expecting, we don't consume it [25:07] yet, and instead we set it aside and ask [25:10] the server for the stream's history [25:12] starting from the last message ID that [25:13] we successfully consumed. [25:16] While we're waiting for the the missing [25:18] message, another newer message might [25:19] arrive. Since it is ahead as well, we we [25:22] set it aside with the others. [25:24] When the history comes back, we consume [25:26] the missing messages in order, three and [25:27] then four, and then advance the expected [25:29] IDs we go. [25:32] Then we work through the messages we set [25:33] aside. We already recovered message four [25:35] from the history response, so we discard [25:37] it. [25:38] Message five is new, so we consume it [25:40] and advance the expected ID again. [25:43] While the waiting queue with the waiting [25:44] queue empty, we're caught back up, so we [25:46] can go right back to listening to the [25:47] for new messages like normal. [25:50] If the website gets disconnected, say [25:52] your internet drops, when the browser [25:53] reconnects and resubscribes, [25:56] the subscription confirmation will tell [25:57] us that the stream is currently on [25:58] message seven, but the browser was still [26:00] expecting message six. So now we know [26:02] we've missed something while the [26:03] connection was down. [26:05] So we can ask the server for the stream [26:06] history again, starting with the last [26:07] message we saw, and we can use that to [26:09] catch back up. [26:11] We consume message six and then seven in [26:13] order, advance what we're expecting [26:14] next, and then go right back to normal [26:15] streaming. [26:18] Ask Chronal Cable. If you've got [26:19] channels where message order really [26:20] matters, give it a whirl and let me [26:21] know. [26:23] And that's it. Here are the repo links [26:25] if you want to check it out and give any [26:26] feedback or issues, and thanks. [26:32] >> [music] [26:36] [music]