Strzibny Details Kamal Engine to Replace Complex Kubernetes Clusters

Kamal Deep Dive - Josef Strzibny — Ruby on Rails

Ruby on Rails

Kamal coordinates zero-downtime blue-green deployments through SSHKit command arrays and an imperative Go proxy, enabling development teams to avoid the infrastructure costs of Kubernetes.

AI-assisted summary of Kamal Deep Dive - Josef Strzibny — original source. Check the original for full context.

Source published:

Key ideas

  • Kamal orchestrates Ruby on Rails deployments from a local machine over SSH using Thor and SSHKit, constructing commands as structured Ruby arrays before dispatching them to remote server hosts. (6:21)
  • Images are compiled via Docker Buildx inside isolated BuildKit containers and pushed to remote servers securely using an SSH reverse tunnel connected to a lightweight local registry. (13:08)
  • Kamal Proxy runs on remote hosts as a Go-based proxy that handles service registration, zero-downtime blue-green switches, request buffering, and connection draining via goroutines and channels. (15:33)
  • Multi-host coordination relies on a deployment 'barrier' invariant that gates background worker deployment until the first primary web server successfully passes health checks. (19:29)

Detailed notes

## Background Published by the official Ruby on Rails channel on YouTube on October 1, 2026, this presentation features Josef Strzibny speaking at Rails World 2026. The session traces what actually happens under the hood across remote infrastructure when executing the 'kamal deploy' command.

## TL;DR Josef Strzibny walks step-by-step through the lifecycle of a deployment orchestrated by Kamal, starting from Thor-based CLI commands and SSHKit execution over pooled SSH connections. He details how Docker Buildx compiles images using isolated BuildKit containers and explains how local registries distribute images across remote hosts via reverse SSH port forwarding. The talk then explores host-level operations, examining how Kamal Proxy manages imperative routing, zero-downtime blue-green switches, shared asset directories, and synchronization barriers across multi-server fleets. Finally, Josef unpacks the internal Go architecture of Kamal Proxy, covering request buffering, maintenance mode coordination, and connection draining primitives.

## Atmosphere & style The presentation blends rigorous, deep-seated technical analysis with self-deprecating comedy and cultural observations from Josef's travels across the United States. His delivery is charismatic and approachable, interweaving explanations of Go channels and concurrency loops with anecdotes about fast-food gluttony, signing reckless Ferrari waivers, and preaching container architecture at a Wendy's counter. Watching the talk feels like an energetic, informative technical seminar where hard-edged systems engineering is enlivened by genuine enthusiasm and playful showmanship.

## Bulletin - [06:21](https://www.youtube.com/watch?v=CsWQD81i5-c&t=381s) Kamal orchestrates Ruby on Rails deployments from a local machine over SSH using Thor and SSHKit, constructing commands as structured Ruby arrays before dispatching them to remote server hosts. - [13:08](https://www.youtube.com/watch?v=CsWQD81i5-c&t=788s) Images are compiled via Docker Buildx inside isolated BuildKit containers and pushed to remote servers securely using an SSH reverse tunnel connected to a lightweight local registry. - [15:33](https://www.youtube.com/watch?v=CsWQD81i5-c&t=933s) Kamal Proxy runs on remote hosts as a Go-based proxy that handles service registration, zero-downtime blue-green switches, request buffering, and connection draining via goroutines and channels. - [19:29](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1169s) Multi-host coordination relies on a deployment 'barrier' invariant that gates background worker deployment until the first primary web server successfully passes health checks.

## Key takeaways - [04:52](https://www.youtube.com/watch?v=CsWQD81i5-c&t=292s) Kamal provides an alternative to complex orchestration tools like Kubernetes by managing deployments via direct SSH communication and Docker containers. - [06:21](https://www.youtube.com/watch?v=CsWQD81i5-c&t=381s) The Kamal CLI uses Thor for subcommand organization and SSHKit for remote host execution, leveraging a customized DSL to execute commands partitioned by roles. - [06:21](https://www.youtube.com/watch?v=CsWQD81i5-c&t=381s) Command instructions are retained as structured Ruby arrays (e.g., [:docker, "-v"]) as long as possible before passing to Net::SSH, ensuring clean logs without shell prefixing issues. - [10:53](https://www.youtube.com/watch?v=CsWQD81i5-c&t=653s) Kamal monkey-patches SSHKit's concurrency loop to accumulate exceptions across all threads rather than aborting instantly, ensuring full visibility into multi-host deployment errors. - [10:53](https://www.youtube.com/watch?v=CsWQD81i5-c&t=653s) Persistent TCP connection pooling across server hosts significantly reduces SSH setup latency during multi-step deployment operations. - [11:53](https://www.youtube.com/watch?v=CsWQD81i5-c&t=713s) Kamal utilizes the 'docker-container' buildx driver to unlock advanced capabilities such as multi-architecture container compilation and isolated builder instances. - [13:08](https://www.youtube.com/watch?v=CsWQD81i5-c&t=788s) Remote servers pull images over an SSH reverse tunnel connected to a local registry container on the developer's machine, eliminating the need for external public registries. - [15:33](https://www.youtube.com/watch?v=CsWQD81i5-c&t=933s) Kamal Proxy is configured imperatively via `docker exec` commands rather than dynamic file watchers, persisting routing state inside a dedicated Docker volume. - [17:09](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1029s) Zero-downtime blue-green deployments run new and old Puma containers in parallel until health checks pass and existing connections are cleanly drained. - [17:09](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1029s) Asset integrity across concurrent container versions is preserved using a shared asset volume with non-overwriting copy semantics (cp -n). - [17:09](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1029s) Concurrent deployment collisions are prevented using a lightweight, filesystem-based mutual exclusion lock on the target server. - [19:29](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1169s) The deployment barrier acts as an invariant synchronization flag, unlocking background worker containers as soon as the first web container passes its health checks. - [19:29](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1169s) Fleet-wide rollouts can be throttled using batch limits configured as absolute integer counts or percentages to prevent simultaneous restarts. - [21:29](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1289s) Kamal Proxy leverages Go goroutines and channels to implement concurrent HTTP middleware routing, periodic ticker-driven health checks, and request buffering. - [25:10](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1510s) In-flight connection draining distinguishes between standard HTTP requests—which are given a parallel waiting grace period—and WebSockets, which are terminated immediately. - [25:10](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1510s) Maintenance mode is efficiently implemented in Kamal Proxy using a synchronization channel whose closure simultaneously unblocks and notifies all waiting request goroutines.

## Detailed walkthrough ### [00:12](https://www.youtube.com/watch?v=CsWQD81i5-c&t=12s) Introduction and Confessions of a "Kamalholic"

Josef Strzibny introduces himself to the Rails World audience with an admission: "Hi, I'm Josef and I'm a Kamalholic." What started out innocently enough—deploying a few applications and writing an initial blog post—quickly escalated. The condition grew into authoring the first-ever book on Kamal, releasing a dedicated video course, getting an article published in *iX* magazine, and delivering talks at conferences such as Wrocław Ruby, where he shared lesser-known operational details about Kamal.

That journey ultimately led him to Austin, Texas, driven in part by his new role at SerpApi. SerpApi provides real-time search engine data for AI workflows, returned cleanly formatted as JSON or Markdown. Josef also works on SerpTrail, an SEO-checking application that he proudly notes remains built on Ruby on Rails—not rewritten in Rust—and comes pre-configured out of the box with Kamal deployment settings.

[01:44](https://www.youtube.com/watch?v=CsWQD81i5-c&t=104s) First Time in America: In-N-Out, Gordon Ramsay, and Hollywood

Reflecting on his first trip to the United States, Josef humorously contrasts the extremes of American coffee portion sizes and recounts attending an unnamed conference in Los Angeles, California. As an unabashed devotee of celebrity chef Gordon Ramsay—not for high-end Michelin-starred dining, but "for all the wrong reasons"—Josef was aware of Ramsay’s well-known appreciation for American fast food, particularly the West Coast staple In-N-Out Burger. Josef made a point to visit In-N-Out and ordered an enormous spread intended for a single person, though he admitted he needed help from his SerpApi colleagues to actually finish it.

Following the conference, he headed to Hollywood Boulevard to view the Walk of Fame. There he encountered a tourist attraction offering rides in a Ferrari California. Despite initially dismissing it as a classic tourist trap, the specific California model won him over. Confronted with a terrifying liability waiver, he realized that reading the fine print would prevent him from signing it, so he signed it unread and took the wheel. The experience made him wonder if he had developed an expensive habit for luxury sports cars. However, after consulting ChatGPT regarding his personal finances, he decided to stick to an addiction he could actually afford: Kamal.

[04:52](https://www.youtube.com/watch?v=CsWQD81i5-c&t=292s) Overview of the Kamal Architecture

Josef outlines the scope of the deep dive, focusing on Kamal's centerpiece command: `kamal deploy`. The walkthrough traces the process from initial SSH execution all the way through to request routing via the proxy. For attendees new to the tool, Josef provides a high-level conceptual model of how Kamal structures applications using Docker containers:

```mermaid flowchart TD accTitle: Kamal Deployment Architecture accDescr: Diagram showing local laptop deploying via SSH to server containers and proxy.

subgraph LocalMachine["Local Machine"] CLI["Kamal CLI ('kamal deploy')"] end

subgraph TargetServer["Server Host"] Proxy["Kamal Proxy"]

subgraph Roles["Application Roles (Shared Docker Image)"] Web["Primary / Web Role (e.g. Puma)"] Worker["Background Jobs / Worker Role"] end

subgraph Accessories["Accessories (Independent Lifecycle)"] DB["Database Container"] Browser["Headless Browser Container"] end end

CLI -->|"SSH Commands"| TargetServer Proxy -->|"Forward Traffic"| Web ```

- **Application Roles**: Containers built from the primary Docker image that share deployment cycles. The primary role is typically the `web` role (such as a Puma application server handling HTTP traffic), alongside optional auxiliary roles such as worker processes for background jobs. - **Accessories**: Long-lived services such as databases (PostgreSQL/MySQL) or headless browser runners. These run on separate life cycles and are managed outside standard application rolling updates. - **Kamal CLI vs. Kamal Proxy**: The CLI operates locally on the operator's laptop, orchestrating actions across remote infrastructure over SSH, while Kamal Proxy runs directly on target hosts to manage traffic cutover.

[06:21](https://www.youtube.com/watch?v=CsWQD81i5-c&t=381s) Deep Dive 1: Under the Hood with Thor, SSHKit, and Command Construction

Executing `kamal deploy` outputs a real-time log of every underlying operation before it takes effect on the infrastructure. Josef breaks down how Kamal translates high-level deployment instructions into shell actions:

1. **CLI Layer via Thor**: Just like the Rails framework itself, Kamal subclasses `Thor` to define its CLI subcommands. When running `kamal deploy`, Thor invokes an internal sequence of nested subcommands via the `invoke` method, all of which are independently runnable directly from the command line. 2. **SSH Execution via SSHKit**: For remote management, Kamal relies on `SSHKit`, the battle-tested library originating from Capistrano. Kamal builds a customized domain-specific language (DSL) on top of SSHKit to run commands locally or across remote server targets partitioned by roles. 3. **Structured Array Commands in `Kamal::Commands`**: Rather than generating raw shell strings directly, Kamal formats command instructions as structured Ruby arrays under the `Kamal::Commands` namespace. - For example, a version check compiles into an array like `[:docker, "-v"]`. - By prepending `:docker` as a distinct symbol, Kamal registers Docker as an explicit SSHKit command runner. This bypasses prefixing commands with `/usr/bin/env`, ensuring remote execution logs remain clean and legible. - Operations that require piping or subshells leverage explicit helpers (e.g., `pipe` or `sh -c`) that still resolve into manageable token arrays. 4. **Handling Sensitive Parameters**: Passwords and secrets (such as container registry credentials) are wrapped in a dedicated `Sensitive` class. When inspected during command logging, the value automatically renders redacted, but delegates to the raw unredacted string when passed to execution. 5. **Dispatching via Net::SSH Backend**: Commands remain in array form for as long as possible before passing to SSHKit's `Net::SSH` backend `execute(*cmd)` method using splat arguments. Because both Thor and SSHKit employ `instance_exec` (which overrides `self`), global context and settings are stored under the top-level `Kamal` namespace constant.

[10:53](https://www.youtube.com/watch?v=CsWQD81i5-c&t=653s) Concurrency, Error Handling, and Connection Pooling

Running commands across fleets of hosts introduces edge cases around failure handling and network overhead:

- **SSHKit Concurrency Patch**: In standard SSHKit, threads are joined simultaneously (`threads.each(&:join)`). If a command errors on one node, standard implementations may abort instantly. Kamal monkey-patches this loop to rescue and accumulate exceptions from all threads. This ensures every host in the deployment role executes the command to completion, providing a full inventory of host failures rather than halting prematurely. - **TCP Connection Pooling**: Constantly opening and closing new SSH connections across multi-step processes—from `docker login` to `docker stop`—incurs high TCP latency. Kamal maintains and reuses an internal connection pool across multiple command invocations for each distinct server host.

[11:53](https://www.youtube.com/watch?v=CsWQD81i5-c&t=713s) Deep Dive 2: Building and Pushing Images with Buildx

The second operational phase covers container compilation and image publishing:

- **BuildKit and Docker Buildx**: Kamal relies on Docker's upstream Moby project tooling, specifically `docker buildx build`. Because `buildx` functions as a Docker CLI plugin, it manages builds through modular driver architectures. - **Docker vs. Docker-Container Driver**: While the standard `docker` driver functions like legacy `docker build`, Kamal explicitly prefers the `docker-container` driver. This spins up an isolated BuildKit container, unlocking advanced features: - Multi-architecture container compilation (e.g., building ARM64 and AMD64 targets). - Multi-node and advanced cache exporter configurations. - **Named Builder Instances**: In Kamal's configuration file, defining a custom `builder` setting maps directly to an isolated, named instance of this `docker-container` driver runtime.

[13:08](https://www.youtube.com/watch?v=CsWQD81i5-c&t=788s) BuildKit and Local Registries: The Remote Forwarding Trick

Josef addresses the initial confusion developers often feel around the build terminology in Kamal. When you run `kamal deploy`, Docker runs `docker buildx build`. Because Buildx uses an underlying builder instance running as a separate container, it is literally called a "docker container" driver. Inside that isolated container, BuildKit executes the project's Dockerfile to produce the image. This entire workflow operates identically whether it takes place on a local development machine or on a designated remote builder server; Kamal simply orchestrates the process remotely as long as Docker is already installed on the remote host.

```mermaid flowchart LR accTitle: Image Build and Registry Forwarding Architecture accDescr: Flow diagram showing Buildx pushing an image to a local Docker registry and hosts pulling it over an SSH reverse tunnel. subgraph LocalDev["Local Machine"] Build["docker buildx\n(BuildKit Container)"] Reg["kamal-docker-registry\n(Local Registry Container)"] end subgraph RemoteHost["Remote Host Server"] Tunnel["Reverse SSH Tunnel\n(Remote Port Forwarding)"] Pull["Docker Engine\n(Pulls Image)"] end Build -->|"push image"| Reg Reg <-->|"forwards registry port"| Tunnel Tunnel -->|"pull image"| Pull ```

The next logistical question is how images reach remote servers without always paying for or configuring third-party hosted registries like Docker Hub. Kamal supports running a lightweight, local in-memory registry—essentially spinning up a standard `registry` container on your local machine. However, this creates a networking challenge: how can remote production servers pull image layers directly from a port residing on your development laptop?

Josef points out that while developers are accustomed to local SSH port forwarding (running `ssh -L` to inspect a remote production database on `localhost`), Kamal uses the inverse mechanism: remote port forwarding (comparable to `ssh -R`), implemented directly through Ruby SSH libraries rather than by shelling out. Remote servers can view and hit the laptop's local registry port through the encrypted tunnel as if it were accessible over their own network, allowing them to pull newly built images seamlessly without ever publishing them to a public registry.

- `docker buildx build` executes inside a dedicated BuildKit container, ensuring identical build mechanics on local machines or remote builder hosts. - Local registries run via a standard `kamal-docker-registry` container. - Remote hosts pull images from the local registry over an SSH reverse tunnel (remote port forwarding) without requiring external exposure.

---

[15:33](https://www.youtube.com/watch?v=CsWQD81i5-c&t=933s) Imperative Service Registration via Kamal Proxy

Once the image arrives on the host, Kamal moves to service registration. Unlike traditional reverse proxies that continuously poll static configuration files or service discovery backends, Kamal Proxy does not have a dedicated configuration file on disk. Instead, Kamal operates purely imperatively: every routing adjustment is issued directly from the Kamal CLI via `docker exec`.

``` docker exec kamal-proxy kamal-proxy deploy web --target <container_id>:<port> [flags...] ```

Every operational parameter—including TLS termination settings, request buffering rules, and logging configurations—is passed directly as CLI flags within that single execution call. The deploy command tells the `kamal-proxy` binary to attach incoming traffic for a given service role (such as `web`) to a specific target consisting of the new container ID and its internal private port. Because pure imperative in-memory state would vanish if the proxy container crashed or restarted, Kamal Proxy mounts a persistent named volume (`kamal-proxy-config`) where it serializes its active state. Beyond initial deploys, the proxy exposes additional lifecycle operations including maintenance mode, stopping, removing, and resuming backends.

- Configuration is pushed imperatively via `docker exec` commands rather than dynamic file watchers. - Active routing targets are defined by dynamic container IDs and internal published ports. - Proxy routing configurations persist across proxy restarts inside the `kamal-proxy-config` named Docker volume. - Supported proxy service operations include `deploy`, `remove`, `stop`, `pause` (maintenance mode), and `resume`.

---

[17:09](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1029s) Zero-Downtime Deployments, Asset Overlap, and Locking

Josef transitions into the core mechanics of zero-downtime blue-green deployments on a single host. When deploying a fresh release, Kamal starts the new container running Puma on an isolated private port alongside the old, currently serving container. Once the new container boots, Kamal signals Kamal Proxy to register the new container ID as the target. The proxy runs internal health checks against the new instance. Once the checks pass, it cleanly shifts routing to the green container and drains existing in-flight connections on the old blue container before decommissioning it.

```mermaid flowchart TD accTitle: Blue-Green Switchover and Asset Sharing accDescr: Flow diagram showing parallel asset directory extraction, proxy health check, and request draining during container cutover. subgraph Storage["Shared Asset Volume"] Assets["/shared/assets\n(Extracted with cp -n)"] end subgraph Routing["Kamal Proxy Routing"] Proxy["kamal-proxy"] Old["Old App Container\n(Draining)"] New["New App Container\n(Passing Health Check)"] end Assets -.->|"serves old assets"| Old Assets -.->|"serves new assets"| New Proxy -->|"drains connections"| Old Proxy -->|"switches live traffic"| New ```

Simultaneously running two application generations introduces an asset problem: active clients might request asset fingerprints generated by either the outgoing or incoming deploy. Kamal resolves this by extracting asset bundles from both container images into a shared asset volume. By utilizing non-overwriting copy semantics (`cp -n`), assets from the prior version remain intact while new fingerprinted files are added alongside them. As long as the assets have unique fingerprints, both containers can serve requests across deploy boundaries without 404 errors.

To prevent catastrophic collisions when multiple engineers or CI pipelines attempt to deploy at the same moment, Kamal uses a lightweight, filesystem-based mutual exclusion lock on the target server. Acquiring the deployment lock is as straightforward as atomically creating a designated lock directory on the server filesystem; if the directory already exists, concurrent deployments halt immediately.

- Blue-green zero-downtime deploys run old and new Puma containers in parallel until health checks pass and connections drain. - Asset integrity across versions is maintained via a shared volume using `cp -n` to prevent asset clobbering. - Concurrent deploy safety relies on atomic server-side directory locking.

---

[19:14](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1154s) Multi-Host Coordination and the Deploy Barrier

When scaling across multiple servers, coordinating rollouts requires more orchestrational rigor than simply looping through hosts one by one. Josef dives into what Kamal calls the deployment "barrier" (or "web barrier"), comparing the deep architectural exploration to navigating the "Great Barrier Reef." The barrier functions as an invariant synchronization flag within Kamal's health-check namespace that can only transition from false to true once.

```mermaid flowchart TD accTitle: Multi-Host Web Barrier Coordination accDescr: Flow diagram showing how web containers gatekeep background worker deployment across multiple hosts. subgraph WebHosts["Web Role Servers"] WebBoot["Boot New Web Container"] Health["Perform Health Check"] Release["Open Web Barrier\n(Set Invariant = true)"] end subgraph WorkerHosts["Worker Roles"] Wait["Worker Roles\nWait at Barrier"] Proceed["Release Worker\nContainer Boots"] end WebBoot --> Health Health -->|"First Web Pass"| Release Wait -.->|"blocked until open"| Release Release --> Proceed ```

The primary role of the barrier is acting as a gatekeeper. Non-web roles (such as background workers or queues) are told to wait at the barrier while web roles begin spinning up. The deployment does not wait for every single web server to be green; as soon as the first web container passes its health checks, it releases the barrier. The moment that single invariant flips, the remaining web servers finish switching over and the waiting background worker containers are unlocked to deploy. For fleet-wide operations, deploys can be batched using rolling limits (expressed either as an absolute integer of hosts or a percentage), preventing an entire fleet from restarting simultaneously.

- The deployment barrier uses an immutable invariant variable that transitions to open exactly once. - The first successfully booted and health-checked web container opens the gate for the rest of the rollout. - Worker roles are blocked behind the barrier until initial web traffic validation succeeds. - Rolling deployments across large server fleets are throttled using batch limits configured as counts or percentages.

---

[21:29](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1289s) Under the Hood of Kamal Proxy: Goroutines, Channels, and Draining

Josef shifts into the internal mechanics of Kamal Proxy itself, which is written in Go. For the Ruby on Rails audience, he explains Go’s core concurrency primitives: goroutines (lightweight functions managed by Go's runtime scheduler across operating system threads) and channels (typed, first-in-first-out communication queues used to pass data between goroutines). While typical Go HTTP servers call `go srv.Serve()`, Kamal Proxy uses standard Go HTTP middleware handlers to intercept, inspect, buffer, and route requests down to backend targets.

```go ticker:= time.NewTicker(interval) for { select { case <-ticker.C: runHealthCheck() case <-ctx.Done(): return } } ```

Channels are heavily utilized throughout Kamal Proxy's internals. Health checking is orchestrated by monitoring a `time.Ticker` channel (`ticker.C`). Inside an infinite loop, a Go `select` statement waits on incoming values: each time the interval timer ticks, a health-check goroutine executes against the backend target.

Kamal Proxy also provides request buffering to protect application workers from slow clients; it buffers incoming payloads in memory and overflows larger requests to disk before handing them off to Puma in a clean burst.

Finally, Josef explains connection draining during zero-downtime cutovers. When an old container is retired, Kamal Proxy must drain in-flight requests. Using context deadlines combined with `select` statements, the proxy monitors whether active requests finish within the grace period (`ctx.Done()`) or whether their deadlines expire, gracefully completing open transactions while terminating hung connections.

- Goroutines and Go channels handle concurrent HTTP traffic, middleware inspection, and background tasks. - Periodic health checks are driven by Go's `time.Ticker` channel via non-blocking `select` loops. - Request buffering absorbs slow client uploads in memory and disk before proxying to Puma. - Connection draining allows existing in-flight connections to finish cleanly under bounded cancellation deadlines.

[25:10](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1510s) Request Draining and Maintenance Mode in Kamal Proxy

Diving into the mechanics of connection draining during deployment, Josef Strzibny explains how Kamal Proxy differentiates between types of in-flight traffic when retiring an old container instance. Persistent connections such as WebSockets—which power Rails features like Action Cable—are handled bluntly: the proxy terminates them immediately without attempting any graceful wait. For standard HTTP requests, however, the proxy enters a dedicated waiting routine that processes in-flight requests in parallel.

```mermaid flowchart TD accTitle: Kamal Proxy Request Draining Flow accDescr: Flowchart illustrating how Kamal Proxy handles WebSockets, standard HTTP requests, deadlines, and maintenance mode.

subgraph Draining["Connection Draining"] Req["In-flight Connection"] Req -->|"WebSocket / Action Cable"| Terminate["Kill Immediately"] Req -->|"Standard HTTP"| WaitBlock["Wait Block (Parallel)"] WaitBlock -->|"Request completes"| Done["Clean Finish"] WaitBlock -->|"Hits deadline"| Cancel["Cancel with Error Draining"] end

subgraph Maintenance["Maintenance Mode"] PauseChan["Pause Channel Created"] -->|"Goroutines wait"| SelectLoop["Select Block"] CloseChan["Close Channel"] -->|"Notifies all workers"| Resume["Unblock & Proceed"] end ```

The proxy tracks in-flight requests concurrently until either all requests finish or a specified timeout deadline is hit. If requests fail to finish before the deadline expires, execution exits the waiting block, and any lingering connections are canceled outright with an `error draining` state.

Josef also highlights the elegant way Kamal Proxy handles maintenance mode using native Go concurrency primitives. Rather than passing specific values or polling state flags across threads, the proxy uses a synchronization channel acting as a pause signal: - Goroutines representing incoming requests wait on a select statement listening to the channel. - Once maintenance mode ends, the channel is closed. - Closing a Go channel instantly notifies every listening goroutine simultaneously, unblocking all requests at once so they can proceed cleanly.

---

[26:20](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1580s) Summary, Kamal Addiction, and Fast Food in America

Wrapping up the technical architecture, Josef recaps the core layers underpinning a deployment. Kamal relies on Ruby gems like Thor for the command-line interface and SSHKit for remote command execution, carefully keeping bash commands as arrays right up until the point they are sent over the wire. Once on the target host, Kamal Proxy takes center stage, registering new container ports and coordinating seamless blue-green zero-downtime rollouts across multiple servers. In the process, the audience got an impromptu deep dive into Go internals, which Josef jokes has probably left everyone's brains completely fried "like a chicken sandwich."

Josef confesses that his obsession with Kamal borders on an addiction. He shares an anecdote from a trip to California where he struck up a conversation with a woman and enthusiastically pitched her on deployment tooling, explaining how Kamal lets developers ditch complex Kubernetes clusters to save massive amounts of infrastructure money. Her deadpan reaction was the classic internet meme: "Sir, this is a Wendy's." Acknowledging that people tell him he has been "doing America wrong" by subsisting on fast food chains, he promises to pivot strictly to Texas barbecue and trade in his figurative Ferrari for a true American muscle car. He concludes his talk with a rousing "Long live Kamal and God bless America!" before walking off stage to warm applause.

- **Foundational Architecture:** Kamal builds CLI workflows with Thor and remote execution via SSHKit, preserving command arguments as arrays until runtime. - **Kamal Proxy Coordination:** Handles automated port discovery, health checks, and traffic shifting to execute zero-downtime blue-green deployments across fleets of servers. - **The Wendy's Pitch:** Josef's running joke about explaining Kubernetes displacement to confused fast-food employees.

## Notable quotes > [00:12](https://www.youtube.com/watch?v=CsWQD81i5-c&t=12s) Hi, I'm Josef and I'm a Kamalholic. — Josef Strzibny

> [01:44](https://www.youtube.com/watch?v=CsWQD81i5-c&t=104s) for all the wrong reasons — Josef Strzibny

> [13:08](https://www.youtube.com/watch?v=CsWQD81i5-c&t=788s) docker container — Josef Strzibny

> [19:14](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1154s) Great Barrier Reef — Josef Strzibny

> [25:10](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1510s) error draining — Josef Strzibny

> [26:20](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1580s) like a chicken sandwich — Josef Strzibny

> [26:20](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1580s) Sir, this is a Wendy's. — Josef Strzibny

> [26:20](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1580s) doing America wrong — Josef Strzibny

> [26:20](https://www.youtube.com/watch?v=CsWQD81i5-c&t=1580s) Long live Kamal and God bless America! — Josef Strzibny

## Mentioned Josef Strzibny, Gordon Ramsay, Ruby on Rails, Rails World, Shopify, SerpApi, SerpTrail, Kamal, Kamal Proxy, Docker, Docker Buildx, BuildKit, Moby, Thor, SSHKit, Capistrano, Net::SSH, Puma, PostgreSQL, MySQL, Go, Action Cable, Kubernetes, ChatGPT, In-N-Out Burger, Wendy's, Ferrari California, iX

## Video description What actually happens when you type 'kamal deploy'? At Rails World 2026, Josef Strzibny, a self-described "Kamalholic" follows a deploy from start to finish.

#RailsWorld #RubyOnRails #Kamal #Docker #DevOps #Deployment

Thank you Shopify for sponsoring the editing and post-production of these videos. Check out insights from the Engineering team at: https://shopify.engineering/

[Source](https://www.youtube.com/watch?v=CsWQD81i5-c)

中文简报

Strzibny详解用Kamal引擎替代复杂的Kubernetes集群 Kamal通过SSHKit命令数组和命令式Go代理协调零停机蓝绿部署，使开发团队能够规避Kubernetes的基础设施成本。 Josef Strzibny逐步解析了由Kamal编排的部署生命周期，从基于Thor的CLI命令和通过SSH连接池执行的SSHKit开始讲起。他详细阐述了Docker Buildx如何利用隔离的BuildKit容器编译镜像，并解释了本地镜像仓库如何通过反向SSH端口转发将镜像分发至远程主机。随后，演讲探讨了主机级操作，剖析了Kamal Proxy如何管理命令式路由、零停机蓝绿切换、共享静态资源目录以及跨多服务器集群的同步屏障。最后，Josef拆解了Kamal Proxy的内部Go架构，内容涵盖请求缓冲、维护模式协调以及连接耗尽原语。 Kamal利用Thor和SSHKit通过SSH从本地机器编排Ruby on Rails部署，将命令构建为结构化的Ruby数组后再分派至远程服务器主机。 (6:21) 镜像通过Docker Buildx在隔离的BuildKit容器内完成编译，并利用连接至轻量级本地镜像仓库的SSH反向隧道安全推送至远程服务器。 (13:08) Kamal Proxy作为基于Go的代理运行在远程主机上，通过goroutine和channel处理服务注册、零停机蓝绿切换、请求缓冲以及连接耗尽。 (15:33) 多主机协调依赖于部署“屏障”不变性，该机制会拦截后台工作进程的部署，直至首台主Web服务器成功通过健康检查。 (19:29)

Reading summary

Josef Strzibny details the operational architecture of Kamal, demonstrating how it combines Ruby CLI orchestration with a specialized Go proxy to deliver zero-downtime, multi-host container deployments without traditional cloud orchestrators. Remote Command Construction and Safe Concurrency (6:21) Kamal orchestrates infrastructure from an operator's local machine using Ruby gems Thor for subcommands and SSHKit for remote command dispatch. Command arguments are structured as discrete Ruby arrays within the `Kamal::Commands` namespace rather than raw strings, avoiding shell clutter like `/usr/bin/env` prefixes and keeping sensitive parameters redacted in logs via a specialized `Sensitive` wrapper class. Top-level settings reside under the global `Kamal` namespace because Thor and SSHKit alter context using `instance_exec`. To overcome latency and failure limitations during multi-host deployments, Kamal maintains a persistent TCP connection pool across successive commands. It also modifies standard SSHKit concurrency behavior: while SSHKit's standard implementation may abort when a thread fails, Kamal monkey-patches the join loop to catch and accumulate exceptions across all threads, ensuring every target host completes its tasks and providing an exhaustive log of all host-level failures. BuildKit Drivers and SSH Tunnel Registries (11:53) For container compilation, Kamal relies on Docker's Moby project tooling via `docker buildx build`, specifically configuring the isolated `docker-container` driver rather than standard Docker build mechanics. This dedicated runtime enables multi-architecture container builds (ARM64 and AMD64) and advanced cache export strategies across isolated builder environments, regardless of whether builds occur locally or on remote builder nodes. To bypass paid third-party container registries, Kamal supports running a local in-memory `kamal-docker-registry` container. Target production hosts pull images directly through a reverse SSH tunnel—remote port forwarding similar to `ssh -R`—implemented inside Ruby SSH libraries. This configuration allows remote servers to pull freshly compiled images through an encrypted tunnel directly from the developer's laptop as if accessing a local network port, removing the need to publish artifacts externally. Imperative State and Blue-Green Zero-Downtime Swaps (15:33) Instead of polling filesystem configurations, Kamal Proxy manages routing imperatively through CLI commands executed via `docker exec`. Routing flags—such as TLS termination, logging, and target container IDs with private ports—are registered dynamically on the fly, while state persistence is handled by serializing configuration to a mounted `kamal-proxy-config` Docker volume. Concurrent deployments from different users or CI pipelines are prevented by an atomic filesystem directory lock on the target server. During zero-downtime blue-green cutovers, new application containers boot in isolation alongside running versions. Kamal prevents 404 errors during rolling releases by extracting asset directories into a shared asset volume using non-overwriting copy semantics (`cp -n`), preserving older asset fingerprints alongside newly deployed ones. Once proxy health checks succeed on the new container, incoming traffic shifts cleanly, and the older container is drained of active connections. Fleet Coordination and the Deploy Barrier (19:14) Coordinating rollouts across multi-server environments relies on a deployment synchronization mechanism known as the deployment barrier. The barrier acts as an invariant flag in Kamal's health-check namespace that can only transition from false to true once. Auxillary services, such as worker and background job containers, are blocked from deploying until the web role verifies its first successful instance. Rather than waiting for an entire fleet to boot, the deployment releases the barrier the moment the very first web container passes its health checks. This event unlocks deployment for waiting worker roles while the remaining web servers finish updating. Large fleets can also leverage configured batch limits, defined as absolute host counts or percentages, to prevent simultaneous cluster-wide restarts. Go Concurrency and Request Draining Internals (21:29) Written in Go, Kamal Proxy uses goroutines and channels to handle request buffering, background tasks, and periodic health checks driven by `time.Ticker` channel select loops. Incoming requests are buffered in memory and overflow to disk to shield downstream application servers from slow client connections. During connection draining, the proxy treats traffic types differently: persistent WebSockets (such as Rails Action Cable) are terminated immediately, whereas standard HTTP connections are granted a grace period using Go context deadlines. If pending requests exceed the deadline, they are canceled with an error draining status. Maintenance mode similarly uses native Go concurrency: incoming requests wait on a pause channel, which unblocks all waiting goroutines simultaneously the moment the channel is closed. Strzibny positions Kamal as a cost-effective, low-overhead alternative to complex container orchestrators like Kubernetes, though effective adoption requires teams to master its specific orchestration mechanics—including SSH command construction, reverse-tunnel image distribution, and proxy draining behaviors.

Source quotes

[0:12] Josef Strzibny: Hi, I'm Josef and I'm a Kamalholic. [1:44] Josef Strzibny: for all the wrong reasons [13:08] Josef Strzibny: docker container [19:14] Josef Strzibny: Great Barrier Reef [25:10] Josef Strzibny: error draining [26:20] Josef Strzibny: like a chicken sandwich [26:20] Josef Strzibny: Sir, this is a Wendy's. [26:20] Josef Strzibny: doing America wrong [26:20] Josef Strzibny: Long live Kamal and God bless America!

Original transcript

[0:06] [music] [0:12] >> Uh [0:13] Hi, I'm Youssef and I'm Camel holic. [0:17] Uh [0:18] nothing seemed wrong at first. I just [0:21] deployed few apps and wrote a blog post. [0:24] But [0:26] it got worse. [0:28] Uh [0:29] I wrote the first ever book on Camel and [0:32] I even made a video course of this [0:34] really close boat video. [0:37] And it become a serious condition. [0:42] I was here this year published in IX [0:45] magazine [0:47] with my Camel article. [0:49] And I started to give Camel talks like [0:52] the one in Wroclaw Ruby where I shared [0:54] some not so known facts about Camel. [1:00] And this brings me to Austin, Texas to [1:02] here. [1:03] But my journey to Austin, Texas started [1:06] earlier because I joined a new company. [1:09] Of course the company is SerpApi. [1:12] If you don't know what we do, we give [1:15] live search data for your AI [1:18] uh [1:19] as a JSON or as a markdown. You can have [1:21] it your way. [1:23] I also [1:24] um [1:26] work on a application called SerpTrail. [1:29] It's like SEO checker. [1:31] And last time I checked it was still [1:34] on Rails, not in Rust. [1:38] Um [1:39] you can check it out. It also comes with [1:41] Camel [1:42] configuration. [1:44] And this brings me also to [1:47] uh the topic of today, my first time in [1:49] America. [1:50] Now this is not a ice slope, uh this is [1:53] two sides of coffee [1:55] in America's. [1:58] Uh some of you know that uh [2:02] uh I went to a conference, but we shall [2:06] not speak its names. Uh and it was in [2:09] LA, California. [2:12] Um [2:14] so [2:15] if you were on my last talk, you know [2:18] that I'm a an unapologetic fan [2:21] of uh Chef Gordon Ramsay, [2:24] who is a celebrity chef. [2:26] And uh [2:29] uh [2:31] why I mention him uh [2:33] for my talk is because [2:36] uh we know that America is known for one [2:40] thing, and it's [2:43] fast food. [2:44] And there is a secret with uh Chef [2:47] Gordon Ramsay that he actually [2:49] uh [2:52] Yeah. [2:53] I actually don't like him for the [2:57] I don't like him for the fancy stuff and [2:59] fancy dining, but for all the wrong [3:03] for all the wrong reasons. [3:07] Uh and he likes fast food, too, right? [3:11] He likes In-N-Out Burgers. [3:14] And because it's a chain in California, [3:16] I had to go and try it. [3:21] So, I ordered essentially a portion for [3:23] one. [3:24] Uh [3:27] um and you know, when in Rome, you [3:29] should also finish. [3:31] But uh no, I I actually got help by my [3:34] colleagues from Serp API. [3:38] Um [3:40] and after the conference, I went to [3:44] Hollywood, of course, because you are in [3:47] LA, right? I was walking this pavement [3:50] with the stars, and then I saw that oh, [3:52] you can actually drive a Ferrari like a [3:55] movie star, right? [3:58] I thought what a tourist trap. [4:01] Uh but actually after closer look, they [4:05] has [4:06] uh Ferrari [4:07] California as a model there. [4:10] So, I got [4:12] I got sold, you know? [4:14] Uh but there was one problem. They give [4:16] you this [4:17] uh very scary-looking waiver [4:20] to sign, and I was like, "Yosef, if you [4:22] are going to read it, you are not going [4:25] to sign it." [4:28] And so, I didn't. [4:30] I didn't read it. I just signed it. [4:33] And next thing I know, I was sitting in [4:35] a Ferrari for the very first time. [4:37] And I realized that maybe I also have [4:40] addiction for [4:42] expensive cars. [4:43] But after chatting with ChatGPT about my [4:45] financials, [4:47] I decided [4:49] I will just stay with the addiction I [4:51] can afford. [4:52] And I will take you uh today with me [4:55] to explore Kamal a little bit. [4:57] Uh [4:59] so, we'll do a couple of deep dives. [5:02] We'll of course explore the most [5:05] interesting command, which is Kamal [5:07] deploy. [5:10] Um [5:13] and we will go all the way from SSH to [5:17] proxying requests, yeah? Now, if [5:21] uh you are here by accident, you don't [5:23] know even what Kamal is, it's important [5:25] to know how Kamal sees your application [5:27] with their application roles and [5:29] accessories. [5:30] So, [5:32] Kamal of course uses Docker, so you have [5:34] your Docker image, and then you would [5:36] have, for example, running Puma as your [5:39] web server. That would be your first [5:41] role, the so-called web role, the [5:43] primary role. And then you can have more [5:45] roles like uh for jobs or for something [5:47] else. [5:49] And [5:50] uh [5:51] then there are some other accessories [5:53] like your database or headless browser [5:56] or whatever. [5:57] Uh and they have different life cycles. [5:59] So, that's not what we will really talk [6:01] about. It's also important to realize [6:03] that we have Kamal CLI [6:05] and Kamal proxy that will run [6:07] on your [6:09] uh on your server. [6:10] So, you are running Kamal deploy in your [6:12] laptop [6:13] and then we have that over SSH uh [6:16] running some commands, Kamal proxy of [6:19] course running there. [6:21] So, let's go and let's go for first uh [6:25] dice. So, that's SSH and Docker. If you [6:28] already deployed with Kamal, [6:30] now [6:32] you're seeing the screen. [6:33] It's my favorite feature of Kamal [6:35] because you see everything that's [6:36] happening, but even before you actually [6:39] run the command. [6:40] Uh [6:41] of course, [6:43] we need something to translate that [6:45] command to code. So, for that you store [6:48] in Kamal. If that sounds familiar [6:50] familiar, yes, it's also used in Rails. [6:53] So, Kamal simply sub classes from Thor [6:56] and then implements the commands. [6:58] Uh [7:00] Kamal deploy [7:01] runs many sub commands that can be also [7:03] invoked from the command line, but if [7:04] it's in code, there will be this invoke [7:07] method and then we go further. [7:10] Everything to SSHKit also might sound [7:12] familiar because we know it's from [7:14] Capistrano times. [7:16] And that actually runs the SSH command [7:19] for Kamal. [7:21] So, Kamal include [7:22] a little bit little small patch that I [7:25] will talk about and small DSL. So, we [7:28] have DSL for running of course some code [7:30] on hosts on servers and some code [7:33] locally. But Kamal because it have roles [7:36] that we also has to have own roles. [7:40] Yeah? [7:42] Uh [7:42] so, we extended just a little bit more. [7:45] So essentially we have this [7:46] configuration for Kamal and then in the [7:49] end we are running some commands. It's [7:51] mostly like Docker commands on the on [7:54] the server and the code itself for the [7:57] commands come from command uh Kamal uh [8:00] commands [8:01] uh namespace. So for example, very [8:04] simple example would be uh Docker V for [8:07] well, it's Docker installed. If you have [8:10] something more complex, it will be full [8:12] Docker run [8:13] and you can notice it almost look like a [8:16] shell. Yeah. Uh there are [8:19] uh some symbols, there are some strings, [8:22] there are some methods, but essentially [8:23] it looks like a shell. That's because [8:25] the Docker method actually builds an [8:27] array and puts Docker symbol at the [8:30] beginning. It does it because it [8:32] actually registered Docker as a SSH kit [8:35] command. And when it does, it means that [8:38] we are skipping the user bin um [8:41] environment. [8:42] binary and the logs get cleaner, right? [8:45] Because it still works even by just [8:47] shelling out to Docker. I talked about [8:49] methods. So for example, let's look at [8:51] how sensitive data are handled. So we [8:53] will have this class for uh sensitive. [8:57] And if you delegate, well, if you need [8:59] to string, right? It will be unredacted. [9:02] But if you are just inspecting, you will [9:04] get the redaction. So this is the [9:06] example for example for registry [9:08] password. [9:10] Yeah, but we are still having this [9:12] string of tokens as an array. [9:17] Now we can combine the commands. We know [9:19] it's array, so this is operations on [9:21] arrays, right? We can compact collect [9:24] type. Now we want pipe which in you [9:27] know, pipe from shell. Well, it's the [9:29] same thing. Just we divide by a pipe. We [9:33] can also have new shell like SHC, you [9:36] know. Or for get docker, we now would [9:39] use that [9:40] shell method with any [9:43] one of them, still just arrays, and we [9:46] could create like docker install [9:48] command, which is like pipe get docker [9:50] to SSH. [9:54] And finally, when we have that, yeah, [9:57] Kamal keeps it for as an array for as [10:00] long as it's possible, but then it has [10:02] to pass it to SSH kit for actually [10:04] running it. [10:06] Uh if you would study the Kamal code, [10:09] you would also see this execute method. [10:12] Now, the execute method actually comes [10:15] from net SSH backend for SSH kit. [10:18] Uh and it would actually really run the [10:21] command. Now, the asterisk is there [10:24] because [10:25] it accepts positional arguments. [10:28] So, we will pass the command like that, [10:31] and it will execute. And we have also [10:34] Kamal as a [10:36] constant because now that we have both [10:40] the tor and SSH kit, which also used [10:43] instant exec, we are losing the self. [10:46] So, we still need to be able to ask for [10:48] global configuration for all the global [10:50] stuff. So, that's why we have that. [10:53] And how is that concurrently on many [10:55] hosts? [10:56] For that, I will return to the SSH kit [10:58] patch. [10:59] So, uh SSH [11:02] kit would normally run threads each [11:04] join, so just joining all threads, but [11:06] Kamal uh does something a little bit [11:09] different. It still joins, but it [11:12] rescues for all the exceptions from all [11:14] the errors. Uh this is so that we really [11:17] run all the code everywhere on every [11:19] host, and we just collect the errors [11:22] instead of just uh bailing out. [11:25] >> [snorts] [11:26] >> And finally, so we have threads, yes, [11:28] but what about TCP connections? We also [11:31] should share them because they're [11:32] expensive to create. So, yes, they are [11:34] shared, because all these little [11:36] commands from Docker login to Docker [11:38] stop, if they are running on concurrent [11:41] um [11:43] uh [11:43] roles, [11:44] uh then they can be actually reused for [11:48] connections. So, there will be a pool, [11:49] and it will get uh [11:51] reused. [11:53] Uh now, the second dive, building and [11:55] pushing images. [11:57] So, of course, Docker is our [11:58] application, but it's buildkit, part of [12:01] Docker kind of or Moby project, which is [12:03] the upstream for Docker, builds your [12:06] image, right? So, essentially, we will [12:08] run Docker buildx build. The buildx [12:11] command is a bit weird, because [12:15] uh this is actually a plugin to to [12:17] Docker. [12:18] So, we need to pull out Docker builder [12:21] drivers. So, there's Docker and Docker [12:23] container. [12:24] Docker is the original one. That was the [12:27] uh Docker build, right? The simple one. [12:30] And we also have Docker container, which [12:33] Kamal uses, because we need [12:35] a little bit more. [12:36] Uh we need multiple architectures, and [12:39] we need [12:40] um [12:41] exports, and we need [12:44] uh [12:44] uh multi-node, and so on. [12:47] So, the builder is the name instance to [12:51] run. So, [12:53] when we are configuring in the animal [12:55] file, maybe you are saying builder [12:57] something, so that would be the instance [13:00] of that Docker [13:01] uh [13:04] uh Docker container [13:06] um [13:07] thing. [13:08] And isn't it a little bit confusing? Uh [13:11] for the first time, I think yes, because [13:13] of the naming. So, we run Kamal deploy, [13:16] as part of that, we are building the [13:17] image, we are running Docker build is x, [13:19] that uses the buildx plugin, that will [13:22] spin up a new container, that's why it's [13:24] called Docker container. And uh the [13:26] buildkit inside that container will then [13:28] run your Dockerfile and actually build [13:30] the image. Now, if that happens on a [13:33] remote builder, it's actually completely [13:35] the same. [13:37] Kamal can just uh do it as long as uh [13:40] Docker is pre-installed on that host. [13:43] And, [13:44] that's it. And, that brings me to [13:46] registries because part of that command [13:49] is of course upload the image to [13:51] registry. So, we have the usual Docker [13:54] Hub, right? Then, you can have different [13:55] provider. But, what's interesting is the [13:57] local registry, right? We can now run [13:59] local registries. So, how it's done [14:02] is that we really just run an extra [14:04] container, which is the Kamal Docker [14:06] registry. [14:08] And, uh we're really running registry [14:10] image. There's nothing special about [14:12] that. [14:14] Except maybe one special thing [14:17] uh because what about the ports, right? [14:19] How our hosts our servers see a local [14:24] port on our laptop. So, [14:27] you might know a local forwarding that [14:29] you might use before when you needed to [14:32] take [14:33] um [14:33] a database port from the server and see [14:35] it locally for some inspection. And, so [14:38] essentially here is just the opposite. [14:40] So, it's remote forwarding. [14:42] Uh something like SSHR. [14:45] Kamal does not shell out SSHR. Of [14:47] course, it's uh implemented than SSH 10. [14:51] Uh but, essentially this is how it [14:54] works. It will be like some kind of port [14:56] for port forwarding [14:58] uh in Kamal. [15:00] And now, the push and pull, right? How [15:02] the image gets on on there. So, for the [15:05] push, [15:07] yeah, we're running Docker build X [15:09] built. [15:11] Uh so, we already seen that, right? Of [15:13] course, because it's push, but it also [15:16] uh it builds and then pushes, right? [15:19] And, for pull, [15:21] if we sort out the port I talked about, [15:25] now the servers see our port so we can [15:27] just pull the image on hosts and there's [15:31] no problem. [15:33] So next we need a service registration. [15:36] So we'll come on has this config camel [15:40] uh, configuration file. But camel proxy [15:42] doesn't have a configuration [15:45] uh, specifically. [15:47] >> [snorts] [15:48] >> Every instruction uh, [15:50] for it would be actually docker exec [15:53] from camel CLI. [15:55] Because we have this imperative [15:57] uh, s- [15:58] way of doing things so we need it for [16:01] the proxy as well. [16:02] And on camel proxy we'll essentially run [16:05] camel proxy deploy. That's the command [16:07] for the proxy. [16:09] Now since we are passing everything in [16:12] one docker exec, we need to pass all the [16:15] flags. So from TSL to logging, [16:19] buffering, whatever we need, we just [16:20] pass it on. So how it looks like, we [16:23] have docker exec camel proxy. [16:26] That means the first one is like, okay, [16:28] run something on this container and then [16:30] camel proxy that will be the binary. [16:33] Deploy our service here such as web and [16:36] then we pass everything else. Now the [16:39] target, that's also important. Uh, that [16:41] will be the container ID and port. [16:45] And now of course, when we are passing [16:47] an imperatively, what happens if the [16:49] proxy goes down? So there's camel proxy [16:51] config uh, named volume. It will keep [16:54] the configuration so [16:56] uh, camel proxy can load it up. [16:59] There are other service actions so the [17:01] deploy, of course we can remove the [17:03] service later or we can stop it, [17:05] maintenance mode, and resume it. [17:09] And uh, that brings me to [17:12] uh, zero downtown [17:14] uh, deployments that are very essential [17:16] to how Kamal works and runs. [17:19] So, let's return to the docker exact [17:22] Kamal proxy [17:23] command. Because when we are running a [17:25] new version, we just run this again [17:28] with the same service name. We just use [17:31] a new deploy target. [17:34] So, we change the tiger target, which is [17:36] the [17:38] container ID and the port. [17:41] And then if you are running proxy, we [17:43] insert the proxy to deploy. So, first we [17:46] run the container. [17:48] It's running and proxy can actually do [17:50] the switch. [17:51] So, in Kamal we use blue green [17:53] deployment. So, simple case, single [17:56] host, we will be booting the new [17:59] uh web rail container, which is our [18:02] Puma, that needs to be served traffic. [18:05] We pass it over to Kamal proxy, which [18:07] will run health check. And once done, it [18:10] can switch over the traffic to the new [18:13] green ones. [18:15] And of course, it has to drain the [18:16] request for [18:18] the previous ones. [18:21] Now, we also have to do something with [18:22] assets because we have two versions at [18:24] the same time. So, essentially we unpack [18:27] the assets to the same directory. [18:31] And then we copy two volumes for both [18:35] the old one and new ones. [18:37] Uh the copy n [18:39] ensures that the first one that gets [18:41] copied over are not overwritten. So, [18:46] we have to just follow the right order [18:48] and [18:50] that's it. Both containers can serve [18:52] both assets. They need to be [18:53] fingerprinted, of course, but then [18:55] everything works. Lock is even easier. [18:59] So, when we would have concurrent [19:01] deploys, we are kind of locking it, so [19:05] nothing bad happens. And this is just [19:07] really creating a directory on on the [19:10] server and that's it. [19:14] >> [snorts] [19:14] >> Now, what happens when we have more [19:16] hosts and we need to coordinate them? [19:20] Right? So, it's kind of the same thing, [19:22] but a bit more complex. For that, we [19:25] will dive even deeper to the great [19:27] barrier reefs. [19:28] So, we can explore the barrier or [19:32] what's also called the web barrier. [19:35] Uh [19:37] that according to the role [19:40] that we are protecting. How it looks [19:42] like? So, it's part of actually health [19:44] check name space, and we will have an [19:46] invariant, [19:48] which is like immutable variable. [19:51] And that's our invariant which we can [19:54] set once. So, the barrier is either open [19:57] or it's closed. And when we set it, we [19:59] don't care about errors, but only once [20:01] it can be set to to true. So, what it [20:05] does is that we will [20:08] start booting all web [20:10] uh roles container, and the first one [20:13] that boots correctly, that will open the [20:15] barrier, and then let everybody [20:18] everything else boots up. So, only one [20:21] is needed, and everything gets gets [20:23] switched over, and the switch happens [20:25] like we talked before. [20:27] Uh how it looks in code? So, we wait [20:30] at barrier if we are curer, which means [20:32] we are not web role. [20:34] Starting new version. [20:36] And of course, if we are web role, that [20:39] we are the gatekeeper, so we release the [20:41] barrier. [20:42] Then we can stop it. The start new [20:44] version. [20:46] Uh also what you would expect, we upload [20:49] the environment, [20:50] uh [20:51] follow the new uh [20:54] secrets. Uh [20:56] we run uprun, [20:58] which means we are starting the [20:59] container, and then if we run if we are [21:02] running the proxy, we let the proxy to [21:04] actually do the change with up-deploy. [21:09] Now, for running deploys, of course, [21:10] Caddy [21:11] makes it easy to run many many servers. [21:14] So, we just slice them up, you know, you [21:16] probably seen it in the config with the [21:18] limit. And this is just in the code, so [21:20] we can provide a number, digit, or [21:23] percentage, and this will work. [21:27] And [21:29] uh finally, for final dive today, we [21:32] have [21:33] uh [21:34] we need to learn how to proxy the [21:36] request, right? [21:38] This is a bit of a go territory. [21:40] It's not to be. It should seem to be the [21:42] topic of of this conference. [21:45] Uh [21:47] but don't worry, because I will teach [21:49] you go [21:50] just now. [21:52] Um so, if you heard about go when it was [21:56] released, it was all about go routines [22:00] and channels, right? So, what are go [22:01] routines? They are concurrent functions [22:05] uh that are scheduled scheduled by go [22:08] scheduler, so they happen automatically [22:11] all over the all available threads. It [22:14] can be one or many CPUs, and it will all [22:17] work. [22:18] So, essentially, there will be somewhere [22:20] in the code will be go keyword. That's [22:22] maybe why go language is named. [22:25] And so, for example, here in the net [22:27] HTTP package, if you are starting the [22:30] new request, it will be like go serve [22:32] and so on. It's simplified. But if you [22:34] go to Caddy proxy code, we will not [22:36] actually find the go keyword at all, [22:39] because it will use this package, [22:41] then it will run the middleware there, [22:43] and then this handler will be called. [22:46] And part of this handler, we will simply [22:48] find the right service for the request [22:50] and service. [22:54] And the second feature of go that's so [22:57] important are channels. So, if you have [22:59] so many running functions, maybe they [23:01] want to coordinate something. [23:03] So, essentially it's something like [23:05] first in first out QE, let's say. Uh [23:09] the go routines can write the channels [23:12] or they can read from channels. Uh we [23:14] don't know which go routine picks up any [23:17] value in the channel. [23:19] Uh [23:20] but if the channel is closed, uh all go [23:22] routines will know it. [23:25] So, first thing that's implemented like [23:26] that is health check because we have [23:28] ticker C and the channel, the values in [23:31] the channels are actually just the [23:32] interval. [23:34] The our specified interval. And uh when [23:36] the channel gets new value, we can [23:38] uh go go and run the check, the actual [23:41] check, right? So, in code, we will have [23:44] a new ticker with our interval from the [23:47] configuration. And then we have this for [23:50] loop that is uh kind of running forever. [23:55] And then we have this get select, maybe. [23:58] And this uh select this very known in go [24:01] code. It's uh it's everywhere there. [24:04] So, basically these cases are waiting [24:07] for some new value or some new [24:10] information from channels. So, here, [24:13] when we get the new value in ticker C, [24:15] we just go and check it. [24:19] Then, the camel proxy also does [24:20] buffering. So, when the request is too [24:22] big, you know, this is kind of [24:24] protection against slow clients. So, [24:27] some part of that [24:29] is in memory, some part is written on [24:31] disk, but in any case, it will be like [24:34] buffered up and sent to Puma later. [24:37] Draining is maybe more interesting. So, [24:41] I told about draining, we have to drain [24:42] requests, but how does it [24:44] look like, actually? [24:47] Uh this is the same select. I just like [24:49] visualized it in different ways, so we [24:51] would have this request context done. [24:53] So, if request [24:54] is done, [24:56] uh we can let it finish. But if we have [24:58] deadline, then we have to cancel it. [25:01] In code, if we look at the drain uh [25:04] function, [25:05] first, there is something about [25:07] in-flight requests, about uh hijacked [25:10] in-flight in-flight request. So, that's [25:12] our web socket side, that's our action [25:14] cable. We just kill it immediately. We [25:17] don't uh bother to do anything. [25:19] But with the others, we have these wait [25:21] for requests to complete, like this [25:23] named block. And uh this will actually [25:26] for all these requests in parallel [25:29] uh wait for either the request is done [25:32] or [25:33] uh we are hitting a deadline. If we are [25:35] hitting a deadline, we go outside of [25:37] that uh named block, and for whatever is [25:41] remaining, we just cancel with error [25:44] draining. Yeah? [25:47] Uh and maintenance mode, that's actually [25:49] also very interesting. So, we have this [25:51] pulse channel. [25:53] And uh when we create it, all these uh [25:56] requests are waiting. These goroutines [25:58] are waiting for the channel. And when we [25:59] close it, all of them know about it and [26:02] can continue. So, we don't need any [26:04] particular value, it's just closed and [26:06] it's done. [26:07] Uh so, such state, there will be this uh [26:10] closed pulse channel. [26:12] And somewhere inside the request, we [26:14] will again have the same select just [26:15] waiting for our channels. [26:20] So, uh what we learned, uh we learned [26:23] that [26:25] uh Kamal is based on Thor and SSH kit, [26:29] how it keeps the commands as arrays [26:32] until the last moment where it has to [26:33] execute it, and then the Kamal proxy [26:36] comes in because we have to set it up [26:38] with a command to register the service, [26:40] so it can then do blue-green style [26:43] deployment, maybe coordinating many [26:45] servers. And then finally, we also uh [26:49] learned Go, so maybe uh [26:53] it's fried already. [26:54] Like a chicken sandwich. [26:56] And [26:57] yeah, I cannot stop talking about Kamal. [27:00] This is my problem. So, when I was in [27:04] California, I talked to this girl, [27:05] right? I was explaining that yeah, Kamal [27:08] is this simple deploy tool. You can [27:09] replace your Kubernetes clusters with, [27:11] you know, and then save a bunch of [27:13] money. And she was like, [27:15] "Sir, [27:16] this is a vendis?" [27:19] And [27:20] some say I'm doing America wrong with [27:22] all these fast foods. [27:24] Uh it's potentially right, but [27:27] uh the good news is that now I will only [27:29] eat Texas barbecue. [27:32] And [27:33] I will also finally drive a proper [27:36] American muscle car. [27:39] So, [27:40] not a Ferrari anymore. [27:42] Uh so, long live Kamal. [27:46] And uh God bless America. So, [27:50] uh [27:51] So, thank you. [27:52] >> [applause] [27:55] >> Enjoy the rest of the rest of the world. [27:57] >> [cheering] [28:05] [music]