Dementyev Expands Active Job to Replace Workflow Engines on Rails

Durable orchestration on Rails: From continuation to workflow - Vladimir Dementyev — Ruby on Rails

Ruby on Rails

Persisting execution state to the database transforms Active Job continuations into native workflow orchestrators, eliminating the need for fragmented job chains and external platforms like Temporal.

AI-assisted summary of Durable orchestration on Rails: From continuation to workflow - Vladimir Dementyev — original source. Check the original for full context.

Source published:

Key ideas

  • Active Job continuations, originally designed to help long-running tasks survive application deployments, are increasingly being adopted by the Rails community as an ad-hoc workflow orchestration primitive. (6:10)
  • Standard Active Job lacks framework-level support for intermediate step observability, cancellation of future-scheduled jobs, and step-level callbacks, forcing teams to rely on fragmented multi-job chains or database state machines. (11:30)
  • Vladimir Dementyev introduced active_job-durable_continuation (aj-dc), an open-source extension that brings database-backed execution persistence, step callbacks, declarative timers, and signal handling to Active Job. (30:30)
  • By delivering 'just enough orchestration' through idiomatic Rails conventions, durable continuations allow entire multi-step pipelines and human-in-the-loop workflows to live within a single job class without requiring external engines like Temporal. (33:49)

Detailed notes

## Background Published on October 1, 2026, by the official Ruby on Rails YouTube channel, this conference talk features Vladimir Dementyev of Evil Martians presenting at a Rails World event sponsored by Shopify. The session explores how Active Job continuations evolved from a deployment resilience mechanism into an orchestration primitive, introducing durable continuations to model complex workflows within standard Rails job classes.

## TL;DR Vladimir examines how Rails developers traditionally tackle complex multi-job workflows like pipelines, batches, scheduled timers, and human-in-the-loop signals, often resulting in fragmented architectures or reliance on heavy external tools like Temporal. He argues that while Active Job continuations initially emerged to survive deployment restarts, grassroots community adoption has turned them into an orchestration tool that still lacks observability, step hooks, and uniqueness. To bridge this gap, he introduces durable continuations (`active_job-durable_continuation`), which persist execution state, cursors, and callbacks directly to a database. This lightweight, Rails-native model delivers 'just enough orchestration' to handle complex sequential, delayed, and agentic workflows inside a single cohesive Active Job class.

## Atmosphere & style The presentation carries a pragmatic, seasoned engineering tone infused with Vladimir's characteristic dry wit and deadpan humor, from venting about LLM reasoning nightmares to joking about failed proposal topics. The narrative flows smoothly from architectural war stories of broken deployments and fragmented workers toward an empowering, principled defense of 'batteries included' framework design. It feels like an authoritative masterclass in Rails architecture that demystifies distributed systems jargon into clean, intuitive Ruby code.

## Bulletin - [06:10](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=370s) Active Job continuations, originally designed to help long-running tasks survive application deployments, are increasingly being adopted by the Rails community as an ad-hoc workflow orchestration primitive. - [11:30](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=690s) Standard Active Job lacks framework-level support for intermediate step observability, cancellation of future-scheduled jobs, and step-level callbacks, forcing teams to rely on fragmented multi-job chains or database state machines. - [30:30](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1830s) Vladimir Dementyev introduced active_job-durable_continuation (aj-dc), an open-source extension that brings database-backed execution persistence, step callbacks, declarative timers, and signal handling to Active Job. - [33:49](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=2029s) By delivering 'just enough orchestration' through idiomatic Rails conventions, durable continuations allow entire multi-step pipelines and human-in-the-loop workflows to live within a single job class without requiring external engines like Temporal.

## Key takeaways - [04:14](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=254s) Active Job continuations were introduced in Rails primarily as a resilience mechanism to let long-running jobs be safely interrupted and resumed across deployments without duplicating side effects. - [06:10](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=370s) Community pull requests demonstrate that developers have repurposed continuations into an orchestration primitive for complex, multi-step operations like LLM pipelines. - [08:22](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=502s) Traditional job chains fragment business logic across multiple distinct job classes, obscuring execution flow, branching logic, and error handling. - [11:30](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=690s) Native continuations store execution state only ephemerally in queue payloads, preventing external callers or monitoring tools from inspecting current step progress or execution history. - [12:26](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=746s) While gems like AcidicJob and Geneva Drive offer durable execution, they either operate outside core Active Job conventions or require adopting foreign domain-specific languages. - [13:42](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=822s) Solid Queue 1.7 introduced native Active Job batches, but parallel fan-out steps cannot yet be coordinated natively inside Active Job continuations. - [16:57](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1017s) Time-delayed background jobs suffer from invalidation challenges because Active Job provides no standard API to cancel or update scheduled jobs when domain state changes. - [20:26](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1226s) Waiting for external signals such as inbound webhooks or human approvals typically fractures cohesive workflows into disjointed controllers, mailers, and workers. - [25:35](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1535s) Durable continuations save workflow state and step transitions directly into database records, ensuring that execution survives worker crashes and restarts deterministically from checkpoints. - [28:41](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1721s) Idiomatic lifecycle hooks such as after_step callbacks and halt_on handlers allow developers to park recoverable workflows without permanently dropping them like discard_on does. - [28:41](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1721s) Declarative step timers and Active Record model associations allow long-running lifecycle schedules to reside in a single class while resetting cleanly on domain updates. - [30:30](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1830s) The wait_step primitive enables human-in-the-loop patterns and agentic conversational loops by pausing execution until a controller signal releases the step or a timeout expires. - [33:49](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=2029s) The active_job-durable_continuation gem provides a lightweight, Rails-native alternative to heavy external workflow engines by providing 'just enough orchestration' inside standard Active Job classes.

## Detailed walkthrough ### [00:10](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=10s) From Action Cable to Background Jobs

Vladimir Dementyev takes the stage, opening with warm thanks to Amanda, the Rails Foundation, and the event organizers for putting together what he jokes could be a "once-in-a-lifetime, twice-in-a-lifetime, or maybe last-in-a-lifetime event and experience." Introducing himself, Vladimir shares that he has been building software with Ruby on Rails and actively contributing to the framework for more than a decade. Over that time, he has authored numerous gems and published a book on Rails architecture. His most recent contributions to core Rails centered on re-architecting Action Cable for upcoming releases to cleanly support alternative server backends—most notably Async Cable—without relying on monkey-patches, alongside a suite of performance and architectural improvements. However, since that particular talk proposal was not selected, he quips that he has to leave Action Cable behind and move to a different topic entirely.

Vladimir introduces his day job as backend lead at Evil Martians, directing a team of engineers helping startups and scaled enterprise companies build software with higher reliability. Most recently, Evil Martians partnered with the Rails Foundation on the Rails AI benchmarks project, which has consumed his past month to the point where his "nightmares are full of LLM session transcripts full of shitty reasoning." Shifting gears away from his own job, he announces the true subject of the presentation: the audience's jobs—specifically, background jobs in Rails.

[02:39](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=159s) The Evolution of Active Job (Rails 4.2 to 8.2)

Looking back over the decade from Rails 4.2 to 8.2, Vladimir traces the evolution of Active Job from its inception in 2014. That was the same year he shipped his very first production Rails application. When Active Job was initially unveiled, his reaction was lukewarm: while it was great that Rails had added background job support out of the box, it felt like nothing more than a thin compatibility layer wrapped around existing tools like Sidekiq and Resque without introducing any novel features. In fact, that original 2014 monolith is still running in production today as a massive enterprise application, and it still does not use Active Job.

Vladimir points out that many legacy applications remain stuck using adapter-specific primitives directly, such as raw Sidekiq workers. He strongly encourages developers to migrate to Active Job because it is no longer just a generic interface facade. Over the past decade, Active Job has steadily matured, introducing foundational abstractions of its own. In the past year, Active Job added a concept that did not exist in any underlying background engine: continuations.

- Active Job was introduced in Rails 4.2 (2014) as a unified wrapper over gems like Sidekiq, Resque, and Delayed Job. - For years, it lagged behind native engine capabilities and offered little reason for mature applications to migrate away from native workers. - The recent arrival of Active Job continuations marked the first time Rails introduced a novel execution primitive that did not originate in the underlying queue engines.

[04:14](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=254s) Continuations: Resilience and Surviving Deployments

Continuations were introduced into Active Job strictly as an infrastructure resilience mechanism. Their intended goal was to make long-running jobs interruptible and resumable by breaking execution into discrete steps tracked by checkpoints and cursors. As documented in the original pull request, the primary driving motivation was surviving application deployments.

To illustrate the problem, Vladimir presents a standard webhook dispatch worker iterating over thousands of records. In a traditional worker without continuations, if a deployment rolls out mid-iteration, the worker process is dropped. When the job processing engine retries the job later, the iteration restarts from the very beginning, leading to duplicate webhook deliveries. For years, Rails developers worked around this by splitting the logic into multiple files: a parent orchestrator job that iterated over records and enqueued hundreds of tiny, atomic per-record jobs. While this fan-out workaround was robust, it was verbose and fragmented cohesive business logic across multiple classes. Continuations solve this by allowing the entire resumable loop—with its cursors and checkpoints—to live inside a single class while ensuring that mid-deploy restarts resume precisely where they left off.

```mermaid flowchart TD accTitle: Deployment Interruption Comparison accDescr: Contrasts traditional background job restart behavior against continuation-based resumption during deployments.

subgraph Traditional["Traditional Background Job"] T1["Process Records 1..50"] --> T_DEP["Deployment / Process Killed"] T_DEP --> T2["Engine Retries Job"] T2 --> T3["Restarts at Record 1<br>(Duplicate Side-Effects)"] end

subgraph Continuations["Active Job Continuation"] C1["Step 1: Checkpoint Saved"] --> C_DEP["Deployment / Process Killed"] C_DEP --> C2["Engine Retries Job"] C2 --> C3["Resumes from Checkpoint<br>(Zero Duplicates)"] end ```

- Without continuations, killed long-running jobs restart from zero, risking duplicate side effects like duplicate outgoing webhooks. - The standard multi-job workaround (orchestrator job enqueuing individual atomic record jobs) adds boilerplate and spreads cohesive logic across multiple files. - Active Job continuations preserve idempotency and resilience within a single job file by persisting cursors as execution checkpoints.

[06:10](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=370s) An Accidental Orchestration Primitive

Paired with Solid Queue, Active Job is now defining modern conventions for background processing in Ruby on Rails. Examining recent community pull requests against Active Job, Vladimir highlights that roughly half of them focus on continuations, submitted not by Rails Core or framework maintainers, but by third-party community developers. This reveals a rapid, grassroots adoption of the feature just one year after its release.

More importantly, how the community actually uses continuations has diverged completely from the authors' original intent. Developers are not reaching for continuations simply to survive deploy restarts; they are using them to coordinate long-running, multi-step business operations requiring isolated state or execution fairness guarantees. Vladimir shares a snippet he relies on in modern AI projects where multi-phase LLM workflows are split into individual continuation steps, allowing worker processes to interleave steps and distribute heavy workloads fairly across the pool. Consequently, a feature created as an infrastructure workaround has effectively transformed into an orchestration primitive—yet it remains too primitive to handle complex workflow patterns cleanly.

[08:22](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=502s) Pipelines and Job Sequences: Why the Primitive Falls Short

To demonstrate the shortcomings of continuations as an orchestration tool, Vladimir examines pipelines and job sequences—common architectural patterns used for file ingestion, ETL data imports, and external synchronization. The standard implementation relies on a chain of distinct job classes: `Job A` executes, and upon completion enqueues `Job B`, which on success enqueues `Job C`, with error handling and branching logic scattered across callbacks. Within a month of writing such a chain, nobody on the team can trace where the flow begins, what path it follows, or how failure states branch out.

Refactoring that pipeline into a single continuation job brings all sequential steps into one file, drastically improving readability. However, major architectural limitations immediately surface:

```mermaid flowchart LR accTitle: Job Chains versus Single-Job Continuation Pipelines accDescr: Compares scattered multi-job pipelines against unified continuation steps inside a single job.

subgraph Chain["Traditional Job Chain (Scattered)"] J1["ExtractJob"] -->|"enqueues"| J2["TransformJob"] J2 -->|"enqueues"| J3["LoadJob"] end

subgraph Continuation["Continuation Job (Single File)"] direction TB S1["step:extract"] --> S2["step:transform"] S2 --> S3["step:load"] end ```

First, to pass data between continuation steps, developers are forced to persist intermediate step states onto database models, turning Active Record records into makeshift orchestration state machines. Second, alternative patterns that use libraries like `activejob-performs` (such as those used in Evil Martians' AnyCable Plus project) introduce sprawling state machine definitions just to re-enqueue a single job across distinct methods.

While Active Job's new attributes API allows keeping ephemeral state inside the job payload across continuations, that payload state remains completely inaccessible to external callers or monitoring tools. Finally, Active Job lacks control-flow step hooks: there are no per-step callbacks (such as `after_step`) and no step-level rescue handlers. Developers must manually write boilerplate—such as invoking Action Cable `broadcast_update` calls—inside every individual step block.

[11:30](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=690s) Limitations of Continuations: Observability and Uniqueness

While continuations offer better developer experience and improve the readability of complex multi-step processes, they still lack critical framework-level capabilities today. The primary missing piece is observability. Because native continuations do not maintain persistent state, there is no way to inspect which step a job is currently executing, how long it has been running, or what the execution history looks like across individual steps.

Beyond step observability and retry logic, long-running workflows inevitably run into the problem of uniqueness constraints. In real-world orchestration, you rarely want duplicate runs of the same multi-step workflow enqueued simultaneously while a previous run is still actively working. Ideally, attempting to enqueue a duplicate should either be rejected or trigger a cancellation and re-enqueue of the active job. However, Active Job provides no native, framework-level API to enforce uniqueness or manage cancellations. While specific queue backends offer proprietary locks or uniqueness add-ons, the lack of a standardized Active Job primitive forces developers to roll their own solutions.

[12:26](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=746s) Existing Workflow Gems: AcidicJob and Geneva Drive

To fill these gaps, the Ruby ecosystem has produced several third-party workflow engines over the years. One notable solution is AcidicJob, which Vladimir notes having used across several client projects. AcidicJob introduces durable state by writing every execution transition directly to the database. This allows developers to restore state, inspect progress, and rely on built-in idempotency and uniqueness guards. Crucially, AcidicJob preserves the familiar Active Job interface: external callers enqueue standard jobs, while the durability mechanics and workflow state remain cleanly encapsulated inside the job class.

Another notable modern entry is the Geneva Drive project. Vladimir praises Geneva Drive for providing virtually every feature expected from a mature workflow engine. However, its primary downside from a Rails perspective is that it operates as an entirely separate abstraction:

> "It's not a Rails thing. It's a custom DSL, and it's a custom entity. It's not strictly compatible with Active Job, so you need to refactor everything to use it."

While Geneva Drive provides robust workflow guarantees, Rails applications benefit far more from tools that integrate directly into the existing Active Job primitives rather than requiring a wholesale architecture rewrite.

[13:42](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=822s) The Batching Pattern and Solid Queue Batches

The next fundamental orchestration pattern is batching: a setup job initializes work, fans out into multiple parallel child jobs, and coordinates a final callback once all child jobs finish. While batches can technically be processed sequentially using continuation cursors—as seen in real-world Spree codebase examples processing CSV rows—batch processing in practice almost always implies parallel execution.

```mermaid flowchart TD accTitle: Batch Coordination Architecture accDescr: Setup job fans out to multiple parallel worker jobs, which report to a coordinator before triggering completion.

subgraph BatchStage["Batch Fan-out & Execution"] Setup["Setup Job"] --> JobA["Worker Job A"] Setup --> JobB["Worker Job B"] Setup --> JobC["Worker Job C"] end

subgraph CoordStage["Coordination & Completion"] JobA --> Coord["Coordinator / Counter"] JobB --> Coord JobC --> Coord Coord --> Complete["On Success / On Failure Job"] end ```

Parallel fan-out introduces the hard problem of coordination. To know when all children have completed, systems traditionally rely on atomic Redis counters or recurring polling sweeps that periodically re-check whether every worker is finished. Historically, this demanded specialized third-party gems, but Solid Queue 1.7 introduced native batch support—a massive achievement credited to outside contributor JP Camara alongside maintainer Rosa Gutiérrez:

- **Solid Queue Batches:** Built entirely around standard Active Job conventions, allowing explicit completion hooks like `on_success` and `on_failure` callback jobs. - **GoodJob Influence:** The API draws architectural inspiration from GoodJob, which has supported native database-backed batches for some time. - **Doximity's Simplekiq:** A Sidekiq-based gem highlighting a syntax very close to Active Job continuations, featuring sequential steps mixed with explicit parallel step blocks. - **Framework Gap:** Parallel steps remain missing from native Active Job continuations because Active Job itself lacks an upstream batching primitive, meaning developers must wait for framework-level batch APIs before continuations can natively coordinate fan-out.

[16:57](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1017s) Timers, Delays, and the Pain of Scheduled Jobs

Orchestration frequently involves timers, delays, and schedule-driven background tasks such as expiration notices, periodic reminders, and follow-up notifications. In an example from AnyCable's license expiration logic, a single domain workflow ended up fragmented across numerous recurring polling jobs. A cron job ran every ten minutes or hourly just to find expiring records and enqueue follow-up background jobs, introducing unnecessary structural boilerplate for what should be a unified linear flow.

While continuations allow consolidating iterative polling sweeps into a single job class via cursors, developers still struggle when attempting to "park" a job into the future using Active Job's `wait:` or `wait_until:` parameters. The core issue is invalidation:

> "The first problem is that we usually need to invalidate—'Oh, is this job still relevant?'—because there is no, at least Active Job level, API to cancel a job scheduled for the future."

Reaching into adapter-specific APIs to purge scheduled job payloads is brittle, driving most teams back to blunt cron sweeps. Several gems try to improve this: - **Conductor / ChronoForge:** Extends Active Job with DSL decorators like `wait`, allowing an execution to pause at a specific line, record itself in a database table, and re-enqueue seamlessly when the deadline arrives. - **Geneva Drive Timers:** Offers step-level timer configurations and associates workflows directly to ActiveRecord models using handles, providing clean lookup APIs to cancel pending timers whenever records update. - **Temporal:** Takes a durable workflow approach where appointment reminder deadlines can simply be modified in-flight as state updates, avoiding the need to cancel and restart workflows from scratch when target dates shift.

[20:26](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1226s) Human-in-the-Loop, Signals, and Webhook Workflows

A broader class of delayed workflows revolves around external signals: events that pause execution not for an elapsed duration, but for an external stimulus such as human approval or an inbound webhook. In typical Rails codebases, waiting for a signal forcefully fractures a single cohesive process into disconnected before-and-after fragments scattered across the application.

```mermaid flowchart LR accTitle: Signal-Driven Webhook Fragmentation accDescr: Comparison of a fragmented three-part webhook flow versus a consolidated continuation-style perform method.

subgraph Fragmented["Fragmented Architecture (Status Quo)"] direction TB J1["1. Request Withdrawal Job"] --> W1["2. External Webhook Controller"] W1 --> J2["3. Process Completion Job"] end

subgraph Unified["Unified Continuation Architecture"] direction TB Step1["Step 1: Request withdrawal"] --> Step2["Step 2: wait_for_signal (payment_receipt)"] Step2 --> Step3["Step 3: Credit account"] end ```

In Mastodon, approval flows are split across separate controllers, mailers, and workers, requiring manual ad-hoc cleanup jobs to purge unapproved tasks that never receive their signal. Similarly, in a withdrawal workflow Vladimir encountered in production, the logic was fragmented across three separate files: an initial withdrawal request job, an HTTP webhook controller receiving the vendor's confirmation, and a third background job applying balance credits.

Tools supporting signal primitives—such as Conductor with its `continue_if` step—allow developers to consolidate that entire distributed transaction into a single readable `perform` method: run step one, pause execution until the payment receipt signal arrives, and execute step three. Yet despite the clear readability benefits of writing this inside Ruby, teams frequently bypass Rails entirely and reach for heavy external orchestrators like Temporal. Vladimir laments this trend, arguing that these business flows could easily be expressed with native Rails ergonomics if the framework provided durable primitives.

[22:39](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1359s) The Philosophy of Durable Continuations

Writing workflows that can be described in plain English in just a few words should not require "hundreds lines of code and dozens of Ruby objects and scary like voodoo magic to make them all work." Yet as workflows grow more sophisticated, Rails developers are increasingly forced to reach for heavy, alien orchestration tooling. This trajectory runs contrary to the core Rails ethos of having "batteries included." Rather than relying on separate orchestrators, external services, or cumbersome proprietary SDKs, Active Job's existing continuation concept can be extended directly into a full-fledged workflow engine.

A year prior, Vladimir floated the idea of evolving continuations into a true orchestration primitive without adding custom gems or bloated external abstractions. The outcome of that concept is **durable continuations** for Active Job. While industry buzzwords—frequently pushed by products like Temporal—often dress durability up as mysterious, advanced tech, the underlying principle is simple: the execution state is saved as records in a database or dedicated store rather than existing purely as a transient payload in a message queue. Surviving hard kills (`SIGKILL`) and node restarts naturally follows once execution state is committed, restorable, and replayable. Crucially, any such extension must adhere strictly to "the Rails way"—feeling like a natural part of the framework that could conceivably be upstreamed into Rails core rather than functioning as a foreign third-party add-on.

```mermaid flowchart LR accTitle: Transient Queue Payload vs Durable Database Execution accDescr: Comparison showing how transient queue payloads lose state on crashes while durable continuations persist execution state to a database. subgraph Traditional["Standard Active Job"] direction TB Q["Queue Payload"] --> W["Worker Memory"] W -.->|"SIGKILL / Crash"| Lost["State Lost"] end subgraph Durable["Durable Continuation"] direction TB DB[("Execution Run DB")] <--> WR["Worker Execution"] WR -->|"Checkpoint Step"| DB DB -->|"Crash Recovery"| Resume["Resume from DB State"] end ```

---

[25:35](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1535s) Persisting State and Resuming Execution

To illustrate how durable execution works in practice, consider an avatar and card generation pipeline from a conference app. Generating customized attendee cards requires sequentially processing original photo uploads, moderating content against inappropriate uploads, and finally generating the finished visual assets. In standard Active Job continuations, state tracking across these intermediate phases is largely ephemeral.

Under durable continuations, every job execution run maps to a dedicated database record that tracks overall status, individual steps, and arbitrary run-level attributes:

* **Enqueued State:** The moment a job is enqueued, a record is written to the database; the system already recognizes the job's existence and planned arguments before worker execution begins. * **In-Progress Execution:** As background workers pick up the job, the run status updates to running, and each step's progress and results are committed sequentially. * **External Observability:** Persisted job attributes remain readable by outside processes throughout execution, allowing other application features to query run data, track percent completion, or trigger downstream operations. * **Direct Inspection and Control API:** Similar to the developer experience of tools like GoodJob, jobs expose an intuitive API to query execution state by identifiers (inferred automatically from arguments or explicitly declared). Runs can be inspected, canceled, or manually retried.

This model significantly hardens resumability. In an un-durable webhook dispatcher job that iterates through subscribers, an unexpected hard crash or unhandled exception between steps will restart the job from the very beginning if the progress cursor was never committed to a database. Under durable continuations, the cursor and step metadata are committed to the database, ensuring that when the worker restarts, execution picks up exactly where it was interrupted.

---

[28:41](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1721s) Rails Ergonomics: Callbacks, Halting, and Timers

Durable continuations elevate workflow definitions from bare-bones background wrappers into rich, first-class Rails objects. Instead of cluttering the bodies of individual step methods with logging, metrics, or auxiliary notifications, developers can leverage familiar lifecycle hooks:

* **`after_step` Callbacks:** Provide idiomatic Rails-style hooks to run cross-cutting logic cleanly after any step finishes. * **`halt_on` Handler:** Unlike Active Job's standard `discard_on` (which permanently drops the entire job upon failure), `halt_on` gracefully parks the workflow after a specific step. If the underlying issue is recoverable, the parked workflow can be cleanly resumed once the environment or data is repaired. * **Built-in Uniqueness:** Because each active execution run is stored in the database, workflow uniqueness can be enforced natively without third-party Redis locks or custom plugins.

```mermaid flowchart TD accTitle: Durable Step Lifecycle and Error Handling accDescr: Flow diagram showing step execution, after_step callbacks, halt_on error handling, and resumption. Start["Job Step Invoked"] --> Run{"Execute Step"} Run -->|"Success"| CB["Run after_step Callbacks"] CB --> Next["Advance to Next Step"] Run -->|"Recoverable Error"| Halt["halt_on: Park Workflow"] Halt -->|"Condition Resolved"| Resume["job.resume"] Resume --> Run ```

Timers and deferred scheduling also condense dramatically. Traditional multi-stage schedules—such as a software license expiration reminder lifecycle—historically required external cron managers triggering multiple independent job classes. With durable continuations, the entire multi-month sequence lives inside a single class using declarative step delays:

* Steps support explicit timing constraints such as `wait:` (duration) and `wait_until:` (specific timestamp). * If a customer renews their subscription mid-lifecycle, re-enqueuing the unique job automatically replaces the pending schedule and resets the lifecycle workflow. * ActiveRecord models can integrate directly via helpers like `has_one_performed`, exposing seamless associations between domain records and their associated durable background runs.

---

[30:30](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1830s) Signals, Human-in-the-Loop, and the Open-Source Release

Complex asynchronous orchestrations often require waiting on external events or manual human intervention. In heavyweight workflow engines like Temporal, implementing signals and pauses requires coordinating multiple external concepts and verbose SDK boilerplate. Durable continuations condense this into standard, readable Ruby code.

In a moderation pipeline where an AI image generator yields a borderline safety score, automated processing must pause until a human reviewer manually approves or rejects publishing the badge to social ads. Traditional Rails implementations lean on messy status enums, polling loops, or fragmented state machines that struggle with concurrency, lack execution history, and cannot enforce review deadlines:

```mermaid flowchart LR accTitle: Human in the Loop Approval Flow accDescr: Pipeline workflow showing automated moderation, a parked wait step awaiting approval, and subsequent controller signal resumption. Mod["Moderation Check"] --> Review{"Borderline Score?"} Review -->|"Yes"| WaitStep["wait_step: Parks Job Execution"] WaitStep -->|"Timeout Exceeded"| Expired["Handle Timeout / Fallback"] Review -->|"No"| Gen["Generate Assets"] Controller["Admin Controller Signal"] -.->|"Approve / Release"| WaitStep WaitStep --> Gen ```

* **`wait_step` Primitive:** Parks the job for a configurable timeout period using empty-step timer mechanisms. If no input arrives before the deadline, it releases automatically. * **Signal Dispatch:** When an administrator approves the action, a simple controller call emits a signal to the waiting execution run, releasing the park and resuming downstream steps immediately. * **Agentic Workflows (RubyLLM 2.0):** Emerging conversational workflows in libraries like RubyLLM frequently require human-in-the-loop loops for approving tool calls or external actions. Instead of juggling disparate background workers, the entire conversational turn loop can be halted with `halt_on` and resumed on approval. While chat messages hold the application-level state, durable continuations manage the underlying control flow state.

Vladimir wraps up by introducing the open-source gem: `active_job-durable_continuation` (or `aj-dc`—humorously clarifying it has "nothing related to music"). Beyond standard workflow features, the gem bundles ready-made AI development skills powered by Rails Hyperdrive, allowing coding agents to inspect existing Rails jobs, identify migration candidates, and refactor traditional background workers into durable continuations automatically.

[33:49](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=2029s) The Rails Way: Bringing "Just Enough Orchestration" to Active Job

Vladimir concludes by encouraging attendees to review the implementation details in their own time, deadpanning that he certainly does not "recommend not reading code." The driving philosophy behind introducing durable continuations to Active Job is pragmatic restraint: delivering "just enough orchestration" so teams can stay true to the Rails way. Rather than taking on the operational overhead of heavy external orchestration engines or introducing cryptic gems to stitch together complex multi-job flows, Rails developers can solve these problems natively within the framework they already know, creating solutions that are cleaner, simpler, and far better integrated.

With additional workflow features already on the roadmap, Vladimir leaves the audience with a clear call to action. He urges engineers to pay closer attention to Active Job continuations and inspect their existing codebases for brittle multi-job patterns—such as manual pipelines, batching hacks, or ad-hoc polling loops—that would be better modeled as cohesive workflows. By consolidating that logic into durable workflows, developers can make long-running processes reliable, deploy-safe, and straightforward to maintain.

```mermaid flowchart LR accTitle: Adopting Durable Workflows in Rails accDescr: Three-step progression from identifying fragmented jobs to adopting durable workflows natively in Active Job. A["Identify Multi-Job Patterns<br/>(Pipelines, Batches, Timers)"] --> B["Extract into Workflows<br/>(Stay on the Rails Way)"] B --> C["Leverage Durable Continuations<br/>(Active Job Primitive)"] ```

- **Avoid External Sprawl:** Eliminates the need to reach for hosted workflow platforms or specialized, unfamiliar gems for standard background job orchestration. - **Audience Takeaway:** Audit existing background job chains, look for hidden orchestration needs, and experiment with durable continuations to keep full end-to-end workflows contained within standard job classes.

## Notable quotes > [00:10](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=10s) once-in-a-lifetime, twice-in-a-lifetime, or maybe last-in-a-lifetime event and experience — Vladimir Dementyev

> [00:10](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=10s) nightmares are full of LLM session transcripts full of shitty reasoning — Vladimir Dementyev

> [12:26](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=746s) It's not a Rails thing. It's a custom DSL, and it's a custom entity. It's not strictly compatible with Active Job, so you need to refactor everything to use it. — Vladimir Dementyev

> [16:57](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1017s) The first problem is that we usually need to invalidate—'Oh, is this job still relevant?'—because there is no, at least Active Job level, API to cancel a job scheduled for the future. — Vladimir Dementyev

> [22:39](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1359s) hundreds lines of code and dozens of Ruby objects and scary like voodoo magic to make them all work — Vladimir Dementyev

> [22:39](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1359s) batteries included — Vladimir Dementyev

> [30:30](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=1830s) nothing related to music — Vladimir Dementyev

> [33:49](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=2029s) recommend not reading code — Vladimir Dementyev

> [33:49](https://www.youtube.com/watch?v=4dcv1a7eeMk&t=2029s) just enough orchestration — Vladimir Dementyev

## Mentioned Vladimir Dementyev, Amanda, Rails Foundation, Shopify, Ruby on Rails, Action Cable, Async Cable, Evil Martians, Active Job, Solid Queue, Sidekiq, Resque, Delayed Job, activejob-performs, AnyCable, AnyCable Plus, AcidicJob, Geneva Drive, Spree, JP Camara, Rosa Gutiérrez, GoodJob, Doximity, Simplekiq, Conductor, ChronoForge, Temporal, Mastodon, Active Record, RubyLLM, Rails Hyperdrive, active_job-durable_continuation

## Video description Active Job continuations were built to help long-running jobs survive deploys. Now they're becoming an orchestration primitive. Vladimir Dementyev (Evil Martians) looks at common multi-job patterns like pipelines, batches, timers and human-in-the-loop approvals, and at the tools Rails developers reach for today. He then introduces durable continuations for Active Job, a Rails-way extension that adds persisted state, step callbacks, uniqueness, timers and signals, so an entire workflow can live in a single job class.

***

Thank you Shopify for sponsoring the editing and post-production of these videos. Check out insights from the Engineering team at: https://shopify.engineering/

#RailsWorld #RubyOnRails #ActiveJob #BackgroundJobs #Workflows #SolidQueue

[Source](https://www.youtube.com/watch?v=4dcv1a7eeMk)

中文简报

Dementyev扩展Active Job以替代Rails工作流引擎 将执行状态持久化至数据库，使Active Job延续机制转变为原生工作流编排工具，无需零碎的任务链以及Temporal等外部平台。 Vladimir探讨了Rails开发者传统上处理流水线、批处理、定时调度以及人工介入信号等复杂多任务工作流的方式，指出这些方式往往导致架构碎片化，或过度依赖Temporal等庞大的外部工具。他认为，虽然Active Job延续机制最初是为了应对部署重启而出现，但社区自发的采纳已将其变成了一种编排工具，只是目前仍缺乏可观测性、步骤钩子和唯一性保障。为了填补这一空白，他推出了持久延续机制（`active_job-durable_continuation`），可将执行状态、游标和回调直接持久化到数据库中。这种轻量级的Rails原生模型提供了“恰到好处的编排能力”，仅在单个内聚的Active Job类中即可处理复杂的顺序、延迟及智能体工作流。 Active Job延续机制最初旨在帮助长时间运行的任务在应用部署中存活，如今正日益被Rails社区作为一种临时工作流编排原语所采用。 (6:10) 标准Active Job在框架层面上缺乏对中间步骤可观测性、未来计划任务取消以及步骤级回调的支持，迫使团队依赖零碎的多任务链或数据库状态机。 (11:30) Vladimir Dementyev推出了开源扩展active_job-durable_continuation (aj-dc)，为Active Job带来了基于数据库的执行持久化、步骤回调、声明式定时器和信号处理能力。 (30:30) 通过符合惯用Rails规范的方式提供“恰到好处的编排”，持久延续机制使完整的多步骤流水线和人工介入工作流能够保存在单个任务类中，无需Temporal等外部引擎。 (33:49)

Reading summary

Vladimir Dementyev argues that Active Job continuations—originally designed as an infrastructure mechanism to survive deployment restarts—are evolving into an orchestration primitive that currently lacks the durability, observability, and control flow needed for complex workflows. To solve this without adopting heavy external orchestration engines, Dementyev proposes extending Active Job with database-backed durable continuations directly within the Rails framework. From Deployment Resilience to Accidental Orchestration (4:14) Active Job debuted in Rails 4.2 (2014) as a unified wrapper around adapters such as Sidekiq and Resque, but for years it offered little reason for mature applications to abandon native adapter features. According to Dementyev, the introduction of continuations marked the first time Rails introduced a novel execution primitive absent from underlying engines. Continuations were initially created to make long-running jobs interruptible across application deployments through checkpoints and cursors, preventing duplicate side effects like repeated webhook dispatches without requiring verbose multi-file fan-out patterns. However, Dementyev observes that community adoption quickly diverged from this narrow operational goal. Developers began using continuations to coordinate sequential business operations, AI processing workloads, and execution fairness across worker pools. While this grass-roots shift turned continuations into an orchestration primitive, Dementyev emphasizes that the native feature remains too low-level to support complex workflow patterns cleanly. Structural Deficits in Native Continuations (8:22) When used for multi-step pipelines, ETL imports, or external synchronizations, native continuations expose major architectural limitations. Dementyev explains that developers must often persist intermediate step states onto Active Record models, turning domain tables into makeshift state machines. While Active Job's attributes API retains data across continuations, that payload state remains inaccessible to external monitoring tools. Furthermore, native continuations lack per-step lifecycle hooks (such as `after_step`), step-level rescue handlers, and execution history tracking. These limitations also affect scheduling, batching, and external event handling. Active Job lacks framework-level APIs to enforce job uniqueness or invalidate scheduled timers when domain data changes. Coordinating parallel batches still depends on queue-specific tooling like Solid Queue 1.7 or GoodJob rather than upstream Active Job primitives. In addition, handling external signals—such as human approvals or inbound webhooks—typically fragments cohesive business logic across disconnected background jobs, mailers, and HTTP controllers, often driving teams to adopt external platforms like Temporal. Database Persistence and Resumable Execution (22:39) To address these shortcomings, Dementyev advocates for "durable continuations." Rooted in the principle that execution state must be committed to a database rather than existing purely as transient queue payloads, durable continuations ensure background processes survive unexpected node crashes, restarts, and `SIGKILL` signals. Every execution run maps to a dedicated database record tracking overall status, job arguments, step-level progression, and run-level attributes. Under this model, the system registers a job's existence before a worker picks it up. External callers gain observability through dedicated inspection APIs, allowing outside processes to monitor progress percentages, inspect step metadata, or trigger manual retries and cancellations. If a worker fails mid-process, the persisted checkpoint guarantees that the retried job resumes precisely from the interrupted step rather than restarting from the beginning. Rails Ergonomics, Signals, and Open Source (28:41) Dementyev outlines how durable continuations bring idiomatic Rails conventions to workflow management. Developers can attach `after_step` lifecycle callbacks for cross-cutting logic, park recoverable failures using a `halt_on` handler rather than permanently dropping jobs via `discard_on`, and enforce execution uniqueness natively. Declarative timing controls (`wait:` and `wait_until:`) allow linear scheduling inside a single class, while Active Record associations like `has_one_performed` link domain records directly to their execution runs. For asynchronous processes requiring external input, the model introduces a `wait_step` primitive. This mechanism parks execution until an external signal—such as an administrator's approval or a webhook confirmation—is emitted from a controller, or until a configurable timeout expires. Dementyev released this architecture in the open-source gem `active_job-durable_continuation` (`aj-dc`), which also includes automated migration tooling built on Rails Hyperdrive to help refactor legacy multi-job chains. Dementyev advocates for "just enough orchestration" within Rails, maintaining that developers can avoid the operational overhead of third-party workflow engines by expanding Active Job. He urges teams to inspect their codebases for fragmented patterns—including manual job chains, ad-hoc polling sweeps, and brittle batching logic—and consolidate them into durable, deploy-safe workflows native to the framework.

Source quotes

[0:10] Vladimir Dementyev: once-in-a-lifetime, twice-in-a-lifetime, or maybe last-in-a-lifetime event and experience [0:10] Vladimir Dementyev: nightmares are full of LLM session transcripts full of shitty reasoning [12:26] Vladimir Dementyev: It's not a Rails thing. It's a custom DSL, and it's a custom entity. It's not strictly compatible with Active Job, so you need to refactor everything to use it. [16:57] Vladimir Dementyev: The first problem is that we usually need to invalidate—'Oh, is this job still relevant?'—because there is no, at least Active Job level, API to cancel a job scheduled for the future. [22:39] Vladimir Dementyev: hundreds lines of code and dozens of Ruby objects and scary like voodoo magic to make them all work [22:39] Vladimir Dementyev: batteries included [30:30] Vladimir Dementyev: nothing related to music [33:49] Vladimir Dementyev: recommend not reading code [33:49] Vladimir Dementyev: just enough orchestration

Original transcript

[0:06] [music] [0:10] >> All right, [singing] [0:11] everyone, [0:12] glad to be on the RailsConf stage again. [0:15] Uh, thanks to Amanda, the Rails [0:17] Foundation, and all the organizers for [0:18] making this conference a [0:20] once-in-a-lifetime or [0:22] twice-in-a-lifetime for someone, maybe [0:24] last-in-a-lifetime event and experience. [0:27] Uh, [0:28] so, um, [0:30] my name is Vladimir. [0:33] And I've been building with [0:35] Ruby on Rails for more than 10 years, [0:38] and about the same amount of time I've [0:40] been contributing to Rails. [0:42] And, uh, over the years I [0:45] managed to publish a lot of gems. You [0:46] probably know a few of them. Some of [0:48] them were mentioned. Even wrote a book [0:50] you probably read, or you haven't read. [0:52] Ask it. By the way, is it good? [0:55] And, um, most recently my Rails [0:58] contributions were all around Action [1:00] Cable. [1:01] And, uh, with the next release, uh, it's [1:04] been re-architected to support [1:06] different, uh, server implementations [1:08] without hacks, like Async Cable, the [1:11] most [1:12] interesting one. And on top of that [1:14] architecture, some fixture fixes and [1:18] improvements have been already prepared, [1:20] and also [1:21] expecting to land in the future [1:23] releases. [1:24] So, but that's not what I'm going to [1:26] talk about today, because that proposal [1:27] hasn't been selected. So, that's it. Uh, [1:30] next topic. [1:33] Sorry. [1:34] I'm first time using this clicker. [1:36] Uh, so, a few words about my job. I'm a [1:40] leading back end team, uh, full of [1:42] brilliant Ruby and not only Ruby [1:44] engineers at Evil Martians, makers on [1:48] our own my team, sorry. And, uh, we [1:51] helping startups and companies at scale [1:55] to you things, you know, and do that [1:58] uh with higher quality. And we also help [2:01] Rails Foundation since recently. Uh you [2:04] probably heard about this Rails AI [2:05] benchmarks project. And that's both my [2:08] primary focus for the last month. And [2:10] now my nightmares are full of LM [2:12] sessions transcript full shitty [2:14] reasoning, so I can sometimes recall [2:17] them uh when I'm awake. Don't be afraid. [2:21] So, uh but I'm not talking about my job [2:23] today, luckily. [2:25] Uh I'm talking about your jobs. Uh more [2:29] specifically, your background jobs. [2:31] And uh that's the topic of my talk [2:34] today. And I'd like to start with some a [2:36] little bit of a little bit of [2:38] retrospective. [2:39] Uh so, I wanted to show you how the role [2:44] of Active Job has changed over the years [2:47] from 4.2 to 8.2. [2:52] So, it all started in [2:54] uh 2014. The same year I shipped my [2:57] first production Rails application. And [2:59] I clearly remember the moment uh Active [3:02] Job has been announced. My thoughts were [3:03] like, "Oh, wow, that's really cool. [3:06] Finally, Rails shipped a background job [3:08] uh kind of implementation on its own. [3:11] And but I'm not going to migrate to it [3:13] because what's the point? It's just a [3:15] thin compatibility layer on top of the [3:17] existing [3:19] uh gems and doesn't add anything and [3:22] missed a lot of stuff." So, the app I [3:25] worked back then, by the way, is still [3:26] alive and now it's a large Rails [3:28] monolith and it still doesn't use Active [3:31] Job. [3:31] So, and I've seen many apps like that in [3:34] my life, unfortunately. But I encourage [3:37] you if you're still using, I don't know, [3:38] Sidekiq workers directly, please [3:40] consider migrating to Active Job because [3:42] today Active Job is no longer just a [3:46] uh common interface. Over the years, [3:49] gradually, slowly, it's been evolving. [3:52] Uh [3:53] it was bringing new solid ideas until we [3:57] got to the point where year ago a new [4:00] concept not present in background job [4:04] provide engines appeared in active job [4:07] itself. So, the framework finally [4:08] started bringing something specific on [4:11] top of [4:12] other works. [4:14] And let's talk about continuation for a [4:16] moment. What is it and why it's [4:18] important? [4:19] So, [4:20] continuation is by design a resilience [4:23] concept. It was added to make [4:26] long-running jobs interruptible and [4:29] resumable by introducing steps and [4:32] cursors that act as checkpoints. [4:35] And the original motivation for that was [4:38] surviving deployments. [4:41] And you can find it in the original PR. [4:42] That's [4:43] the actual reason why the feature was [4:45] brought to Rails. [4:46] And what does it mean surviving [4:47] deployments? Let's imagine the same [4:49] webhook dispatch job from the previous [4:50] slide, but without continuation. It [4:53] looks [4:54] pretty similar, same shape, same logic, [4:57] but different guarantees. [4:59] Whenever deploy happens in the middle of [5:02] the iteration, [5:03] the job is dropped [5:05] and it's [5:06] retried by the [5:08] background processing engine, but it [5:10] start over the iteration from the first [5:13] step and could result in sending the [5:16] same webhook event twice. [5:18] So, it's not a [5:20] cool for your application. That's why [5:22] for many years we've been using some [5:23] workarounds like this one. That's the [5:25] most popular. Having several jobs, one [5:28] is just orchestrating and queuing atomic [5:31] jobs per record. So, making this [5:34] approach robust, reliable, maybe a bit [5:37] more verbose and less cohesive than a [5:40] single job version, but it worked. Now, [5:42] let's again compare it to the [5:43] continuation version. We have everything [5:46] in one place. We have the same [5:47] guarantees. [5:49] And we have the whole workflow, let's [5:51] call it workflow, defined in a single [5:53] file. And to my taste, it's much better [5:56] than having it scattered across files. [5:58] It's more resilient to [6:00] human or agent errors, whatever you [6:02] prefer. [6:03] So, this is a canonical canonical [6:05] example of using Active Job, and this is [6:07] what it was created for. [6:10] Uh now, back to the timeline. So, as I [6:13] already said, Active Job started as a [6:15] tiny facade over other runtimes. But [6:19] over the years, it evolved into a [6:21] foundation of its own. And together with [6:24] Solid Queue, which is an important [6:25] component of the ecosystem today, uh [6:28] they lead the way for everyone on how [6:30] to, [6:31] you know, uh implement background jobs [6:34] and maintain them on Rails. But what's [6:36] coming next, right? Uh what's What [6:39] awaits us in the next version? [6:41] Uh I picked some [6:43] PRs to Active Job, some merged, some not [6:46] yet, uh but the most interesting because [6:49] they bring in new stuff to Active Job. [6:51] And half of those PRs are [6:53] continuation-related. [6:55] And what's more important, they [6:57] contributed by the community members, [6:58] not by the core team or by the core [7:00] company team members. [7:02] So, and that's an interesting fact [7:04] because continuation is just uh year [7:07] old, [7:08] but it's being adopted quickly. [7:11] But the way [7:13] regular users, like not the ones [7:14] contributed it originally, use [7:16] continuation differs from the original [7:18] intention. [7:19] It's no longer about surviving [7:21] deployments. It's about orchestrating [7:23] long-running multi-step operations with [7:26] their state or with their kind of [7:29] fairness guarantees. That's the snippet [7:31] I use a lot these days to split a [7:34] long-running LLM-related job into [7:37] multiple steps and run them in isolation [7:39] so we can more fair distribution of work [7:42] across workers. [7:44] And that means that [7:47] the teacher that was born as a work [7:49] around for infrastructure quirks [7:52] actually became an orchestration [7:54] primitive. [7:55] But this primitive is still too [7:58] primitive to be a building block for the [8:00] common job orchestration scenarios. [8:03] Uh let's look at some examples how we [8:05] coordinate multiple background jobs [8:08] today and what are the common patterns [8:11] and tools to implement them before we [8:15] and and how and how we can use [8:17] continuation for that, maybe a little [8:20] bit upgraded version of it. [8:22] So, let's start with pipelines or job [8:25] sequences. [8:27] So, this is a pattern that's common for [8:29] processing files or importing data, [8:33] syncing, whatever. It's There were many [8:35] a lot of use cases, but they have [8:37] similar shape. Uh one of the shape is a [8:40] chain of jobs. [8:42] So, we have one job performing the first [8:44] step of the operation and then on the [8:46] successful result we in queue another [8:48] job. We can terminate the process by [8:52] failing it somehow, but then there could [8:54] be another job and the number of such [8:57] jobs in a chain and possibly branching [8:59] could be enormous and it's very typical [9:02] for large Rails applications. And the [9:05] problem with that it that no one like, [9:08] you know, in a [9:09] months after implementing that, no one [9:11] knows where the start of the chain, how [9:13] it goes, what is the direction, what are [9:15] the outcomes, what are the failure modes [9:17] and so on. [9:18] So, we can refactor this chain of jobs [9:22] into a chain of steps [9:24] with continuation. We can encapsulate [9:27] the whole workflow in a single place [9:30] and there's this no longer about inter- [9:32] interruptibility and resumability, [9:34] right? This is just about organizing [9:36] code in the application. And [9:38] there are some quirks still left because [9:40] we rely on the states saved on the model [9:44] to control the execution of the steps, [9:46] which is not the best way to do that, [9:49] but we don't have any other state to [9:51] rely upon. So, it's still just a [9:53] background job running by Active Job. [9:57] Uh [9:57] another example of the kind of pipeline, [10:00] same idea, multiple jobs running one [10:02] after each other, but now [10:04] we have actually the single job which is [10:06] being re-queued to perform the next step [10:09] controlled by a separate state machine. [10:11] This is example from our AnyCable Plus [10:13] project. There is not even no even [10:16] background job class visible, right? [10:18] Because we use Active Job Performs [10:19] plugin that just turns [10:22] method into background job, very [10:24] convenient. [10:25] But the problem here is that we have too [10:27] many entities to orchestrate the work. [10:30] And moreover, [10:32] yeah, so that's how it works. And it [10:35] could be also written into a kind of [10:37] state machine to make it more clear what [10:40] the order of the step, what are the [10:41] guard guards for each step. [10:44] And it could be done today, especially [10:46] with the addition of this new attributes [10:48] API that allows us to keep some state uh [10:52] during the execution of the workflow, [10:54] but this state is not accessible [10:56] anywhere outside because it lives in the [10:58] job's payload. That's there are some [11:00] limitations. [11:01] Uh another problem that [11:03] unlike with many things in Rails, like [11:07] performing actions and whatever, we do [11:09] not have control flow hooks for steps. [11:13] There are no step callbacks, no per step [11:17] rescue handlers, and so on. So, every [11:19] step has to deal with it [11:21] manually. Like here, we do broadcast [11:23] update in every step because, well, we [11:26] need to do that. If we had like after [11:28] step broadcast, that would be much [11:30] better [11:31] uh experience and much better [11:33] readability of the whole process. [11:35] But, that's something that's missing [11:37] today. [11:38] So, yeah, uh we mentioned a couple [11:40] things uh and I [11:42] I slightly mentioned that we don't have [11:44] observability for continuation process. [11:46] We don't know which step the job is [11:48] currently in, how long has it been [11:50] running, what is the history of steps, [11:52] just because there is no persistent [11:54] state. [11:55] And the rules and steps and retries I [11:58] already talked about. One other thing [12:00] related to such workflows as uniqueness [12:03] constraint. For long-running workflows, [12:05] we typically don't want them to be [12:07] re-enqueued multiple times when the [12:08] previous one is still running. Or if [12:10] it's still running, it's better to [12:11] cancel it and re-enqueued. But, there is [12:14] no such API in Active Job natively. Some [12:17] providers have something to do with it, [12:19] but it's not a framework-level concept. [12:21] And we also need it to for many or [12:24] workflows and pipelines. [12:26] So, what about the third-party [12:28] solutions? Uh they exist. They've been [12:30] existing for a while. There's, for [12:31] example, Acidic Jobs. Uh we used them in [12:33] a couple of projects. It introduces [12:35] durable state, so everything happening [12:39] with the workflow is stored in the [12:41] database. We can restore it, we can [12:43] observe it. Uh there is a built-in [12:45] uniqueness, and so on. And what's [12:48] important for me, at least, that it has [12:51] uh Active Job interface. So, from the [12:53] outside caller, the job is still Active [12:55] Job. Everything you can use with Active [12:57] Job works. And the whole durability of [13:00] the workflow is hidden within the [13:02] workflow class. [13:04] Another most more recent development is [13:07] Unix Geneva Drive project. Um it [13:11] Actually, it comes with almost [13:13] everything uh I wish a workflow [13:16] implementation had. And we're going to [13:19] refer to it a few times in the talk. [13:21] But, the biggest problem to me, it's not [13:24] a Rails thing. It's a custom DSL, and [13:27] it's a custom entity. It's not surely [13:30] compatible with active job, so you need [13:32] to refactor everything to use it. And I [13:35] wish uh we had something more kind of [13:38] framework friendly, but with the same [13:40] guarantees. [13:42] Uh okay, next pattern I want to [13:44] touch a bit is batching. So, when we [13:46] have one job performing the setup, many [13:50] jobs performing their own task, and then [13:52] something running on completion. [13:55] Uh and batches could be actually [13:57] executed sequentially, and we can say [13:59] that uh [14:01] well, uh we can use continuation for [14:03] that, and that's again an example from [14:05] the wild world from Spray. Uh [14:08] we begin processing, we create a a job [14:11] per uh CSV file row row, and then we do [14:15] some [14:16] finalization. But, [14:18] actually, whenever we talk about [14:20] batches, we usually imply parallel [14:21] execution, right? [14:23] And that's a problem because [14:25] uh whenever we start parallel and we [14:27] need to coordinate the [14:29] end of the execution of many jobs, [14:32] we need some coordinator. And typical [14:35] coordinators are counters, possibly [14:38] stored in Redis, and um [14:40] maybe polling like in this example. We [14:42] just [14:43] uh have a job that periodically checks, [14:45] "Oh, everyone's ready? No, let's let's [14:47] try again in, you know, 5 seconds." [14:49] And for this use case, we typically use [14:53] third-party gems. But, the good news is [14:54] that since very recently, we can use [14:58] third-party gem. And uh Solid Queue 1.7 [15:02] just got Solid Queue batch support [15:04] thanks to JP Camera and [15:08] uh Rosa, of course, who helped a lot [15:10] with that. And it's been a long and [15:13] winding road to batches in Solid Queue, [15:15] and I highly recommend to read the story [15:17] because it demonstrates Uh, what amount [15:21] of effort is required to ship a major [15:24] feature [15:25] to a cornerstone library like Solid [15:28] Queue when you are not a core [15:30] contributor, outside contributor. That's [15:33] massive work and thanks everyone [15:35] involved into that. That's cool. [15:38] So, that's a just a quick review of how [15:39] it does it look. Everything is a job, so [15:42] whenever we want to run something on [15:43] completion, we create a job on success, [15:45] on failure, and so on. [15:47] Uh, this interface is inspired by good [15:50] job [15:51] that had batches for a while. And we had [15:55] many more examples of batching. Uh, with [15:58] everyone comes with their own interface. [16:00] That's a problem. But still, uh, the the [16:02] problem kind of solved. It just needs [16:05] your choice on which library to choose. [16:09] And I want to highlight, uh, [16:11] the one from Doximity, Simple Kick. It's [16:13] Sidekiq specific, but what's interesting [16:15] here is that interface is very similar [16:18] to continuation. We have steps, cold [16:21] runs. We have parallel steps, something [16:23] that's missing in continuation, but [16:26] it hints, at least to me, that, uh, we [16:29] can reuse the same idea of parallel [16:32] steps, probably, [16:34] in continuation. But there is just one [16:36] blocker. [16:38] And, um, what's the blocker? We don't [16:40] have it in Active Job, and [16:42] we'll probably have it at some point [16:44] because there were some discussions [16:46] about it. And as soon the interface [16:48] landed in Active Job, we can bring it to [16:50] continuation and make our workflow [16:52] batches aware. [16:54] But, uh, [16:55] we need to wait. [16:57] And speaking of waiting, that's another [16:58] pattern, [16:59] uh, another group of orchestration [17:00] scenarios, a background task that [17:02] involve all kind of timers, like [17:05] reminders, [17:07] uh, ex- expirations, notifications, and [17:10] so on. Uh, for example, again from our [17:13] AnyCable project, uh, this is a typical [17:15] flow. We have a bunch of recurring tasks [17:19] like polling tasks that run every 10 [17:21] minutes or every hour or whatever [17:23] depending on the scenario. For the same [17:25] process actually, this is a license [17:27] expiration [17:28] workflow, but it's split across multiple [17:30] background jobs and each background job [17:33] is actually an entry point to enqueue [17:35] more background jobs and it's introduced [17:38] a lot of [17:40] you know, code concepts just to [17:42] implement a single workflow. We need to [17:43] repeat it for every workflow. [17:46] With continuation, it could be [17:48] simplified a bit because we don't need [17:49] this nesting with we can use iteration [17:52] within steps and cursors and [17:56] limit the amount of [17:58] job classes to one. Still we need the [18:01] scheduling thing and [18:03] but that's already it looks better [18:06] than without it. [18:07] Another pattern related to time is [18:10] parking job someday in the future with [18:14] wait or wait until and then executing [18:16] there. But the problem with this [18:18] approach, actually there are a lot of [18:19] problems with this approach. The first [18:21] problem is that we usually need to [18:22] invalidate oh, if this job is still [18:24] relevant because there is no at least [18:27] active job level API to cancel a job in [18:30] the that is scheduled for the future. [18:32] Yes, you can reach out for the provider [18:34] specific API, but usually that's just a [18:36] lot of work and validation is [18:38] simpler and more reliable. There are [18:41] many questions related to this approach, [18:43] but conceptually it's sounds good, but [18:46] technically it's hard to implement and [18:47] we [18:48] typically fall back to the cron driven [18:50] sweeps. [18:52] Let's take a look at [18:54] the gems that implement something that [18:57] makes this easier. This is another [19:00] relatively new project called Connor [19:02] Forge and it's it extends actually [19:06] active job with custom, let's say, [19:09] DSL method decorations including wait. [19:12] So, whenever you enqueue Java, Patrice [19:13] the wait statement and says, "Oh, I'm [19:16] stopped here. I add myself to the [19:18] waiting [19:19] queue or table, and then I'm being [19:22] re-enqueued when time comes and continue [19:24] the execution." [19:25] Um [19:27] Geneva Drive also has this concept of [19:30] timers. It's just an option for the [19:32] step, and it also has this [19:35] beautifully designed uh [19:37] uh mechanism of associating workflows [19:39] with [19:41] models through heroes. [19:43] Uh so, you can easily find a workflow [19:45] associated with some active record model [19:47] and cancel it when needed. That's very [19:50] useful. So, there is an API for that. [19:53] And another option is actually using [19:55] third-party services, like Temporo, a [19:57] pretty popular one recently. It has a [20:00] lot of stuff. And what's interesting [20:03] here is that uh that's kind of the same [20:05] example with appointment reminder, but [20:08] instead of canceling the workflow and [20:11] restarting it from scratch when the date [20:14] has changed, for example, we can just [20:16] update the deadline for the reminder. [20:18] So, it's also part of the state of the [20:21] workflow when we want to trigger it. [20:26] So, uh kind of related to timers is a [20:29] more generic uh group of workflows or [20:32] features. [20:33] Uh [20:34] approval workflows or signal-driven [20:37] workflows or like human-in-the-loop [20:39] scenarios, right? [20:41] So, [20:42] uh a signal is something that a workflow [20:44] waits, uh not that isn't time. It may be [20:48] a user action. And typically in code, [20:51] signal splits the implementation of the [20:53] workflow into before and after parts, [20:56] uh living in different places in code, [20:57] like here it's a from Mastodon example. [21:00] It's the same process. It should be read [21:02] as a one, you know, uh process with a [21:06] break in the middle, but we don't have a [21:09] way to implement that in Rails. [21:12] And given that signal is not guaranteed, [21:14] we need to take care of cleaning up uh, [21:18] what task that hasn't been [21:20] uh, you know, uh, signaled yet. That [21:22] adds a lot of [21:24] ad hoc logic to your application. [21:26] Uh, [21:27] another example, another type of signal, [21:29] very popular is a webhook event. This is [21:32] a real example from one of the apps I [21:33] worked on, and we see that workflow is [21:36] is scattered across three files, so we [21:38] have the one job that request the [21:40] withdrawal, then we have a webhook, uh, [21:43] that send the confirmation or rejection, [21:47] and then finally have another [21:49] background job that process the result. [21:52] If we migrate, for example, to [21:54] Conductor, because it also has this [21:56] signaling feature with continue if step, [21:59] not an obvious naming, but anyway, we [22:01] could uh, rewrite the workflow, well, [22:03] this this one, uh, in to just a single [22:05] perform method with three steps. [22:08] Run third step, wait for the payment [22:11] receipt, run credit step. And it's much [22:13] easier to reason about than [22:16] finding all the logic in between [22:17] multiple files. [22:19] But unfortunately, the project I [22:20] borrowed this example from [22:22] uh, picked another solution to implement [22:25] this workflow. [22:27] Uh, sorry, it's loading. [22:29] Yeah. So, uh, they picked Temporal, and [22:32] I can't say I'm happy with that. So, I [22:34] don't think uh, implementing such uh, [22:37] things that could be expressed, you [22:39] know, in English in few words, should [22:41] cost us hundreds lines of code and [22:43] dozens of Ruby objects and [22:47] scary like voodoo magic to make them all [22:49] work. So, [22:52] that's the problem the problem. The more [22:54] complicated, uh, the more sophisticated [22:56] your workflow becomes, the more [22:58] sophisticated tools you have to use. And [23:00] I don't think it's the right, uh, way [23:03] of, you know, [23:05] going into the future with Rails, right? [23:07] Uh, we need more batteries included. We [23:10] We are missing some, uh, [23:12] some of them, clearly. And why why not [23:15] about why not adding them? Uh, about a [23:17] year ago at a Sapphire B conference, [23:19] which is happening this year again in [23:21] November, by the way, don't miss out. I [23:23] already shared this idea, uh, that, [23:26] well, now we have a cool concept for [23:28] building more than just single jobs in [23:32] active job, so we can build something [23:33] bigger, some workflows. [23:35] And let's [23:37] not invest into, you know, building [23:38] custom gems or [23:41] ugly SDKs. Let's just build on top. [23:44] Let's turn continuation into something [23:45] bigger. [23:46] And I've been keeping this idea in mind [23:49] for almost a year. And today, I I want [23:53] to present the first implementation of [23:54] it. [23:55] So, let's Let me introduce the durable [23:59] continuation as part of the job. [24:02] Ooh. [24:03] All right. What is durable, by the way? [24:05] What do we mean by that, right? [24:07] Everyone's talking about that, [24:08] especially since the bureau coined this [24:10] term and started using it everywhere, [24:12] trying to make it look like, oh, it's [24:14] something's very special, you know? It's [24:16] nothing special. Uh, the main difference [24:18] is that, basically, uh, a job, the [24:20] execution process, is a record or [24:23] multiple records in a database or some [24:25] other store, [24:26] and not just a payload in the queuing [24:28] system. That's the primary difference. [24:30] Everyone else is, uh, consequence of it. [24:34] So, like, surviving sick kills and every [24:36] and and so on and so forth. They're just [24:38] the properties we get if we carefully [24:41] commit the state and make it restorable [24:44] and re- replayable. [24:46] So, to turn active job continuation [24:49] into, uh, durable continuation, we need [24:52] to implement a few things. Uh, [24:55] the ones I've been already talking about [24:57] in the previous section, right? Though [24:59] that's missing. But the most important [25:01] requirement, which I already always [25:05] add to anything I work on, any library, [25:08] that it must be as close to Rails way as [25:11] possible. It must feel like using the [25:13] framework, not a third-party library. [25:16] So, it should be an extension, not a [25:17] replacement. Something that could be [25:19] merged into Rails in the future. If you [25:22] build your library like that, then you [25:25] kind of get a, [25:26] you know, some kind of style and taste [25:29] constraints by default, and they usually [25:33] help you to get with a better [25:34] implementation. [25:35] So, let's move to the durable [25:37] continuation proposal. And let's start [25:39] with the state. [25:40] So, back to the one of the examples. [25:43] This is a also from a separate via our [25:46] cards generation job, the previous [25:48] version, the one that generates cute [25:49] characters based on the [25:52] photos. [25:54] And it has multiple steps because we [25:55] need to process the original image [25:56] first, we need to moderate it. We do not [25:58] allow any type of images downloaded and [26:01] show to everyone in, you know, the [26:03] community. So, we have some [26:06] checks there, moderation step. And [26:08] finally, we generate the final content. [26:11] So, that's how it looked like uh [26:14] in the original version. It's not [26:16] durable yet. [26:17] What does it mean to make it durable? [26:19] And what what changed? So, we have a [26:23] table with the [26:25] execution runs and their state and step [26:28] state and whatever. [26:30] And we can see how it's being populated [26:32] and updated as as the execution [26:35] progress. So, first, we enqueue the job, [26:38] and we got a record on the table. So, [26:42] the process hasn't started yet, but we [26:43] already know that it's being enqueued. [26:46] Then, as soon as we hit [26:48] um [26:49] the worker, we change the state to to [26:52] then we update each step as we progress. [26:55] We keep the attributes in the state as [26:58] well, so they're also durable. So, you [27:00] can observe them from the outside, so [27:02] you can build some features that use [27:04] these attributes as data for your, you [27:07] know, other operations. [27:09] And yeah. [27:11] And and finally, you have an API to [27:15] query this data easily, kind of similar [27:17] to what Geneva Drive has, but we don't [27:19] have concept of heroes. We just infer [27:22] identifiers from arguments or [27:25] specifically defined identifiers on the [27:27] job. And you can see what's happening [27:28] with the job. You can [27:30] cancel it if you want, or retry, or [27:32] whatever. [27:35] Uh let's see how this state, durable [27:39] state that we add, affect our [27:41] resumability properties. This is the [27:44] original webhook example and using [27:46] continuation. Imagine that we run the [27:49] job, it triggers webhooks, but then it's [27:52] crashed. [27:53] And [27:54] the job restarts, [27:57] and the cursor value hasn't been [27:59] committed because it was a hard crash or [28:01] some exception that hasn't been caught, [28:04] and we start over [28:06] because we don't have any memory of [28:08] what's been in between the steps and [28:12] within the step. [28:14] When we switch to durable continuation, [28:16] just one word changed, right? Nothing [28:18] changed in the class, just a single [28:21] model. We have a different picture. We [28:23] have cursor committed, not every step, [28:26] actually, but you can configure it [28:28] whatever you want. And that gives you [28:30] better guarantees in case of [28:33] uh [28:34] different problems that could occur to [28:35] your app. So, it just improve the [28:38] reliability of the operation. [28:41] Uh now um [28:43] let's talk about the life cycle Uh, and [28:46] uh, else do we add to make workflow more [28:49] useful? Remember I mentioned that while [28:51] we don't have callbacks, we have to [28:52] inline everything in our step bodies. [28:55] And now, uh, with durable execution, we [28:57] also ship callbacks, after step [28:59] callbacks. That makes your workflow look [29:02] like, you know, a typical Rails object [29:06] with all the functionality typically [29:08] available for models, controllers, [29:10] whatever. It's no longer just a tiny [29:12] wrapper over background job execution. [29:15] Similarly, we can have fix, uh, the [29:17] problem with terminating the workflow [29:20] after a particular step, not just the [29:22] whole thing, uh, not just discarding [29:25] everything. And there is a halt on [29:28] rescue handler. And it differs from [29:30] discard on because it allows you to [29:32] restart the job when the condition is [29:35] fixed because some errors are [29:38] recoverable. There is also uniqueness [29:40] built in just because we have a database [29:42] of runs, we can easily achieve [29:44] uniqueness. [29:46] And [29:47] let's me show you how timers evolve. So, [29:50] previously we had a scheduler and many [29:52] jobs. Now we can have a single job with [29:54] steps, and now steps have wait and wait [29:57] until, and they describe the whole [29:58] process. And again, we have uniqueness [30:01] here, and when we want to restart the [30:03] life cycle workflow for a license, uh, [30:06] because it'd been renewed, we just [30:08] re-enqueue the job and it replace the [30:10] previous workflow and continue working. [30:14] All right. Uh, yeah, that's another [30:16] example with, uh, uh, [30:19] timers and it has active record [30:21] integration has one performed. So, you [30:23] can easily access associated runs. [30:26] And, yeah, we kind of a bit out of time. [30:30] I expected that. [30:32] So, we just have couple minutes left, [30:34] uh, to show what's uh, the other [30:36] features. And I think the most favorite [30:38] one. So, this is a previous Temporal [30:40] example with signals. And when we [30:42] convert it into durable continuation, it [30:44] becomes just this. [30:46] It's just a single file with a few [30:48] configuration things, and and that's [30:51] basically it. The whole logic lives in a [30:53] single rails [30:54] Ruby class, and you can observe it, you [30:57] can control it. It's It's I don't know. [31:00] It looks much better. And it looks like [31:03] better than English, you mean? It's [31:05] Ruby. It has structure and clear [31:07] meaning. [31:09] So, uh [31:11] yeah, a couple of examples with signals. [31:13] Um [31:14] The The again generation example, but [31:17] now an upgraded version when we decide [31:19] whether the [31:20] photo is good enough. Uh [31:23] If it's not, you know, uh it has a [31:26] uh border score, we add human to the [31:29] loop and ask, "Oh, do you need Does Do [31:32] you allow to post this image on ads?" [31:34] And now we do that by keeping a lot of [31:37] states uh in the enum on the model, and [31:40] just, you know, checking them and [31:42] iterating in the job. Uh [31:45] but, there are problems with that [31:46] because there could be concurrent [31:48] executions. We don't have history, we [31:50] don't have deadlines for the review. [31:52] With durable continuation, we can change [31:53] it with just single line. It's a a wait [31:56] step. It's a just a special kind of step [31:58] that parks the job for a given period of [32:00] time. It could be released after [32:02] timeout, and then we can send the [32:04] signal, say, from the controller or [32:06] wherever we want to continue the [32:08] execution. That's the whole API. [32:11] And it works [32:12] Yeah, on top of the same primitives as [32:16] timers. Um with just steps being empty [32:19] bodies. [32:23] So, one bonus example, recently Ruby [32:26] alarm 2.0 was released, and it has a lot [32:28] of [32:29] uh example snippets with continuations [32:31] involved, but they were they not as good [32:34] if uh they were using durable [32:36] continuation because now the whole loop [32:39] could be implemented in a single job [32:41] with uh, restarting after approval [32:44] because we just have this health method [32:46] and you can call resume whenever you [32:49] approve or not the action and um, that [32:53] also simplifies this workflow. We don't [32:55] need durable state for real here because [32:57] Ruby on Rails has its own durable state [32:59] as chat messages but still we need the [33:01] control flow durable state [33:04] to exist to implement it this way. [33:08] All right. Uh, so [33:10] that was a quick overview of the [33:12] features we need to add to continuation [33:13] to make it durable and more useful and [33:16] drop all the stuff you using for that [33:19] and it's available today. Like in a [33:21] minute I'm going to open source it. [33:23] Uh, it's not yet available actually uh, [33:25] as a gem AGDC active job durable [33:28] continuation nothing related to music [33:31] and uh, yeah, it comes with everything I [33:33] mentioned in the talk and more features [33:34] coming. It comes with AI scales [33:37] backed by Rails hyperdrive project. So [33:40] all we need is to install gem and tell [33:42] your agents, oh, configure durable [33:44] continuation find job that would be nice [33:46] to migrate and do the migration and then [33:48] that's it. [33:49] Uh, probably you'd like to read the code [33:52] afterwards. I don't recommend not [33:53] reading code. And um, the idea behind it [33:56] is to bring just enough orchestration to [34:00] active job to stay on the Rails way to [34:03] not reach out for services or cryptic [34:06] gems to implement these features. We can [34:10] do that in Rails and then you can you [34:11] can see that it could be much better [34:14] than using any of those. So yeah, more [34:16] features coming I mentioned that. [34:19] That's the plan for you. [34:21] Pay more attention to continuation. [34:23] Try to see if you have these patterns in [34:26] your code and they were extracting into [34:28] something looking more like workflows [34:30] and give durable workflows a try. [34:33] That's it. Thank you for your attention. [34:38] >> [applause] [34:44] [music]