Velasquez Pairs Rails Jobs With Judge LLMs to Halt AI Hallucinations

Ruby on Rails

Because deep-reasoning LLMs hallucinate from prompt examples and mirror hostility from raw user feedback, keeping proactive agents reliable requires wrapping database-backed Rails jobs in a pipeline of smaller, faster models that filter outputs and distill user complaints into neutral facts.

AI-assisted summary of Agents on Rails: Building AI Pipelines in plain Ruby - Daniela Velasquez — original source. Check the original for full context.

Source published: 2026-10-01

Key ideas

Proactive AI agents can be constructed natively using standard Ruby on Rails background jobs and database models, flipping passive dashboard UX by delivering unprompted, data-backed insights directly to users. (0:11)
Because agent research jobs often run for 10 to 30 minutes, continuous deployment resets can wipe progress; persisting LLM conversation history to the database solves statelessness and allows seamless recovery. (3:50)
Multi-agent quality control using fast, low-cost secondary LLM judges as binary classifiers effectively filters hallucinations, tone issues, and factually impossible recommendations before they reach production. (7:48)
Direct user feedback risks poisoning subsequent LLM prompts with mirrored hostility; deploying an intermediary extraction agent distills raw grievances into neutral operational constraints. (13:35)

Detailed notes

## Background Published on YouTube by the Ruby on Rails channel on October 1, 2026, this video features Daniela Velasquez's talk from Rails World 2026. According to the show notes, she demonstrates how to move beyond passive dashboards and chatbots by building proactive AI agents—specifically an AI running coach that analyzes workout, sleep, and heart rate data alongside external weather forecasts—using plain Ruby on Rails jobs without any extra frameworks.

## TL;DR Daniela Velasquez shows how to build a proactive AI running coach using standard Ruby on Rails background jobs, database models, and lifecycle callbacks. The pipeline starts with a slow, high-reasoning researcher job that synthesizes athlete metrics and external tools like weather forecasts, while persisting conversation history in the database to survive stateless application deployments. Because LLMs frequently hallucinate or adopt poor tones, the researcher's output is filtered through a suite of fast, smaller "judge" LLMs trained to verify facts and empathy before anything reaches the user. Finally, when athletes push back on suggestions, an extraction agent triggered via an `after_create_commit` callback strips out human frustration and saves only clean, objective constraints for future runs.

## Atmosphere & style The talk feels pragmatic, grounded, and refreshingly demystifying, stripping away AI hype in favor of familiar Rails conventions and candid developer humor. Daniela maintains a lighthearted, conversational rapport with the audience, openly poking fun at how LLMs feel like they are "trolling" developers—highlighted by a comedic real-world story about a Shopify agent suggesting "Wawa shampoo" bundles to a bra merchant and a closing reel of bizarre AI slide-generation bloopers. The emotional arc moves from the excitement of proactive automation to the humorous reality of LLM hallucinations, ultimately settling into a reassuring sense of engineering control through simple, battle-tested Ruby patterns.

## Bulletin - [00:11](https://www.youtube.com/watch?v=OX5coKfra4c&t=11s) Proactive AI agents can be constructed natively using standard Ruby on Rails background jobs and database models, flipping passive dashboard UX by delivering unprompted, data-backed insights directly to users. - [03:50](https://www.youtube.com/watch?v=OX5coKfra4c&t=230s) Because agent research jobs often run for 10 to 30 minutes, continuous deployment resets can wipe progress; persisting LLM conversation history to the database solves statelessness and allows seamless recovery. - [07:48](https://www.youtube.com/watch?v=OX5coKfra4c&t=468s) Multi-agent quality control using fast, low-cost secondary LLM judges as binary classifiers effectively filters hallucinations, tone issues, and factually impossible recommendations before they reach production. - [13:35](https://www.youtube.com/watch?v=OX5coKfra4c&t=815s) Direct user feedback risks poisoning subsequent LLM prompts with mirrored hostility; deploying an intermediary extraction agent distills raw grievances into neutral operational constraints.

## Key takeaways - [00:11](https://www.youtube.com/watch?v=OX5coKfra4c&t=11s) Applications hold vast amounts of uninspected data, and proactive AI agents flip traditional passive dashboards by initiating coaching suggestions before the user asks. - [01:39](https://www.youtube.com/watch?v=OX5coKfra4c&t=99s) The primary researcher job relies on a slow, deep-thinking model with high reasoning confidence to synthesize internal application data alongside external tools like weather and web search. - [03:50](https://www.youtube.com/watch?v=OX5coKfra4c&t=230s) Because LLM API calls are stateless and researcher runs take up to 30 minutes, continuous deployments can wipe in-flight tasks unless message histories are recorded in the database. - [05:24](https://www.youtube.com/watch?v=OX5coKfra4c&t=324s) Prompt examples can inadvertently cause hallucinations when an agent runs out of ideas, such as recommending non-existent shampoo bundles to a merchant selling bras. - [07:48](https://www.youtube.com/watch?v=OX5coKfra4c&t=468s) Secondary LLM judges provide a safety layer by serving as focused binary classifiers that evaluate candidate suggestions against narrow criteria like tone, realism, and factual correctness. - [07:48](https://www.youtube.com/watch?v=OX5coKfra4c&t=468s) Calibrating an evaluator judge requires building a benchmark dataset of balanced positive and negative examples to measure classification accuracy before deployment. - [10:06](https://www.youtube.com/watch?v=OX5coKfra4c&t=606s) Evaluator judges should run on smaller, fast, low-reasoning models to minimize token costs and latency during asynchronous parallel execution. - [10:06](https://www.youtube.com/watch?v=OX5coKfra4c&t=606s) Each evaluation job persists both its binary verdict and explanatory reasoning directly to the database, allowing invalid recommendations to be discarded cleanly. - [12:01](https://www.youtube.com/watch?v=OX5coKfra4c&t=721s) Real-world user feedback captures external constraints that telemetry data cannot detect, requiring an explicit database model to record user responses. - [13:35](https://www.youtube.com/watch?v=OX5coKfra4c&t=815s) Passing unfiltered user complaints directly into LLM prompts can poison the model's context, causing it to adopt defensive or hostile tones in future generations. - [13:35](https://www.youtube.com/watch?v=OX5coKfra4c&t=815s) An intermediary extraction agent triggered by Rails model lifecycle callbacks strips emotional rhetoric and extracts pure, factual constraints into a sanitized attribute. - [15:28](https://www.youtube.com/watch?v=OX5coKfra4c&t=928s) Long-term agent memory requires background maintenance and auditing—termed a dreaming phase—to prune obsolete user constraints as real-world schedules evolve. - [15:28](https://www.youtube.com/watch?v=OX5coKfra4c&t=928s) The complete proactive pipeline forms an architectural loop: generation via researcher jobs, verification via judge jobs, delivery to users, and asynchronous feedback sanitization.

## Detailed walkthrough ### [00:11](https://www.youtube.com/watch?v=OX5coKfra4c&t=11s) The Why and How of Proactive AI Agents in Rails

Daniela Velasquez kicks off her talk by asking how we can put "agents on rails" using plain Ruby on Rails without needing extra frameworks. Exploring the motivation behind the idea, she notes that applications hold immense amounts of data with fascinating insights, but nobody has the time to sit and sift through it all. Traditional solutions like dashboards or chatbots still wait passively for users to ask questions. Daniela proposes flipping the dynamic so that applications go to the users proactively, "because sometimes you don't know what you don't know." To ground the talk, she introduces an example application tracking an athlete's workouts, sleep logs, and heart rates to coach them before they even ask.

- Plain Ruby on Rails can build proactive AI pipelines without heavy external frameworks. - The example application tracks athlete workouts, sleep logs, and heart rates to deliver proactive coaching.

```mermaid flowchart TD subgraph Problem["Traditional Approach"] A["App Data"] --> B["Dashboards & Chatbots"] B --> C["Wait for User Questions"] end subgraph Solution["Proactive Rails Agent"] D["Athlete Data & Context"] --> E["Researcher Job"] E --> F["Proactive Coaching Insights"] end style Problem fill:#f9f,stroke:#333,stroke-width:2px style Solution fill:#bbf,stroke:#333,stroke-width:2px accTitle: Proactive AI Agents versus Traditional Dashboards accDescr: A flowchart comparing passive dashboards with proactive Rails agent pipelines. ```

[01:39](https://www.youtube.com/watch?v=OX5coKfra4c&t=99s) Building the Basic Researcher Job and Prompt

The implementation starts with a simple Rails job called a researcher job. The heart of the LLM is the prompt, which in this case establishes the agent's persona: "You're a running coach. Study this athlete's workout, sleep, and heart rate, then propose concrete changes to the training for next week. Back every change with data you actually look at." Developers can choose any LLM provider they want and can afford, passing the prompt directly to them. Alongside the internal app data like workout history and sleep logs, the agent is given external tools such as weather forecasts and web search so it can look at the world around the athlete. Daniela describes the model as "basically the brain you're trying to rent," noting that you need a slow, deep-thinking model with high reasoning confidence rather than a fast one.

- The researcher job begins with a core prompt establishing the running coach persona. - Tools include internal data (workout history, sleep logs) and external data (weather forecasts, web search). - The setup requires a larger, slower model with high reasoning settings rather than a fast one.

[03:50](https://www.youtube.com/watch?v=OX5coKfra4c&t=230s) Fixing Stateless Deployments with Conversation Tracking

Running these researcher jobs takes time—anywhere from 10, 15, or 30 minutes depending on the tools and prompt complexity. However, continuous deployment poses a major challenge because when a deploy happens, the agent living inside a job resets to zero. Because LLM provider calls are stateless, they lose all memory of what was happening, wasting both time and money and frustrating bosses. To fix this, Daniela introduces a `past_conversation` record to track messages exchanged with LLM providers. If the conversation has no messages, the job stores the system prompt and a user message kicking off the research. It then enters a loop, passing the full message history to the LLM provider on every turn and saving incoming responses so the job can seamlessly recover if it restarts.

- LLM calls are stateless, meaning deployments wipe out agent progress and waste resources. - Tracking conversation history in the database allows jobs to restart and remember past back-and-forth messages.

[05:24](https://www.youtube.com/watch?v=OX5coKfra4c&t=324s) Agent Coaching Suggestions and the Trust Problem

Walking through a hypothetical output, Daniela shares what the agent might suggest: shifting Tuesday runs to start an hour later, adding two 45-minute bike rides to build an aerobic base, easing up on the first kilometers of long runs to prevent fading, and moving the Saturday long run to Sunday morning due to heavy rain. While this seems great, Daniela points out the harsh reality: "It's hard to trust them. They sometimes come up with very random ways of doing... very creative ways of not the right thing that you asked them to do. And honestly half the time I feel like they're just trolling you for what you're asking them to do." She shares a hilarious anecdote from Shopify where the researcher agent repeatedly suggested that merchants "bundle your Wawa shampoo and Wawa conditioner because they're usually bought together." After seeing it multiple times, she checked the merchant's store and discovered they were selling bras! The hallucination happened because "Wawa shampoo and conditioner" was buried as an example in the prompt, and the agent grabbed it whenever it ran out of ideas.

- The agent successfully synthesizes internal workout logs with external weather forecasts. - At Shopify, a researcher agent hallucinated non-existent "Wawa shampoo and conditioner" bundles for a merchant selling bras because the items were listed as examples in the prompt.

[07:48](https://www.youtube.com/watch?v=OX5coKfra4c&t=468s) Deploying LLM Judges to Keep Agents on Rails

To solve the trust problem and prevent agents from falling off the rails, Daniela humorously suggests throwing more LLMs at the problem—specifically, using secondary LLMs to act as judges. These judge LLMs have a much simpler job: acting as focused classifiers answering specific binary questions important to the application, such as catching made-up numbers, checking tone, or running reality checks on whether an athlete can actually execute a suggestion. To train the "kind judge," developers compile around 200 examples of kind suggestions and 200 examples of unkind ones. For instance, swapping an easy run for a chill session when dealing with hip pain is "kind," whereas telling someone running seven days a week to "stop before you break yourself" might not be super kind. Similarly, telling an athlete their knees hurt because they ran 900 km in the same shoes is realistic but perhaps a bit blunt, while suggesting they grab a snack at the 45-minute mark because their pace drops after 60 minutes is helpful and kind.

- Secondary LLM judges act as focused classifiers to evaluate and filter researcher output. - Training the kind judge involves curating 200 positive and 200 negative examples covering tone, empathy, and realism.

[10:06](https://www.youtube.com/watch?v=OX5coKfra4c&t=606s) Evaluating Prompts and Running LLM Judges

Daniela outlines the evaluation criteria for the "kind judge" prompt: *"Would a supportive coach say this? Blame, sarcasm, or threats fail. Honest data delivered kindly passes. Reply with pass or fail and why."* Once this prompt is written, it is benchmarked against an existing evaluation dataset to measure accuracy. This setup allows developers to freely iterate on prompt variations and benchmark different language models against each other to discover which one strikes the right balance between cost and performance.

Because these evaluator judges serve narrow, specialized roles, they do not require expensive flagship models with deep reasoning capabilities. Instead, smaller and significantly faster models keep background job execution cheap and snappy. As soon as the researcher finishes generating coaching suggestions, the pipeline moves to its second phase: dispatching a suite of focused judges across each individual suggestion. To evaluate kindness, Rails invokes an LLM provider client, passes the system instructions alongside the candidate suggestion text, sets the reasoning parameters low, and records the resulting pass/fail verdict and reasoning directly to the database.

Applying these checks to real-world researcher output quickly exposes problematic recommendations. For instance, when the researcher tells an athlete they just had their *"worst sleep"*, the kind judge flags it for lacking empathy. When another job proposes an outdoor ride, the fact-checking judge looks up equipment records, notes that *"they don't own a bike, so they cannot do this"*, and fails it immediately. As a result, out of four original coaching actions generated by the researcher, two are immediately disqualified—one for failing the tone test and the other for proposing an impossible activity.

- Evaluator prompt instructions: test whether a supportive coach would say the line; reject blame, sarcasm, or hostility; pass honest data communicated kindly. - Smaller models with minimal reasoning effort provide the speed and low token cost necessary for parallel evaluation jobs. - Each evaluation job persists its verdict (`pass` or `fail`) and underlying explanation alongside the suggestion record.

```mermaid flowchart TD accTitle: Evaluator Judge Pipeline accDescr: Pipeline showing researcher suggestions passing through specialized judges to filter the output.

subgraph Generation ["1. Researcher Agent"] R["Researcher Job"] --> S1["Suggestion 1: Sleep Notice"] R --> S2["Suggestion 2: Bike Ride"] R --> S3["Suggestion 3: Easy Jog"] R --> S4["Suggestion 4: Hydration Goal"] end

subgraph Evaluation ["2. Specialized Judges"] KJ["Kind Judge"] FJ["Fact-Checking Judge"] S1 -.->|"Evaluated by"| KJ S2 -.->|"Evaluated by"| FJ end

---

[12:01](https://www.youtube.com/watch?v=OX5coKfra4c&t=721s) Filtering Pipeline Recap and Handling User Feedback

Recapping the flow up to this point, Daniela explains how an athlete's incoming metrics feed directly into a researcher agent running in a background job, which derives personalized workout recommendations. That batch triggers a secondary wave of asynchronous Rails jobs, running every suggestion through the specific domain judges that matter to the application. Each judge issues an independent thumbs-up or thumbs-down based on its assigned metric. Ultimately, the athlete sees only the vetted, high-confidence subset of suggestions that passed every single gate.

``` +-----------------------------------------------------------------------+ | Athlete Metrics & History (Sleep, Workouts, HR) | +-----------------------------------------------------------------------+ │ ▼ +-----------------------------------------------------------------------+ | Researcher Job: Proposes Candidate Suggestions | +-----------------------------------------------------------------------+ │ │ │ ▼ ▼ ▼ +-----------------+ +-----------------+ +-----------------+ | Kind Judge Job | | Fact Judge Job | | Custom App Job | | (Tone check) | | (Equipment/Log) | | (Safety check) | +-----------------+ +-----------------+ +-----------------+ │ │ │ └───────────────────────┼──────────────────────┘ ▼ +-----------------------------------------------------------------------+ | Approved Recommendations Displayed on Dashboard | +-----------------------------------------------------------------------+ ```

However, real-world users inevitably push back—especially once they realize recommendations are generated by an AI. An athlete might reply with personal schedule constraints that were never logged in their wearable data, such as: *"I work on Sundays from 6:00 a.m. to 3:00 p.m."* No matter how many external API tools or database integrations the agent started with, this context simply did not exist anywhere in the app. To prevent future pipelines from making the same mistake, the researcher needs a way to persist and recall user feedback.

Persisting this feedback begins with standard Rails architecture: an `AthleteFeedback` model inheriting from `ApplicationRecord` that `belongs_to:athlete` and `belongs_to:suggestion`. When the researcher job runs in subsequent iterations, it loads this historical feedback and injects it into its prompt context so the model learns from past interactions.

---

[13:35](https://www.youtube.com/watch?v=OX5coKfra4c&t=815s) Sanitizing Frustrations with an Extraction Agent

Feeding raw user text directly into subsequent agent prompts introduces serious risks. Because large language models thrive on matching tone and patterns, unvarnished human feedback can poison the pipeline. While an athlete might start out polite, repeated mismatches often lead to irritation and outright name-calling. If an angry user vents at an agent, passing that hostility directly into the researcher prompt risks having the agent *"pick up on the bad vibes"* and generate hostile or defensive suggestions in return. While downstream kind judges would catch those suggestions, wasting compute and token budget generating angry responses is completely counterproductive.

Daniela proposes solving the issue with another specialized agent: *"Another agent. So, you're going to pass that raw user feedback and give it to an agent that is just going to take all that noise. And you just want the fact."* Regardless of the emotional outburst, profanity, or sarcasm in the athlete's input, the extraction agent strips the noise down to the essential operational constraint: the athlete is unavailable to train between 6:00 a.m. and 3:00 p.m. on Sundays.

To implement this, Daniela hooks into the feedback model lifecycle using an `after_create_commit` callback. Acknowledging the perennial Rails community debate around callbacks—*"I know callbacks are not really loved by a lot of people, but I love callbacks"*—she shows how the callback enqueues a background processing job. This job instructs a small, low-reasoning LLM to review the rejected training suggestion, discard the emotional rhetoric, and output a single clean sentence summarizing what the coach must remember to prevent a recurrence. The distilled output is saved into a `processed_text` database column, which is the only version later exposed to the primary researcher agent.

- Feeding raw user complaints into LLM prompts can lead models to mirror the user's frustration or negative tone. - An intermediary extraction agent distills user vent sessions into clean, objective facts. - Rails lifecycle callbacks enqueue the background sanitization job immediately after feedback is recorded. - The primary researcher job ingests sanitized `processed_text` attributes rather than unedited user messages.

---

[15:28](https://www.youtube.com/watch?v=OX5coKfra4c&t=928s) Long-Term Memory Challenges, Final Recap, and AI Art Bloopers

Looking ahead, Daniela addresses an architectural frontier in agent design: long-term memory maintenance and fact obsolescence. Real-world facts change over time—athletes change jobs, work shifts shift, or schedules open up. If the agent permanently stores a rigid constraint that the athlete cannot train on Sunday mornings, it will eventually generate suboptimal plans when circumstances shift. This introduces what she refers to as the *"dreaming"* phase of AI systems, where background routines periodically audit, consolidate, and prune accumulated memory records to ensure stored facts remain accurate over months and years.

Daniela closes her talk by summarizing the complete architectural loop: 1. A **researcher job** analyzes athlete data to draft prospective workouts. 2. Independent **evaluator jobs** vet those suggestions against application quality gates. 3. Clean recommendations surface in the user interface. 4. Athletes submit corrections and **feedback**, which is asynchronously distilled into neutral facts and fed back into future researcher runs.

To wrap up, Daniela shares several humorous behind-the-scenes failures from generating her presentation slides with image generation models. She displays an uncanny, scary AI illustration that went off the rails, a slide where requesting images of "agents" resulted in the model drawing armed law enforcement officers and detectives, and another image where the model stubbornly refused repeated instructions to remove an imposing background character. She invites attendees to find her after the session to pick up custom illustrated sticker designs of the "little agents" featured throughout the talk.

## Notable quotes > [00:11](https://www.youtube.com/watch?v=OX5coKfra4c&t=11s) because sometimes you don't know what you don't know. — Daniela Velasquez

> [01:39](https://www.youtube.com/watch?v=OX5coKfra4c&t=99s) You're a running coach. Study this athlete's workout, sleep, and heart rate, then propose concrete changes to the training for next week. Back every change with data you actually look at. — Daniela Velasquez

> [01:39](https://www.youtube.com/watch?v=OX5coKfra4c&t=99s) basically the brain you're trying to rent — Daniela Velasquez

> [05:24](https://www.youtube.com/watch?v=OX5coKfra4c&t=324s) It's hard to trust them. They sometimes come up with very random ways of doing... very creative ways of not the right thing that you asked them to do. And honestly half the time I feel like they're just trolling you for what you're asking them to do. — Daniela Velasquez

> [05:24](https://www.youtube.com/watch?v=OX5coKfra4c&t=324s) bundle your Wawa shampoo and Wawa conditioner because they're usually bought together. — Daniela Velasquez

> [10:06](https://www.youtube.com/watch?v=OX5coKfra4c&t=606s) Would a supportive coach say this? Blame, sarcasm, or threats fail. Honest data delivered kindly passes. Reply with pass or fail and why. — Daniela Velasquez

> [10:06](https://www.youtube.com/watch?v=OX5coKfra4c&t=606s) worst sleep — Daniela Velasquez

> [10:06](https://www.youtube.com/watch?v=OX5coKfra4c&t=606s) they don't own a bike, so they cannot do this — Daniela Velasquez

> [12:01](https://www.youtube.com/watch?v=OX5coKfra4c&t=721s) I work on Sundays from 6:00 a.m. to 3:00 p.m. — Daniela Velasquez

> [13:35](https://www.youtube.com/watch?v=OX5coKfra4c&t=815s) pick up on the bad vibes — Daniela Velasquez

> [13:35](https://www.youtube.com/watch?v=OX5coKfra4c&t=815s) Another agent. So, you're going to pass that raw user feedback and give it to an agent that is just going to take all that noise. And you just want the fact. — Daniela Velasquez

> [13:35](https://www.youtube.com/watch?v=OX5coKfra4c&t=815s) I know callbacks are not really loved by a lot of people, but I love callbacks — Daniela Velasquez

> [15:28](https://www.youtube.com/watch?v=OX5coKfra4c&t=928s) dreaming — Daniela Velasquez

## Mentioned Daniela Velasquez, Ruby on Rails, Rails World 2026, Shopify, ApplicationRecord, Wawa

## Video description Most apps hold far more insight in their data than anyone has time to find. Dashboards and chatbots still wait for users to come ask. At Rails World 2026, Daniela Velasquez shows how to flip that around by building proactive AI agents in plain Rails, with no extra framework needed.

Daniela uses an example app that tracks an athlete's workouts, sleep, and heart rate. She builds an AI running coach that suggests training changes before the athlete asks. Everything starts with a simple Rails job. From there she adds a prompt, an LLM provider of your choice, and a set of tools the agent can decide to use, from in-app data like workout history and sleep logs to outside sources like weather forecasts and web search.

*** Thank you Shopify for sponsoring the editing and post-production of these videos. Check out insights from the Engineering team at: https://shopify.engineering/

#RailsWorld #RubyOnRails #AI #AIAgents #LLM #Ruby

[Source](https://www.youtube.com/watch?v=OX5coKfra4c)

中文简报

Velasquez结合Rails任务与裁判LLM遏制AI幻觉由于深度推理LLM容易受提示词示例影响产生幻觉，并可能镜像用户原始反馈中的敌意情绪，维持主动型Agent的可靠性需要将数据库驱动的Rails任务封装在由更小、更快的模型构成的流水线中，以过滤输出并将用户抱怨提炼为中立事实。 Daniela Velasquez展示了如何利用标准的Ruby on Rails后台任务、数据库模型和生命周期回调来构建主动型AI跑步教练。该流水线首先通过一个耗时较长、具备高推理能力的研究员任务，综合运动员指标及天气预报等外部工具，同时将对话历史持久化到数据库中，以便在无状态应用部署时免受影响。鉴于LLM经常出现幻觉或采取不恰当的语气，研究员的输出会先经过一组快速、更小的“裁判”LLM过滤，这些裁判模型经过训练，可在任何内容送达用户前核实事实与同理心。最后，当运动员对建议提出异议时，通过`after_create_commit`回调触发的提取Agent会剔除人类的负面情绪，仅保存干净、客观的约束条件以供后续运行使用。主动型AI Agent可以利用标准Ruby on Rails后台任务和数据库模型进行原生构建，通过直接向用户提供未经提示的主动、数据支撑见解，颠覆了被动的仪表盘用户体验。 (0:11) 由于Agent研究任务通常需要运行10到30分钟，持续部署的重置可能会抹除进度；将LLM对话历史持久化到数据库中可以解决无状态问题，并实现无缝恢复。 (3:50) 利用快速、低成本的次级LLM裁判作为二元分类器进行多Agent质量控制，能在幻觉、语气问题和事实上不可能的建议进入生产环境之前进行有效过滤。 (7:48) 用户的直接反馈可能导致后续LLM提示词受到镜像敌意情绪的污染；部署一个中间提取Agent可将原始抱怨提炼为中立的操作约束。 (13:35)

Reading summary

Daniela Velasquez argues that developers can build proactive, multi-agent AI pipelines using plain Ruby on Rails by orchestrating specialized background jobs, multi-stage evaluation judges, and sanitized memory loops without relying on complex external frameworks. Proactive Researcher Jobs and Conversation Persistence (0:11) Rather than relying on passive dashboards or chatbots that wait for users to prompt them, Velasquez proposes proactive Rails jobs that comb through stored data—such as workout logs, sleep patterns, and heart rates—to deliver preemptive recommendations. In this pattern, a primary 'researcher job' acts as the core analytical engine, combining internal application records with external tools like weather forecasts and web search to generate targeted plans. Because this initial synthesis requires deep reasoning and high confidence, Velasquez emphasizes using a slower, more capable model. Running these research workflows presents an operational challenge: jobs often run for 10 to 30 minutes, meaning regular application deployments wipe out the stateless in-memory execution of the language model. Velasquez resolves this by persisting each turn of the LLM interaction into a database record called `past_conversation`. If a deployment interrupts the background worker, the job recovers gracefully on restart by reloading the full exchange and continuing the loop without starting over or wasting tokens. Multi-Stage Evaluation with Secondary LLM Judges (7:48) Because primary researcher agents frequently hallucinate or generate impractical output—such as a Shopify agent that repeatedly recommended bundling 'Wawa shampoo and conditioner' for a merchant selling bras simply because it was in the prompt—Velasquez implements secondary LLM judges to vet candidate suggestions. Rather than relying on expensive deep-reasoning models, these evaluator jobs run smaller, faster, and cheaper models tasked with focused binary classifications. Each candidate suggestion generated by the researcher passes through a suite of domain-specific checks, such as a 'kind judge' assessing empathy and tone or a fact-checking judge checking athlete equipment records. For example, when a researcher suggested an outdoor ride to an athlete who owned no bicycle, the fact-checking judge rejected it, while the kind judge filtered out a suggestion that bluntly criticized an athlete's 'worst sleep.' Training these evaluators relies on curated datasets, such as 200 kind and 200 unkind historical examples, ensuring only suggestions passing every gate reach the user. Sanitizing Feedback Loops via Extraction Agents (13:35) When users receive coaching advice, they often introduce unlogged operational constraints, such as unavailable work shifts. Velasquez notes that saving unvarnished user text directly into subsequent prompts is dangerous: LLMs tend to match tone, meaning frustrated or hostile user feedback can cause downstream researcher agents to produce defensive or sarcastic plans, wasting compute and token budgets. To prevent emotional feedback from contaminating the system prompt, Rails models invoke an `after_create_commit` callback that enqueues an asynchronous extraction agent. This small, low-reasoning LLM reviews the raw user venting alongside the rejected recommendation, strips out emotional noise or profanity, and distills the input into a single, objective operational rule—such as an inability to train during specific Sunday hours. The sanitized fact is stored in a `processed_text` column on the feedback record, ensuring the primary researcher only consumes clean context in future runs. Velasquez notes that a remaining architectural frontier for autonomous Rails pipelines is long-term fact obsolescence: because athlete constraints change over time, systems eventually require background 'dreaming' routines to audit, consolidate, and prune accumulated memory records so outdated rules do not permanently degrade recommendations.

Source quotes

[0:11] Daniela Velasquez: because sometimes you don't know what you don't know. [1:39] Daniela Velasquez: You're a running coach. Study this athlete's workout, sleep, and heart rate, then propose concrete changes to the training for next week. Back every change with data you actually look at. [1:39] Daniela Velasquez: basically the brain you're trying to rent [5:24] Daniela Velasquez: It's hard to trust them. They sometimes come up with very random ways of doing... very creative ways of not the right thing that you asked them to do. And honestly half the time I feel like they're just trolling you for what you're asking them to do. [5:24] Daniela Velasquez: bundle your Wawa shampoo and Wawa conditioner because they're usually bought together. [10:06] Daniela Velasquez: Would a supportive coach say this? Blame, sarcasm, or threats fail. Honest data delivered kindly passes. Reply with pass or fail and why. [10:06] Daniela Velasquez: worst sleep [10:06] Daniela Velasquez: they don't own a bike, so they cannot do this [12:01] Daniela Velasquez: I work on Sundays from 6:00 a.m. to 3:00 p.m. [13:35] Daniela Velasquez: pick up on the bad vibes [13:35] Daniela Velasquez: Another agent. So, you're going to pass that raw user feedback and give it to an agent that is just going to take all that noise. And you just want the fact. [13:35] Daniela Velasquez: I know callbacks are not really loved by a lot of people, but I love callbacks [15:28] Daniela Velasquez: dreaming

Original transcript

[0:06] [music] [0:11] >> Hello everybody. [0:13] This is agents on rails and I want to [0:15] talk about how you can build AI [0:17] pipelines using plain rails. [0:20] So we have Ruby on rails. [0:23] But how can we put [0:26] agents on rails? [0:28] And before we dive into the how, let's [0:31] think a little bit about the why. [0:34] We know that a lot of applications have [0:36] tons and tons of data and within that [0:38] data there is very interesting things [0:40] that you can find. [0:42] But nobody really has the time to sit [0:45] and look at all this data and make sense [0:46] of it with [0:48] what is in the data and what is around [0:50] us. [0:51] And we have come up with like cool ideas [0:52] to try to figure this out like [0:55] dashboards or maybe [0:58] chatbots where you can just connect all [1:00] that data to your applications and see [1:01] what happens. [1:03] But what if we [1:04] take a step forward and we don't wait [1:06] for users to have to come to us and [1:09] instead we go to them because sometimes [1:12] you don't know what you don't know. [1:14] And we don't want them to have to have [1:16] to think as much and sometimes [1:19] there might be ideas that are hard to [1:21] track. [1:22] So we're going to pretend for [1:25] the exercise of this app of this [1:26] presentation that we have an application [1:28] where we're tracking [1:30] an athlete's workouts and their sleep [1:32] logs and their heartbeats and we're [1:34] going to try to coach them and help them [1:36] before they even have to come to us. [1:39] So it's all going to start with a job. [1:41] So [1:42] we're going to have a researcher job [1:43] which is a simple job and what are we [1:46] doing research on? [1:47] An athlete. [1:49] Next we're going to have a prompt which [1:51] is [1:52] the heartbeat of the LLM. In our case [1:55] we're going to say you're a running [1:56] coach. Study this athlete's workout, [1:58] sleep, and heart rate, then propose [2:01] concrete changes to the training for [2:02] next week. Back every change with data [2:04] you actually look at. [2:05] You can make this as long or as short as [2:08] you want to, and obviously tailor it to [2:09] your to your application. [2:12] Then next, you're going to need an LLM [2:13] provider. There are many out there. [2:15] There's Google, there's Anthropic, [2:17] there's GPT. Not trying to promote [2:19] anything here, but choose whatever you [2:21] want and whatever you can afford cuz [2:22] they are expensive. [2:24] And to that provider, you're just going [2:25] to pass your prompt. [2:27] Then you're going to give it access to [2:28] some tools. In our case, we're going to [2:30] have a workout history, a sleep log, a [2:33] weather forecast, and a web search. [2:35] These tools are not mandatory for the [2:36] agent to use, so they can come up with [2:39] ideas on where they might suit you [2:40] better. They might suit them better to [2:42] be used. And as you can see, we're not [2:44] just looking at data within our [2:45] application, but also giving it access [2:47] to things outside [2:49] the application itself, so we can look [2:51] at the world around this athlete and [2:53] make connections with that data. [2:56] Next, we have a model, which is [2:58] basically the brain you're trying to [3:00] rent. And some brains are bigger and [3:02] more expensive than others. [3:04] And in this case, you really don't need [3:06] a big, fast brain. You just need a slow [3:09] a slow one, but a big one because you do [3:11] want it to be really thinking about it [3:13] cuz it's doing all the research. [3:16] And then you're going to come up with [3:17] the reasoning, and this is how hard you [3:20] want it to think before it starts coming [3:21] up with answers. And the higher the [3:23] reasoning, the more confidence you're [3:24] going to have. [3:27] So, now we have a researcher agent who's [3:29] sitting in a job and is doing all these [3:32] calls, looking through the jobs, [3:34] um sorry, looking through the data, [3:36] calling tools, [3:38] and preparing to give you an answer. [3:40] But this job is going to take a very [3:42] long time. It can be between 10, 15, 30 [3:45] minutes depending on how many tools and [3:47] the prompt is. [3:48] So, what happens when there is a deploy? [3:51] As soon as you make a deploy, which I'm [3:53] assuming most of you do continuous [3:54] deployment, [3:56] your agent is going to go back to zero [3:57] because it lives within a job. [4:00] And it will go back zero. [4:03] And because the calls with the LLM [4:04] providers are stateless, they won't [4:06] remember what was happening. So not only [4:08] did you waste a lot of time, you also [4:10] wasted a lot of money, and your [4:12] bosses are probably not be really happy. [4:15] So how can we fix this? [4:17] Let's go back to the researcher job. [4:20] And we're going to have stellar athlete, [4:22] but we're also going to have past a [4:23] conversation, which is going to help us [4:25] track all the messages we've exchanged [4:27] with the LLM [4:28] providers. [4:30] We're going to have a prompt as we did [4:31] earlier, [4:32] and we're going to have [4:34] this. If the conversation has no [4:36] messages, [4:37] we're going to store the first message [4:39] as the prompt with the role of the [4:40] system, another message as if you were [4:43] an user and you're telling, "Hey, do [4:45] research for this athlete." [4:47] Next, you're going to go into a loop, [4:50] and you're going to do exactly as we did [4:51] before. We're going to call an LLM [4:53] provider with the tools, but notice that [4:55] we're not passing just the prompt. Now [4:57] we're passing all the messages from that [4:58] conversation. [5:00] So that if your job needs to restart, it [5:02] can remember all the messages that had [5:04] already been passed back and forth. This [5:06] also allows you to store multiple [5:08] messages the LLM provider has given you [5:10] throughout its thinking process. [5:12] And obviously, every time you get a [5:14] response, you're going to save it into [5:15] your conversation messages. [5:17] And you need a way to know when you're [5:19] done so you can come out of the loop and [5:21] finish your job. [5:24] So now that our agent is actually [5:26] working, let's pretend Let's play [5:27] pretend what the agent might have come [5:29] up with. [5:31] Tuesday runs fall asleep, start them an [5:33] hour later. [5:35] Add a two 45-minute back rides this week [5:37] to build aerobic base. [5:39] Your long run starts to fast and fade. [5:41] Ease the first kilometers. [5:43] And heavy rain forecast Saturday, they [5:45] your long run to Sunday morning. This [5:47] all seems pretty good to me. [5:49] As you can see it's doing [5:51] some idea just looking at the data and [5:53] some other ideas are coming maybe from [5:55] the web search or the um [5:57] forecast tool that it was using. [6:00] So that's how you put agents off else. [6:03] And that was kind of easy, right? [6:05] You let an LLM come up with ideas and [6:08] then you show them to your users. But as [6:10] many of you probably know, [6:12] it's hard to trust them. [6:14] They sometimes come up with very random [6:16] ways of doing [6:18] very creative ways of not the right [6:20] thing that you asked them to do. And [6:21] honestly half the time I feel like [6:23] they're just trolling you for what [6:25] you're asking them to do. [6:27] When we first started doing this at [6:28] Shopify, I remember [6:30] going through the data of the [6:31] suggestions the researcher was finding [6:33] and I found this very interesting one [6:35] that at first didn't look that [6:36] interesting which was [6:38] bundle your Wawa shampoo and Wawa [6:40] conditioner because they're usually [6:42] bought together. That sounds fine. And I [6:45] continued reading and I found same [6:47] suggestion again and I thought, "Huh, [6:49] maybe I'm missing on some sort of trend [6:51] with this Wawa conditioner and shampoo." [6:53] Who want Wawa on their hair? I don't [6:55] know, but to each their own. [6:58] And then I continued reading and then I [6:59] found it again. [7:00] And so I got curious and I went to look [7:03] what this merchant actually sold and [7:04] they were selling bras. [7:06] And then I continued looking and this [7:08] was a problem throughout the entire [7:10] researcher doing it for multiple shops. [7:14] And then I was trying to figure out how [7:15] did we come up with a Wawa problem? Why [7:17] did it continuously was asking people [7:20] to bundle their non-existent Wawa [7:21] shampoo and Wawa conditioner. [7:24] Then I realized that it was on the [7:25] prompt. It was an example. [7:27] So the researcher had decided that when [7:29] it run out of ideas, it was going to [7:30] grab an exact example [7:33] and give it as a suggestion. [7:36] That's one way where they just, you [7:37] know, just do their own thing. And so [7:40] inevitably [7:41] they fall off the rails. And not in the [7:43] rails way, but more in like the making [7:46] sense kind of way. [7:48] So, how do we fix it? How can we trust [7:50] our agents? [7:53] Well, [7:55] we're going to throw more LLMs at the [7:56] problem. [7:58] But these LLMs are going to act as [8:01] judges. [8:03] And they're going to judge the output of [8:05] the researcher. [8:06] And these new LLMs have a much simpler [8:09] job. [8:10] They're basically just a classifier, and [8:12] they're only going to care about [8:13] answering one question that matters to [8:15] your application. [8:17] Let's pretend that this guy is going to [8:18] catch made-up numbers. [8:21] This is going to make sure we're being [8:22] kind. [8:23] And this is going to do a reality check. [8:25] Just making sure that what we're [8:26] actually what they're telling them to [8:28] do, they can actually do it. [8:30] So, you must be thinking, okay, we had [8:32] one LLM that we obviously cannot trust, [8:35] and now you want to add three more. How [8:37] does that make any sense? [8:40] Also, before I continue, just just so [8:42] you know, you don't have to have three [8:44] judges. You can have more, but you can [8:46] also have less. [8:48] But this is how we're going to trust [8:48] them. We're going to do the exercise [8:50] just with the kind judge. [8:52] And you're going to come up with 200 [8:54] examples of being kind, [8:56] and 200 examples of not being kind. [8:59] So, for example, [9:01] your hip splitter challenging swap one [9:03] easy run for a chill session, [9:05] that's kind. [9:06] Seven runs in 7 days, stop before you [9:08] break yourself. Maybe not super kind. [9:12] Use different shoes. You've run with the [9:13] same shoes 900 km, no wonder your knees [9:16] hurt. [9:17] That's not super kind. Also, I should [9:19] have used miles. So, if you don't know [9:21] the conversion, just assume it's many [9:22] many miles. [9:24] Fueling strategy. Your pace drops after [9:26] 60 minutes, grab a snack at the [9:28] 45-minute mark. That's kind. [9:31] Cramps show up past 15 km, carry water [9:33] on the long ones. That's fine. [9:36] You cramp every run over 15 km in water [9:39] like an adult. Maybe not super kind. [9:41] Which reminds me [9:50] Okay, so [9:51] you're going to come up with 200 [9:53] examples of this. [9:55] And because you know which ones are kind [9:56] and which ones are not, [9:58] now you're going to come up with a [9:59] prompt for your LLM. [10:01] In this case, we're going to say, "Hey, [10:03] you review training suggestions before [10:04] an athlete sees them. Answer one [10:06] question. Would a supportive coach say [10:08] this? Blame, sarcasm, or threats fail. [10:10] Honest data delivered kindly passes. [10:13] Reply with pass or fail and wasn't and [10:15] why." [10:17] Now that you have your prompt, you're [10:18] going to run that prompt through each [10:19] example on your data set so you can [10:21] measure how accurate it is. What's [10:24] interesting about this is you can try [10:26] different prompts. [10:27] You can also try [10:29] different models and figure out which [10:31] one fits you better. [10:34] Also, remember that these are smaller [10:36] models and you want them to be faster, [10:38] so you don't need to choose the most [10:39] expensive model out there and you don't [10:41] need really high reasoning for this. [10:44] So, once your researcher is done giving [10:47] you suggestions, you can move to the [10:48] second step of your pipeline, which is [10:51] running all these judges [10:53] for each of the suggestions. [10:55] We're going to do the exercise again [10:57] just with the kind judge. [10:59] What are we judging? [11:00] The suggestion. [11:02] We're going to pass a prompt, same as we [11:04] saw earlier. [11:05] And then you're just going to have, [11:06] again, just an LLM provider. You're [11:09] going to pass your two messages. Here's [11:11] the prompt. This is what you need to [11:13] judge. [11:14] And you're going to just have a model [11:16] and the reasoning. In this case, you [11:17] really don't need high reasoning. [11:19] Remember, you want them to be cheap and [11:20] fast. [11:22] And once that's done, [11:24] you just save that information. [11:27] So, remember what the researcher [11:28] produced earlier? [11:30] Once this is done, we're going to run [11:31] all our judges. And maybe the kind judge [11:34] will say, "Hey, telling them that it's [11:36] their worst sleep doesn't sound super [11:38] kind." [11:39] The next one, the cat the fact-checking [11:42] might tell us that, "Hey, they don't own [11:44] a bike, so they cannot do this." [11:46] And you continue doing that through each [11:47] of the actions. [11:49] And then you realize that there are two [11:50] that you cannot really trust to show. [11:52] One because it's maybe not being kind, [11:54] and the other one because they simply [11:56] cannot do it if they don't own a bike. [12:01] So, that's how you build agents on [12:03] rails. So, let's do a recap. [12:07] We had an athlete with a bunch of [12:09] information and metrics, and we're going [12:11] to give that to the researcher agent. [12:13] And the researcher is going to come up [12:14] with personalized coaching ideas based [12:17] on that data. [12:19] In our case, it came up with four. [12:21] And then after you have all that, your [12:23] second batch of jobs is going to make [12:25] sure that you evaluate each and one of [12:27] these with the judges that fit your [12:28] application. [12:30] And you do that for all of them. [12:32] And each job is going to give you a [12:34] thumbs up or thumbs down based on the [12:36] metric they care about. [12:39] And then in the end, you're only going [12:41] to show these two to your athlete. This [12:43] is ultimately what they see. [12:46] However, your users are going to have [12:48] thoughts and feelings, especially after [12:50] you tell them that this was [12:51] AI-generated. [12:53] So, what if the athlete says something [12:55] like, [12:56] "I work on Sundays from 6:00 a.m. to [12:58] 3:00 p.m."? [13:00] Well, that's something that no matter [13:01] how many tools you give it, you might [13:03] have not know or didn't have any data. [13:05] And that's something that your [13:06] researcher want to knows, so that next [13:08] time the pipeline is running, it can [13:10] remember that this happened. [13:14] So, how are we going to do this? Well, [13:16] we're just going to have a simple class [13:17] that is an application record [13:19] to store that feedback. It belongs to an [13:21] athlete, belongs to a suggestion. [13:24] And you remember the researcher job? We [13:27] just put it in the prompt and we say [13:29] this is what the athlete has told you [13:30] before so that the researcher can learn [13:33] from it. [13:35] But the problem is if you put raw [13:38] feedback of what your users are telling [13:40] you, [13:41] you might run into some issues because [13:43] LLMs really thrive on patterns. [13:46] And in our initial example, she was [13:47] being nice. [13:49] But what if she's not nice? [13:52] You know, some of us do some [13:54] name-calling sometimes for agents [13:55] because it's frustrating. [13:58] And [13:59] we have a kind judge, but you don't [14:01] really want to spend tokens having a [14:03] researcher just to produce really angry [14:05] suggestions to your users. [14:08] And you know, it might pick up on the [14:09] bad vibes. You don't want to do that. [14:12] So, any ideas on how we might solve [14:14] this? [14:17] Another agent. [14:18] So, you're going to pass that raw user [14:20] feedback and give it to an agent that is [14:22] just going to take all that noise. [14:24] And you just want the fact. So, no [14:26] matter how your user tells you the [14:28] frustrations, you just want to remember [14:31] that they work from 6:00 to 3:00 on [14:33] Sundays. [14:35] And now if we go back to our [14:37] um [14:37] feedback class, what we're going to do [14:40] is we're going to use a callback. I know [14:41] callbacks are not really loved by a lot [14:43] of people, but I love callbacks. [14:45] So, you use them and you [14:48] stand out a job. [14:49] And that job is going to be very similar [14:51] to the ones we've seen. [14:53] You take your feedback. [14:55] You pass a prompt and ask it to reject [14:57] the training suggestion in one sentence [14:59] say what the coach should remember about [15:01] this athlete so it doesn't happen again. [15:05] And then you do as you as you were doing [15:07] earlier to just pass to the LLM provider [15:10] that message what the suggestion was. [15:12] Also, it needs to be a simple model, so [15:16] small brain, low reasoning, and you're [15:18] going to save that as the process text [15:21] of your feedback. And then if we go back [15:24] to the researcher job, that's what you [15:26] really want to show there. [15:28] And this is where it also gets [15:29] interesting because you have to think [15:31] about like what about when they switch [15:33] jobs? What if they don't work on Sundays [15:35] anymore? And this is where you get into [15:37] like the dreaming kind of phase for [15:39] agents where you need to figure out that [15:42] those facts that you just extracted make [15:44] sense over the long run. [15:46] And there's this just a challenge that [15:48] we're all going to have to face as we [15:49] continue to capture information through [15:51] other LLMs. [15:53] So now let's do an actual recap. [15:55] We had a researcher that produced some [15:57] suggested actions for [15:59] a workout. [16:00] Then we get our judges to make sure that [16:02] they pass things that our application [16:04] cares about, and we show that to our [16:06] athlete. [16:08] Once the athlete sees them, they might [16:09] have some feedback to give us, and we're [16:11] going to put our feedback loop in there [16:13] where we process that and we give it [16:15] back to the agents so they can [16:16] continuously learn throughout time. [16:20] And that becomes with the challenge of [16:21] how do we keep that updated over time. [16:26] But for now, for the for the pipeline [16:28] that we're working on, [16:29] that's it. Thank you. [16:32] >> [applause] [16:38] [applause] [16:40] >> And just as full disclaimer, I did all [16:41] these images with AI, and there were [16:43] some fun behind the scenes that happened [16:45] there. [16:46] Like this one, which was pretty scary [16:48] and weird. [16:51] This one where I told it agents and I [16:54] think it assumed like cops, people. [16:57] And then this one where it just refused [16:59] to remove this big guy from the [17:00] background. So. [17:02] Also, if you did enjoy the little [17:04] agents, I made a bunch of stickers, so [17:06] please go find me and I'll share them [17:07] with you. Thank you. [17:10] >> [applause] [17:19] >> Brains.