RubyEvents Finds Small AI Models Beat Frontier Rivals With Rails Tools

Ruby on Rails

Pairing compact language models with prescriptive command-line generators eliminates hallucinations and yields zero-error codebase updates at a fraction of frontier model costs.

AI-assisted summary of Custom Rails Generators in RubyEvents - Rachael Wright Munn — original source. Check the original for full context.

Source published: 2026-10-01

Key ideas

Empirical benchmarking demonstrates that smaller language models paired with custom Rails generators and an AI Skill achieve zero errors at a cost of $0.39, outperforming unconstrained frontier models that incurred four errors and cost $1.83. (23:50)
Teams do not need to choose between CLIs and the Model Context Protocol; Rails generator class options and type metadata can be metaprogrammed directly into dynamic MCP tools to serve both human contributors and AI agents. (20:06)
Rails generators built on Thor provide self-documenting CLI flags and USAGE files that serve as low-token, dynamic context injection vectors for autonomous coding agents. (7:53)
Generator test flakiness in parallel test suites stems from the default shared tmp/generators directory, which can be permanently resolved by isolating runs in dynamic temporary directories via Dir.mktmpdir. (15:10)

Detailed notes

## Background Published by the Ruby on Rails YouTube channel on October 1, 2026, this presentation features RubyEvents.org maintainer Rachael Wright-Munn speaking at Rails World 2026. According to the video description, the talk covers building custom Rails generators from scratch, fixing flaky generator test suites, converting generators into MCP tools, and benchmarking AI models to demonstrate that smaller models with well-designed tools can match or exceed frontier models at lower costs.

## TL;DR Facing schema consistency challenges across both human open-source contributors and AI coding agents on RubyEvents.org, Rachael Wright-Munn implemented custom Rails generators built on Thor to standardize YAML data generation. She demonstrates how to structure generator classes, configure self-documenting class options, leverage ERB templates, and isolate generator test suites to prevent parallel execution flakiness. The talk concludes with empirical benchmarks comparing raw frontier models against generator-assisted runs, AI skills, and MCP tools, revealing that smaller, budget-friendly models paired with structured CLI tools can achieve flawless task execution for pennies.

## Atmosphere & style The session is pragmatic, engaging, and punctuated with dry developer humor, as Rachael casually jokes about acquired tech stacks and quirks in LLM behavior. Her presentation style is brisk and hands-on, transitioning smoothly from granular code walkthroughs and terminal debugging quirks to hard empirical benchmarking data. The atmosphere feels like an honest dispatch from the open-source trenches, balancing technical precision with accessible, hype-free insights on modern AI-assisted engineering.

## Bulletin - [23:50](https://www.youtube.com/watch?v=aornGet-EoI&t=1430s) Empirical benchmarking demonstrates that smaller language models paired with custom Rails generators and an AI Skill achieve zero errors at a cost of $0.39, outperforming unconstrained frontier models that incurred four errors and cost $1.83. - [20:06](https://www.youtube.com/watch?v=aornGet-EoI&t=1206s) Teams do not need to choose between CLIs and the Model Context Protocol; Rails generator class options and type metadata can be metaprogrammed directly into dynamic MCP tools to serve both human contributors and AI agents. - [07:53](https://www.youtube.com/watch?v=aornGet-EoI&t=473s) Rails generators built on Thor provide self-documenting CLI flags and USAGE files that serve as low-token, dynamic context injection vectors for autonomous coding agents. - [15:10](https://www.youtube.com/watch?v=aornGet-EoI&t=910s) Generator test flakiness in parallel test suites stems from the default shared tmp/generators directory, which can be permanently resolved by isolating runs in dynamic temporary directories via Dir.mktmpdir.

## Key takeaways - [00:06](https://www.youtube.com/watch?v=aornGet-EoI&t=6s) RubyEvents.org relies on flat YAML files in version control for spam prevention and open community archiving, but this creates high submission friction for external contributors. - [01:11](https://www.youtube.com/watch?v=aornGet-EoI&t=71s) AI coding agents and human open-source contributors experience identical friction in unfamiliar codebases, including schema errors, missing required comments, and ignored file lifecycle transitions. - [03:15](https://www.youtube.com/watch?v=aornGet-EoI&t=195s) Combining an AI Skill with a CLI generator provides an accessible interface for both automated agents and human contributors without requiring proprietary tool SDKs or token budgets. - [05:10](https://www.youtube.com/watch?v=aornGet-EoI&t=310s) Inheriting from Rails::Generators::Base instead of NamedBase allows developers to bypass mandatory positional name arguments and inflection helpers when generating files with predetermined paths. - [06:48](https://www.youtube.com/watch?v=aornGet-EoI&t=408s) Using class_option flags instead of positional arguments eliminates ordering mistakes and provides flexible defaults for both human users and AI agents. - [07:53](https://www.youtube.com/watch?v=aornGet-EoI&t=473s) Rails automatically appends generator USAGE file contents to --help output, allowing AI agents to discover syntax, flag schemas, and realistic edge-case examples through a single CLI command. - [12:21](https://www.youtube.com/watch?v=aornGet-EoI&t=741s) ERB trim tags (<%- and -%>) are necessary in Thor templates to suppress extraneous whitespace and blank lines when generating strict flat data formats like YAML. - [15:10](https://www.youtube.com/watch?v=aornGet-EoI&t=910s) Default generator test harnesses wipe out a shared tmp/generators directory between tests, introducing race conditions in parallel test runs unless isolated temporary directories are used. - [15:10](https://www.youtube.com/watch?v=aornGet-EoI&t=910s) Interactive debuggers like Pry and IRB freeze silently during generator tests because Rails suppresses standard output redirection by default. - [20:06](https://www.youtube.com/watch?v=aornGet-EoI&t=1206s) Metaprogramming Thor class_options directly into Model Context Protocol schemas allows generators to serve as dynamic MCP servers that update automatically whenever CLI options change. - [21:57](https://www.youtube.com/watch?v=aornGet-EoI&t=1317s) Unassisted frontier models produced high error rates, including hallucinated sponsor logos, casing errors, and unauthorized web requests, consuming 136k tokens and costing $1.83. - [23:50](https://www.youtube.com/watch?v=aornGet-EoI&t=1430s) Smaller sibling models beat massive frontier models when equipped with structured tools because they strictly followed CLI constraints instead of rewriting content or deviating from prompts. - [25:14](https://www.youtube.com/watch?v=aornGet-EoI&t=1514s) Ultra-budget models like Luna completed an entire experimental suite across multiple configurations for under $0.25, demonstrating viability for scoped, generator-backed automation. - [26:15](https://www.youtube.com/watch?v=aornGet-EoI&t=1575s) Benchmarking revealed that concerns over MCP token costs are largely unfounded, as MCP configurations consistently produced among the lowest token expenditures across models.

## Detailed walkthrough ### [00:06](https://www.youtube.com/watch?v=aornGet-EoI&t=6s) Preserving Community History: Why RubyEvents.org Runs on YAML

Rachael Wright-Munn, one of four maintainers of RubyEvents.org, opens her presentation by explaining the foundational architecture of the project. RubyEvents.org is an open-source platform indexing Ruby events worldwide entirely through YAML files. Showing the concise YAML file representing Rails World 2026, she quips to the audience, "You are here."

Anticipating the obvious question of why an application would rely on flat YAML files rather than a standard relational database, she walks through four explicit reasons. First, it is an inherited architectural decision. Second, managing data through version control allows maintainers to thoroughly review every submission for spam. Third, it lowers the barrier of entry by enabling any developer in the community to submit updates directly via pull requests. Lastly, and most critically for community longevity: if the application ever goes offline or gets acquired by Meetup.com, the event data will persist forever as an open, accessible historical record of the Ruby community.

```mermaid flowchart TD accTitle: RubyEvents Data Pipeline accDescr: Visual representation of how external community updates flow from social feeds into YAML git commits.

subgraph Sourcing["Event Discovery"] A["Bluesky / Web Announcements"] end

subgraph Automation["AI Agent Processing"] B["GitHub Copilot / Agent Session"] C["Parse Unstructured Event Info"] end

subgraph Repository["Git Version Control"] D["YAML File Index"] E["Community Spam Review & Merge"] end

A --> B B --> C C --> D D --> E ```

[01:11](https://www.youtube.com/watch?v=aornGet-EoI&t=71s) The Unstructured Data Challenge: AI Agents and 180+ Human Contributors

Relying on YAML introduces a major friction point: every data change requires a git commit. For maintainers looking to automate updates, taking unstructured web text and turning it into clean commits was an uphill battle before modern AI tools. Rachael shares her ideal workflow: while scrolling Bluesky, she spots an event announcement, kicks off an agent session using GitHub Copilot, and receives a clean pull request ready to merge into the repository. This exact approach was used to add the Rails World 2026 CFP, but making it work reliably required solving several subtle hurdles.

Initially, AI agents struggled with the specific conventions of the codebase. Despite being capable, the models had no internal mental model of RubyEvents.org. They frequently hallucinated non-existent keys, reused deprecated attributes (such as `event_name`, which is reserved exclusively for recurring meetups), and omitted required `# TODO` comments. Furthermore, intermediate document states proved nearly impossible for AI to deduce; for instance, the `videos.yml` file transitions through three distinct phases across an event’s lifecycle—first when speakers are announced, second when the schedule is published, and third when talk recordings are uploaded. Because agents primarily encounter completed files, they consistently neglected intermediate lifecycle states as well as valuable niche keys like `cancelled` or `original_title` (used for talks given in languages other than English). Furthermore, feeding full YAML examples to agents burned an excessive number of tokens.

Crucially, this was not just an AI issue. RubyEvents.org has more than 180 open-source contributors who face identical frustrations. Contributors know when local events update and have the raw information, but they do not know the repository's internal schema. They misspell keys, miss required fields, and spend hours parsing existing examples just to submit a single event.

- **Intermediate Data States**: AI fails to navigate files with progressive lifecycles, such as `videos.yml` evolving from speaker announcements to schedules to video URLs. - **Neglected Schema Keys**: Useful metadata fields like `cancelled` and `original_title` were frequently overlooked. - **Token Inefficiency**: Having models inspect existing YAML files consumed massive token budgets just to deduce schemas.

[03:15](https://www.youtube.com/watch?v=aornGet-EoI&t=195s) Architecture Dilemma: MCP Tools vs. Skill + CLI

To address both AI and human contributors simultaneously, Rachael evaluated two primary paradigms for teaching agents how to interact with the project: Model Context Protocol (MCP) tooling versus an AI Skill paired with a Command Line Interface (CLI). An MCP tool is an AI-native SDK designed to fetch context and execute actions directly on behalf of a model. Conversely, an AI Skill consists of discoverable Markdown documentation paired with a conventional CLI utility.

While AI agents naturally prefer and reach for native MCP tools, Rachael points out that a Skill paired with a CLI bridges the gap between machines and humans: "anybody can use that. We can all read markdown and then CLI... anybody can call them." A common critique of MCP tools is the significant context window overhead required simply to register and describe the tool schemas—and in LLMs, "context is tokens is money." Choosing the Skill plus CLI route offered a unified interface that supports both automated agents and human contributors who may not have access to paid AI tokens.

```mermaid flowchart LR accTitle: Tooling Architecture Tradeoff accDescr: Comparison of MCP tools against a Skill plus CLI approach for agents and human contributors.

subgraph MCPOption["MCP Tooling"] M1["Model Context Protocol"] --> M2["AI-Native SDK"] M2 --> M3["Heavy Context Token Overhead"] M3 --> M4["AI Agents Only"] end

subgraph SkillOption["Skill + CLI (Chosen)"] S1["Markdown Skill File"] --> S2["Standard Rails CLI Generator"] S2 --> S3["Low Token Usage via Help Docs"] S3 --> S4["AI Agents & 180+ Human Contributors"] end ```

[04:17](https://www.youtube.com/watch?v=aornGet-EoI&t=257s) Choosing Rails Generators Built on Thor

When deciding on a CLI framework, Rails generators stood out as an intuitive choice that every Rails developer already understands from running migrations or scaffolding resources. Beyond familiarity, adopting Rails generators eliminated the burden of designing CLI boilerplate from scratch. Built-in solutions already existed for:

- Argument and option parsing - File templating and directory generation - Self-generating help documentation - Automated test suites - Interactive file conflict resolution (an edge case Rachael admits was not even on her radar initially)

Rails generators achieve this by building directly on top of Thor, the battle-tested Ruby toolkit designed for authoring sophisticated command-line applications.

[05:10](https://www.youtube.com/watch?v=aornGet-EoI&t=310s) Generating a Generator: `NamedBase` vs. `Base`

To scaffold a generator for Call for Proposals (CFP) YAML files, Rachael starts with Rails' own meta-tooling: "I'm going to generate a generator because of course there's a generator to generate your generators." Running `bin/rails generate generator cfp` creates three core files:

1. `lib/generators/cfp/cfp_generator.rb` (the executable generator class) 2. `lib/generators/cfp/USAGE` (the documentation and help file) 3. `test/lib/generators/cfp_generator_test.rb` (the test suite)

By default, the scaffolded class inherits from `Rails::Generators::NamedBase`. Under `NamedBase`, the generator requires a single mandatory positional argument (`name`) and generates a vast suite of inflection helper methods based on that name (e.g., `singular_name`, `plural_name`, `file_name`), which are implemented under the hood with straightforward string interpolation.

However, because RubyEvents.org already has rigid, predetermined file paths and does not need name inflections, Rachael discarded the scaffolded defaults: "Let's throw all of that away... I know where I want my file to be and I don't need any of those helper methods." She swapped the superclass from `Rails::Generators::NamedBase` to `Rails::Generators::Base`, choosing a clean slate without forced positional arguments.

[06:48](https://www.youtube.com/watch?v=aornGet-EoI&t=408s) Arguments vs. Options: Designing the CLI Interface

When designing how users pass input to a generator, Rails provides two approaches: `argument` (positional arguments) and `class_option` (named flags). Positional arguments are declared as `argument:event, type::string, desc: "event slug"` and invoked via `bin/rails g cfp rails-world-2026`. Like positional parameters in standard Ruby methods, their meaning depends strictly on order.

Class options, on the other hand, behave like keyword arguments: ```ruby class_option:event, type::string, desc: "event slug" ``` Users call them explicitly with flags: `bin/rails g cfp --event rails-world-2026`. Class options can be marked as required, given fallback defaults, or treated as optional, and they remain completely order-independent. Rachael opted for class options across the CFP generator because they eliminate ordering mistakes for both human contributors and AI agents while providing flexibility over which values are supplied or omitted.

- **Positional Arguments**: Order-dependent, invoked without flags, functionally equivalent to positional method arguments. - **Class Options**: Order-independent, flag-driven (`--event`), support explicit flags, defaults, and optional flags.

[07:53](https://www.youtube.com/watch?v=aornGet-EoI&t=473s) Self-Documenting CLIs and Context Injection via `USAGE`

Rails generators provide built-in documentation that Rachael had never fully utilized before this project: calling `bin/rails g cfp --help` dynamically displays all configured `class_option` flags alongside their types, aliases, and descriptions. Whenever maintainers adjust class options in the generator code, the CLI help documentation updates automatically.

Even more powerfully, Rails appends the contents of the generator's `USAGE` file directly to the end of the `--help` output. Rachael recognized this as an ideal vector for context injection when guiding AI agents: "one of the things I find is that AI is reluctant to actually call documentation and files that I reference, but it's pretty good at running commands." By executing `--help`, the agent receives both syntax definitions and rich practical examples in a single command output.

The current `USAGE` file for the RubyEvents CFP generator contains four targeted examples designed to prevent common errors: 1. **Minimal example**: The baseline configuration required to generate a valid CFP file. 2. **Ongoing meetup CFP**: Demonstrates how to handle recurring meetups that do not feature an end date. 3. **Lightning talks**: Illustrates fields specific to talk tracks and open mic formats. 4. **Updating or extending a CFP**: Guides agents and humans on how to mutate an existing file when deadlines shift, addressing the difficulty AI models face with evolving lifecycle states.

[10:05](https://www.youtube.com/watch?v=aornGet-EoI&t=605s) Thor Templates (.tt) and ERB

With inputs defined and documentation in place, the generator needs to render actual files. Rails generators rely on Thor templates, designated by the `.tt` file extension. Thor templates are standard ERB templates, making file definition simple. Rachael took the raw YAML from the Rails World 2026 CFP, placed it directly into the `.tt` template file, and replaced static values with ERB tags reading from the generator's `options` hash.

As an aside, she highlights that this templating mechanism is not limited to custom generators: developers can customize Rails' built-in default generators (such as scaffold or model templates) by placing customized `.tt` templates under the application's `lib/templates` directory, as outlined in Section 6 of the official Rails Generators Guide.

[10:49](https://www.youtube.com/watch?v=aornGet-EoI&t=649s) Wiring the Template to Generator Methods

With the ERB template written, the next step is wiring it directly into the generator class by introducing two core methods: `cfp_file_path` and `create_cfp_file`. The implementation of `create_cfp_file` is clean and declarative: it invokes Thor's `template` method, passing the template source file and the destination path where the resulting file should live. Rails generators automatically handle where to locate template files via a boilerplate snippet placed at the top of generated generator classes: `source_root File.expand_path("templates", __dir__)`. Capturing this kind of repetitive glue code automatically is precisely where generator tooling shines.

```mermaid flowchart LR accTitle: Rails Generator File Execution Flow accDescr: Flow diagram showing how templates and options resolve into generated files in RubyEvents. subgraph Inputs["Inputs & Templates"] direction TB SR["source_root: templates/cfp.yml.erb"] OPT["Generator Options: --event, --open-date, etc."] end subgraph Engine["Generator Execution"] direction TB TMPL["template helper"] DEST["cfp_file_path(destination_root)"] end subgraph Output["Target Filesystem"] direction TB OUT["data/{event_series}/{event}/cfp.yml"] end SR --> TMPL OPT --> TMPL TMPL --> DEST DEST --> OUT ```

For the target destination, RubyEvents organizes its file tree by placing CFP data under the directory path `data/{event_series}/{event}/`. The generator relies on `destination_root`, a value provided by Rails that defaults to whichever working directory the command was invoked from. Running the generator for the first time with all required inputs yields instant gratification: it executes, injects all passed values into the template, and outputs a complete, valid CFP file.

- `template` helper: Takes the source ERB template filename and destination path as arguments. - `source_root`: Configures Thor's lookup path using `File.expand_path("templates", __dir__)`. - `destination_root`: Contextual root path supplied by Rails that defaults to the invocation directory.

[12:21](https://www.youtube.com/watch?v=aornGet-EoI&t=741s) Handling File Conflicts, Optional Fields, and ERB Whitespace

Calling the generator a second time without start and end dates exposes two real-world edge cases. First, because a `cfp.yml` already exists on disk from the previous execution, Rails triggers an interactive file conflict prompt. Thor's `template` helper handles this automatically out of the box, allowing the developer to inspect the diff or confirm an overwrite directly from the command line. Second, RubyEvents enforces strict schema validations: if `open_date` and `close_date` keys are present in the YAML file, they cannot be blank and must be formatted as valid dates. For open-ended community meetups with rolling CFPs, those keys need to be omitted entirely when omitted on the command line.

The template needs conditional ERB logic wrapping both date attributes: `<% if options[:open_date] %>` and `<% if options[:close_date] %>`. While standard ERB works as expected, it leaves behind unwanted blank newlines in the raw text output. In browser HTML rendering, extraneous blank lines are invisibly collapsed, but in structured data formats like YAML, messy whitespace creates visual noise and formatting inconsistencies. By switching from standard ERB tags to ERB trim tags (`<%-` and `-%>`), trailing and leading whitespace are stripped out cleanly, leaving behind properly formatted YAML with exactly one terminal newline.

- Thor handles existing file collisions natively by pausing execution and asking the user whether to overwrite, skip, or diff the file. - Optional YAML attributes require conditional guard clauses around keys to prevent empty values from failing application data validations. - ERB hyphen/trim tags (`<%-... -%>`) suppress formatting gaps and blank lines that would otherwise pollute generated flat data files.

[13:51](https://www.youtube.com/watch?v=aornGet-EoI&t=831s) Testing Generators with `run_generator` and `assert_file`

With conditional logic living inside the template, the generator requires automated test coverage. Rails provides a generator test harness out of the box (`CFPGeneratorTest`). Simply uncommenting the default generated test and executing it immediately results in a failing test reporting that no value was provided for the required `--event` option.

```ruby # Passing CLI flags to the generator test harness run_generator(["--event", "rails-world-2026", "--open-date", "2026-06-01"]) ```

To exercise the generator correctly in tests, arguments are passed to the `run_generator` helper as an array of strings that mirror exact CLI flags and arguments. Pairing flag names and values explicitly into paired elements makes test setup clear and readable. Once executed, Rails provides assertions such as `assert_file` to verify file creation. Assertions can match raw file contents directly or check output using regular expressions. While tests might produce a passing green dot, verification feels incomplete without checking the generated disk output directly. Digging into the file tree reveals that generated files actually persist on disk under `tmp/generators/` even after the test suite completes.

- `run_generator`: Accepts an array representing command-line invocation tokens (e.g., `["--event", "rails-world"]`). - `assert_file`: Validates the existence of a file at a specified relative destination path and verifies its contents via regex or string matching. - Persisted test output is stored by default inside the application's `tmp/generators` directory.

[15:10](https://www.youtube.com/watch?v=aornGet-EoI&t=910s) Taming Test Flakiness and Silent Debugging Failures

Investigating why files persist in `tmp/generators` without triggering file conflict prompts reveals the internal mechanics of Rails' generator test harness:

```mermaid flowchart TD accTitle: Generator Test Isolation Architecture accDescr: Comparison of naive default generator setup versus isolated temporary directory approach. subgraph DefaultHarness["Default Setup (Flaky in Parallel)"] D1["Fixed directory: tmp/generators"] D2["setup:prepare_destination"] D3["Entire tmp/generators wiped and recreated"] D1 --> D2 --> D3 end subgraph IsolatedHarness["Recommended Isolated Setup"] I1["Dir.mktmpdir created per test run"] I2["destination = isolated_temp_dir"] I3["teardown: FileUtils.remove_entry temp_dir"] I1 --> I2 --> I3 end ```

The standard generator test template sets `destination File.expand_path("../tmp/generators", __dir__)` and invokes `prepare_destination` inside a `setup` block. Under the hood, `prepare_destination` recursively deletes the entire directory and recreates it before every test. While simple, this shared-folder approach creates dangerous race conditions whenever tests run across parallel workers—one thread wipes out `tmp/generators` while another is actively writing or asserting against it.

To eliminate flaky generator specs completely, the test class should allocate a unique temporary directory via `Dir.mktmpdir` for each test run and clean it up inside `teardown`, omitting `prepare_destination` entirely.

Attempting to inspect intermediate generator state by inserting a `binding.irb` or `binding.pry` triggers an unnerving freeze: the test hangs silently, displaying no prompt or terminal output until the user blindly types `quit` and hits enter. Rails suppresses standard output during generator runs by default. As documented in Section 10 of the official Rails Generator Testing guide, standard output redirection must be explicitly overridden:

- Configure `Rails.application.config.rails_log_to_stdout = true` (or pass standard out logging flags) to enable interactive terminal debuggers like IRB and Pry. - Leaving stdout uncaptured in normal runs will flood test suites with generator terminal noise, which is why Rails silences output in the first place. - Replace static destination paths with isolated dynamic temp directories to guarantee parallel test isolation. - Essential generator resources: Garrett Dimon's blog, the official Rails Guides, and source code from Rails itself and production open-source apps like RubyEvents.org.

[19:11](https://www.youtube.com/watch?v=aornGet-EoI&t=1151s) Structuring AI Skills for CLI Generators

Having hardened the CLI interface, the focus turns to the other half of the agent integration equation: the AI "Skill". A skill acts as an instructional markdown document that guides an AI coding agent on how to interact with the repository. It begins with YAML front matter detailing when and how the agent should activate the skill, followed by markdown instructions highlighting codebase conventions and edge cases that LLMs frequently botch. Because RubyEvents functions as an archive and index of global community events, maintaining exact accuracy in event metadata and CFP timelines is non-negotiable.

The skill document outlines concrete checklist items for the agent's workflow: 1. Run `bin/rails g cfp --help` first so the model parses the auto-generated documentation, flag descriptions, and usage examples directly into its context window. 2. Formulate and execute the specific `bin/rails g cfp` invocation matching the user request. 3. Generate a screenshot or run data verifications to confirm visual and functional correctness before finishing.

By structuring the document around CLI help discovery, the model discovers its own parameters dynamically rather than hallucinating options.

[20:06](https://www.youtube.com/watch?v=aornGet-EoI&t=1206s) From "CLI or MCP" to "CLI AND MCP": Generating Model Context Protocol Tools

The original design question was framed as an either/or dilemma: use a skill with a CLI, or build custom Model Context Protocol (MCP) servers. However, during a presentation at RubyConf, Andy Croll demonstrated techniques for leveraging OpenAI JSON schemas through metaprogramming to build dynamic Ruby LLM MCP tools. This sparked an immediate realization: the exact same `class_option` metadata, type contracts, descriptions, and required flags already defined on the Rails generator can be metaprogrammed into an MCP tool automatically.

Rather than maintaining separate tools, RubyEvents uses a tool-generation script that reflects over Rails generator classes (such as `CFPGenerator`), extracts their Thor `class_options`, and synthesizes corresponding MCP tool definitions for a Ruby LLM MCP server. Any time an engineer or open-source contributor updates a generator's flags or validations and merges the PR, the corresponding MCP tool updates automatically upon server restart.

This bridge allows direct empirical benchmarking between the two paradigms: testing Skill + CLI versus native MCP tools across real models, starting tests against frontier models by rolling back manual codebase improvements to measure raw tool efficacy.

[21:57](https://www.youtube.com/watch?v=aornGet-EoI&t=1317s) Benchmarking Frontier Models: Raw AI vs. Generators, Skills, and MCP

To measure the real-world value of custom generators and AI tooling, Rachael established a rigorous benchmarking suite centered on a single end-to-end task: adding Rails World 2026 to the RubyEvents repository. To isolate the effects of the tools and prevent network fluctuations from skewing the data, she provided each run with a static Markdown source file rather than allowing live web scraping. After each run completed, the resulting output was checked against the test suite using `bin/rails validate:all`. She began by putting a frontier model through four distinct environments: purely unconstrained AI with zero specialized tooling, an AI instructed to use custom Rails generators, an AI running with an explicit skill wrapping the CLI generator, and finally an AI operating via the Model Context Protocol (MCP).

```mermaid flowchart TD accTitle: Frontier Model Benchmark Configurations accDescr: Comparison of four tooling approaches tested with a frontier AI model adding event data. subgraph Raw["No Tooling"] A["Frontier Model (Raw)"] -->|"Unconstrained Web & Edits"| E1["4 Errors / $1.83<br/>136k tokens"] end subgraph Generator["Generator Only"] B["Frontier + CLI Generator"] -->|"Standard CLI Execution"| E2["2 Errors<br/>131k tokens"] end subgraph SkillCLI["Skill + CLI"] C["Frontier + Skill + CLI"] -->|"Scoped Workflow Guide"| E3["1 Error (Em Dash)"] end subgraph MCPTool["MCP Integration"] D["Frontier + MCP"] -->|"Direct Tool Protocol"| E4["Quit Prematurely<br/>Lowest Token Count"] end ```

The baseline test—relying entirely on the frontier model with no generators, no custom skills, and no MCP—proved chaotic. Lacking structural guardrails, the model pulled in stale historical context from the existing repository. It hallucinated old 2023 sponsor logos instead of the 2026 roster, introduced casing blunders such as lowercasing "1Password" and splitting "TableCheck" into two words, and performed an unauthorized web fetch to grab a URL for the Palmer Events Center. All told, the raw approach notched four distinct errors while burning through 136,000 tokens and costing $1.83.

Adding the custom generator without a dedicated skill immediately halved the failure rate. Prompted simply to "add Rails World 2026" using custom generators, the model dropped to two errors: it omitted PlanetScale and replaced em dashes (—) with standard hyphens in talk descriptions—leading Rachael to joke dryly that "the big AI companies are really listening to us." This run consumed 131,000 tokens. When she layered the AI skill on top of the CLI generator, accuracy climbed higher still, producing only a single error (the recurring em dash substitution).

The MCP evaluation, however, yielded an unexpected stumbling block: the frontier model simply gave up midway through execution, leaving behind no videos, no schedule, and no sponsors. While that made it technically the cheapest run with the lowest token footprint, it was an outright operational failure.

- **Baseline (No Tools):** 4 errors (2023 sponsor logos reused, casing mistakes on 1Password and TableCheck, unauthorized URL fetch for Palmer Events Center), 136,000 tokens, $1.83 cost. - **Generator Only:** 2 errors (missed PlanetScale, converted em dashes to hyphens), 131,000 tokens. - **Skill + CLI:** 1 error (em dash converted to hyphen). - **MCP Tooling:** Premature termination without writing videos, sponsors, or schedule; lowest token count and cost, but failed execution.

[23:50](https://www.youtube.com/watch?v=aornGet-EoI&t=1430s) Smaller Models Outperform Frontier Giants with Better Tools

Turning her attention to the frontier model’s smaller sibling, Rachael discovered an even more compelling trend. When left completely unassisted without tools, the smaller model failed spectacularly: it renamed keynote speeches, stripped talk descriptions entirely, and assigned the formatting normally reserved for lunch breaks and coffee pauses to all the speaker sessions, ensuring none of them would link properly. When allowed to use generators alone, it struggled with repo-level context—copying over historical involvements (including former MCs and volunteers), wiping out `TODO` comments, and inserting an unnecessary event name parameter.

The tables turned completely once structured tooling entered the picture. With MCP, the smaller sibling performed admirably, making only two mistakes: an erroneous involvement addition (which was at least the Rails Foundation) and the ubiquitous em dash conversion. But the breakthrough came with the **Skill + CLI** configuration: the smaller model executed the entire Rails World 2026 addition with **zero errors** at a total cost of just **$0.39**.

This stark result exposed an architectural reality of modern LLMs: smaller models paired with well-defined tools consistently beat massive frontier models on scoped, repeatable coding chores. Because the smaller model lacked the conversational ambition of a flagship model, it followed instructions obediently. It never went "off script," never tried to rewrite speaker descriptions according to its own editorial tastes, and did not waste tokens inspecting adjacent files to second-guess whether the generated scaffolding was correct.

[25:14](https://www.youtube.com/watch?v=aornGet-EoI&t=1514s) Ultra-Budget Testing: The Luna Evaluation

Encouraged by these results, Rachael extended her experiment to include Luna, an ultra-low-cost model highlighted in a recent Rails Foundation AI model report for its surprisingly high accuracy score relative to its rock-bottom operating expense.

Luna showed acute weaknesses when dealing with complex scheduling logic. In earlier, less constrained configurations, it populated incorrect timestamps across the timetable, and when running under MCP, it failed to produce a schedule at all. When paired with the Skill + CLI pipeline, however, its only structural omission was dropping Day 0 from the event schedule. Its remaining slip-ups mirrored the other models: replacing full-length talk descriptions with clipped, one-line summaries and turning em dashes into hyphens under MCP.

What made the Luna benchmark remarkable was the economics. Despite running through the entire gamut of multi-step evaluation setups, the cumulative bill for every single Luna run combined came in under twenty-five cents. For high-volume maintainers and automated CI pipelines, such extreme cost efficiency demonstrates that budget models can handle substantial code generation workloads if paired with robust validation and generators.

- **Baseline / Early Configurations:** Inaccurate schedule timestamps, talk descriptions compressed into one-liners. - **MCP Configuration:** Failed to output a schedule; swapped em dashes for hyphens. - **Skill + CLI Configuration:** Omitted Day 0, but completed the rest of the generation cleanly. - **Total Suite Cost:** Less than $0.25 across all experimental variations combined.

[26:15](https://www.youtube.com/watch?v=aornGet-EoI&t=1575s) Key Findings and Actionable Takeaways for Rails Developers

Synthesizing her data across all three model tiers and four tooling configurations, Rachael drew two major analytical conclusions. First, while custom skills delivered the lowest aggregate error rates, they came with higher token overhead and financial expense. Second, the widespread anxiety that MCP tooling is inherently cost-prohibitive is overblown; across all benchmarks, MCP runs consistently ranked among the lowest in token consumption and raw expenditure.

To close out the talk, Rachael left the Rails World audience with five core recommendations:

1. **You are now dangerous enough to write custom generators:** Building generators using `Rails::Generators::Base` or `NamedBase` requires minimal code, handles file conflicts elegantly, and generates its own help documentation. 2. **Explicitly instruct your agents to use generators:** Reminding AI assistants in system prompts and skill instructions to lean on native Rails generators dramatically boosts output reliability while suppressing token costs. 3. **Reach for small models with superior tools:** Instead of defaulting to flagship, expensive engines like Claude Opus for every software engineering task, deploy compact models backed by prescriptive CLIs and generators. 4. **Tailor existing generator templates to corporate standards:** Maintainers and teams should customize the default templates shipped with Rails to match company style guides, eliminating tedious code review iterations on first-pass PRs. 5. **Build for humans and AI alike:** Open-source contributors and autonomous agents face the exact same challenge: stepping into an unfamiliar codebase without context. Tooling that makes human onboarding effortless will make AI agents significantly more effective, and vice versa.

Rachael concluded her presentation to applause, introducing herself as Rachael Wright-Munn (often going by "Chael" or "Rachael"), pointing attendees to her social presence at `@chaelcodes`, and inviting everyone to catch her regular live streams on Twitch and YouTube every Sunday at 18:00 UTC.

## Notable quotes > [00:06](https://www.youtube.com/watch?v=aornGet-EoI&t=6s) You are here. — Rachael Wright-Munn

> [03:15](https://www.youtube.com/watch?v=aornGet-EoI&t=195s) anybody can use that. We can all read markdown and then CLI... anybody can call them. — Rachael Wright-Munn

> [03:15](https://www.youtube.com/watch?v=aornGet-EoI&t=195s) context is tokens is money. — Rachael Wright-Munn

> [05:10](https://www.youtube.com/watch?v=aornGet-EoI&t=310s) I'm going to generate a generator because of course there's a generator to generate your generators. — Rachael Wright-Munn

> [05:10](https://www.youtube.com/watch?v=aornGet-EoI&t=310s) Let's throw all of that away... I know where I want my file to be and I don't need any of those helper methods. — Rachael Wright-Munn

> [07:53](https://www.youtube.com/watch?v=aornGet-EoI&t=473s) one of the things I find is that AI is reluctant to actually call documentation and files that I reference, but it's pretty good at running commands. — Rachael Wright-Munn

> [21:57](https://www.youtube.com/watch?v=aornGet-EoI&t=1317s) the big AI companies are really listening to us. — Rachael Wright-Munn

> [26:15](https://www.youtube.com/watch?v=aornGet-EoI&t=1575s) You are now dangerous enough to write custom generators — Rachael Wright-Munn

## Mentioned 1Password, Andy Croll, Bluesky, Claude Opus, Garrett Dimon, GitHub Copilot, Luna, Meetup.com, Model Context Protocol (MCP), OpenAI, Palmer Events Center, PlanetScale, Rachael Wright-Munn, Rails Foundation, Rails World 2026, Ruby on Rails, RubyConf, RubyEvents.org, Shopify, TableCheck, Thor, Twitch, YouTube

## Video description How do you teach both AI agents and 180+ open-source contributors to work with a codebase they don't know? At #RailsWorld 2026, RubyEvents.org maintainer Rachael Wright-Munn shows how custom Rails generators became the answer. She walks through building one from scratch, covering NamedBase vs. Base, arguments vs. options, auto-generated help docs, Thor templates, and file conflicts. She also shares practical fixes for flaky generator tests. Rachael then pairs the CLI with an AI skill, turns the same generators into MCP tools, and benchmarks every approach across models. The results show that smaller models with better tools can match or outperform frontier models on scoped, repeatable tasks for a fraction of the cost.

Slides found here: https://speakerdeck.com/chaelcodes/custom-rails-generators-in-rubyevents-rails-world-2026

***

Thank you Shopify for sponsoring the editing and post-production of these videos. Check out insights from the Engineering team at: https://shopify.engineering/

[Source](https://www.youtube.com/watch?v=aornGet-EoI)

中文简报

RubyEvents发现小型AI模型借助Rails工具表现优于前沿模型将轻量级语言模型与规范性命令行生成器相结合，可消除幻觉并以极低的前沿模型成本实现零错误代码库更新。面对RubyEvents.org上人类开源贡献者与AI编程智能体所共同遭遇的架构一致性难题，Rachael Wright-Munn采用基于Thor构建的自定义Rails生成器来标准化YAML数据生成。她演示了如何组织生成器类结构、配置具备自解释功能的类选项、利用ERB模板，以及隔离生成器测试套件以避免并行执行的不稳定性。演讲最后通过实证基准测试，对比了前沿模型直接运行与借助生成器、AI技能及MCP工具运行的表现，结果表明，经济型小模型搭配结构化CLI工具，仅需极低成本即可实现零失误的任务执行。实证基准测试表明，小型语言模型搭配自定义Rails生成器与AI Skill时，以0.39美元的成本实现了零错误，表现优于出现4处错误且成本达1.83美元的无约束前沿模型。 (23:50) 团队无需在CLI与Model Context Protocol之间二选一；Rails生成器的类选项和类型元数据可以直接通过元编程转换为动态MCP工具，同时服务于人类贡献者和AI智能体。 (20:06) 基于Thor构建的Rails生成器提供了自解释的CLI标志和USAGE文件，可作为自主编程智能体的低Token消耗动态上下文注入载体。 (7:53) 并行测试套件中生成器测试的不稳定性源于默认共享的tmp/generators目录，通过Dir.mktmpdir将运行隔离在动态临时目录中即可彻底解决该问题。 (15:10)

Reading summary

Rachael Wright-Munn demonstrates that pairing custom Rails generators with AI skills creates a self-documenting interface that prevents schema errors for human contributors and coding agents alike. By enforcing strict scaffolding, compact language models can execute complex repository updates with zero errors at a fraction of the cost of unassisted frontier models. Schema Friction in Flat-File Architectures (1:11) RubyEvents.org maintains its global index of community events through flat YAML files stored in Git to guarantee long-term data preservation, spam mitigation, and open pull requests. However, managing structured data through commits creates severe operational friction. The project's more than 180 open-source contributors and AI agents lack internal mental models of the repository's schema, often misspelling keys or failing to deduce intermediate document states—such as a talk tracking file evolving from speaker confirmations to schedules and final recordings. Without explicit tooling, automated agents burned substantial token budgets parsing sample files, hallucinated obsolete keys such as the recurring meetup attribute `event_name`, and routinely stripped necessary `# TODO` comments. Human contributors faced identical hurdles, spending excessive time reverse-engineering existing YAML files just to submit single-event updates. Leveraging Thor and Custom Rails Generators (4:17) To eliminate custom CLI boilerplate, Wright-Munn implemented custom generators built on Thor. Scaffolding via `Rails::Generators::Base` instead of `NamedBase` discarded unwanted positional argument inflections in favor of explicit `class_option` flags. These flag-based options mimic keyword arguments, eliminating input-ordering mistakes for humans and machines while allowing optional fields to be safely omitted. Thor ERB templates (`.tt`) using hyphenated trim tags (`<%-` and `-%>`) prevent blank lines from breaking YAML data validations, and Thor natively handles interactive file conflict resolution. Rails generators also provide a built-in vector for agent context injection through their help output. When invoked with `--help`, the CLI dynamically outputs option flags alongside the contents of the generator's `USAGE` markdown file. Because AI models reliably run command-line tools even when reluctant to read project documentation files, this command feeds the agent syntax parameters and targeted examples—such as ongoing meetups or schedule revisions—directly into its context window. Metaprogramming Model Context Protocol Tools (20:06) Rather than choosing between a Markdown skill paired with a CLI utility or native Model Context Protocol (MCP) tooling, Wright-Munn unified both architectures. While MCP provides direct SDK access for AI agents, it historically introduced context overhead and excluded human contributors who lack AI tokens. Conversely, CLI tools serve all 180+ contributors but require explicit agent invocation. Drawing on techniques demonstrated by Andy Croll, Wright-Munn automated MCP tool creation by metaprogramming over the generator classes. A reflection script inspects Thor `class_options`, argument types, and descriptions directly from the Rails generator, synthesizing schema definitions for a Ruby LLM MCP server. Any schema adjustment made to a generator updates the MCP server upon restart, eliminating duplicate maintenance across interfaces. Benchmark Results Across Model Tiers (21:57) Wright-Munn benchmarked four tooling configurations across different model tiers on an identical task: parsing a static Markdown file to add Rails World 2026 data. An unassisted frontier model committed four distinct errors—including reusing 2023 sponsor logos, altering brand casing, and attempting an unauthorized web fetch—costing $1.83 across 136,000 tokens. Adding custom generators reduced errors to two, and combining the generator with an AI skill left only a single punctuation discrepancy. In contrast, running the frontier model under native MCP halted execution prematurely without writing schedule, sponsor, or video data. The most notable efficiency occurred with the frontier model's smaller sibling. While unassisted smaller models failed entirely, pairing the smaller sibling with a Skill and CLI generator completed the entire task with zero errors at an operating cost of $0.39. Wright-Munn noted that smaller models followed prescriptive generator rules without attempting to editorialize descriptions. Further evaluation using the ultra-budget model Luna completed tests across all pipelines for under $0.25 total, while MCP configurations across models consistently recorded the lowest token expenditures. Wright-Munn concludes that engineering teams should customize built-in Rails generator templates and explicitly instruct AI assistants to use them. Because external contributors and automated agents encounter the same friction when entering unfamiliar codebases, building robust, self-documenting command-line tools provides a single architectural solution that improves contributor onboarding while making automated generation cheaper and more reliable.

Source quotes

[0:06] Rachael Wright-Munn: You are here. [3:15] Rachael Wright-Munn: anybody can use that. We can all read markdown and then CLI... anybody can call them. [3:15] Rachael Wright-Munn: context is tokens is money. [5:10] Rachael Wright-Munn: I'm going to generate a generator because of course there's a generator to generate your generators. [5:10] Rachael Wright-Munn: Let's throw all of that away... I know where I want my file to be and I don't need any of those helper methods. [7:53] Rachael Wright-Munn: one of the things I find is that AI is reluctant to actually call documentation and files that I reference, but it's pretty good at running commands. [21:57] Rachael Wright-Munn: the big AI companies are really listening to us. [26:15] Rachael Wright-Munn: You are now dangerous enough to write custom generators

Original transcript

[0:06] [music] [0:12] Starting off the right foot there. Hello [0:14] everybody. My name is Rachel Wrightman [0:16] and as Michelle said, I am one of four [0:18] maintainers on the rubyvents.org team. [0:21] Now rubyvents.org org is an open-source [0:23] application where we index Ruby events [0:26] from around the community using YAML. [0:30] So, this is all of the YAML that's [0:31] required to describe Rails World 2026. [0:35] You are here. [0:39] Okay, so you're probably wondering why [0:42] we use YAML and I'm just going to get it [0:44] out of the way really quick so we can [0:45] move on to what we're here for. First, [0:48] it's an inherited architectural [0:49] decision. Second, it allows us to review [0:52] events for spam. Third, it allows any [0:55] developer to update events in the index. [0:58] And last, but perhaps most importantly, [1:00] if our app ever goes offline or is [1:02] acquired by meetups.com, [1:05] our data will still persist and be part [1:07] of the historical record of the Ruby [1:09] community. [1:11] Now using YAML introduces an interesting [1:14] technical challenge in that data changes [1:17] require commits [1:19] and if we wanted any automation then I [1:23] needed a tool that would take [1:25] unstructured data and turn it into [1:27] commits and this was a rather [1:29] challenging proposition prior to the age [1:33] of AI. GitHub Copilot is a tool that [1:36] will take an issue and convert it into a [1:39] nice little PR for me. So I can scroll [1:41] blue sky, I can see an update, and then [1:44] I can fire off a quick agent session and [1:46] get a PR that I can merge into Ruby [1:48] events. Spoiler alert, this will work [1:51] and is how the CFP for Rails World 2026 [1:54] was added. But it didn't work at first. [1:58] AI is super smart, but it doesn't know [2:00] our system. It would invent keys. It [2:03] would use old and deprecated formats [2:05] like event name which we only use for [2:07] meetups. It didn't include to-do [2:09] comments. Our videos file goes through [2:12] three different phases. The first when [2:14] the speakers are announced, the second [2:16] when the schedule is released, and the [2:17] third when the actual videos are [2:19] released. Intermediate states like that [2:21] are challenging for AI to capture [2:23] because they mostly see the finished [2:25] result. We also found that useful keys [2:28] like cancelled or original title which [2:30] is used for talks in other languages [2:32] would be neglected because it was [2:35] unaware or it didn't catch those when it [2:38] was uh checking our data. The other [2:41] problem is that it spends lots of tokens [2:43] understanding examples. [2:46] We have this problem with contributors [2:48] as well. We have 180 plus contributors [2:52] on rubyvents.org [2:54] and increasing all of the time. And they [2:57] have the exact same problem. They're [2:58] super smart. THEY KNOW WHEN EVENTS are [3:00] updating. They have data that we need, [3:02] but they don't know our system. They [3:05] would typo keys and they would spend [3:06] lots of time trying to understand our [3:09] examples. So, I had the question, how do [3:12] I teach them both about Ruby events? For [3:15] AI, it's kind of a solved problem. It [3:18] comes down to MCP or skill versus skill [3:21] plus CLI. So an MCP is an AI native SDK [3:24] that fetches context or executes [3:26] commands for a model. And then a skill [3:29] is markdown documentation discoverable [3:31] by AI that's paired with a normal CLI. [3:34] Now MCP tooling is undeniably preferred [3:36] by AI and it tends to reach for them. [3:39] But the skill plus CLI, I mean anybody [3:42] can use that. We can all read markdown [3:44] and then CLI or CLI anybody can call [3:48] them. Now, one of the biggest arguments [3:51] that I see in the debate between MCP and [3:53] skill plus CLI is the context that's [3:56] consumed in describing an MCP tool and [3:59] obviously context is tokens is money. Uh [4:02] so in the end I looked at it and I made [4:05] the decision to go with the scale plus [4:06] CLI not just for the benefits that are [4:09] built into it but also because it allows [4:11] me to support those 180 contributors and [4:15] I know that not all of them have tokens. [4:17] So the next step was picking my CLI [4:20] format. This is where the generators [4:22] come into play. So Rails generators are [4:24] a really familiar pattern. Most of us [4:26] have run a Rails generator to generate a [4:28] migration or to scaffold an endpoint. [4:33] And there was another major benefit [4:36] which is that I didn't have to make any [4:37] of these decisions. arguments and [4:40] options, templating and file generation, [4:42] documentation, testing, and file [4:45] conflicts, which wasn't even on my radar [4:47] when I started working on the CLI, were [4:49] all handled for me, so I could focus on [4:52] customizing and working with the actual [4:53] tooling. As a bonus, I learned more [4:56] about a fundamental piece of Rails [4:57] infrastructure, starting with Rails [5:00] didn't have to make these decisions [5:01] either. Rails generators are based on [5:03] Thor, which is a toolkit for building [5:07] CLI uh applications. [5:10] All right, so we're all together on the [5:12] same page. I want to make a generator to [5:15] make my YAML files that describe events. [5:18] So let's start. I'm going to generate a [5:20] generator because of course there's a [5:22] generator to generate your generators. [5:26] It's going to give us all of these [5:28] files. It's going to give us the CFP [5:30] generator. It's going to give us a usage [5:33] file. And then it's also going to give [5:35] us a CFP generator test. [5:38] Now, you might be wondering like how [5:40] does it name things? Because it's weaved [5:42] that CFP through pretty thoroughly. And [5:44] the first hint is in the class that it [5:46] generates with for the CFP generator, [5:48] named base. So named base takes one [5:52] argument and then it has all of these [5:54] methods associated with it that output [5:57] different versions of CFP. [6:00] We can actually take a little peek at [6:01] that argument and see that it says [6:03] argument name type string. And then oh [6:07] it gave us even more methods. That's [6:09] fun. Let's take a cheeky little peek at [6:11] how those are defined in Rails. They're [6:13] pretty straightforward. Mostly string [6:15] interpolation. And oh, it gave us even [6:18] more. Wow. We got so many methods from [6:21] named base. Let's throw all of that [6:24] away. So, I don't need a name. I know [6:26] where I want my file to be and I don't [6:28] need any of those helper methods. So, [6:30] we're going to substitute named base for [6:32] base here. [snorts] Let's kind of [6:34] summarize the differences between these. [6:36] So, named base is going to accept one [6:38] argument name. It gives you tons of [6:40] simple helper methods based on the name [6:42] and it inherits from base. Base has none [6:46] of that and it's the one that I picked. [6:48] Now I mentioned that we have that [6:50] argument name. That's how a CLI will [6:53] accept input. But there's actually two [6:55] ways to do it. There's arguments and [6:56] then there's options. [snorts] Now the [6:59] arguments look like this. You have [7:00] argument event type string description [7:02] event slug and then when you're calling [7:04] the CLI it just goes bin rails gcfp [7:08] rails world 2026. [7:10] And the options can be either required [7:12] or default and it's dependent on order. [7:13] So you can think of this a lot like [7:15] positional arguments in a method. Next [7:18] we have options. So the class options is [7:20] structured almost exactly the same but [7:22] with class option instead. And what you [7:24] can see is that when we call the CLI we [7:26] now have this flag event that we're [7:29] passing the value to. This means that it [7:31] can be required default or optional and [7:33] it's order independent. So you can think [7:35] of it like the quark instead of method. [7:38] So, if I'm looking at these two options, [7:40] for me, I'm going to go with the options [7:42] because I feel like that makes it easier [7:43] for my contributors and agents to avoid [7:45] issues with ordering, and it gives me a [7:48] lot more flexibility when I'm deciding [7:49] what values I want to pass into it and [7:51] exclude. [7:53] I'm going to go here and I'm going to [7:54] add all of the class options that I need [7:57] to describe a CFP to my generator. There [8:00] are some little extras in here like the [8:02] aliasing and the banner that [8:04] unfortunately I don't have time to [8:06] describe today, but these options do [8:09] something else that's really really [8:10] cool. Have you ever called bin rails gfp [8:14] help or not cfp, but have you ever [8:16] called help on one of the default [8:18] generators? Honestly, I didn't know this [8:20] existed before this, but there's [8:22] actually documentation for each of the [8:25] generators. And if we look at this [8:26] section right here, our class options [8:29] have generated documentation based on [8:32] that. And every time we update those [8:34] class options, these are also going to [8:36] get updated. And if you map this back to [8:39] the description of class options that I [8:41] had before, you can kind of see that uh [8:44] you can see that laid out here. [8:48] There's something else in here um that [8:50] looks kind of familiar at the bottom. [8:53] This is our usage documentation. It gets [8:56] appended to the end of our help call. [8:59] I've been using usage file to inject [9:01] context because one of the things I find [9:03] is that AI is reluctant to actually call [9:07] documentation and files that I [9:09] reference, but it's pretty good at [9:10] running commands. [snorts] So, let's [9:14] time travel to the present. This is the [9:15] modernday help docu or usage file for uh [9:19] the Ruby event CSP generator. We've got [9:22] a couple of different sections in here. [9:24] I've got this minimal example and then [9:26] I've got an example about how to [9:28] generate an ongoing meetup CFP. So for [9:31] our meetups, we oftentimes don't have an [9:34] end date because they'll just be open [9:35] and ongoing. I've got one for lightning [9:38] talks and then I've got another one [9:40] that's about how to update or extend the [9:42] CFP because again capturing those [9:45] changing states is a bit challenging [9:47] when it comes to AI. [9:49] So, all of this context just gets [9:51] automatically injected when I have it [9:53] call help. Humans [snorts] struggle with [9:56] reading documentations, too. But [9:58] honestly, I didn't even know this [9:59] documentation existed, so I'm not sure [10:01] how much it helps with that. [10:05] All right, we've got our inputs, we've [10:07] got our documentation. It's time to [10:10] generate some files. So, we're going to [10:12] start with real world RailsWorld 2026 [10:15] CFP. I'm going to take this and I'm just [10:17] going to paste it into our TT file. This [10:20] is our template file. TT stands for Thor [10:22] template. And then these THOR templates [10:26] just use regular herb. So you can just [10:29] go in here and replace each of those [10:31] sections with the uh herb tags and then [10:34] the options. [10:36] And uh that's our template. By the way, [10:39] you can customize the templates for the [10:41] default generators as well. There's an [10:44] entire section for it in the Rails guide [10:46] for generators section six, but [10:48] basically you go into lib templates and [10:49] you can replace those default templates. [10:52] Um, [10:53] and most of them use name based so you [10:55] have access to all of the helper methods [10:57] that I showed you before. [10:59] All right, so now it's time to actually [11:01] use the template that we created. So [11:03] we're going to introduce two new methods [11:04] to our generator. We're going to [11:05] introduce CFP file path and create CFP [11:08] file. Create CFP file is pretty [11:10] straightforward. we have template and [11:12] then we pass in our template file and [11:15] then we pass the destination we want it [11:17] to end up in. Now you might be wondering [11:19] how it knows where to find our template [11:22] file from and the answer is in the [11:25] generator that was generated. Uh at the [11:27] very top it added source root and then [11:30] file.expand [11:32] path templates and that's one of the [11:34] things generators are really good at is [11:36] capturing blue code like that that needs [11:37] to be added in. The destination is [11:40] something that I added. So in Ruby [11:42] events, we store all of our CFPs in data [11:45] folder and then event series and then [11:47] event. And then we have this destination [11:50] root which is passed in by Rails and [11:53] defaults to where the generator was [11:55] called. [11:57] Let's do that now. Let's actually run [11:58] the generator that we've just created. [12:01] All right. So we're going to run this [12:03] and it's going to create it and it's [12:05] going to work and it's going to pass in [12:08] like all of the values that we asked for [12:10] and this is the output of that. So we [12:12] have just created a working generator. [12:14] Yay. [12:19] I think next I mentioned that we have [12:21] those ongoing meetup CSPs. I think we [12:23] should try calling this generator [12:25] without that start and end date and [12:26] seeing what happens. Oh, I forgot that [12:30] we already had like a cfp.l created. So, [12:33] we ran into a file conflict, but [12:35] thankfully template has native support [12:37] for file conflicts by default. So, it's [12:40] going to pop this up and I can just say [12:41] yes, overwrite. Oh, okay. So, uh in Ruby [12:47] events, if we pass an open date and a [12:49] close date, we need to have values in [12:51] there and they need to be formatted as a [12:53] date. So, we're going to have to remove [12:55] those fields from the template. All [12:57] right. So, I'm going to go here and I'm [12:59] going to say if options open date and [13:01] I'm going to pass that. And again, this [13:03] is just regular herb, which you're [13:05] already familiar with from your [13:06] templates. And then I'm going to do the [13:08] same thing for options close date. And [13:10] it's going to look like this, which is [13:12] good and passes validation. Um, but it [13:15] has a bunch of weird extra white space [13:18] at the bottom. [13:20] Normally when I'm working with like [13:22] HTML, Erb, all of that kind of gets [13:24] smooshed out by the browser and I don't [13:26] have to see it. Uh, all right. So, [13:29] instead what I'm going to do is I'm [13:30] going to use ERB minus tags in order to [13:32] remove that extra white space that was [13:34] added. Okay, that looks much better. And [13:38] now I only have one extra line at the [13:39] end. So, we trimmed whites space using [13:43] these erb minus tags. Uh, but now we [13:46] have some conditional logic. So, I think [13:48] it's time for us to write a test. [13:51] All right. So, good thing we created [13:53] one. The CSP generator test. I'm just [13:55] going to uncomment the code here and run [13:57] it and see what happens. [14:00] And there we go. Uncomment it and run [14:03] it. And [14:05] green dot. Yay. Uh, it says no value [14:09] provided for required options event. So, [14:13] I think we should probably pass some [14:15] options to this. [14:17] Okay, so when I'm calling run generator, [14:20] I'm going to pass an array that are the [14:22] exact values that I would have passed to [14:24] my CLI. And what I like to do is I like [14:26] to pair them with flag and value just to [14:28] make it easier to read. All right, let's [14:31] run this again. [14:33] [snorts] Green dot, [laughter] [14:35] not bad. Um, maybe we should try like [14:38] asserting the file. So assert file is [14:40] another testing helper that's provided [14:42] by Rails. We're gonna put our file path [14:44] in here and then we're going to take [14:46] content. A really common pattern is just [14:48] checking this against kind of some reax. [14:50] But you can do whatever you want to [14:52] assert content and [14:56] green dot. [14:58] Green dots don't inspire much [14:59] confidence. [15:01] Good news though, if we go to temp [15:03] generators, we can actually see the [15:06] file. It persists after the test run. [15:10] Wait, if it persists, why doesn't it run [15:13] into a file conflict like we saw [15:15] earlier? That's kind of weird. Let's [15:17] take a look at what that test is [15:19] actually doing. All right, so this is [15:20] the default generator test that it [15:22] created. Um, we've got this destination [15:25] here where we put in temp generators. [15:28] That explains why the file was in temp [15:29] generators. Ah, and when I did the cfp [15:32] file path, I passed in destination root. [15:35] So that's how it ends up knowing that [15:37] temp generators goes there. Okay, good. [15:40] And then setup prepare destination. This [15:43] must be what does the cleanup. Oh wow, [15:46] that's so straightforward. We just [15:48] remove the entire destination root [15:50] folder and then remake it. Perfect. [15:53] Wait, hang on. If we're removing the [15:58] temp generators uh directory and then [16:01] recreating that [16:03] like wouldn't this be flaky if we had [16:05] parallel test cases? [16:08] Yeah, yeah, [16:11] yep. Yep. If you just uh uncommented [16:14] that and like rolled forward, then I [16:16] suppose at some point you might end up [16:18] with flaky generator specs. So instead, [16:21] I recommend this. when you're setting [16:24] your classes destination, use a temp [16:26] directory. That way, you don't run into [16:28] the same errors and you don't end up [16:31] running into that same uh issue. And [16:33] then the other thing I recommend is [16:34] cleaning it up in tear down. We've been [16:36] using this in Ruby events for a while [16:38] and we haven't had any issues with it. [16:40] So, just a reminder, when you're working [16:42] with those tests, try and use Ruby's [16:44] temp directories as the destination and [16:46] cleanup in tear down. [16:49] But now I can't see the file, right? And [16:52] so I'm back to my original asserts and [16:53] confirmation. [16:56] I'll just put a binding in here. No big [16:58] deal. And then I'll go check the file [17:00] that way. Ah, perfect. Okay, so we can [17:03] actually see our temp directory. And it [17:06] worked. That's great. Let's go back to [17:08] the console. And where did my green dog [17:11] go? [17:14] Uh, where's my binding be? Uh, hang on a [17:18] second. Uh, let me just do like a [17:20] control C. Maybe a little quit. Panic. [17:22] Okay, quit plus enter got me out of that [17:25] mess. That's weird. That's so weird. But [17:29] I can see the IRB output now that I've [17:32] done quit and enter. It's almost like it [17:34] captured my binding. IRB output. And you [17:38] might be baffled by this if you didn't [17:40] read section 10 of testing Rails [17:42] generators very thoroughly. It turns out [17:45] that you need to pass rails log to [17:46] standard out true in order for your [17:49] debugging tools to work. All right, [17:51] let's try that out now. Oh, perfect. [17:54] Okay, that makes so much more sense and [17:56] everything looks fine and I can see it [17:57] now. Good. Oh, wait. Why don't we do [18:01] that for everything? Uh, well, if I run [18:04] it on the current test file that we have [18:06] in Ruby events, this is what one of our [18:09] tests look like. So if you're not [18:11] capturing that output, it's going to [18:12] look really messy when you're running [18:14] your entire test suite, which is why we [18:16] need that rails log to standard out [18:18] true. [18:20] Okay, so the roundup of the feedback for [18:22] testing generators is make sure to use [18:24] rails log to standard out true to see [18:26] output, replace your destination route [18:29] with a temp directory, and then don't [18:32] prepare destination. [18:34] All right, I have now taught you enough [18:36] about generators to be dangerous. [18:47] There's a lot more to learn about [18:48] generators. There's a ton that I [18:50] couldn't cover. I strongly recommend [18:52] Garrett Demon's blog. Obviously, I [18:54] recommend the Rails guides. They're [18:56] fantastic. But also, the Ruby events [18:59] generators. It's an open source app. You [19:01] can go look at them. Rails is open [19:03] source. You can go look at Rails as [19:05] generators and read them and see what [19:07] their testing patterns are and how they [19:08] handle things. [19:11] Okay, so we have spent an awful lot of [19:13] time on the CLI. So I think it's time [19:15] for us to start talking about the skill [19:16] part of that equation. [19:19] Skills are pretty straightforward. We [19:21] start with front matter about how to use [19:22] it. Then it's just regular markdown. I [19:25] like to include important notes about [19:27] things that the AI commonly gets wrong. [19:29] We are an index in archive for Ruby [19:32] events. So, it is very important that [19:33] our descriptions and our data is [19:36] accurate. [19:38] Next, I recommend adding some to-do list [19:41] items. Those tend to get loaded into the [19:42] workflow, you know, if you're lucky. And [19:45] then here is our actual section on using [19:48] a generator. We suggest that it runs Bin [19:50] Rails GCFP help. That way, it loads all [19:53] of that context and then it actually has [19:56] a sample so it can run that and create [19:58] it. And then we ask it to make [19:59] screenshot. [20:01] All right. So that's the skill part of [20:03] skill plus CLI. [20:05] And now [20:07] we've done it. We built the thing. We [20:11] built the skill and we built the CLI. We [20:14] have handled the or part of this [20:16] equation flawlessly. And really we're [20:19] we're we're we're we're done. [gasps] [20:22] It's fine. I don't need an MCP. I have [20:25] the skill plus CLI. [20:28] I said or MPC. [20:32] What if it were and MCP? [20:36] So at Ruby comp, Andy did this talk [20:39] about abusing schemas using metaroming [20:42] where he talked about how to use an open [20:43] AI schema in order to generate a Ruby [20:46] LLM MCP tool. So I was delighted and [20:50] excited and I wanted to pair with him [20:52] and we did to build MCP tooling for our [20:56] generators. we have these really [20:58] specified class options that describe [21:00] everything that we need for an MCP tool. [21:05] And yeah, so the full story is on the [21:08] Ruby on Rails podcast. If I don't have [21:10] time for class option options, I don't [21:12] have time for this story eventually, but [21:15] [snorts] [21:16] we now have a tool generation script [21:18] that will take a CFP generator or any [21:21] generator and turn it into a Ruby LLM [21:24] MCP tool for our server. It's a CLI and [21:27] an MCT. [21:34] I haven't even got to the best part. [21:36] Anytime we merge generator changes, it's [21:39] reflected in MCP tools when we restart [21:41] the server. [snorts] [21:43] It's so nice. But this also means that I [21:45] can actually test skill plus CLI versus [21:48] MCP, which is what I did. So, I started [21:52] with GBT 5.4 four and I deleted all of [21:54] the improvements that I made to the [21:56] codebase and started rolling them back [21:57] in slowly. The first one I had no skill, [22:01] no generator, no MCP, just trusting the [22:03] AI and I told it to add Rails World 2026 [22:06] and then I gave it a markdown file. I [22:09] wanted to make sure that it was [22:10] consistent and different web fetches [22:12] weren't affecting my results. And then I [22:14] also used bin rails validate all to [22:16] validate because that could have a major [22:18] impact on our results as well. um [22:21] because it pulled in that old context. [22:23] It decided to use old sponsor logos from [22:25] 2023 instead of the 2026 ones. It got [22:28] one password lowercase instead of one [22:30] password table check instead of table [22:32] check. And it added a URL for the Palmer [22:34] event center which tells me that it [22:36] decided to do a web fetch that I didn't [22:38] authorize. [22:40] I'm going to count that as four errors [22:42] and it cost me $183 and 136,000 tokens [22:46] to do that. Now this next one I didn't [22:49] add the skill yet. I just added the [22:51] generator. So this is the status that [22:53] you are in with the Rails generators [22:55] right now. Using custom generators add [22:57] Rails world 2026 and everything else is [23:01] the same. This time it only had two [23:03] errors. It forgot planet scale and uh it [23:05] replaced mdash with hyphens for [23:07] descriptions. So you can tell the big AI [23:09] companies are really listening to us. [23:14] This one was 131,000 tokens and two [23:17] errors. And then this last one, we're [23:20] finishing the scale plus CLI. And this [23:22] time it only had one error, which was [23:26] replacing the M dashes with hyphens. [23:29] Now, I wanted to see how the MCP went. [23:32] Uh, and no videos, no schedule, no [23:36] sponsors. It just kind of quit in the [23:38] middle of the run. So, that was a fun [23:40] thing to discover. [23:42] Uh, however, it was the lowest cost and [23:45] the lowest token usage out of all of the [23:47] options. [23:50] I didn't just run this with a Frontier [23:53] model though. I ran it with its smaller [23:55] sibling too. So, in this one, uh, the [23:58] worst one was no tools. So, it renamed [24:01] the keynotes. None of the talks had [24:03] descriptions. And, uh, for the schedule, [24:06] it decided to use the format we use for [24:09] lunch and breaks for everybody's talks. [24:11] they wouldn't link [24:13] generators but nothing else. Uh, it [24:16] copied involvements including the old MC [24:19] involunteers, it deleted all the to-dos [24:21] and it added the event name parameter [24:23] that I mentioned before. [24:26] And uh, yeah, for the MCP one, um, that [24:31] one was pretty good. It did add an [24:34] involvement, [24:36] which was kind of a problem, but it was [24:37] just the Rails Foundation, so that's [24:39] better. And then it also did our M dash [24:41] for hyphens mistake. The interesting one [24:44] to me, however, [24:46] is 39 cents for zero mistakes [24:51] because I was starting to see kind of a [24:53] pattern here, which was that smaller [24:55] models with better tools outperformed [24:58] larger models for scoped tasks and [25:00] repeatable tasks like this one because [25:03] they didn't go off script. They didn't [25:05] try and rewrite my descriptions. They [25:07] didn't try to look at other files to [25:09] decide if the file that the generator [25:11] had created was appropriate. [25:14] Which brings us to something very [25:15] interesting. The Rails Foundation [25:16] recently published uh a set of model uh [25:21] they did an AI model report and they [25:24] found that Luna, which is right over [25:26] there, had a shockingly good accuracy [25:30] rating given its incredibly low costs. [25:33] So, I wanted to test that one as well. [25:37] Okay, it had a lot of trouble with [25:38] schedules. At the earlier ones, it would [25:41] do wrong times. It had no schedule for [25:44] the MCP option. Um, and for the skill [25:48] plus CLI, the only error that it had was [25:51] that it didn't include day zero in the [25:54] events. Uh, and then the other errors [25:56] were all related to replacing [26:00] uh either descriptions with oneline [26:02] summaries. And for the MCP, it replaced [26:06] M dashes with hyphens. [26:09] So after this, oh, and by the way, all [26:11] of these together are less than a [26:13] quarter. [26:15] So I cut a few conclusions after this, [26:17] which is that skills had fewer mistakes, [26:19] but were more expensive. Um, the cost of [26:22] MCP tooling might be overblown because [26:25] MCP tooling was one of the lowest [26:26] options in all or one of the lowest cost [26:29] options for all of these. Um, and then [26:31] there are a couple of takeaways I'd like [26:33] to leave you with. One is you are now [26:36] dangerous enough to build a custom [26:37] generator. I'd like you to remind your [26:40] agents to use Rails generators to get [26:43] more reliability and to lower your token [26:45] costs. I think you should try using a [26:48] small model with better tools instead of [26:50] leaving things to Opus all the time. I [26:52] think you should try customizing [26:54] existing generator templates for your [26:56] company's standards to see if you can [26:58] get closer to what you're looking for in [27:00] a first pass. And then the last one is [27:03] to make sure that you build for humans, [27:05] too. Humans and AI need help [27:07] understanding our code bases. So, let's [27:09] make sure that as we're building, we [27:10] build tools for both. I want to thank [27:13] you all so much for your time. My name [27:14] is Rachel Wrightman. I go by Chail or [27:16] Rachel. You can find me at Chail Codes [27:18] and all the places. I stream on Twitch [27:19] and YouTube Sundays at 1800 UTC. Thank [27:22] you. [applause] [27:26] [music]