Eddie Galindo & Kagen Hearn: Rate Limiting For Your Customers

Eddie Galindo 与 Kagen Hearn：面向客户的速率限制实践 Eddie Galindo 和 Kagen Hearn 深入解析了 Ascend 财务对账平台在实际生产环境中的扩展挑战。该平台作为单体 Rails 应用运行，已处理超过 10 亿个 Sidekiq 任务。当企业客户的激增给下游老旧 API 带来巨大压力时，团队通过部署采用事务池模式的 PgBouncer，并构建受 Stripe 启发的自定义 Redis 并发速率限制器，化解了吞吐量和数据库瓶颈。他们还详细介绍了 PostgreSQL 与 Elasticsearch 协同工作的双数据库架构、采用明确的数据库列替代非结构化 JSON 以遏制配置泛滥的策略，以及使用 jemalloc 调优 Puma 的方法。在整个讨论中，他们强调了约定优于配置、务实的 gem 维护方式，以及在发布涉及金融交易的软件时严格依赖工程师专业判断的必要性。 Ascend 在五年内持续扩展其 Ruby on Rails 单体架构，仅凭约 20 人的工程团队就处理了超过 13 亿个后台任务和数百万笔保险对账业务。 (0:00) 为防止客户老旧的代理机构管理系统（AMS）API 在高吞吐量数据摄入时崩溃，Ascend 借鉴 Stripe 的架构实现了一个基于 Redis 的并发槽位速率限制器。 (25:39) 针对 Puma 并发量上升和大体量报表导出导致的 Dyno 内存耗尽问题，团队引入 jemalloc 作为 Ruby 内存分配器，并将内存直接生成 CSV 的方式重构为流式处理架构。 (47:11) Ascend 特意避免使用非结构化 JSON 块来存储租户专属配置，而是选择在组织模型中添加显式 PostgreSQL 列，借此增加迁移阻力以防止配置无节制泛滥。 (32:03)

Eddie Galindo and Kagen Hearn explain how financial automation startup Ascend scales a 20-person engineering team and over 1.3 billion background jobs on a single Ruby on Rails monolith. By adhering strictly to standard Rails conventions and deploying focused architectural controls, the team processes enterprise volumes across fragile external integrations without adopting microservices. Maintaining Velocity via a Shared Rails Monolith (13:08) Ascend avoided microservices to keep operational and compliance overhead low while building multi-product insurance financial tooling. Eddie Galindo estimates that 80% to 85% of their daily backend challenges are resolved by standard Rails guides and established gems, allowing engineers to focus on business domain logic rather than infrastructure wiring. Core entities—including policies, clients, and lines of business—are shared across agency billing and direct bill reconciliation modules, centralizing the compliance surface area required for financial audits. Five product teams maintain their respective features while collectively owning the core platform and data models. Kagen Hearn notes that onboarding is smooth because code resides strictly where standard Rails conventions dictate. This architectural predictability enabled engineers hired without prior Ruby experience to ramp up quickly without bespoke framework abstractions. Mitigating Ingestion Bottlenecks and External Rate Limits (25:39) Ascend's direct bill reconciliation engine ingests millions of records from legacy Agency Management Systems (AMS), which often rely on slow, aging XML and SOAP endpoints. When worker concurrency was initially scaled to accelerate data ingestion, Sidekiq background jobs quickly exhausted PostgreSQL database connections. To solve this, the team deployed PgBouncer in transaction pooling mode strictly as a sidecar for ingestion workers, avoiding incompatibilities with session-level locks or prepared statements in other application domains. To prevent high-volume ingestion from overwhelming brittle customer endpoints with 504 timeouts, Ascend built an opt-in Redis concurrency limiter based on a design published by Stripe. Workers acquire a slot by adding their identifier to an organization-specific Redis set before issuing requests, retrying briefly or re-enqueueing the job if limits are exceeded. This mechanism guarantees that jobs remain idempotent and granular, while interactive UI requests bypass the queue to preserve human responsiveness. Enterprise Customization and Technical Debt Governance (32:03) Enterprise customer onboarding brought divergent accounting rules and reconciliation edge cases. Ascend separates transient product rollouts, managed via LaunchDarkly feature flags, from permanent business rules stored as dedicated columns on the database's Organization model. Hearn notes that while this model has expanded into a god object, requiring formal ActiveRecord migrations introduces helpful friction against configuration sprawl, with plans underway to split it into domain-specific configuration models. To prevent customer flexibility from turning into technical debt, engineers participate directly in pre-sales conversations to assess technical feasibility and guide prospects toward operational workarounds instead of bespoke features. When custom business rules are unavoidable, developers utilize Ruby's duck typing to write modular adapter classes conforming to existing interfaces, isolating bespoke customer logic from shared core code. Dual Data Synchronization and Infrastructure Optimization (60:50) Because insurance accounting requires dynamic filtering across tens of thousands of statement line items, Ascend maintains PostgreSQL as its relational authority while powering search and list views with Elasticsearch. To avoid user confusion caused by indexing latency or N+1 queries during asynchronous statement extraction, the team coordinates workflows using the aasm state machine gem and Sidekiq batch callbacks, locking UI views until parsing and policy matching finish. The team is also evaluating ParadeDB as an in-database search extension to eliminate external synchronization pipelines. At the server layer, Ascend optimized Heroku dyno performance by increasing Puma concurrency for I/O-bound requests and adopting jemalloc to curb memory fragmentation. Memory safety also dictated an overhaul of CSV exports: replacing in-memory report generation with batched streaming prevented dyno memory exhaustion when exporting statement datasets containing up to one million records. Contextual AI Adoption in Financial Systems (75:36) Hearn frames the application of AI tools along a risk spectrum guided by the financial consequences of error. Autonomous generation and rapid prototyping tools are permitted for low-stakes internal tooling where software defects carry minimal risk. In contrast, core money movement and accounting pipelines require strict human-led architecture and code review, confining LLMs to boilerplates,Console-driven stress testing, and routine version upgrades. Hearn argues that Rails conventions naturally constrain language models to standard patterns while using fewer context tokens than unopinionated frameworks. However, Galindo insists that AI-assisted output must be reviewed with the same scrutiny as human code, maintaining that engineers remain fully accountable for every committed line. Ascend demonstrates that disciplined adherence to Rails conventions, paired with targeted mitigations like Redis concurrency limits and streaming exports, allows a lean team to operate high-throughput financial infrastructure. However, evaluating alternatives to Heroku and addressing dual-store Elasticsearch synchronization remain ongoing technical challenges.

[1:17] Eddie Galindo: It's been sort of part of my whole career. I stopped using it for a few years and it was nice coming back and like having it be the same. I think I kind of bought into the philosophy on Ruby and Rails and all of that. [1:17] Kagen Hearn: I really appreciate how much Ruby on Rails and Ruby as a language lean into their own design principles rather than trying to hedge against them. It's very much a dynamic object-oriented programming language and it doesn't try to be anything else, and because of that, it lends a lot of expressiveness to the language and very elegant designs. [8:27] Eddie Galindo: It was just really like, hey, what's the fastest way that we can put something out... not having to spend time making a hundred decisions of like, 'Oh, what am I going to use here? What am I going to use over here?' [9:22] Eddie Galindo: I think the interesting pieces are on like figuring out the product side of things rather than like, oh, how do you validations, like how do you send a JSON response back... those things like we've been doing it the same way and we don't have to keep making that decision. [32:03] Kagen Hearn: it's columns on a database model. It's not a JSON. The conventions exist for a reason. They help you find where things are, reason about where things work when you follow them. If you don't, some of the magic, so to speak, can kind of take over and make it a little bit more difficult to reason about what's going on. [60:50] the UI will disagree on the state of the world in two different places. [69:21] Kagen Hearn: They will do exactly the thing that you ask them to do to solve the problem at hand. [69:21] Eddie Galindo: Whatever output comes from AI, like I still consider it the output of the peer that I'm working with. [72:51] Kagen Hearn: the cost of writing code is heading towards zero, but the cost of producing meaningful software is not. [72:51] Kagen Hearn: fully vibe it out and have the AI ship the code and barely review it [84:18] Robby Russell: Again, I'm Robby Russell. Thanks for riding along. See you next time.

[0:00] [music] [0:05] Welcome to On Rails, the podcast where [0:07] we dig into the technical decisions [0:08] behind building and maintaining [0:10] production rub on Rails apps. And I'm [0:11] your host, Robbie Russell, and I run [0:13] Planet Argon. And for over 21 years, [0:14] we've helped teams maintain and evolve [0:16] their long live Rails apps. So, I tend [0:18] to approach these conversations through [0:19] that lens. [music] [0:20] In this episode, we're joined by Eddie [0:22] Galindo, who's an engineering lead at [0:24] Ascend, and Kagan Hearn, a senior [0:26] software engineer at Ascend. Ascend [0:28] builds software that helps insurance [0:30] [music] agencies automate the movement [0:31] and reconciliation of money. Today, [0:33] we're going to explore how their team [0:35] has evolved the Rails app from its [0:36] earliest days into a platform supporting [0:38] multiple products, increasingly large [0:40] customers, and billions of background [0:41] jobs, all while continuing to invest in [0:43] our shared Rails monolith. Eddie and [0:45] Kagan join us from San Francisco, [0:47] California. All right, check for your [0:49] belongings. All aboard. [0:54] Eddie and Kagan, welcome to On Rails. [0:57] Hey Robbie, happy to be here. [0:59] >> Hey Robbie, thanks for having us. Glad [1:01] to be here. [1:01] >> We've had a chance to talk a little bit [1:03] ahead of time, you know, over the last [1:04] couple month or two in prepping for this [1:06] conversation. So I'm really excited to [1:07] kind of dig into things. So before we [1:09] get into some of the topics I wanted to [1:10] bring you on, first question I want to [1:11] ask each of you is what keeps you on [1:14] rails? How about you Eddie first? [1:18] >> Yeah. Um so I think for me it was [1:20] actually the first uh framework I use in [1:22] the last year of college and the I think [1:25] right after uh after graduating. So it's [1:28] been it's been sort of part of my my [1:31] whole uh career. I stopped using it for [1:33] for a few years and it was nice coming [1:35] back and like having it be the the same. [1:38] I think I I kind of bought into the [1:41] philosophy on like Ruby and Rails and [1:43] all of that. So I think it's just been [1:44] part of like how I've been building [1:46] things uh forever. So it's like a lot of [1:48] the principles kind of resonate with [1:50] with me. [1:51] >> What about you, Kagan? [1:53] >> Yeah, for me um it's a more recent [1:56] introduction to my career. I started [1:59] working with it in my last position at [2:01] Midesk, but I really fell in love with [2:03] it. I really appreciate how much Ruby on [2:08] Rails and Ruby as a language lean into [2:11] their own design principles rather than [2:13] trying to hedge against them. It's a [2:15] very much a dynamic object-oriented [2:18] programming language and it doesn't try [2:20] to be anything else and because of that [2:22] it it lends a lot of expressiveness to [2:25] the language and very elegant designs [2:29] >> and out of curiosity [2:31] did what what did you work with prior to [2:34] getting introduced to rubs? [2:36] >> Yeah, I've worked with a bunch of [2:37] things. Um I've done Node.js on the back [2:40] end. I've done Python and Flask on the [2:42] back end. And I had a little bit of a [2:44] phase where I was doing some data [2:45] engineering work which involved uh [2:47] Python, Scala, a bit of SQL. [2:50] >> Oh, interesting. And and Eddie, you [2:52] mentioned you had started off in Rails [2:54] and then you kind of went away for a [2:56] little bit to do some other things. What [2:57] was what were you where did you wander [3:00] off to in the in the tech programming [3:02] language world? [3:04] >> Uh so I joined the the big tech uh [3:06] world. I I used to work at LinkedIn and [3:09] I was primarily just doing front-end uh [3:10] work uh there and then from there uh [3:13] another startup that I joined right [3:14] after it was just a note JS shop. So it [3:18] was all uh uh JavaScript and stuff. [3:20] >> What framework [3:22] front end frameworks were you working [3:23] with over at LinkedIn? Was that like [3:24] Ember there or [3:27] >> it was a lot of stuff. So when I first [3:28] joined they were they were actually [3:30] still in YUI and then from there it was [3:32] like a little bit of jQuery and then we [3:34] migrated to Ember.js JS and then at the [3:37] time I was leaving I think there was [3:38] also like some conversations about like [3:40] going over to React. So it was a little [3:43] bit of everything. [3:44] >> Okay. I think I I knew a few people that [3:46] were kind of wandered over to LinkedIn [3:47] at some point with the with the Ember [3:49] with the Ember crew that I think went [3:51] over there at the time. So [3:52] >> anyhow, so that's a conversation for [3:54] another day. So like uh I got a lot of [3:56] things I want to talk with both of you. [3:57] So but before we getting get into those [3:59] things, I feel like we should probably [4:00] provide a little context for our [4:01] listeners for what is a send? Like what [4:04] is a Maybe uh Eddie you can give [4:06] listeners a quick overview of Ascend and [4:09] some of the problems that you're solving [4:10] for your customers. [4:11] >> Yeah, so um um Ascent is a a company [4:15] that started around five uh years ago [4:17] with the mission of building a financial [4:19] platform for the insurance industry. So [4:21] we specialize on all like money related [4:23] problems specific to to the insurance uh [4:26] world. So we primarily work with like [4:28] insurance agencies. across the country [4:30] trying to solve two money related [4:33] issues. We focus a lot on like on like [4:36] small [clears throat] and mediumsiz [4:37] agencies helping them collect money, [4:39] disperse those funds and then on more on [4:41] the enterprise uh side we kind of [4:43] facilitate workflows for them to operate [4:45] their bank accounts and just money uh [4:47] reconciliation when like commissions [4:49] that they're receiving and stuff. So a [4:50] lot of it the the big focus of the [4:52] application is around money and how can [4:54] we make you more efficient when it comes [4:55] to dealing with those uh type of [4:57] problems. What's Ascend's like business [4:59] model? Is it a like a broker in between [5:01] that or how does Ascend make money with [5:03] with that being part of that equation? [5:06] >> Um, so it's a few uh a few channels. So [5:08] one is we're we're a SAS uh platform. So [5:11] that's uh part of it on our uh on our [5:14] transaction uh product. It's really just [5:17] on hey on on our financing uh book. So [5:19] like we help uh people also to finance [5:22] the policies that they're purchasing. So [5:24] that's part of it. um and and more on [5:26] the money transactions uh side of things [5:28] too. So we we have a few different [5:30] channels [5:31] >> roughly how large of an engineering team [5:34] is there at this point in time. [5:36] >> Uh today we have around 20 which is [5:38] probably the the largest it's uh it's [5:40] been [5:41] >> okay nice and you said there's about [5:42] five years in in the development now as [5:44] a as an application. So so Rail's new [5:47] was five six years ago or so. [5:49] >> Uh yeah it's been uh real since the [5:51] start and we haven't moved away uh [5:53] since. Were were any of the any of you [5:55] there around that period of time or how [5:57] soon after did you did you join? [6:00] >> I've been there since since the start. [6:02] Yeah. [6:02] >> What about you, Kagan? [6:03] >> So, yeah, I'm fairly new. I joined the [6:06] team just a few months ago, right? As [6:09] our SAS side of the product is like [6:11] hitting its hockey stick in terms of uh [6:13] customer growth. [6:14] >> Interesting. So, it's kind of fun. You [6:16] get the two different perspectives for [6:18] our audience here. You got someone [6:19] that's been at the beginning and someone [6:20] that's kind of new. Um, so in a few [6:22] months, how Kagan, maybe if I recall [6:24] from our conversation, you work more on [6:25] the direct bill side of things. So, [6:27] anything you can add to help kind of [6:29] further paint that picture for or for [6:31] our audience here? [6:32] >> Yeah, so the SAS side of the product is [6:34] mostly tailored towards the accountants [6:36] that work at the insurance agencies. So [6:39] for a direct bill, the insurance [6:41] carriers that underwrite the policies [6:43] will collect the premium and then they [6:45] will pass the commission for that policy [6:48] along to the insurance agency [6:50] afterwards. they will receive the [6:53] deposit for the commission [6:56] asynchronously [6:57] um relative to like and then they'll get [6:59] a statement for it either before or [7:01] after they receive the money and they [7:02] need to reconcile where all of the money [7:04] comes from from their deposits versus um [7:07] what's in all of the statements and do [7:09] some matching and the direct bill [7:11] reconciliation team helps with that. Are [7:13] either of you allowed to or are you are [7:15] you allowed to share any terms of like [7:17] scale like how many how much you're [7:19] processing on a regular basis in terms [7:21] of number of transactions or is a little [7:24] off the off the books at the moment for [7:26] us. [7:28] >> I think the money stuff we've probably [7:30] uh uh uh be but I think a numbers of [7:33] segments uh probably can mention a bit [7:36] on that. [7:37] >> Yeah. um we are processing um more [7:42] recently like tens of thousands of [7:44] statements in the past few months and [7:46] some of those statements can have tens [7:48] of thousands of entries on them also [7:50] okay [7:51] >> that need to be reconciled so [7:53] >> I think the number just of um statement [7:57] entries that we've reconciled has more [8:00] than doubled just in the past few months [8:02] over the this period where we're [8:03] onboarding lots of new customers and [8:05] that number is going to continue to grow [8:08] um pretty dramatically many more [8:09] multiples um as those customers ramp up [8:13] because a lot of them are just getting [8:14] started and there are also more in the [8:16] pipeline. So the primary scaling factor [8:18] is number of statement entries that [8:21] we're ingesting or uh that we're [8:23] processing and that is growing by the [8:26] millions. Okay. Well, I'm I know we're [8:30] going to want to dive into some of that [8:31] those those kind of fun scaling [8:33] challenges they might be navigating. So [8:35] I'm curious before we kind of get into [8:36] that since Eddie you were kind of around [8:38] at the beginning do you recall why Rails [8:41] was considered uh for for this [8:44] particular platform? [8:46] >> Yeah. Um so I think I mean a big factor [8:49] of it uh one of the co-founders used to [8:51] work at Instacart. He had Rails [8:53] experience. I was like oh what's the [8:54] easiest way to get started? So he [8:56] bootstrap the app and uh that was kind [8:58] of it. And then from there I think a few [9:01] of the of the attributes from from Rails [9:03] really helped us at the beginning was [9:05] like it's just the speed to like [9:06] actually having a product uh out like it [9:08] was really easy to kind of get the [9:11] initial thing uh going not having to [9:14] spend time making a hundred decisions of [9:15] like oh what am I going to use here what [9:17] am I going to use uh over here. First, [9:19] it was just really like, hey, what's the [9:20] fastest way that we can put something [9:22] out in front of uh customers and make [9:24] sure that we're going down the right uh [9:26] track and focus more on the product side [9:28] of things rather than like making a [9:30] hundred technical decisions that might [9:32] not be relevant if we don't actually [9:33] survive. [9:35] >> Sure. So, so it was a little bit of a [9:37] familiarity thing, you know, or and so [9:40] there must have been probably probably [9:41] was more maybe more than just a [9:42] familiarity thing, but were you [9:44] recruited because you specifically you [9:46] had worked with Rails and they were [9:47] trying to bring in people that had [9:48] worked with Rails before or how was that [9:50] kind of what did that look like? I'm [9:51] just thinking for anyone listening that [9:52] might be thinking of starting a new [9:54] startup venture and they're like, do we [9:56] actually need Ruby on Rails developers [9:58] to be able to build this Ruby on Rails [10:00] application and platform? Um, it's [10:02] something I talk about with a lot of [10:03] potential business owners or budding uh [10:06] business owners, but I'm kind of what [10:08] what's your take on that? Do you think [10:09] that's that's true? Like you need to [10:11] hire people with experience early on or [10:13] do you think you can kind of like let [10:14] people from other skill sets kind of [10:16] come in and pick it up pretty quickly? [10:19] >> I don't think necessarily if if uh if [10:21] anything actually at the at the [10:22] beginning was probably like 30% of the [10:25] team that actually knew uh knew Rails. [10:27] It was more of like hey we we used to [10:29] work together at the previous startup. [10:31] it was more of a people thing of like, [10:33] oh, we like working with each other, so [10:34] we'll continue working out together. Um, [10:37] and then from there, I think Rails for [10:39] the people that uh hadn't used it, it [10:41] was like easy to uh to pick up. Um, I [10:44] think we've we've also tried to not like [10:46] deviate from like your typical Rails [10:48] app. So, we like relied heavily on the [10:50] on the guides and just like stuff that [10:52] was out there, right? I think there's a [10:54] lot of uh content that uh that people [10:57] can look up to kind of familiarize [10:59] themselves with with the framework. [11:01] >> Sure. Kagan, since you joined more [11:03] recently and coming to an established [11:05] Rails application, what was your [11:06] onboarding experience look like? Does it [11:08] did it look and feel like a pretty [11:09] typical Rails application from your [11:11] perspective? [11:12] >> Yeah, it really did. Um I was pretty [11:15] impressed by actually by how well um [11:18] Rails conventions have generally been [11:20] followed. it was pretty easy to onboard. [11:23] Whenever wherever I expected to find [11:25] things, I would generally find them. Um, [11:28] so I do think that we have benefited [11:30] from staying mostly on Rails conventions [11:34] over time, especially if you're somebody [11:35] that's already familiar with Rails and [11:37] you're onboarding. [11:39] >> Okay, that kind of makes a lot of sense. [11:41] I'm I'm curious, uh, Kagan, like how [11:43] much do you feel like the Rails guides [11:46] has provided? you mentioned like it was [11:49] really helpful in you too as Eddie in [11:52] terms of like do you feel like there's [11:54] there's aspects to the Rails guides that [11:57] your team was able to kind of quickly [11:58] pick up on most things or where do you [12:00] how does your team kind of like decide [12:02] like when you're going to do something [12:03] like the Rails way versus like coming up [12:05] with your own innovative solution [12:06] because we know that Rails doesn't [12:07] provide all the solutions necessarily [12:10] that an organization like yours might [12:11] need. [12:12] >> Yeah, I think uh um I can speak on kind [12:15] of more of the things from the from the [12:17] start. So I think we we use the guide as [12:19] a way of like hey somebody probably [12:22] already thought through like oh how do [12:23] you do this? So it's like this is the [12:25] place where we where we start. Um and [12:28] we've actually haven't run too into too [12:30] many situations where it's like oh this [12:33] doesn't exist on on Rails. If anything [12:35] like you can like very easily find a a [12:38] gem. But I would say like maybe 80 85% [12:41] of the work like you can just go to the [12:44] guides and see how something is done [12:45] like in in terms of the the application [12:48] I think the interesting pieces are on [12:50] like figuring out the product side of [12:52] things rather than like oh how do you uh [12:54] validations like how do you send a JSON [12:57] response uh back right like those things [12:59] like we've been doing it the same uh uh [13:01] way and we don't have to keep making [13:03] that that decision. Um so I would say [13:05] the majority of things are kind of [13:07] covered uh there. [13:08] >> Is this platform primarily a monolith? I [13:11] know you work on kind of like separate [13:12] teams so to speak within the same or [13:14] engineering organization. What's that [13:16] kind of structure look like at today? Um [13:18] so we do have a monolith uh application [13:21] for our what we call like like our core [13:23] platform one because the the product is [13:26] meant to be a a platform and you have [13:27] customers kind of using various pieces [13:29] of it that has kind of kept us on that [13:31] monolith route and it has also been like [13:34] way easier to to manage than having uh [13:36] multiple microservices. [13:37] >> I'm curious because you mentioned with [13:39] Kagan coming in is kind of working on a [13:41] different part of the platform. you [13:43] mentioned I guess maybe the SAS part of [13:45] the platform was there conversations [13:47] around splitting that up and having them [13:49] separate or was there enough shared what [13:52] what's the kind of rationale for not [13:54] separating it for anyone listening I'm [13:55] like well there seem like different [13:56] teams why wouldn't you [13:58] >> at the foundational level like some of [14:00] our modeling that we do we do share a [14:02] lot of uh uh concepts um so for example [14:04] we we integrate with external systems [14:06] and we run like some normalization of uh [14:09] data there and actually the the whole [14:12] product benefits from uh from that. [14:14] Aside from that, there's like other core [14:16] components like your your user [14:17] management, your organization management [14:19] and and all of that that are still uh [14:21] sharing. I think a big reason why they [14:23] still make sense to be in the same uh in [14:25] the same place and at the same time I [14:27] think it's also a lot of the [14:28] infrastructure uh side of things like [14:30] it's been way easier to to manage and I [14:32] think for a small team that's like [14:34] pretty uh important especially in the [14:36] world that that we're in and the the [14:38] money side of things where how we have [14:39] to be compliant and all of that. there's [14:41] not a 100 applications that we have to [14:43] keep monitoring uh closely. So that that [14:45] part is also uh important. It's just [14:47] like it's easier to to do and I think [14:49] we'll try to push it as far as we as we [14:51] can. Like once it gets hard to manage, [14:53] we'll probably think think about it [14:54] again. [14:55] >> Sure. I'm sure I'm sure it'll come up [14:56] again, but how does your team kind of [14:58] establish boundaries at this phase of [15:00] it? because you're still kind of 20 [15:02] people and maybe I don't know if you're [15:03] going to be hiring more people in the [15:04] near future or not but like how do you [15:06] kind of think about boundaries of like [15:09] which which part of those two teams is [15:11] responsible for what or is it kind of a [15:13] little bit implicit? [15:14] >> Yeah. So um at the at the product level [15:17] like each uh team kind of owns their uh [15:20] their their product like um and they're [15:22] they're responsible for for that more of [15:25] the like the platform side of things and [15:27] like things that are share across is [15:29] really more of a team uh a team effort [15:31] or like hey we all kind of own it uh uh [15:34] together and are responsible for keeping [15:36] that uh uh clean. I think that [15:38] eventually as we add more people it [15:39] might uh break but I think for now it [15:41] has worked well for for us. What about [15:44] for you Kagan? As someone that's been [15:45] primarily working within the direct [15:47] build side of things, does the shared [15:49] monolith feel like a superpower or a [15:51] constraint at this point? [snorts] [15:53] >> Um, it I think it provides a lot more [15:55] benefits right now than it does [15:57] constraints. In my opinion, a lot of the [15:59] benefits of microservices [16:01] is organizational. It makes it easier to [16:04] deploy changes um to a specific team's [16:07] domain um independently without [16:10] interfering with anything that other [16:12] teams depend on, right? like you can [16:13] isolate dependencies um and deploy [16:16] independently for each team's domain. [16:19] But I think right now we're still at the [16:21] point where we share a lot of the [16:22] models, right? Like a lot of things in [16:24] the insurance industry are common [16:26] regardless of whether it's agency bill [16:28] or direct bill or whatever other [16:30] accounting task that you have going on. [16:32] Um such as the policies, any lines of [16:35] business, clients, etc. and all parts of [16:38] the shop share those models um and [16:41] continue to benefit from being able to [16:43] add to them in one place rather than [16:45] being able or needing to update them [16:47] everywhere anytime you want to extend [16:49] functionality. Will the direct build [16:52] team be growing more as this you [16:53] mentioned this hockey stip growth type [16:55] of thing and like is that part of like [16:56] the plans right now or are you trying to [16:58] stay kind of in a realm of and what and [17:00] how's the team divvied up? Is it like [17:02] 50/50 approximately at this point or is [17:05] it just a few people working on one side [17:06] of it? [17:07] >> So, we we we're actually like split up [17:09] into like five uh uh different uh teams [17:13] uh today. Okay. For like five uh [17:16] >> kind of products that we have in the in [17:17] the market and the teams are pretty like [17:20] I would say like evenly uh uh split. [17:22] There might be some that have uh less [17:24] people uh right now just because they're [17:25] newer products, but they're pretty like [17:27] evenly split uh today. Yeah, I think in [17:30] terms of uh like we're looking forward [17:32] to like continue to growing each each [17:34] one because we usually like start up [17:36] most of these products are just like [17:37] tackling the service and I think as you [17:40] like go on and like discover new things [17:42] like there's always uh areas of more [17:44] that you can go into. [17:46] >> Sure. That makes a lot of sense. So [17:48] Kagan, you know, in our prior [17:49] conversation, you had mentioned that, [17:51] you know, you'd already talked about how [17:53] you've been onboarding new customers on [17:55] the direct pay side of things. So uh [17:58] maybe some of them were dramatically [17:59] larger customers and so what sort of [18:02] impact did that have on the platform [18:04] that your team had started building [18:05] already and how much larger relative to [18:08] prior customers are we talking? [18:11] Yeah. Um so on the direct bill side of [18:16] the house up until recently we had um [18:20] two relatively large organizations [18:23] um using the product and the product was [18:27] like fairly hastily put together. Um I [18:30] think Eddie uh what was it about a year [18:33] ago or so? One or two years. Um [18:35] >> are you blaming Eddie? [18:37] >> Yes. No. Um of course not. Um, [18:40] >> don't be so hasty, Eddie. [18:43] >> But, [clears throat] [18:44] um, there's this interesting dynamic [18:47] going on where we're trying to ship at [18:49] startup speeds for enterprise large [18:52] enterprise customers. Um, and as we've [18:56] started to onboard lots of customers, [18:58] some of the prior assumptions that were [19:01] made um, started to break down under the [19:05] pressure um, especially ingestion [19:07] scaling. So um in order to service [19:11] direct bill customers we need to ingest [19:13] their policies, their lines of business [19:16] um any of their clients etc from the um [19:20] agency management system or AMS that [19:22] they work with. At the time that we [19:24] started getting this um deluge of new [19:27] customers, we were able to process maybe [19:31] like 500,000 to a million new policies [19:34] in a day and the limit was just latency. [19:37] It's a lot of external requests that we [19:39] have to make. They take a lot of time [19:41] and each one uh holds like a a sidekick [19:46] thread. We use sidekick for asynchronous [19:47] processing. And each one holds a [19:49] sidekick thread the entire time that [19:50] it's waiting on the request. And so [19:52] there's just a strict limit to with how [19:54] many sidekick workers we had, how much [19:56] volume we could process in a given day. [19:58] Um and so we needed to start driving [20:01] that up. [20:03] Out of curiosity, are the you say what's [20:04] an AMS for anyone out there listening [20:06] that doesn't know what that is? [20:07] >> Yeah, it's an agency management system. [20:09] >> Okay. Um, and are these more modern [20:13] platforms that you're integrating with [20:14] or are these older? [20:16] >> Most of them have been around for a [20:17] while. [20:18] >> Okay. We're talking as old as like XML, [20:21] SOAP services. [20:23] >> Yes. [20:23] >> Okay. So, [20:24] >> but SOAP services, XML, all that. [20:26] >> Okay. [20:28] Um, any CSV transfers over FTP or [20:31] anything like that or [20:33] I I don't think we're ingesting anything [20:35] that way. Okay. Then you're not going [20:38] too far back in the in the the eras of [20:41] data transfer between different [20:42] platforms. But uh so yeah, so you got [20:44] all these background jobs using Sidekick [20:47] and they're happening asynchronously. [20:50] Well, can't you just add more workers? [20:52] Like what's what's the what's the [20:54] solution there? Like I mean when you say [20:55] you couldn't scale up more, is it just [20:56] that there could only handle so many in [20:58] a day? [20:59] >> Yep. So when we first started facing [21:01] this problem um we weren't sure actually [21:04] what the scaling boundaries were because [21:06] we had never tried to ingest more as we [21:08] had had never really needed to. And so [21:11] the first thing we needed to do was try [21:13] to understand what the actual scaling [21:15] limiters were. So during off business [21:18] hours we would just kind of turn the [21:20] knob up on the number of workers to see [21:21] what would happen. And the first [21:23] limitation that we noticed was just the [21:26] number of database connections [21:28] um that those would check out. Sidekick [21:31] will take a database connection for each [21:34] thread the first time that it does any [21:36] database work in that thread. And all of [21:38] these workers are doing database work. [21:40] And so we would start pushing pretty [21:42] close to Heroku Postgres's connection [21:46] limit pretty quickly when we started [21:48] trying to up the number of sidekick [21:50] workers. Are you still hosting this on [21:53] Heroku [21:54] Post right now? [21:57] >> And did you come up with any interesting [21:59] solutions to that or is it something [22:00] that you just had to like figure out how [22:02] to throttle at this point in time? [22:04] >> Yes. Um there's the standard solution to [22:09] connection limits is just uh transaction [22:11] pooling. So we brought in PG balancer [22:14] for client side as a sidecar for the [22:17] sidekick workers specifically that are [22:19] doing ingestion. We didn't apply it to [22:21] all of our workers because we weren't in [22:23] order to implement transaction pooling. [22:25] There are some Postgress features that [22:27] you actually can't use and it can be a [22:30] bit of a heavy lift to actually like [22:32] comb through the application and make [22:34] sure that none of those places are [22:35] currently relying on it. So we [22:37] introduced it in a limited capacity for [22:40] ingestion workers which really helped us [22:42] out when you're processing all this and [22:45] like say I'm trying to just walk through [22:47] the workflows. Let's say someone you're [22:49] ingesting maybe for a customer and going [22:52] through their their things. Are you able [22:55] to then run another one for another [22:58] customer in parallel and or is it then [23:00] have to wait for the other one? Like [23:01] what's that? Have you come up with some [23:04] what how is that currently looking or [23:05] how did it originally look and how have [23:07] you kind of been evolving and and [23:08] morphing that? Yeah, the way that it [23:11] looks is um we have a cron job that runs [23:15] every 10 minutes that will pull each of [23:19] our customers APIs to for any change [23:21] events. So this will tell us which [23:24] policies have changed and the last x [23:28] time the window is configurable how far [23:30] you want to look and for any of those [23:33] policies that we find that have change [23:34] events we'll create a record in our [23:36] postgress database um to represent that [23:39] change event and then we have another [23:41] scheduled sidekick worker that runs on a [23:43] cron that will pick up change events and [23:45] then push them into the respective [23:47] processing cues. So like uh the policy [23:51] sync workers for example um this is the [23:53] primary one will be incued by this [23:55] worker that picks up change events and [23:58] that allows us to limit the number of [23:59] change events that we process at at a [24:01] given time. That's one lever to limit [24:03] concurrency is how much this worker how [24:05] many change events this worker will pick [24:06] up at a time. So we can stack up a [24:08] backlog of change events and then [24:10] process them as quickly as we're able to [24:12] or as we need to. And then the policy [24:16] sync workers will pick up those change [24:18] events. Those have some of their own [24:20] side effects. Those will sync uh [24:22] transactions that are related to those [24:24] policies for example as a secondary side [24:26] effect. But all of the customers can run [24:29] on the same shared resources at the same [24:32] time. [24:32] >> When it comes to like that workflow, out [24:34] of curiosity, if you checking every [24:37] polling every 10 minutes, what if the [24:39] policy changes 12 minutes later? Are you [24:42] having to cycle through? Does it is [24:44] there anything about those policies that [24:46] you're having to then go back and check [24:48] things as you're actually processing [24:50] that policy again to like double check [24:51] the data or is it kind of do you pull [24:53] and just know that and then you fetch it [24:54] and you handle that transaction then and [24:56] then it it's so it's safe enough that it [25:00] can keep running 12 minutes later and [25:02] it's not going to be an issue. [25:05] >> It's yeah it's completely fine for it to [25:06] run 12 minutes later. We the window that [25:09] we look back is not 10 minutes. we look [25:11] back further than that to give us some [25:13] buffer room to make sure that we're not [25:14] missing policies or missing any change [25:16] events. [25:17] >> Sure. Sure. [25:17] >> We have a daily sync that will look back [25:20] over the past day. And then over the [25:22] course of the day, we also every 10 [25:24] minutes will look back at policy change [25:27] events from I'm not sure what the latest [25:29] is, the past hour. [25:30] >> I see. So, so you have you kind of have [25:32] a a little bit of a fail backup [25:35] situation happening just in case to [25:36] catch things. Hopefully catch things. [25:38] Yes. [25:39] And how has that you running all this [25:43] you know fetching all this stuff from [25:44] these different systems has that all [25:46] been super smooth in terms of like those [25:48] platforms have been able to handle the [25:50] load that you're requesting from them? [25:53] >> Yeah that's a great leading question. Um [25:56] the [25:57] >> thank you. [25:57] >> What what we have observed after we [26:00] solved our transaction our database [26:03] connection problem and with transaction [26:06] pooling the next limiter that we [26:08] observed is that our users own APIs are [26:13] only able to handle so much ingestion [26:15] traffic that we send at them. Um, if we [26:18] send too many concurrent requests at a [26:19] time for policies, we'll start getting [26:22] like 504s, server failures. In the worst [26:25] case, we might even cause them [26:26] disruption on their own service if we're [26:28] not careful. [26:29] >> How have you handled that? You just a [26:31] lot of retries or [26:33] >> Yeah. Well, we don't want to bring the [26:34] customer down, so we can't just retry [26:36] it. Sure. Um, so we implemented a [26:39] concurrency rate limiter and this is [26:41] inspired by a Stripe blog post on rate [26:43] limiting from 2017 inspired our [26:46] implementation specifically one on [26:50] concurrency rate limiting. They have a [26:51] really nice GitHub gist that gives you a [26:54] sample implementation. Um and we use [26:57] that and the way that works is [27:02] it puts a set in Reddus for each [27:04] individual [27:05] uh integration that has outgoing [27:07] requests and each time that we want to [27:10] make an outbound integration request say [27:12] to fetch a policy um that worker has to [27:16] try to obtain a slot and obtaining a [27:19] slot is just adding an item to that set. [27:22] Each organization has some configuration [27:25] that tells how much concurrency they're [27:27] allowed to they're able to withstand. So [27:29] how much we can send them. Um [27:32] and if the number of items in that set [27:35] is equal to or greater than the limit [27:37] they have configured then the worker is [27:40] unable to acquire a slot. And so it'll [27:43] keep trying once every like 100 [27:45] milliseconds or so. And then eventually [27:47] it'll time out and it'll just reincue [27:49] the job. [27:50] >> Interesting. What's what's your what's [27:52] what's been the kind of behind the [27:53] scenes? What's the the high level [27:57] how does that what does that look like [27:58] from uh within your Ruby code? [28:01] >> Yep. Um so the [28:04] one thing this forces us to do is make [28:06] sure that all of the jobs that we inq [28:08] are item potent because a job could fail [28:10] to acquire a slot at any time in its [28:13] processing. In any rate limited task we [28:15] try to just put one request in the job [28:18] make sure it only has one side effect. [28:20] These are general sidekick principles, [28:21] but they're not always super easy to [28:23] follow. And so at the beginning of a [28:27] policy sync, for example, we will [28:29] attempt to acquire a slot. If we're [28:32] unable to acquire a slot, then we'll [28:33] just reincue the job at the back of the [28:35] queue. This helps make sure that other [28:37] organizations are able to continue [28:38] processing volume even if one is [28:41] currently being rate limited. And then [28:44] that will encue some side effect workers [28:46] of its own like syncing transactions for [28:48] example that is also rate limited. And [28:50] because all of this is implemented using [28:52] a Ruby class with a context block it's [28:55] optin. So requests that are executed [28:58] from like production user paths for [29:01] example are not subject to the rate [29:03] limiting. If we're doing something where [29:05] the user is clicking buttons and that [29:06] needs to make external API requests [29:08] that's not subject to rate limiting but [29:10] ingestion requests are subject to rate [29:12] limiting. [29:14] And do do you reserve a certain [29:16] allocation of slots then for userdriven [29:19] things then? [29:21] >> So user driven things actually don't [29:23] aren't subject to the slots at all [29:25] because they tend not to push so much [29:27] volume that they put the customer's API [29:29] under stress. Okay. [29:30] >> As much as the ingestion ones do. Um so [29:33] the slots typically are only used for [29:35] ingestion. [29:36] >> Okay. Interesting. And Eddie, when [29:38] you're thinking about designing systems [29:39] like that, how do you think about [29:40] protecting [29:42] does this do you have are you using [29:43] these types of functionality in other [29:46] parts of the application as well? [29:48] >> Uh no, I think uh today probably that [29:50] this uh this product is the one that's [29:52] pushing the the boundaries on our [29:53] application in terms of uh scalability [29:56] and uh and performance. So kind of [29:58] connected back to one of your initial [29:59] questions, it is probably one of the of [30:01] the cons on the monolith of like hey the [30:03] different products might have different [30:05] uh requirements. we're just living with [30:06] the fact of like hey we need to pay [30:08] attention to info on some side of the of [30:11] the system like we'll have to do it for [30:13] everything but uh but yeah I think this [30:15] is probably the one that has been [30:16] pushing at the uh the most and I think [30:20] also the another important aspect is [30:22] like it's probably things that we didn't [30:24] consider at the at the beginning like I [30:26] think everything that can describe is [30:28] like hey we didn't start this way we [30:31] just kind of slowly uh got in there like [30:33] >> we take it back to the basics I think [30:35] what he desri I've started really as [30:36] like, hey, it's just a worker that's [30:38] just going to make an API call ingest [30:40] stuff and we just kept running and [30:41] running until we like kind of hit the [30:43] limit. So, [30:44] >> have you been able to work out with with [30:46] your different with your customers and [30:48] figure out what these uh do they do they [30:51] do they make it very obvious what the [30:52] the limits are of how often you can be [30:54] hitting their their app platforms or [30:56] these just trial and error and you're [30:58] like well we seem to be able to do this [30:59] many and then it starts to we start [31:00] getting a bunch of errors so we'll we'll [31:02] just kind of we'll pencil in this number [31:04] right now until we see there's an issue [31:06] again. [31:07] >> Yep. Most customers limits start at a [31:10] fairly safe and conservative value. Um [31:13] and then if we feel like we need to push [31:15] more ingestion volume than what that [31:17] conservative value allows, we will [31:20] assess based on the size of the customer [31:23] um whether they have like onrem [31:26] infrastructure or they're using [31:28] something that's uh shared for example [31:30] to try to estimate uh what we can [31:33] actually push through. And there's a few [31:35] customers that are only able to take [31:37] like a very limited amount of volume as [31:40] well. And so we actually have to scale [31:42] them down. [31:43] >> Interesting. And it's not been as easy [31:45] for you to convince them. Can you just [31:47] increase your your side of the equation [31:49] and add some more hardware over there? [31:50] >> Yeah. Just spend more money. [31:52] >> Yeah. Uh [31:54] you're you're you're trying to provide [31:56] them a service, right? So you're having [31:57] to do all this stuff to work around [31:59] their limitations and uh anyway, like [32:03] save maybe save that for another day. I [32:05] know that uh I think I don't know I I [32:07] can't remember which one of you were [32:08] mentioning this. might have been you [32:09] Kagan but I think you mentioned that in [32:12] the platform uh there's a lot of feature [32:14] flag and maybe configuration complexity [32:18] in the platform already so is that is [32:20] that is that true [32:22] >> that's definitely true that has become a [32:24] lot more true over the past few months [32:26] as we've been onboarding all of these [32:28] large customers this is I think natural [32:32] to like enterprise B2B contexts each [32:35] customer wants their own version of the [32:38] product. There's different things that [32:39] they need. They all have their own [32:41] accounting workflows and ways that they [32:43] think about reconciliation, different [32:46] processes. Um, some of them have [32:49] acquired other agencies and that affects [32:51] their processes as well. And so the set [32:54] of features that we expose to each [32:57] customer and the configuration that we [33:00] allow them is is different depending on [33:02] uh which one it is. So there's a lot of [33:05] there's a lot of conditionals in your [33:07] code to do more than maybe just swap out [33:09] a logo. [33:10] >> We we try to avoid continually adding [33:13] them, but there are necessarily a lot of [33:16] conditionals. Yes. [33:18] >> How do you see that scaling? like you're [33:20] kind of if you're months into this [33:22] process right now and I've I've talked [33:24] to a lot of companies that will be years [33:26] into the process and they find [33:27] themselves with this tangled mess of [33:29] being like it's really challenging to [33:31] debug things in different environments [33:33] like and maybe part of that was because [33:35] back in the back when they started they [33:36] maybe they didn't get to leverage a lot [33:38] of things like feature flags or things [33:40] like that. So, are you using much [33:42] feature flag type approaches to this? [33:44] And is this just some big config files [33:46] or how does your team kind of think [33:48] about this? Thinking about the long [33:50] term, but also trying to you're [33:51] obviously trying to solve problems today [33:53] for the customers you're bringing in and [33:54] you're still learning. I would imagine [33:56] you probably don't have a clear vision [33:58] on how this would look like in two years [33:59] at this point. [34:01] That's right. We have two primary [34:03] levers, two mechanisms that we can use. [34:07] um we use feature flags pretty [34:08] extensively and the idea for those is [34:12] gating actual product functionality. So [34:16] what views are available to who um who [34:18] sees which buttons in the UI stuff like [34:21] that. And then we also have organization [34:23] configurations which live in our [34:26] database and this is to tweak [34:29] functionality for products generally [34:32] that everybody is on. Um, so this is [34:35] things like for direct bill [34:37] reconciliation for example, we need to [34:40] match the entries on a statement to the [34:42] policy that the entry is for. And every [34:45] organization has different rules for [34:47] exactly how they want to match those [34:49] entries. And or configurations allow us [34:52] to provide a way for customers to to say [34:55] we want to match it this way in this [34:58] context and this way in that context and [35:00] allow everybody to uh have it the way [35:02] they need it. When you say like you have [35:04] things in your database, is just is this [35:06] like one potentially growing god object [35:10] at some point or or [35:12] >> Yes, the organization configuration. [35:14] Exactly. It is currently a large growing [35:16] god object. Um [35:18] >> are these a bunch of like attributes? Do [35:20] you like actual columns or is this like [35:22] a JSON like what's that look like behind [35:23] the scenes right now? If you don't mind [35:24] sharing behind the the the little bit of [35:27] the the spiderweb. [35:30] >> Yeah. Um it's it's columns on a database [35:32] model. It's not a JSON. Okay, [35:34] thankfully. [35:35] >> Um, [35:36] >> it's so anytime you want one of the [35:39] things that adds a bit of friction and [35:41] makes it harder to continuously [35:42] introduce new organization [35:44] configurations, discourages it a little [35:46] bit is that you need a database [35:48] migration for each organization [35:50] configuration that you want to add. And [35:52] so each one is a database column on a [35:54] model that belongs to the organization. [35:55] >> I see. Is this a multi-tenant [35:58] application? And do you keep the datab [36:00] data very separate or is if you're [36:03] allowed to kind of share that? [36:05] >> So organizations uh like their data is [36:08] split up but we still have a single uh a [36:10] single uh uh database. Uh okay one more [36:13] thing to add on the configuration. I [36:14] think we we are thinking of like [36:17] actually start splitting it up more on [36:19] kind of domain that it's uh configuring [36:22] like oh we have I don't know payment [36:24] configuration or we have uh our [36:26] reconciliation uh uh configuration or [36:28] for other products so to kind of ease uh [36:31] uh the modeling uh there and not having [36:33] like one huge like go object for for [36:36] configuration and I think the decision [36:38] for us on like hey actually putting in [36:39] the database because those ones are [36:41] meant to actually be part of the product [36:43] and like how they work and like they're [36:45] going to stay there for forever if uh if [36:48] that's the case. Um too [36:50] >> that's interesting. How do you decide [36:51] when to use a feature flag? Maybe maybe [36:54] putting something configuration and what [36:55] are you using for feature flags that [36:57] across [36:58] >> uh for feature flags we're using launch [36:59] darkly today. [37:00] >> Okay. Um um so yeah, I think it's kind [37:03] of that principle like hey whatever is [37:05] meant to be uh shortlived and like hey [37:08] as we're working more enterprise [37:10] customers some things uh might require [37:12] like approval from from them or slower [37:15] rollout right but eventually the goal is [37:17] like hey everybody would get this uh [37:19] feature and clean up the feature flag [37:21] and the branching in the code is gone. [37:24] Uh versus in the configuration model is [37:26] like hey those are actually going to [37:27] change your business logic. Um, and one [37:30] example is like what uh u were [37:32] describing earlier like oh you might [37:34] want like different granularity on like [37:36] when you're reconciling something and [37:37] stuff like that and that's just part of [37:39] the like actual behavior in your [37:41] product. [37:42] >> This episode of on rails is brought to [37:44] you by unsafe save. [music] The fastest [37:46] way to skip validations and write [37:48] questionable data with confidence. Are [37:50] you tired of pesky [music] validations [37:51] getting in the way of your momentum? [37:53] Just add validate false and boom your [37:55] record is saved. Empty fields no [37:57] problem. Missing associations? Who [37:59] cares? But wait, there's more. Act now [38:02] and we'll throw an update column totally [38:04] free. Skip validations and callbacks. [38:06] That's right. Bypass your business logic [38:08] entirely. Update your database like no [38:10] one's watching. Why wait for your code [38:12] to behave when you can just skip the [38:13] parts you don't care about. Unsafe save. [38:15] Because sometimes you just need that [38:17] record in the database consequences [38:19] pending. [38:22] When you think about how the [38:23] application's been evolving over the [38:25] last few years, how do you decide, and [38:28] this may be more directed for you, [38:29] Eddie, when that customer flexibility [38:31] can start to become, for lack of a [38:33] better term, technical debt of some [38:35] sort? [38:37] >> Um, it's a good question. I think it's [38:39] probably something that everyone uh uh [38:42] struggles with. I think we we do try to [38:46] push back and kind of and think about [38:48] the product in generic terms, right? [38:50] like obviously you don't want to be [38:51] building like one-off uh uh things. So I [38:54] think it's more of a putting yourself in [38:57] their shoes and then also like looking [38:59] at it from our perspective like hey this [39:00] is this something that makes sense in [39:02] the product uh long term and it's [39:04] something that it's part of our process [39:05] that you can standardize or do you [39:07] actually work with the customer to be [39:08] like hey like this is maybe another [39:10] process you can follow so we don't need [39:12] to like add this uh feature and stuff. [39:14] So I think it's it's kind of like both [39:16] ways and like having empathy for like [39:18] where they're coming from and like why [39:19] they're asking for for something. So [39:21] with that like I mean I think we don't [39:22] have all the answers. So sometimes they [39:24] might ask for a feature and be like oh [39:25] this actually makes uh makes sense like [39:27] we we need to uh we need to add it. So I [39:29] think it's probably not the best answer [39:31] that everybody would want to hear but I [39:32] think it's really on a case by case [39:33] basis right like you just have to sit [39:34] there and like think through the through [39:36] the requests and uh think about what [39:39] makes sense for all of your customers [39:40] and then where's your customer coming [39:42] from. No, that's true. I I always think [39:45] about how there's that balance of like [39:47] you're trying and I don't know if you're [39:49] like a it's like a salesdriven [39:52] conversation like hey we we can land [39:54] this new customer and they're going to [39:55] break you know like yeah we we're 90% of [39:57] what they think they need but there's [39:59] these few new things so we're going to [40:01] we've committed to building these things [40:02] like well how is that going to impact [40:04] the rest of our customers we already [40:05] have and will they need it or not and [40:07] how do you roll that out and like every [40:08] one of those I'm assuming becomes its [40:10] own uh interesting conver set of [40:12] conversations and things to think about, [40:14] but do you have like a pretty smooth [40:17] workflow for like is it do you come in [40:19] and like all right, we're going to build [40:20] out a new is that is that first of all [40:21] is that assumption true that you need to [40:24] do that kind of might be help bring in a [40:26] new customer and you're like we have to [40:27] kind of change this a little bit for [40:29] this customer. Is that is that true? [40:31] >> Yeah. So um um I think fortunately for [40:34] for u uh most of it is actually like a [40:37] conversation with uh in some case with [40:39] engineering and and product like [40:41] actually jumping in with the customer [40:42] and try to understand what they're uh [40:44] what they're asking uh for. So we're [40:46] able to kind of work through the [40:47] features uh a little bit more and not [40:49] just essentially be like oh [40:51] >> we we promise [40:52] >> we're trying to sell this so you have to [40:53] you have to do it kind of thing. [40:54] >> Yeah. So, so you're part of the you get [40:57] to the the engineers get to be part of [40:59] those conversations before that even [41:01] like typically before the ink even gets [41:03] signed and you're like this is what [41:04] we're we can do and we'll make some [41:06] adjustments to make this work for you or [41:08] if if necessary or like oh we don't need [41:10] that right now but this is coming down [41:11] the road this is on the road map for the [41:13] next 12 months and we'll get there with [41:15] >> uh yeah yeah so a combination of uh uh [41:17] both so we um we do try as as much as we [41:21] can to like actually have engineers in [41:23] the in the calls and so so that you're [41:26] close to the customer and you can [41:27] actually understand what they're asking [41:29] for. [41:30] >> Out of curiosity for for you know just [41:32] because I think a a lot of people that [41:34] work in certain types of SAS businesses [41:36] that don't work with like enterprise [41:38] customers where you may maybe you're [41:39] selling $30, $50 $100 a month [41:42] subscription service. the there's like [41:44] maybe some tiers. Presumably, you're not [41:46] doing a lot of high-end customization [41:48] for specific customers because the idea [41:50] is you're trying to get bulk customers, [41:53] not necessarily larger type clients. Did [41:55] either of you work at SAS platforms [41:56] before working at [41:59] >> I have it? [41:59] >> Yep. Most most of my experiences at [42:02] different kinds of SAS platforms. Yes. [42:04] >> And was it very was it similar where [42:06] there was a lot of this kind of uh per [42:08] customer customization in those other [42:10] environments as well? Yes, in particular [42:12] for the largest contracts, we would be [42:14] especially willing to to do uh more [42:17] custom things. Although I think [42:19] something that holds for those companies [42:21] as much as it does here is that [42:23] generally before we go about agreeing to [42:26] do something custom um we try to think [42:29] about if that's going to generalize well [42:33] to use cases for anybody else in the [42:35] future. um whether this makes sense as [42:38] part of the product in general. And if [42:40] it doesn't, we do generally try to push [42:42] back on like super bespoke custom [42:45] features that won't ever apply to anyone [42:47] else. That there are exceptions. [42:49] >> Sure. Sure. Um money's money is an [42:51] attractive thing to to businesses at [42:53] times. [laughter] [42:55] Depends if if they'll pay for it, right? [42:56] Uh or pay pay for some pay towards that, [42:59] I suppose, in there. I can appreciate [43:01] that. But yeah, I was just always [43:02] curious like how that what that actually [43:03] looks like in different in different [43:05] environments. [43:06] How how do you think Rails make [43:08] simplifies that? Do you feel like this [43:10] is more of a is do you feel like Rails [43:12] is giving you enough tools to help work [43:15] on these types of platforms like that [43:16] where you are customizing it in that [43:18] sort of fashion or do you think Rails [43:21] should think about that a little bit [43:22] more from a the framework itself to have [43:24] some more baked in tooling of that sort? [43:28] Um I think [43:31] maybe contrary to the expected um [43:36] criticism that Rails's convention over [43:38] configuration makes it more difficult to [43:40] do something custom. I think that's more [43:42] true on the tech side than it is on the [43:44] product side. And because of the ability [43:47] to spin up something new fairly quickly [43:49] relying on those conventions, it [43:51] actually makes it a little bit easier to [43:53] build out something bespoke pretty [43:55] quickly as well as and the duct typing [43:59] also goes a long way in that respect [44:01] too. [44:01] >> Can you give me an example of where duct [44:03] typing might come into play in that in [44:05] that environment for our listeners that [44:06] might not be that familiar with the [44:07] term? Yeah, the duct typing helps with [44:11] taking something that we have already [44:14] built out, for example, that's shared [44:16] across a lot of customers and then we [44:18] want to do something weird with it that [44:22] is maybe oddly specific to a particular [44:25] customer, but we can rely on a lot of [44:27] the logic that's already been written [44:29] because with duct typing, what duct [44:32] typing essentially says is that the type [44:34] of the object is its interface. It's [44:36] what what methods does it respond to [44:38] rather than a statically declared type [44:41] information. And so if we're able to [44:44] write some kind of adapter or interface [44:46] to our already existing code that [44:49] responds correctly to the pre-existing [44:51] interface, we can leverage a lot of the [44:55] work that's already been done on past [44:57] projects. [44:58] >> And is that like as much as just like [45:00] overwriting a method to do something [45:01] that and you can kind of rely on [45:04] everything else that's already there? [45:05] Yeah. Um overwriting a method or writing [45:08] a class um that matches the interface of [45:11] some other module that we already have [45:14] that uh does most of the work that we're [45:15] looking for and expects that interface. [45:17] For example, [45:19] um [45:21] it it really helps um with it with hacky [45:25] things like that in particular. It kind [45:26] of sounds like you have a positive [45:28] favor. I've heard some people say like [45:29] this is a uh can feel a little too [45:32] magical or a little weird to kind of [45:34] diagnose and if for people coming in and [45:36] they don't understand how these things [45:38] work. [45:38] >> Yep. [45:39] >> What's your take on that? [45:41] >> That that's a real trade-off. Um, [45:43] especially if [45:45] the module that you're writing is [45:50] you can definitely write it in a way [45:51] that makes it harder to figure out [45:53] what's going on. If you're especially if [45:55] you're not following Rails conventions, [45:56] like the the conventions exist for a [45:59] reason. They help you find where things [46:00] are, reason about where things work when [46:02] you follow them. If you don't, some of [46:05] the magic, so to speak, can kind of take [46:07] over and make it a little bit more [46:08] difficult to reason about what's going [46:10] on. But that's just one of the [46:12] trade-offs that you make when Ruby and [46:15] Rails are fully leaning into the design [46:18] philosophy that it believes in which is [46:20] like duct typed OO. [46:22] >> Eddie, you have any thoughts on that as [46:23] well or [46:24] >> uh nothing to add? [46:25] >> Nothing to add. Okay. Well, let's shift [46:26] gears a little I want to kind of pivot [46:28] over to talk about infrastructure [46:29] because you mentioned that you're [46:30] running on Heroku and I think maybe for [46:33] listeners out there maybe an obvious [46:35] question might be is like or do you [46:37] think there's a good chance you're going [46:38] to try to stick around as Heroku as long [46:40] as they provide support or I know [46:42] there's a little bit of ambiguity about [46:45] their long-term support right now that [46:46] I've heard from we've been hearing from [46:48] their blog posts. Yeah, I think we've [46:50] been as uh confused as everyone uh [46:52] everyone else in terms of like I think I [46:54] saw it on Reddit of like you released a [46:56] blog post saying you're uh going into [46:59] maintenance mode, but you keep releasing [47:00] features and people were confused. So I [47:02] think we're kind of in the same uh camp. [47:04] I think we start exploring uh options [47:07] and I think we'll we'll move if we if we [47:09] have to. I mean in some areas we're kind [47:11] of hitting the the limits but uh we [47:13] still have to do the homework of like [47:15] actually exploring uh other options. uh [47:18] we use uh uniscale for autoscaling of of [47:20] our rails uh application and um I [47:23] remember those guys said that oh we're [47:25] gonna look into all the vendors and kind [47:27] of write up a blog post to uh suggest [47:29] you something that's like heroku like so [47:32] I don't know if that has been uh posted [47:34] already but we're kind of waiting on on [47:36] that but I think we'll also do all [47:37] homework uh when the time comes [47:40] >> so when we had prep for this [47:41] conversation you mentioned that you [47:42] needed to do some tuning with Puma can [47:44] you speak to that spec specifically [47:46] within the Heroku environment Uh yeah, I [47:49] think we we there we actually like took [47:51] the time to uh probably take advantage [47:53] of all the uh concurrency that you can [47:55] do in uh uh in Puma. So we were running [47:57] into issues where we're kind of taking [47:59] in requests and just like sitting idle [48:01] and not doing anything uh in the in the [48:04] process. So we start to take a look into [48:06] into that a little bit more uh and uh [48:09] actually leverage that Puma does for you [48:12] which is like hey you can handle uh [48:14] requests more more concurrently and like [48:16] scale a little bit uh a little bit [48:17] better and that turned out to be like a [48:20] pretty easy uh fix like it was just like [48:22] a configuration change like hey increase [48:24] the concurrency and all of a sudden you [48:26] can uh have a little bit uh better uh [48:28] performance with the trade-off of like I [48:30] think we also had to make some like [48:32] memory uh modifications because then [48:34] that would obviously start consuming [48:36] more memory in your in this case in our [48:38] Herokua dinos. So we also make like a [48:40] change there to start using uh I don't [48:42] know if I'm pronouncing it correctly but [48:43] Gemma lock for Ruby memory management. U [48:47] so yeah I think that those two have been [48:49] uh working good for for us so far and [48:51] the the change uh really helped us uh [48:54] increase our our request uh our request [48:56] per second processing. [48:58] >> Did you how did you go about testing [49:00] that out? Did you just just tweak it a [49:02] little bit and watch or did you were [49:05] able to kind of simulate this a little [49:06] bit in like a a different environment [49:08] before you rolled it out to the [49:09] production stack? [49:10] >> Yeah, so um we were able to simulate it [49:13] a little bit. So we um we we have like a [49:17] a sandbox environment in in Heroku [49:19] that's like very configure pretty much [49:21] like similarly to our production [49:23] applications. So we were able to test [49:24] some of these uh uh changes. I think [49:26] we've made the mistake in the early days [49:28] of like I think we actually tried to uh [49:30] turn it on a while back and I think the [49:33] application just stopped uh working. So [49:35] we learned our lesson. [49:37] >> That happens. You you learn you you [49:39] figure it out and sometimes sometimes it [49:40] works. Uh quite often it worked. Are you [49:43] relatively up to date with versions of [49:44] Ruby and Rails and and Puma? [49:47] >> Uh yeah. I think actually uh thanks to [49:49] the fact that we haven't deviated a lot [49:50] from like just your standard Rails [49:52] application. I think the the upgrades [49:54] have been pretty easy and we're able to [49:55] like keep up to up to date. I think now [49:57] with uh all the coding agents and stuff, [50:00] it's like pretty easy to like hey [50:01] >> can you bump to this version then have [50:04] it run tests then we run our own uh [50:06] verification just for uh our own uh [50:09] sanity and from there it's like they're [50:11] pretty easy to to get through. And out [50:13] of curiosity, is your team's kind of [50:15] standardized on AI tooling at this point [50:18] that you're using or you're kind of an [50:19] experimental everybody's doing a little [50:21] bit of trying different things at the [50:23] moment? [50:24] >> Um, I think more of the trying different [50:26] things at the at the moment. I would say [50:28] the majority are kind of center around [50:30] cloud code, but yeah, I think that's [50:32] still pretty open for the team and [50:34] you're also not enforcing like, hey, you [50:36] have to use this or that. And for our [50:39] listeners and watchers, we're recording [50:41] this in uh second or the later half of [50:43] June. So by the time this gets [50:45] published, who who knows where that what [50:47] where things might be. Um these things [50:48] are changing quickly. So I'm kind of [50:50] curious. You were talking about your [50:52] your deployment workflows there. And so [50:54] you're using Heroku, so sounds like [50:55] fairly kind of straightforward [50:57] deployment and you got sandbox [50:59] environments. Have you built any [51:01] interesting tooling internally to help [51:03] you simulate realistic data in your [51:06] local development environment or what's [51:08] that workflow look like? So you can have [51:10] that sort of volume. Do are you able to [51:12] even test locally or in your sandbox, [51:15] but it's like a pretty big volume of [51:18] transactions where you're bringing in [51:19] you mentioned like like an account or a [51:21] customer might have transaction that or [51:23] 10,000 transactions in a single uh I [51:26] forget the word you're using [51:28] >> a supplier statement. [51:29] >> Supplier. Yep. Yep. So, do you have [51:32] tooling locally for that that you've [51:33] been bu building out for yourselves or [51:35] bunch of rake tasks or what's this look [51:37] like? [51:38] >> Yeah, we have uh seating uh rake tasks. [51:42] Um, but we we haven't to my knowledge [51:44] Eddie correct me if I'm wrong, we [51:46] haven't invested tremendously in a lot [51:48] of uh guardraily type AI specific [51:52] technology um such as like [51:54] self-verification stuff um or [51:58] some of the things that you'll uh that [52:00] some other organizations are working on [52:02] to make it so that AI can fully develop [52:06] and verify its own work. We haven't gone [52:08] fully in on that. However, um I think on [52:12] a personal level, [52:14] I've used it uh a lot for that purpose. [52:16] A good example of exactly just what you [52:18] were talking about. Um a little while [52:20] ago, I needed to migrate one of our CSV [52:26] builders over to streaming from like [52:29] building a CSV file in memory because [52:31] the scale that we were hitting was [52:32] causing some of the reports that we were [52:34] CSV reports we were having customers [52:36] build. they can click a button [52:38] essentially to export a CSV of any list [52:40] view that they're looking at and the [52:43] volume was starting to get to the point [52:44] where building it in memory would become [52:46] insanely slow or sometimes even crash [52:49] >> and so I needed to migrate it over to [52:50] streaming. I went ahead and implemented [52:53] that and needed to figure out how to [52:56] test it. And so I was just like, Claude, [52:58] go into open a developer Rails console [53:01] and just instantiate a statement with a [53:03] million transactions and then we'll [53:05] generate a CSV to see if the memory [53:06] profile of the generator is stable over [53:09] time and that we got like a reasonable [53:14] decrease in the amount of latency that [53:15] it takes to process. [53:17] >> That's that's kind of a cool use case. [53:20] What about um are you able to test [53:22] things out in like your sandbox [53:24] environment to kind of before you push [53:26] things out to production for customers [53:28] when you're rolling things out? Like how [53:29] how much does that mirror our production [53:32] data if you don't if you're allowed to [53:34] share? [53:35] >> Yeah. Um no, so all of the data there is [53:37] just mock uh uh data, but we actually [53:40] rely a lot more heavily on on the tests [53:43] that we that we uh write especially on [53:45] our on a Rails application. It's not so [53:47] much like seating uh data and like [53:49] clicking around primarily because the [53:51] Rails application serves as a um there's [53:54] a JSON API uh uh today and I think when [53:57] we're testing the boundaries more of [53:59] data scale I think we we do those more [54:01] as uh well so like oh we we have some [54:03] scripts like benchmark this and um I [54:06] think just measuring uh uh memory [54:08] pressure and in uh in like different uh [54:11] places but yeah I think we we rely a lot [54:13] more heavily on like hey like making [54:15] sure um that we that we follow a test [54:18] first approach. Um so that's kind of [54:21] what prevents us more from the from the [54:23] box rather than like oh let's go into SM [54:25] box and test something out. [54:26] >> What are you using on the the view layer [54:29] then? Are you mentioned as a JSON API? [54:31] Are your customers using the JSON or is [54:33] that something you're using internally [54:34] with a different front-end framework or [54:36] something or or are you using Rails [54:37] views at all or [54:39] >> uh so we use it in both uh internally [54:41] and externally too because we also have [54:43] a an API that customers can can use. So [54:45] we use uh active model serializers for [54:48] our uh that side of the of the view uh [54:51] layer. [54:51] >> And what about when it comes to like [54:53] rendering out HTML and CSS? What's [54:55] >> uh so that one we actually run a a NexJS [54:58] application? [54:58] >> Okay. So there's so you have a next JS [55:00] involved as well. [55:02] >> Is that for the the customerf facing [55:04] side of things or [55:05] >> uh yes they they have like a dashboard [55:08] where they they do most of their tasks. [55:10] uh since that part of the application [55:12] what was the rationale for using that [55:14] versus using what kind of comes with [55:17] rails at the time? Do you recall that? [55:20] >> Um I think a lot of that was actually [55:22] during also the the beginning from uh [55:25] the engineer that was kind of selling [55:26] the the front end uh of things. It was [55:28] it was more of the ease of like I think [55:31] at that point in time I don't remember [55:33] if hot wire was already out or or not uh [55:37] when we were starting uh uh that um but [55:39] I think a lot of the dynamic uh work on [55:42] the on the UI was a little bit more [55:43] challenging to to do especially I think [55:46] just like finding uh engineers in the in [55:48] the market that were like oh I think [55:52] that the rails sorry at that point on [55:54] the on the UI was a little bit more uh [55:56] involved. Um, so is this what's easier [55:59] to uh speed up and get something [56:00] >> historically been been a little [56:02] complicated over the years? Uh, like [56:03] what's the new hot thing and is this [56:05] going to stick around and uh what's the [56:08] job? [56:08] >> It has gotten a little bit uh uh better. [56:12] >> Um, [56:12] >> it's true. Um, but you're always kind of [56:15] like wondering. I'm like, well, maybe [56:16] maybe enough people using this. Well, [56:18] hopefully. Uh, yeah. You know, another [56:21] thing you mentioned um when we were [56:23] talking is like you mentioned the [56:24] sidekick. Could you recall approximately [56:26] how many how many job or sidekick jobs [56:30] has your system been processing to date [56:32] approximately? [56:34] >> Yeah, I think we're more than one uh [56:37] billion. I think we're around like 1.3 [56:39] 1.4 uh uh billion. Um so we I think [56:44] psychic is it's a it's a really [56:46] important part of our of our [56:47] infrastructure. We rely uh heavily on it [56:49] and I think we we liked it a lot. [56:51] >> That's great. And I'm I'm assuming [56:52] you're doing that on a pro account. I [56:54] don't remember. I think we're actually [56:55] on the on the enterprise uh icon by now. [56:58] >> Enterprise. Yep. Sorry. Sorry. Um that's [57:00] awesome. When you're thinking about you [57:03] mentioned like using gems and stuff like [57:05] that in the community earlier, do does [57:07] your team kind of have an ethos or [57:09] approach about how you vet potential [57:12] gems you might bring in and do you feel [57:13] like that's different? Do you feel like [57:15] your thoughts on that has changed now? [57:17] We're we're talking about the AI era now [57:19] where u that becomes a potential risk of [57:23] bringing in some external dependencies [57:25] now like so historically what has been [57:28] your ethos about like when do you when [57:30] do you decide to build something [57:31] yourselves versus bring something into [57:33] the into the repository into your [57:35] project and throw in your gem file and [57:37] then two is your thoughts on that [57:39] changing at all at the moment [57:41] >> I think at the very beginning we were [57:43] following more of the approach of like [57:45] hey whatever we bring in it's still like [57:47] code that we own even if it's not like [57:49] on our code base. So you have to [57:51] understand that you cannot bring a a gem [57:53] that's like super complex for maybe a [57:55] tiny little problem that you're trying [57:57] to solve. So I think that that was a big [57:59] uh piece of like hey it's still code [58:01] that we have to maintain like whenever [58:03] we have to do an upgrade like we have to [58:04] understand what's going on if there is a [58:06] bug there like we have to be able to [58:08] figure out what's uh what's happening. [58:10] Um so I think that has been a big uh uh [58:14] piece of it. But I think with that uh [58:16] said we we don't shy shy away from like [58:18] especially for for things that are [58:20] already uh implemented right like we use [58:22] for example paper trail for our uh audit [58:25] uh logs on on models like hey we know [58:28] it's like something that has been around [58:29] for for a while is stable and we feel [58:32] comfortable using it. So I think it's [58:33] probably a combination of of that of [58:35] like hey uh we're going to own it and [58:37] then like hey is it something that other [58:40] people are using is uh stable and uh you [58:42] feel comfortable uh adopting. [58:45] Have you needed to navigate in these fi [58:48] past five years many gem migrations [58:50] where you maybe the support was [58:52] discontinued and you needed to migrate [58:54] to allow you to take advantage of some [58:56] new functionality with from the from [58:58] those gems or maybe it was a blocker for [59:01] an upgrading Ruby or Rails or Ruby [59:03] itself. [59:04] >> I think we've been lucky that every time [59:05] we try to do something like there's [59:07] already a PR open on the repository. So [59:09] it has been mostly like hey we need to [59:11] cut our own version until that gets uh [59:14] uh released. Uh so I think that has been [59:16] probably most common uh situation that [59:18] we've encountered rather than like oh [59:19] this is not supporting like I don't know [59:21] the latest Rails version or or whatnot. [59:23] >> Okay that tracks. Do you have many fork [59:26] gems right now that you've need you're [59:27] maintaining? You don't have to disclose [59:29] which ones they are but [59:31] >> I think uh I think now we've been able [59:34] to cut it down to maybe just two. [59:36] >> Okay. [59:36] >> Yeah. I don't actually don't remember [59:38] the number of how many how many we [59:40] depend on today. Speaking to that the [59:42] the point about bringing in dependencies [59:44] whether that be on the even sounds like [59:46] you're even maybe on the npm side like [59:48] how do you your team think about [59:50] bringing in dependencies is do you feel [59:52] like you're are you having active [59:53] conversations about that right now about [59:55] like hey like this is this is a way that [59:57] we could be potentially at risk that not [60:01] that you weren't before but it feels [60:02] like things could sneak in a little bit [60:04] quicker under the under under our nose [60:08] >> you mean from the existing ones that we [60:09] use or from uh new ones potentially from [60:11] new ones or [60:12] >> I think it doesn't happen that often [60:15] because we actually don't add new gems [60:17] uh that much at this point in uh in [60:20] time. I would say like the PG balancer [60:22] example that we were walking through [60:23] like that was probably the most uh uh [60:25] recent one. But aside from that, like I [60:27] would say the the problems throughout [60:29] the application are very similar. I [60:30] think we've been able to kind of [60:31] constrain ourselves to like a handful of [60:33] uh of gems. I think with that said like [60:35] with the ones that we have it is a a [60:37] constant thing that's on our mind of [60:38] like oh how do we make sure that those [60:40] ones don't get uh compromised. I think [60:43] uh that's something that's a top of mind [60:46] especially on on the field that we're [60:48] that we're in. [60:50] >> I appreciate that. Kean from your side [60:52] of the world I'm thinking about [60:54] infrastructure and decisions that kind [60:56] of affecting your day-to-day. One of the [60:58] things in our conversations, prior [60:59] conversations we were talking ahead of [61:01] this was that you mentioned that you're [61:03] needing to also maintain uh elastic [61:06] search I think was is part of the [61:07] equation with your Postgress. What are [61:09] you using elastic search for and what [61:11] and what sort of problems was it solving [61:14] and [61:15] are there any new challenges that that [61:16] has since introduced as well? [61:18] >> Yeah, that's right. We we maintain uh [61:21] two primary data layers. So we have uh [61:24] postgres and elastic search. The reason [61:27] we have elastic search is previously [61:30] alluded to many customers. They have [61:32] millions of policies. Supplier [61:34] statements can come in with like tens of [61:36] thousands of entries on them. And most [61:39] of the fields that exist, especially on [61:42] a transaction, our users at some point [61:44] will want to be able to filter or sort [61:46] by that value. And elastic search is [61:49] really good for that on large data sets [61:51] especially because when it comes to uh [61:54] supplier statements for example the [61:55] number of those in production is just [61:57] going to continue to grow unbounded. So [62:00] we rely on postgres for more um detail [62:04] view rendering and elastic search [62:07] renders our list views and provides uh s [62:10] filtering sortability search. [62:15] there any interesting challenges with [62:17] that or like if you have a pretty [62:19] growing database data set, how much of [62:22] that data is getting indexed and how do [62:24] you keep that in sync? [62:26] Yeah, both of those are are big [62:28] challenges. Um the first one that's an [62:32] evergreen ongoing challenge is when [62:34] you're trying to maintain um the same [62:37] data in both Postgres and Elastic [62:39] Search, you need to make sure that you [62:41] keep it in sync. Um we use active model [62:44] serializer to serialize the documents [62:47] into ES [62:49] >> and [62:50] um a couple of complications there. If [62:53] some of the fields rely on other models [62:56] in order to render their values, one we [62:58] need to make sure that when those [63:00] dependent models get updated, the [63:02] parents document gets reindexed so that [63:05] the list view and the filters and [63:06] sorting is is all accurate. And the [63:09] other is I'm sure you've seen lots of [63:12] active model serializers that are not [63:14] very performant. Um lot N plus1 queries [63:18] with relations and such like that. Um [63:21] and when those things happen, it can put [63:22] a lot of strain on the database. It can [63:24] slow the cues down. So we have to be [63:26] very careful. What would what would be [63:27] the impact of it not being totally in [63:30] sync right away? Like what sort of is [63:32] this like a customerf facing issue or is [63:34] it just it'll eventually get updated? is [63:36] just like a UI primarily issue. [63:39] It's it's definitely can be userf facing [63:42] um if severe enough because elastic [63:46] search is used to render the list views. [63:49] The worst case scenario is if something [63:51] doesn't get updated, the list view will [63:54] continue to render an outdated value or [63:56] if it's a newly created record, it just [63:58] won't show it at all. And then when you [63:59] click into the detail view for that, [64:01] it'll show the updated information. And [64:03] so the UI will disagree on the state of [64:06] the world in two different places. This [64:09] can happen if for example the cues get [64:11] backed up um for processing elastic [64:14] search indexing. Um it can happen if we [64:18] forget to reindex a document somewhere [64:20] that's important um like on one of those [64:23] dependent models for example. [64:26] Do you have any ways to like convey that [64:29] to the user if if there is things are [64:31] are a little out of sync or is it just [64:33] something how would you how would they [64:34] how would you know or they know unless [64:36] they don't raise a support request or [64:38] something like hey I noticed that the [64:40] dollar amount or whatever in the when I [64:43] did a search was this and it's showing a [64:45] different number here and that I would [64:47] imagine if that's there's a little bit [64:49] of a versus like they just the data [64:51] changed and they just happened to search [64:52] like immediately at the same time and [64:54] they Is that pretty obvious or like how [64:56] would you know that that's happening? [64:58] >> Yeah. Um I I think when whenever there [65:02] is a failure in this regard, it can be [65:03] visible to um end users pretty quickly. [65:06] They'll notice it. They'll report it to [65:08] us and we will know. Um [65:12] because in some contexts, for example, [65:15] we're extracting a new supplier [65:17] statement and all of its entries. [65:19] There's an expectation of eventual [65:21] consistency. And so there's a little bit [65:24] of a grace period that we're allowed [65:26] before like items populating into the [65:28] list view um will show up. We'll [65:31] generally try to block UI elements when [65:33] we're waiting on something to reindex [65:36] >> um flows. What's what's that what's that [65:39] workflow look like? How do you is it [65:41] just like it knows that it's going to be [65:43] rebuilt and so it just waits or [65:45] >> Yeah. Uh in the case of a supplier [65:47] statement when we're extracting [65:48] extraction for the record is um a PDF [65:52] document comes in and we use um AI to [65:57] parse the document and extract all of [65:59] the individual entries that are on that [66:01] statement [66:02] and create new uh models in our database [66:05] for each one. There's a record for each [66:07] transaction etc. Um, and while that's [66:10] completing, we mark a status on the [66:13] supplier statement that says that it's [66:15] currently extracting or matching. Um, [66:19] and [66:20] we have we use sidekick batching to [66:22] orchestrate all of the things that [66:24] happen as a result of extraction. So, [66:27] uh, matching, for example, will happen [66:29] automatically after extraction. We'll [66:31] try to match each of the entries to the [66:32] policies that they belong to. And after [66:35] all of the sidekick jobs have completed, [66:37] um the sidekick batch call back will [66:40] fire and it'll set it'll unblock the [66:42] supplier statement for user viewing. [66:44] >> Okay. Are you using any sort of like [66:46] state machine approach to processing all [66:49] that or [66:50] >> Yeah, we do. Um for extraction in [66:52] particular, there's state machine [66:55] transitions. We use um what's the module [66:58] called? It's AASM for short, I forget. [67:01] >> Access machine. [67:02] >> Yes. Okay. How does that kind of match [67:05] up from on your perspective, Eddie, when [67:07] you think about product requirements and [67:09] things around architectural decisions? [67:12] Was you already have patterns like that [67:13] that you're already using elsewhere in [67:14] the in the codebase and or is these kind [67:17] of more newer things that you've been [67:19] adopting as a as a as a team? [67:21] >> No. So, I think uh those patterns are [67:24] like pretty like center across our our [67:26] application especially because we do [67:28] depend uh heavily on uh on elastic [67:30] search. It is it is something that we [67:32] keep an eye on so that the the UI is [67:35] actually reflecting what the customer is [67:36] expecting to to see otherwise it creates [67:39] a bunch of uh confusion on their side [67:40] and and on our side as well. So yeah, I [67:42] think those those patterns are just like [67:43] share across the application. Yeah, we [67:46] we have some patterns that help us [67:48] enforce that um like callbacks on our [67:50] models for example on save or on update [67:53] that will run any related ES [67:54] serializers. But it's still tricky when [67:57] the document depends on other models for [68:00] some of its values to make sure that [68:02] when those change any necessary [68:04] documents that depend on it are also [68:06] updated. [68:07] >> That's that that's a challenging. Out of [68:09] curiosity, have you looked at [68:11] alternatives to Elastic Search since you [68:13] decided to first use it? I'm I'm not [68:15] saying I know something off top of my [68:16] head, but this seems like an interesting [68:18] challenge for that other teams might be [68:20] faced with. And um so if anyone's [68:22] listening out there and has got ideas [68:23] and want to respond or write something [68:24] in the comments there, but is this [68:26] something you're you're actively [68:28] previewing or is just like no, this this [68:31] will probably be fine for now, but I'm [68:32] just thinking you've mentioned hockey [68:33] stick growth data consumption. So I'm [68:36] like that's sounds like a pretty big [68:38] index to be managing. [68:39] >> We have. Yes. Um recently we started [68:43] looking at parade DB which is a Postgres [68:46] extension. We haven't made any decision [68:48] to migrate or anything like that. Um [68:50] it's a broader organizational decision [68:52] but it's something we started looking at [68:53] because it's a Postgres extension and so [68:58] you get the indexing uh the document [69:00] reindexing for free as part of just [69:03] saving uh records to Postgres in the [69:05] first place. Something interesting about [69:08] it is that it's also relied on by um [69:12] modern treasury who do similar money [69:15] accounting workflows and they rely on it [69:18] for like elastic search type work as [69:20] well. [69:21] >> Interesting. I'm not familiar with that [69:22] one. I'll have to check that out. You [69:24] always learn something new. Um I'm kind [69:27] of curious, you know, one of the other [69:28] things I wanted to kind of come back to. [69:30] you mentioned a little bit earlier [69:31] around using AI and Keagen, you [69:34] specifically had mentioned like [69:37] finding like you're you're able to [69:39] you're using AI a bit more. How would [69:41] you describe it from within the realm of [69:43] using it specifically working on Ruby in [69:46] particular like when it comes to using [69:49] these LLM tooling? Yeah, I think there's [69:52] a consensus opinion developing that AI [69:56] models, LLMs tend to work best in the [69:59] context of strongly typed languages [70:02] because it provides more guard rails um [70:04] and automatic verification for the AI to [70:06] check its work. But I think on the other [70:10] side of that, the conventions, the [70:15] expressiveness of Ruby and Rails also [70:18] lend pretty well to AI usage because [70:23] they provide a ready-made set of rules [70:25] for LLMs that kind of keep them on the [70:28] rails that are also token inexpensive. [70:31] As anybody that has worked a lot with [70:35] LLM coding knows, they are not very [70:40] proactive about developing abstractions [70:42] upfront or thinking about how things [70:44] should scale into the long term or [70:47] setting conventions. They will do [70:48] exactly the thing that you ask them to [70:50] do to solve the problem at hand. and [70:53] Rails and Ruby's conventions and [70:56] expressiveness help keep the token [70:59] expenditure down so that the reasoning [71:01] stays better and also provide a a good [71:04] set of conventions that are already made [71:06] for it to rely on. We definitely have [71:08] been hearing that more as of late. Um, [71:11] I'm I'm looking forward to seeing some [71:13] like more like some studies on this so [71:15] we can say this with a some confidence [71:17] like we know what we're talking about [71:18] and I know there's been some useful uh [71:21] anecdotes being shared out there. So I [71:23] want to take that with a grain of salt [71:24] for anyone listening. This isn't a [71:26] scientific perspective but something [71:27] we're we're trying to read the tea [71:28] leaves as best we can here. And Eddie, [71:31] what's your kind of take on what's your [71:33] perspective here? How are like AI [71:35] tooling changing like your workflows [71:37] over at end? I think it helps uh uh like [71:41] being a little bit more efficient with [71:42] the the task. And I think for actually [71:46] one great output from all of this is I [71:48] think people are kind of spending more [71:50] time on expressing their thoughts of [71:52] like oh I want to do uh this and uh this [71:55] is why and like this is what I'm [71:56] thinking. So I think that part has been [71:58] uh great because you you have more of [72:00] that clarity up front of like hey this [72:01] is what I'm like trying to do. On the [72:03] other side, I've been following more of [72:05] the approach and I think my my opinion [72:07] more on like hey whatever output comes [72:09] from AI like I still consider of the [72:12] output of the peer that I'm working with [72:14] like I I think I'm kind of against it [72:17] like oh I'll just put some code there [72:18] and like uh pray that somebody else will [72:21] review it for me uh and stuff. So I [72:24] think that that part I'm still like [72:25] mindful of of like hey whatever output [72:27] is coming out like I don't care if you [72:28] wrote it yourself or if you use a tool [72:30] to uh uh write it like I still expect uh [72:33] for the person writing it to like [72:35] understand it and and know what's uh [72:37] what's going on and and all of that. [72:39] Like I think I kind of see it the same [72:41] way as like back in the day when you [72:42] would like copy something from [72:44] Sackleflow like you still have to [72:45] understand it. You can just like copy [72:47] paste and like I'm done and move on. [72:51] There's an uh a blog from Ashb about AI [72:54] usage actually that really resonates [72:56] with me on philosophy about AI usage in [73:00] software development in general. It's [73:02] from this month and it's called AI ashby [73:06] engineering and the future. [73:09] And the the way that it positions it the [73:12] thesis is that the cost of writing code [73:15] is heading towards zero but the cost of [73:17] producing meaningful software is not. [73:20] >> Um and that the differentiators for good [73:24] engineers which are judgment, taste, [73:26] customer understanding, those things are [73:28] becoming more important rather than less [73:31] important. and we have to think more and [73:33] think harder than we did before because [73:36] LLMs make it very easy to avoid the [73:39] cognitive burden of doing the work. [73:41] That's the um some of the appeal [73:43] sometimes is being able to offload the [73:45] the thinking part. But when LLMs are [73:49] producing very plausible but not quite [73:52] correct or not quite the right pattern [73:54] code, it requires us to think harder and [73:57] better than we did before. And uh I [74:00] really appreciated that framing. Any [74:04] follow-ups on that, Eddie? [74:06] >> No, I think it uh it it's going to align [74:08] with my my thinking as well. I think [74:11] it's it's an interesting time right now [74:12] and for for teams and I've you know [74:15] interviewed a number of people in some [74:16] other upcoming episodes that'll be [74:18] published in the near future or I guess [74:20] actually before this episode but uh just [74:23] was always like trying to track like [74:24] where where different teams are at and [74:26] like you talk to some teams and they're [74:27] like fully like oh we're just letting [74:29] the the as Kempck is now calling them [74:32] the genie is just just he's this is not [74:35] what he he wouldn't say this but like [74:37] I've talked to some people who are like [74:38] AI is cranking out the code and they're [74:40] barely even reviewing the code anymore. [74:42] And I'm just like, that's wild. And I'm [74:45] like, how what kind of guard rails you [74:47] have in place? You feel so confident [74:48] about that. And then I'm like, it's just [74:51] interesting. Like, and some people like [74:52] we got to be super careful. I'm not [74:55] saying that that's you, but it's just [74:56] like where where is what's what's and [74:59] maybe is everybody's reality reality? [75:02] And that's okay. And like maybe it's [75:03] always been uh we're always we're always [75:06] working a little bit differently. I [75:07] don't know. I'm [75:08] >> I'm trying to be optimistic and [75:09] skeptical at the same time and it's an [75:11] interesting place to be and [75:13] >> I I think there's a lot of nuance in it [75:16] that's not always acknowledged by either [75:19] side having the conversation. It's very [75:21] context dependent. If the cost of being [75:24] wrong is low, then it makes a lot more [75:27] sense to just [75:30] >> fully vibe it out and have the AI ship [75:32] the code and barely review it because if [75:34] if it's wrong, then you can just iterate [75:36] and ship again. But if the cost of [75:38] failure is high and you need to get it [75:39] right the first time, it makes a lot [75:42] less sense to operate that way. Um, and [75:45] the the article that I mentioned, the [75:46] Ashby engineering one, also uh frames [75:49] the AI usage on a spectrum. Um, full [75:52] vibe coding for like low stakes tasks [75:54] like a throwaway script or some kind of [75:57] internal tool. And this we we do this at [76:00] we use um levelable for internal tooling [76:03] because the cost of making a mistake is [76:05] fairly low. [76:06] >> Yeah. Yeah. Um, but on the other side, [76:09] when the stakes are high, I think human [76:13] driven with like AI as more of a [76:14] sidekick makes more sense. You can have [76:17] AI type out some of the code, but you [76:20] can uh review it very carefully, think [76:22] through the problem, make sure you're [76:23] using the right abstraction, [76:25] etc. In our context, a lot of times the [76:27] stakes are high because we're we're [76:30] dealing with money. our customers, our [76:32] accountants, they care about the details [76:35] and and so even within the same company, [76:37] you see the the different spectrum of [76:39] what's acceptable. [76:40] >> I think that's I think that's a good [76:42] place to end the AI topic. I appreciate [76:45] that. I'm gonna I'll I'll find out find [76:46] that link and share that in the show [76:48] notes for us for as you said AI ashby or [76:52] >> I'll have you send that over to me and [76:54] we'll include that for everybody in the [76:55] show in the show notes. um kept you guys [76:58] long enough so I want to kind of start [76:59] working towards kind of few last things [77:01] I want to I'm kind of curious about and [77:04] this maybe for you Eddie you know you [77:06] talked earlier about coming back to Ruby [77:08] on Rails as as part of the secret sauce [77:10] of Asen's success to date and it's it's [77:14] a scent uh so to speak uh and what other [77:18] technical decisions do you feel like [77:19] your team's made that you feel like is [77:22] you really glad you did aside from Ruby [77:24] on Rails obviously is there something [77:26] else you feel like you you you were able [77:27] to do early on that you felt like has [77:29] been a really good anchor for the [77:31] organization on the engineering side of [77:33] things? [77:34] >> I mean I think tied to kind of the the [77:36] Rails uh uh stuff I think on the on the [77:38] infrastructure uh side we've also tried [77:41] to sort of keep things uh minimal. I [77:44] think from the very beginning our main [77:46] focus was like hey how can we spend more [77:49] time thinking about the product and and [77:50] building uh uh stuff. So, we try to [77:53] minimize the extra stuff as much as uh [77:56] as possible. Um, and it doesn't mean [78:00] like, oh, we're not going to pay [78:01] attention to it, but it's just more like [78:03] we'll pay attention to it when it uh [78:04] when it matters. Um, so I think we we [78:07] operate a lot under that principle of [78:10] like it [78:11] is not engineering for the sake of [78:13] engineering. I think it's just more like [78:15] if we have a a product and a customer [78:17] base to serve. So like we want to [78:20] dedicate our time to to that. Um so I [78:23] think a lot of the infrastructure [78:24] decisions come with uh with that of like [78:26] hey you have the trade-off of like oh [78:28] some of them might be more expensive [78:29] that what do you pay if you uh roll out [78:31] everything on on on your own but at the [78:34] end of the day like hey you don't have a [78:36] dedicated like infra or devops uh uh [78:38] person so we have to keep things uh [78:41] simple so that we can keep uh moving uh [78:42] forward. So I think we operate a lot [78:44] under that principle and I think also on [78:48] like kind of enforcing uh senders I [78:50] think that was something that we tried [78:51] to do at the at the beginning like hey [78:54] we want things to kind of keep a [78:56] cohesive uh approach on and stuff and I [78:59] think with that also came the sort of [79:00] the the testing uh philosophy of like [79:02] hey we're like writing things like we we [79:05] want to start uh uh writing tests for uh [79:08] for everything that we do especially on [79:10] our on our on our back end when we're [79:12] like dealing with uh money and all of [79:14] that. [79:14] >> So I think that kind of been great with [79:16] the uh the team and it's something that [79:18] we keep uh uh up until this uh day and I [79:21] think [79:22] >> something that has saved us also a lot [79:24] from like I think just having bugs in [79:26] production and stuff. Sure. But I think [79:28] just allows you to move with uh velocity [79:32] at the same time. [79:34] >> That's great. Out of curiosity, I don't [79:35] think you touched on this earlier. Uh [79:37] what what testing framework are you [79:38] using on within your Rails app? [79:40] >> Today we use uh RSpec. Okay, using [79:43] respect. All right, [79:44] >> I think that decision is probably [79:45] debatable, but [79:46] >> okay, [79:47] >> have enough, so we're sticking with [79:49] >> uh Well, I I've heard some people have [79:51] been using um their LLMs to to migrate. [79:54] Uh and putting that like we'll just [79:56] offload some of that stuff. I don't know [79:57] how conf I guess if you can find some [79:59] good uh parody there that would be not [80:01] the bad not a bad thing necessarily but [80:03] um I'm a longtime RSpec user and fan [80:06] myself but uh I've been using mini test [80:08] more recently on some some smaller [80:10] projects and like I could I could do [80:12] this I can probably get behind this as [80:13] well. Um but we don't need to have a a [80:17] religious war about tabs or spaces or [80:19] RSpec and uh [80:22] >> mini test there. But um no that that's [80:24] useful and I think you know your your [80:25] point there around maybe intentionally [80:28] uh not having certain types of roles [80:31] maybe around like infrastructure and [80:32] stuff like that and using something like [80:34] Heroku and trying to use that as an [80:36] interesting constraint for your [80:37] organization. Maybe there's a tendency [80:39] to kind of like well we'll add more [80:40] complexity to the infrastructure if we [80:42] have people that know how to do that and [80:45] maybe there's a really good reason for [80:46] it that but then there's the cost of not [80:48] maintaining that and people like maybe [80:50] people on your team may or may not have [80:51] the time and bandwidth to learn all [80:54] those things to also be able to [80:55] participate in some of the [80:56] infrastructure management that might go [80:58] get involved in that. Is that is that a [80:59] safe kind of read on what you were [81:01] saying there? [81:02] >> Uh yeah. No, I think that's uh that's [81:04] right. I think I feel like all of the [81:05] things that we do need a reason of like [81:07] oh why are we doing it but some of the [81:10] problems like for example on like the [81:11] data ingestion elastic uh search is like [81:14] hey we have a reason because we're like [81:15] now getting more more customers so like [81:17] now it makes sense for us to spend time [81:19] there versus like before if you had like [81:21] very little traffic like why would you [81:23] have like a super complex uh uh set of [81:26] infrastructure when it's not actually [81:28] necessary. As someone that works in the [81:30] consulting space where we get brought [81:32] into projects, there have been many many [81:34] projects where uh there'll be three [81:37] developers that had worked on an app and [81:38] there'll be five repositories, Terraform [81:41] involved and I'm like you've got no [81:43] customers. What are you doing? Like so [81:47] that that's a thing. Um all right. Well, [81:49] with that couple quick questions. Uh [81:51] we'll start with you Kagan. Is for both [81:53] of you though. Is there a programming [81:55] book or technical book that you find [81:57] yourself recommending to peers? [82:00] >> Oh man, tough question. Um I I think I [82:04] need a second program. [82:06] >> I I do not as [82:09] not at the velocity that I wish that I [82:11] did, but I do read them. One of my [82:14] favorites is the classic I'm forgetting [82:16] actually the title of it embarrassingly. [82:18] Um, but it's an introductory programming [82:22] book uh through the perspective of [82:24] Scheme uh from an MIT course. I don't [82:27] know if if that rings a bell for you, [82:29] >> but that that's a personal favorite of [82:31] mine. Eddie, do you know the the title [82:33] of that one? [82:34] >> I I know, but I I don't know the title. [82:36] >> What about you, Eddie? Do you have a [82:38] >> I read uh I think a few years back [82:40] already the it's a philosophy of [82:42] software uh design. It's a pretty like [82:45] small uh uh book, but has a lot of [82:49] interesting uh uh pieces. One of those [82:51] that you could probably read in a in a [82:52] in a day or or even in a week. Like it [82:55] doesn't have a lot, but I think it just [82:56] has like ideas around like how to design [82:58] software that are pretty I find pretty [83:00] uh valuable. [83:01] >> Structure and interpretation of computer [83:03] programs. That's the one. [83:05] >> Okay. It's a it's a real nerds book. [83:08] >> Okay. Real nerds book. All right. Well, [83:09] great. We'll defin we'll definitely [83:11] include links to the both of those in [83:12] the show notes. And out of curiosity, [83:14] where does the Ascend have like an [83:17] engineering blog or anything like at [83:19] this point or have you talked about that [83:20] at all or to share what you're you're [83:23] cooking up over there? [83:24] >> We've uh talked about it. We've never [83:26] done it, but uh yeah, [83:28] >> probably something that we will look [83:29] into. Yeah, [83:31] >> maybe you can have one of your AI bots [83:33] build you one and then when your first [83:34] post could be like, "Hey, we were on the [83:36] on rails podcast and you check out the [83:38] video. Um, here's some of the things we [83:39] talked about. So, I'm giving you your [83:41] first blog post right there. Um, [83:43] [laughter] [83:44] but but I think, yeah, I think there's [83:45] something you you you folks are doing [83:47] some really interesting stuff there that [83:48] I think the community would benefit from [83:50] hearing more about. So, um, I want to [83:52] thank you both for taking time to come [83:54] on on Rails and talk shop with us and [83:56] share some things that are going on [83:58] behind the scene over at Ascend. I [83:59] really appreciate that. And on behalf of [84:00] our audience, thank you. [84:03] >> Thanks for having me on. [84:06] That's it for this episode of On Rails. [84:09] This podcast is produced by the Rails [84:11] Foundation with support from its core [84:12] and contributing members. If you enjoyed [84:14] the ride, leave a quick review on Apple [84:16] Podcast, Spotify, or YouTube. It helps [84:18] more folks find the show. Again, I'm [84:21] Robbie Russell. Thanks for writing [84:22] along. See you next time.