Hussain Akram & Antonio Carlos de Souza: Rails Scales for the Rest of Us

Hussain Akram 与 Antonio Carlos de Souza：面向大众的 Rails 可扩展性实践 Donorbox 依托由 Heroku、Cloudflare 和 PostgreSQL 构成的简单架构，有效减少热路径上的数据库争用，承载了数亿次请求与高峰捐赠流量。为保持研发效率并消除部署瓶颈，团队通过定制的本地工具，将测试从远程 CircleCI 流水线迁回开发者本地设备，加速了代码合并周期，并将测试覆盖率提升至 100%。这两位架构师还分享了他们对原生 Rails 模式的严格恪守——例如放弃 Slim 模板回归标准 ERB、避免随意设立 service 对象目录，以及借助 ActiveModel 处理复杂的搜索表单。最后，他们介绍了务实的开发者工具使用方式，即通过 AGENTS.md 文档与 Model Context Protocol 服务引导 AI 助手，同时未放弃人工对系统架构的把控。 Donorbox 运营着一个高流量筹款平台，在 Cloudflare 边缘缓存的协助下，仅靠平均 2.5 个 Heroku web dyno 即可每月处理 1.88 亿次非静态资源请求，证明了标准 Rails 单体架构无需微服务也能有效支撑常规业务负载。 (25:29) 工程团队借助内部 Rake 工具将 CI 流水线从 CircleCI 迁回开发者的本地机器，使测试运行时间从一小时大幅缩短至 8 分钟以内，消除了竞态条件，并将测试覆盖率从 86% 提高到 100%。 (44:52) Donorbox 恪守严格的 Rails 默认规范，不设立顶层的 app/services 目录，将第三方集成封装在 lib/ 内部，并特意逐步淘汰 Slim 视图模板以回归标准 Rails ERB。 (61:01) 该团队通过集中的 AGENTS.md 文件以及内部 Model Context Protocol (MCP) 服务器来引导 Claude 和 Cursor 等 AI 编程工具，使 AI 模型能够直接访问设计系统组件库。 (81:27)

[81:00] Antonio Carlos de Souza: I see the LLM agents as a amplifier of yourself. So it will amplify your knowledge and also amplifies your lack of knowledge. [73:00] Hussain Akram: The power that ActiveModel and ActiveSupport has, it's super underrated.

[0:00] [music] [0:04] Welcome to On Rails, the podcast where [0:06] we dig into the technical decisions [0:07] behind building [music] and maintaining [0:08] production Ruby on Rails apps. I'm your [0:10] host, Robbie Russell, and I run Planet [0:12] Argon. And for over 21 years, we've [0:14] helped teams maintain and evolve their [0:15] long live Rails apps. So, I tend to [0:17] approach these conversations through [0:18] that lens. In this episode, I'm joined [0:21] by Hussein Aram, an engineering manager [0:23] at Donorbox, and Antonio Carlos Duza, [0:25] Donorbox's chief architect. Donorbox [0:27] helps nonprofit organizations raise [0:29] money online and in person. And behind [0:31] that platform is a Rails application [0:33] that has been evolving for more than a [0:34] decade. We'll dig into how their team [0:36] keeps that application relatively simple [0:38] while handling serious traffic. How they [0:40] think about Rails conventions and code [0:42] design and some of the ways that they're [0:43] pulling older parts of their application [0:45] towards patterns they want the rest of [0:47] their team to follow. Hussein joins us [0:49] from Virginia and AC joins us from [0:51] Brazil. [music] All right, check for [0:53] your belongings. All aboard. [0:57] Hussein and AC welcome to on rails. [1:00] >> Hi Robbie. Thanks for having us. [1:02] >> Hey thanks for having me. [1:05] >> So let's get things off to start. I'm [1:07] going to start with you Hussein first. [1:08] What keeps you on rails? [1:11] >> I would say typical conventions over [1:13] configuration stuff because I started [1:15] working with Rails when I was in my [1:17] third or fourth semester in my [1:19] bachelors. So it was very easy for me to [1:22] just follow the conventions rather than [1:24] going deeper into the stuff because I [1:27] was still uh studying the other courses [1:30] in parallel like object-oriented [1:32] programming, data structures, assembly, [1:35] C language. So it was so easy for me to [1:38] build stuff so quicker for my term [1:40] projects. So that's where I fall in love [1:43] with this framework and kept using it. [1:45] It's so easy for me to solve hard [1:49] problems rather than solving the [1:51] infrastructure problems if that makes [1:53] sense. So if I have some problem, I [1:57] don't have to think about how I'm going [1:59] to go about it infrastructure wise. So [2:02] it's just having a mental model and just [2:05] solving it using Ruby code which is so [2:08] much easy to read and write, you know. [2:10] So yeah, those those would be the most [2:12] prominent things that I love about Rails [2:15] and that keeps me using it. [2:17] >> Going through uh the educational process [2:19] of that and and learning to be a [2:20] software engineer. Did you know you [2:21] wanted to primarily work in the web [2:23] space or were you kind of just [2:25] interested in software engineering in [2:26] general? [2:27] >> Yeah, I was very picky uh on selecting [2:30] my first job after my bachelors. So I [2:32] searched around what are the companies [2:34] that are using Rails and the guy who [2:37] interviewed me was super impressed. uh [2:40] you know by the knowledge that I already [2:42] have about rails and and it was smooth [2:45] experience for onboarding because I [2:47] already knew the framework. So training [2:49] was like two weeks and then I just [2:51] jumped right into the project. Yeah. [2:53] >> Wow. And and just to provide some [2:56] context for how long ago was that like [2:59] what what version of Rails do you recall [3:01] that was? [3:02] >> It was Rails 3 or four. I it was in 2018 [3:06] when I graduated and I got the job even [3:09] before my last semester so mid 2018. [3:13] Yeah, [3:14] >> that's great. That's it's great to hear [3:16] people be able to graduate and then and [3:18] land a job getting to use Ruby on Rails [3:20] right out of school. Uh what about you [3:23] AC? What keeps you on Rails? [3:26] >> Uh by far the convention. I was a little [3:29] bit older than Hussein. So I started [3:32] working in 2002 [3:35] and at the time I was working with Java [3:37] and for the web. I really enjoyed those [3:42] information systems in the grad school. [3:45] I was more like uh trying to get what I [3:48] can learn and one thing that always [3:51] struggles with me was the roll out [3:54] process of non web applications and I [3:58] was always wondering how people do that [4:00] in real life. So at some point in life [4:03] my life I got to know about the web and [4:06] how it works. So, it's kind of a [4:11] I got in love [4:15] and what happened is I was looking [4:17] around on things to use for the web. I [4:20] work with flash. I don't know if you you [4:22] know ask your parents what it means [4:25] >> and [4:27] I use a Java I struts and his it's for [4:32] action. So it's a long time and one [4:37] thing that happens in the Java world is [4:39] that we have infinite discussions about [4:42] how to build the same thing. The most [4:46] extreme and I got was when we were [4:51] discussing which logging framework we [4:54] should be using. [4:56] The problem for this conversation is [5:00] well there are three main competitors in [5:02] the Java landscape at the time commerce [5:05] login log forj and something [5:07] [clears throat] else I think it's the [5:09] default login for the Java uh itself the [5:14] thing is you need all of them because [5:16] you have dependencies and those [5:18] dependencies [5:20] >> will depend on them so even yeah even [5:23] the discussion is pointless [5:26] And and what happened is in 2009 [5:31] I guess [5:34] uh I was looking for a platform that I [5:37] don't have those kind of pointless [5:39] discussions so that I can actually do [5:42] something and that's when I met Rails. [5:46] Uh at that point I was learning about [5:50] two different things that I don't know [5:51] if ever heard about it. the Tobu's gears [5:55] and jungle both from Python [5:59] and I learned about raves [6:02] and guess what they have a ton of [6:05] conventions like you don't need to [6:07] discuss that anymore [6:09] >> it's kind of a blessing and of having [6:13] those conversations I read saddle for [6:16] this particular [6:19] environment where I work so it's more [6:21] like a Yeah, I finally found it, you [6:24] know, and I keep on that since then. [6:30] >> That's great. I think about, you know, [6:32] that like AC like my story is probably [6:34] more similar to yours. I wasn't doing [6:36] much Java at the time, but like there [6:38] was a lot of these other it felt like [6:40] every project it was like, all right, [6:41] what are we doing on this project? What [6:42] do we want to try this time? Because [6:44] maybe we weren't happy with the choices [6:46] the last time or someone was super [6:48] curious or like you didn't work on a [6:50] project someone else built. like what [6:51] what decisions did they make? And it was [6:53] always trying to put together this [6:55] puzzle of like, all right, well, okay, I [6:57] kind of understand how these things work [6:58] together and then like [7:00] >> next time I start a new project, I'm [7:01] going to do it this way and like and it [7:04] felt like every project was an [7:05] experiment a little bit and and in taste [7:08] and that's [7:10] >> Yeah, absolutely. I I [7:11] >> and I don't know if you you got this as [7:15] well but in 2008ish [7:20] the Java world is getting a lot of the [7:23] design patterns debates. [7:26] >> So instead of reading the design [7:29] patterns and understanding the concept [7:31] of well if you have this problem you can [7:34] implement the solution and this are the [7:37] cons for using that solution. [7:41] people jumped directly to the UML models [7:46] and said, "Well, we need to implement [7:48] that." [7:49] >> Yeah. [7:49] >> You [clears throat] don't even have this [7:50] problem. Why are you doing that to you? [7:54] And [7:55] >> that's that's wild. Do you think you [7:57] both generally agree on what good Rails [8:02] looks like when you when it comes to [8:05] looking at I don't know how many [8:06] different applications you each worked [8:07] on and we'll talk in a moment about [8:09] donor box as well but just like in terms [8:11] of like knowing that you both kind of [8:13] like found Rails in very different paths [8:16] and you know AC you come from this [8:17] background of like working with [8:18] different platforms and for a several [8:20] years not that you weren't Hussein but [8:22] it just like in that you were in that in [8:24] the I'm air quoting the real world for [8:25] audio listeners here of like you're [8:27] working on different projects and you're [8:28] like okay was there also a moment of [8:30] like once you started working with Ruby [8:32] on Rails you saw like you removed some [8:34] of the decisions but then as you work on [8:37] across different Ruby on Rails code [8:39] bases are you getting a sense of like oh [8:41] this is like a good approach to building [8:44] and scaling a Ruby on Rails application [8:47] versus what doesn't look like a good [8:49] approach to that with Ruby on Rails [8:52] >> the Rails build unless you're not paying [8:55] attention at Oh, what are you doing? It [8:58] builds rails building in you a sense of [9:01] familiarity. [9:03] So when you jump from one application to [9:05] another, even in different teams or even [9:08] different companies, you kind of can [9:12] guess where things are for the shape you [9:16] expect them to be. [9:18] And [9:20] this feeling kind of it's what guides me [9:23] to the that's a good approach. So using [9:26] Rails that's a weird approach. I working [9:30] on companies that they are constantly [9:32] fighting Rails as I got to a point at [9:35] asking why exactly you pick Rails again [9:39] because every single decision Rails has [9:42] they try a way to change it. [9:47] I love people get bored because well you [9:51] know like you don't actually have those [9:52] problems anymore so you just need to you [9:56] know actually deliver something [9:59] or if something like they [10:02] they keep saying that they need to build [10:04] the the application [10:07] and has this feeling of they built it [10:10] you know like they took the decisions [10:12] >> people problem maybe or [10:16] who who knows I I get that sense. [10:18] >> But that's the amazing thing. This [10:21] problem only exists [10:23] because hers allow you to actually [10:25] deliver in the end of the day. [10:28] >> So even after that you end up is still [10:32] delivering somehow. [10:35] >> What about you? Yeah, I would just say [10:38] if you're using RS, I think try not to [10:41] reinvent the wheel every time you solve [10:43] a problem and just go around and see the [10:46] community, talk to the other the person [10:48] who are using Rails or just go search [10:50] online and see how people solve the [10:52] issue. Most probably 99% this problem [10:56] specific problem is already solved by a [10:57] simplest way. So the code bases I have [11:00] struggled with. This is the issue that I [11:03] saw in the codebase. they were just [11:04] reinventing the the stuff and it just [11:07] made codebase so complex which is hard [11:10] to if you just using the conventions [11:13] that Rails provides. Yeah, it's very [11:16] hard to you know diverge from that line [11:20] if you're using Rails because everything [11:22] is in front of you. You just need to put [11:24] the pieces in right place and you're [11:26] done you know. So yeah that that's [11:28] that's a main uh point I wanted to [11:32] mention here. Yeah, I'm sure we'll [11:34] probably get to dig into some more maybe [11:36] some tangible examples in a little bit, [11:37] but I think for provide some context for [11:40] our listeners whose maybe you can give [11:42] us a kind of a quick highle tour of [11:44] donor box from a engineering perspective [11:47] and what does the rails application [11:49] actually have to support day in and day [11:51] out at this point in time. [11:53] >> Sure. So our engineering team is divided [11:56] into five or six teams and those teams [11:58] surround and have responsibilities on [12:02] particular aspect of service that we [12:04] provide to nonprofit organizations. For [12:06] example, my team is um in-person [12:09] fundraising team which is basically uh [12:11] includes everything that has to do with [12:13] inerson. It can be live kiosk right in [12:16] and on my back. uh it can be uh [12:18] ticketing events um text to give and all [12:22] that stuff. Other teams are online [12:24] fundraising team. They handle online [12:27] fundraising form our main donation form [12:29] and then we have or experience team that [12:32] handles and provides best tools for us [12:35] for other teams to use and then we have [12:37] platform team integrations team for [12:39] Salesforce and HubSpot and all other [12:41] stuff. So this is how uh we have divided [12:44] our teams and each team has one project [12:46] manager, designer and QA. [12:49] >> So in person literally means like being [12:51] at a real live event for people. So so [12:54] online you know we can talk about donor [12:57] and in the intro I provided a little bit [12:58] of an overview of what donor box is. But [13:00] I I think about so you've got online [13:02] donations. I think people can wrap their [13:03] head around it. But then you've got you [13:05] pointed it behind you for our listeners. [13:08] There's a kiosk in the back corner [13:09] behind Hussein in the in his in his home [13:11] office there. Um, so it's like [13:13] facilitating that. Does that have to do [13:15] anything like maybe get into the weeds [13:17] of some of the fun details, but I've [13:19] worked on some like event management [13:21] tools in the past and with some some of [13:23] different clients. Some of those require [13:25] having to work around maybe unreliable [13:27] internet and at times in those spaces do [13:30] you have to do any sort of like offline [13:32] processing or anything like that or at [13:34] this point? We also had a similar issue [13:37] where because uh kiosk are being used by [13:40] mosques and churches across the US and [13:43] some of the places they don't have a [13:45] stable connection. So we ran into this [13:48] uh and stripe does provide a way to [13:51] process transactions in an offline mode. [13:53] So our kiosk also handles those [13:56] transactions in an offline mode. So [13:58] basically we use um devices uh local [14:02] storage uh store them and then when the [14:04] device gets online we start processing [14:07] uh those transactions and just sends [14:09] them to the server to process. [14:11] >> Nice. [14:12] >> Yeah. [14:12] >> And uh you mentioned like five or six [14:14] teams approximately how many people I [14:16] don't know if you remind did you say [14:17] that how many approximately how many [14:19] engineers are part of that team right [14:21] now? [14:21] >> AC I think port 50 [14:25] >> 50 I guess. [14:26] >> Yeah. Okay. I I didn't mention the sport [14:28] team. Amazing sport team. Uh they they [14:30] serve our customers very closely. So [14:34] yeah, they [14:35] >> was sport team. [14:37] >> Yeah. So sport agents they are also [14:39] engineers uh and they kind of do um high [14:44] level debugging on stuff. [14:46] [clears throat] So if they can [14:47] understand fix a bug and then just push [14:49] it without disturbing the you know uh [14:51] relative team. Um but yeah if if if the [14:55] issue is so big or needs a team [14:57] collaboration then obviously they [14:59] escalate it to the respective team [15:02] >> and is this primarily a like monolith [15:05] rails app or do you have is this broken [15:07] up into multiple things or is that a [15:10] touch [15:10] >> it's a monolith monolith app but we do [15:14] have some small repos for example for [15:16] our kiosk is has um code on Android and [15:20] iOS repos and then our integr ations [15:22] team they have their own repo because [15:25] you know they have to communicate with [15:26] Salesforce and all other stuff. So they [15:28] have a very simple uh repo which [15:30] communicates through the main monolith [15:32] using a our internal API. [15:34] >> Yeah you can think about this as a we [15:37] have this model ship and some support [15:40] ships around it. [15:43] The live kiosk we have the integrations [15:45] application that is also separated. So [15:48] it's essentially a background job for [15:51] different integrations. [15:53] This thing lives aside. [15:57] >> And how do you think about maybe AC on [16:00] your side like how when you zoom out [16:01] across the different product areas, what [16:03] has this allowed you to remain primarily [16:05] in like the one Rails application and [16:07] when you how do you think about when [16:09] you're going to carve off say like [16:10] something like this integrations [16:13] part of it? [16:15] Uh it's essentially a can I [16:20] delete this whole application if some [16:23] business change can delete this whole [16:26] application without anyone noticing. [16:28] >> It's essentially this is the question [16:32] >> is that a have you ever needed to delete [16:34] anything like that? [16:36] >> Uh in a way yes [16:38] it's not common behavior. So the the [16:41] default answer for this is uh no. We [16:44] probably if that happens we probably [16:48] need to delete the team as well because [16:50] it will be a huge business change [16:55] >> and [16:57] and what happened is this thing [16:59] especially the the integration app for [17:01] example it's essentially a background [17:03] job. So something happens in the [17:07] platform [17:08] and this something needs to be [17:10] communicated to a third party tool that [17:13] our customers log in there to do [17:16] something. There's no [17:19] UX essentially besides mapping what [17:23] happens on Donbox to a third party to [17:25] integration. [17:27] Those are the cases that we think about [17:30] okay maybe this can live in a different [17:32] application because Dbox has a ton of [17:37] PCI sock and whatever certificates we [17:41] require it's required from us [17:44] and because of those certifications [17:47] our process needs some double checks [17:54] >> that prevent us to walk faster. [17:57] The easier example we have is for the [18:01] it's very easy to explain to to an [18:03] auditor [18:05] that we create a pull request. We [18:09] require someone else that not the author [18:11] of the pull request to approve that pull [18:12] request before merging. [18:15] That's unbelievable easy to explain to [18:17] the auditor. [18:19] If I tell the auditor that well we do [18:22] have this home brew integration process [18:26] that runs a ton of sha against the your [18:29] pull request even before you share it [18:32] with someone else. [18:34] It requires the auditor to understand [18:37] what you're talking about. It makes the [18:39] audit process way more complicated or [18:42] longer. [18:43] balance between wasting time in audit [18:47] wasting time on the process daily. We [18:51] found out during our time that the [18:56] auditor is way harder than just ask for [18:59] someone that is already working with you [19:00] to approve this pull request. [19:02] >> That's interesting. So you're needing [19:05] there are some scenarios where you're [19:07] making architectural choices just to [19:09] make it simpler to work through the [19:12] auditing process. [19:15] >> Yes. Dobbot has this problem of not [19:19] following the the full and expected path [19:23] for a startup. Instead of breaking and [19:26] vanishing the sand of time, we're [19:28] striving. So we need to accommodate old [19:32] habits into a new set of requirements [19:35] that was unexpected. [19:38] >> How has Rails been part of that growth [19:42] story? Do you feel like Rails has helped [19:44] enable you or is it been able to stay [19:46] out of the way so to speak? So as your I [19:48] know when we were prepping for this [19:49] conversation like it started off with [19:51] one thing and you've been adding all [19:52] these other approaches to fundraising [19:55] and handling donations and things like [19:57] that through your platform whereas rail [20:00] has been really beneficial to you there [20:03] leave over to you AC to start with that [20:05] >> we fall back to the story lessons that [20:08] it was usually QT on the box I've been [20:11] box since 2019 [20:14] so I end up getting a lot of context of [20:18] how things happens in the past. What [20:21] happened is Dunbox was built by its own [20:24] founder. He learned Braos because it's [20:27] sexy at the time and that's the cool [20:31] kids are using. [20:33] The thing is he's not as a a programmer [20:36] by trade. He became a programmer by [20:40] necessity [20:42] before Claude, which is a big [20:44] difference. And the thing is, Rails has [20:49] a learning path, a clear learning path [20:52] for someone that never done a web [20:55] application before, but is willing to [20:58] learn and he walked his path. So he [21:03] managed to build a company through [21:08] essentially upworks tasks that he [21:12] managed himself while he's wasn't able [21:15] to build the the stuff. When I joined [21:18] the company the founder and owner of the [21:21] company is still doing product [21:23] deployments. [21:25] So rails literally enable the company to [21:28] be built by someone passionated about [21:33] what they're trying to achieve [21:36] and who wants to actually do something [21:38] that is not only useful but [21:42] secure [21:44] and manageable by one person. [21:48] >> That's great. So they the folks building [21:51] the application weren't historically [21:54] super experienced software engineers [21:56] were able to build the initial versions [21:58] get this thing bootstrapped pick ruban [22:01] rails because that was what the cool [22:02] kids were using or at least the other [22:04] people that they saw were being [22:05] successful is probably another way to [22:06] frame that like oh those people are [22:07] doing that I'll just I'll just follow in [22:09] their footsteps. I meet a lot of people [22:11] over the years who were like musicians [22:13] like I was a musician you know musician [22:15] myself as well but I think about like [22:17] people that didn't have a super deep [22:20] resume of like architecture planning [22:22] building startups necessarily like but [22:24] they had an idea [22:26] looked and saw what people were using [22:27] pick Ruby Rails off the shelf and be [22:29] like all right I'll pick this up as I go [22:31] and then you know not every product like [22:33] that ends up being successful you know [22:35] that that's okay but then the journey [22:38] was that eventually they realized like [22:40] it was kind of outgrowing them and like [22:41] maybe we should hire [22:43] >> some people that have even more [22:44] experience who can continue moving this [22:46] thing forward, improve it, make it more [22:49] stable and help it scale with as as a as [22:52] an organization which is I think a great [22:54] testament to Ruben Rails that it can do [22:56] that. Was that a interesting transition? [22:58] You know, we talked a little bit about [22:59] what was good Rails or not. I I'm not [23:01] saying you need to like complain about [23:03] like with the state of the codebase [23:04] because I'm sure every codebase has [23:06] their problems and if it's someone's [23:08] first major mature Rails app that [23:10] they're building, I'm sure there's going [23:11] to be some like things you maybe maybe [23:13] not want to uh not be super proud of [23:16] necessarily in retrospect, but you learn [23:18] from that and I think that's okay, too. [23:20] But what were some of the types the [23:22] types of things that you were like [23:23] coming in and being like okay this is [23:25] how we're going to strategize on an [23:26] iterating from this point forward that [23:29] you can recall? [23:30] >> Well a couple of them uh Hussein I [23:33] actually have a very good story on that. [23:36] So basically most of the code bases that [23:38] I have interacted uh they were built in [23:41] a rapid fashion because the owner or [23:43] co-founder wanted to solve the problem [23:45] as soon as possible and rails gives you [23:48] that power to you know just build and [23:51] worry about the optimizations and all [23:54] the other stuff later. So one of the [23:56] fine example was our events product [23:59] which was a kind of experimental feature [24:01] at that time. So our founder wanted to [24:04] give it a try because he heard a lot of [24:06] nonprofit organizations they want to [24:08] organize events for their main [24:10] fundraisers and this also relates to our [24:14] you know fundraising goal and providing [24:16] services around that area. So events [24:19] product was built in a very rapid [24:21] fashion but it was I think built within [24:24] three or four months and that came with [24:28] uh this cost of you know doing [24:31] refactoring and code that and everything [24:33] but still we we were able to solve that [24:37] issue gradually because we knew that the [24:39] events product is serving the nonprofit [24:42] very well. it's generating a good amount [24:45] of revenue for the company and now we [24:47] can invest on the optimizations but [24:50] still I would say it's a like positive [24:52] side of Rails that it still gave you [24:54] tools to build something cool in a very [24:56] small amount of time before like a gala [24:58] season and then it performed very well [25:00] and then we built like whole team around [25:03] this so you know now we have inerson [25:05] team of like four or five developers [25:08] which are managing other stuff along [25:09] with events so this happened with the [25:13] product and I have seen it even with the [25:15] main projects which was built a very in [25:18] a very fast fashion and then the owner [25:21] or the founder wanted to invest in [25:22] optimizations later on. I I I don't [25:25] think there's any shame uh like [25:27] following that path. [25:29] >> That's great. And then out of curiosity, [25:31] do you have any approximate stats you [25:33] can share with us on like how many [25:35] >> Yes. like donations are getting [25:37] processed like on an annual or maybe [25:39] even daily basis just to pro provide [25:41] some context for listeners. [25:43] >> Sure. Uh I'll start from the edge until [25:48] the donations. So [25:49] >> okay, [25:50] >> two very interesting numbers. We [25:53] processed in the last 30 days [25:57] 188 million requests. [26:00] >> Okay. [26:02] >> Filter out any assets. No fave icons. [26:07] >> Yes. No fave icon, no JavaScript, [26:08] [snorts] [26:09] no CSS, no image. [26:12] That's our volume of requests in a [26:14] month. [26:15] >> Okay. So about 120 almost 120 million. [26:20] >> Yes. [26:21] >> 30 days. Okay. How big of a platform are [26:23] you hosting this on? Is this like spread [26:25] out across several AWS data centers or [26:29] >> Yeah, it's kind of sad because you only [26:32] we use Heroku for the same reason we use [26:35] Rails [26:36] >> conventions. [26:38] They're already there and working. [26:40] >> You know the story about uh Heroku being [26:43] very expensive and stuff but well we [26:46] don't have a dedicated infrastructure [26:48] team to keep that thing running. In the [26:52] last 60 days, we have a spike of 200 [26:57] donations. 204 donations [27:01] made in 1 minute in July 15 [27:04] >> and no one knows. Literally, no one's [27:08] noticed. It's not a particular day for [27:11] anyone in the company. It's just [27:13] happened. [27:14] >> Yeah. AC keeps sharing this information [27:16] in our Slack channel. there would be [27:18] like a big spikes and nobody you know [27:21] noticed and he would show the cloudflare [27:24] stats that while you were sleeping this [27:27] this is what happened on our server [27:30] >> exactly [27:31] >> so you've got your Heroku dinos and then [27:33] you've you mentioned background jobs [27:35] what is that actually request like life [27:37] cycle look like so maybe from the web [27:38] request someone makes a donation is that [27:40] then immediately hitting a stripe thing [27:43] or something or is that and are you also [27:46] integrating with several different [27:47] payment payment processors as well to [27:49] make that work. I mean, I'm not sure [27:51] where your your target customer base is. [27:54] Is that very regional and like do you [27:56] have to have set up separate things in [27:57] different countries because not [27:58] everybody has a credit card everywhere [28:00] or is that primarily are you primarily [28:02] focused in environments? I only bring [28:03] that up because I know that some people [28:04] might integrate with Stripe in some [28:06] places, but not everybody has a credit [28:08] card in all places and it's not as [28:10] accessible and they might need to use [28:12] PayPal or some other sort of payment [28:14] methods or I'm assuming that kind of [28:16] complexity might show up in this kind of [28:18] platform, right? [28:19] >> Absolutely. So, we do have both PayPal [28:24] and a partner call it the giving block. [28:27] The gaming vlog help us with the stock [28:29] donations and cryptocurrencies. [28:33] And of course, Stripe has more than [28:35] credit card [28:37] like way more than that. [28:39] >> Yeah. Yeah. ACA only. [laughter] [28:42] >> What it do is [28:44] a majority of our donations came through [28:47] the US [28:49] but [28:51] we have around 40% outside of the US [28:55] Europe, United Kingdom, India, we have a [28:59] lot of donations. Besides [29:02] the majority of the nations came through [29:05] the United States, we do have a global [29:08] presence and Europe being the second [29:12] most and their own payment methods like [29:16] sea and it's not a they call it is ax [29:20] debit or something close to that. I see [29:23] [music] [29:25] you've worked hard to build your Rails [29:26] app, but have you thought about what [29:28] happens after you call [music] dossave? [29:30] An email gets sent, a job gets cued, [29:32] three other records update, and [29:34] somewhere a web hook fires for reasons [29:35] nobody remembers. That's why there's [29:37] call back insurance. Callback insurance [29:40] protects your team from unexpected side [29:41] effects caused by before save, after [29:44] commit, and the dreaded after save [29:46] inside a concern. So when someone asks [29:48] why did changing their address cancel [29:50] their subscription, you can rest easy [29:52] knowing you're covered. Call back [29:54] insurance because you shouldn't have to [29:55] know what happens after save. Coverage [29:57] [music] does not include recursive [29:59] callbacks, callbacks added by gems or [30:00] anything described in the code as [30:01] temporary. Existing condition may apply. [30:05] >> Back to that other part of my question [30:06] there was I was running around. So if [30:08] someone makes a web request, how does [30:10] that like that life cycle kind of look [30:12] like right now? Is that from there and [30:14] then maybe we can talk about from [30:15] Hussein your perspective like what [30:16] happens on the kiosk you mentioned like [30:18] things might be later sync when does [30:20] that all kind of connect to from both [30:21] sides. [30:22] >> Sure. So the request is started on [30:24] Cloudflare because we have it in front [30:26] of us and Cloudflare delivers the the [30:31] form [snorts] itself. It's either an [30:32] embedded form or the hosted forms that [30:35] you have a kind of a web page for the [30:37] forms. Most of the time you sketch it. [30:40] By the way, those are not counted in the [30:41] previous numbers. The user fulfilled the [30:44] information, send the request for make [30:46] the payment from the user perspective. [30:50] But for the user browser, [30:52] we request a stripe for a tokenized a [30:55] token of the payment method. And this is [30:58] the thing that lands on the box [31:01] infrastructure. So we essentially only [31:03] have the tokens from Stripe or PayPal. [31:06] From this moment on it's essentially [31:08] Heroku writing down in a post and that's [31:12] it you know like can't be more simple [31:16] than that. We do have background jobs [31:19] for the things that happens after the [31:20] donation like we try to create the [31:23] donation as a one insert in a one table [31:27] >> and it makes it faster because it's less [31:31] locks to create during this process. And [31:35] what happen is after the donation is [31:38] created, we create background jobs to [31:41] extract from the donation informations [31:44] like the donor. Do we already have this [31:47] don in our database or is a new one? So [31:50] if it's a new one, create it. Those are [31:53] what those background jobs does. [31:57] >> I see. That's interesting. That feels [31:59] like if you were starting to build the [32:01] app up kind of incrementally, you might [32:04] not think to do it that way initially, [32:06] right? If someone were to say, "Hey, [32:08] we're going to be we're going to be [32:10] building a thing. We're going to like I [32:11] don't know, maybe you have one big table [32:13] with a lot of columns and then you [32:14] insert that and that feels a little [32:17] atypical from like a the Rails way. You [32:19] might have [32:20] >> you might try to do that stuff in and in [32:22] in your initial create statement and [32:24] then [32:25] >> it might go off and create a couple [32:26] different objects or something like [32:27] that. Save other data in different [32:28] tables. So you have this relational [32:30] thing. Was that something that you were [32:31] just seeing as a scaling thing and like [32:33] that's where we're going to iterate from [32:34] and we evolved it that way or was that [32:36] an upfront plan because I think that's [32:38] where people start to maybe they [32:40] prematurely do that and think that like [32:41] well we're going to deviate from Rails [32:43] conventions in that way but how do you [32:46] wrap your head around that part of thing [32:47] if anyone listening to be like well [32:48] that's that doesn't smell like a typical [32:50] Rails thing but maybe that actually is a [32:51] really good use case if there's a high [32:53] velocity of of throughput. The [32:56] domination is our hottest path [33:00] by far. [33:03] >> And what happened is [33:06] the the application was built having in [33:09] mind one individual making one donation [33:14] to an organization. Through time what [33:17] happened is this concept of one [33:19] individual we started to getting [33:21] stretched [33:22] for yeah if it's a company make a [33:25] donation is that does it count as an [33:27] individual is still a donor the system [33:30] start grew around this concept of one [33:35] person making one donation and what we [33:37] measured [33:39] way later on like years before that is [33:42] that keeping things this Okay. Help us [33:47] on the scale. And by the way, we're not [33:50] Shopify, okay? So, we're not [33:52] >> Yeah. [33:53] >> scale for human beings for for the rest [33:55] of us. And I I was in way too many [33:59] discussions where people studying by [34:02] Rails doesn't scale. And I was like, [34:06] dude, you have more microservices in [34:08] your application than customers. What [34:12] are you talking about? [laughter] [34:14] Yeah, [34:15] >> those conversations are really hard and [34:19] and for the majority of cases [34:23] ra scale really well and you don't need [34:25] to do anything. You just need to pay [34:27] attention. So we got an issue on scale [34:32] because we use touch everywhere like you [34:37] create a donation we starting touch the [34:38] whole database. Why are you doing that? [34:41] because we are trying to update the [34:44] counter caches. [34:46] >> What is the problem? [34:48] When you have a really hot path that [34:51] needs to keep the database inserting [34:55] one donation after the other, if you [34:58] stack up a lot of updates, your database [35:02] is starting to complain. We have a lot [35:04] of out of the DOS because we created [35:08] this problem for ourselves. When you [35:11] have, I don't know, 100 customers, [35:14] that's fine. You don't even notice. You [35:16] have a thousand customers, [35:19] you see some spikes. If you're really [35:22] paying attention in your logs, don't box [35:25] right now have more than 30,000 one [35:29] database for writing. [35:31] >> Oh, wow. This is interesting. I think um [35:34] I think about that that path and I think [35:35] maybe for people listening I think you [35:37] know you mentioned you have like this [35:39] really hot path where you need to keep [35:41] the inserts as streamlined as possible. [35:44] Maybe you have this benefit of like [35:46] maybe if you bring up something like a [35:47] Shopify where a lot of things have to [35:49] happen because you have to double check [35:50] is there inventory, [35:52] you know, of the product, right? And [35:54] maybe in a donation thing, it's like I [35:56] don't know if there's like campaign type [35:58] things where there's some specific rules [36:00] like, well, we can't have this many [36:01] donations of this tier or if it's just [36:04] like, no, no, the more donations the [36:05] better. So, let's let's just make sure [36:07] money can come in as smooth as possible [36:09] and we'll make sure that we audit this [36:11] thing and we can track where it came [36:12] from so we can properly map this later [36:15] and like the the point is make it easy [36:17] to get money to the organizations you [36:19] want to help out and so like we'll [36:20] figure the rest of the stuff in the [36:22] post-processing phase of that. Yeah, we [36:25] are in the the more the donations the [36:28] better, but we do have very specific [36:30] campaigns like political ones that [36:32] require you some information that maybe [36:36] is not there [36:38] >> and those are more tricky in the sense [36:40] of well we need to run a validation [36:42] before actually accepting the donation [36:44] and report back to the the donor. [36:47] >> That makes sense. Those are the most [36:49] complicated cases in this for this [36:52] >> and part of the this validation is [36:54] already been done on the organization [36:56] level if the organization is valid and [36:59] they are compliant with 501c nonprofit [37:02] organization. So it's easy to assume [37:04] that their donors will also be legit but [37:07] we do have fraud prevention in place [37:10] that's working fine. It can easily block [37:12] donors who are just you know messing [37:14] with the donation form. There's not much [37:16] of validation that we need to do. So [37:18] it's just very simple. You send the JSON [37:21] parameters, incue the donation job and [37:23] yeah just write in couple of database [37:26] tables. Yeah. [37:26] >> What are you all using for background [37:28] job processing at this point? [37:30] >> We're using sidekick in revits for now. [37:34] >> Okay. [37:34] >> We do have plans to migrate to to the [37:38] froze rails [37:39] >> solid cube. [37:40] >> Yeah, solid cube. Thanks. Uh we have two [37:44] tests with solid Q on the D box right [37:46] now. One for the communication team and [37:51] we do have one test in the integration [37:54] team. So essentially the idea is we have [37:57] those supports ships around the mother [38:00] the mother ship and [38:03] >> we try there first. while you're [38:06] thinking about and running those tests [38:07] right now or is there anything that [38:08] you're encountering when it comes to [38:10] like using Reddus or Sidekick that's [38:12] like kind of like well is it that you're [38:13] just like let's just get on board with [38:15] what the general Rails conventions like [38:17] this is what's getting shipped by [38:18] default or is is there anything that [38:20] you're like some limitations or some [38:21] tooling that you kind of aren't seeing [38:24] or benefits if that you're you're [38:25] looking at. [38:26] >> Uh one of the main benefits is space for [38:30] the jobs. [38:32] the thing that happens. keep nudging [38:35] your head on that thing like hey think [38:38] about next time you need to do this and [38:42] increasing radius is okay in this site [38:44] especially on Heroku is really expensive [38:48] >> while having a proper database and I'm [38:51] not bashing on radius here proper [38:54] database I mean a relational database [38:56] that's all not judging anyone [38:59] >> the thing is [39:02] having a relational database to do that [39:04] is way easier especially when you're [39:07] starting to [39:09] to query the jobs that run. [39:13] >> Yeah, [39:14] >> one of the other pain point was the job [39:16] continuations. I think solid Q solves [39:19] that issue. Um so yeah that's a one [39:23] motivation because currently if we [39:25] restart the server make a new deployment [39:27] all the jobs have to stop and rerun [39:30] again. There are very few jobs that take [39:33] a lot of time but most of the jobs they [39:35] get processed within a second or two. [39:37] Some jobs get stuck and then we have an [39:40] amazing guy as an infrastructure person. [39:43] So he solves these kind of issues and [39:46] notifies the team. His name is Akrias. [39:48] >> Thank you for your service. [39:50] >> Yeah, sir. Mhm. [39:53] >> So yeah, he recently mentioned that we [39:55] also downgraded our sidekicks dyno from [39:59] using 15 gigs of RAM to 2.5 gigs of RAM [40:02] because we weren't utilizing anything on [40:06] that scale. [40:08] Is that one of those types of things [40:09] where just like somebody slid the thing [40:11] up at one point like this seems safe and [40:13] nobody complained about it and someone [40:15] was able to start gradually moving that [40:17] down and be like let's see what breaks [40:18] or or they felt like they were confident [40:20] enough to slide it way down to two and a [40:22] half like no no based off of what we're [40:24] seeing you know you mentioned you don't [40:26] have like a dedicated DevOps team [40:28] necessarily there but it sounds like you [40:30] do have at least one person there that [40:32] maybe knows a little bit more and solves [40:34] the day a little bit here and there. [40:35] What was that that workflow look like? [40:37] >> Uh we were talking about Zakarius. He's [40:40] our sever [40:43] shepherd. [40:44] Essentially what happened is Zakarius is [40:47] really really good on infrastructure [40:49] like he likes this. [40:52] So what happened is that he keep an eye [40:56] on what's happening on the database like [40:59] do you have any queries [41:01] running for more than 30 seconds? Do you [41:04] have queries, I don't know, not hitting [41:08] the the caches? Do we have queries that [41:13] loads half of the database to the memory [41:16] before trying to do something? He do [41:18] that because he actually likes it. [41:20] >> Yeah. [41:20] >> So what happened is the whole [41:23] infrastructure run by its own song. What [41:26] Sakaras does is making sure we're not [41:30] over reaching of this capabilities [41:34] and we can actually been using I have [41:38] the number for the last week for the [41:40] last month of our medium usage of web [41:44] dinos was two and a half. [41:46] >> Oh okay. For the majority of the month, [41:50] we go from two dinos because I'm too [41:52] paranoid to have only one to three dinos [41:55] and back to two. Most of the time you [41:58] see the shards. Heroku increase the [42:00] dinos when you make a production [42:02] deployment. So you made a production [42:06] deployment, all the cash is invalidated. [42:08] Heroku thinks you getting a huge spike [42:12] of requests. They add you one more dyno [42:16] 15 minutes later. Oh, okay. Sorry, [42:18] overreacting. They get back to two. [42:21] That's how we got to two and a half [42:22] dinos. [42:24] >> So, it's your deploys are triggering [42:26] that more than anything. [42:27] >> Yeah. [42:28] >> You know, when I when you think about [42:30] your your hosting infrastructure there, [42:32] how are you does your team what's your [42:34] what's your deployment process look like [42:36] right now? Are you before you deploy [42:38] something to production, are you doing a [42:39] lot of things in like a QA environment? [42:41] you have like a dedicated team that are [42:42] testing. You mentioned earlier I think [42:44] it was AC talked about your platform [42:46] allows your customers to like embed like [42:49] a form in their website. So maybe like a [42:51] nonprofit maybe you can put a embed a [42:53] form so they can kind of it feels like [42:55] it's part of their website but it's an [42:57] embed maybe it's I don't know are these [42:58] like iframes are these JS snip maybe a [43:02] couple different versions of those that [43:03] you might be doing. you can't really [43:05] predict how much traffic those places [43:07] are going to get, right? And then [43:08] necessarily because and that's that's [43:10] great that you're not having to think [43:11] about it that much, but I'm curious [43:12] about what's that QA workflow look like [43:14] when you're dealing with like these [43:16] forms elsewhere and like are you using [43:18] like Heroku preview environments or [43:20] something like that for that or is [43:21] relying a lot of just internal testing [43:23] there? How how do things get shipped [43:25] from a new feature or a new button [43:28] getting added and getting shipped to [43:29] production? [43:30] >> Hussein, that's all yours. [43:32] >> Yeah. [laughter] So our deployment [43:34] process is very simple uh and classic in [43:37] terms of we raise a PR and then like we [43:41] have JR columns every team has their own [43:44] Heroku review app where you can deploy [43:46] and notify the your QA to test it out. [43:49] Um and then we have really good close to [43:53] 100 I think 100% code coverage and some [43:55] of the tests that we have written are [43:57] system tests that actually test your [44:00] embed versions different type of [44:01] configurations and all that stuff. Um [44:04] and QA does another excellent job to [44:06] retest it uh re-mbbed it in different [44:09] like WordPress website or some other [44:11] places to see if it all works well. They [44:14] are not like frequent moments when we [44:17] make uh changes on our donation farm. It [44:20] happens like very rarely like when we [44:22] are like pushing major features to the [44:24] donation farm. For example, we re [44:26] launched a new version of donation farm [44:29] which was omni uh which needed uh [44:32] thorough testing from QAS and other [44:35] teams to make sure that everything works [44:38] fine on different versions of embedded [44:41] you know versions. After the QA is done, [44:43] we do code reviews. Every team is, you [44:46] know, is welcome to review the code of [44:48] their peers and then we have one command [44:52] which runs all the tests and pushes the [44:55] results to GitHub all green. You make [44:58] run another command deploy similar to [45:01] capestrano that we used to do. So it's [45:03] like internal shell command that we just [45:06] run and it deploys to our main staging [45:09] server which is used for demo purposes [45:10] and on our production very quick very [45:13] simple and boring. [45:15] >> You mentioned keeping it kind of boring. [45:18] What do you think that allows you to [45:20] focus? [45:21] You know, I think some teams are [45:23] struggling or maybe not struggling, but [45:25] they're kind of weighing up like how [45:27] much they're relying on their local CI [45:30] versus like doing like all the CI and [45:33] say on GitHub or stuff like that, GitHub [45:34] actions or GitHub actions has had its [45:37] reliability issues in recently and [45:39] hopefully by the time this airs they'll [45:40] have solved a lot of that. But you know [45:42] what it is what it is and are there any [45:44] other interesting tooling that your team [45:45] has built internally you feel like as a [45:48] as a result that because you've kind of [45:50] kept your infrastructure relatively [45:52] boring? Uh yeah, so we were using circle [45:55] CI before using local CI. With CircleCI, [45:59] we used to run into this issue where the [46:02] code coverage would drop if the two PRs [46:04] get merged simultaneously and when the [46:06] code is accumulated on the main branch, [46:08] the coverage would drop on in fractions [46:12] and people would be fighting with each [46:14] other like on slack that I'm unable to [46:16] you know deploy the next PR. at the [46:18] during the same time DHH posted this [46:21] post that he also shifted onto local CI. [46:25] So that was a big blessing for us to an [46:28] excellent idea for us to use that and [46:30] then we developed this rake gem which [46:34] basically runs the linting breakman runs [46:37] the test and everything and pushes the [46:39] results to GitHub PR. Now only one P can [46:42] be merged at one time and they we didn't [46:45] have any issue related to code coverage [46:46] dropage after that. [46:48] >> Interesting. [46:49] >> I have a couple of numbers about this [46:51] that maybe sound interesting. Uh [46:55] we keep track of the durometrics in dab [46:58] box. Durometrics is the effort from [47:00] Google for measuring the devops teams [47:04] inside Google. Because are for metrics [47:07] and they are depending on each other. [47:09] they are less capable of being cheated [47:12] or gified. One of the the metrics is the [47:17] deployment frequency. Last week we are [47:20] around 10 deployments a day considering [47:24] 15 developers. It's kind of nice. this [47:27] integration process that Hussein [47:29] mentioned. I have a slightly different [47:32] view on that because I was guilty for [47:35] introducing the this process. What we [47:37] had before, as Hassan already explained, [47:40] was a outsourced CI and it took around [47:44] one hour to run the the tests [47:48] >> and only run the tests and I have a [47:51] different view because I was the person [47:54] pushing to run the tests locally [47:58] so that the person doing the thing [48:01] experience the problems they are [48:03] creating. Another way and less sad way [48:08] of seeing this is that is that I'm [48:10] giving you back the feedback you lost [48:13] when you put that on a Circle CI and [48:16] this integration steps is pretty funny [48:19] because they are a little bit more than [48:21] just run the tests. We run the [48:23] statistics analysis. So we enforcing the [48:26] Ruby on Rails directory compliance. [48:29] So that's how we can have 15 people [48:32] without exploding different design code [48:35] designs. [48:37] We run Robocop breakman. We make sure [48:40] that feature branch is updated bundle [48:44] style because you know things happens [48:46] and then we test the migrations. [48:50] We drop the developer database and [48:53] recreate it running the migrations from [48:55] scratch. Why doing that instead of just [48:57] using setup? So that we check if the [49:00] migrations are actually working in the [49:02] process of migrating from a previous [49:05] version 21. Funny thing is the first [49:08] [snorts] version did exactly that. They [49:11] dropped the database and recreed. [49:16] The first thing people complain [49:19] was about well you dropped my database. [49:23] I lost my test environment, my local [49:27] test environment. [49:28] >> Yeah, happened a lot. My first question [49:32] is hey sir do you have time to listen to [49:35] our lord and saber fixtures [49:39] and it's pretty funny because people [49:42] didn't understand that fixtures is not [49:44] only about running tests is having a set [49:48] of known [49:51] situations scenarios for a database I [49:55] could convince people that features and [49:58] fixers load [49:59] is a way better solution that every [50:03] single person need to learn about this [50:05] new scenario and create the scenario [50:08] locally. The problem is that I have this [50:11] conversation like 10 years after the [50:13] application was built. So we kind of [50:15] need to catch up fixing the features. [50:19] >> I I would also admit that after doing it [50:22] locally, it also motivated developers to [50:25] re-evaluate the data tests. So because [50:28] initially the tests were taking more [50:30] than 20 minutes and with that time every [50:33] developer thanks to all the developers [50:35] at Donbox every developer was kind of [50:38] proactively working through the slow [50:41] test and the they were like optimizing [50:43] it sharing it with peers and doing it [50:45] across the project and now we reduce the [50:49] time to have still we are continuously [50:52] uh working on improving the time to you [50:55] know run the the whole test if there are [50:58] any warnings or any deprecation warnings [51:00] developers because you are running the [51:02] test on your machine so it's easier for [51:04] you to just see and ping the relative [51:06] developer or can you check if there are [51:08] deprecation warnings should we upgrade [51:09] this gem or something like that rather [51:11] than just outsourcing to CI and just [51:14] continuing on your work [51:16] >> what's always interesting about hearing [51:17] these these types of stories is like [51:19] that's how we used to do it and then you [51:22] know we used to run it all locally and [51:24] then there was like oh great GitHub [51:26] provided us a way to run it on their [51:28] servers or we had CircleCI or Jenkins or [51:32] and it was also another way to like be [51:34] another backup thing like well we assume [51:36] people are running their test. We didn't [51:37] have I think enough confidence to time [51:41] that our peers were actually running [51:43] their tests before they would push a [51:45] change up. And so it would require the [51:49] piece of like if you're doing like a P [51:51] before we had PR requests even you know [51:53] before we had pull requests uh when you [51:55] get a patch or whatever whatever the [51:56] process what I can't even remember [51:58] exactly how we used to do it but you [52:00] would have to pull down the changes you [52:02] run your tests locally and you you were [52:03] assuming that everybody was running [52:05] through the steps before they deployed [52:07] things and that wasn't always true that [52:09] was that wasn't always true and so there [52:11] were definitely times like well I just [52:13] we shipped this out really quickly [52:14] because there was a quick bug and then [52:15] that starts to degrade or that that [52:17] consistency kind of goes away. So, it [52:20] was nice to have a nice little green [52:21] check mark on GitHub to be like, well, [52:23] this thing says it's it's happening. We [52:25] didn't break anything hopefully. But [52:27] then you're saying it was taking an hour [52:28] to run your test and then maybe locally [52:31] it was taking 20 25 whatever minutes and [52:33] then maybe you cut that down even more [52:34] 10 minutes or so. Is that [52:36] >> Yeah, eight eight minutes. Seven to [52:37] eight. [52:38] >> Eight minutes. [52:38] >> Okay. And you would have to wait an hour [52:40] before for CI to do that. [52:42] >> Yeah. So, does that even mean that if [52:44] you're running it, like if you sent a PR [52:46] for somebody, then you're waiting at [52:48] least an hour before someone's going to [52:49] review it? [52:50] >> That that's exactly the problem. [52:52] >> Because the tests are taking so long, [52:55] they became uh [52:59] check mark you add to the thing, not a [53:03] developing tool. [53:06] So bringing back the CIS to the [53:09] developers [53:11] reminds the developers that hey guys [53:15] that's for you you know like the [53:17] customers completely don't care about [53:20] automated tests this feature the the [53:24] company because I worked on companies [53:26] that I need to fight for writing [53:27] automated tests [53:30] >> I'm not that old okay just happens to be [53:32] there [53:34] And [53:36] the thing is people forgot [53:40] that the automated tests are a tool for [53:44] developments and for developers and more [53:47] than that [53:49] they didn't saw that tool as they see [53:53] their food editor for example and this [53:57] is the main analogy I keep doing over [54:01] and over like dude you have tool that [54:04] the company not only allows you to to [54:07] create, they demand you to create just [54:11] to make your life easier. [54:13] Writing test doesn't make your PM lives [54:16] easier. It doesn't make your manager [54:19] life easier. It makes your life easier [54:23] because you are the person that's not [54:24] waking up to fix a bug that end up in [54:28] production and no one have no idea how. [54:31] We mentioned the task cover version [54:33] earlier that we reached 100%. First [54:37] guess who pushed this this number up of [54:40] course was because he is the person that [54:43] in the whole team care the most about [54:45] those details and the way you see the [54:48] test coverage is the information is not [54:52] the coverage itself but what's not [54:56] coverage. So 100% test cover is more [54:59] noise than signal. The signal is the [55:02] list of things that if you change it [55:07] don't have any way to automatically [55:10] check if that thing breaks. [55:13] That's the way we using this [55:15] measurement. Not if you're reaching [55:17] 100%. But if the list of things that [55:21] breaks if you change something and you [55:24] don't have an automated test for [55:26] checking it out and the reason for [55:29] reaching this 100% is well I cannot [55:32] [clears throat] guarantee the test will [55:35] exercise that thing but I can guarantee [55:38] that one of the tests pass it through [55:42] that path. So if even this is breaking, [55:47] something's really odd in the tests. [55:50] >> I'm curious about when you when your [55:52] team was putting more of an effort to [55:54] and and and kind of making everybody run [55:56] their tests themselves locally and [55:58] having to live with that and learned [56:00] that like it's going to blow away your [56:01] database. So you need your fixtures or [56:04] your seeds or whatever your team is [56:06] using locally to make that optimize that [56:08] workflow. Did your team have to go [56:09] through any training together or like [56:12] like was there like here follow what AC [56:15] is doing with this type approach that [56:16] seems to be the right path or is there [56:18] like workshops that you were doing to [56:20] get everybody kind of on the same page [56:21] or did everybody kind of pick it up over [56:23] time and at their own pace? What would [56:26] you replicate and what would you maybe [56:27] do differently next time? [56:29] >> I think you'd love to see the different [56:31] perspectives on this. So Hussein, if [56:33] you'd like to go yours please. at Donald [56:36] Box. It was like like getting everybody [56:39] on board on this was very easy because [56:41] we or AC built this shell tool on GitHub [56:45] shared it with everybody to share their [56:48] opinions. If there were any suggestions [56:50] those were addressed and there was just [56:52] one single command that we asked rather [56:54] than expecting CI to response to you. [56:57] You just run this command and then it [57:00] will respond to. Yes, there were some [57:02] instances where it was frustrating uh [57:05] because if there is another server [57:07] running on another terminal, it would [57:10] create a conflict and sometimes drop the [57:12] database. But it doesn't happen every [57:13] time you run the test. Now mostly I I [57:16] don't encounter this issue at all. [57:18] That's a signal uh that you need to fix [57:21] something rather than just replacing the [57:22] whole process, [57:24] >> you know. So that happened initially [57:26] tests were slow and with the time [57:29] developers were motivated I think we [57:32] wouldn't have this time without moving [57:35] the signals test to local if we were [57:36] just doing onci so now our tests are [57:39] faster we improved our coverage and when [57:42] I joined donbox the coverage stayed I [57:45] think at 86% for a while but I saw the [57:50] spike to 100% very quickly within I [57:52] think 6 to 8 months when I shifted it [57:54] from CI to local. So that's like a [57:57] positive thing, but it came with some [57:58] cost, you know. [58:00] >> Yeah. [58:01] >> Tell me more about this one shell [58:02] command. So you mentioned is it just [58:04] running a bunch of these different tools [58:06] at that point or what what else is in [58:08] there? AC you mentioned breakman and [58:11] rubocop and things like that. Is it is [58:12] is it running all those types of things? [58:14] Is it kind of is it similar to the bin [58:16] CI script but just like some other [58:18] >> exe? What happened is that we have that [58:21] very creative name of integrated box and [58:25] what happened is we have 12 different [58:28] steps [58:30] before you share your code. So [58:34] run rubocop they fix the things for you. [58:36] You don't need to worry too much. We [58:38] don't have crazy requirements on [58:40] rubocop. We're running the the full [58:42] Omakazi rubocop version right now after [58:46] a lot of dispute on what you should run. [58:49] We run the test uh rig task. We add a [58:53] bunch of steps there and we keep [58:56] evolving this task but only when we [58:59] actually have a requirement for that. We [59:01] don't spend too much time on that to [59:04] create the first version. It takes some [59:06] time to list the things you want to do, [59:09] but it's a Rick task. Very [59:11] straightforward, [59:12] not fancy. The things first thing you do [59:15] here is requiring the GitHub app [59:20] installed in your computer. It could be [59:22] the CLI because we do the the [59:25] authentication to add a mark on the P [59:28] request that someone run this and we use [59:31] this as a requirement for the for the [59:33] merge. For the process of doing that, I [59:37] presented to the engineering managers of [59:39] this idea [59:41] and my main argument is people are [59:46] losing track of why you write automated [59:50] tests. Some developers are getting way [59:53] too close to see the tests [59:56] as a check mark for someone else [59:59] checklist to run this test again in your [60:03] machine. It reminds you that you are the [60:08] responsible one for that not somebody [60:11] else. [60:12] And what I have to do was [60:18] first [60:20] convince the engineering managers [60:22] of having faith and try to see what [60:26] happens. [60:28] But again, the test runs takes one hour. [60:31] It's really really hard to convince [60:33] someone of doing that. And I worked with [60:37] the company management of requesting [60:40] time for them to do that. So if you're [60:44] not there yet in the my [60:47] test which takes one hour to run, it's [60:50] easier if you get to this point is [60:53] doable. You just need to commit [60:56] yourself. No, [60:57] >> I appreciate that perspective. So, I [60:59] want to pivot the conversation a little [61:01] bit. And one of the things I've been [61:02] thinking about was in our prep [61:04] conversation, I think Hussein, you [61:05] mentioned that you're in the process of [61:08] moving parts of donor box from slim [61:12] templates back to using ERB. Uh so why [61:17] out of curiosity, what what's kind of [61:19] motivating that to start off and then [61:21] maybe like what's that process look like [61:23] over on your end? Yeah. So this is also [61:26] uh our kind of effort to go with pure [61:31] rails defaults. Also we started seeing [61:34] issues with slim files uh related to [61:36] formatting and everything. So we want [61:38] our new developers or when we scale our [61:41] development team to make it easier for [61:43] them to just onboard them with pure [61:47] rails defaults easy to convey that this [61:50] is this is how we work. So slim files [61:53] was just adding another complexity and [61:55] it was also slower than ERB. So there [61:58] was no point of uh having this extra [62:01] pretty layer on top of what we already [62:04] have with Rails. That was a kind of main [62:06] thought behind it in my opinion. AC do [62:09] you want to add anything into it? [62:11] >> It also was an effort from the [62:14] engineering department perspective. [62:18] Yeah. We're not saying that you should [62:20] commit to raise the f as a rhetorical [62:24] commitment, [62:26] but [62:27] this is our efforts for making this [62:30] commitment more tangible for everyone. [62:33] If you need to change a link file, [62:35] convert that to ERB first and at some [62:39] point we could finally remove this ling [62:43] gem. Again, it's not about this ling [62:46] itself. is [clears throat] that on [62:48] donoxy wasn't necessary. were around [62:51] when they just started using slim or was [62:53] that just a decision that was made very [62:54] early on and like well that was was it [62:56] just a choice like well that's what the [62:58] cool kids are doing so I'll just use [62:59] slim because that seems to be what my [63:01] peers are using or I think about like [63:03] that whole HTML templating abstractions [63:06] that we've we've gone through it back [63:08] and forth a couple times and I was [63:09] always a little personally a little like [63:11] I don't what are we doing here because I [63:12] always felt like it was easy for for me [63:14] as someone that hires software engineers [63:16] and front-end developers in particular [63:17] what we used to call front-end [63:18] developers before as much of the [63:21] JavaScript side of things came into [63:22] play. But people that could take designs [63:24] and make HTML and CSS out of that and [63:26] like great well now I let's also then [63:28] introduce now you need to learn this [63:31] Ruby code to generate the thing that you [63:32] already know how to work with. It always [63:33] felt a little like I felt like it was a [63:35] developer solving a problem of them not [63:37] wanting to write HTML even though you [63:39] needed to understand HTML to to use it. [63:41] So it just felt like an extra layer of [63:43] not saying complexity but it maybe it [63:45] look cleaner in your when you're looking [63:47] at it. I guess some people might be [63:49] listening be like, well, it's slower, [63:50] but is it maybe it actually looks [63:53] cleaner to them? I I don't know. It's a [63:55] some guy can argue that it's easier. It [63:58] makes it easier for a developer to write [64:00] it and understand it. But at the same [64:02] point, I would argue that we need to [64:05] replicate everything that we show and [64:07] render on on the browser in our code [64:10] editor. So it's easier to see the [64:12] structure with all the indentation and [64:13] everything rather than configuring [64:16] different settings for the slim in your [64:18] code editor to not remove the spaces at [64:21] the end and all that stuff. So yeah, [64:23] that was kind of another pain that you [64:25] need to go through. [64:26] >> Is that not something that you can just [64:28] run through some sort of like conversion [64:31] script and just call it a day? The [64:33] workflow right now is just if you're [64:35] going to have to make changes in your [64:36] slim file, let's convert it first and [64:38] then make sure things will look okay. [64:40] You do you feel like the automatic [64:43] conversion through would be kind of a [64:44] little too risky? Yeah, I would say that [64:47] because some parts of the app were very [64:49] delicate and we had to involve QA or at [64:52] least a developer to test it before [64:55] doing anything or trusting too much on [64:57] online convers converter [clears throat] [64:58] tools or uh there are some gems as well [65:01] to convert the slim files to ERP but [65:04] sometimes they make mistakes. It was [65:06] always a good idea to test it after you [65:09] convert the slip file. Has your team [65:11] started uh playing with any of uh [65:14] Marco's [65:16] um Marco Roth's like herb at all or [65:19] anything to [65:20] >> Yes. Yes. I I've been using it. I really [65:22] really appreciate his efforts on herb. [65:25] It's amazing. Uh I really enjoyed his [65:27] talk last year in Amsterdam. He's doing [65:30] amazing job at that. [65:32] >> Thank you Marco on behalf of all of us. [65:35] >> Yes. I'm curious about like this is like [65:37] a particular project that your team's [65:38] working on a migration and so it's it's [65:40] happening over a period of time and [65:42] presumably it'll get done. How does your [65:43] team stay motivated to keep working on [65:45] that? Do you like is there someone that [65:47] kind of owns that on your team and like [65:48] reports like hey we're 60% of the way [65:50] there or what's that kind of like look [65:52] like from a team culture perspective of [65:55] like how you champion these types of [65:56] things? [65:58] >> This particular migration I wanted to [66:00] use as a team building process. So I [66:05] don't want to affect the things you want [66:07] to you need to deliver but I do want [66:10] that to be a collective ownership [66:14] of what you do. So what I did [66:19] is requesting the management to add this [66:23] as a departmental PR for the rest of the [66:25] company. the information that the team [66:28] the the department as a whole is looking [66:31] for doing this as a internal requirement [66:36] for the team. It gives them the piece of [66:39] mind that they're not doing something [66:41] else not connected to the company. [66:44] So you're doing your job by doing this. [66:46] We announce it to the whole company like [66:48] yeah the team will reserve some of their [66:52] time to do this. If they need to change [66:54] something on the link file they convert [66:57] to ERB first. The purpose of doing that [67:01] is remember that we are one team [67:04] not different people working in their [67:07] own home because Dirtbox has this very [67:11] specific characteristic of being a [67:14] remote first company. We have people [67:17] from [67:19] California [67:21] up to Cambodia [67:23] and very other layers in between and [67:27] very other different time zones in the [67:28] between being Pakistan the most famous [67:32] one of them or most populated at this [67:35] point. The point of those those link [67:38] files was is more about the guys we [67:41] working together here and the way of [67:43] motivating people is [67:46] a weekly reminder of how many link files [67:50] were migrated the last week how many [67:53] would change without the migration. I'm [67:55] not trying to build that as a a knife on [67:59] people's neck but uh as a if you need [68:03] time let me know. It's pretty [68:05] interesting how it worked because in the [68:09] couple in first weeks when I started to [68:12] post that on the dev's channel would [68:14] have people that needs to change his own [68:16] files without converting them and [68:21] I reach out try to help after a couple [68:25] of weeks [68:28] this behavior was gone and people do [68:31] bought the the idea of let's [68:35] update them and that's it. That's the [68:38] thing with behavior like once you change [68:42] you don't even remember why you're not [68:43] doing that in the first place. [68:46] >> It's useful p perspective there. I think [68:48] about how teams try to [68:51] like all right we're going to make this [68:52] we're going to start working through [68:53] this migration or you kind of will it in [68:56] way or like sometimes like well that [68:58] team require like an owner or someone [68:59] championing it like and maybe in this [69:01] scenario like if you're personally [69:02] reaching out to people and be like hey I [69:04] noticed you did something and like do [69:05] you have any I'm happy to help provide a [69:07] little bit of a see if there's something [69:09] we can do to make that easier in the [69:11] future you know what have you and then [69:12] like that starts to remind people of [69:14] that that process and stuff like that I [69:16] think it's helpful to hear like these [69:17] different perspectives how teams [69:18] approach these types of things because I [69:20] would imagine like if an application's [69:21] been around for a while like seems like [69:23] a a good good tangible example we can [69:25] wrap our head around migrating back to [69:27] ERB or maybe just to ERB maybe there [69:29] wasn't an ERB before maybe it's not a [69:31] migration back but but a lot of us deal [69:33] with a lot of interesting challenges on [69:35] the front end and our Rails apps because [69:36] of just the generational things of when [69:40] what was the new hot cool thing of the [69:42] JavaScript framework at the time or you [69:44] built one part of thing and you're like [69:45] we're going to build some new thing [69:46] we're going to maybe use a new [69:47] JavaScript framework for this thing to [69:49] experiment. Maybe that's going to make [69:50] things better. And the Rails doesn't [69:52] always get those it it's changed its [69:54] minds, right? And so we we all can't [69:56] keep up maybe as fast as uh as Rails [69:59] would like us to at times or or at least [70:01] it was like this is where we're going. [70:02] And so not everybody's rebuilding their [70:05] their app every couple years like 37 [70:07] signals and they take a new approach and [70:09] we get to benefit from seeing that in [70:12] how they're approaching things. I'm [70:14] curious like when you're looking and [70:17] maybe the question for you Hussein [70:18] initially is uh the more the donor boxes [70:20] application can be more Rails like or [70:23] following Rails conventions does that [70:25] make it easier for you to bring on new [70:27] engineers into your teams to be like and [70:29] look at other comparison projects be [70:31] like look let's see how they're doing [70:33] that as there's open source projects or [70:35] things like that and or other projects [70:36] you've worked on to to to weigh that up. [70:40] Yeah, hiring is also a kind of [70:43] underrated feature of Rails. It makes a [70:46] lot easier for you to uh set the [70:48] expectations because the guy with like [70:51] three, four or two or three years of [70:53] experience, you already know that he [70:54] knows the basics uh conventions and [70:57] everything. So, and onboard Yeah. Yeah. [71:00] [laughter] [71:00] And onboarding goes very smoothly like [71:03] in terms of you just need to pass some [71:05] type of u documentation how we work. We [71:08] do have like our own document. It's like [71:10] a I think six to eight minutes read that [71:14] you can go through that this is how we [71:16] work and considering that it only takes [71:20] like one or two weeks for a person to [71:23] get an overall idea how donor box how [71:25] developers are using rails and donobox [71:28] you know so for me at least it it it [71:31] made so much easier to onboard a new [71:33] person to the team they already have the [71:36] basics because I have heard stories from [71:38] different people using different [71:40] frameworks that it takes a lot of time [71:42] because take an example of NodeJS. [71:44] Everybody has their own way of building [71:47] a project. So for I think first few [71:50] weeks you will be just juggling around [71:53] to get an idea the founder of this [71:55] project how he was thinking why he did [71:57] this configuration particularly we don't [71:59] have this issue on rails we should be [72:01] thankful for that you know [72:03] >> very similar to what AC was describing [72:05] in the the Java land before he got [72:07] introduced to rails and we just have to [72:09] make all these decisions like or your [72:11] app needs three log logging libraries [72:14] because of the dependencies they they [72:17] depend on different logging [72:18] tooling and rails has removed a lot of [72:22] those decisions for us. Something else [72:23] that I wanted to make sure you know with [72:25] and being mindful of time is you know [72:28] one of the things we had talked about [72:29] prior to this conversation was I think [72:31] there was something related to was it [72:33] like search and filter pages [72:35] >> in your in your application and was that [72:39] >> yes [72:39] >> tell us more about this like I think [72:41] what what were you dealing what sort of [72:43] challenges were you having with that [72:44] over at donor box? uh data box requires [72:47] a lot of filtering for da segmentation. [72:51] The people working on that particular [72:54] feature, they're trying to create the [72:59] the filters and creating some ways of [73:04] preventing a a user [73:08] to creating possible filters. like we [73:11] have a filter for the amount of [73:13] donations you receive it. The amount of [73:16] donations a donor made. So I want to see [73:21] all the donors that donated more than [73:24] five donations [73:26] and less than two. This kind of logical [73:30] >> complex. [73:31] >> Yeah. You know, but but do you see that [73:33] there is a logical failure here? Like [73:35] more than five and less than two. [73:38] [clears throat] [73:38] >> Yep. You need some way for preventing [73:41] this [73:44] mistakes because someone you you don't [73:47] necessarily need trying to doing [73:49] something wrong. Maybe you're just [73:51] writing things down. You rule out your [73:54] mouse over the the increment decrement [73:57] stuff. So to avoid those problems, we're [74:00] trying to create a self validation of [74:05] the futures. The thing is the way it was [74:08] created, [74:10] we're trying to use the flash message to [74:14] explain what went wrong. [74:15] >> Oh, I see. [74:16] >> To do that, you rise an error. [74:20] So essentially they are controlling the [74:23] flow of the user by rising exceptions. I [74:27] tried a couple of times sharing rails [74:31] guides, sharing the thing about the [74:34] active model and it didn't work. So what [74:38] I did is [74:40] I would like to ask you for three hours [74:44] of [clears throat] your time today. I [74:46] sit down with you and try to show you an [74:49] alternative way of implementing the same [74:53] set of filters. [74:56] But I need you to do a leap of faith [75:01] because it will sounds unbelievable [75:04] tedious and you have the urge to keeping [75:07] asking me but it's already working why [75:09] why you're doing that and I was like [75:11] yeah it's not only working it's deploy [75:13] it you know like you already delivered [75:16] the value I just want to show you there [75:19] is another way I never managed to do [75:22] that in a huge audience every time it [75:25] worked for me is was when I sit down [75:28] with the person and I say I got what he [75:31] did. Let me show you another way of [75:34] doing that. I'm not saying it's better. [75:36] I'm not saying I like more mine. I'm [75:39] just trying to say that this is [75:40] different of what you did. I want you to [75:42] judge the merger button is yours. You do [75:45] whatever you want with that. But I need [75:47] to give you this three hours. [75:51] So I sit down deleted the whole code [75:54] from the controllers. [75:57] It kind of calls the attention when you [75:59] do that. You like people wait a minute. [76:03] And what I did was implementing the [76:06] whole filters as one active method with [76:10] a very creative name of search donors [76:13] because you know like I'm very creative [76:15] of those names. So given your searching [76:18] donors that's the name of the filter. [76:21] plain Ruby class and I created a method. [76:27] I don't remember exactly what was the [76:29] name of the method. I don't know if it's [76:30] all operator or something like that. But [76:32] the thing is I added the first very very [76:36] simple method of [76:39] filtering by dates and you add the first [76:42] date and the last date of the donation [76:44] and you filter your donors by that. [76:47] So it feels kind of awkward in the [76:50] beginning of the exercise [76:52] but I asked them how do you implement [76:54] the validation of this and oh you can [76:58] throw an arrow yada yada and I was yeah [77:00] yeah you also have this rails feature [77:03] called active model active model is [77:07] every nice stuff you have on active [77:09] record without the database so let's [77:12] start with the validations so I create a [77:14] validation using [77:17] And when they saw [77:20] the errors appears in the form itself [77:24] without requiring them to do anything, [77:28] it landed different, you know, like it's [77:31] like, oh, wait a minute. [77:33] How did that do anything? I just follow [77:36] the results, [77:39] you know, like it's way less effort [77:44] and I keep doing that until I [77:45] implemented all the filters. [77:48] The thing is, and it took me three [77:50] hours. And what happened is during this [77:53] process, I saw people having the same [77:57] because it's two guys, they have the [78:00] same feeling that people keep having [78:02] when I do that this way. [78:05] One of them get back to me and said, [78:08] "You're right." I was actually through [78:11] the whole process thinking that why are [78:13] you doing that? It's already working. [78:15] But you gave me this feeling that if [78:18] it's taking too long or it's too hard, [78:21] maybe I'm doing something wrong. [78:23] >> Yeah. [78:24] >> Because it's not possible that in more [78:26] than 20 years of Rails, I am the first [78:29] person with this problem. [78:33] So maybe Ry has already thought about [78:35] that [78:37] you know this feeling [78:39] >> the power that uh active model and [78:41] active sport has it's it's super [78:43] underrated. So yeah you can do a lot of [78:46] stuff using those two modules. [78:49] >> I think that's probably a common thing [78:51] that a lot of teams struggle with is [78:53] like you might have there's already a [78:55] lot of complexity maybe in a controller [78:58] action somewhere. Um, I think search is [79:00] like a really good example of that where [79:02] probably the one of the most common [79:03] things I' I've seen hundreds and [79:04] hundreds of Ruby on Rails applications [79:06] and those really big controller method [79:08] and sometimes people abstract that a [79:10] little bit and and come up with some way [79:12] to approach that and it's not always [79:14] super consistent but uh but seems to be [79:17] like a little tricky. It ends up they [79:18] can be a little fickle and complicated [79:20] to debug like why am I getting this [79:21] weird issue or uh you end up with a lot [79:24] of weird like nil messages showing up. [79:26] One of the probably most common things I [79:28] see when it comes to like uh weird [79:30] things that people like, well, it's [79:31] probably an issue, but maybe not an [79:32] issue. I'm not going to look into that [79:33] particular error today, but uh we'll [79:36] look at that at some point. It's [79:37] probably it could it could be like [79:38] someone just crawling the website, you [79:39] know, and hitting a search page or [79:41] something and then it triggers some [79:42] weird error messages because uh it's a [79:44] bot or something. But knowing when to [79:46] leverage Rails for these types of things [79:48] and if the code is already seemingly [79:49] working and has already, as you said, [79:51] provided the value, when do you come [79:52] back and revisit these things? And so I [79:54] I appreciate donor box has like a an [79:57] approach like in your culture there of [79:59] let's go back and revisit these things [80:00] like we it's okay we don't have to be [80:02] scared to redde this and like [80:04] reimplement this in three hours that [80:06] could be you know and that could unlock [80:08] things in people's minds. I'm curious, [80:11] you know, I think maybe Hussein was [80:13] mentioning like that you have like a [80:14] thing that takes like six to eight [80:15] minutes to read. What's and how your [80:17] donor box approaches are these like your [80:20] team's engineering principles and like [80:21] when to reach for things in like does [80:23] Rails even mention to that or [80:26] >> Yes, basically it handles everything uh [80:28] for a new hire to let them [80:31] >> Yeah. onboarding guide. It mentions the [80:33] commands that you need to run basically [80:35] there those are all already mentioned on [80:37] the GitHub repo as well. But overall [80:39] idea how the teams are structured, how [80:42] the code works, what are the main parts [80:43] of the app, some tips and tricks for [80:46] adding new columns in the database uh [80:48] like how to do any kind of optimizations [80:50] on database levels. A lot of those and [80:53] we kind of try to evolve this document [80:55] uh with the time. So we revisit it [80:57] sometimes even uh it's been like two [80:59] years one not sure but I I still revisit [81:03] the same document whenever I'm you know [81:05] having any issues. Maybe there's a new [81:07] developer who raised us some questions. [81:09] I try to see if if it has been answered [81:12] already in our onboarding guide or not [81:14] and then just I just link that share the [81:16] link to that part of the document to [81:18] this person. So this way we keep [81:20] evolving this document and it makes [81:22] easier for us to onboard a new guy like [81:24] frictional way. I'm curious like given [81:29] that we're having this conversation in [81:31] it's August 2026 and the uh this episode [81:37] will probably come out like in the next [81:38] month or so, but I I'm curious about [81:41] does this tooling also that you've been [81:44] investing in? Are you bringing on agents [81:47] into your development workflow? Are you [81:48] using any LM AI agentic development in [81:51] your like locally in your to help you [81:54] and your engineers work on this? And how [81:56] is these tooling and the onboarding [81:58] documentation part of that? [82:00] >> Yeah, so we we we have put similar [82:02] efforts into the agent side as well. So [82:04] we do have cloud and cursor. Some people [82:07] prefer cursor uh in their development [82:09] flow and some some people are using [82:11] cloud. And then we have thanks to AC we [82:14] added agent empty and another couple of [82:17] files for overall view for agent before [82:21] writing anything and how to comment how [82:23] to write comment messages and all that [82:25] stuff. Um and thanks to our orc [82:29] experience team we do have our internal [82:30] jam for adding design components which [82:33] is powered by MCP server. So your agent [82:35] can ask questions how to add a new [82:38] button, how to add a model, uh you know [82:40] these type of questions and then it it [82:42] will make easier for you to not spend [82:45] too much time on the front end and just [82:47] solve the main problem on the back end [82:49] you know. So with the design components [82:51] it it has made huge difference in saving [82:53] so much time and making all the products [82:57] of the same nature. Previously they were [83:00] like a mixture of pages like using old [83:02] design and new design and now it feels [83:05] very easy for to add a new feature. [83:08] Right now I'm my team is at the level [83:10] that when we are going through Figma [83:12] design we are just thinking and modeling [83:15] in our mind okay I need to add this [83:17] model this connection association and [83:19] that's it you know and it doesn't take [83:22] much time to develop as well. So because [83:25] you have design components in place, you [83:27] just need to put them in right place and [83:29] just focus on the backend strategy and [83:31] structure and you're done. Do you have [83:33] much tooling that or come up with any [83:36] ways to like say like that you're to to [83:39] feel pretty confident that someone is uh [83:42] that like maybe not someone but the [83:44] agent or the PR is using the the ideal [83:48] path going forward because maybe they're [83:50] you know a lot of code bases have three [83:52] or four equally plausible historical [83:55] ways of approaching similar things and [83:56] you might be like as a team being like [83:58] we're going to start moving more in this [83:59] direction but the agents might be like [84:02] getting to source just a lot of things. [84:03] How are you wrestling with that right [84:05] now? [84:05] >> Personally, I would uh prefer your first [84:10] thing as a readable code and it should [84:12] be domain driven driven way because it's [84:15] a legacy project and most of the code is [84:18] built in this way. So those are the two [84:20] things I want my agent or anybody's [84:22] agent to focus on that code should be [84:25] readable and it should follow domain [84:27] driven design pattern. But there there [84:30] could be different ways. The ways that [84:32] we can't compromise they are already [84:34] mentioned in MD files. So that will give [84:37] you if you are asking your agent to do a [84:39] quick quick review. It will ask you okay [84:42] this is what is mentioned on your MD [84:44] file. Do you want still want me to write [84:46] it this way or we should rethink about [84:49] it? You know that's like a soft [84:50] warnings. [84:51] >> Sure. Sure. [84:52] >> Yeah. What happened is we're documenting [84:55] the things we don't want to follow [84:57] anymore for the agents. [84:59] >> Is that just something that you're kind [85:01] of as things pop up you're like oh yeah [85:03] we should add it to the list of things [85:05] to stop following in that direction. [85:07] >> Who's deciding that? Uh [85:10] >> maybe that's a complicated question but [85:12] >> I don't know if everyone will have this [85:14] view that it is collective [85:17] because there is one particular name [85:19] that keep popping up every time you see [85:22] the commit messages. [85:27] But what happened is that [85:30] I reach out for people before changing [85:33] this documents. Instead of creating a [85:35] big [85:37] committee of what are the the decisions [85:40] we should take for this particular [85:41] thing, I reach out for the people that [85:43] is actually working on that and request [85:46] for comments like we're doing this way [85:50] in this particular side of the of the [85:52] code. We have this other way of doing [85:55] it. What do you think about requesting [85:58] the LLM agents to every time we stump on [86:02] that? Help us to move to this other [86:05] direction we're looking for. I see the [86:08] LLM agents as a amplifier of yourself. [86:13] So it will amplify your knowledge and [86:16] also amplifies your lack of knowledge. [86:18] What we're trying to do is having a [86:20] place for the lack of knowledge or lack [86:24] of context of one individual [86:28] has a backup in an agent document. So [86:33] that's the overall idea on how this is [86:36] being built. It's helpful to hear how [86:38] different teams are navigating that [86:40] right now because I I I hear you on the [86:42] uh it's a collective thing and we're all [86:44] participating but that you know maybe [86:46] there is one person that's being a [86:47] little bit more vocal or being like hey [86:49] what do you think and you're it sounds [86:50] like you have a very you're inviting [86:53] people you're not you're not uh you're [86:55] not dictating this is how we're going to [86:57] do our things now this is AC's way [86:59] everybody follow along maybe I can have [87:01] a side conversation with Hussein later [87:03] and he can tell me otherwise but I [87:04] assume that take that take that for uh [87:06] we'll just trust that That's that's [87:07] actually the case there. But uh I [87:11] believe you there. But uh you know I'm [87:12] thinking about as team do you have any [87:14] recommendations for teams right now that [87:16] are that haven't quite figured out is it [87:20] just as simple as like hey in your [87:22] agents MDs like there's some you're [87:24] annotate annotating certain types of [87:26] patterns like hey we've done it like [87:27] this and so when the agents are looking [87:29] at things here's how to avoid [87:32] here at least to ask the soft question [87:34] like are you sure you want us to maybe [87:36] as a better tangible example do you have [87:39] something in there for like the slim and [87:40] erb thing and they're If if a slim file [87:42] is being modified, raise the question in [87:45] the chat with like maybe we should [87:48] should we convert first. [87:51] The sad story here is that we didn't [87:54] have quad when implementing we're moving [87:57] from Z files to GRB which kind of sad [88:01] because it helps so much and what you're [88:06] doing here on the agents is trying to [88:09] describe the pieces of the puzzle in [88:13] their own MD files. I do have a big [88:16] agent MD and for those who use claude, [88:20] you can always create a aim symbolic [88:24] link which is pretty funny when you need [88:27] to update the your cla MD file because [88:30] air quotes because claude tries to [88:34] update the file. He said, "Oh, wait a [88:36] minute. This file doesn't exist. It's a [88:38] symbolic link. Yeah, sorry about that." [88:41] But for reading, he didn't notice. So [88:45] what I do is having in the agent MD file [88:48] the core things that you need to to use [88:52] in every question you make about the the [88:55] the codebase itself. So how do I run a [88:58] test? How do I run the the servers? In [89:02] our case it's pretty boring because it's [89:04] a rails test rail server. So yeah, you [89:07] got it. [89:09] And we keep the the text tag. So it [89:12] helps on the building the context for [89:14] the LLMs. We added the donbox components [89:18] gems and how you update them because it [89:22] helps when they're creating something [89:24] that requires UX. In the same file we [89:28] have the application structure like how [89:32] our directories looks like and what is [89:35] there [89:36] >> is there a lot of interesting things [89:38] that are different than a I'm going say [89:40] a typical Rails app that you've done [89:42] differently in your Rails directory [89:44] structure anything for our listeners [89:46] that's stands out kind of interesting [89:48] >> for example we don't have services [89:50] folder you know so that's one example we [89:54] want every class that you want to [89:56] required it can be if it's associated [89:58] with the business logic it can stay in [90:00] the models directory so no need to add [90:02] another directory uh in app folder you [90:06] know so [90:07] >> do you name space those at all or [90:09] >> yeah we do but it it is still in the [90:11] models directory so it's just uh [90:14] conventional stuff like app models views [90:17] helpers and those defaults [90:20] >> anything interesting in your lib [90:22] directory or [90:23] >> we have a lot of different integrations [90:26] for example uh Cloudflare because [90:29] Cloudflare maintains the cached forms [90:33] and hosted pages. So every time user [90:38] uh in our case an organization our dire [90:41] customers they update their form we need [90:44] to invalidate the cache. We use the lib [90:47] as a wrapper around those [90:51] integrations. [90:53] So we literally have a code like [90:55] Cloudflare cache purge [91:00] this form right here this campaign form [91:04] and the translations happens from [91:08] donabox [91:09] language to cloudflare language happens [91:12] in this call of I tell you the name of [91:16] the form [91:18] and by this this name the cloudflare [91:22] cache purge [91:23] knows how to get that URL. The cool [91:26] thing about this is when you're reading [91:27] the code inside the box where there's [91:30] this code. It sounds that that [91:32] Cloudflare is just another library that [91:34] you're using, you know, like when you [91:37] have the update of the form, you have a [91:39] Cloudflare cache version with the name [91:42] of the of that form. So, we usually call [91:46] it a slug because you have a [91:48] dropbox.orglug org/lug [91:50] of the hosted page or embed is lug to [91:54] get the embedded version. The cloudflare [91:58] details [92:00] didn't leak inside of the box code and [92:04] we have a couple of examples like this. [92:06] The other one is pleo because we send [92:08] text message for donations and we don't [92:12] send text message for requesting [92:14] donations. We do have this feature of [92:19] your text donor box asking for a form. [92:24] If you already made animation in that [92:25] form, you can make another one through [92:27] text. But it's always a donor initiated [92:31] process not a [92:32] >> I see any other fun interesting things [92:34] in your in your rails directory [92:36] structure there or [92:37] >> also we we I wanted to mention we do [92:39] have one thing uh that we are kind of [92:42] demotivating agents to do is adding any [92:45] new CSS files. So the whole point of [92:49] having design components is to [92:53] not write or at least uh write minimal [92:57] CSS and that CSS should live in the same [93:00] HTML file. So there shouldn't be like a [93:02] directory or pathway of all the CSS [93:05] files. So if there's something missing [93:07] in design components then this is a good [93:09] point to reach out to the team that I [93:12] I'm using this design component but it's [93:14] not supporting and designer built it [93:16] this way. So it's just like a 15 minutes [93:19] talk with you all three and then uh you [93:23] make a decision okay we need to evolve [93:24] our design component or we need to ditch [93:26] this new requirement. [93:28] >> That's interesting. Is that typically [93:29] coming up when you're working on some [93:30] new feature request or something? You're [93:32] like like how do we [93:33] >> mostly new features [93:35] >> you mentioned you're using Jira out of [93:36] curiosity when the when a developer gets [93:38] a ticket the engineering team has a [93:41] ticket is it already kind of determined [93:44] like you should be pointing to this in [93:45] our design library or is that something [93:48] the developer is figuring out as they [93:50] start working on the ticket. So [93:51] basically in the start of the task uh we [93:54] do like a R&D phase where designer would [93:58] do like a schedule couple of meetings to [94:01] get an overall idea what we want to [94:02] achieve what we need to skim before we [94:05] actually start the design process and [94:08] after the design process is done I [94:10] involve one developer who is motivated [94:13] to work on that feature or product and [94:15] then we would set up a meeting and then [94:18] ask any questions. So this developer [94:20] would take like two or three hours to go [94:22] through the design see if there are any [94:25] limitations if this is doable if we need [94:28] to skim seeing the deadline and [94:30] everything and put it in perspective uh [94:33] if it will be done uh before that if all [94:35] good he has answers to all his questions [94:38] then I would do another meeting with him [94:40] just to deise a development plan it's [94:43] not a long meeting like 3 hours but it [94:45] it's like a 30 minutes meeting we would [94:47] just quickly go through [94:49] This is the migration that we need to [94:50] add like overall idea because engineers [94:52] are also like mid to senior level [94:54] engineers at Donov. So you just need to [94:57] give an overall idea that this is my [94:59] approach. If you have ideas then we [95:02] share it with each other and get to [95:04] work. [95:04] >> I see. [95:05] >> Yeah. [95:06] >> And how much do you feel like you how [95:08] far away do you feel like you are right [95:09] now? Maybe we think about Hussein on [95:12] your on your side where you have your [95:15] linear or your Jira ticket or whatever [95:16] your team is using and be able to be [95:18] like all right a developer takes that [95:21] pops it into their cloud or cursor and [95:23] be like all right implement this and not [95:26] needing to do a lot of code review [95:29] themselves like how far away do you feel [95:30] like you are right now with that [95:32] culturally at the moment or personally [95:34] maybe as well? Uh no I think uh we are [95:38] blessed in in this regard that [95:40] developers at Donboxer they are not [95:42] trusting too much uh on their agents [95:46] for now at least. So I'm not very uh [95:50] cautious about uh this particular issue. [95:53] Uh I do use it for code reviews and all [95:55] that stuff but I become hesitant just [95:58] sharing that review from claw to you [96:01] know uh the developer. I need to go [96:03] through it ask questions. why did you [96:05] post this and maybe update the MD file [96:09] so the next time it doesn't make that [96:11] mistake and it consider yeah it [96:14] considers the convention that we are [96:16] following or the direction that we are [96:18] following in terms of uh code quality or [96:20] code debt or anything like that. So yeah [96:22] I'm I'm I'm I'm not much worried about [96:26] uh this particular issue. Do you feel [96:28] like you're seeing a lot of throughput [96:32] improve like you're able to get through [96:33] more as a team at this point or is it I [96:36] feel like talking to different people on [96:38] their podcast and some people like oh [96:39] yeah we're like the number of PRs has [96:41] drastically increased and we have other [96:43] issues now as a result but uh now we [96:46] have more PRs to review or something [96:47] like that. Is that is that showing up [96:49] yet or [96:50] >> I think the biggest advantage for us [96:53] that we have got so far is because the [96:56] donor box uh project is very big. So it [96:59] has made and empowered you so much to [97:02] know what other teams are doing just by [97:04] asking AI rather than pinging other [97:07] devs. You can just have a conversation [97:09] with it how the payments are working, [97:10] how we are charging the invoices, how we [97:13] are you know scheduling everything. [97:15] those have empowered you to understand [97:18] how other team if if your code or [97:20] feature is related is touching some [97:23] parts of their code. So rather than [97:25] having a long meetings uh when AI [97:28] >> something I I I'm curious about with [97:30] that do you foresee like fast forward [97:32] this progression a couple years [97:35] regardless of whether or not we're going [97:36] to get to a point where we can just [97:38] assign an agent to task and be like [97:39] great and then ship start sends TED PRs [97:42] where you're not even looking at [97:43] anything. But if if your developers are [97:45] maybe spending a little less time in [97:47] those maybe longer meetings, do you see [97:49] there potentially be any interesting [97:51] things culturally where your team's not [97:54] checking in with each other more but [97:56] themselves and they're just asking the [97:58] AI for information on what their peers [97:59] are doing? And do you feel like that [98:00] could cause any interesting maybe not a [98:03] good or bad thing, but just more of like [98:04] something I'm I'm always curious about [98:06] when people are like I'm not I'm not [98:08] needing to ask my co-workers for their [98:11] input like how how have we done this in [98:12] the past because I can just ask Claude [98:14] or Curser or whatever. And if that those [98:17] conversations start to get reduced and I [98:20] feel like there's got to be some sort of [98:21] like interesting unexpected consequence [98:24] of that. Maybe not a bad or a good [98:25] thing, but it's just it'll be different. [98:27] Like maybe we don't ask maybe will it [98:29] feel like a cohesive team in the same [98:31] way or maybe it'll just it'll feel [98:33] different. [98:33] >> Yeah, I think we haven't gotten to that [98:35] point that AI would can answer every [98:38] question because there were so many [98:39] decisions that people made considering [98:42] the limitations at that time. So AI [98:44] doesn't have that information. It it can [98:46] go to the comment and PR and all that [98:48] stuff but not at the discussion what was [98:51] made. For example, why events had a [98:54] different way of uh you know using [98:56] rails? Why it was so much different? So [98:59] only a handful of people at Don Box know [99:01] this story. So AI can't so unless you [99:04] feed it into the MD file, AI can't know [99:09] that why it was built this way. So this [99:11] is kind of a contextual history. I think [99:13] you need to fade it and you can't trust [99:15] it too much with everything. So but you [99:17] can ask it like what this class do or [99:20] you know trivial stuff and which which [99:22] takes time for human to understand. [99:25] >> Any thoughts on that AC? [99:26] >> Yeah the the agents are are [99:31] very enhanced Google search. [99:35] So the same way you don't approach to a [99:40] to a colleague to ask for uh what is the [99:44] name exact of that validation for [99:46] numeric values that is positive and only [99:49] integers you just search for that [99:52] looking into the future questions like [99:55] okay where should I looking for some [99:58] information [100:00] you can start the conversation with the [100:01] AI and get the information you need to [100:05] actually have a conversation with a [100:06] human being and use that as a oh now I I [100:10] did my homework this is what I found can [100:14] you help me understanding what the hell [100:16] you did here why is this way what are [100:18] you trying to prevent because this is [100:21] are the information that the AI doesn't [100:22] have [100:24] >> is there anything that you're kind of [100:25] like you've you you've maybe for each of [100:28] you that you've yet to be able to [100:30] experiment with that you're would love [100:33] to be able to get an opportunity at some [100:34] point to like you're something like kind [100:35] of coming down the road or something [100:37] that's been introduced that you're like [100:38] well we haven't been able to start doing [100:39] that donor box but maybe at some point [100:41] it would be kind of I'd be really [100:43] interested to be able to have a chance [100:44] to play with that something in the in [100:46] the Ruby or Ruby on Rails ecosystem and [100:48] maybe we'll start with you and AC [100:50] >> uh right now we were using this this FA [100:54] editor and I really interested in the MD [100:59] editor that was released recently [101:03] as recently as possible. But you got the [101:06] point. It sounds interesting to check it [101:09] out. And it's a a move back to the well [101:13] if if you're going to write JavaScript, [101:15] just run JavaScript. Don't try to [101:17] recreate JavaScript in another language [101:19] behavior that the tricks didn't have. It [101:22] was a that TypeScript stuff. I would [101:25] love to have the opportunity to test [101:27] that because it sounds very interesting. [101:30] >> That's Lexi. [101:32] >> Yeah. Yeah. What about you, Hussein? [101:35] >> I think I'm excited to use Solid Q in [101:38] our code base very much, especially the [101:41] job continuations. [101:43] It's been a while since Fizzy released [101:45] their code base online. So, it's just so [101:48] much fun for me to casually read the [101:51] codebase. And another thing I noticed [101:54] was how they are using the identity. So, [101:58] identity is basically is equal to email. [102:01] And if you do identity current to your [102:03] user, you know, and if you do identity [102:05] do user, it tells you your role. [102:08] Basically, this this was very [102:09] unconventional for me to absorb and [102:11] digest because coming from the [102:13] background using device and session [102:15] controllers and all that stuff and I [102:18] think this can play very well at Donald [102:21] terms of logging in can be done from [102:24] different roles. You can be a [102:25] fundraiser, you can be or get me on [102:28] behalf of organization, you can be a [102:29] donor. Implementing this would uh would [102:32] be fun for me and I would raise my hand [102:35] to do this. Out of curiosity, I I didn't [102:39] want to go too far down the weeds there [102:40] into your platform there, but if if like [102:43] a single person like let's say Robbie [102:46] Russellgmail.com [102:47] is a is a donor to one organization [102:52] um but I've donated to like 10 different [102:54] organizations, 10 different customers [102:55] there. Is that linked at all in your [102:57] system or are those treated as separate [102:59] donors per customer type of scenario? [103:02] >> I think you're going to laugh for the in [103:04] the for my answer, but the answer is [103:06] yes. [103:07] The thing [laughter] [103:09] >> the thing is [103:12] a from a don perspective a donor is a [103:15] contact of a person [103:19] and that organization agenda [103:23] the person itself is another thing [103:26] >> I see [103:27] >> if donated to 10 different organizations [103:30] you have 10 different donors to [103:32] represent yourself [103:35] but the cool thing is You can manage all [103:38] those donations through your donor [103:43] credentials. [103:44] >> So you can see all of them, [103:47] but they can only see their own context [103:50] for you. [103:51] >> Yeah. Yeah. Okay. And there's no Okay, [103:53] that that's helpful. I was kind of [103:55] curious about that, but Hussein, do you [103:57] feel like if if you there's a there's a [103:59] scenario where you might look to migrate [104:03] away from something like a device? [104:05] Um we're not using device ad. I just [104:08] gave an example because typically on [104:11] device or any you try to use everything [104:15] in the user model like authentication [104:17] pairs and everything but fizzy gave a [104:20] very different uh perspective on how [104:23] they solve this issue. Yeah. [104:25] >> Using identity you know so that was [104:27] interesting. It's just like an analogy [104:31] of entering a building. There's a [104:33] security guy. He's he just wants to [104:34] check your ID. Doesn't care about who [104:36] you are, where you work, and he just let [104:39] you in. And then there's different types [104:41] of areas in the building that you can go [104:43] in, you know. [104:44] >> Yeah. All right. So, before I let you [104:46] both go, here's a question that [104:47] hopefully is a little less complicated. [104:49] Uh I'll start with you, Hussein. Uh is [104:52] there a programming or software book [104:53] that you find yourself recommending to [104:55] other engineers? [104:58] I would just recommend the old bible for [105:01] rails developers is uh agile web [105:04] development. Uh that's a good one. Uh I [105:07] I loved eloquent ruby and practical [105:11] objectoriented programming by Sandy I [105:14] think. [105:14] >> Sandy. Great. What about you AC? [105:17] >> Well, Hussein stole all my [105:20] recommendations. So plus one. [105:22] [clears throat] [105:24] Uh I goes with the the pragmatic [105:27] programmers from journeyman to master. [105:29] It's not a a new book. It's kind of old [105:33] from Andrew Hunt and David Thomas. Uh [105:36] you probably heard about Davids before. [105:39] The discussions in this book is about [105:43] everything you do in a programming [105:45] environment professionally. You need to [105:49] weigh pros and cons. And they give you a [105:52] thousand examples of that through the [105:54] book. But the way they they describe it [105:57] is programming is a tool for achieving [106:01] something you want to do. Sometimes the [106:03] the the your goal is the journey of [106:06] programming itself and that's fair but [106:10] they call your attention to things that [106:12] you just take for granted. So that would [106:15] be the reason around my my book [106:18] recommendation. That's a good addition [106:21] to that. Well, I'll definitely include [106:22] links to all of those in the show notes [106:23] for listeners. And with that, I want to [106:26] thank you both, AC and Hussein, for [106:28] stopping by to talk shop with us on [106:30] Rails. It's been a delight getting to [106:31] chat with both of you about all the fun [106:33] stuff you're doing at Donor Box and [106:34] hopefully we'll get to chat again some [106:36] point and maybe at a conference or [106:37] something. [106:38] >> Sure. It's been a pleasure. [106:41] >> By the way, will you be on the real [106:43] world in Austin? [106:45] >> Yeah, good. [106:46] >> We we'll look forward to Yeah. getting [106:48] catch up there. U I'm speaking at the [106:50] end of day one I just learned. So um [106:52] well thank you so much both of you. [106:54] Thank you again for stopping by and for [106:55] joining us on Rails. [106:57] >> Thank you so much. [106:59] >> That's it for this episode of On Rails. [107:02] This podcast is produced by the Rails [107:04] Foundation with support from its core [107:05] and contributing members. If you enjoyed [107:08] the ride, leave a quick review on Apple [107:09] Podcast, Spotify, or YouTube. It helps [107:12] more folks find the show. Again, I'm [107:14] Robbie Russell. Thanks for writing [107:16] along. See you next time.