Hot Cell: Securing Active Storage in the age of AI - Mike Dalessio

Hot Cell：AI 时代的 Active Storage 安全防护 —— Mike Dalessio 在今年夏季对 CVE-2026-66066 采取紧急应对后（该漏洞中，Active Storage 和 libvips 中的七环利用链将环境变量机密直接泄露到生成的缩略图中），Mike Dalessio 解释了为什么在 AI 驱动的漏洞发现呈抛物线式增长的时代，永无止境的修补打补丁已经过时。Dalessio 没有继续依赖脆弱的上游 C 语言库，而是介绍了 Hotcell，这是一种受核工程“热室”（hot cell）启发的隔离执行架构，可在剥离权限且断网的容器中运行不受信任的变体处理。通过 Unix 域套接字传递打开的文件描述符，Hotcell 消除了对宿主机机密的访问权限，并以仅 8 毫秒的开销实现了深度防御。Dalessio 分享了 37signals 的生产环境部署经验，并敦促开发者以对抗性红队思维审视自己的代码库，同时构建具有弹性的架构，推动行业迈向乐观的“星际迷航”式未来。 Active Storage 漏洞 CVE-2026-66066 表明，图像处理流水线可能会被 libmatio 等下游 C 语言库欺骗，进而将本地机密和环境变量直接泄露到生成的缩略图中。 (6:16) 该漏洞利用由 AI 发现的包含七种不同系统行为的链条促成——包括默认直接上传、受信任的 MIME 类型以及 libvips 标头伪造——这反映出开源依赖项中的链式漏洞呈抛物线式激增。 (8:03) 为消除脆弱 C 语言解析器的爆炸半径，37signals 开发了 Hotcell，这是一种开源沙箱架构，在通过 Unix 套接字文件描述符传递进行通信的网络隔离容器内执行图像处理。 (14:22) Hotcell 在生产环境中运行，每次操作的开销约为 8 毫秒，其即将推出的 1.0 路线图旨在集成 Linux Landlock，以永久限制工作进程中的内核级能力。 (21:34)

Mike Dalessio argues that the rise of automated, AI-driven vulnerability discovery has rendered routine patching insufficient for C-based parsing libraries, requiring Rails applications to contain blast radiuses through strict process isolation. The Vulnerability Cascade of CVE-2026-66066 (2:28) The systemic risk of default image processing in Active Storage emerged during the triage of CVE-2026-66066. In unpatched Rails installations, `ActiveStorage::Transformers::VipsTransformer` routed untrusted uploads to libvips, which pulled in over 100 dynamic dependencies inside standard Ruby container environments. Among these was `libmatio`, an unvetted MATLAB parser. By submitting spoofed image headers, attackers coerced the library into reading local files and container environment variables, rendering sensitive Rails secrets directly into generated thumbnail images. The vulnerability registered a CVSS severity score of 9.5 because extracted secrets—such as the `secret_key_base` or database credentials—readily escalated to remote code execution. Dalessio observed that autonomous tools chained together seven separate design quirks across Active Storage, libvips, and libmatio. Despite deliberately writing an opaque disclosure advisory to stall malicious actors, working exploits were reverse-engineered and published online within an hour, illustrating that traditional vulnerability lifecycles are collapsing under automated discovery. Shifting from Patching to Blast Radius Containment (11:22) Evaluating risk as the product of probability and impact, Dalessio asserts that the likelihood of zero-day flaws in C-based binary parsers like libvips or ImageMagick cannot be meaningfully lowered. Across libvips and its dependencies alone, 378 vulnerabilities were published in a single year. Consequently, defensible application security requires reducing the potential impact of compromise to near zero rather than attempting to prevent every parsing flaw. To eliminate the consequences of arbitrary code execution, Dalessio defined an execution model where untrusted files run inside an isolated sandbox. The requirements mandate zero changes to core Rails application code, no access to host environment variables or file systems, hard caps on memory and CPU, a total lack of network interfaces, and disposable worker processes that are terminated immediately after handling a single payload. Hotcell Architecture and File Descriptor IPC (14:22) Modeled after the lead-shielded hot cells used to manipulate radioactive isotopes at Oak Ridge National Laboratory, Dalessio built 'Hotcell' to isolate Active Storage operations. The architecture mirrors Rails' native configuration points—variant processors, previewers, and analyzers—across modular gems, such as `hotcell-core` and drop-in `active_storage-hotcell` adapters. Because the sandbox container runs without network capabilities (`network: none`), Hotcell avoids conventional network-based remote procedure calls. Instead, inter-process communication relies on Unix domain sockets mounted via a shared directory. Hotcell utilizes the `sendmsg` system call to pass open file descriptors directly across the socket. The host Rails application opens the target file and transmits the file descriptor handle, enabling the sandboxed supervisor and ephemeral worker process to read input and write output without gaining access to the broader host filesystem. Production Deployment, Overhead, and Kernel Isolation (18:10) Deployed via container orchestrators like Kamal as an accessory container, Hotcell separates telemetry from untrusted execution. Rails logs calls like ActiveRecord queries, while the external Hotcell supervisor emits queue depths and JSON event logs to Grafana. Hotcell adds roughly 8 milliseconds of overhead per call, an expenditure Dalessio views as an acceptable operational cost for defense-in-depth against parser exploits. Real-world production rollouts across 37signals applications revealed that modern architectures like HEY and Fizzy integrated seamlessly, whereas Basecamp encountered outages due to legacy variations and heavy attachment traffic. Ahead of a formal 1.0 release, Dalessio is investigating Linux Landlock—a kernel module enabling processes to irrevocably restrict their own rights, preventing compromised workers from executing `fork` or `exec` system calls even if an attacker achieves arbitrary code execution. Dalessio views the surge in automated vulnerability discovery as a chaotic transition toward an eventual plateau in undiscovered flaws. He advises engineering teams to prepare by publishing machine-readable remediation skills alongside CVE disclosures, subjecting internal code to adversarial AI red-teaming, and systematically isolating aging parsing infrastructure.

[0:06] [music] [0:12] Hey everyone. Uh so the original [0:15] intention for this talk was um to talk [0:18] about a markdown API for action text [0:22] [clears throat and cough] and it was [0:23] like aspirational in nature. It was like [0:25] conference driven development like I'll [0:27] get the talk accepted and then I'll [0:28] write the code in the meantime. [0:31] Uh and something strange happened over [0:33] the year which was I spent most of my [0:36] summer heads down working in reactive [0:38] mode on security issues in Rails. Uh and [0:42] I worked on that instead of the markdown [0:43] API because it felt more important and [0:46] more urgent. And then when it came time [0:48] to prepare the talk I was like well [0:50] maybe I should give a talk about the [0:51] thing that's important and urgent. And [0:53] so instead of an active uh instead of a [0:56] markdown API talk, you're going to hear [0:58] a security talk. Thank you in advance [1:00] for your patience. [1:04] In the beginning, the earth cooled. [1:08] Dinosaurs roamed the earth. [1:12] Humans started building things and [1:15] started writing software. [1:18] >> [sighs] [1:19] >> A lot of the software written by humans [1:21] was not great quality. Some of it had [1:24] mistakes. Some of it had bugs. [1:27] Corner cases weren't totally explored. [1:30] Maybe the design of the software was [1:32] very trusting in nature. [1:35] [clears throat] [1:36] You are here. AI has created all kinds [1:39] of chaos because AI is really good at [1:42] finding these problems and [1:45] security in particular is really chaotic [1:48] right now and I'm a worrier. A lot of [1:51] what I do is security and so I worry a [1:54] lot and I worry about the future. Uh I [1:58] don't know what's coming. This morning [2:00] David talked about maybe it's going to [2:02] be Terminator, maybe it'll be a more [2:04] optimistic sci-fi movie. [2:07] But at this moment right now, I'm very [2:09] worried and you should be too. And so [2:11] I'm going to give you a talk that will [2:13] get you scared before I equip you to go [2:17] and be proactive and fix things. [2:20] [clears throat] [2:23] So you may say to me, Mike, you worry [2:25] too much. But I'm saying to you, you're [2:26] not worried enough. [2:28] My summer started with uh CVE 2026 [2:33] 66066. Don't read this, by the way. [2:36] Focus on me right now. Okay, I'll [2:38] explain it all. [2:41] The um when it came time to write this [2:44] CVE and publish it, I intentionally [2:47] chose the most opaque, obtuse language I [2:50] could and we did not disclose any of the [2:53] details because I wanted to buy you time [2:57] to upgrade Rails. [2:59] I was really worried that the nature of [3:02] this is so bad that there would be uh [3:05] attackers immediately going after your [3:07] systems before you had a chance to [3:09] upgrade. So I I wrote it to be very [3:11] obtuse. Four sentences. Let's take them [3:14] one by one. Inversions prior to blah [3:17] blah every Rails version ever. Uh active [3:21] storage does not disable lib vips. So [3:23] there are some underlined terms here. [3:25] You're free to read now. [3:28] Uh, I just want to maybe define those [3:31] terms. It gives me an opportunity to go [3:33] into exposition mode and explain a [3:35] little bit about active storage too. If [3:37] you're not familiar with active storage, [3:39] it's basically Rails's subsystem for [3:43] storing images, blobs, whatever you [3:45] want, and for transforming them. So, the [3:49] typical case is I'm using Lexi. I upload [3:52] a file, Active Storage saves it, and [3:54] then I want to make a thumbnail for my [3:56] activity feed, a little version of it, [3:58] and Active Storage transforms it and [4:00] does that. And so this CVE is about the [4:02] second half of this. It's about [4:04] transforming. [4:05] And specifically, uh, [4:09] Rails calls VIP transformer is the class [4:13] that calls out to libs on the bottom. [4:16] Okay, this is the default configuration [4:19] for Rails and um Lib Vips does something [4:22] really interesting which is it [4:23] outsources the work, right? The image [4:26] gets handed to lib VIPS. Lips looks at [4:28] it, tries to figure out who should [4:30] handle it and routes it to the right [4:32] library. And some of these libraries you [4:33] know and love, lib ping, lib JPEG, old [4:37] reliable lib magic maybe a little less [4:40] reliable. It's got a history of security [4:42] problems. But there are also libraries [4:44] that you don't even know about. Lib [4:47] Matio. Does anybody know what that [4:49] library is? [4:51] Go ahead, shout it out. [4:53] >> Matt. [4:54] >> Mat Lab. For all of your mat lab Rails [4:56] apps. [4:58] So, libips knows about lib mat.io. You [5:01] don't. And in fact, lib vips don't read [5:04] this slide either. But when I install [5:07] libips on the Ruby slim image, [5:12] this is the dependency list that it [5:14] installs. It's over a 100 libraries. And [5:16] I'm here to tell you that there's no way [5:19] that that those are all secure. There [5:21] are bugs, there are vulnerabilities in [5:23] some of those libraries right now. [5:26] And um libips has a concept of like [5:29] trusted library versus untrusted, which [5:32] is why that sentence I showed you [5:33] earlier mentioned untrusted libraries. [5:36] So like Lib Magic and Libatio are marked [5:38] as untrusted. But we're starting to get [5:41] a sense for the shape of the [5:42] vulnerability now, right? It's like, oh, [5:43] an image is uploaded that's carefully [5:45] crafted to try to trick those libraries [5:48] into doing something. Cool. What are [5:50] what are we tricking the library into [5:52] doing? Um oh well first let's talk about [5:56] um who's vulnerable um when you're [5:59] configured to use lib VIPS and you [6:01] accept image uploads from untrusted [6:03] users. So this is basically again every [6:05] Rails app uses libs by default and every [6:09] user is an untrusted user. Remember that [6:13] don't trust anybody. [6:16] So you can exploit the library behavior [6:18] to read files, read arbitrary files, [6:22] including environment variables and [6:23] application secrets. Like how does that [6:25] work? [6:27] Remember this is all running in a [6:28] container along with your source code. [6:31] Maybe that's sensitive. [6:33] You have config files, maybe your [6:34] credentials. You've got your environment [6:36] variables too. So if you're injecting [6:38] secrets into your container uh through [6:41] environment variables, those are all [6:43] going to be available to lib mat.io in [6:46] this case. And the real trick is like [6:49] what's the exploit? Because you can [6:51] trick this system into then the [6:53] thumbnail image that comes back out is [6:55] going to contain your secrets. And like [6:58] that's pretty scary. I think that's [7:00] scary. [7:02] Um the last sentence um is the real [7:04] impact. [7:07] Once you have secrets, you can do remote [7:09] code execution on a server and you can [7:12] do what's called lateral movement. And [7:13] that is like you have a database secret, [7:16] you can go mess with the database. You [7:18] have the Reddus login, you can go mess [7:19] with the cache and you can maybe even [7:22] get into corporate networks. I don't [7:23] know how you guys all have your secrets [7:26] set up, but this is really bad. This is [7:28] about as bad as it gets. This is really [7:30] bad news. This was a 9.5 on the CVSS [7:34] score. Um, about as bad as it gets, but [7:38] I haven't gotten to the best bit yet. [7:42] Once we published the CVE based on this [7:45] obtuse, opaque information, people [7:48] reverse engineered this hack in under an [7:50] hour and published their results on the [7:52] web. I bought you 60 minutes of time to [7:56] upgrade. [7:58] Super scary. Um, not done yet though. [8:03] The other interesting thing about this [8:04] is that there were there was actually no [8:06] big vulnerability here. There were seven [8:08] small vulnerabilities that got chained [8:11] together by the agents. I'll run through [8:14] them real quick so you won't get bored. [8:17] First of all, active storage direct [8:19] uploads are enabled by default. They're [8:21] not anymore. That's been fixed. But it [8:23] meant that even if you weren't using [8:24] active storage, that route was still [8:26] available to attackers. [8:28] Direct uploads never examine the bytes, [8:30] right? Direct upload means it goes from [8:32] the user's browser to the blob store and [8:34] the Rails app never sees it. And so it's [8:36] forced to trust the content type [8:38] reported. That's the vector in is by [8:41] lying about the content type. [8:43] Um oh, and that's just like a [8:45] performance thing, right? Like it would [8:47] cost you performance if you had to [8:48] download the file and reanalyze it. Uh [8:51] signed variation keys. This one is a [8:53] little opaque. It makes the attack [8:55] easier to pull off. Um, and it's not [8:58] even a security problem. So, we're [9:00] probably never going to change that in [9:01] Rails. [9:03] Uh, [9:05] this is the actual fix. We decided to [9:07] call that block untrusted method. And [9:09] it's notable that this came out in libs [9:12] as a feature that was buried in a change [9:14] log that you had to opt into the secure [9:17] behavior. Weird. Libs. This is the [9:21] actual flaw. Libs was fooled by a [9:25] spoofed mat lab header. And this was [9:26] fixed and no CVE was published. It was [9:29] so small it wasn't even worth a CVE. [9:32] Libatio [9:34] pointed the finger back at libs and said [9:36] this is functioning as designed. This is [9:37] not a bug. This is not a security [9:39] problem. You shouldn't be handing [9:40] untrusted files to me. Cool. [9:44] And the last thing was is once you get [9:46] the secret key, you can um use a couple [9:48] of there's a couple of known ways that [9:50] you can get um root like um remote [9:53] command execution based on the secret [9:56] key. You can escalate that and these are [9:59] we know about these on in the Rails [10:01] security team, but they're all breaking [10:02] changes to fix and so we've been [10:04] dragging our feet. Um but this is the [10:06] really bad one which is once you get the [10:08] secret, you can then leverage that. So, [10:11] all seven of these were chained together [10:14] by AIS based on very little information. [10:17] This should really scare you. They're [10:18] getting really good at this. [10:21] If you're not worried yet, here's a [10:23] video of the exploit script running in [10:25] real time, pulling out all of my secrets [10:28] from a development app. [10:32] Pretty scary. Once you know what to do, [10:35] it happens really fast. [10:38] It should also scare you that these libs [10:41] vulnerabilities sorry lib's dependencies [10:44] so far this year uh there have been 378 [10:47] vulnerabilities published in these [10:49] dependencies in libipss or its [10:50] dependencies. So there's a lot of them [10:53] and there's more being reported every [10:55] day. This is um 37 signals hacker one [10:59] account data. This is when you have a [11:01] bounty program. This is what the valid [11:03] reports are looking like now. This is [11:05] like going parabolic at this point. [11:08] Super scary. So, there's a whole [11:11] category of problems here. Like, I can [11:13] fix this one CVE, but like what's going [11:15] to happen when there's a zero day [11:16] vulnerability found in the next image [11:18] processing library. Like, we're going to [11:20] have to go through all of this again. [11:22] And so, this is when I started thinking [11:24] about how [snorts] to fix it. You're [11:26] probably all screaming inside, "Fix it, [11:27] Mike. You're on the security team. Fix [11:29] it. Fix it." [11:31] Um, when I think about fixing it, I [11:34] think about risk and how do we decrease [11:36] risk? And here it's the risk of having a [11:40] uh a vulnerability in one of those image [11:42] libraries. [11:44] And I just covered this territory where [11:46] we know the probability of finding [11:47] another vulnerability in these libraries [11:49] is super high and probably not going [11:52] down anytime soon. So that we're forced [11:54] to think about the impact of blast [11:56] radius. How do we minimize the amount of [11:58] damage that can be done if one of these [12:01] vulnerabilities is found tomorrow? [12:04] And that's what part two of this talk is [12:06] about. How do we actually reduced the [12:09] blast radius? [12:12] Specifically, the question I want to [12:13] answer is how can we run active storage [12:15] so that it doesn't matter if there's a [12:18] vulnerable library [12:20] that's running this untrusted uh [12:22] untrusted code on a untrusted file. [12:26] And the solution that you probably all [12:28] would reach for just like me is some [12:30] kind of a sandbox. We want to make sure [12:31] this runs somewhere. [12:34] Um, and so I started thinking about like [12:36] what are the five easy steps? Five easy [12:39] steps to secure active storage. Step [12:41] one, let's define the requirements. And [12:44] this will actually become the prompt [12:46] that I feed into my agent as we build [12:48] this system together. Um, so the grab [12:51] bag of requirements I had is I want it [12:53] to be a drop-in replacement for Rails. [12:55] You shouldn't have to make any code [12:56] changes. It should just be a [12:57] configuration change. [12:59] I want to run everything with the least [13:01] privilege, right? I want it to be a very [13:03] limited user in a file system that [13:07] doesn't allow suid. I want to drop all [13:11] the uh capabilities in the docker [13:13] container like least privilege. I don't [13:15] want it running with the rails [13:17] privileges, right? Um hard limits on the [13:19] container. I want to limit CPU time. I [13:21] want to limit memory. I want to limit [13:22] file size. I want to limit number of [13:24] files, number of processes. Let's make [13:26] it as hard as possible to do damage if [13:28] there is an exploit. [13:31] I don't want to give any access to the [13:32] app file system or the app environment. [13:34] So there are no secrets to exfiltrate. I [13:37] don't even want there to be a network [13:38] interface [13:39] because if something happens in that [13:41] box, I don't even want them to be able [13:43] to make a network call back out. I don't [13:45] want them to be able to move laterally [13:46] within your network. [13:48] I want it to be disposable. like we're [13:50] going to start up a process to handle uh [13:53] an image and I want to kill that process [13:55] when it's done so that if there is a [13:57] remote command execution the lifetime of [14:00] that is limited I want the deployment [14:02] life cycle to be very familiar I want it [14:04] to be easy to deploy with Kamal or [14:06] Kubernetes [14:08] uh and I want you to be able to build [14:10] your own things like I built active [14:11] storage on top of this system but you [14:13] should be able to build whatever you [14:14] want if you want a secure place to unzip [14:16] files you should be able to build that [14:19] so this is the requirements list and I [14:22] started thinking about real life [14:24] analoges to this and this is a um photo [14:28] from Oakidge National Labs which deals a [14:30] lot with radiation radioactive materials [14:33] and they have a device that they call a [14:34] hot cell and basically what it is is [14:37] it's um it's a big lead box and even the [14:40] window is like mineral oil so the [14:42] radiation can't get out and you have to [14:44] do everything with these little [14:45] manipulators right um and anything that [14:49] comes in or out of that box is tightly [14:52] controlled and this is exactly what I [14:54] want but a fractive storage. So this is [14:56] where the name comes from and that by [14:58] the way is also item number two on the [15:00] five easy steps [15:04] to fixing active storage. [15:07] And this is the system as we can see so [15:09] far like there's a container and there's [15:11] like something going on but like let's [15:13] get more specific. How do we jam this [15:15] into Rails? Now thankfully Rails has a [15:19] config for this already. This is [15:21] amazing. So there are three different [15:24] config parameters. [15:26] Um each of which controls one aspect of [15:29] active storage. There's um analyzing [15:32] which gets back metadata like how many [15:33] pages are in this PDF. There is a [15:36] variant processor which is [15:37] transformations. It's for rotating or [15:39] cropping or resizing. [15:42] And previewers are for generating a [15:45] thumbnail for non-image files like PDFs [15:48] or movies. [15:50] And so really all we all I want to do [15:53] here is I want your config change to be [15:54] like that. I want to be able to like go [15:57] back and forth, [15:59] right? That should be all you need to do [16:01] here. So we know that now there's the [16:04] shape of this is like there's an [16:05] existing class with an existing API. I [16:08] can do that. [16:10] And so hotell the project is actually [16:12] five different gems. There's hotell core [16:14] which is the common bits and then [16:16] there's client and server on the hotell [16:18] level. And this kind of defines how does [16:20] work get done and how does how do error [16:24] codes get returned and some of the [16:26] communications. And then there's an [16:27] active storage client and server gems [16:30] which subclass and that's what allows us [16:34] to do a drop-in replacement for rails. [16:37] And so now this is what the system looks [16:39] like right in my in my head where we got [16:41] a hot cell client and a hot cell server [16:43] and I have to communicate somehow. And [16:45] I've kind of handwaved this so far [16:46] because I said I don't want a network [16:48] interface on this. So how do we actually [16:50] get inputs and outputs [16:53] working? And the answer is Unix sockets. [16:56] Unix sockets are great. It's a little [16:57] file on disk that processes can read and [17:00] write to, but it's not routable on the [17:02] network. But it's a it's a very common [17:04] way to do interprocess communication. [17:06] And the really interesting bit is send [17:09] message the system call supports passing [17:12] open file descriptors. And this is [17:14] really cool. This is like magic. So the [17:17] way this works is on the left hand side [17:19] the image upload goes in the hotell [17:21] client class opens the file on disk and [17:24] then hands the file descriptor over to [17:27] the hotell container. So the the file [17:29] system is not accessible inside the [17:32] container but that file is because I've [17:34] given it the handle. And this is what we [17:36] use for all the inputs and outputs. It's [17:37] pretty good. [17:39] But we still don't know how the work is [17:41] going to get done. And that is step [17:42] five. How to actually do the work. [17:44] Unsurprising. [17:46] It's a supervisor worker style thing [17:48] just like active job or Puma or anything [17:51] else that you can think of. Like this is [17:53] pretty straightforward and probably not [17:54] worth spending too much time on. And [17:57] this is the finished system. And this is [17:58] the code that's actually running right [18:00] now in production at 37 signals. And I'm [18:03] here now that I've scared you to say to [18:05] you, you should be running hotell in [18:07] your apps. And I'm going to tell you how [18:10] rolling it out. [18:13] Once you've installed the gem, you can [18:15] just run hotell install. It'll create [18:16] three files in one empty directory. And [18:19] that's pretty much it inside of a hotell [18:22] subdirectory. [18:24] So the gem file is exactly what you [18:26] think it's going to be. Um, hotell [18:28] config is a file where you can actually [18:30] configure hotell. It's just Ruby code. [18:33] It's just Ruby code. It's not a Rails [18:35] process, but it is a Ruby process. [18:38] Um, and you can set the limits. Um, some [18:40] of the things that are available are [18:41] listed here. I suggest you tune those [18:44] once you get going. But this is where [18:46] you do that initial configuration. [18:49] Those are just global limits. But then [18:51] we can actually uh get more refined [18:54] along the way as well. When you deploy [18:56] your app, you've got to set up the um [19:00] the socket directory. So the only thing [19:02] that's shared is uh the a directory with [19:05] two sockets in it, two Unix sockets in [19:07] it. You just have to configure that. And [19:10] you do the same thing in your accessory [19:12] configuration. I'll I'll break this one [19:14] down. It's a lot of code. Um you set up [19:17] the volumes to match what you have in [19:19] your application. [19:21] You set up your Docker container limits. [19:24] Um, and you set up any of your [19:26] capabilities that you want to have too, [19:28] including network none, which is the [19:30] proof that none of this actually needs [19:32] the network to work. [19:34] The drop in replacement for Rails is, [19:37] you know, you could be using any number [19:38] of these Rails classes on the left and [19:40] all you have to do is change your config [19:42] to use the corresponding twin on the [19:44] right that's provided by Hotell. So, [19:47] this is what your typical application [19:48] configuration would probably look like. [19:51] Uh, just set the three configs. um set [19:53] the two properties you have for the um [19:57] Unix sockets and you're good to go. [20:01] When you um configure the hot cell [20:05] itself, you just have to load the [20:08] classes because remember this isn't a [20:09] Rails app. So Zite work does not work [20:11] here. So you require the classes you [20:13] need and you configure them. So here's [20:15] where you can do per operation [20:17] configuration. So maybe the cell itself [20:19] has a deadline of 10 seconds, but you [20:21] know this operation you want it to be [20:23] like 2 seconds. You can do that kind of [20:25] a configuration here. [20:27] And that is all it takes to port a kind [20:30] of a vanilla active uh action text [20:33] active storage application. Like that's [20:36] it. You configure active storage, you [20:37] configure the hotell, you deploy it. [20:40] That's it. [sighs] [20:42] It also comes with great observability. [20:44] Um, even though we can't trust the [20:46] worker in the container, we don't know [20:47] if that's been hacked, we can trust the [20:50] supervisor process and we can obviously [20:53] trust your Rails app. And so we log [20:56] events in the Rails app just like it's a [20:58] database hit. You'll get good metrics [21:00] around that. We periodically pull [21:03] metrics from the supervisor, things like [21:05] Q depth that we can't get from the [21:07] requests. And then the um supervisor [21:10] itself will do a little JSON log to [21:13] standard out. So you can actually see if [21:15] children are killed, why were they [21:17] killed, that kind of information. And [21:19] you can light up a graphana screen. This [21:21] is what we use uh internally for our [21:23] dashboard. Uh and I won't go through it [21:25] all, but you can set up alerting just [21:27] however you want to do it in your uh in [21:29] your environment. [21:32] The interesting bit is this only cost us [21:34] 8 milliseconds per call, which is lower [21:36] than I expected it would be. Um still [21:38] maybe higher than you'd like. We could [21:40] probably bring that number down. Right [21:42] now it's related to I think uh garbage [21:44] collection. But um that's a pretty [21:46] that's pretty cheap to know that you [21:48] won't be zeroed the next time image [21:50] magic has a problem. [21:54] It's also very flexible. If you want to [21:56] have multiple cues, you can just stand [21:58] up multiple cells that are all listing [22:00] on different sockets and have different [22:01] q depth and different timeouts and [22:02] different deadlines. [22:04] Um you can have multiple input and [22:06] output files if you have a custom [22:08] operation you want to write. You can [22:10] bring your own container if you work in [22:11] a corporate environment. There's a [22:12] conformance test that will tell you [22:14] whether your container is configured [22:15] properly to work well. Um, and again, [22:19] all the per operation limits are totally [22:21] configurable. [22:22] If you want to write your own operation, [22:24] so you have your own thing you want to [22:27] jam into hotell, this is what it looks [22:30] like. It's probably familiar to some of [22:32] you. Uh, I drew heavily from active job. [22:36] There's a perform method and the perform [22:38] method can take whatever arguments you [22:39] want. Uh so here we see we're giving it [22:42] a name operation extract text. We set [22:44] some default limits that can be [22:46] overridden and then the perform method. [22:48] And to call that from the Rails app, we [22:51] simply have a thin shim called extract [22:54] text and we call perform in hotell. The [22:57] same way for a job you would say perform [22:59] later. Here you can say perform in hot [23:01] cell and it will serialize the arguments [23:03] and then bring whatever the file is [23:05] back. And you'll notice that we have to [23:08] do a file open on the input and the [23:10] output uh because we need to hand a file [23:13] descriptor, a live file descriptor to [23:15] the uh to the call. But that's really [23:17] the only hoop you have to jump through [23:19] and generally speaking that's going to [23:21] be hidden by whatever class you write. [23:24] So that's kind of how custom operations [23:27] work. And when I rolled this out in Hay [23:29] and in Fizzy, which are both relatively [23:32] modern apps that use vanilla active [23:35] storage, this was absolutely no problem. [23:38] Base [clears throat] Camp was harder. [23:40] For those of you in the know, Base Camp [23:42] predates Active Storage. Active Storage [23:44] was extracted from Base Camp and we [23:45] never went back and cleaned it up. [23:48] Um, so there were a lot of custom [23:50] operations I had to write and there's [23:52] also really heavy usage of attachments. [23:54] Like for some reason our user base loves [23:57] to send each other memes. Don't ask me. [24:00] Um but we had a couple of outages along [24:02] the way that taught me a lot. Um and I [24:05] won't um spend too much time on these. [24:07] Just know that there were outages and I [24:09] learned lessons and I rolled that back [24:11] into the library. [24:13] Um tuning is something that you should [24:15] probably allocate time for. [24:18] But really this app is ready to go right [24:20] now. It's not a 1.0 yet because I'm [24:21] waiting for Rails to ship. I also want [24:24] to provide some a little more [24:26] observability features, some um [24:28] syntactic sugar so to speak and I may [24:30] add Linux landlock if anyone is [24:33] interested in talking to me about that. [24:35] I would love to know more. Uh landlock [24:38] is a kernel extension that allows you to [24:40] ratchet down permissions within a [24:42] process so that once you've ratcheted it [24:44] down, there's no way for the process to [24:46] regain those. So you could say, I don't [24:48] want this process to be able to fork [24:51] exec and no matter what happens, uh, if [24:54] an attack takes over that process, it [24:56] won't be able to, um, to regain any of [24:59] those privileges. That's like a nice [25:01] like built-in suspenders kind of a thing [25:02] I'd like to add. [25:05] Okay, so I scared you. Now I've given [25:09] you the tools to go and roll out secure [25:13] active storage in your apps. [25:15] Now, I just want to like reflect for a [25:17] moment. I started with, "Oh, sorry. It's [25:21] really easy to do this. Is this QA's [25:23] fault?" And blame it. It's really easy, [25:26] and you would be forgiven for doing [25:27] that, but I want to have a little bit [25:29] more nuanced view. Like, again, Earth [25:32] cooled, yada yada yada. We're in this [25:35] time of chaos right now. [25:37] And I think that I'm seeing things trend [25:40] in a good direction. So, I want to be [25:41] optimistic, but I also want to give you [25:43] a call to action. Like, there are things [25:46] that we need to do to get through this [25:48] chaos and come out the other side. [25:51] The alternative is, I don't know, we all [25:53] quit our jobs, turn off our computers, [25:55] and become sheep farmers or something, [25:57] right? I want to get through this [25:59] chaotic period. I want you guys to help. [26:02] And I want the future to be that [26:04] optimistic future. I don't want [26:05] Terminator. I want Star Trek. [26:08] So, here are the things that I'm seeing. [26:10] I showed you this chart before to scare [26:13] you, but now I'm going to reassure you [26:15] with it. This is the system working as [26:18] designed. People are finding [26:20] vulnerabilities and they're reporting [26:22] them. That's great. That's exactly what [26:23] we want. We know the vulnerabilities are [26:25] out there. We want them to be reported [26:27] responsibly and that's happening. And [26:29] this is frustrating to deal with, but [26:30] it's the system working well. And [26:32] there's a finite number of [26:34] vulnerabilities, right? Right. Um, and [26:38] at some point we're going to have like [26:40] the last bug party. We'll have the last [26:42] vulnerability party. Like this this is [26:44] unsustainable. It can't go on like this [26:45] forever. The number has to come back [26:47] down at some point. [26:49] When I said 378 vulnerabilities found, [26:52] what I meant to say was 378 [26:55] vulnerabilities were fixed. They were [26:58] fixed. All of these libraries are [26:59] trending in the right direction. [27:01] responsible maintainers fixing the [27:03] problems rolling out software updates [27:06] system working as designed this is great [27:08] building confidence [27:10] what can you all do [27:13] here's the thing I think vulnerability [27:14] reports should all come with their own [27:16] agent skills and so this is actually [27:18] what I did in response to the CVE is I [27:21] published a repo that was extracted from [27:24] my own work at 37 signals to examine our [27:27] apps and see if we had been [27:29] [clears throat] exploited [27:30] And so you can just point your agent at [27:32] this repo and at your application and [27:34] you can ask it am I vulnerable? [27:37] Probably. [27:38] And you can ask it was I exploited and [27:40] it will actually go through all the [27:42] active storage database records and tell [27:44] you if you were exploited and if you [27:46] need to rotate your secret key then you [27:48] can point it at like the Rails guides [27:50] and say well then help me rotate my key. [27:52] So this is the future and I think that's [27:53] going to help us all get through this [27:55] period of chaos. [27:58] attack your own work. When I was [27:59] building hot cell, I did this [28:01] religiously, which was uh adversarial [28:04] review on almost every commit and Jeremy [28:08] helped me out with some red teaming [28:10] where we actually set up agents and told [28:12] them to break in and they found some [28:14] gaps. This is great. So, doing this [28:17] doesn't mean that it's going to be a [28:18] perfect library, but it means that it's [28:20] going to have fewer problems with it. [28:22] So, if we do this on our own software, [28:24] we'll get through this period of chaos [28:25] faster. [28:27] And last of all, like improve or replace [28:29] existing systems. Yay, hot sell baby. Um [28:33] [gasps] the uh the only way through is [28:36] that software has to get better or it [28:38] has to be replaced. That's the only way [28:40] through this period of chaos. And so I [28:42] hope that you all will um [28:46] will take some of this advice to heart. [28:48] Don't be too scared. Be proactive and [28:50] constructive. And the future will be [28:52] Star Trek, not Terminator. Uh live long [28:55] and prosper. And thank you for listening [28:57] to my talk. [applause] [29:01] [music] [29:06] >> [music]