WalmartLabs Scales Retail Pipeline Across 5,000 Stores on Clojure

WalmartLabs基于Clojure将零售数据流水线扩展至5000家门店 只有当团队通过生命周期组件消除周围的全局状态，并严格审计第三方网络库以防止线程耗尽时，企业级Clojure系统才能实现可靠扩展。 Anthony Marcar分享了WalmartLabs电子收据（eReceipts）平台的实用架构模式。该平台跨20个微服务和70个Leiningen项目，摄取约5000家零售门店的数据。他展示了采用Stuart Sierra的Component库如何消除脆弱的全局状态、组织领域逻辑，并实现无需Mock的健壮通道式测试。Marcar还探讨了单一代码库（monorepo）的工具应对技巧，警示了第三方库中隐藏的网络超时问题，并概述了包括统合SLF4J/Logback日志记录、Apache Commons Daemon部署以及基础设施自动化在内的运维最佳实践。 WalmartLabs用Clojure重构了其核心电子收据平台，涵盖20个微服务、70个Leiningen项目和5万行代码，仅由八名工程师组成的核心团队进行维护。 (2:00) 该团队通过采用Stuart Sierra的Component库消除了周围的顶层状态，从而支持多实例数据库连接、确定性生命周期排序以及隔离的词法传递。 (4:52) 为了解决多项目monorepo中的Maven jar屏障，WalmartLabs部署了一个聚合脚本，将70个独立的project.clj文件整合到一个合成根项目中，以实现统一的REPL工作流。 (27:01) 一个未经审查的HBase库含有四个隐藏的超时参数，叠加形成了长达20分钟的阻塞重试窗口，在每秒1000次请求的负载下，可能每分钟产生60000个阻塞线程。 (30:38)

[20:48] Anthony Marcar: Mocking libraries can get pretty powerful, but if you can restrict yourself to the core Clojure ecosystem, like the core Clojure libraries, you're going to find yourself in a much more flexible area. [2:00] Anthony Marcar: Java is painful to write, but there are a lot of awesome technologies there, and it's easy when you come in from a non-Java background kind of like be scared of all that Java stuff, but when you reach these kinds of scales, then Java becomes your friend. [24:42] Anthony Marcar: I would highly recommend using Component if you're not already and you expect your codebase to scale in any way.

[0:00] right so my name is Anthony Makar I work [0:02] at Walmart Labs is a closure developer [0:04] and I'm going to talk a bit about [0:06] closure at scale so when I talk about [0:11] closure at scale I'm talking about we've [0:14] kind of been using closure at Walmart [0:15] for about three years now like a lot of [0:18] people in the closure community when [0:20] you're operating at a little bit of [0:21] scale a larger than normal it's a bit of [0:24] a frontier there's a lot of tutorials [0:26] out there on how to create small closure [0:29] applications you know little libraries [0:31] here and there but in terms of resources [0:33] on how to create larger applications [0:35] that's something that we didn't find too [0:37] much of so this talk is basically about [0:41] our experiences that we've had some of [0:44] the I guess things that we've been able [0:47] to find out that somewhat work so I [0:50] don't want this to be like this is how [0:51] you should do it this is just our [0:53] experiences and recommendations that [0:55] we've come up with everyone should be [0:56] doing these things differently it's a [0:57] frontier but anyway this is what we've [1:00] come up with so a little bit about our [1:04] system we run a team called ear receipts [1:07] within Walmart to the customer at a [1:09] store it's it's a few things it's [1:12] basically every receipt that's printed [1:14] in the store has this QR code down the [1:16] bottom that QR code encodes the unique [1:19] identity of a receipt so that you can [1:21] scan it with the Walmart app or any app [1:23] for that matter and it will download [1:25] that receipt on to your phone associate [1:27] it with your account so that you can now [1:29] do returns you'll never lose that [1:31] receipt again in the future we hope to [1:33] go purely paperless with this as well as [1:37] well as that there are some other teams [1:38] within Walmart who can use this [1:40] information provide other benefits to [1:41] the customer savings catcher is [1:44] basically getting money back if it was [1:46] cheaper at other stores insta watch is [1:48] getting a digital copy of a movie that [1:50] you purchased in a store and the one [1:51] hour guarantee is the largest sale in [1:53] the world and all of these kind of [1:55] partners inside the company use our data [1:57] as well so within the data center where [2:01] it starts to get more interesting so we [2:04] have approaching 5,000 Walmart stores I [2:07] think now we get data from all of those [2:10] stores every time a receipt is printed [2:12] that transaction [2:13] sent up to our data centers where a [2:16] whole bunch of stuff happens a whole [2:17] bunch of side effects these could be [2:19] saving it to the database [2:21] it could be forwarding on to other [2:23] partners within Walmart [2:24] it could be sending push notifications [2:27] to customers it's essentially just a [2:29] one-way pipe of data coming in and stuff [2:33] happening so of course in the internals [2:36] we use closure it's essentially a [2:39] one-way directed graph of data we have [2:42] durable queues in between all of our [2:44] consumers and producers so that because [2:48] when you're operating in a Walmart data [2:49] center stuff goes up and down and goes [2:52] missing and everything that Zack was [2:54] just talking about is very true [2:56] especially at this kind of scale so all [2:59] the edges we have keys if Apple is down [3:03] which I hope it wouldn't be but if it [3:05] does then we actually have a queue that [3:07] kind of blocks you know protects us from [3:09] that bit of realm and vice-versa for all [3:12] these other side effects we have a lot [3:16] of other closure as well [3:17] we have closure to do REST API s2 called [3:20] access we have closure to do deployments [3:22] we have closure to monitoring its [3:26] closure all the way down we use a lot of [3:28] closure all in all this is about 20 [3:32] services that we run it's about 50 [3:36] thousand lines of code it's about 70 [3:40] Leiningen projects these are all ways to [3:42] get a sense of roughly how big it is [3:44] it's I don't think it's the biggest [3:46] closure system out there but it's a [3:47] little bit bigger than normal and we run [3:49] this with about eight developers and [3:51] have been doing it for about three years [3:52] and all started from a java application [3:55] back in a startup and then we've [3:57] rewritten it all to be big in glory yeah [4:02] so basically the stuff I want to talk [4:05] about today is all the things that we [4:07] run into as we grew the team and grow [4:09] our code base how do you structure code [4:10] how do you deal with multiple projects [4:13] like learning and projects how do you [4:15] evaluate what project what libraries to [4:18] use inside your applications and bit of [4:20] a framework for thinking about that and [4:22] also just embracing the Java ecosystem [4:24] we build on the shoulder [4:26] of giants Java is painful to write but [4:29] there are a lot of awesome technologies [4:30] there and it's easy when you come in [4:32] from a non Java background kind of like [4:35] be scared of all that Java stuff but [4:37] when you reach these kinds of scales [4:39] then Java becomes your friend and if I [4:42] get time every time I give this talk it [4:44] goes a different length but if I get [4:46] time I'll go and talk that [4:47] infrastructure as code as well and some [4:48] of our ideas around that so yeah [4:52] structuring code there are a lot as I [4:56] mentioned there are lot of tutorials [4:57] online you know little quick namespaces [5:00] showing you how to write applications [5:01] almost all of them have def connection [5:06] start connection and then functions that [5:09] kind of like reference that global State [5:11] one thing with us now we started over [5:14] time we've tried to eliminate as much [5:16] global state as possible and a lot of [5:18] this is driven by Stewart Sierra's talks [5:20] on closure in the large and component [5:23] and basically the recommendation I can [5:24] give that has worked really really well [5:27] for us is embrace component if component [5:31] is a library it's a tiny framework that [5:34] helps you manage runtime state and gives [5:36] you a place to put it so that you can [5:38] instantiate it you know where it is and [5:41] it's not just kind of some global var [5:43] sitting off in the distance so I'm gonna [5:46] show a little bit of code about examples [5:50] of how we actually use component before [5:54] I do that who is already using component [5:57] in some way okay [6:00] so I'll walk through a little bit of how [6:02] component works in and what it is so [6:06] this is a component this is a Cassandra [6:10] component that we have it is essentially [6:13] holds a database connection and [6:16] configuration on how to create that [6:18] connection so it's just a record in this [6:21] case a component can actually just be a [6:22] map it's basically just some place to [6:25] put data and so on so we have hosts [6:29] which is just going to be a vector of IP [6:31] addresses where the Cassandra connection [6:33] is we have a key space which kind of [6:35] tells us which key space we want to [6:37] access inside Cassandra [6:39] and when we first instantiate this [6:41] record we don't give it a connection the [6:44] connection is something that we want to [6:45] be put into this record after it started [6:48] so how does that happen so there is this [6:50] protocol called life cycle in component [6:53] and component provides two methods start [6:56] and stop you call call start and we're [7:01] in closure it's immutable so this is [7:03] going to return a new version of this [7:06] record the old version still exists but [7:08] after we call start we're going to it'll [7:10] go and do all this work to create a [7:12] Cassandra connection and it'll return [7:14] that with by just as searching the [7:17] connection back on to this record and so [7:19] that's what you get this isn't some kind [7:21] of immutable thing that you call start [7:23] on it mutates the state inside and then [7:25] you just keep the same reference every [7:27] time you call start you need to grab the [7:29] result of that and component gives you [7:32] ways to do that very very easily so [7:34] within here we have we say when we call [7:36] start if the connection is on this [7:39] record that means it's already started [7:41] and we want to do item put in operations [7:43] here so we are going to just return [7:46] immediately but it's going to send back [7:48] these same record as we started with if [7:50] it hasn't been started if that [7:52] connection object is nil then we are [7:54] going to call this start connection [7:55] thing that could be whatever in the case [7:58] of Cassandra probably just returns a [8:00] connection object so we're going to [8:02] search that object back onto the record [8:04] and return the record first first so for [8:08] stopping we have if the connection is [8:10] present it means that it has been [8:12] started so we do want to stop it so [8:15] we're going to call stop connection [8:16] which does something stops the [8:18] connection and then associate is [8:20] connection back on is nil so that we [8:22] know that this is not started anymore [8:24] it's stopped and vice-versa [8:27] if it if the connection was nil that [8:30] means already stopped so we can just [8:32] return the record as is and for each of [8:36] these components we usually just have a [8:37] helpful constructor in our applications [8:40] there is a config map it's like [8:43] basically an Eden file that sits on the [8:45] machine and there's a loaded when the [8:46] application starts and this config [8:49] contains all the configuration like IP [8:51] addresses usernames passwords all that [8:53] kind [8:53] configuration we throw that into the [8:55] constructor with a key space and then [8:58] this constructor is responsible for [9:00] pulling out the information from that [9:02] configuration map that it cares about in [9:04] this case we're going to assume that [9:06] config is a map it has a key Cassandra [9:09] and under that it has a hosts map or [9:12] whatever it is and so we use that to [9:16] call the record constructor and passing [9:19] that information that we need so this is [9:21] what a component looks like for us and [9:24] this is there are different types of [9:26] components if anyone seen Stewart [9:27] Sierra's component talk he talks about [9:29] three different types of components [9:31] stateful service and I think it's a [9:35] composite or something like that this is [9:37] an example of a stateful component it [9:38] there is some state state could be a [9:41] database connection a thread pool a Atem [9:44] like anything that is inherently not in [9:47] not immutable even the core async [9:50] channel is state in some way so that's [9:53] what this does this is encapsulating [9:55] that state somewhere that we can think [9:56] about and reason about and instantiate [9:59] another type of component is push so [10:06] this is a service component it kind of [10:09] encapsulate some service over here which [10:12] does something in this case it's an HTTP [10:15] SMS service that we might use to go and [10:17] send an SMS to a customer and by the way [10:19] most of this code is actually really [10:21] close to how it looks in our code base [10:23] and like in a production code so in this [10:26] case you'll notice that we have a record [10:28] this is our component it has a little [10:30] bit of configuration a host and a port [10:32] that we want to go and send information [10:32] to and we have naively decided to use a [10:36] blocking HTTP client which doesn't give [10:40] us any kind of access that connection or [10:42] how to stop and start it but that's fine [10:45] in this case there is no lifecycle so [10:48] we're just going to say a record is [10:49] constructed it doesn't need to be [10:51] started or stopped it's just a record [10:53] and what component will do is it will [10:55] say all right well if there is no [10:57] lifecycle created for this particular [10:59] record then we're just going to return [11:02] the record there there's nothing to stop [11:03] and start and that's fine so in this [11:05] case we just have the record and it's [11:07] just implementing a protocol which is [11:09] just going to send an SMS to a customer [11:11] given a phone number and a bit of text [11:14] to send them and again we just have a [11:17] constructor function just to make our [11:19] lives a little bit easier we pass in a [11:20] configuration map it's going to pull out [11:22] the SMS configuration from that map and [11:24] use that to construct the record so [11:27] that's that kind of component that's a [11:30] service component the last component [11:32] that I'll mention is a composite [11:34] component and this is you can think of [11:37] it like a business logic place you know [11:39] there we've got our kind of underlying [11:41] components that deal with state service [11:43] providers all these kind of raw [11:45] operations that we want to do a [11:46] composite component is something that [11:49] where you can actually put in the [11:50] business logic so this one again it's a [11:53] record with no start and stop because [11:54] there's no state we care about we're [11:57] going to give it a little bit [11:57] configuration some text that we would [12:00] want to send in the SMS and then we're [12:03] going to give it some components of its [12:05] own to use in this case the Cassandra [12:07] component because we need to persist [12:08] receipts and SMS because we need to send [12:11] messages to customers here's our [12:15] constructor this one's a little bit [12:17] different we've got our usual calling [12:20] the default construct that sorry the [12:21] constructor for the actual record up [12:24] here inside the uploader in that config [12:27] map there's going to be that SMS text we [12:28] presume but also you'll notice that [12:31] we're telling we're using a component [12:34] system to say you're going to need a [12:37] Cassandra component and you're going to [12:39] need an SMS component we're not going to [12:42] tell you what they are right now like [12:44] you don't have to construct them before [12:45] you pass them into this continue [12:47] uploader constructor we're just going to [12:49] assume that there is some component that [12:51] has been created which you can reference [12:53] by this name here receipt Cassandra and [12:55] there is also going to be some component [12:57] called SMS so when you find those [12:59] when component tells you what they are [13:01] and this is just classic dependency [13:02] injection just slot amine into Cassandra [13:05] and SMS up here in the record so finally [13:09] we end up being able to use these [13:11] components in this case we have an [13:14] upload receipt function basically when [13:17] receipt is uploaded into our system we [13:18] need to go and do some stuff [13:20] and that stuff is going to be inserting [13:22] the receipt to Cassandra and sending an [13:25] SMS to the customer so the important [13:28] thing here is that we've passed in the [13:30] upload a component into this function [13:32] usually what you'd see is something a [13:35] bit like this you have got old upload [13:38] receipt because this is actually what we [13:39] used to do we would call this upload [13:43] receipt function we would pass it the [13:45] receipt and then we would just assume I [13:47] shouldn't go to that meeting I thought [13:52] I'd turn a felt look I did yeah we use [13:55] that look at one month so yeah you're [14:00] going to insert a receipt to Cassandra [14:02] and you going to send SMS but you'll [14:03] notice that we haven't told it what that [14:06] Cassandra object is where the connection [14:08] is you know this is kind of very typical [14:10] code that you'll see around the closure [14:12] community where it is assumed that there [14:14] is a connection object in some namespace [14:15] that has been started but we don't know [14:18] and so if we were to just call this [14:20] function as is like who knows what might [14:22] have happened that connection object [14:23] might have been stopped there might be [14:26] some kind of race condition that it goes [14:28] there because now it's like a single bit [14:29] of state that you know everyone's trying [14:31] to use so there's that there's a lot of [14:35] problems with this and while the code [14:37] looks nicer you know that there's less [14:38] like you know D structuring of kind of [14:41] various other sub components and so on [14:43] it it does hired a lot of complexity [14:46] that's going on here [14:47] so this is how we write it we pass in [14:49] the uploader we pull out the information [14:52] from that uploader that we care about [14:53] which in this case is the SMS text [14:55] configuration the Cassandra object the [14:58] Cassandra component and the SMS [14:59] component and so therefore now the [15:03] Cassandra function just takes a [15:04] Cassandra component and does something [15:06] with it and vice versa with that send [15:08] SMS this doesn't need to know about the [15:10] internals of Cassandra or SMS you know [15:12] it's just going to pass it through the [15:13] bit that cares but the important thing [15:14] is that we have access to all these and [15:16] there is zero global state right now [15:18] everything is within lexical scope [15:21] within this function so we find that a [15:23] huge win so let's go and one step [15:29] further how do we construct this system [15:32] here is our constructor for our whole [15:35] system ours is about ten times bigger [15:38] than this because we have a lot of [15:39] components but essentially what we're [15:41] doing here is we are going to say [15:43] constructor system assign these names to [15:47] the various components so that we can [15:48] reference them in other places and then [15:50] go and construct them so we're going to [15:53] construct the Cassandra I'm going to [15:55] pass in the configuration map which [15:57] we're just pulling into this here [16:00] and we're going to say this particular [16:01] Cassandra connection is for the receipt [16:03] database we're also going to construct [16:05] another Cassandra component this one's [16:07] for the customer database and you can [16:10] see here we've already kind of gone [16:13] further than what you could do just [16:14] using a symbol a single global var for [16:16] the Cassandra connection [16:17] you can't instantiate that connection [16:19] multiple times here you can because it's [16:21] it's just in this record so just create [16:23] multiple versions of the record so for [16:26] us when we have multiple databases and [16:27] and this is something which you know I [16:29] think is code bases get bigger and [16:30] bigger you run into this is this is [16:33] where the avenge comes in another thing [16:36] to note here is that we don't need to [16:37] care about ordering component we'll [16:39] figure out the order of dependencies [16:41] when they need to be started and stopped [16:42] so all we need to do is say like you [16:45] know we could swap all these around it [16:46] could put you know Redis down here you [16:48] know there's nothing that you know that [16:50] we need to worry about here the actual [16:54] here's where it actually we tell [16:57] component how things are ordered we say [16:59] this customer API component depends on [17:01] Cassandra customer and store Redis [17:04] so therefore component will do the job [17:06] of constructing those first calling [17:08] start first so that everything has [17:10] started stopped in the right order [17:14] so this is this is our constructor [17:17] system we've got services here the [17:19] receipt service might bind to port 8080 [17:20] that that would be kind of its start [17:22] operation and then push would usually [17:25] just be like starting connections and [17:27] doing various configuration so as I [17:30] mentioned hours are a lot bigger than [17:31] this but this is how you can start a [17:33] system and I can show you what that [17:34] actually looks like so I've got a little [17:43] so I've got a bit of configuration here [17:46] this is pretty much what a lot of our [17:48] stuff looks like you have some SMS [17:50] configuration including the host and [17:51] port you have the SMS text we referred [17:54] to area strewth you've got a receipt we [17:57] have some Cassandra hosts that we're [17:59] going to care about this is what our [18:01] configuration map looks like when we [18:02] pass it into our application and then if [18:05] we want to start the system or create a [18:06] system we can call the so this is this [18:10] new system function which I mentioned [18:13] earlier so let's create that and see [18:16] what it creates nothing oh yeah I need [18:18] to borrow some configuration ok so here [18:26] you can see we've created a receipt [18:27] Cassandra record it has hosts the key [18:31] space and a nil connection and that's [18:33] because we haven't started the system [18:34] yet we have a customer Cassandra again [18:37] with a nil connection we have our [18:39] uploader here which importantly doesn't [18:41] have Cassandra or SMS yet because we [18:43] haven't started the system so this is [18:45] what the system looks like when you [18:46] construct it and remember it's just a [18:48] map there's nothing special going on [18:50] here yet so let's take that it's call [18:54] that system and now let's start it and [19:03] I'm gonna pre print this okay so this [19:08] time we can see that receipt Cassandra [19:10] has a connection object it's here we can [19:13] see that the customer Cassandra one has [19:15] a different yep a different Cassandra [19:18] object the uploader oops yes so the [19:28] uploader now has the text and it has a [19:31] reference to the other standard object [19:33] and remember we're talking about a big [19:34] persistent map here so where it looks [19:36] like it's all copied around that this is [19:38] actually just referencing the same [19:39] actual bit of data but again this [19:42] connection object is the one which was [19:43] created earlier so now we've kind of [19:45] filled in all the blanks [19:46] everything's been starting the right [19:47] order we've got our because our [19:49] connection objects created all of our [19:51] can our business logic components that [19:54] relied on Cassandra and [19:55] and everything they're all filled in now [19:57] like this system is ready to go and we [19:58] can start using it and just to hammer [20:02] this in home the system that we created [20:03] originally is still the same thing it [20:07] the uploader oh no wait I didn't write [20:17] the original system that we created [20:20] hasn't been mutated it's still in a [20:21] mutable record so this makes reasoning [20:23] about your system very easy so this is [20:27] kind of a great way to structure your [20:29] applications that we've found it really [20:31] helps in testing as well we do a lot of [20:35] side effect testing we will say run [20:39] upload receipt and make sure that an SMS [20:41] was sent to a customer well we really [20:43] shouldn't send an SMS to a customer [20:44] we're running a test that's not a good [20:46] idea and this is a pretty common [20:48] occurrence that people run into and [20:49] usually the traditional way you do this [20:50] is you would mock you would with read if [20:53] you would kind of take the function that [20:55] you were going to run and now you just [20:56] say instead of running it just do [20:58] nothing [20:59] mocking libraries can get pretty [21:02] powerful but if you can restrict [21:04] yourself to the core closure ecosystem [21:07] like the call closure libraries you're [21:09] going to find yourself in a much more [21:11] flexible area so what we do is we had [21:15] our push protocol here SMS and obviously [21:20] here we're going to send a post to some [21:22] SMS provider they're going to do the [21:23] worker sending the SMS to the customer [21:25] so what we can do is we can instead of [21:27] creating that record we can create our [21:29] own record that still emits SMS but [21:32] simply puts that result onto a channel [21:33] and we've kind of experimented with this [21:35] a little bit you can put it into an atom [21:37] you can use various other kind of [21:40] synchronization mechanisms the great [21:42] thing about putting this information [21:44] into a channel is that you can block [21:45] somewhere lower down the test so that [21:47] you can immediately know when a [21:49] particular operation occurred as opposed [21:51] to when you perform a side effect and [21:53] you just put that information into an [21:54] item or something like that you've now [21:56] got a pole at the end of the test or you [21:58] know do a thread sleep for about 10 [22:00] milliseconds while you wait for that [22:02] particular side effect to occur because [22:03] by definition like everything we do here [22:05] is there's cues involved there's side [22:07] effects like everything is occurring [22:09] in concurrent ways so this is how we do [22:12] it we will say when the Sen SMS [22:14] operation occurs just go and put that [22:17] information in channel and you're done [22:19] so here we have our same constructor for [22:22] it for this particular component and [22:24] when we actually create the test this is [22:27] the full test that we're going to create [22:28] we're going to have some SMS text that [22:31] we want to send the customer we create [22:33] our configuration map we create a [22:36] channel upon which those SMS information [22:38] will be put eventually as part of the [22:41] side effect we're going to construct the [22:43] SMS component using that SMS channel and [22:46] again this would have been HTTP if we'd [22:48] been using our usual system that I [22:50] showed you earlier and now we're going [22:53] to construct the uploader so this is [22:56] we've seen this one before it's going to [22:59] take the SMS text and bubble bar and [23:01] expect that there's a Cassandra and SMS [23:03] component handed to it and because [23:05] components are just Maps we can do that [23:07] we can just say construct the system [23:09] because we construct this uploader don't [23:11] start it yet but just associate the SMS [23:14] on to that particular uploader so what [23:16] is telling it just use this one now so [23:19] continue on we're going to create a [23:21] phone number we're going to create a [23:22] receipt which is just a map for now and [23:25] we're going to call upload receipt now [23:27] we're going it we want to test here that [23:28] the SMS is sent so now we can just sit [23:31] here blocking waiting for that [23:33] particular channel event to occur and [23:35] then get the value out and then BOOM we [23:37] can just say here is phone number and [23:39] text and is it correct one little [23:44] challenge here is that if that side [23:46] effect doesn't occur then that's [23:47] obviously going to block forever so you [23:49] usually want to put a little bit of kind [23:50] of you know sugar around this to say [23:52] timeout after a second and if the [23:54] timeout occurs that means the test has [23:56] failed but that's all stuff you can add [23:58] pretty easily so I want to stress that [24:01] this is the whole test there is no [24:03] global state still everything is here so [24:05] if you wanted to run these tests in [24:07] parallel you can do that now because [24:09] you're constructing completely separate [24:10] systems if you're doing things with [24:12] ports you know you can just have in [24:15] incrementer so that every single time a [24:17] system is constructed uses a different [24:19] port now you can like blast off ten of [24:21] these tests in parallel [24:23] you could very easily use this to [24:25] generate property based testing because [24:28] this looks an awful lot like properties [24:30] for all so all of this stuff becomes [24:33] very easy you don't have to worry about [24:34] state stepping on each other or [24:36] different threads you know stepping on [24:37] each other [24:38] everything's here and you can [24:39] instantiate it which we find pretty cool [24:42] the other the final thing they'll say [24:44] about this is for I showed you the big [24:48] system constructor before with all the [24:50] different services again we have [24:51] something like 50 components or [24:54] something like that a lot of them are [24:55] side effect upon your running tests you [24:58] don't want to perform those side effects [24:59] so what we end up doing a lot of is [25:02] creating a mock system which just has [25:05] the components that we don't want stuff [25:08] to happen to so in this case SMS and [25:10] Apple and Google push so now the next [25:12] time we run the test because a system is [25:15] just a map-like component is really nice [25:17] like this it's 200 something lines of [25:19] code most of which is comments like it's [25:21] really small it's just kind of some dust [25:24] around the edges of maps and records so [25:27] here we're just going to construct a [25:29] system which returns a map record we're [25:31] going to merge in the mock components [25:34] just over the top of it so we're saying [25:35] for those particular components just [25:36] overwrite them we don't care about them [25:38] and now we can call start so here this [25:42] is what a lot of our code looks like we [25:43] have a full system like this is all the [25:46] production code that we want to test [25:47] just with the very edges mocked out and [25:50] we can run upload receipt passing in [25:53] that in entire system and test like 99% [25:56] of our code all the way through and so [25:58] yeah this is this is what a lot of our [26:00] tests look like so yeah I would highly [26:04] recommend using component if you're not [26:05] already and you expect your codebase to [26:07] scale in any way [26:08] even if you don't expect your code to [26:09] scale component is just a really nice [26:11] way to reason about your code and it [26:13] also answers a few questions about when [26:15] you put code which is the other thing we [26:17] struggled with like the namespace is not [26:20] a class a namespace is you know just [26:22] some place to put stuff so one pattern [26:24] we've been using a lot of is you've got [26:26] a component for Cassandra [26:28] it has like insert put all the kind of [26:30] like you know local functions to it so [26:33] just put all that in a namespace and [26:34] coloca Sandra [26:35] and vice-versa for SMS like this is a [26:37] really great way to separate your [26:39] application and it also means that [26:42] eventually if you do want to split your [26:43] application into multiple projects then [26:45] you kind of components become a natural [26:48] way to separate that because it's state [26:50] and States a great way to separate [26:51] things so that is right so let's go back [27:01] to multiple projects so this is [27:06] something which a lot of that we get a [27:09] lot of you create a project it's one [27:11] Lanigan project we've looked at brood a [27:13] little bit we haven't looked enough to [27:15] kind of talk about it much so I'll just [27:16] talk about line but yeah you create a [27:18] project and you want to split it out and [27:23] when you do you kind of hit this problem [27:25] of all right let's let's assume that we [27:28] have system we're using HTTP let's [27:32] assume that's one of our own libraries [27:33] and we want to make some changes to it [27:36] so we're going to go into it in cider [27:39] and let's just go in how to println ah [27:43] we can't this is buffer is read-only and [27:45] the reason is is that it's actually in [27:47] your local maven repository so you [27:49] obviously can't edit it because it's [27:51] inside a jar file um who actually has [27:53] hit this like yeah it gets really [27:56] annoying and this is a side effect of [27:58] having a maven repository and Java which [28:02] closure is based upon so if you have [28:04] multiple projects and you depend on one [28:05] project down here and that means that [28:07] when you try and navigate into that [28:10] particular project and make some code [28:12] changes it's actually referencing your [28:14] local maven repository it's not [28:15] referencing your local source code I [28:18] won't go on too much about this I will [28:21] say that it's been a pretty annoying [28:24] issue for us we don't really have any [28:26] answers on how to best deal with [28:28] multiple lining and projects the only [28:31] thing we can say is that we're in a [28:34] better position to deal with multiple [28:35] projects in the closure community [28:36] because a project clj for instance is [28:39] just data it's just this you know thing [28:42] that you can pull into memory you can [28:44] parse you can pull out all the [28:45] dependency lists you can pull out the [28:47] source directories and [28:48] can just kind of merge it together to [28:49] create some kind of zombie Frankenstein [28:52] project and that's something that we did [28:54] which I'm just going to show you what it [28:56] looks like this is a generated project [29:00] clj you know we've got some kind of name [29:03] for it we've got a version but [29:06] importantly we've got all the [29:07] dependencies from all of the projects in [29:10] our well that's a lot when you look at [29:15] it in a small buffer yeah and like we [29:19] have all the sauce directories from all [29:20] of our projects or 70 of them in here [29:23] all the resource paths so you obviously [29:25] won't want to do like update this every [29:27] time you add a new project so we just [29:30] generate this and I won't show the code [29:32] for it like it's it's a it's a good [29:33] exercise and something to do but you [29:35] just like traverse the whole directory [29:37] structure pull out all the products your [29:38] J's merging together somehow then you [29:40] can do stuff like run all the tests in [29:42] your system within the one rep within [29:44] the one session you can have access to [29:46] all of your code in the one repple [29:47] session like it's it's kind of a I don't [29:49] know it's a total hack but uh you know [29:53] we can we can hack it this is this is [29:55] pretty like accessible to us as closure [29:57] developers because it's just data that [29:59] we can work with try doing this in Java [30:01] or something else and it's pretty [30:02] difficult right I've heard I was going [30:09] to lead up to this but choosing [30:11] libraries is something that we also [30:13] struggle with a lot how do you choose [30:16] what library to use when you're reaching [30:18] kind of some kind of scaled application [30:20] there's a few ways we think about this [30:23] but the fundamental thing is just be [30:25] extremely cynical and yeah rich Hickey [30:27] said this best not everything is awesome [30:31] it's you know you've got there's so many [30:34] different closure libraries out there [30:35] but when they're there you've got to ask [30:38] a few questions you've got to ask does [30:40] this force me into a particular style of [30:41] programming like an idiom you know is it [30:43] full of macros and if it is how is that [30:45] going to affect how we like build on [30:48] this library in the future is it going [30:50] to mean that we're just stacking on code [30:51] left right and center like simply to you [30:55] know when we could have just cut it out [30:56] and like toned ourselves so in general [30:59] if a library does one thing really well [31:01] if it's [31:02] small if it's just input/output it [31:04] doesn't have any state doesn't have any [31:05] network connections like pausing CSV you [31:09] probably don't want to implement that [31:10] yourself like you know how about it go [31:11] and go and grab a lottery for it if it [31:14] forces unit was some kind of idiom be [31:17] really sure that that's what you want to [31:19] do core async is a pretty good example [31:20] of a library that does something which [31:22] we just couldn't do so we're quite happy [31:24] to use it it forces into a style of [31:27] programming but it solves a really cool [31:28] problem the that one to be really [31:31] worried about though is any library [31:33] which does any network operation so [31:35] we've been hit pretty hard by this we've [31:37] you know there's let's take an example [31:39] like I know accessing Dropbox or [31:41] something like that you know if there's [31:42] some closure library which maybe does [31:44] this and you can just kind of call it [31:46] give it a data structure and maybe it [31:47] just creates a file on a Dropbox server [31:49] somewhere if it doesn't give you timeout [31:52] guarantees if it doesn't tell you how [31:54] the queuing works behind the scenes if [31:57] if it's using blocking i/o instead of [31:59] non-blocking i/o these are all questions [32:01] that you need to ask pretty early on [32:03] because the second your server starts to [32:05] scale into any you know reasonable [32:07] number of requests per second these [32:10] things can can hit you pretty hard we [32:12] look at example we had we were using [32:15] HBase for a while and didn't do our [32:17] research on the library that we were [32:19] using and so turned out that when we hit [32:22] you know a thousand transactions a [32:23] second which happens on Black Friday and [32:26] thankfully this one didn't happen on [32:28] Black Friday [32:30] basically the network slowed down you [32:32] know requests we're only going at about [32:34] a hundred bytes a second or so and so [32:36] when you've got you know all these [32:38] transactions coming in and then it slows [32:41] down it turns out that our timeouts [32:43] weren't working as we thought they were [32:44] we had specified a timeout of one second [32:46] I think but it turns out that there were [32:48] like four other timeout parameters that [32:49] we had no idea about and it wasn't your [32:51] typical connection versus you know [32:54] socket kind of timeout there was like a [32:57] retry timeout so if it did timeout then [32:59] it would retry itself but it wouldn't [33:02] show that to us as the end user so it [33:04] turns out that through a weird [33:05] amalgamation of stuff you get 20 minutes [33:09] of default timeout on this library and [33:11] this is something that we found out the [33:12] hard way and it and it was blocking [33:15] so if you can imagine thousand requests [33:18] a second sixty seconds you've got sixty [33:20] thousand threads open yeah so do your [33:23] due diligence on libraries libraries are [33:25] awesome especially at the early startup [33:27] stage like you know just how about it [33:29] it's great but the second you think that [33:31] you're going to be scaling then just get [33:32] I mean just be cynical about libraries [33:34] it helps a lot [33:36] logging so who who is like understands [33:43] Java looking well actually no I'm [33:46] looking there are there's a lot of [33:49] wedding stuff out there there's log back [33:51] there's love for Jay there's Java [33:52] logging thankfully there is some kind of [33:55] facade right in the middle [33:56] SL foj don't be too afraid of it if you [34:01] are getting to scale then you're going [34:04] to start to want to debug a lot of the [34:05] java libraries that underlie your system [34:07] and and most like a lot of the libraries [34:09] that we use which we take the ground [34:11] enclosure like clj HTTP it's using a [34:14] patchy HTTP you know like most of the [34:15] stuff that we're using is still [34:16] underlying is Java if you don't [34:19] understand how this works then you're [34:20] going to be able to bug to bug this in [34:22] production so yeah like understand it [34:26] it's worth it it gives you a lot of [34:28] power you know we're doing stuff now [34:30] like we log you know just using a seller [34:34] for jay logging and then we we configure [34:37] a Pender's using log back zero so we [34:39] might append in JSON to a file so the [34:41] log stash can put it up to a server we [34:44] might also append by UDP to a ream on [34:46] sober you know there's lots of like this [34:49] is once you're using a logging framework [34:51] these are all things you can configure [34:53] at runtime very very easily and I've got [34:56] a few like little code samples around [34:58] this so it's just finished at ten forty [35:02] or ten fifty no fourteen okay cool I [35:06] have five minutes so actually I'll skip [35:11] this I will say I've got all this code [35:13] up on github I've got a link to it at [35:15] the end of it so you can have a look at [35:16] it but here's like some basic stuff that [35:19] you should add here your application to [35:21] ensure that all logging from Java util [35:23] logging from SOF AJ from log back [35:26] it all goes through the same facade [35:27] otherwise you're going to [35:29] yourself having a log4j xml i'll log [35:32] back XML a java.util properties file so [35:35] it's really nice to kind of put them [35:36] into the one bit of configuration and we [35:39] have that inside a project clj as well [35:42] so check this out afterwards if you're [35:45] interested in this kind of looking stuff [35:47] another quick thing that we do is [35:54] demonising so when you are deploying an [35:58] application usually end up with an uber [36:00] jar honest on a system somewhere and [36:02] there's quite a few ways of deploying it [36:05] i've actually running that Ruby jar the [36:07] way that we do it which will kind of [36:09] like is we use Commons daemon it's say [36:12] Apache project that has an interface [36:14] that you implement with four methods in [36:17] it [36:17] start stop destroy pretty pretty basic [36:19] stuff it's not like Tomcat or anything [36:22] like that it does just one thing very [36:24] very simply so you implement this and it [36:27] turns out that this works really well [36:29] with component because in in it you can [36:32] construct your system using some kind of [36:34] config file coming in from the file [36:36] system and then you can store that and [36:39] this is the one bit of global state we [36:41] have we have no global state in our [36:42] system except for this one system [36:44] reference and the only reason we have [36:46] that is because Commons daemon calls in [36:48] at first and assumes that you mutate [36:50] some global state to create the system [36:52] so that afterwards it can start that [36:55] system so here's our system which is no [36:57] when we start and then we initialize it [37:00] and then we call start and all we're [37:02] doing here is we're just altering that [37:03] root bar and calling components start on [37:05] the entire system which will then go [37:07] through every component in the right [37:08] order start everything up so it's all [37:10] ready to go and then is also stop and [37:13] destroy and so your access to this is [37:15] via the program JCC which is just a [37:18] binary running on the system you can [37:20] tell it what user to run as you can tell [37:22] it what out file to use you know Java [37:25] configuration and then finally you just [37:27] give it the class where that interface [37:30] is implemented and started so yeah [37:34] Commons is cool [37:36] I think yeah well I had a lot more stuff [37:40] to talk about but I'll have to cover [37:42] short [37:43] I guess the one other thing I do want to [37:45] mention is that infrastructure is code [37:50] gets a lot more important as you grow [37:52] when you've got one application one over [37:54] charter deploy you can SCP it to a [37:57] server and run it but when you've got 20 [38:00] services running on hundreds of nodes [38:01] with queues and databases and load [38:05] balancers and monitoring and logging and [38:07] all this stuff you want to be able to [38:10] spin up a new service like that you want [38:12] to be able to say alright I'm gonna pull [38:14] these components together and create a [38:16] new Bazaar out of them and then that's a [38:18] new service that I'm going to provide to [38:19] my customer if you're not automating the [38:22] creation of the log back XML of the load [38:25] balancer configuration of how many [38:28] machines to deploy it onto if you're [38:29] doing that via hand you're going to be [38:31] scared of creating new services it's [38:33] gonna be like a time consuming thing [38:35] full of you know errors automate as much [38:38] as that as possible as early as possible [38:40] and it'll make your lives a lot easier [38:42] so as an example like here's the stuff [38:46] that we have to do is part of [38:47] configuring a new service and we [38:49] automate all of it so that we can and we [38:51] actually deploy all of this from a [38:52] ripple and if you want to hear how we do [38:54] that you can come and find me afterwards [38:55] because I'm not going to recommend the [38:57] technology in public so anyway yeah [39:03] thanks listening here is the code [39:05] samples I would highly recommend reading [39:08] release it it's a book by Mike Nygaard [39:10] it'll scare the crap out of you when [39:12] you're thinking about creating big [39:14] systems because it tells you all the bad [39:15] things that can happen but you should [39:17] read it [39:17] watch component by Stewart Sierra it's a [39:20] great it goes deeper on what component [39:22] is language of the system by Richie key [39:24] was behind a lot of our design of our [39:27] overall architecture and if you do want [39:30] to learn more about infrastructure as [39:31] code and DevOps which I highly recommend [39:33] you do the Phoenix project is a great [39:35] place to get started thank you