Ollama Cloud Token Volume Jumps 150-Fold as Firms Shift to Open AI

中国开源模型主导云端，Ollama报告Token消耗激增150倍 企业正将高达90%的Token用量转向用于自主编程的低成本开源模型，使闭源前沿实验室退居至窄范围的推理任务。 Jeffrey Morgan深入剖析了推理成本的下降、DeepSeek和Qwen等实验室快速发布的开源模型，以及自主编程智能体的兴起如何推动企业工作负载转向开放权重架构。尽管闭源前沿模型仍主导高杠杆推理，但预计开源模型将在日常运营和边缘计算中吸收高达90%的Token用量。Morgan详细阐述了Ollama作为连接模型权重、测试框架与硬件运行时的操作系统层所扮演的角色，同时回顾了自己的创业历程——从打造Docker Desktop，到在创业荒野中度过两年，直至推出Ollama。 自年初以来，Ollama Cloud的总Token消耗量激增了150倍，其驱动力在于开源模型达到了支持自主编程以及OpenClaw和Hermes等企业级智能体所需的能力门槛。 (3:03) 企业AI经济学正在趋于一种混合稳态：开放权重模型处理80%至90%的总Token用量，却仅消耗10%至20%的AI预算，昂贵的闭源前沿模型则退居为高层编排器。 (17:37) 虽然笔记本电脑本地推理在西方模型（Llama、Gemma）与中国模型之间平分秋色，但在用于自动化编程工作流的云端托管开放Token消耗中，Qwen、DeepSeek和GLM等中国开源系列占比接近100%。 (20:41) 随着闭源前沿实验室与开放权重之间的能力差距缩小至三个月以内，像DeepSeek Flash这样超高效的“flash”架构已成为Ollama Cloud增长最快的工作负载，以传统API极小比例的成本处理了约80%的日常编程任务。 (29:16)

Ollama co-founder Jeffrey Morgan argues that open-weight models are reshaping enterprise software economics by turning token generation into an abundant commodity, even as proprietary labs retain budgets for complex reasoning. The Budget Inversion: Token Volume Versus Dollar Spend (17:37) Enterprises initially adopt open models to cut proprietary API expenses, but cost relief serves primarily as a pathway toward internal customization, autonomy, and data governance. Morgan projects that enterprise usage will settle into an economic inversion where an 80% to 90% supermajority of token volume flows through low-cost open models, while 80% to 90% of actual enterprise budgets remain dedicated to frontier closed labs for top-tier reasoning and high-level orchestration. This dynamic mirrors historical computing and corporate patterns, such as senior law partners delegating discovery to associates, or systems routing specialized computations away from central CPUs. Rather than locking into a single closed garden, enterprises are deploying routing layers to delegate line-item execution and high-volume tasks to fast, open models while escalating difficult edge cases to closed frontier architectures. Autonomous Agents and the Rise of Efficient Flash Architectures (3:01) The expansion of open token traffic is largely driven by coding assistants and multi-domain coworkers like OpenClaw and Hermes. As context windows expanded to over one million tokens and agents adopted iterative loops—retrieving files, running commands, and replanning—single-task consumption by individual power users jumped to upwards of 15 million tokens, driving a 150-fold aggregate throughput increase on Ollama Cloud. To manage this demand, enterprise architectures are shifting away from monolithic 'God models' toward modular task orchestration powered by lightweight 'Flash' models, such as DeepSeek Flash. Because these models perform roughly 80% of routine programming tasks at a fraction of the cost, developers can decompose complex workflows into parallel subtasks handled by cheap, specialized workers without rationing token consumption. Hybrid Workflows Across Local Workstations and Cloud Clusters (20:42) Developer inference is settling into a split between local desktop execution and cloud clusters based on workload intensity. Routine document processing and bounded workflows execute locally on developer laptops at zero marginal token cost, while multi-step coding agents tackling architectural problems rely on cloud compute. On local runtimes, model origins remain evenly split between Western releases (like Meta's Llama and Google DeepMind's Gemma) and Chinese models, whereas Ollama Cloud's coding-agent workloads are overwhelmingly dominated by Chinese-origin open models. Hardware advances are facilitating a desktop inference renaissance. Hardware releases such as NVIDIA's DGX Spark (featuring 128 GB of unified memory) and desktop GB300 workstations enable engineers to run 20B to 120B parameter models—and daisy-chain hardware locally for 400B+ parameter engines—bringing latency down to the sub-100-millisecond threshold required for fluid development loops. Security Refusals, Provenance, and Supply Chain Governance (7:27) Open weights hold a categorical operational advantage in sensitive environments like offensive security and automated penetration testing. Strict alignment guardrails often cause frontier closed models to refuse legitimate vulnerability testing, prompting security researchers to turn to uncensored or abliterated open weights. In production environments such as power grid monitoring, enterprises balance privacy benefits against scrutiny over model lineage, training data mixture, and potential behavioral drift. Addressing enterprise concerns that foreign-trained foundation models might conceal backdoors, Morgan notes that enterprise security treats models like traditional open-source dependencies. Although deep neural networks are structurally complex, their runtime inference is deterministic, allowing organizations to deploy standardized evaluation suites and screening checks to verify behavior prior to deployment. Infrastructure Abstraction and Surviving Open-Source Monetization (11:31) Drawing on systems engineering backgrounds from Docker and VMware, Ollama positions itself as an operating-system layer mediating between low-level hardware drivers and fragmented application harnesses. While classic DevOps orthodoxy warned against occupying thin middleware layers between infrastructure and apps, Morgan argues that artificial intelligence inverts this rule: staying directly adjacent to developer workflows captures value by insulating engineers from upstream provider inconsistencies, undocumented API changes, and non-deterministic model outputs. Having experienced Docker's historical monetization hurdles, Ollama delayed monetization until open models matured enough to execute complex agentic workloads. Morgan highlights the danger of viewing an open-source user base as an anonymous community blob, stressing that enduring infrastructure companies must translate viral enthusiasm into concrete enterprise cloud workflows. While open models provide unprecedented cost efficiencies and local autonomy, production ecosystems remain challenged by hardware procurement volatility, rapid base-model version churn, and an uneven geographic distribution where large open-weight cloud models remain heavily dependent on overseas labs.

[0:00] Jeffrey Morgan: by far the largest pain point that open models can jump in and solve [7:27] if you try to get Claude to like pentest your product it will just refuse [17:37] Jeffrey Morgan: super majority of tokens [17:37] but the labs who own the models presume you don't want that [20:42] Jeffrey Morgan: not the lowest memory MacBook, but the second lowest memory MacBook you can buy from the store [23:12] recoloring the x-axis [42:15] Jeffrey Morgan: throw everything out—if we had to start from scratch and we were just joined right now, what would we do? [43:52] like meeting a rockstar [44:49] Jeffrey Morgan: Things kind of just started working and started taking off, and you're really as a founder just beside yourself because you can't totally explain why. [44:49] Homebrew Computer Club to broad computer adoption speedrun [47:02] Jeffrey Morgan: One of the biggest risks of having an open-source project that takes off is you consider your user base and customer base like your customer just a blob on the internet, which is a really risky way to think about customers because you want to meet them, figure out their needs. What are they doing? What do they want to do in six months? What's their story?

[0:00] cost is by far the largest pain point [0:03] that open models can jump in and solve. [0:05] But you know, every business has a [0:07] vision of getting better control over AI [0:09] and customizing it for their business. [0:11] And that's really their north star. You [0:13] know, cost is something they can solve [0:14] in the short term, but it then enables [0:16] them to then go and and customize these [0:18] models for their unique use case. [0:20] >> Early 2024, there was lots of interest [0:23] in fine-tuning your own custom models. [0:25] Then it sort of went away and all of [0:26] that it will just be wasted effort. [0:28] it'll get stomped by the next model [0:29] release. Seems like it's coming back [0:31] now. You have a front seat to all of it. [0:33] Do you think we're going through like [0:34] another cycle or is it here to stay this [0:36] time? [0:37] [music] [0:43] Welcome back to another episode of the [0:45] Lite Cone. Today we're talking to [0:47] Jeffrey Morgan, co-founder and CEO of [0:49] Olama, the easiest way to run open [0:51] source AI models locally and in the [0:53] cloud. Alama is used by 9 million [0:56] developers, has 178,000 GitHub stars, [0:59] and is used by 85% of the Fortune 500, [1:03] which means Jeff knows a lot about the [1:05] state-of-the-art of AI, what models [1:08] score highest on benchmarks, and what [1:10] developers actually download and keep [1:12] using. Jeff, welcome to the Lite Cone. [1:16] >> Thank you for having me. [1:17] >> We're down to here. Like, what is the [1:18] state-of-the-art? What are you seeing [1:19] out there? Well, I think the biggest [1:21] thing we're seeing is a shift to open [1:23] models, especially in enterprise and [1:25] that's from a mix of US and and Chinese [1:28] origin models and it's predominantly [1:30] driven by coding agents and also AI [1:32] assistants more co-work cases like [1:34] openclaw and Hermes [1:36] >> and because you sit in the token flow of [1:38] like so many tokens you have really good [1:41] data on what models people are actually [1:43] using and how it's changing. What are [1:44] the trends that you're seeing? Yeah, you [1:46] know, started as a way to run open [1:48] models on your MacBook or other [1:50] hardware, Nvidia, AMD, Intel, and um [1:54] earlier this year, we launched Alama's [1:56] cloud. And what we're seeing there is [1:58] that it's predominantly Chinese models [2:00] right now of Chinese origin. Um but [2:02] they're being accessed by businesses all [2:04] over all over the world. Um especially [2:07] US and and Germany is actually a big [2:10] source of where open model tokens are [2:12] being accessed. [2:13] >> Is it all about cost? Is it are [2:15] enterprises coming because they just [2:16] want to get the cost down or is there [2:17] anything more to it? [2:18] >> Cost is by far the largest pain point [2:21] that open models can jump in and solve [2:24] but you know every business has a vision [2:26] of getting better control over AI and [2:28] customizing it for their business and [2:30] that's really their north star. You know [2:31] cost is something they can solve in the [2:33] short term but it then enables them to [2:35] then go and and customize these models [2:37] for their unique use case. [2:39] >> Is there a particular uh large [2:40] enterprise that you can name that has [2:43] done this? I think there was a great [2:45] article in the information yesterday uh [2:47] from AT&T and it ends up they've already [2:49] shifted 40% of their token consumption [2:51] to open models and that's right now [2:54] predominantly through US uh and and [2:57] Europe models but they're also [2:59] evaluating the Chinese models. [3:01] >> What kind of workflows do they run? [3:03] >> Predominantly coding agents. I think [3:04] what we've seen just from the extreme [3:07] growth and you know per developer or per [3:10] user token usage has predominantly been [3:13] from coding agents and then earlier in [3:15] March and April we saw open claw take [3:17] off and subsequently the Hermes project [3:19] the Hermes agent project take off which [3:22] has then opened up that ability to [3:24] automate a huge chunk of work over a [3:26] long span of time to non-developers too [3:28] whether it's like finance or support or [3:30] marketing or sales. you had this [3:32] actually very cool graph on the takeoff [3:35] exponential for for OpenClaw. [3:37] >> Yeah. So earlier this year, you know, [3:39] this is a graph of token usage by [3:42] developer on Olama's cloud. You know, [3:44] the average amount of tokens they use [3:46] per week and we kind of [3:48] >> this is per developer. So like it looks [3:51] this graph looks like it should be an [3:52] aggregate of Lama's growth, but this is [3:54] actually the per user. [3:55] >> Exactly. [laughter] [3:56] >> This is on an individual user user [3:59] basis. how many tokens are they using a [4:01] week? And so there's kind of like two [4:03] big inflection points. One is that [4:05] initial runup at the start of the year [4:06] which was driven by coding agents. So we [4:08] saw Kimmy, the GLM models, Miniaax [4:11] launch. Finally, we had open models that [4:13] could power coding agents. And then in [4:15] April, we saw this incredible growth [4:17] from OpenClaw really, which was then not [4:20] just developers, but the rest of the [4:22] world could take a hard problem, give it [4:24] to an open model, and let it go complete [4:27] the task, which obviously consumes a ton [4:28] of tokens as it's figuring out what [4:30] tools to use, what data to go fetch. We [4:33] went from a context window of 128K to a [4:35] million plus with open models. And so [4:37] all that enabled this explosive growth. [4:39] So it went from uh roughly 5x so under [4:42] somewhere around I don't know 15 million [4:44] tokens before before all these core work [4:47] type of use cases. [4:49] >> I think that's about right for that open [4:50] claw jump we saw in April. Um obviously [4:54] in aggregate it's you know in the 10 to [4:56] 20x if not more as a whole through cloud [4:59] we saw 150x since the start of the year [5:02] and so it's just the big you know [5:04] interesting thing here is this huge [5:06] surge in demand for open models right [5:08] whereas open models I'd say in 2024 2025 [5:11] from the large models being served were [5:13] mostly being served as custom models so [5:15] you'd take an off-the-shelf model like [5:17] deepseek or kimmy and you'd fine-tune it [5:19] for your use case um like for example [5:21] you know cursor had famously done and [5:25] from there, you know, you could serve [5:26] that at scale. But seeing out of the box [5:29] open models being served, that really [5:30] only took off at the start of this year. [5:31] Even since we started this podcast, [5:33] these things sort of come in cycles [5:35] maybe. Like I feel like early 2024 there [5:38] was lots of interest in fine-tuning your [5:41] own custom models. Then it sort of went [5:42] away and it's like all that it will just [5:44] be wasted effort. It'll get stomped by [5:46] the next model release. Seems like it's [5:48] coming back now. You have a front seat [5:49] to all of it. Like what's do you think [5:52] we're going through like another cycle [5:53] or is it here to stay this time? [5:55] >> I think the release of these models the [5:57] cadence is only speeding up which makes [5:59] it ever more harder to you know stay on [6:02] top of that and have a a post [6:04] >> you mean the the the latest closed [6:07] frontier models are releasing faster [6:08] than ever. I [6:09] >> think on the open source side it's [6:10] getting faster and faster. You know this [6:13] summer we've already seen three [6:14] iterations of the Deepseek Flash model [6:16] as an example. um what used to be more [6:19] of a six-month cycle. [6:20] >> Yeah. So the gap is like closing [6:22] essentially [6:22] >> closing and I think that makes it even [6:24] harder to custom train models. On the [6:26] flip side, I think the tooling is [6:27] getting better and so it allows teams [6:29] that want to fine-tune their models to [6:31] stay on top of it. [6:32] >> I mean, we're also entering this moment [6:33] where AI safety is becoming more and [6:35] more of an issue at uh the Frontier [6:37] Labs. So the frontier uh may well slow [6:41] down to figure out its alignment and [6:42] containment issues. And then meanwhile [6:45] the open source models and openw weight [6:47] models are continuing to grow and get [6:50] better. [6:51] >> Yeah. And you know we saw the [6:52] announcement and the release of the [6:54] GLM53 model and its capabilities from a [6:56] cyber security standpoint you know being [6:58] extremely impressive. it creates a big [7:01] opportunity for whether it's startups or [7:03] existing businesses in the security and [7:05] governance space to really jump in and [7:06] help because I think if you look at you [7:07] know that AT&T article I was speaking [7:09] about it largely the the blocker to [7:12] adopting open models is largely around [7:14] security and safety [7:16] >> um but from our experience talking to [7:17] customers whether it's in Europe whether [7:18] it's here in the US if you can solve the [7:20] safety problems by and large adopting [7:22] the Chinese origin model labs is [7:25] completely on the table and it's really [7:26] exciting for these businesses [7:27] >> there's sort of this interesting moment [7:29] right now where a hugging face had to [7:31] use openweight models to actually even [7:33] detect the hack from the frontier. [7:36] >> A common question we get is like well [7:38] where where can I use open models that [7:40] are leaps and bounds of an advantage [7:42] over uh using a frontier closed model [7:45] and one of the key use cases is security [7:48] testing and making sure that you know [7:50] your software is secure. [7:51] >> Yeah. Because like if you try to get [7:52] claw to like pentest your product it [7:55] will just refuse to do that. [7:56] >> Correct. Yeah. By and large. Whereas [7:58] there are literally obliterated uh [8:00] security researcher models that you can [8:03] find on hugging face that allow you to [8:04] do it. [8:05] >> That is true. There are ones that are [8:06] are you know customtrained to be you [8:08] know even more you know liberal to to go [8:10] and attack these problems. But even the [8:12] out of the box models um they they do [8:14] come with safety training but they're a [8:16] little better understanding if you're [8:17] doing this for you know a good you know [8:21] use case versus versus one that's more [8:24] of a negative or you know malicious use [8:26] case. A cool thing about is that because [8:28] you guys are such a key distribution [8:31] channel for these models, my [8:32] understanding is that typically the [8:34] model developers are contacting you [8:36] before the general release to like [8:38] coordinate launches and stuff like that [8:40] and you often get sort of like previews [8:41] of what's about to about to happen and [8:44] you get these incredible growth spikes [8:45] when a new model drops. Like I wonder if [8:48] you could like tell us a bit about like [8:50] what it's like to operate this thing at [8:51] scale. [8:52] >> Yeah, absolutely. And like I said [8:53] before, the models are coming out faster [8:55] and faster and faster. And so we've [8:57] developed a playbook to what does a [8:59] successful day zero model launch look [9:01] like? And there are a lot of things to [9:02] get right. There's making sure that it's [9:04] supported in your favorite inference [9:06] engine, which is generally sometimes a [9:08] multi-week process to make sure it's [9:09] fast, to make sure it's accurate, to [9:10] make sure it's up to spec with the [9:12] reference. There's also finding the [9:14] right use cases and harnesses for [9:16] developers and users to make use of this [9:18] model. Generally, these models have new [9:19] capabilities. This morning, you know, [9:21] there's announced Deepseek launched [9:23] their first multimmoal model um from a [9:25] large model LLM standpoint. They had [9:27] previously had some smaller OCR models, [9:29] which unlocks a whole bunch of use [9:31] cases. But what they also changed was [9:33] the Deep Seek harness, which launched [9:34] recently so that it could support this [9:36] capability. So step two is then to find [9:38] the harnesses, make sure they're [9:39] prepared to actually run this model and [9:41] to do that effectively. But you know, [9:43] every model's different. They all have [9:45] different challenges and architecture [9:47] changes and tool calling mechanics and [9:50] getting all this right is really hard. I [9:53] think the key thing to do at the end of [9:54] the day is to run benchmarks against the [9:56] final product ahead of release and make [9:58] sure that you know it's it's running as [10:00] as it as the research team at the lab [10:02] specified. So I think one very important [10:05] part that you play in the whole [10:06] ecosystem is you kind of create a very [10:08] legible standard to be able to make sure [10:12] you have the best way to use the [10:14] hardness for each new model each new [10:16] tool call and all of it is consistent [10:18] across all the different models which is [10:20] pretty hard to do. Yeah, I think there [10:22] are three things really that you know we [10:24] try to package together. One is a [10:26] harness for example, you know, whether [10:28] it's an off-the-shelf harness or an SDK [10:30] to help use the model. Some existing [10:32] harness that is designed for this and [10:34] and what's great now there's so many [10:36] great open source harnesses. Um the [10:38] codeex harness is open source. Uh open [10:41] code's a great one and one of the most [10:42] popular harnesses from OAMI users. Step [10:45] one's getting that right, but then you [10:46] got to package it with the model, making [10:47] sure that it's, you know, available, [10:49] it's reliable, if it's in the cloud, [10:51] that there's enough capacity for it [10:53] because day zero tends to be the largest [10:55] growth day obviously. And then [10:57] importantly, there's the hardware and [10:59] the providers. And that's actually where [11:01] there's a lot of collaboration to be had [11:02] whether it's like an inference provider [11:03] optimizing the model or you know with [11:05] some of our partners whether it's Nvidia [11:08] or or you know for example working with [11:09] the Apple silicon stack making sure that [11:12] it actually runs the model really fast [11:14] because if the model's capable but it's [11:16] really slow that's not a great [11:16] experience. So getting those three [11:18] things packaged into a box and generally [11:20] you get the model you're lucky if [11:22] there's a you know the ability to access [11:24] the model a few weeks in advance. A lot [11:26] of this stuff comes together in the last [11:27] 24 hours before the model gets released [11:30] and so it's generally a fire drill. [11:31] >> So you're kind of almost a like an [11:33] operating system in the old world where [11:34] you needed to really integrate very [11:36] tightly with all the drivers, all the [11:38] hardware and then at the application [11:40] level to make sure that all the apps [11:41] were really tuned up well and you are [11:44] the glue for all of it. Right? I do [11:46] think the OS, which is generally a [11:48] cliche analogy to use, is a good one [11:50] because you've got the drivers for the [11:52] hardware and the providers and the [11:53] inference layer, but you also have the [11:56] application runtime and making sure that [11:57] the harness works. Gluing that together, [12:00] it's a very combinatorically large [12:03] problem to solve. And so doing that well [12:05] is really difficult. And so, but you [12:07] know, over time you develop pieces that, [12:10] you know, allow you to quickly develop [12:12] that, test it, release it, um, and kind [12:15] of have this common runtime that can [12:17] match any harness to any model. And [12:19] that's kind of the role we're really [12:20] playing for developers. [12:22] >> You guys are very hardcore engineers and [12:24] you fine-tune things all the way to from [12:27] the origins with Apple silicon all the [12:29] way now to DJX. How do you build such a [12:33] deep technical bench with that? I think [12:35] a lot of our team, you know, we aren't [12:36] AI researchers by background. We're from [12:38] VMware and Docker, um, and from, you [12:41] know, other networking companies. And so [12:43] the classic compute problems are kind of [12:46] reinventing themselves in the inference [12:48] land, whether and and so largely, you [12:50] know, that's where we like to focus our [12:52] time. But I think at the end of the day, [12:54] you know, it all comes down to the [12:55] developer experience. What is it like [12:58] when the developer makes a call to the [12:59] API and gets tokens back? What happens? [13:01] And it there's more and more happening [13:03] in that layer right now. And getting [13:05] that right needs all the layers of the [13:06] stack to work well together. And I think [13:08] you know one of the challenges with open [13:10] models has been that hasn't been [13:12] happening at the rate of what a frontier [13:14] model lab puts out where they have you [13:17] know the classic five layer cake right [13:18] that Jensen mentioned which is like the [13:20] apps you have you know the model you [13:23] have the infrastructure and inference [13:25] you've got the chips and you got the [13:27] energy and they've got all that ready to [13:29] go for developers on day zero. And [13:30] that's really the thing that we're [13:32] trying to reproduce for open models. Of [13:34] course, we're not going to do every [13:35] layer of the stack, but we can help [13:36] orchestrate that. [13:37] >> You layer the lay layers. [13:39] >> Yeah. And maybe in open models, there's [13:41] more than five layers. Like that model [13:42] layer actually has a lot to it, right? [13:44] There's the model weights, but there's [13:45] also a lot of the orchestration [13:47] components that each model is uniquely [13:49] good at. There's this developer API [13:51] layer. There's so many opportunities to [13:52] build between the model and the [13:54] application layer that are kind of [13:55] hidden in today's, you know, five layer [13:57] cake stack. [13:58] >> That's interesting. Do you do you think [13:59] any of those like hidden layers might [14:01] get unbundled and become their own [14:03] companies or providers? [14:06] >> Absolutely. There was a really good talk [14:08] from some of the anthropic team, the [14:09] platform team, and they talked about [14:11] three big things. One was knowledge. How [14:13] do you connect your company's data and [14:14] context to the model? One is [14:16] coordination. So, as as you know, when [14:18] you make a request to in your cloud app [14:21] or codecs, it goes off and spins out a [14:23] bunch of sub aents, some of those in the [14:24] cloud, some locally, there's a [14:26] coordination problem. Um and then lastly [14:29] there's an execution problem which you [14:30] know we talk a lot about as sandboxes [14:32] but these agents more and more of them [14:34] are moving to the cloud. There's a huge [14:36] compute problem to be solved there. And [14:38] if you look at what's happened [14:39] classically and in the cloud business [14:42] open source has meant that there are [14:45] best of breed companies for each of [14:46] those problems. Whereas you might have [14:48] had kind of the if you look back to the [14:50] original generation cloud products you [14:52] have like the Herokus of the world you [14:54] have Google app engine all those things [14:56] were bundled together. But what [14:57] developers ended up preferring is best [14:59] of breed products for each one. [15:00] >> It's the nalism which is uh all things [15:03] are just bundling or unbundling like the [15:05] frontier model frontier model labs want [15:07] you to be totally in their walled garden [15:09] of manage agents and their context their [15:12] memory layer. And then meanwhile little [15:14] tech and all the founders out there and [15:16] all the open source developers uh don't [15:18] want to be caged in so we're going to [15:19] make all this other stuff and it'll [15:22] it'll be an interesting moment to figure [15:24] out like what ends up winning. I mean [15:25] it'll probably be some mix of both. [15:27] >> I think so. And you know you've got this [15:29] abundance of open model tokens that's [15:31] being created. There are just dozens of [15:32] open model providers that are able to [15:34] serve these tokens. The new scarcity the [15:36] problems now are what's above the tokens [15:39] right? You know how do you orchestrate [15:41] an agent from you know A to B. These are [15:44] problems that are have tons of new you [15:47] know systems and and engineering [15:49] problems that are just really hard to [15:50] solve for an individual dev. Like [15:52] there's no way they're going to build [15:52] all those layers. Well, the interesting [15:54] thing now is um because the coding [15:56] agents themselves are getting a lot [15:57] better and ostensibly this is the worst [15:59] the models will ever be. The classic [16:02] reason why there was a moat here was it [16:04] was just too hard to have really [16:06] well-maintained software that was [16:08] properly tested that actually satisfied [16:10] user need. And what if that goes away? [16:14] Like we're literally at this moment [16:16] where actually maybe the a you know [16:18] you'll just have a curron and it runs a [16:20] markdown file in some TypeScript and [16:22] it'll just you know there's no lock in [16:25] anymore right like you could be using [16:27] OpenAI's memory system one day and then [16:30] actually you know you could have an [16:32] agent be uh constantly syncing that [16:35] against your own memory system and [16:38] actually it just works it's fine like it [16:39] works over MCP like there's all you know [16:42] there's plenty of like runtime testing [16:44] and then there's no lock in. [16:46] >> Yeah, I think a lot of the, you know, [16:48] what we know of as harnesses today, a [16:50] lot of those pieces will go down into [16:51] the model. But, you know, as you kind of [16:55] push a lot of the core loop of the model [16:58] and hooks down to the model itself, [17:00] there are these pieces that kind of come [17:02] out like memory is a great one. And the [17:04] the general guidance we're using is if [17:07] it's a stateful problem like there's [17:08] storage involved that's something that [17:11] you know in the end can't go into the [17:13] model because the model's trained and [17:14] it's you know as we know training runs [17:17] are now happening on like a monthly [17:19] basis but it's still not up to date with [17:21] the latest data. So generally storing [17:23] data is like a huge problem space that I [17:25] don't think will ever make its way down [17:27] into the model layer [17:28] >> like managing credentials and that kind [17:29] of stuff isn't [17:30] >> security credentials safety um open [17:33] models don't have all the safety tooling [17:35] that closed model providers give you out [17:37] of the box but that's super important [17:38] especially for businesses to adopt them. [17:40] I'm curious where you see sort of the [17:41] the end or future state for enterprise [17:44] on the balance between sort of frontier [17:46] closed models and open source model um [17:49] especially on spend like my it feels [17:51] like you know initially it was just the [17:54] everyone's just like allocating all of [17:55] their budget to anthropic or um open AI [17:59] uh my sense now is yes open a open open [18:02] source is clearly growing but like so [18:04] are so is like anthropic spread so like [18:06] the things seem to be growing together [18:08] like does that continue or Do you think [18:10] there's sort of like a steady state [18:11] where it's like I don't know like half [18:13] the budget's going to be on the closed [18:15] source frontier model and half's going [18:16] to be in open source or or something [18:18] different? [18:19] >> The super majority of tokens and this is [18:21] our take it will be open models within a [18:23] business. Call it 80 90%. That doesn't [18:26] mean 89% of the the budget will go to [18:28] open models. In fact, I think what the [18:30] open model community is doing incredibly [18:32] well together is lowering the cost to [18:34] make it more accessible. And so maybe [18:35] you'll only pay 10 to 20% uh of of the [18:38] cost towards open models, but your token [18:40] >> most of your tokens will be going [18:42] through the open models, [18:43] >> right? Which will enable a whole bunch [18:44] of use cases on top because you have [18:45] this abundance of tokens. You're not [18:47] thinking about taking away token access [18:49] from your team. You're giving more and [18:50] more access. I think for the hardest [18:52] tasks that's reserved for these frontier [18:53] labs where a lot of the best researchers [18:55] are. And then from there, there's a [18:57] whole bunch of problems in the middle, [18:58] right? Where maybe it's a combination of [19:00] open and closed models working together. [19:02] I think the steady state is that most of [19:05] the software is most of the models are [19:07] open. It's kind of an interesting idea [19:08] because as the models get more powerful, [19:10] ideally you just want to delegate to [19:12] like your smartest model to figure out [19:14] when to go to like an open model, but [19:17] the labs who own the models presume you [19:19] don't want that. [19:20] >> And I think look, I think that all the [19:22] labs are aligned in many ways to one [19:23] thing, which is how do you serve the [19:25] customer? And I think it will be up to [19:26] the customer to decide if I have a [19:28] router where some of the scheduling and [19:30] harder you know orchestration happens [19:32] through a frontier model so be it. But a [19:35] lot of the kind of [19:38] line item work can happen through open [19:39] models and the collaboration of the two [19:41] together. I think we've seen a ton of [19:43] projects whether it's from Sakana AI or [19:44] open router that have combined the two [19:46] and it's seen really good results. [19:47] >> This is not too dissimilar from a human [19:50] organization, right? Like you have you [19:52] know like a law firm there's like a [19:53] partner and then there's like a bunch of [19:54] associates and like the partner farms [19:56] out the work to the the associates, [19:57] right? It's like the same. [laughter] [19:59] >> We saw the same thing with cloud [20:01] computing where it was really a blend of [20:03] proprietary software. some of them [20:05] provided by the the cloud providers [20:06] themselves. For example, you know, AWS [20:08] had Dynamo DB which was kind of their [20:10] proprietary scale out database. But then [20:13] a lot of customers use that in [20:14] conjunction with Postgress DB and in the [20:17] end you know what we see is customers [20:18] will will use a combination of the two. [20:20] >> I think it's a very common pattern. This [20:22] is also the same design for why the [20:24] Apple silicon is actually more superior. [20:26] the special spe special special [20:28] accelerators for different kinds of [20:29] workloads for let's say image processing [20:32] supposed to audio that's been or even go [20:35] way back in in in the PC era you had [20:38] like your standalone audio card right [20:40] graphics card and all that [20:42] >> and speaking of Apple Silicon should we [20:43] talk about local models because you're [20:45] you're in a bit of a unique position [20:47] because you have large businesses both [20:49] in cloud hosted models and locally [20:51] hosted models that will run on your [20:53] laptop what are you seeing in those two [20:55] worlds and What what do you think is [20:56] going to happen? [20:57] >> I think it's incredibly exciting because [20:59] it's similar to the closed versus open [21:01] model question. It'll be a mix in our [21:04] mind and and that's what we hear from [21:06] customers as well where for easier tasks [21:09] you could run them locally and with [21:10] lower latency and of course lower cost [21:13] when it comes to the per token cost. [21:15] Ultimately, you're buying hardware up [21:16] front and you'll use that in conjunction [21:18] with these cloud models. What's exciting [21:20] about this next generation of hardware, [21:22] which we've had for a few years now, is [21:24] just how good they are at running the 20 [21:27] billion parameter to 40 billion [21:29] parameter range of models. Sometimes up [21:30] to 128 billion parameters. [21:32] >> Yeah. Quinn 3.8 38B is now as good as [21:36] Opus 4.6 for coding. Is that right? [21:38] >> That's what the benchmark show. [21:40] >> It's incredibly exciting because you can [21:42] run that on not the the the lowest [21:45] memory MacBook, but the second lowest [21:47] memory MacBook you can buy from the [21:48] store. So it's incredible. [21:50] >> And are you seeing your users do that? [21:52] Like what are you seeing people use the [21:53] local models versus the cloud hosted [21:55] models for in practice [21:56] >> from the side of which models they're [21:58] running? We're seeing a really solid mix [22:00] of US and Chinese trained models being [22:02] used for local and we have the [22:04] incredible models from you know the [22:06] original llama models of course but also [22:08] the Gemma models from DeepMine. These [22:10] are great choices for local. But when it [22:12] comes down to use cases coding agents by [22:14] and large are most effective with the [22:16] large cloud models. You're solving [22:17] really hard problems. you're writing [22:19] code tests, it's really difficult uh [22:22] versus some of the document processing [22:24] workflow use cases that run extremely [22:26] well locally because they don't have as [22:28] difficult of a of a task in the [22:30] endto-end, you know, problem you're [22:32] trying to solve. And so that's where we [22:33] see this hybrid execution model where [22:36] some of the easier, more straightforward [22:37] tasks run locally and then you have a [22:39] router that can help decide, hey, we [22:41] need to go to a large cloud model for [22:42] this. And I think what that means for [22:44] customers is that you're you're really [22:45] dropping the costs even further when [22:49] you're going from open models. Not just [22:50] because they're cheaper to run in the [22:52] cloud, but because now you can run them [22:53] effectively for free on the hardware [22:55] you're buying for your business anyways. [22:57] And what we're seeing ultimately from [22:58] the cloud coding agent models is it is [23:01] predominantly Chinese models being [23:03] consumed today. And for the local [23:06] models, it's a really strong blend of uh [23:09] US, Europe, and and Chinese origin [23:11] models. Yeah, this these two graphs are [23:13] pretty stunning in comparison. Like [23:14] basically for local models, the US and [23:16] Chinese models are neck andneck. We're [23:18] like tied. And for cloud hosted models, [23:20] like the US is like recoloring the [23:23] x-axis. It's like 100% Chinese models. [23:26] Basically, we need more US labs to make [23:28] large models. Is is is is that what this [23:30] graph is showing [23:32] >> effectively? And you know, with the [23:33] launch of the Neotron Ultra model, we're [23:35] seeing kind of the first wave of that [23:37] and it's really exciting. Nvidia as a [23:39] company is so interesting because um [23:42] their moat is not like trying to start [23:44] new software businesses or sell you know [23:47] tokens. They seem to be quite interested [23:50] in just releasing a lot of open- source [23:53] and helping the ecosystem and then the [23:55] fact that they do that then helps them [23:58] stay ahead of the game on the hardware [24:00] side. [24:00] >> I think so. And you know ultimately [24:03] Nvidia what's so incredible is there's [24:05] helping power an ecosystem around open [24:06] models whether that's the hardware the [24:08] models you know we've seen the uh new [24:11] DGX station uh computers that they're [24:14] working on which have GB300 on your [24:16] desk. [24:17] >> How do we get on that list that isn't [24:19] deafening loud? [24:20] >> Yeah. Do you know the price point on [24:22] that thing yet? [24:23] >> I don't know it off the bat. [24:25] >> Gary's buying it. [laughter] [24:25] >> I know, right? Well, I looked it up. [24:27] It's like I mean you can probably run a [24:29] Frontier model for like I mean very [24:31] slowly for like two two $300,000. Is [24:35] that is that right? [24:36] >> I think it's even more competitive than [24:37] that. Yeah. [24:38] >> And you can run more than a Frontier [24:40] model at high speeds at a price point [24:43] that isn't very far off what you can buy [24:46] from a classic workstation computer. [24:48] >> Oh, no way. [24:48] >> You think about, you know, quite a few [24:50] of the customers we talked to, some of [24:51] them are banks, for example, or [24:52] industrial businesses. They already have [24:54] these Nvidia workstation GPUs in every [24:57] single engineer's desk. Some of them [24:59] tens of thousands of them. And so this [25:01] is one. [25:02] >> They've been selling them for all the [25:04] CAD work. [25:05] >> Well, this is for this is for the kind [25:06] of original RTX A6000s, but this is the [25:09] next generation. [25:10] >> Yeah. Yeah. I'm going to have to email [25:12] Jensen. [25:12] >> This was at this was at GTC. So, what we [25:14] see here was at GTC. And we were one of [25:16] the first uh people along with Elon and [25:19] a few others to receive the DJX Spark as [25:22] well [25:22] >> which sits on your desk and provides [25:25] >> you know 128 GB of unified memory to run [25:28] that kind of 20 to 120B model range but [25:32] obviously that's just the beginning of a [25:33] whole new range of hardware that can run [25:34] the biggest models. [25:35] >> So did you say you can buy a bunch of [25:36] these and chain them and actually run a [25:38] 400B model? [25:40] >> You can absolutely. They they have this [25:42] really fast network link and so you can [25:44] stack them on your desk almost like a [25:46] miniature uh data center rack and [25:49] >> people do with the Mac minis. [25:51] >> Absolutely. [25:51] >> Now this is like the production version [25:52] of it. [25:53] >> I think that's what's so exciting is [25:54] you're seeing both from Apple and Nvidia [25:56] this incredibly incredible leap to next [25:59] generation hardware that's built for [26:01] these models and is effective at running [26:03] them. So you'd say like this is this is [26:05] the platform to get like you could make [26:07] stu apple studios work but like if you [26:10] want something that just can work get [26:12] DGX spark [26:13] >> from our testing both are very [26:14] competitive. Got it. [26:15] >> So I think a lot of it come down to what [26:17] you can get and and then also um the [26:20] tech stack you're looking for. I think [26:22] there's an incredibly uh mature tech [26:24] stack through the MLX project with Apple [26:26] where they've done some amazing work to [26:27] run LLMs on the Mac Studio but also the [26:30] smaller Macs and of course the DJX Spark [26:32] stacks. just incredible. We're super [26:33] excited uh as partners with Nvidia for [26:35] that. [26:35] >> I think it's going to be a cool [26:37] renaissance for personal desktops. [26:40] >> I think so. And you know, it's it's [26:42] funny with Olama's journey. We started [26:44] local. Clearly, the coding agent demand [26:48] is in the cloud, but that's going to [26:50] come back locally in our minds because [26:52] the hardware will catch up. When you [26:54] have a GB300 on your desk and you want [26:56] the fastest coding loop, that's as fast [26:58] as running your tests or as fast as [27:00] making code editor change. We all [27:01] remember the GitHub copilot experience [27:02] of having the autocomplete come up in a [27:04] few millisecond at 100 milliseconds. [27:06] That experience will make its way back [27:07] to the desk which is which has been a [27:09] journey of starting local going to the [27:10] cloud and then we think that'll come [27:12] back local and you'll end up using the [27:14] two together. [27:15] >> Speaking of like what you can get in [27:18] order to run a Llama cloud, you need [27:20] like a ton of GPUs. What are you [27:22] seeing in the GPU market? [27:24] >> I think what we're seeing is ultimately [27:25] the prices are changing very quickly and [27:27] the supply and demand volatility is is [27:30] very high there. And so, you know, I [27:32] think ultimately [27:35] for if you're a startup, getting access [27:37] to some of the B200, B300 GPUs you need [27:40] to run these latest models is very hard. [27:42] Thankfully, there's a great set of [27:43] inference providers building on top of [27:45] that. And so we're seeing this extreme [27:47] demand on like you know and and when [27:50] >> are you able to get all the GPUs that [27:51] you need? Are you constantly like like [27:54] growth limited by how many GPUs you can [27:56] get your hands on? What's the what's the [27:58] current state? We're lucky that we've [27:59] partnered with quite a few providers to [28:01] work together to pull a bunch of GPUs [28:03] together, which allows us to stay on top [28:06] of our our demand, but that's a lot of [28:08] work and it's definitely a lot of [28:11] spending time thinking through, you [28:13] know, which model will get run where, [28:15] how fast should it be, which region is [28:17] it in, um what will the latency be for [28:19] the customer? There's a lot of hard [28:21] problems to solve in that stack. And I [28:23] think what's really exciting about [28:25] products like Open Router, Alama, the [28:28] Open Code project is for an end user [28:30] developer, they can sign up and get [28:32] access to this without having to go [28:33] negotiate prices on a B200, B300. You [28:36] know, think about their 24-month [28:38] forecast in order to get access to some [28:40] of these these uh you know, GPUs. [28:43] >> YC's Next Batch is now taking [28:45] applications. Got a startup in you? [28:47] Apply at y combinator.com/apply. [28:50] It's never too early. and filling out [28:52] the app will level up your idea. [music] [28:55] Okay, back to the video. [28:56] >> Suppose you were like a a startup [28:59] founder and you were just starting out [29:00] now and you're building some AI company [29:03] and you haven't raised a lot of money [29:04] and so you like want to like use as many [29:07] tokens as possible like inexpensively [29:09] like what would your advice be to that [29:11] person about like how they can get like [29:13] huge mileage with like a limited budget? [29:16] There's this new class of models like [29:18] DeepSeek Flash is a great example and I [29:21] think there'll be quite a few more where [29:22] it's ultra low cost per token. It's also [29:25] low cost per task which is a really [29:27] important metric and that class of [29:30] models in my mind will be the first ones [29:33] that come down to this idea of like [29:34] unlimited tokens. We all remember Chad [29:36] GBT. You didn't really have to think [29:37] about how many tokens you were using. [29:39] You would just use it every day. You had [29:40] unlimited. Ultimately, I think we return [29:42] to that, but it's going to take a lot of [29:44] work in the model, the architecture to [29:46] be customtrained for high volume token [29:48] usage. And if you think about the start [29:50] of the year, we really wanted open [29:51] models got to the frontier of [29:53] intelligence. We bridge the gap. We're [29:54] maybe like less than 3 months behind [29:56] between the frontier closed models and [29:58] the open models. But the next problem to [30:00] solve is extreme efficiency. Seeing, for [30:03] example, the GBT Luna model become very, [30:06] very price effective for customers has [30:08] been a huge boom. We talked a ton of [30:09] customers where that kind of pricing [30:11] enables widespread adoption within a [30:12] team. I think we're going to see that [30:14] with open models. We already are seeing [30:15] that with open models. I think the [30:16] Deepseek flash model is leading that [30:18] charge. If we go back to the model [30:21] breakdown on OAM's cloud, the highest [30:24] growth area is definitely the Deepseek [30:25] model. And this is largely powered by [30:27] the Deepseek Flash uh adoption. And so [30:30] this new class of flash models where [30:31] they're good enough for 80% of the [30:33] tasks, they're really fast and they're [30:35] ultra cheap. this new class of model [30:37] that I think will enable some of those [30:39] use cases. [30:40] >> Yeah. Those are going to be like the [30:41] workhorse models to do like all the [30:42] grunt work. [30:43] >> Exactly. [30:43] >> Yeah. You won't have to be thinking [30:45] about how many requests am I making, how [30:46] many tokens. You'll be much more [30:49] inclined to consume as much as you can [30:51] because you know it's able to solve [30:54] >> the hardest the not the hardest but you [30:56] know difficult problems. If you think [30:58] back to the coordination layer we were [31:00] talking about earlier too. Being able to [31:02] coordinate these flash models together [31:03] to do different tasks can also yield [31:06] great results that a bigger model can. [31:09] So by having these cheaper models, not [31:12] only are they more accessible, they can [31:15] run faster and you can access them in [31:17] higher volume, but you can start to [31:18] chain them together and build new [31:20] problems that are solved by [31:21] orchestration on top. And that's a [31:22] really exciting area for new startups, [31:24] for existing inference providers, um for [31:26] some of the larger businesses today that [31:28] solve workflow problems. ultimately [31:31] being able to chain these models [31:32] together is going to be super helpful [31:33] and you won't have to think about the [31:34] underlying costs. [31:35] >> Yeah, I guess you know when we first [31:36] started talking about AGI even on this [31:39] podcast um there was this sort of debate [31:41] about you know and I think a lot of AI [31:43] researchers would come out and say like [31:45] there's just going to be a giant god [31:47] model and it's going to do everything. [31:48] But you know I think so far like it [31:51] hasn't quite worked out that way. Like [31:52] obviously you still have you know if [31:54] you're if you have to literally hack the [31:56] NSA maybe you need mythos or something [31:59] but um for the majority of use cases [32:02] like you're talking about orchestration [32:04] and you're talking about um like smaller [32:06] models you you you know that the task [32:09] composition actually probably gives you [32:11] a bunch of ways to make it more [32:12] repeatable. It's more trustworthy like [32:14] it actually does work at a cost that is [32:17] like possible. So, you know, if it was [32:19] going to be um God model versus like [32:22] lots of, you know, smaller special [32:24] purpose or even just like simpler [32:26] models, um it's turning out to be the [32:28] latter. So far, [32:30] >> I think for most customer use cases, [32:32] there's a you know level at which a [32:35] model becomes good enough and then they [32:37] can continue using that level of [32:39] intelligence. Maybe the model will get [32:41] faster, it'll have better architecture, [32:42] it'll be'll have new capabilities, but [32:44] they won't have to reach for the god [32:46] model. But I do think there are use [32:47] cases where the most powerful models [32:50] unlock them and that'll continue to be a [32:52] thing and it'll be really exciting, you [32:54] know, and sometimes scary as well on on [32:56] what they can do. But for the [32:57] run-of-the-mill use cases where open [32:59] models really shine, I think that's [33:00] where, you know, we're we're hitting a [33:02] point where, you know, you're not [33:04] solving necessarily the hardest problems [33:06] within the business, but they're hard [33:07] enough where it's now unlocked by open [33:09] models. Will there be an open model that [33:11] becomes a god tier model? Um I think [33:14] it's possible and we're seeing you know [33:16] really exciting developments from you [33:18] know Zoo AI and GLM where some of the [33:21] tasks they are frontier and we all saw [33:23] with the Kimmy model how for web [33:25] development it became the best model and [33:27] that sent this new shock wave across the [33:29] market which is it's less about a gap [33:31] and it's more about a head-to-head [33:33] competition which I think makes all of [33:35] this even much more exciting. [33:37] >> This is a bit of a sensitive question [33:38] but what do you think about this and the [33:40] geopolitics around it? I think you know [33:42] a lot of the geopolitical angles around [33:45] this um start with you know where the [33:48] where the model's from and the more we [33:51] spend time with customers and users a [33:53] lot of it's actually how the model's run [33:55] where it's run how it's run is it run in [33:57] a secure environment um and that starts [34:00] to matter a lot more um but I do think [34:03] you know look it it's super important [34:05] that a customer in the US can use a [34:08] model trained in the US and we they have [34:11] two kind of classes of of customers we [34:14] speak to. One is they don't really care [34:15] where the model's from. They care about [34:16] where it's run. But for every one of [34:18] those, there's, you know, a customer [34:20] that's saying, I really care about where [34:22] the models from because it's data the [34:24] way it it's not even just a a security [34:26] issue as much as how does the model [34:29] speak. You know, we all go through mo [34:31] and and communicate. We all go through, [34:33] you know, a lot of the models go through [34:35] phases where they sound more robotic, [34:36] they sound more friendly, and a lot of [34:38] that matters, too. Um but [snorts] I [34:40] think the highest order bit is obviously [34:42] making sure that you have a model that [34:43] end to end you understand where the data [34:45] is from which is great from the neotron [34:47] models you can go and introspect what [34:49] made this model because if you're [34:50] putting in a mission critical task which [34:52] people are absolutely using open models [34:54] for mission critical tasks there's a [34:56] post online about how llama powers the [34:58] analytics of a power plant to detect [35:00] surges in Finland to make sure that the [35:02] lights stay on. That's where these [35:04] models the the model origin really [35:06] matters [35:07] >> for for like critical [35:09] tasks like that. How do you ensure that [35:12] a Chinese model [35:14] even if it's hosted in the US isn't [35:17] basically like booby trapped to like [35:18] cause problems? [35:19] >> Yeah, the menuranian candidate problem. [35:21] >> Exactly. [35:22] >> Have there any been any like known cases [35:24] there? The menurian candidate yet? [35:26] >> I think not that I can think of off the [35:28] top of my head. I haven't heard. I feel [35:29] like I would have heard about it, but [35:31] >> you know what you don't see a lot on [35:32] some of the press articles is how robust [35:35] some of the the IT and security teams [35:37] are at the businesses that we know of, [35:40] the top, you know, Fortune 500 [35:42] businesses, they're really used to this [35:44] already because open source software, if [35:46] you think the average application has [35:47] thousands of dependencies. This is like [35:49] isn't a new problem and all it takes is [35:51] one dependency for there to be a major [35:53] security issue in the entire [35:54] application. [35:56] >> Clain poisoning is insane. [35:57] >> It's a thing. It's been a thing for [35:59] decades and it's not new in that sense. [36:03] It's a little more opaque because you [36:04] can't like dig into the model. It's [36:06] >> it is deterministic [36:07] >> but it's deterministic and if you screen [36:08] the model properly with safety checks by [36:11] and large at least what we're hearing is [36:12] from customers is that can be solved. [36:14] >> Do you want to talk about the origins of [36:16] Lama? you you know you guys came up [36:18] through the Docker ecosystem and uh a [36:22] lot of people watching you know would [36:23] love to be in the position you're in [36:26] where you have this sort of enduring [36:27] brand moat that looks like it will [36:30] extend for you know really till the end [36:32] of time. No just it's it's a very [36:35] powerful situation to be in. um you [36:38] basically found yourself on top of a [36:40] giant oil well, right? Um for those out [36:43] there wildcatting, you know, can you [36:45] tell us that story? You you were [36:47] actually working with Jared in uh 2021. [36:51] >> Yeah, you know, my my co-founder and I [36:53] previously built Docker Desktop while at [36:55] Docker. So we really got a understanding [36:58] of like what makes a great developer [36:59] experience. But I have to say the first [37:01] few years of Ola as a company was really [37:04] in search for what's the right problem [37:06] to solve with this muscle we've built of [37:09] trying to design a great experience for [37:11] developers. [37:12] >> You applied to YC with a very different [37:13] idea. Right. [37:14] >> For sure. [37:14] >> Do you remember what the like tagline [37:16] was when you guys applied to YC in [37:18] Winter 21? [37:18] >> I think it wasn't well defined. I think [37:21] we realized let's go back to building a [37:24] really great desktop experience for [37:26] containers and Kubernetes. [37:28] >> I remember what I wrote down on the on [37:30] the application. It was a kitematic for [37:33] Kubernetes or a Docker desktop for [37:36] Kubernetes. [37:36] >> Yeah. Which was effectively Docker [37:38] Desktop. They had a great Kubernetes. [37:39] [laughter] [37:40] I think you know it's one of the [37:43] challenges as a second time founder that [37:45] you know Michael and I have told [37:46] ourselves. We we tried to overengineer [37:48] the idea in many ways and I think even [37:51] the two to three years after like we did [37:53] YC in 2021 and Olama wasn't launched [37:55] until July of 2023 after we raised our [37:59] series A after obviously after we had [38:01] done YC that journey was one of really [38:05] in search for a customer problem that [38:06] could delight a developer and in some [38:08] ways it was almost a good thing that [38:11] we tried different ideas and pivoted [38:14] until 2023 because that's when Mama [38:16] Llama came out and started the open [38:18] model with [38:19] >> which is why it's called O Lama. [38:21] >> Not necessarily. [38:22] >> Okay. No. Oh, really? [38:24] >> Llama means generally from our [38:26] experience whether you think of local [38:28] llama as the subreddit lama really just [38:30] stands for open models. You know, as we [38:34] were looking through the name, it wasn't [38:35] necessarily from an existing model. [38:37] >> It was more LLM and it's like the animal [38:40] plus LLM. [38:41] >> Yeah. And I think having that character [38:43] was important. We like what's a good [38:44] name for a character, a face you can put [38:47] to the name because models animal mascot [38:50] sometimes. [38:50] >> Docker had one GitHub. [38:52] >> He still hasn't taken my advice to have [38:54] llamas come to actual O Lama events. [38:56] >> Oh my god. [laughter] [38:59] >> I'm curious what your series A pitch was [39:01] because you raised from Benchmark like [39:02] fantastic investor but all of this the [39:05] future we're in now hadn't quite taken [39:06] off in 2023. So what was like the pitch [39:09] and the vision back then? [39:11] >> Yeah. and we partnered with Benchmark in [39:12] 2022. So it was Deli days but pre-C Chad [39:16] GBT when it came down to the pitch I [39:19] think we weighed so much on like hey [39:21] we're trying to build this great [39:22] developer experience we're solving this [39:24] security problem and we had known u [39:27] Peter the partner at benchmark from our [39:29] previous lives building building a [39:30] docker because he was the series A [39:32] investor in docker and so a lot of it [39:34] was weighted on you know the people and [39:37] also why we exist I think the what I [39:40] mean you know solving SSO for [39:43] Kubernetes, which is a real problem, [39:45] wasn't really our passion. I think we [39:47] were really lucky to find a partner that [39:48] could could see us for what we stood for [39:50] and what we were trying to do versus the [39:52] point in time, you know, problem we were [39:53] solving at that point. [39:54] >> Yeah, I see. So, you as the a sort of [39:56] pre- pivot, [39:57] >> correct? Okay. I didn't realize that [39:58] actually. H [39:59] >> I was just looking at this cloud tokens [40:01] by model family graph and like basically [40:03] if you just look at this graph it looks [40:05] like the Olama story begins on in [40:07] February 2026 and it like explodes [40:11] thereafter which is like so funny [40:13] because like of course it actually goes [40:14] back to like 2021. [40:18] What was it like to be like sort of lost [40:20] in the wilderness for like many years [40:22] working on stuff that was like kind of [40:24] working but like not really taking off [40:26] and then all of a sudden to have things [40:28] like just like explode like how did how [40:30] did it affect you and your co-founder [40:32] psychology and the team and the [40:34] employees? What what was the experience [40:36] like? [40:36] >> It was definitely scary and and and for [40:39] a few reasons. You know, one is like [40:42] when you're when you're trying to solve [40:43] a problem for devs or for a customer and [40:46] you're just getting on the phone with [40:47] them over and over again and it's not [40:49] totally clicking. That's, you know, it's [40:52] less about the are we in the headlines [40:54] or is is the project taking off the [40:57] product we're building. It was just are [40:58] we truly actually solving a problem for [41:00] somebody? And I think being lost in the [41:01] wilderness like what's your north star [41:03] that customers are generally a great [41:04] north star, but not seeing the north [41:06] star is even scarier, right? Because [41:08] often you know what problem you want to [41:09] solve, you just haven't figured out what [41:11] problem and I I think the you know [41:13] Michael my co-founder and I we started [41:14] this company because we had built a [41:16] company in the past and we ended up [41:18] being acquired by Docker very early. It [41:20] was just the founding team and Arnorstar [41:22] was saying we want to go solve a great [41:24] experience for developers with something [41:25] they find really hard but man in the two [41:28] years where we're just finding that [41:30] problem it's really scary. you know, we [41:32] had a team of more than 10 people which [41:34] made that really hard. And I'm so [41:36] thankful to that team for staying by our [41:37] side as we went through different ideas, [41:40] you know, and and what what's not really [41:42] obvious is we went from this security [41:44] for Kubernetes to then like security for [41:46] developers on the desktop, which is like [41:48] the pivot that we've never spoken about. [41:50] And then we kind of took that form [41:53] factor when models came out. We said, [41:54] well, it was a leap, but it was we knew [41:57] kind of the the kind of problem and the [41:59] feeling a developer wanted to have, but [42:01] LMS finally made it realize like it was [42:03] it was crystal clear at the point when [42:05] we tried running the llama model and it [42:07] was really hard and we're like, okay, [42:09] this is a problem and it's really [42:10] impressive when you get it working and [42:12] it's kind of just a [42:14] zero to one moment. I'm curious for the [42:16] story of that pivot because there there [42:17] actually like many pivots in the Olama [42:20] story, but probably like the most [42:21] critical one was like the pivot to Olama [42:23] to doing like locally hosted LLMs. Like [42:26] how did that come about? Were you just [42:27] like tinkering with ideas on the side? [42:29] And when you found the idea, was it [42:31] really obvious to everyone in the [42:32] company that that was the thing to do or [42:34] was there like like a a big debate? And [42:37] it wasn't until it took off that it [42:38] became clear. [42:39] >> We you know sat in a room together. I [42:41] remember we were in Toronto because we [42:42] had a team split across Toronto and Palo [42:44] Alto and now we're predominantly in Palo [42:46] Alto and we were saying throw everything [42:50] out like if we had to start from scratch [42:51] and we were just joined right now what [42:53] would we do and you know we had seen two [42:55] big problems because we had talked to [42:57] some users and LMS we tried using open [42:59] source LMS ourselves or just LMS in [43:01] general one problem was could you build [43:03] a gateway to access any model and host [43:05] that and make that really seamless back [43:06] then we were thinking of it as like the [43:08] segment for you know LLMs [43:10] >> it's a good way to think about which I [43:11] think has become really this big router [43:13] idea which is only at the beginning. [43:14] It's a massive opportunity. And the [43:16] other problem was we were a bunch of [43:17] XVMware Xdoccker folks like we know how [43:19] to make things run and so like let's [43:21] help make things run with open models [43:23] and then we kind of [43:24] >> systems [43:25] >> yeah systems and so we kind of tried to [43:27] really introspect our team which I wish [43:30] we had done sooner because security is a [43:32] very different team and sale than [43:34] developer tools and just by doing that [43:36] we gravitated towards saying let's just [43:38] try this thing let's give ourselves two [43:39] weeks to launch the first version of [43:41] Llama and then Llama 2 came out and we [43:43] said that was right at the end of the [43:44] two weeks we said okay we're launching [43:46] And we just had a bias to action. And if [43:48] you think back like in 2 weeks all of [43:51] that happened, going from idea to [43:52] shipping it to getting to more users [43:54] than we had ever had with our previous [43:55] stuff. And before that was two years of [43:57] just frankly overthinking the customer, [44:00] the product, and just not getting [44:01] something out there. [44:02] >> The first time I actually heard about [44:04] was on Reddit. I didn't realize it was [44:05] you guys. I was on like that logo. I [44:07] just was interested in like running [44:08] local models and it was on like the I [44:10] think the local LLM subreddit or [44:12] whatever and everyone was just raving [44:14] about OAMA and how great it was. [44:16] >> I was like, "Oh, it's a YC company." [44:17] >> Yeah. I found I found out later actually [44:19] cuz you you were called a different [44:20] company. You weren't in our internal [44:22] system and [laughter] [44:24] >> Yeah. I remember catching up with Jared [44:26] and saying, "Oh, hey, by the way, [44:28] there's all that security stuff. We have [44:30] this thing now. I think you were [44:33] catching up with Jared and then I bumped [44:34] into you on the stairs and [laughter] I [44:36] think you had your t-shirt or some swag [44:37] or something and I was like like you [44:39] guys are a lot like [44:41] >> you know I [44:42] >> meeting a rockstar or something. [44:43] >> It would have been really hard to time [44:45] this but I wish we had taken that leap [44:46] much sooner. I mean the best time to do [44:48] it was during YC. [44:49] >> It wasn't possible didn't exist. [44:51] >> Exactly. [44:52] >> Lama didn't launch yet. I guess you were [44:54] also one of the first GitHub projects [44:57] that very quickly got to 100,000 GitHub [45:00] stars, right? Do you remember how long? [45:02] It was like very quick. [45:03] >> Yeah, I can't remember exactly how fast, [45:05] but it was much faster than Docker and [45:08] Kubernetes. To your point, it it things [45:11] kind of just started working and started [45:12] taking off and you're really as a [45:15] founder just beside yourself because you [45:17] can't totally explain why. I think it's [45:18] the best way to explain product market [45:20] fit. And there are different levels of [45:22] product market fit. you know, we only [45:23] started monetizing earlier this year [45:26] with Llama's Cloud, but just to see [45:28] people fall in love with the product. [45:30] It's such a zero to one moment that um I [45:34] wish we had done it during YC, but in [45:35] some ways it wasn't possible. I also [45:37] think it's just kind of wild to put into [45:38] perspective like you sort of went from [45:40] being in sort of like the cranks on [45:42] Reddit like interested in running their [45:44] own like rigs at home to like 85% of the [45:48] Fortune 500 in like 2 years or something [45:51] like that. That's [laughter] like that's [45:52] like a pretty [45:53] >> That's the homebrew computer club to uh [45:55] broad computer adoption like speedrun [45:58] that took 10 years for the PC took like [46:01] >> 18 months 12 months. [46:03] >> Yeah. And that's one of the things that [46:04] surprised us the most because I think [46:05] look I think open models the original [46:07] users very much hobbyists just [46:09] tinkering. Oh my god this is even [46:10] possible but very quickly because you [46:13] know two things one is they were free to [46:15] get started with and you could run them [46:16] anywhere. that is incredibly helpful to [46:20] a Fortune 500 IT developer team because [46:23] they don't have to ask for permission to [46:24] use it. And so it just happened what was [46:26] really good for a hobbyist user [46:27] translated very quickly to a developer [46:31] within a business. Um it just happened [46:33] to be a case where that was that was it. [46:35] You know, for example, databases, we saw [46:37] some of this too where a database that [46:39] started for devs like MongoDB very [46:42] quickly also moved to enterprise, but [46:44] because LLMs are stateless, it made for [46:47] such an easy transition. Now, moving to [46:49] the cloud, there's a lot more in play. [46:50] There's an economic question. If you're [46:52] a customer, there's obviously security. [46:54] Where's the model running? But what's [46:56] beautiful about open models that both [46:57] hobbyists and IT developers loved is you [47:00] could just get started. You didn't need [47:02] permission. [47:02] >> Can we talk about the monetization [47:04] angle? cuz this is this is interesting [47:05] too. So like in 2023 Llama 2 takes off [47:09] all of a sudden you got all these users [47:11] 100,000 GitHub stars like you've clearly [47:13] found something but it's basically like [47:16] Reddit cranks who are using it. You're [47:17] making no revenue and there's no obvious [47:19] path for how you will ever make any [47:21] revenue from all these like cranks on [47:22] Reddit. It was 2 years before you [47:25] actually figured out a business model [47:26] for it which funny enough is exactly the [47:29] position that Docker was in. Like how [47:30] did you think about it during those two [47:32] years? Were you were you worried about [47:33] it? Were was the team asking like what's [47:35] the business model going to be? How did [47:37] you think about like figuring out how to [47:39] make money from it? [47:40] >> I think there's always two ways that we [47:43] saw open models um being able to [47:46] monetize in a way that's great for the [47:48] company, great for the developer and [47:50] great for the the customer. And one of [47:52] them was a privacy focused AI product [47:56] which Ola started really started with [47:58] that in its open source incarnation. But [48:00] we always felt that there was this [48:02] moment where you know you weren't using [48:04] llama with uh the llama models for [48:07] example with with tool calling right [48:09] away when they came out. So there were [48:11] use cases where it was still reserved [48:12] for the frontier models. And again with [48:14] at the risk of overthinking it we kind [48:17] of saw that there wasn't the product [48:19] level of product market fit with open [48:21] models that closed models had. And in [48:23] some ways philosophically we want to [48:25] align with when that happens we want to [48:26] be there to capture that. I think it [48:27] happened this year with coding agents [48:29] running with open models because you had [48:30] the largest consumption of AI being [48:32] matched with finally open models being [48:34] able to service that. There are a lot of [48:36] opportunities along the way to do it [48:37] privately securely. Again, a lot of the [48:39] Fortune 500 have already adopt Lama. But [48:42] we really asked ourselves what would be [48:43] the most important problem we could [48:45] solve for a customer. And the local [48:48] piece, while an important part of that [48:50] story, never felt like the whole story, [48:52] which was how do you access open models [48:55] for the hardest problems? And so, in [48:58] some ways, waiting. We knew we had to [48:59] wait a little bit for the market to [49:01] mature. At the same time, what are the [49:03] risks of waiting? Well, you build a [49:05] culture if if not careful. And we had [49:08] learned a lot of this from our Docker [49:09] days where you don't think about [49:11] monetization. It's not a priority. I [49:13] think from our previous battle scars as [49:15] a team, we we kind of had we knew about [49:16] that. But I think the other component [49:18] which is really important is making sure [49:21] you keep in touch with your customers. [49:22] Like one of the biggest risks of having [49:23] an open-source project that takes off is [49:26] you consider your user base and customer [49:28] base like your customer just a blob on [49:30] the internet [49:31] >> which is a really risky way to think [49:32] about customers because you want to meet [49:34] them figure out their needs. What are [49:35] they doing? What do they want to do in [49:36] six months? What's their story? And I [49:39] think that's the thing I wish we had [49:40] done a little more in the last few years [49:41] and we're doing a ton of that now. [49:43] >> One thing I'm curious when you went [49:45] through IC, you guys were second time [49:47] founders. I'm curious what got you to [49:49] decide to do YC. Actually, [49:51] >> you know, we we went back and forth on [49:53] this for a lot, which we shouldn't have. [49:54] We should have just said, of course, [49:55] we're doing YC. Um, but by and large, [49:59] starting a company is a really lonely [50:01] experience. Even if you have a great [50:02] co-founder, and Michael, my co-founder, [50:05] was the co-founder of my first company. [50:06] He was my college roommate at University [50:08] of Wateroo. Um, but it's still lonely [50:11] and I think just having a set of peers, [50:13] even though we did it during the [50:14] pandemic, just talking to Jared and like [50:16] five [snorts] other groups of founders [50:18] every week really helped you feel less [50:20] lonely. And I think that's such an [50:24] important part of it. And then of course [50:26] when we finally moved down here and [50:28] there was no more co the network was [50:30] just incredible. And the fact that we [50:33] could meet founders building on open [50:35] models, building on any kind of AI, you [50:37] know, we kind of knew that was going to [50:38] happen cuz we had known so many founders [50:40] from the University of Water who had [50:42] done YC pre-COVID and they were like, [50:44] it's really about getting together and [50:46] like that was a big part of it and we [50:47] knew that was there and you know, I [50:49] think that's that's what made it a [50:51] no-brainer. But also just I think there [50:54] are a lot of mistakes you can repeat [50:57] that you don't have to. And what I love [51:00] about the YC community is how [51:01] transparent founders are with each other [51:03] about those and you know I still keep in [51:05] touch with the founder of Docker uh [51:06] who's an investor in our company and [51:09] we're able to talk about some of these [51:10] challenges we saw in the previous [51:12] generation of companies that you know we [51:14] don't necessarily have to repeat or [51:16] things that worked and we can bring you [51:17] know into the future. [51:18] >> Yeah. If you just uh don't repeat one of [51:20] those mistakes that you know sometimes [51:22] is the mistake that would have killed [51:23] the company [51:24] >> potentially. Yeah. Yeah. our our famous [51:26] saying, you know, a bunch of our team is [51:28] from uh companies that ended up working [51:31] great and Docker is doing phenomenal [51:32] now, but whether it's, you know, some of [51:34] our team was early early at VMware and [51:37] it there are always ups and downs and I [51:40] think just having a group of people [51:41] around the table who have a collection [51:43] of those and also what worked actually [51:45] what worked is actually even more [51:46] important and just being able to like [51:47] have that muscle memory is a big part of [51:48] it. Oh man, I was just thinking about [51:50] this because we obviously hang out with [51:52] and work with a lot of 18 year olds or [51:55] 19 year olds and then sometimes they're [51:56] always asking like, "Well, what should I [51:58] do?" And then I'm starting to realize [51:59] like one of the more important things is [52:01] if you've never worked on a team that [52:03] shipped really amazing technology to [52:05] like a lot of people or like like just [52:08] real clear product market fit, like do [52:10] that once. Like even if it's a month, [52:12] even if it's like three months, you [52:14] would learn more in those three months [52:16] because then you know what good looks [52:17] like. And then without that it's like I [52:20] mean it's not like it's impossible like [52:21] people at YC do figure it out because [52:25] you but it's that much harder. Like the [52:27] difference between having seen something [52:29] that actually works from like beginning [52:31] to like some form of like this is what [52:34] the bug database looks like and this is [52:36] how we release and this is the quality [52:38] that's necessary and here's like the bar [52:41] that we hold each other to. Having seen [52:43] that it just like multiplies the chance [52:45] that people succeed. So it makes sense [52:47] that you know starting off with a [52:49] co-founding team that has seen a lot of [52:51] that pretty powerful. [52:52] >> Yeah, I think it it provides you a set [52:54] of values you can work around especially [52:56] when you have so much power in your [52:57] hands with AI. There just parts of it [52:59] that I can help you with but it won't [53:01] hold you accountable to it and you know [53:03] how does software work and look at for [53:05] example I'm sure there's versions of [53:07] running in the wild from two years ago. [53:08] How will your software work when [53:10] somebody falls in love with it and [53:12] continues using it for two years? Is it [53:13] still going to be working well? [53:14] hopefully they update to the latest [53:16] software or it's you know a cloud [53:17] service but I think you build that [53:18] muscle memory and we definitely have [53:20] that from a lot of our more senior [53:22] engineers on the team who are at VMware [53:23] or NERA for example but at the same time [53:26] I think there are a lot of lessons we [53:27] learned in the previous generation of [53:29] DevOps and infrastructure that aren't [53:32] valid anymore in the AI world [53:34] >> oh yeah tell us about it what have you [53:36] found [53:36] >> what is not valid anymore [53:38] >> I think a good example that I [53:40] classically used is there was this [53:42] generation of companies called platform [53:44] platform as a service [53:45] >> and in the contents the Heroku of the [53:48] world was a great example of this. [53:50] >> I mean Docker started out as that [53:51] >> Docker started out as a platform as a [53:52] service and there's this concept that if [53:54] you're a layer on top of something else [53:57] that you're in kind of a vulnerable [53:59] position as a startup which is [54:00] absolutely not true in the eye world and [54:02] in fact going up the stack can sometimes [54:04] be even better because you're closer to [54:05] the customer. in an infrastructure world [54:08] that that's also the case and that that [54:10] was like an analogy that we had to like [54:11] so many of these muscles we actually had [54:12] to break building a lama. Another one [54:15] was you know these LMS are never perfect [54:17] and like in the systems world you want [54:19] everything to be exactly as it's [54:20] designed to run. It's tested it's [54:22] validated but LMS by definition are not [54:25] >> that's a feature not a bug. [54:26] >> Exactly. It's a feature. [54:27] >> You want it to be a little [54:28] non-deterministic I suppose. And I think [54:30] from building a team too, it's that you [54:33] know with AI now there are just problems [54:35] that you don't need to [54:39] staff as heavily whereas you you you did [54:41] 10 years ago, right? If you think about [54:43] what does your customer support pipeline [54:45] look like? [54:46] >> Um what does it look like to deliver a [54:47] cloud service? Like it's a very [54:49] different world with AI because how do [54:51] you build a service where no engineer [54:53] knows exactly how all the code works [54:54] which is obviously the case now. And so [54:56] there's just new lessons we're learning [54:58] going from like a, you know, some of our [55:01] team from infrastructure 1.0 in the [55:02] 2000s to cloud in the 2010s to now the [55:05] AI space. There are a lot of rules that [55:07] break. [55:07] >> I mean, you're probably actually doing [55:08] an incredible service to um like both [55:12] sides of the ecosystem and that like the [55:14] end users get this like very clean thing [55:16] that just works, especially like the [55:18] tokens just come out and they're very [55:20] clean and the API makes sense and it's [55:23] rational and logical. And then on the [55:25] flip side like I mean if you don't have [55:27] a layer like I've directly experienced [55:29] this where it's like oh yeah the [55:30] underlying inference provider has a [55:32] weird error for you know if you put this [55:35] parameter in this way or it expects JSON [55:37] and you know it's not documented. It's [55:39] just like this insane minefield. Like, [55:41] you know, the agents can kind of figure [55:42] it out, but like you're going to like [55:43] bang your head into the wall for like a [55:46] couple hours before, you know, the agent [55:48] figures it out. And in the meantime, [55:49] you're like, "This is a terrible [55:50] experience." You know, and so you're [55:52] like in there probably helping the [55:54] inference providers fix all these [55:56] fundamental bugs, too. [55:57] >> Yeah. And it's part of the the job we [55:59] do. And I think one of the big [56:00] opportunities in the open model [56:02] landscape is curation and taking a [56:05] fragmented universe of models and [56:09] inference technology and cloud services [56:11] and harnesses and like making that [56:14] actually just work is a really valuable [56:17] problem because the end developer to [56:19] your point they just want to build their [56:20] software right they just want to build [56:22] stuff they want to build their next [56:23] company their next application and I I [56:25] think that's where we come in but it's [56:26] where a ton of great services also come [56:29] in and we saw you know open router [56:31] obviously is a good example of that from [56:32] a wide model selection so developer [56:34] doesn't have to sign up for you know [56:36] hundred different providers they can [56:38] just go to one they can pay in one place [56:40] I think we've seen with open code you [56:42] know you can have one harness that [56:43] integrates with any model it's a really [56:45] powerful experience for developers just [56:47] looking to try the next model to see if [56:49] it solves their use case better so this [56:51] curation and you know when there's a [56:53] abundance of models and providers now [56:55] there's a scarcity in bringing that [56:57] together into something that works. [56:59] >> Thank you so much for joining us. That's [57:00] all we have time for. [57:01] >> Thank you guys for having me.