## Detailed walkthrough
Here is the detailed, step-by-step walkthrough of the presentation:
[00:04](https://www.youtube.com/watch?v=SHOzagRY3vw&t=4s) The Paradox of AI Inference and GPU Waste
Developer Advocate Sheilah Kirui opens the session by addressing a stark contradiction in the current artificial intelligence ecosystem: despite widespread complaints about severe GPU shortages, millions of compute clusters around the world sit completely idle or remain locked to single, underutilized production models.
She explains that modern inference workloads operate under a challenging baseline reality:
* **Global Users with High Expectations:** Users are distributed worldwide and demand sub-second, instant response times.
* **Multi-Model Pipelines:** Driven by the rapid growth of open-source models and agentic frameworks, modern applications have shifted away from single monolithic models. A typical voice assistant pipeline now requires distinct models working in sequence for audio transcription, reasoning, embeddings, and audio generation.
* **VRAM Constraints and Pre-Deployment Guesswork:** GPUs are scarce, expensive, and constrained by finite VRAM, making it virtually impossible to keep every model loaded everywhere at all times.
Because of these physical limitations, engineering teams traditionally make rigid pre-deployment choices. They pick specific geographic regions, hardcode which models are preloaded on which GPUs, and "hope for the best" based on predicted traffic patterns. However, because real-world traffic unpredictable, this static approach consistently fails in production.
---
[00:44](https://www.youtube.com/watch?v=SHOzagRY3vw&t=44s) The Latency vs. Utilization Dilemma
Every AI inference platform struggles to reconcile two competing operational priorities:
```
THE INFERENCE DILEMMA
Option A: Minimize Latency Option B: Maximize Utilization
+--------------------------+ +--------------------------+
| Preload models globally | | Centralize GPUs into |
| on distributed GPUs. | | hyperscale data centers. |
+--------------------------+ +--------------------------+
| |
v v
+--------------------------+ +--------------------------+
| - Fast user response | | - High GPU utilization |
| - Wasted idle GPUs | | - Network latency spikes |
| during off-peak hours | | for distant users |
+--------------------------+ +--------------------------+
```
1. **Option A (Optimizing for Latency):** Preloading models on GPUs scattered across global regions eliminates cold-start penalties and reduces network lag by keeping compute close to the user. However, during local off-peak hours (e.g., nighttime), these pre-allocated GPUs sit idle while still consuming power and budget.
2. **Option B (Optimizing for Utilization & Cost):** Consolidating GPUs into a few centralized hyperscale data centers allows operators to batch requests from all over the world, ensuring high GPU utilization. The tradeoff is severe network latency for distant users, whose requests must traverse long physical distances to reach the central facility.
---
[05:06](https://www.youtube.com/watch?v=SHOzagRY3vw&t=306s) The Solution: AI Inference as a Global Compute Cache
To break this architectural deadlock, Kirui proposes treating AI inference as a **global compute cache** rather than a set of static, permanently pinned applications.
Drawing an explicit parallel to Content Delivery Networks (CDNs), the system treats GPU VRAM as dynamic cache storage:
* Models are fetched, swapped, and loaded onto GPUs strictly on demand.
* Popular, frequently requested models remain hot in VRAM.
* Unpopular or inactive models are automatically evicted from VRAM when capacity is reached, freeing up hardware resources for incoming active workloads.
---
[06:00](https://www.youtube.com/watch?v=SHOzagRY3vw&t=360s) Architecture of Akamai’s Serverless Inference Prototype
To prove the concept, Akamai engineers built a prototype serverless inference platform designed to eliminate wasted GPU cycles while preserving low user latency. The architecture consists of three core components:
```
+---------------------------------------------------------------+
| 1. Global AI Router |
| (Edge Proxy: Inspects headers & queries control panel) |
+---------------------------------------------------------------+
|
v
+---------------------------------------------------------------+
| 2. Control Panel |
| (System Brain: Tracks VRAM, active models, node health) |
+---------------------------------------------------------------+
|
v
+---------------------------------------------------------------+
| 3. Autonomous GPU Nodes |
| (Runs vLLM engine: Dynamic VRAM loading & LRU eviction) |
+---------------------------------------------------------------+
```
1. **Global AI Router:** Implemented as an edge function running on Akamai's globally distributed network, this proxy serves as the initial entry point for user requests. It inspects incoming HTTP request headers to extract target model metadata.
2. **Control Panel:** Acting as the central brain of the system, the control panel maintains a real-time global map of all GPU nodes, tracking VRAM utilization, currently loaded models, and node saturation levels.
3. **Autonomous GPU Nodes:** The underlying GPU nodes run a specialized serving engine (such as vLLM) capable of dynamically managing VRAM and executing requests autonomously.
#### Request Execution Flow:
- When an API request hits the **Global AI Router**, the router inspects the request headers, extracts the requested model name, and queries the **Control Panel** for the optimal routing destination.
- The **Control Panel** runs its routing algorithm and returns the destination node address to the router.
- The **Global AI Router** proxies the payload directly to the selected GPU node for execution.
---
[08:26](https://www.youtube.com/watch?v=SHOzagRY3vw&t=506s) Node-Level Dynamic Caching and Routing Signals
On the individual compute nodes, models are never preloaded by default. When a request targets a cold model residing on local disk, the serving engine loads it into VRAM on the fly. When VRAM capacity is reached, the node applies a **Least Recently Used (LRU)** eviction algorithm—specifically targeting the least popular models—to clear space for fresh workloads.
When evaluating where to send an incoming inference request, the Control Panel balances three primary routing signals:
* **Tiered Availability (Model Warmth):** The system prioritizes nodes where the requested model is already loaded in VRAM.
* **Geolocation:** The system prefers GPU nodes physically closest to the requesting user.
* **Saturation:** The system checks node capacity and active workload pressure to avoid overloading busy hardware.
#### Routing Behavior in Practice:
- **First-time cold execution:** When a model is requested for the first time globally, initial latency is slightly higher because the system must load the model from disk, prioritizing local node placement.
- **Warm vs. Cold tradeoffs:** If a user's nearest local GPU node does not have the target model loaded, but a more distant node already holds the model warm in VRAM, the control panel prioritizes routing to the farther, warm location to eliminate the cold-start loading delay.
Kirui wraps up the session by offering to show attendees a side video demonstration of the prototype in action.
- **Key Takeaways:**
- Traditional static model deployments force an artificial choice between high user latency and poor GPU utilization.
- Treating GPU VRAM as a dynamic global compute cache enables automated model swapping and eviction based on real-time request patterns.
- Combining edge-level request inspection, centralized state routing, and node-level LRU eviction maximizes GPU usage without compromising the end-user experience.
## Video description
Lena Hall - Akamai
Sheilah Kirui - Senior Developer Advocate, Akamai
With the explosion of AI models, keeping every model constantly loaded across every global region is no longer viable: it leaves expensive GPUs sitting idle, plummets utilization, and skyrockets costs. Conversely, centralizing your inference infrastructure destroys the user experience by introducing massive latency spikes. How do you architect a globally distributed AI inference platform that balances low latency with high GPU efficiency?
This session explores a modern approach to solving this dilemma: treating inference like a global compute cache rather than a static system.
We will break down the architectural patterns needed to make this possible, focusing on dynamic model caching, intelligent request routing, and distributed global infrastructure.
告别 GPU 浪费：借助 Akamai 构建 AI 推理全局计算缓存
针对当前 AI 推理中全球边缘预加载显存浪费与集中式数据中心网络延迟难以兼顾的困境，Akamai 借鉴 CDN 架构提出将推理系统设计为“全局计算缓存”，将 GPU 显存作为动态缓存层实现模型的按需加载与自动淘汰。该无服务器推理原型由边缘全局 AI 路由器、监控显存状态的集中式控制面板以及运行 vLLM 的 GPU 节点构成，通过结合模型预热状态、地理距离和节点负载进行动态路由，优先调度至已预热节点以消除加载延迟。
当前 AI 推理面临低延迟与高 GPU 利用率无法兼顾的难题：全球边缘预加载导致显存闲置浪费，而集中式数据中心则引发严重跨地域网络延迟。 (0:44)
Akamai 提出将推理系统设计为“全局计算缓存”，借鉴 CDN 架构将 GPU 显存视作动态缓存层，实现模型的按需加载与自动淘汰。 (5:06)
该无服务器推理原型由边缘全局 AI 路由器、实时监控显存状态的集中式控制面板以及运行 vLLM 的自主 GPU 计算节点协同构成。 (6:00)
系统根据模型预热状态、地理距离和节点负载进行动态路由，在本地冷启动与远端已预热节点之间优先选择已有预热模型的节点以消除加载延迟。 (8:26)
Reading summary
Developer Advocate Sheilah Kirui argues that static model deployments force an inefficient tradeoff between latency and GPU utilization, proposing instead that AI inference be structured as a dynamic, CDN-style global compute cache.
The Latency Versus Utilization Dilemma (0:44)
According to Kirui, the AI ecosystem faces a paradox: widespread complaints of GPU shortages coexist with millions of compute clusters sitting idle or restricted to underutilized, static workloads. Modern applications increasingly rely on multi-model pipelines—such as voice assistant systems coordinating transcription, reasoning, embeddings, and audio generation in sequence—while users demand sub-second latency. Because finite GPU VRAM prevents organizations from keeping every model preloaded in every location, teams often resort to static pre-deployment configurations that fail under unpredictable real-world traffic.
This creates an operational dilemma. Preloading models on distributed GPUs across multiple geographic regions reduces network lag and avoids cold starts, but leaves GPUs idle and costly during local off-peak hours. Conversely, centralizing GPUs in hyperscale facilities maximizes batch utilization and lowers idle waste, but introduces substantial network latency spikes for distant users.
Treating GPU VRAM as a Dynamic Cache (5:06)
To resolve the tension between speed and hardware efficiency, Kirui proposes modeling AI inference after Content Delivery Networks (CDNs). Rather than treating models as permanently pinned services, the proposed architecture handles GPU VRAM as dynamic cache storage. Models are fetched and loaded onto hardware strictly on demand based on live traffic. Frequently requested models remain hot in VRAM, while inactive or less popular workloads are automatically evicted once memory capacity is exhausted.
Three-Tiered Serverless Inference Architecture (6:00)
Akamai designed a prototype serverless platform around three core components: an edge proxy, a central controller, and autonomous compute nodes. At the network perimeter, a Global AI Router implemented as an edge function intercepts user HTTP requests and inspects request headers to extract the target model's metadata.
The router queries a central Control Panel, which acts as the platform's brain by monitoring real-time VRAM allocation, active model locations, and node saturation across the global fleet. Once the Control Panel selects the optimal destination, the router proxies the payload directly to an autonomous GPU node running a serving engine such as vLLM to process the inference request.
Intelligent Routing Signals and Node-Level Eviction (8:26)
At the hardware level, nodes do not preload models by default. When an incoming request targets a cold model stored on local disk, the node loads it into memory dynamically, applying a Least Recently Used (LRU) algorithm to evict inactive models as VRAM fills.
To balance workloads across the fleet, the Control Panel routes traffic using three primary signals: tiered availability (whether the model is already warm in VRAM), geolocation (proximity to the user), and node saturation. Kirui notes that when trade-offs arise, the system favors routing a request to a more distant node where the model is already warm in memory rather than incurring the cold-start delay of loading the model from local disk on a closer node.
Kirui presents this dynamic caching model as a viable method to eliminate idle GPU waste without compromising latency, though the approach requires balancing the latency cost of cross-region routing against local disk-to-VRAM loading times during cold starts.
[0:04] Hi everyone. Um, welcome to my session. My name
is Sheila Curie. I'm a developer advocate at
[0:09] Akamai where I focus on AI um use cases around
infra and inference. And today I want to talk
[0:16] about a massive paradox in the AI space. Um, so
we're constantly hearing about um how there's GPU
[0:23] shortages. Um, yet at this moment there's probably
millions of clusters that are sitting idle right
[0:28] now or maybe locked to sync I and utilize models
in production. Um, so today I want to talk about
[0:35] how we can fix this by changing how we look at um
infrastructure when running inference. Um, yeah.
[0:44] Um, but first let's look at the Oh. It's
[0:51] First let's look at the baseline reality of
running model inference today. Is that the
[0:57] right slide? Yeah. Um, so first you have users who
are scattered all over the world and um no matter
[1:03] where they are they're expecting um sub second
instant responses. Um, secondly with the explosion
[1:08] of open source models um and agentic frameworks in
the past couple years, um applications are moving
[1:14] from, you know, being like monolithic to like
running um multi-model pipelines, right? Um, so
[1:20] for example like a standard voice assistant could
rely on multiple models for audio transcription,
[1:25] reasoning, embedding, and so forth. Um, and this
is important because um GPUs are incredibly scarce
[1:33] and expensive and they're also constrained by
finite VRAM. Um, so you do it is kind of like
[1:39] virtually impossible to load every single model
everywhere at all times. Um, yeah. And because of
[1:46] these constraints, engineering teams are usually
forced to make uh permanent decisions or rigid
[1:52] decisions before the deployment or maybe
not permanent but they still have to make
[1:56] decisions before running the deployment. Um
for example, they have to decide where should
[2:01] the GPUs live. Um they have to hardcore like
which models should be preloaded onto the GPUs
[2:06] and then kind of like hope for the best. I mean,
usually it's based on where they predict like
[2:11] demand will be, but um this rigid approach is not
practical because in reality like things don't go
[2:17] as expected, right? Um so this brings a dilemma
in every inference platform that every inference
[2:23] platform tries to balance these two. Um so on one
hand, you have latency. Like I mentioned earlier,
[2:31] um you do have users everywhere. They're expecting
[2:35] fast responses, especially for interactive
applications like chats, um chatbots, um voice
[2:40] assistants. Um and on the other side, you do have
to optimize uh consider taking into consideration
[2:46] the utilization of GPUs. As I mentioned, this
it's a finite resource and in the entire stack,
[2:51] GPUs are like the most constrained. Um so this
Ideally, you would want to optimize for both,
[2:57] but the challenge is that um usually
trying to optimize for one pull it they're
[3:02] opposing forces. Like trying to optimize for one
will um probably make you compromise on the other.
[3:09] Um on this slide, I was saying that because of
these constraints, um engineering teams are forced
[3:13] to make decisions in advance. Um usually they have
to decide where the GPUs should live, which models
[3:19] they should preload, and then kind of like just
hope for the best. Um this approach is not i- it
[3:26] it's not ideal because like I mentioned in uh in
production, things don't always go as pre-planned,
[3:31] right? Um and these are the tradeoffs when you're
thinking of serving inference. Um on the one hand,
[3:37] you have latency. You want to make sure there's
fast responses. On the other side, you're
[3:41] optimizing for those GPUs. Ideally, you would want
to optimize for both, but they are they get pulled
[3:45] in different directions. So there's usually some
tradeoffs, which is the next slide here. Um well,
[3:51] the first one is if you wanted to minimize user
latency, you would probably want to preload the
[3:56] models onto globally distributed GPUs. This
will be able to reduce eliminate the cold start
[4:03] and it will also reduce the network lag by
processing compute much closer to the user,
[4:07] so they will get fast responses. Um but it does
introduce an efficiency problem because when you
[4:13] load GPUs in different regions and sometimes
the traffic in these regions fluctuate.
[4:18] So, your GPUs could be sitting in idle, let's say
night time and you've already preloaded the model,
[4:22] so you're just wasting resources. On the other
hand, if you decide to maximize for utilization
[4:27] or to cut cost, one way to do it is to pull all
the GPUs into a few massive hyperscale centers.
[4:33] And then you would be able to batch the traffic
from different locations. So, by doing this,
[4:37] your GPUs stay busy, so this is good for
cost. But on the trade-off of this is that
[4:43] users who are much further from those central
locations have to wait longer for responses
[4:48] because the data has to travel further just to
reach those models in those specific regions.
[4:54] Three Y. So, which is the question here? So, how
can we break this deadlock? How can you architect
[5:01] a system where it can somehow balance
both of them, right? The opposing forces?
[5:06] And the way to do this is to treat AI
inference as a global compute cache
[5:12] and I will explain what this is in a second. What
I mean by this is instead of thinking as models as
[5:19] static applications that you pin permanently
into memory, what if instead you treated them as
[5:25] resources that can be loaded, swapped, and
scaled based on demand? So, with this shift,
[5:31] the GPUs are what's acting as a compute
cache. If you're familiarized with CDN, it
[5:36] caches the most frequently requested data and
then when it's no longer requested, it will evict
[5:42] it from the cache. So, similarly, you only load
the models as they are needed and then over time
[5:48] if they become less popular, then you can kick
it off to be able to serve more requests with
[5:52] with a more popular or more requested models.
Great. So, to bring this concept to life,
[6:00] some engineers at Akamai created a prototype to
showcase how you would be able to solve this.
[6:05] Um this is a fully managed serverless inference
platform. It leverages the Akamai edge for the
[6:11] intelligent and the dynamic routing. And the goal
basically is to be able to stop wasting GPU cycles
[6:17] by making sure that you're routing the right
models at the right time in the right location.
[6:22] And in doing so, you will be able to balance
latency, and you'll also, like I said,
[6:27] make sure you're not wasting GPUs. Um So, at a
high level, um this system is split into three
[6:33] different layers. You have the AI router,
which is the global entry to the system.
[6:39] This serves as the proxy for user request. And
it it because it's serving at the Akamai edge,
[6:44] it's it's the it's distributed, so it's closest
to the user, so it's the first point that users
[6:48] will interact with. In the middle here, you have
the control panel, and this acts as the brain of
[6:53] the system. It has a real-time view of all the GPU
nodes in the system, like what their availability
[7:00] is, what are the active models loaded, what is the
the VRAM capacity. Um and then at the foundation,
[7:06] you have the those GPU nodes, and the key
part here is they act autonomously. Um
[7:11] And inside of those GPU nodes, we're running
a specific servant serving engine called Muon,
[7:16] because it's a multimodal support, so it
can dynamically be able to manage the VRAM.
[7:23] Yeah, and um once again, when the API request
comes in, the edge function will inspect the
[7:27] custom header, it will extract the model name,
and then will instantly ask the control panel
[7:33] where should this request go. And then once
the control panel, um like I said, it has a
[7:38] global map of the entire system, it will use some
routing algorithms to be able to decide, and I'll
[7:42] talk about what those signals are. And then, yeah.
And then it will be And then it will send it send
[7:48] the location to the router, and then the router
will actually be able to direct it where needed.
[7:53] Um So, of course, for this idea to work, you do
need to consider these three architecturals. Um
[7:58] first one, like I mentioned, is the distributed
architecture by having those GPUs in different
[8:02] regions. The at my edge, um you need to have
global this um infrastructure. And then the other
[8:09] thing you need to be able to have is a system that
can dynamically load and evict models, so dynamic
[8:14] model caching. And you also need an intelligent
routing system, so it's cuz the goal here is to
[8:20] minimize latency, so you want to make sure
you're routing to the ideal locations.
[8:26] Uh take a pause. Talking too fast. Um so, yeah,
let's look more in the how the GPU nodes operate
[8:33] within the system. So, as I mentioned,
the models are never preloaded by default.
[8:38] When the request hits the node for a cold model
sitting on the disk, the engine will load it
[8:43] instantly. And then as more models cycle through,
um this to serve other requests, the VRAM is
[8:49] treated as a finite resource. So, this means
that when it reaches capacity, it uses an LRU um
[8:55] algorithm specifically, the least popular
algorithm, and this is how it's able to kick
[8:59] out them be able to detect which model hasn't been
requested for. Um so, by doing this, um this does
[9:05] maximize utilization because you're making sure
that your GPU always stays busy by kicking out,
[9:10] you know, the ones that are not being the models
that are not um being requested for. Um hm
[9:18] And then for the routing level, um here um
this is the control panel is distributed
[9:25] and the implementation is distributed across the
edge and core. And I think I've already mentioned
[9:29] this. The only key to mention here is the signals
that I mentioned. How is it able to determine
[9:34] the most optimal optimal location to route?
And it's based on these three signals. Tiered
[9:39] availability basically warmth and prioritizes
if the model is already warm somewhere.
[9:45] The second one is geolocation. Nearby locations
are preferred but near the closest is preferred
[9:51] but sometimes in some situation everything
else can be fair game. It just balances based
[9:56] on saturation how how much load does a engine
have how much capacity for load it has it has
[10:04] and so I think to the things to mention here
is that for example with the demo that I will
[10:10] showcase for example initially the very first
time you launch the model obviously the latency
[10:16] will be higher because you're loading it and it
prioritizes location because you don't have it
[10:20] loaded anywhere but if you have a situation where
you you have a location that's closer to the user
[10:26] but doesn't have the model and another one that's
farther it will prioritize the farther location.
[10:32] Great um and with that we'll see. This is
the takeaways. I have a demo to show but
[10:41] I don't think the demo will showcase. Should
I I should probably should have sent the video
[10:45] to showcase as well. I'm just realizing that.
But um since it's just you guys I can show you
[10:49] the video like here on the side just to see like
how it's working right. Um thank you. [laughter]
