Cloudflare Workers AI Optimizes Large Model Serving with KV Cache Quantization, Weight Compression, and Integrity Checks

Cloudflare Workers AI Optimizes Large Model Serving with KV Cache Quantization, Weight Compression, and Integrity Checks Cloudflare detailed how it efficiently serves large, long-context mixture-of-experts models like Moonshot's Kimi K-series and Z.ai's GLM on Workers AI using SGLang. By decoupling prefill and decode pools, Cloudflare layers FP8 KV cache quantization, INT4 weight compression, and KV cache integrity verification to scale throughput and reduce memory costs without sacrificing model accuracy. Storing Kimi K2.6's KV cache in FP8 doubles capacity to roughly 1.37 million tokens and enables 64 concurrent requests at 2,192 tokens per second on H200 decode pools. Compressing GLM 5.2 weights to INT4 shrinks checkpoints by about 40% from 705 GB to 421 GB, freeing memory for 1.18 million KV cache tokens while keeping accuracy within 0.8 points of FP8. Prefill and decode are disaggregated to use FP8 weights and BF16 cache where compute-bound, reserving memory-saving INT4 weights and FP8 cache for bandwidth-bound decode. A tagged KV cache integrity checking mechanism prevents cross-request data leaks on shared GPUs with under 1% overhead on throughput and tail latency. FP8 KV Cache Quantization for Decode To counter memory exhaustion from long context, Cloudflare converts the KV cache from BF16 to 8-bit FP8 (e4m3). On Kimi K2.6 running on disaggregated H200s, this raises context capacity from roughly 686,000 to 1.37 million tokens. While BF16 has slightly higher raw token speed per request, it caps out at 32 concurrent requests. FP8 scales to 64 concurrent requests, achieving 2,192 tokens per second—about 41% higher peak throughput for 30% lower token cost—with indistinguishable accuracy. INT4 Weight Compression Across Disaggregated Pools For GLM 5.2, weights are compressed from FP8 to INT4, lowering checkpoint size from 705 GB to 421 GB and reducing per-GPU memory across an 8-way tensor-parallel setup from 88 GB to 52 GB. Because decode is memory-bandwidth bound, smaller INT4 weights accelerate token generation. Because prefill is compute-bound and slowed down by INT4 dequantization (8,660 tokens/s in INT4 versus 10,160 tokens/s in FP8), Cloudflare deploys INT4 for decode nodes and FP8 for prefill nodes. KV Cache Integrity Protection Denser request packing increases the risk of memory bookkeeping errors across paged attention and continuous batching. Cloudflare introduced a tag-based integrity system where physical cache pages receive tags on reallocation. Decode operations verify expected mappings in a separate batch check to avoid GPU thread-group races. The check aborts any mismatching requests with under 1% impact on throughput and tail latency.

Cloudflare优化大模型推理：Kimi与GLM的高效部署实践 Cloudflare在Workers AI上结合开源框架SGLang，针对Moonshot的Kimi与Z.ai的GLM模型，通过解耦预填充与解码阶段，并叠加KV缓存量化、模型权重压缩与缓存完整性校验三项技术，在维持模型精度的同时大幅降低成本并提升吞吐量。 将KV缓存量化为FP8使Kimi K2.6的上下文容量翻倍至约137万Token，解码并发量上限提升至64，峰值吞吐量提高约41%，每Token成本降低约30%。 将GLM 5.2权重压缩至INT4使体积缩减约40%，单卡显存占用降至52 GB，解耦架构在解码阶段利用低带宽需求加速，预填充阶段维持FP8以确保吞吐效率与精度。 设计了开销低于1%的KV缓存动态标签校验机制，防止高并发共享硬件下的分页簿记错误导致数据混淆。 KV缓存量化与解耦调度 长上下文模型的GPU显存瓶颈通常在于KV缓存。Cloudflare将默认的BF16缓存压缩为8位浮点（FP8 e4m3），在Kimi K2.6上把驻留上下文容量从约68.6万Token提升至137万。在H200部署测试中，BF16在32并发时耗尽缓存，而FP8可支撑64并发并达到2,192 Token/秒的吞吐量。得益于预填充与解码解耦部署，计算密集型的预填充阶段保留BF16的高吞吐，内存密集型的解码阶段采用FP8，且评测显示精度未发生改变。 模型权重INT4压缩加速解码 针对GLM 5.2，团队将权重由FP8压缩为INT4，检查点体积从705 GB降至421 GB，8卡张量并行环境下单卡显存占用从88 GB降至52 GB，为KV缓存腾出约118万Token的空间。解码速度受显存带宽限制，轻量化权重显著降低了延迟；而针对需要将INT4反算展开的计算密集型预填充阶段，系统保留FP8权重以维持约10,160 Token/秒的速率（INT4为8,660 Token/秒），全评测基准下精度差异在0.8分以内。 高并发下的KV缓存完整性校验 随着大量并发请求共享物理显存，分页注意力与缓存复用机制一旦出现微小簿记差错就可能泄露数据。Cloudflare建立了物理页动态标签匹配机制，在每次重新分配时更新标签，解码读取前由服务器校验映射关系，遇不匹配则立即中止请求。该校验作为独立的批量检查运行以避免GPU线程竞争，对吞吐量和尾部延迟的影响均低于1%，并在默认路径提供无开销的空操作跟踪器。

Workers AI runs inference for some of the best open models in the world on GPUs in Cloudflare data centers close to your users. Two of the most capable, and most demanding, are Moonshot's Kimi K-series and [Z.ai](http://Z.ai)'s GLM. They are large, long-context, mixture-of-experts models, and they are wonderful to use. They are also very hard to serve efficiently because of memory constraints. We've written before about how we [serve large models on Workers AI](https://blog.cloudflare.com/workers-ai-large-models/) and about separating [the prefill and decode phases](https://blog.cloudflare.com/high-performance-llms/) of inference to get more out of each GPU. This post looks at three techniques we layer on top of that to fit these models into memory and keep them fast: quantizing the KV cache, compressing the model weights, and, because both of those pack more requests onto shared hardware, protecting the cache those requests share. These optimizations enable us to support more customers at lower costs, with no change in model accuracy. All our experiments and production traffic are running and benchmarked with [SGLang](https://github.com/sgl-project/sglang), an open-source inference serving framework. We found that SGLang offers the best performance in the market, and we work closely with the SGLang team to upstream patches and new features to make our work available to the open-source community. As a model generates text, it stores the attention keys (K) and values (V) for every token it has already processed in a structure called the KV cache. The cache is what lets the model extend a long conversation without re-reading the entire context on every new token. For a long-context model, it grows quickly, and it is usually the KV cache, not the model's weights, that fills up GPU memory first. By default, the cache is stored in 16-bit precision (BF16). We store it in 8-bit floating point instead (FP8, e4m3), which halves its size. On Kimi K2.6, that raises the amount of context we can hold in memory from roughly 686,000 tokens to about 1.37 million, twice as much. It's worth being precise about where the benefit comes from, because it isn't raw speed. Quantizing the cache adds a small amount of work per token, since the FP8 attention kernel has to convert values as it reads them. What it changes is how many requests we can keep resident at once. The following measurements are for Kimi K2.6 decoding on a disaggregated H200 deployment, comparing the attention kernels directly: At any single concurrency level, BF16 is a few percent faster per token. But BF16 runs out of cache at 32 concurrent requests and can't admit a 33rd, while FP8 keeps going to 64 and reaches 2,192 tokens per second, about 41% higher than BF16's peak, for roughly 30% less cost per token. Because we run prefill and decode as separate pools, we can apply this where it helps most: prefill is compute-bound rather than memory-bound, so there we leave the cache in BF16 and keep its slightly higher throughput. None of this would matter if it changed the model's answers, so we checked. Across our evaluation suite, FP8 and BF16 caches are indistinguishable: The KV cache is one demand on GPU memory; the model's weights are the other. For GLM 5.2, we compress the weights from 8-bit floating point down to 4-bit integers (INT4) with no loss in accuracy. The checkpoint shrinks from 705 GB to 421 GB, about 40%, and per-GPU memory across an 8-way tensor-parallel deployment drops from roughly 88 GB to 52 GB, which leaves room for around 1.18 million tokens of KV cache on the same hardware. Across our evaluation suite, INT4 and FP8 weights are indistinguishable: Smaller weights make the decode phase faster, and for a clear reason: generating each token means streaming the model's weights out of GPU memory, so decode speed is limited by memory bandwidth. Move less data and every token arrives sooner. The effect is largest at low concurrency, where per-request latency matters most: Prefill behaves differently. It is compute-bound, and INT4 weights have to be expanded back out before the model can multiply with them, so that extra step makes prefill slower rather than faster, GLM sustains about 10,160 tokens per second of prefill in FP8 versus 8,660 in INT4. As with the KV cache, the disaggregated design turns this into a choice rather than a compromise: we run INT4 for decode, where it wins, and FP8 for prefill, where it wins. Model accuracy stays within 0.8 points of the FP8 model across every benchmark we run, making its quality indistinguishable. Both techniques above have the same effect: they let many more requests share one GPU's memory at the same time. That efficiency is the whole point, but it also means hundreds of requests are reading and writing pages of the same physical KV cache. The mechanisms that make this fast, paged attention, continuous batching, cache reuse, all rely on getting the bookkeeping exactly right, and at our request volumes, even a one-in-a-billion mistake would show up regularly. So we built KV cache integrity checking as a layer of defense. The idea is straightforward: every physical cache page gets a tag that changes whenever the page is reallocated, and the server records which pages and tags each request expects to use. Before supported decode operations read from the cache, those mappings are checked. If anything doesn't match, the affected request is aborted rather than allowed to return data from the wrong page. The question that decides whether a safety check ships is what it costs. We measured it on a mid-sized production model in a two-prefill, two-decode configuration, with 8,192-token inputs and 1,000-token outputs: The cost is under 1% on both throughput and tail latency, and even the upper bound of the 95% confidence interval stays near 1%. We kept it computationally cheap by running the validation as a separate batch check rather than fusing it into the attention kernel, which would have introduced a race between GPU thread groups. It's enabled per deployment, and the default path uses a no-op tracker with no measurable overhead, so deployments that don't need it pay nothing. Serving frontier models efficiently is a moving target, and this is the ongoing work behind it. We're expanding FP8 KV caches across more of the fleet, validating NVFP4 weights on Blackwell (NVIDIA’s GPU architecture), and working toward making integrity checks something we can leave on everywhere at negligible cost. These optimizations will allow us to continue to support more customers at a lower cost and at the same accuracy. If squeezing the best open models onto GPUs and serving them to millions of developers sounds like your kind of problem, [come work with us](https://www.cloudflare.com/careers/).