Intel CPUs Beat Nvidia GPUs on AI Inference Under Load, Deloitte Finds

Innovation in Model Compression for AI Efficiency With Multiverse And Deloitte — Ai4

Ai4

Compressing models with quantum techniques lets Intel CPUs outperform Nvidia GPUs by 39% at high concurrency while cutting cloud hosting costs by 56%.

AI-assisted summary of Innovation in Model Compression for AI Efficiency With Multiverse And Deloitte — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by the Ai4 channel on August 28, 2025, this presentation features Chris Zaharias of Multiverse Computing alongside Robert Simmons and Saaket Varma of Deloitte. According to the page description, the session covers a collaborative study evaluating the viability, cost reduction, and efficiency gains of running generative AI inferencing on Intel CPU architectures and OpenVINO rather than conventional GPU setups in the AWS cloud.

## TL;DR The presentation examines how combining quantum-inspired model compression with commodity CPU infrastructure addresses enterprise bottlenecks around soaring GPU expenses and hardware scarcity. Multiverse explains its physics-based compression technique that reduces model footprint by up to 90% without sacrificing accuracy, while Deloitte illustrates the operational, financial, and regulatory drivers pushing organizations toward localized CPU inferencing. Technical benchmarks testing an 8-billion-parameter Llama 3.1 model on 4th Gen Intel Xeon processors demonstrate a 48% to 56% cost reduction compared to dedicated cloud GPUs. Crucially, empirical testing reveals that while GPUs lead at single-user loads, CPU architectures with OpenVINO achieve superior throughput and stability under high concurrency scaling past 256 and 512 simultaneous users.

## Atmosphere & style The presentation maintains the pragmatic, professional cadence of an enterprise conference briefing, shifting smoothly from strategic hardware economics to granular benchmarking data. The team dynamic is collaborative and lightly humorous, punctuated by self-deprecating remarks about Multiverse's low profile and an enthusiastic 'show you the money' transition into empirical charts. It feels grounded, analytical, and refreshingly devoid of generic AI hype, delivering a concrete engineering case study aimed directly at budget-conscious enterprise IT leaders.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=4s) Multiverse Computing utilizes mathematical frameworks from quantum many-body physics to map neural network correlations, enabling 80% to 90%+ model compression with near-lossless precision compared to traditional pruning or uniform quantization. - [02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) Dedicated enterprise cloud GPU instances ($20,000–$25,000/month) create massive cost barriers compared to standard CPU infrastructure ($3,000–$5,000/month), pushing organizations to explore CPU-based generative AI inferencing. - [02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) Internal compute contention frequently starves support, HR, and back-office teams of GPU hardware as mission-critical and revenue-generating units monopolize available clusters. - [02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) Strict client NDAs in auditing and security classifications in defense require isolated, on-premises, or localized CPU environments to avoid exfiltrating sensitive prompt data into external or shared cloud models. - [05:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=329s) Deploying compressed models on CPUs eliminates exotic hardware dependencies, enabling organizations to leverage existing fleets of corporate laptops, virtual machines, and tactical edge devices. - [05:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=329s) Task-specific compressed models provide clear governance and auditability, allowing developers to trace input-output behavior more reliably than with monolithic, generalist LLMs. - [07:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=449s) Benchmarking Llama 3.1 8B on 4th Gen Intel Xeon CPUs using AMX and OpenVINO demonstrated a 48% reduction in cost-per-token and a 95% throughput increase over an NVIDIA A10G baseline at 128 concurrent users. - [07:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=449s) Multiverse compressed the Llama 3.1 8B model footprint by 60%, delivering a 2x inference speedup across tested environments. - [12:15](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=735s) While GPUs provide superior raw throughput at very low concurrency (e.g., ~70 tokens/sec for 1 user), GPU latency deteriorates rapidly between 256 and 512 concurrent users compared to steady CPU response profiles. - [15:18](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=918s) At 256 concurrent users, Intel CPU infrastructure achieved a 56% monthly cost reduction ($1,027 vs. $2,912) while matching GPU throughput (7.35 vs. 8.17 tokens per second). - [15:18](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=918s) Under high concurrency loads of 512 users, CPU throughput (4.16 tokens/sec) outperformed GPU throughput (2.99 tokens/sec), with model compression driving a 127% efficiency boost on the CPU platform.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=4s) Multiverse Computing and Quantum-Inspired Compression

Chris Zaharias introduces Multiverse Computing as a Spanish AI company that has raised roughly $250 million, joking self-deprecatingly that the audience likely hasn't heard of them yet despite their disruptive work in artificial intelligence and large language model (LLM) compression. He frames the core problem around enterprises and device engineers confronting soaring inference budgets—often running into millions or billions of dollars annually—alongside stringent performance requirements around time-to-first-token and inter-token latency. Rather than relying on traditional compression techniques such as uniform quantization, structured pruning, or knowledge distillation—which often indiscriminately degrade precision or chop away entire sets of neurons across the board—Multiverse relies on an approach rooted in quantum many-body physics.

This mathematical framework acts as a lens to simplify exponential modeling problems, identifying the intricate mathematical correlations across neural networks, convolutional neural networks (CNNs), and LLMs that actually drive model functioning versus those that are redundant. By isolating and preserving meaningful correlations, the technique compresses models by upwards of 80% to 90%+ while sustaining near-lossless precision. Zaharias hands over the session to partners from Deloitte and Intel to demonstrate how this compression translates into real-world architectures.

- **Capitalization and Origins:** Multiverse Computing is a Spain-based AI firm with approximately $250 million raised. - **Core Technology:** Applies quantum many-body physics to map and isolate essential versus non-essential parameter correlations across neural networks. - **Differentiator vs. Standard Methods:** Avoids broad-brush reductions seen in standard quantization, pruning, or distillation, achieving 80%–90%+ compression with minimal drops in accuracy.

---

[02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) Operational Realities: GPU Bottlenecks, Costs, and Security

Robert "Bob" Simmons from Deloitte takes over to lay out the practical, day-to-day drivers compelling organizations toward running small language models (SLMs) and compressed large models on CPUs. GPUs cannot be treated as a "one-size-fits-all" answer. Running full-scale LLMs on dedicated GPU instances is extraordinarily expensive: standing up an enterprise cloud GPU instance typically runs between $20,000 and $25,000 per month, whereas a standard CPU server or virtual machine costs roughly $3,000 to $5,000 per month. For budget-conscious commercial departments and government agencies, that price difference is massive.

```mermaid flowchart TD accTitle: Organizational Resource Crunch subgraph Enterprise["Enterprise Infrastructure Resources"] GPool["Dedicated GPU Cluster<br/>($20k-$25k / mo per instance)"] CPool["Standard CPU Infrastructure<br/>($3k-$5k / mo per instance)"] end

subgraph Allocation["Resource Contention"] Mission["Mission-Critical Teams<br/>(Monopolizes GPU allocation)"] Support["Support & Operations Teams<br/>(Starved of GPU compute)"] end

subgraph Solution["Target Deployment"] CPool --> Support Support --> Edge["Compressed Models & SLMs<br/>(Local VM, Laptop, Edge)"] end

GPool --> Mission ```

Beyond raw pricing, Simmons highlights the organizational friction of compute availability. In both government and commercial sectors, an internal resource tug-of-war exists: the revenue-generating or mission-critical teams inevitably monopolize the available GPUs, leaving internal support units—such as HR, administration, and back-office analytics—starved for hardware. When support personnel want to deploy AI to streamline workflows, waiting indefinitely for an on-premises GPU cluster is unviable.

Simmons stresses the acute security and confidentiality risks inherent in shared, externalized AI systems. In high-stakes settings like commercial audit engagements, client NDAs legally forbid enterprise prompts from ever leaving the audited company's boundary. A parallel wall exists in government and defense: classified intelligence and sensitive procurement contract negotiations cannot risk leaking into external models. Furthermore, if continuous feedback learning is enabled, sensitive input data stored back into model weights could inadvertently be extracted by third parties who know how to prompt the model correctly. Containing AI within dedicated, on-premises or virtual CPU environments eliminates these data exfiltration vectors.

- **Infrastructure Cost Disparity:** Cloud GPU servers run roughly $20,000–$25,000/month versus $3,000–$5,000/month for CPU-based servers. - **Internal GPU Contention:** Mission-side demands consistently starve operational and support teams of dedicated GPU hardware. - **Data Protection & Legal Liability:** Strict NDAs in corporate auditing, defense classification, and government contract shops mandate air-gapped or localized environments to avoid leaking proprietary prompts into shared cloud models.

---

[05:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=329s) Architectural Simplicity, Model Governance, and Edge Deployment

Simmons turns to the functional advantages of shifting to smaller, highly compressed models running on commodity CPU hardware. Deploying on CPUs sidesteps exotic hardware dependencies: organizations can instantly take advantage of their vast, existing fleets of virtual machines, standard corporate laptops, and edge hardware without overhauling their operational architectures. Scaling becomes a straightforward matter of provisioning additional general-purpose CPU instances that cloud providers and internal IT teams already have in abundance.

Another critical advantage is explainability and governance. Massive LLMs are sprawling, opaque generalists that are difficult to audit when deployed on a single, isolated task. In contrast, smaller compressed models can be tailored, fine-tuned, and verified for specific tasks. When an internal auditor asks what the system did, engineers can cleanly trace the input-output relationship, isolate errors, and patch them immediately. Coupling this explainability with accelerated inference speeds opens the door to fielding portfolios of focused models across low-power edge devices and tactical field units.

- **Deployment Flexibility:** Compressed models run natively across laptops, standard cloud VMs, and remote edge hardware. - **Explainability:** Task-specific small models provide clear transparency for compliance, allowing developers to trace exact inputs and outputs without unexpected generalist behaviors. - **Fleet Management:** Organizations can operate diversified portfolios of smaller, reliable models instead of depending on a single monolithic LLM.

---

[07:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=449s) Intel Benchmarking Methodology: GPUs vs. 4th Gen Xeon CPUs

Bernie, a solution architect with Intel's Government Technologies group, steps up to detail the technical testing pipeline constructed alongside Deloitte and Multiverse Computing. The objective was to evaluate whether an 8-billion-parameter open-weights model could run natively on Intel CPU infrastructure while matching or exceeding the economics and throughput of dedicated GPU hardware.

```mermaid flowchart LR accTitle: Three-Step Intel Benchmarking Workflow subgraph S1["Step 1: GPU Baseline"] A["Llama 3.1 8B Model"] --> B["AWS Instance<br/>NVIDIA A10G GPU"] end

subgraph S2["Step 2: CPU Optimization"] C["Llama 3.1 8B Model<br/>(Base & Multiverse Compressed)"] --> D["AWS 4th Gen Intel Xeon"] D --> E["Intel AMX Engine +<br/>OpenVINO Model Server"] end

subgraph S3["Step 3: Comparative Analysis"] B --> F["Benchmarking Engine<br/>(1 to 256 Concurrent Users)"] E --> F F --> G["Metrics: Throughput, Cost,<br/>Latency & Compression Ratio"] end ```

The experiment evaluated a Llama 3.1 8B parameter model deployed across two primary hardware environments in AWS. First, the team established a baseline by running the uncompressed model on an AWS instance equipped with an NVIDIA A10G Tensor Core GPU. Second, they deployed the model on a 4th Generation Intel Xeon processor utilizing Intel Advanced Matrix Extensions (AMX)—built-in hardware accelerators designed to execute heavy matrix math operations within fewer clock cycles—managed via the OpenVINO Model Server as an optimized inference and containerized serving frontend. Tests were executed using both the full-sized model and the Multiverse-compressed version.

The benchmarking suite tested concurrency scaling from a single user up to 256 simultaneous concurrent requests to simulate enterprise demand:

- **High-Concurrency Throughput:** At 128 concurrent users, the CPU-based architecture delivered a 95% throughput increase over the GPU baseline. - **Cost Efficiency:** Running on Intel CPUs produced a 48% reduction in cost-per-token performance compared to running on GPU infrastructure. - **Concurrency Crossover:** For a single isolated user, CPU performance trailed slightly, achieving 89% of the GPU's throughput; however, under heavy load (256 concurrent users), the CPU instance outperformed the GPU by 50%. - **Multiverse Compression Factor:** Multiverse compressed the model footprint by 60%, unlocking a 2x inference speedup across tested environments.

---

[11:47](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=707s) Transition to the Hard Data with Saaket Varma

Saaket Varma from Deloitte takes the stage to unpack the technical details behind the benchmark curves. Emphasizing his role as a data scientist, he quips that it is time to "show you the money" by walking through the empirical numbers and charts collected during the AWS testing runs. He points the audience toward the first comparative graphs, setting up an in-depth breakdown of how token latency, concurrency, and compute costs scale across CPU and GPU architectures.

[12:15](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=735s) Benchmark Analysis: CPU vs. GPU Performance and Scalability

The presentation evaluates the performance differences between running compressed generative AI models on Intel CPUs versus traditional GPUs under varying load conditions. In the benchmark visualizations, tokens per second are plotted in blue, while total response time is shown in orange, with circular markers representing GPU measurements and square markers tracking CPU performance. The benchmark scales the workload across a concurrency spectrum starting from a single concurrent user and ramping up to 1,024 concurrent users hitting the model simultaneously.

At very low concurrency, GPUs deliver higher raw throughput, reaching approximately 70 tokens per second for a single user. However, this throughput degrades sharply as concurrency climbs toward 256 users. In contrast, the CPU architecture maintains a notably steady, stringent performance profile. Although a slight throughput dip occurs between 30 and 64 concurrent users—due to the benchmark utilizing 48 CPU cores—the CPU line remains stable across higher loads. A critical inflection point emerges between 256 and 512 concurrent users: beyond this threshold, GPU total response time deteriorates at a much steeper rate than CPU response time.

```mermaid flowchart TD accTitle: Concurrency Scaling Behavior Comparison subgraph LowConcurrency["Low Concurrency (1–64 Users)"] A["GPU: High raw tokens/sec (~70 tps)"] B["CPU: Moderate throughput, low cost"] end subgraph Inflection["Inflection Range (256–512 Users)"] C["GPU: Rapid throughput drop & latency spike"] D["CPU + OpenVINO: Resilient throughput curve"] end subgraph HighConcurrency["High Concurrency (512–1,024 Users)"] E["GPU: Degraded response time & throughput"] F["CPU: Superior scaling & lower latency per token"] end LowConcurrency --> Inflection --> HighConcurrency ```

- **Low-concurrency trade-offs**: Organizations with modest concurrency requirements can swap GPUs for CPUs to realize substantial cost savings while accepting only marginal differences in initial latency. - **High-concurrency resilience**: Deploying Multiverse-compressed models on Intel CPUs via the OpenVINO Model Server delivers faster, more consistent throughput once traffic scales past 256 users. - **Enterprise deployment fit**: For large enterprise implementations where more than 500 concurrent internal employees or external clients query the model, CPU architectures offer more predictable and stable response profiles than GPU setups.

---

[15:18](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=918s) ROI Demonstration: Cost and Throughput Scaling across Concurrency

To quantify the economic and operational advantages of CPU-based inferencing, an interactive ROI modeling tool demonstrates real-time differences in cost, token throughput, and latency when toggling between Intel CPUs and GPUs across user loads ranging from 1 to 1,024 concurrent sessions.

```mermaid flowchart LR accTitle: ROI Tool Architecture and Evaluation Flow subgraph InputConfig["Input Configuration"] U["Concurrency Slider<br/>(1 to 1,024 Users)"] M["Model Variant<br/>(Compressed vs. Uncompressed)"] end subgraph ComputeEngine["Compute Infrastructure"] CPU["Intel CPUs + OpenVINO"] GPU["Standard GPU Infrastructure"] end subgraph MetricsOutput["Evaluated Metrics"] C["Monthly Cost ($)"] T["Throughput (Tokens/Sec)"] R["Total Response Time"] end InputConfig --> ComputeEngine --> MetricsOutput ```

At 256 concurrent users, running the compressed model on Intel CPUs incurs a monthly infrastructure cost of $1,027. Transitioning the same workload to a GPU-based infrastructure elevates the monthly bill to $2,912, demonstrating an immediate 56% infrastructure cost reduction simply by opting for CPU compute. In terms of throughput at 256 users, the performance remains virtually neck-and-neck: the CPU setup delivers 7.35 tokens per second compared to 8.17 tokens per second on GPU.

| Concurrency Level | Hardware Setup | Monthly Cost | Throughput (Tokens/Sec) | Cost / Efficiency Delta | |:--- |:--- |:--- |:--- |:--- | | **256 Users** | Intel CPU + OpenVINO | $1,027 | 7.35 tps | 56% cost reduction vs. GPU | | **256 Users** | Standard GPU Setup | $2,912 | 8.17 tps | Baseline infrastructure cost | | **512 Users** | Standard GPU Setup | $2,912 | 2.99 tps | Throughput collapses under load | | **512 Users** | Intel CPU + OpenVINO | $1,027 | 4.16 tps | 39% higher throughput than GPU |

Once concurrency expands to 512 users, GPU throughput drops sharply to 2.99 tokens per second, while the Intel CPU configuration powered by OpenVINO maintains a superior 4.16 tokens per second without increasing hardware expenditure. Furthermore, comparing uncompressed versus compressed models directly on the CPU platform at 512 users shows that model compression boosts high-throughput execution efficiency by 127%, illustrating the dual advantage of algorithmic compression combined with optimized host runtimes.

---

[18:04](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=1084s) Session Wrap-up and Concluding Remarks

As the presentation transitions to summarize key engineering and architectural takeaways with Bob Simmons, the session reaches its allocated time limit. The panel moderator steps in to close the session, prompting the audience to give a warm round of applause to the speakers from Multiverse Computing and Deloitte.

## Notable quotes > [02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) standing up an enterprise cloud GPU instance typically runs between $20,000 and $25,000 per month, whereas a standard CPU server or virtual machine costs roughly $3,000 to $5,000 per month. — Robert Simmons

> [02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) revenue-generating or mission-critical teams inevitably monopolize the available GPUs, leaving internal support units—such as HR, administration, and back-office analytics—starved for hardware. — Robert Simmons

> [02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) In high-stakes settings like commercial audit engagements, client NDAs legally forbid enterprise prompts from ever leaving the audited company's boundary. — Robert Simmons

> [02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) if continuous feedback learning is enabled, sensitive input data stored back into model weights could inadvertently be extracted by third parties who know how to prompt the model correctly. — Robert Simmons

> [05:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=329s) organizations can instantly take advantage of their vast, existing fleets of virtual machines, standard corporate laptops, and edge hardware without overhauling their operational architectures. — Robert Simmons

> [05:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=329s) When an internal auditor asks what the system did, engineers can cleanly trace the input-output relationship, isolate errors, and patch them immediately. — Robert Simmons

> [07:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=449s) At 128 concurrent users, the CPU-based architecture delivered a 95% throughput increase over the GPU baseline. — Bernie

> [07:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=449s) Running on Intel CPUs produced a 48% reduction in cost-per-token performance compared to running on GPU infrastructure. — Bernie

> [11:47](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=707s) show you the money — Saaket Varma

> [12:15](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=735s) A critical inflection point emerges between 256 and 512 concurrent users: beyond this threshold, GPU total response time deteriorates at a much steeper rate than CPU response time. — Saaket Varma

> [15:18](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=918s) At 256 concurrent users, running the compressed model on Intel CPUs incurs a monthly infrastructure cost of $1,027. Transitioning the same workload to a GPU-based infrastructure elevates the monthly bill to $2,912 — Saaket Varma

> [15:18](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=918s) Once concurrency expands to 512 users, GPU throughput drops sharply to 2.99 tokens per second, while the Intel CPU configuration powered by OpenVINO maintains a superior 4.16 tokens per second without increasing hardware expenditure. — Saaket Varma

## Mentioned Multiverse Computing, Deloitte, Intel, Intel Government Technologies, AWS, Ai4, Chris Zaharias, Robert Simmons, Saaket Varma, Bernie, Llama 3.1 8B, OpenVINO Model Server, Intel Advanced Matrix Extensions (AMX), 4th Gen Intel Xeon, NVIDIA A10G Tensor Core GPU

## Video description Chris Zaharias, VP U.S. Operations, Multiverse Computing Robert Simmons, Specialist Leader, Deloitte Saaket Varma, Senior Solution Specialist, Deloitte

Abstracts

In 2025, Deloitte, Multiverse Computing, and Intel initiated a collaborative study to assess the viability of CPU-based IT architectures for generative AI inferencing, with the goal of assessing cost reduction and efficiency improvement impacts. By combining Intel’s advanced processors and OpenVINO hosting technology, Multiverse’ Computing's AI model compression methodologies, and Deloitte’s AI engineering experience, the collaboration conducted a rigorous evaluation of CPU-based AI performance deployed in the AWS cloud. The study demonstrated that CPU-based inferencing can achieve accuracy and performance on par with traditional GPU-based approaches, while delivering notable efficiency gains as user demand scales. These findings can potentially have meaningful implications for organizations facing cost or resource constraints, and open new opportunities for deploying generative AI in edge computing and on-premise environments.

[Source](https://www.youtube.com/watch?v=28hTh1Gr_9w)

中文简报

德勤发现：高负载AI推理中Intel CPU性能超越Nvidia GPU 采用量子技术压缩模型使Intel CPU在高并发下性能超越Nvidia GPU达39%，同时降低56%的云托管成本。 本次展示探讨了如何将受量子启发的模型压缩技术与通用CPU基础设施相结合，以解决企业面临的GPU费用激增与硬件短缺瓶颈。Multiverse阐述了其基于物理学的压缩技术，该技术在不牺牲精度的前提下可将模型体积缩减高达90%；德勤则说明了促使机构转向本地化CPU推理的运营、财务和合规驱动因素。在第4代Intel Xeon处理器上针对80亿参数Llama 3.1模型的技术基准测试表明，与专用云GPU相比成本降低了48%至56%。实证测试关键显示，虽然GPU在单用户负载下处于领先，但在并发扩展至256及512个同时在线用户以上的高负载下，结合OpenVINO的CPU架构实现了更高的吞吐量与更强的稳定性。 Multiverse Computing 利用量子多体物理学数学框架映射神经网络相关性，可实现 80% 至 90% 以上的模型压缩率，并在大幅降低资源消耗的同时保持近乎无损的精度。 (0:04) 企业云端 GPU 实例月租高达 20,000 至 25,000 美元，远超仅需 3,000 至 5,000 美元的标准 CPU 基础设施；同时受审计保密协议和国防安全限制，企业与机构正加速转向隔离的本地化 CPU 推理方案。 (2:17) 测试显示，通过 AMX 和 OpenVINO 在第四代 Intel Xeon CPU 上运行 Multiverse 压缩 60% 的 Llama 3.1 8B 模型，单 token 成本降低了 48%，并在 128 并发用户下吞吐量超越 NVIDIA A10G 基准 95%。 (7:29) 当并发规模达到 512 名用户时，Intel CPU 方案凭借 4.16 tokens/秒的表现超越 GPU 架构（2.99 tokens/秒），且每月基础设施开销从 2,912 美元大幅降低 56% 至 1,027 美元。 (15:18)

Reading summary

Multiverse Computing, Deloitte, and Intel present an alternative to costly GPU-centric artificial intelligence deployments by combining quantum-inspired model compression with commodity 4th Gen Intel Xeon CPUs. Benchmarks on an 8-billion-parameter model indicate that compressed open-weights models running on CPUs can lower infrastructure costs by over 50% while outperforming dedicated GPUs under high-concurrency enterprise workloads. Quantum-Inspired Physics for Model Compression (0:04) Chris Zaharias of Multiverse Computing identifies soaring inference budgets and tight token-latency requirements as a critical bottleneck for enterprise AI adoption. Standard compression techniques—including uniform quantization, structured pruning, and knowledge distillation—frequently degrade model precision or broadly remove neuron structures. In response, Multiverse employs mathematical formulations derived from quantum many-body physics to model exponential parameter correlations across large language models and convolutional neural networks. According to Zaharias, mapping and preserving essential correlations while pruning non-essential redundancies allows models to be compressed by 80% to more than 90% with minimal loss in precision. In benchmark testing conducted with Intel and Deloitte on the Llama 3.1 8B model, Multiverse achieved a 60% footprint reduction, which yielded a 2x inference speedup across test configurations. Enterprise GPU Scarcity, Economics, and Security Constraints (2:17) Robert Simmons of Deloitte outlines the operational realities compelling organizations toward small, compressed language models on CPU hardware. Cloud GPU instances typically cost between $20,000 and $25,000 per month, compared to $3,000 to $5,000 per month for standard CPU servers. This wide disparity creates severe internal compute competition, where mission-critical or revenue-generating units monopolize GPU resources and leave operational teams, such as human resources and administration, compute-starved. Simmons also stresses compliance, privacy, and governance challenges. In corporate auditing and defense contexts, strict non-disclosure agreements and security classifications prohibit transmitting proprietary prompts to shared cloud services. Localized CPU environments mitigate data exfiltration risks and prevent sensitive input data from entering feedback loops embedded in model weights. Additionally, deploying smaller, domain-tailored models across existing enterprise fleets of laptops, virtual machines, and edge devices improves explainability and enables straightforward auditing of input-output behavior. Benchmark Dynamics: Concurrency and CPU Crossover (7:29) To evaluate whether general-purpose processors can match dedicated accelerators, Bernie of Intel Government Technologies and Saaket Varma of Deloitte tested an uncompressed and compressed Llama 3.1 8B model across two AWS environments. The baseline setup used an NVIDIA A10G Tensor Core GPU, while the comparative setup ran on a 4th Gen Intel Xeon processor utilizing Intel Advanced Matrix Extensions (AMX) and the OpenVINO Model Server across concurrency levels ranging from 1 to 1,024 users. The benchmark demonstrated a significant performance crossover tied to user load. At low concurrency, the GPU delivered higher initial throughput, achieving approximately 70 tokens per second for a single user compared to the CPU's 89% throughput parity. However, beyond an inflection point between 256 and 512 concurrent users, GPU latency and response times deteriorated sharply. At 128 concurrent users, the CPU setup delivered a 95% throughput advantage over the GPU baseline, and under a 256-user load, the CPU architecture outperformed the GPU by 50%. High-Load ROI and Cost-per-Token Economics (15:18) Deloitte's financial and throughput analysis quantified the operational trade-offs across concurrent loads. At 256 concurrent users, hosting the compressed model on Intel CPUs incurred a monthly infrastructure expense of $1,027 while maintaining 7.35 tokens per second. Running the same workload on GPU infrastructure cost $2,912 per month for 8.17 tokens per second, representing a 56% infrastructure cost reduction for near-equivalent performance. Across all benchmark testing, running on CPUs produced a 48% reduction in cost-per-token. Under heavier load at 512 concurrent users, GPU throughput fell to 2.99 tokens per second, whereas the Intel CPU configuration with OpenVINO sustained 4.16 tokens per second—a 39% throughput advantage at less than half the monthly cost. Furthermore, evaluating uncompressed versus compressed models directly on the CPU platform at 512 users revealed that model compression improved high-throughput execution efficiency by 127%, demonstrating the combined impact of algorithmic compression and matrix-optimized runtime engines. Deloitte, Intel, and Multiverse conclude that while GPUs remain superior for single-user, low-concurrency tasks, compressed models on CPU architectures with hardware matrix acceleration provide a more economical, secure, and scalable profile for enterprise concurrency above 256 users. The presentation did not address performance scaling across models larger than 8 billion parameters, nor did it evaluate accuracy retention across diverse qualitative reasoning benchmarks.

精炼阅读摘要

Multiverse Computing、Deloitte 与 Intel 联合展示了如何结合量子多体物理启发的高比例模型压缩与通用 CPU 硬件加速，以显著降低大语言模型的推理成本，并在高并发企业级场景中获得超越传统 GPU 的吞吐表现。 量子多体物理驱动的高效模型压缩 (0:04) Multiverse Computing 介绍了一种基于量子多体物理数学框架的压缩方法，用于替代传统的均匀量化、结构化剪枝和知识蒸馏。传统方法往往会导致精度下降或粗暴切除神经元网络，而该技术通过识别并保留神经网络、CNN 及 LLM 中真正驱动运行的复杂数学关联，剔除冗余连接。演讲者指出，该算法能够在保留近乎无损精度的同时，实现 80% 至 90% 以上的模型压缩率，从而缓解企业每年数百万至数十亿美元的推理预算压力与延迟瓶颈。 企业 GPU 算力瓶颈与数据隔离现实 (2:17) Deloitte 专家指出，专用 GPU 实例难以作为所有业务的通用解法。在基础设施成本方面，企业云 GPU 实例通常每月需要 20,000 至 25,000 美元，而标准 CPU 服务器或虚拟机仅需 3,000 至 5,000 美元。在组织架构中，核心业务或营收部门往往垄断了有限的 GPU 资源，导致人力资源和后台管理等运营支持团队缺乏可用算力。将压缩后的轻量化模型部署在 CPU 上，可直接复用企业现有的笔记本电脑、虚拟机及边缘设备基础设施。 此外，合规审计、国防和政府采购等高安全要求环境对数据外泄存在严格限制。商业审计中的客户保密协议（NDA）通常禁止输入提示离开企业边界；如果开启持续反馈学习，敏感信息还可能被写回模型权重并在日后被外部提示词套取。通过在本地专用 CPU 或边缘环境中运行压缩模型，能够在物理上杜绝数据向公有云模型泄露，同时小型专用模型也比庞大的通用黑盒 LLM 具备更强的可解释性与审查追踪能力。 高并发下 CPU 与 GPU 性能基准对比 (7:29) Intel 联合 Deloitte 及 Multiverse 对 Llama 3.1 8B 模型进行了基准测试，对比了 AWS 上的 NVIDIA A10G GPU 实例与搭载 AMX 矩阵加速引擎、经 OpenVINO Model Server 优化的第 4 代 Intel Xeon CPU 实例。在低并发场景下，GPU 展现出较高的单用户吞吐量（单用户约 70 tokens/秒，CPU 达到 GPU 吞吐的 89%）。然而，随着并发用户规模扩大至 256 人，CPU 架构的吞吐量保持相对平稳，并在 128 用户时比 GPU 基准高出 95%，256 用户时超出 50%，每 Token 成本降低 48%。 测试数据显示，当并发请求突破 256 达到 512 至 1024 用户时，系统出现明显的性能拐点。由于资源争用，GPU 的总响应时间急剧恶化，吞吐量出现滑坡；而基于 Xeon CPU 和 OpenVINO 的架构则展现出更稳定的延时表现与更高的吞吐弹性，适用于超过 500 名内部员工或外部客户同时发起查询的大型企业级部署环境。 并发规模与软硬件协同的投资回报分析 (15:18) 现场演示的 ROI 测算模型量化了在 1 至 1,024 并发请求下的经济性。在 256 个并发用户时，运行压缩模型的 Intel CPU 方案每月计算开销为 1,027 美元，而 GPU 架构则需 2,912 美元，单纯通过切换硬件底层即可削减 56% 的计算成本，两者的吞吐量此时大致持平（CPU 为 7.35 tokens/秒，GPU 为 8.17 tokens/秒）。 当并发扩展至 512 个用户时，GPU 架构的吞吐量骤降至 2.99 tokens/秒，而 CPU 架构在保持 1,027 美元月成本不变的情况下，吞吐量维持在 4.16 tokens/秒，性能高出 GPU 39%。与此同时，基准测试表明 Multiverse 的压缩算法将模型体积缩减了 60%，并在 512 并发下直接推动 CPU 上的执行效率提升 127%，体现了算法层级压缩与底层计算运行时优化的协同收益。 本场研讨展示了模型压缩与现代 CPU 运行时结合在降低企业 IT 预算和保障数据主权方面的技术可行性。不过现场演示集中在 8B 规模参数模型的测试，并未探讨百亿或千亿级前沿超大模型在此类架构上的迁移瓶颈与精度稳定性。

Detailed notes — zh

## Background 本内容由 Ai4 频道于 2025 年 8 月 28 日发布在 YouTube 平台。视频简介指出，德勤（Deloitte）、Multiverse Computing 与英特尔（Intel）展开联合研究，基于 AWS 云环境评估纯 CPU 架构运行生成式 AI 推理的可行性，旨在结合英特尔至强处理器、OpenVINO 运行时优化与量子多体物理启发的模型压缩技术，验证 CPU 推理在成本削减与扩展效率上匹敌甚至超越传统 GPU 的潜力。 ## TL;DR 演讲围绕企业落地大模型推理面临的高昂成本与 GPU 资源紧缺痛点展开，Multiverse Computing 率先介绍了源自量子多体物理的高效模型压缩技术，可在极低精度损失下实现 80% 至 90% 以上的体积精简。随后德勤与英特尔展示了在 AWS 上针对 Llama 3.1 8B 模型的联合基准测试，证明在 256 及以上高并发用户场景下，搭载 AMX 加速的英特尔第四代至强 CPU 结合 OpenVINO 实现了吞吐量对 GPU 的反超，总响应时间表现更为稳健。现场通过实时 ROI 工具演示证实，该方案不仅能降低 56% 以上的基础设施月度支出，还极大化解了企业内部门争抢 GPU 及敏感数据合规出境的严峻挑战。 ## Atmosphere & style 整场演讲呈现出务实、严谨且富有工业级说服力的专业氛围，讲者之间切换流畅，从商业痛点、物理数学算法层层推进至实测架构。随着数据科学家打出“用数据说话”的幽默口号并展示详尽的性能拐点折线图与实时 ROI 测算表，现场情绪逐步走向高潮，技术反常识的震撼感油然而生。最后在演示正酣之际被主持人因时间见底礼貌切断，在意犹未尽与全场热烈掌声中干脆利落地收尾。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=4s) Multiverse Computing 借鉴量子多体物理数学框架开发模型压缩技术，能够实现大模型 80% 至 90% 以上的体积压缩且几乎无损核心精度。 - [02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) 相比 GPU 服务器每月高达 20,000 至 25,000 美元的租用开销，CPU 服务器月租仅需 3,000 至 5,000 美元，可大幅缩减硬件开支超过 75%。 - [02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) 采用 CPU 部署紧凑型模型能打破组织内核心业务与支持部门对稀缺 GPU 算力的配额争夺，实现弹性即时可用。 - [02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) 在本地受控的 CPU 基础设施上运行压缩小模型，消除了敏感审计提示词或政务机密数据上云的合规与安全泄露风险，并支持边缘终端即时推理。 - [07:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=449s) Intel、德勤与 Multiverse 在 AWS 云端构建了三步基准测试方法论，横向对比 Llama 3.1 8B 原始与压缩模型在 GPU 和搭载 AMX 的第四代至强 CPU 上的性能表现。 - [07:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=449s) Multiverse 压缩技术对 Llama 3.1 8B 实现了 60% 的体积压缩，在 GPU 与 CPU 测试路径下均带来了 2 倍的推理速度提升。 - [12:15](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=735s) 在并发扩展评测中，256 至 512 并发用户成为关键性能拐点，GPU 性能出现急剧衰减，而 48 核 CPU 展现出更强的吞吐韧性与稳定性。 - [13:33](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=813s) 结合 Intel OpenVINO Model Server 优化引擎，CPU 架构在 500 名以上并发用户的大型企业级高负载场景中可输出反超 GPU 的并发吞吐表现。 - [15:18](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=918s) 实测 ROI 工具显示，在 256 并发用户下切换至 Intel CPU 架构运行压缩模型可直接降低 56% 的月度基础设施成本（从 $2,912 降至 $1,271）。 - [15:18](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=918s) 当并发规模上升至 512 用户时，Intel CPU + OpenVINO 的吞吐量稳定在 4.16 Tokens/s 并反超 GPU（2.99 Tokens/s），压缩模型的高吞吐效率提升达 127%。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=4s) 量子多体物理启发的模型压缩新范式：Multiverse Computing 登场 来自 Multiverse Computing 的美国运营负责人 Chris Zaharias 为本次演讲开场。他幽默地调侃 Multiverse 是一家“大家可能从未听说过的公司”，但这家总部位于西班牙的 AI 新创企业已经完成约 2.5 亿美元的融资，并掌握着极具颠覆性的 AI 与大语言模型（LLM）压缩技术。对于硬件设备制造领域的从业者，或者对首字延迟（Time to First Token, TTFT）及词元生成延迟（Inter-token Latency）有严苛要求的团队，抑或是每年在云端算力和模型推理上投入数百万乃至数十亿美元的企业而言，高昂的推理开销一直是难以承受的沉重负担。 为了解决这一行业顽疾，Multiverse Computing 从量子多体物理（Quantum Many-Body Physics）领域汲取灵感，构建了一套独特的数学计算框架。该框架能够将庞大且指数级复杂的建模问题大幅简化，精准识别出神经网络（无论卷积神经网络 CNN 还是大型语言模型 LLM）中真正对模型输出起决定性作用与无关紧要的深层关联。Chris 指出，传统的模型瘦身技术如量化（Quantization）、剪枝（Pruning）或知识蒸馏（Distillation），通常是一刀切地全盘降低模型精度或粗暴删减神经元；而 Multiverse 的方法则截然不同，它通过精细解耦和筛选网络内部关联，通常能够实现高达 80% 至 90% 以上的模型体积压缩，并且“极其关键的是，精度的下降幅度微乎其微”。 - **前沿压缩技术**：脱胎于量子多体物理数学理论，精准挖掘神经网络中的权重关联度。 - **压缩比与精度**：针对大型模型实现 80%~90%+ 的体积极致压缩，几乎无损核心推理精度。 - **行业痛点覆盖**：显著缩短首字延迟并降低词元间延迟，直接击中超大规模推理算力成本痛点。 --- ### [02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) 为什么告别昂贵的 GPU？企业与政务场景中的成本、可用性与数据安全痛点 随后，来自德勤（Deloitte）的专家领导 Robert (Bob) Simmons 登台，从实际落地架构与复杂业务场景出发，深入探讨了为何企业亟需在 CPU 上运行小型语言模型（SLM）或经过深度压缩的大模型。Bob 直言，基于 GPU 运行 LLM 绝非“放之四海而皆准的万能方案”；为了承载大模型的显存占用和算力需求，企业不得不采购昂贵的 GPU 服务器。在成本对比上，一台 GPU 服务器的月租成本高达 20,000 至 25,000 美元，而同等规格的 CPU 服务器每月仅需 3,000 到 5,000 美元。对于成本极度敏感的商业组织或政府机构而言，这笔开支差距关乎项目能否真正规模化铺开。 除了账面资金，GPU 的资源可用性也是组织内部的重大摩擦点。在典型的政府机构或大型商业公司（如审计事务所）中，核心业务线（Mission side）往往会优先霸占所有稀缺的 GPU 算力配额，导致后台支持部门（Support side）陷入严重的资源匮乏。如果组织依赖私有化部署或本地集群，支持部门想要利用 AI 提升内部效率就必须苦苦排队等待 GPU 释放；而将模型部署在 CPU 服务器或 CPU 云虚拟机上，则能彻底消除这种算力排队瓶颈。此外，数据安全与合规同样是一道不可逾越的红线：审计公司受限于保密协议（NDA），绝对不能将客户审计提示词（Prompts）发送到公有云外部环境；政府涉密数据或合同采购细节更无法承受数据泄露风险。一旦大模型引入反馈学习机制，存入模型的敏感数据甚至可能被恶意提示词刺探诱导出来。将压缩后的小模型完全封装在本地可控的 CPU 基础设施上，便成了解决安全合规的最佳途径。 Bob 进一步强调了紧凑型模型的其他关键优势：高透明度与可解释性。相比试图处理所有复杂任务却宛如黑盒的通用大模型，针对特定垂直任务训练的小模型在输入输出上有极其清晰的因果链条，一旦审计员核查逻辑或出现异常，开发人员可以精准定位并迅速修复。加之 CPU 原生架构部署简易、维护平滑，企业不仅能将其灵活部署在云端 CPU 虚拟机上，还能直接下沉到员工笔记本电脑乃至野外前线边缘设备（Edge Devices），实现高速、低延迟的现场即时响应。 ```mermaid flowchart TD accTitle: CPU vs GPU 部署决策权衡 accDescription: 展示企业在算力成本、资源可用性与数据安全三维度下选择 CPU 架构的驱动逻辑 subgraph GPU_Pain ["传统 GPU 方案痛点"] G1["高昂硬件/云成本<br/>每月 $20,000 - $25,000"] G2["资源争夺与短缺<br/>业务端霸占，支持端排队"] G3["数据合规风险<br/>提示词出境泄露、NDA 违规"] end subgraph CPU_Benefits ["CPU + 压缩模型优势"] C1["大幅缩减开销<br/>每月 $3,000 - $5,000"] C2["弹性即时可用<br/>利用现有通用 CPU/虚拟机"] C3["本地可控与高解释性<br/>私有闭环、边缘终端即时推理"] end GPU_Pain -->|驱动架构转型| CPU_Benefits ``` - **服务器成本骤降**：云端或机房月度开销从 20,000~25,000 美元剧降至 3,000~5,000 美元，降幅超过 75%~80%。 - **算力分配解耦**：解决内部支持部门与核心业务部门的 GPU 配额争夺，摆脱硬件排队限制。 - **合规与边缘下沉**：消除 NDA 及涉密数据上云风险，完美支持单机笔记本与边缘端快速响应。 --- ### [07:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=449s) Intel 与 Deloitte 的三步基准测试方法论：CPU 推理实测与吞吐量跃升 来自 Intel 政府技术事业部（Intel Government Technologies）的解决方案架构师 Bernie 随后登台，详细拆解了 Intel、Deloitte 与 Multiverse Computing 联合开展的三步走（Three-Step Approach）基准测试实操与测试成果。为了验证纯 CPU 架构运行大语言模型推理的可行性与极限性能，联合团队在 AWS 云端环境搭建了严密的对比测试矩阵： ```mermaid flowchart LR accTitle: 联合基准测试三步实施路径 accDescription: 从 GPU 基准构建，到 Xeon 加速 CPU 部署，再到多并发性能对比的三阶段流程 subgraph Step1 ["第一步：GPU 基准构建"] S1["Llama 3.1 8B 原始模型"] --> G_Inst["AWS 实例<br/>NVIDIA A10G Tensor Core GPU"] end subgraph Step2 ["第二步：CPU 硬件加速与优化"] S2["Llama 3.1 8B 原始 / 压缩模型"] --> C_Inst["AWS 实例<br/>Intel 第四代 Xeon (AMX)"] C_Inst --> OV["OpenVINO Model Server<br/>优化引擎与容器化服务"] end subgraph Step3 ["第三步：高并发横向评测"] G_Inst --> Bench["吞吐量、词元延迟与成本核算"] OV --> Bench end Step1 --> Step2 --> Step3 ``` 1. **第一步（构建 GPU 参照系）**：选取主流开源的 Llama 3.1 8B 参数模型，部署在搭载 NVIDIA A10G Tensor Core GPU 的 AWS 云实例上，运行标准推理基准并记录性能基线。 2. **第二步（构建 Intel CPU 优化环境）**：在搭载 Intel 第四代至强可扩展处理器（Xeon）的纯 CPU 云实例上部署相同的 Llama 3.1 8B 模型。第四代至强原生集成了高级矩阵扩展（AMX）硬件加速单元，可成倍提升每时钟周期的矩阵计算能力；同时引入 OpenVINO 模型优化器及 OpenVINO Model Server，作为容器化和虚拟化云端环境的高性能推理前端服务。在该环境中，团队分别测试了未经压缩的原始模型与经过 Multiverse 压缩处理后的模型。 3. **第三步（多维度横向比对与并发压测）**：模拟真实企业场景中从单用户到多用户同时访问 LLM 的并发压力（逐步递增用户数），全面比对词元吞吐量、响应延迟与整体运行成本。 评测结果展现出了极具颠覆性的数据表现： - **高并发吞吐量反超**：在 128 个并发用户的高负载场景下，基于 CPU 的基础设施实现了高达 95% 的吞吐量提升；当并发用户数进一步拉升至 256 时，CPU 架构的性能相较 GPU 实现了 50% 的显著领先。 - **单用户与高并发的分水岭**：在仅有 1 个用户访问的极低负载下，纯 CPU 实例的表现略逊于 GPU，约为 GPU 推理性能的 89%；但随着并发请求攀升，CPU 凭借内存通道带宽与 AMX 并行加速机制展现出更强的扩展弹性。 - **综合成本削减**：以单位成本输出的词元数（Tokens per Second per Dollar）衡量，纯 Intel CPU 架构带来了 48% 的大幅成本缩减。 - **Multiverse 压缩带来的倍增效应**：Multiverse 对 Llama 3.1 8B 实现了 60% 的体积压缩，且在 GPU 与 CPU 测试路径下均带来了整整 2 倍（2x）的推理速度提升。 分享完核心数据指标后，Bernie 将麦克风转交给德勤的高级解决方案专家 Saaket Varma。作为数据科学家，Saaket 带着轻松幽默的口吻接棒表示，他将用详实客观的可视化图表“用数据说话”（"Show me the money, show me the data"），为现场观众深度拆解各项实验曲线的具体走向。 ### [12:15](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=735s) CPU 与 GPU 性能基准对比：并发扩展与拐点 在模型压缩效果评估中，团队深入对比了压缩模型在 CPU 与 GPU 架构下的实际表现。测试图表中通过蓝色指标展示每秒处理 Token 数（Tokens per Second），橙色指标展示总响应时间（Total Response Time），圆形代表 GPU 指标，方形则代表 CPU 指标。测试场景模拟了从 1 个并发用户逐步递增至 1024 个并发用户的真实高负载压力。 在单用户低并发场景下，GPU 展现出明显的原始速度优势，初始吞吐量高达每秒 70 个 Token。然而随着并发用户数快速上升至 256 人，GPU 的性能出现了急剧衰减。相比之下，基于 48 核 CPU 构建的架构表现得极为平稳与严苛（Stringent），仅在并发数由 30 增至 64 时因核心数限制出现轻微波动。当并发量达到 256 至 512 个用户这一关键“拐点”（Inflection Point）并进一步向 1024 用户扩展时，GPU 的总响应时间劣化程度显著超过了 CPU，CPU 在大规模并发下反而展现出了更强的吞吐韧性。 ```mermaid flowchart LR accTitle: CPU vs GPU Concurrency Scaling Performance subgraph LowConcurrency["低并发阶段 (1 - 64 用户)"] A["GPU: 初始高吞吐 (70 tokens/s)"] B["CPU (48核): 吞吐适中平稳"] end subgraph InflectionStage["拐点阶段 (256 - 512 用户)"] C["GPU 性能快速劣化"] D["CPU 表现恒定且超越 GPU"] end subgraph HighConcurrency["高并发阶段 (512 - 1024 用户)"] E["GPU 响应时间严重拉长"] F["CPU + OpenVINO: 吞吐与稳定性占优"] end A --> C --> E B --> D --> F ``` ### [13:33](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=813s) 架构部署策略与 OpenVINO 模型服务优势 基于基准测试结果，团队提炼出了企业大模型部署的三大核心策略。在低并发场景下，如果控制成本是企业的核心考量，直接切换到 CPU 能够为企业节省巨额的基础设施开支，同时提供与 GPU 相当可用的性能指标，用户仅需付出微不足道的额外等待时间。 而当企业负载扩展至 256 到 1024 个并发用户时，部署在 CPU 上并通过 Intel OpenVINO Model Server 驱动的压缩模型展现出了决定性优势。OpenVINO 的加速支持使推理过程更加迅捷、稳定且严苛，甚至能输出高于 GPU 的并发吞吐表现。对于拥有庞大员工基数、日常面临 500 名以上用户同时调用模型的大型企业级应用而言，CPU 方案不仅具备压倒性的成本优势，更能随着并发规模的扩大提供更为一致、可靠的性能输出。 - **低并发部署法则**：用户量较少时，CPU 作为 GPU 的低成本替代方案，以微小的延迟代价换取极高的资金利用率。 - **高并发企业方案**：超过 500 用户同时访问时，CPU 展现出更平稳的抗压表现，避免了 GPU 在高负载下响应时间大幅滑坡的问题。 - **软硬协同加速**：Multiverse 的模型压缩技术与 Intel OpenVINO 运行时环境结合，释放了通用计算硬件在生成式 AI 推理中的深层潜力。 ### [15:18](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=918s) 实时 ROI 工具演示：成本与吞吐量实测数据 现场演示了一款动态投资回报率（ROI）计算工具，通过滑动并发用户滑块（1 至 1024 用户），实时对比切换 Intel CPU 与 GPU 架构时的成本、Token 吞吐量及响应时间变化。 在 256 个并发用户场景下，采用 Intel CPU 运行压缩模型时的月度基础设施成本仅为 1,271 美元。团队指出，仅通过将大型语言模型（LLM）精简为小型语言模型（SLM），在模型压缩前就已经降低了 56% 的费用；而当在工具中一键切换至 GPU 架构时，月度费用立刻攀升至 2,912 美元，证明仅改变底层计算架构便能实现 56% 的成本削减。在此并发量下，CPU 的吞吐量为 7.35 Tokens/s，与 GPU 的 8.17 Tokens/s 极为接近。 当并发用户数进一步拉升至 512 时，硬件性能格局发生逆转：GPU 的吞吐量急剧跌落至 2.99 Tokens/s，而运行 OpenVINO 的 Intel CPU 吞吐量则稳定在 4.16 Tokens/s。在月度基础设施费用保持不变的前提下，对比未压缩模型与压缩模型在 512 用户下的表现，压缩模型带来的高吞吐效率提升高达 127%，以严密的数据印证了单位 Token 成本的显著下降。 | 并发用户规模 | 部署架构与方案 | 月度基础设施成本 ($) | 吞吐量 (Tokens/s) | 成本削减 / 效率表现 | |:--- |:--- |:--- |:--- |:--- | | **256 用户** | Intel CPU + 压缩模型 | $1,271 | 7.35 | 相比 GPU 架构节省 56% 成本 | | **256 用户** | GPU 方案 | $2,912 | 8.17 | 吞吐量略高但成本昂贵 | | **512 用户** | GPU 方案 | $2,912 | 2.99 | 性能大幅劣化 | | **512 用户** | Intel CPU + OpenVINO | $1,271 | 4.16 | 吞吐量反超 GPU，高吞吐效率达 127% | ### [18:04](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=1084s) 结语与现场掌声 在展示完详实的数据与 ROI 计算模型后，演讲者 Saaket Varma 准备将发言权移交给德勤的 Bob（Robert Simmons）以对整场联合研究的核心结论做最终总结。此时现场主持人介入提醒活动时间已到："We are actually short for time."（我们的时间确实不够了），并随即号召全场观众为台上的 Multiverse 与德勤专家团队献上热烈的掌声，演讲在全场的掌声中圆满结束。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=4s) 大家可能从未听说过的公司 — Chris Zaharias > [00:04](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=4s) 极其关键的是，精度的下降幅度微乎其微 — Chris Zaharias > [02:17](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=137s) 放之四海而皆准的万能方案 — Robert Simmons > [07:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=449s) Show me the money, show me the data — Saaket Varma > [07:29](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=449s) Tokens per Second per Dollar — Bernie > [12:15](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=735s) Tokens per Second — Saaket Varma > [12:15](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=735s) Total Response Time — Saaket Varma > [18:04](https://www.youtube.com/watch?v=28hTh1Gr_9w&t=1084s) We are actually short for time. — 主持人 ## Mentioned Multiverse Computing, Deloitte (德勤), Intel (英特尔), AWS, Chris Zaharias, Robert Simmons, Bernie, Saaket Varma, Llama 3.1 8B, NVIDIA A10G Tensor Core GPU, Intel 第四代至强可扩展处理器 (Xeon), AMX (高级矩阵扩展), OpenVINO, OpenVINO Model Server ## Video description Chris Zaharias, VP U.S. Operations, Multiverse Computing Robert Simmons, Specialist Leader, Deloitte Saaket Varma, Senior Solution Specialist, Deloitte Abstracts In 2025, Deloitte, Multiverse Computing, and Intel initiated a collaborative study to assess the viability of CPU-based IT architectures for generative AI inferencing, with the goal of assessing cost reduction and efficiency improvement impacts. By combining Intel’s advanced processors and OpenVINO hosting technology, Multiverse’ Computing's AI model compression methodologies, and Deloitte’s AI engineering experience, the collaboration conducted a rigorous evaluation of CPU-based AI performance deployed in the AWS cloud. The study demonstrated that CPU-based inferencing can achieve accuracy and performance on par with traditional GPU-based approaches, while delivering notable efficiency gains as user demand scales. These findings can potentially have meaningful implications for organizations facing cost or resource constraints, and open new opportunities for deploying generative AI in edge computing and on-premise environments. [Source](https://www.youtube.com/watch?v=28hTh1Gr_9w)

Source quotes

[0:04] Chris Zaharias: 大家可能从未听说过的公司 [0:04] Chris Zaharias: 极其关键的是，精度的下降幅度微乎其微 [2:17] Robert Simmons: 放之四海而皆准的万能方案 [7:29] Saaket Varma: Show me the money, show me the data [7:29] Bernie: Tokens per Second per Dollar [12:15] Saaket Varma: Tokens per Second [12:15] Saaket Varma: Total Response Time [18:04] 主持人: We are actually short for time.

Source quotes

[2:17] Robert Simmons: standing up an enterprise cloud GPU instance typically runs between $20,000 and $25,000 per month, whereas a standard CPU server or virtual machine costs roughly $3,000 to $5,000 per month. [2:17] Robert Simmons: revenue-generating or mission-critical teams inevitably monopolize the available GPUs, leaving internal support units—such as HR, administration, and back-office analytics—starved for hardware. [2:17] Robert Simmons: In high-stakes settings like commercial audit engagements, client NDAs legally forbid enterprise prompts from ever leaving the audited company's boundary. [2:17] Robert Simmons: if continuous feedback learning is enabled, sensitive input data stored back into model weights could inadvertently be extracted by third parties who know how to prompt the model correctly. [5:29] Robert Simmons: organizations can instantly take advantage of their vast, existing fleets of virtual machines, standard corporate laptops, and edge hardware without overhauling their operational architectures. [5:29] Robert Simmons: When an internal auditor asks what the system did, engineers can cleanly trace the input-output relationship, isolate errors, and patch them immediately. [7:29] Bernie: At 128 concurrent users, the CPU-based architecture delivered a 95% throughput increase over the GPU baseline. [7:29] Bernie: Running on Intel CPUs produced a 48% reduction in cost-per-token performance compared to running on GPU infrastructure. [11:47] Saaket Varma: show you the money [12:15] Saaket Varma: A critical inflection point emerges between 256 and 512 concurrent users: beyond this threshold, GPU total response time deteriorates at a much steeper rate than CPU response time. [15:18] Saaket Varma: At 256 concurrent users, running the compressed model on Intel CPUs incurs a monthly infrastructure cost of $1,027. Transitioning the same workload to a GPU-based infrastructure elevates the monthly bill to $2,912 [15:18] Saaket Varma: Once concurrency expands to 512 users, GPU throughput drops sharply to 2.99 tokens per second, while the Intel CPU configuration powered by OpenVINO maintains a superior 4.16 tokens per second without increasing hardware expenditure.

Original transcript

[0:04] Uh my name is Chris Harius. I run uh US [0:07] operations for a company called [0:08] Multiverse. And um my colleagues at [0:12] Deote and Intel um are going to be [0:14] presenting on um model compression and [0:19] making AI models more efficient. Uh I'll [0:21] introduce myself. Um I already did that. [0:25] Um company you've never heard of. Uh, [0:27] Multiverse Compu Computing is a Spanish [0:30] AI company that um has raised probably [0:34] about 250 million and that has extremely [0:38] um disruptive technology for the [0:40] compression of AI models and LLMs. Um [0:45] for those of you who work in the device [0:48] space or who have efficiency [0:50] requirements for time to first token [0:53] inner token latency uh if you're [0:55] spending uh millions to billions on [0:59] compute and inference on an annual basis [1:01] uh we've developed really interesting [1:03] technology that uh comes from the world [1:05] of quantum manybody physics and it's [1:08] essentially a math framework that [1:10] simplifies uh massive exponential [1:13] modeling problems. [1:14] uh and which works very well at finding [1:17] the correlations uh in a neural network, [1:22] a CNN or an LLM that do and don't matter [1:25] to the functioning of a model. And so if [1:28] you're familiar with methods like [1:29] quantization, pruning, uh distillation, [1:33] which tend to either across the board [1:35] reduce um the precision of a model or [1:39] the number of neurons in a model, our [1:41] approach is is very distinct in that we [1:45] find the uh the the uh correlations [1:48] within the network that do and don't [1:51] matter. And we're therefore able to [1:52] typically achieve upwards of 80 90% plus [1:57] compression of of AI models and LLM but [2:00] critically with very very minimal drop [2:03] in precision. Um and so um that said um [2:07] I'm going to hand it over to my [2:09] colleagues at um at Deote and Intel who [2:12] are going to walk you through what [2:13] they've been doing with uh with this [2:15] compression technology. [2:17] Hi, Hanky already introduced me Bob [2:20] Simmons um Deote and what I'd like to go [2:23] over is um some of the requirements that [2:26] uh look at the smaller language models [2:29] or compressed language models, [2:31] compressed larger language models for [2:33] more complex use cases. Um one of them [2:36] being costs. So it's like LLMs are not a [2:41] on GPUs are not a one-sizefits-all. They [2:43] cost a lot and if you're going to run an [2:45] LLM, you're going to have to put it on a [2:47] GPU just for the memory size, the [2:49] processing power and everything else. [2:51] Um, looking at the cost, if you ever [2:56] stood up or you know installed the [2:58] servers and everything else and or you [3:00] got to pay a cloud service provider, a [3:02] GPU may run 20 $25,000 a month. A CPU [3:07] server may run three 4,000 $5,000 a [3:10] month. So if you have a costconscious [3:13] part of an organization commercial or uh [3:17] government which where I am in um that's [3:19] a big consideration [3:22] um and depending on the organization [3:25] um I'm going to you know look at a [3:26] commercial organization like an audit [3:28] company um you have infrastructure um [3:33] concerns in terms of I may have a the [3:37] mission side of the house is going to [3:38] take your GPUs I'm just going to put it [3:40] that way. Then you have the support side [3:41] of the house which always gets squeezed [3:43] for resources to go put those over on [3:45] the mission side. That's government or [3:47] commercial. If you have a set number of [3:50] GPUs in your organization, instead of [3:52] using an outside cloud service provider, [3:54] you installed it on your premise in your [3:57] cloud or in your servers, you're going [3:59] to have a GPU availability issue. If [4:02] you're on the support side and you're [4:03] trying to do some AI work to make your [4:06] job more, you know, efficient, [4:09] do I wait for a GPU to become available [4:12] to load my language model on it that [4:14] does my task or can I just have it on a [4:17] CPUbased server but less expensive or on [4:22] the a cloud a CPUbased virtual machine? [4:25] Um, so you have availability issues and [4:27] I get that a lot where I work. So, and [4:30] then um on the security concerns. [4:33] So, let's say you if you're an audit [4:36] company, you don't want your prompts [4:38] leaving your company. You may not want [4:41] your prompts leaving this particular [4:44] audit of this particular company out of [4:47] their environment because, you know, you [4:50] signed an NDA. Um it's a legal issue. If [4:53] the data gets leaked, same thing. If [4:55] you're in the government, you're working [4:56] with classified information, you can't [4:58] have those prompts leave. If you work [5:00] within the contract shop of a government [5:02] organization, you don't want your [5:04] contract information, the prompts. If [5:06] you do feedback learning into your [5:09] language model and you're putting data, [5:10] now that data is being stored, [5:13] you know, stored into the language [5:15] model, a different prompt can come in [5:18] and possibly get some of that [5:19] information out if they know how to [5:20] prompt it correctly. So you want that to [5:22] be protected within your own environment [5:24] and not out within the company itself. [5:27] So [5:29] going through that um so this is about [5:32] either running SLMs on CPUs or larger [5:36] compressed models more complex use cases [5:38] on CPUs. The cost, like I said, the [5:41] costs are going to drop from 20 $25,000 [5:44] a month to four maybe $5,000 a month [5:47] depending on the size of the uh virtual [5:50] server you may be using. Um, you can put [5:52] it on laptops, you can put it on, you [5:54] know, virtual server clouds, you can [5:55] take it out on edge devices. [5:58] Um the the other thing is the increased [6:01] transparency of having a smaller model [6:05] is you can train it to do very specific [6:07] things and it's very explainable. So it [6:11] does something and you know the auditor [6:13] comes in what did it do? It does this [6:16] here's the input here's the output. If [6:18] it does slightly different you know [6:20] exactly what that is. You can go in [6:21] there and fix it. But so the [6:23] explanability of what it does is much [6:25] easier than when you have a large [6:27] language model that does many complex [6:28] things and you're trying to use it for [6:30] one specific thing. [6:33] So and also with this you can have a [6:35] portfolio of these things. So we'll get [6:36] into that. So on the deployment and [6:39] maintenance you know it's a simple [6:40] architecture if you're going to be using [6:42] CPUs you can you it's existing [6:45] architecture you're going to be a [6:46] company it's either the cloud service [6:48] provider that may have a whole bunch of [6:50] virtual machines running on CPUs versus [6:53] the virtual machines running GPUs so [6:56] scaling is easier so looking at that um [7:00] this whole this presentation is for [7:03] running the small language models or [7:04] compressed larger language models [7:06] actually with more complex use cases on [7:08] more abatable, more costefficient [7:11] resources. Besides the other benefits [7:12] that Chris talked about, you got faster [7:14] inference speeds. So if you're going to [7:16] go put on in, you know, edge devices and [7:18] you need answers quickly, you're out in [7:20] the field. [7:22] Bill that tackles that. So with that, [7:25] I'll hand this off to Bernie from Intel. [7:29] >> Yeah. So as mentioned, my name is [7:30] Bernie. Um I work with uh Intel. I'm a [7:33] solution architect uh within the Intel [7:35] government technologies group and I was [7:38] brought in to help uh Deote in [7:40] Multiverse um run some experiments on uh [7:45] doing some uh inferencing running LLM on [7:48] uh natively on a GPU on a um CPU excuse [7:51] me. So we the methodology and the um [7:55] performance measurements we took a [7:56] three-step approach. Um so in step one [8:00] uh we took a llama um 3.1 8 billion [8:04] parameter model um and we ran that on uh [8:08] AWS instance that was running a uh A10G [8:13] tensor core GPU. Um so we we we did that [8:17] and we ran we benchmarked uh on that [8:21] configuration. [8:22] So on the second part of the um the [8:26] approach uh we took the same uh llama [8:29] 3.1 billion uh 8 billion parameter model [8:34] and uh we ran it on a uh fourth [8:37] generation Xeon. So for those of you [8:39] that don't know uh with starting with [8:41] the fourth generation uh Intel added [8:44] some acceleration features um we have a [8:48] acceleration feature called um uh Max [8:52] that um basically accelerates your [8:54] computations per clock cycle. Um in [8:57] addition to that we used uh open veno [9:00] model server. So we have openVO which is [9:03] a um model optimizer [9:07] uh in addition to um an inferencing [9:10] engine and uh the model server version [9:13] is as as the front end to allow you to [9:17] run that in a cloud environment in a [9:19] virtualized or containerized [9:20] environment. [9:22] So with those two um features uh in mind [9:27] uh we saw some pretty good results. So [9:29] we ran we repeated the tests. We ran the [9:32] um uh both the compressed and [9:34] uncompressed models on uh the CPU only [9:40] uh again using the AWS cloud instance. [9:43] Uh and then step three uh we compared [9:46] the results. So we did the benchmarking [9:48] and we did the comparison and we found [9:50] some some some very interesting [9:52] findings. [9:54] um number one on uh you know we found uh [9:59] 95% uh throughput increase um on the CPU [10:05] infrastructure [10:07] um when there was 128 users. So in other [10:11] words um if you have an LLM running in a [10:14] cloud what we did is we went from one [10:16] and we increased that and kept measuring [10:19] the performance. Uh so if you're at an [10:21] organization and uh the organization is [10:23] simultaneously hitting an LLM, uh we're [10:26] trying to evaluate the performance. [10:29] Um number two, we saw um 48% reduction [10:33] in cost, right? As expected, right? [10:35] You're you're running a an LLM in a [10:38] cloud infrastructure [10:40] um without a GPU. Um and then uh [10:46] you know contrarily we we ran on on an [10:48] Intel only and the cost reduction as [10:51] measured by tokens per second we saw 48% [10:54] reduction in cost performance gains um [10:58] so uh when there's one user uh on the [11:01] CPU instance um it doesn't perform as [11:04] well it was 89% of the performance [11:08] uh compared to that of a GPU running it [11:10] on the GPU view. However, um as you [11:14] increased in the total number of users, [11:17] uh so at 256 users, we saw a 50% [11:20] increase [11:22] uh in in um the performance. And then [11:26] lastly, um 60% model per um compression [11:30] when using Multiverse. [11:32] And we found that with the compressed uh [11:35] models uh from Multiverse [11:38] um on both cases the the speed increase [11:41] was 2x. With that I'll turn over to [11:43] Saket who's going to dive a little [11:44] deeper into uh the results. [11:47] >> Thanks Bonnie. [11:52] >> I'm Sake Warmer for Deote. Um and um I'm [11:57] I'm just going to show you the money, [11:59] right? Show me the data, right? It's uh [12:02] I'm u data scientist right I believe in [12:05] numbers right and uh this is something [12:07] that we want to share with the community [12:10] um what we have found right um what [12:13] you're looking at in a graph here and as [12:15] as Bernie said that we are comparing [12:18] compressed models in a CPU and a GPU [12:22] um we're looking at uh sorry about that [12:27] we're looking at um tokens per second in [12:30] blue. We're looking at total response [12:32] time in orange, right? And these circles [12:36] that you see here, these are G uh GPU [12:38] metrics and squares are CPU metrics, [12:42] right? And what we're going from is one [12:44] user to,024 users, right? Uh and these [12:48] are uh uh uh concurrent users that are [12:52] hitting the model, right? What we see is [12:55] that when we start from one users, yes, [12:57] the GPUs you do have increased tokens [13:00] per second. It's 70 tokens per second, [13:02] but you can see that how fast it [13:04] degrades as soon as we get to 256 users. [13:07] And in a CPU, you see that it's [13:09] stringent. It stays constant. There is a [13:13] slight drop from 30 to 64 because we're [13:15] using 48 CPUs. Um and so there's a [13:19] slight drop but you can see that there's [13:21] an inflection around 256 to 512 users [13:25] and beyond that GPUs actually start to [13:28] degrade when you look at the total [13:29] response time more than CPUs [13:33] right um so three things that we found [13:37] um from our work um when we did this [13:40] analysis was one um in at at low [13:44] concurrency levels uh if cost is the [13:48] factor right um you can switch to a CPU [13:51] you'll be saving tons of money there are [13:54] comparable metrics uh as compared to [13:57] GPUs yes you can wait a little bit [13:59] longer but once you scale up to 256 [14:03] 20024 users with these compressed models [14:07] that we are um pushing in in uh uh in a [14:11] CPU and using a open veno model server [14:14] that just makes it more faster and more [14:16] stringent and uh you are able to get u [14:19] higher performance right as compared to [14:21] even GPUs. Uh so as far as uh um uh [14:26] deployment strategy goes um there uh if [14:30] if you have um uh your uh number of [14:34] users are less you can use CPUs uh as a [14:38] cost alternative to GPUs and um if [14:42] you're looking at um you know if you [14:45] have a huge enterprise you have more [14:46] number of employees you you have a use [14:48] case where you're going to have more [14:50] than 500 users um using um um your um uh [14:57] they're uh hitting the model that's when [14:59] uh CPU um uh is is um uh it provides [15:05] more custo uh consistent performance um [15:08] and um uh it's it's u u you can see that [15:13] um it it also provides a better [15:15] performance as as we scale up right um [15:18] so with that what I'm going to do is um [15:21] I also have a small ROI tool um to just [15:24] display. Um I'm just going to pop up [15:27] here and uh showcase this. So what [15:31] you're seeing at the top is concurrent [15:33] users and I'm just going to slide that [15:35] down from one concurrent user 2024 and [15:39] I'm going to show you real time what's [15:40] going to happen when I switch from Intel [15:43] CPUs to GPUs. And we're going to look at [15:45] the cost. We're going to look at the [15:46] high uh throughput tokens and the [15:48] response time, right? And and as we [15:50] reach that inflection point from 256 to [15:53] 512 users, you're going to see the [15:55] difference and the cost. Um I'm just [15:58] going to move this here. Uh this is just [16:00] at one users. So I'm just going to move [16:02] this to um 256 users. Right now we're [16:06] using Intel CPUs and we're using [16:08] compressed model. It gives you a monthly [16:11] cost of,27. [16:14] uh uh we've already saved 56% of money [16:18] by uh moving from LLM to SLM right and [16:21] this is before compression right you're [16:23] seeing a 56% gain um and u to make that [16:28] point what I'm going to do is I'm just [16:29] going to switch to GPUs here you can see [16:32] that the cost changes to 2912 and this [16:35] is a monthly cost um so the [16:37] infrastructure just by changing the [16:39] infrastructure you're decreasing the [16:41] cost by 56% right Uh, and we are we're [16:45] still at 256 users. With Intel CPUs, [16:48] you're going to see 7.35 tokens per [16:51] second. With GPUs, yes, it's slightly [16:53] more, 8.17. It's comparable. As soon as [16:57] you jump to 512, [16:59] you're seeing that uh throughput tokens [17:01] per second is 2.99. The cost doesn't [17:04] change. When you move to uh Intel CPUs [17:07] with open veno, you're you have [17:10] increased uh throughput token 4.16. [17:13] Right? So as you scale up there's this [17:15] that differential right now if I were to [17:17] go down and compare between uncompressed [17:20] and compressed model and the savings uh [17:23] with uh 512 users uh you're seeing that [17:28] it's running at 4 uh um uh the tokens [17:31] per second is 4.16. [17:34] uh your monthly cost doesn't change [17:35] because you're still running both the [17:37] models in CPUs but efficiency wise [17:40] because the model is running faster the [17:43] high throughput uh efficiency is 127% at [17:47] um uh 512 uh users right so that's how [17:51] we are actually calculating um uh the [17:55] entire cost and the cost reduction per [17:57] tokens and that's that's how u uh we are [18:00] u uh looking at the benefits [18:04] So with that, what I'm going to do is [18:06] I'm just going to pass on um um to um [18:11] Bob to summarize the takeaways. [18:14] >> We are actually short for time. Everyone [18:17] give it up for our lovely panel members. [18:19] Let's give him a warm round of applause.