Stanford Finds Local AI Handles 88% of Queries, Cuts Power 80%

Xiaol.x

Routing the bulk of everyday workloads to small on-device models cuts computing power demand by up to 80%, even as centralized data-center chips remain far more energy-efficient on identical tasks.

AI-assisted summary of Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — original source. Check the original for full context.

Source published: 2026-09-17

Key ideas

Stanford and Together AI researchers benchmarked over 20 small local models across 1M real single-turn queries, finding that an ensemble of local models under 20B active parameters can satisfactorily answer 88.7% of workloads. (0:01)
Intelligence per watt (accuracy divided by accelerator power) improved by 5.3× over two years, driven primarily by algorithmic model advances (3.1×) rather than silicon hardware efficiency gains (1.7×). (5:45)
Data-center accelerators running identical models remain 1.4× more power-efficient per watt and up to 7.4× more efficient per joule than high-end laptop chips like Apple's M4 Max due to massive memory bandwidth disparities. (9:01)
Deploying an intelligent router that sends ~80.7% of common queries to local edge models and reserves cloud frontier models for hard queries reduces total inference energy by up to 80.4%, compute by 77%, and cost by 74%. (10:44)

Detailed notes

## Background Published on YouTube by channel Xiaol.x on September 17, 2026, this video is Episode 04 of the 'Loop State' series. The show notes describe it as a two-host Q&A between Mara and Theo breaking down the Stanford and Together AI research paper 'Intelligence per Watt: Measuring Intelligence Efficiency of Local AI' (arXiv 2511.07885), which tests over 20 small local models against nine accelerators across one million real-world queries to evaluate the viability of edge inference.

## TL;DR The discussion analyzes empirical findings showing that small local model ensembles can answer 88.7% of everyday chat and reasoning queries, driven by a 5.3× increase in intelligence per watt over two years. However, dedicated data-center accelerators like Nvidia's B200 and SambaNova's SN40L still outclass laptop chips like Apple's M4 Max by 1.4× to 7.4× on energy efficiency per joule due to severe memory bandwidth bottlenecks. True efficiency gains emerge not from laptop silicon beating server chips, but from intelligent routing: resolving roughly 80% of routine queries locally while offloading complex reasoning to the cloud slashes net energy, compute, and cost by 60% to 80%. The hosts also highlight methodological caveats, including single-turn evaluation limits and an apparent anomaly where the paper's data implies a ~353 W power draw for the M4 Max despite Apple's 145 W wall-power rating.

## Atmosphere & style The presentation carries the crisp, intellectual rhythm of a technical deep-dive podcast, pairing Mara's pragmatic, user-focused questioning with Theo's granular, paper-in-hand analysis. Visualized with retro pixel-art avatars and clean animated SVG charts, the dialogue feels structured, lucid, and measured rather than hype-driven. The emotional trajectory shifts from initial astonishment at local AI's rapid capability gains into a hard-nosed engineering audit that critically unpacks telemetry discrepancies, memory constraints, and the systemic realities of power consumption.

## Bulletin - [00:01](https://www.youtube.com/watch?v=1fgiout-14U&t=1s) Stanford and Together AI researchers benchmarked over 20 small local models across 1M real single-turn queries, finding that an ensemble of local models under 20B active parameters can satisfactorily answer 88.7% of workloads. - [05:45](https://www.youtube.com/watch?v=1fgiout-14U&t=345s) Intelligence per watt (accuracy divided by accelerator power) improved by 5.3× over two years, driven primarily by algorithmic model advances (3.1×) rather than silicon hardware efficiency gains (1.7×). - [09:01](https://www.youtube.com/watch?v=1fgiout-14U&t=541s) Data-center accelerators running identical models remain 1.4× more power-efficient per watt and up to 7.4× more efficient per joule than high-end laptop chips like Apple's M4 Max due to massive memory bandwidth disparities. - [10:44](https://www.youtube.com/watch?v=1fgiout-14U&t=644s) Deploying an intelligent router that sends ~80.7% of common queries to local edge models and reserves cloud frontier models for hard queries reduces total inference energy by up to 80.4%, compute by 77%, and cost by 74%.

## Key takeaways - [00:01](https://www.youtube.com/watch?v=1fgiout-14U&t=1s) The migration of AI workloads toward the edge mirrors the historic shift from mainframes to PCs: edge computing thrives when local devices become 'good enough' within personal power budgets rather than beating supercomputers head-to-head. - [02:54](https://www.youtube.com/watch?v=1fgiout-14U&t=174s) The paper formalizes 'intelligence per watt' (IPW) as expected accuracy divided by mean accelerator wattage, complemented by 'intelligence per joule' (IPJ) to account for inference latency. - [02:54](https://www.youtube.com/watch?v=1fgiout-14U&t=174s) Evaluation spans one million real-world queries across WildChat, NaturalReasoning, MMLU-Pro, and SuperGPQA, utilizing exact-match ground truths alongside reference LLM judges. - [05:45](https://www.youtube.com/watch?v=1fgiout-14U&t=345s) A standalone local model handles up to 71.4% of queries (GPT-OS 120B), but ensembling diverse small local models pushes coverage to 88.7% by covering each other's domain blind spots. - [05:45](https://www.youtube.com/watch?v=1fgiout-14U&t=345s) Local models excel at everyday conversational chat (88.9% accuracy) but degrade significantly on complex reasoning (64.9%), with performance on the hardest reasoning tier barely rising from 3% to 5% over two years. - [05:45](https://www.youtube.com/watch?v=1fgiout-14U&t=345s) Consumer accelerator memory capacity expanded 126× between 2012 and 2025, enabling unified-memory workstation chips to host 120B-parameter models locally. - [09:01](https://www.youtube.com/watch?v=1fgiout-14U&t=541s) The Nvidia B200 and SambaNova SN40L data-center chips substantially outpace the Apple M4 Max in energy efficiency per joule due to high-bandwidth memory (4.8–8.0 TB/s vs. 546 GB/s), with server batching widening the gap further. - [10:44](https://www.youtube.com/watch?v=1fgiout-14U&t=644s) Overall energy savings in hybrid architectures arise from model right-sizing—avoiding running 200B+ frontier models for simple prompts—rather than superior local silicon efficiency. - [10:44](https://www.youtube.com/watch?v=1fgiout-14U&t=644s) Even with an imperfect router operating at 80% accuracy, routing workloads between local devices and cloud models captures 64% energy savings and a 59% cost reduction. - [12:20](https://www.youtube.com/watch?v=1fgiout-14U&t=740s) Study limitations include focusing primarily on single-turn completions, minor evaluator discrepancies, and software telemetry errors estimated between 10% and 15%. - [13:08](https://www.youtube.com/watch?v=1fgiout-14U&t=788s) The paper's implied data suggests a ~353 W accelerator draw for the M4 Max, which conflicts with Apple's official 145 W wall-power rating for the entire Mac Studio chassis, suggesting local chip efficiency may be underestimated. - [14:02](https://www.youtube.com/watch?v=1fgiout-14U&t=842s) Jevons paradox looms over local AI adoption: dramatic efficiency gains could encourage always-on background agents, potentially increasing aggregate global energy consumption. - [14:50](https://www.youtube.com/watch?v=1fgiout-14U&t=890s) The long-term architecture of AI deployment will not be a local-only replacement of the cloud, but a local-first system managed by an intelligent router that delegates the complex reasoning tail to frontier data centers.

## Detailed walkthrough ### [00:01](https://www.youtube.com/watch?v=1fgiout-14U&t=1s) The 88.7% Horizon: Local AI and the Mainframe Parallel

The episode opens by tackling a central dilemma in modern computing: can a localized language model running on your personal hardware truly handle day-to-day intelligence workloads, both today and as models evolve? Host Mara introduces the focal research—the Stanford Hazy Research Group and Together AI paper titled *"Intelligence per Watt: Measuring Intelligence Efficiency of Local AI"*, co-authored by fifteen researchers including John Hennessy and Chris Ré. Co-host Theo encapsulates the paper's headline finding: across a benchmark of one million real-world, single-turn chat and reasoning prompts, an ensemble of small local models can satisfactorily answer 88.7% of queries. This represents an enormous leap from just two years prior, when the best local model could handle a meager 23% of the workload, translating to a 5.3× overall improvement in "intelligence per watt."

Yet, this trajectory carries a counter-intuitive tension. Despite massive gains, the paper's empirical measurements reveal that a modern high-end laptop chip remains at least 1.4 times less power-efficient than a hyperscale data-center accelerator running the exact same model, and up to seven times worse when evaluated on total energy per joule. The path forward is therefore not an outright replacement of the cloud, but rather "a yes with a shape": local hardware handling the bulk of common queries, frontier cloud models absorbing the long tail of heavy reasoning, and an intelligent router arbitrating between them.

The economic and physical impetus for offloading inference from centralized clusters comes down to physical scale. With global AI demand accelerating from billions toward trillions of queries per day, data center providers are hitting severe power and construction bottlenecks. The authors point to historical precedent: from 1946 to 2009, computing performance per watt doubled roughly every eighteen months, ultimately transitioning primary workloads from massive centralized mainframes to personal computers. As Theo points out, PCs never needed to outperform mainframes in raw throughput; they succeeded simply because they became "good enough inside a desk's power budget." Language models are now approaching an identical inflection point driven by three converging trends: small models under 20 billion active parameters matching earlier state-of-the-art checkpoints, edge chips like Apple's M4 Max providing unified memory pools large enough to serve them at conversational speeds, and a growing ecosystem of local-first agent frameworks.

- **Two-Year Jump:** Coverage of real queries jumped from 23% in 2023 to 71% on a single model in 2025, and reached 88.7% when routing dynamically across local models. - **The Mainframe Rule:** Edge computing proliferates when localized devices reach acceptable performance under personal thermal and electrical budgets, not when they defeat industrial supercomputers head-to-head. - **Hardware Realities:** Running identical models on an Apple M4 Max still expends substantially more energy per query than doing so on dedicated data-center silicon like Nvidia's B200.

---

[02:54](https://www.youtube.com/watch?v=1fgiout-14U&t=174s) Defining Intelligence Per Watt and the 1M-Query Benchmark

To turn loose speculative marketing into rigorous engineering science, the researchers formalize "intelligence per watt" (IPW) as a precise mathematical ratio: expected accuracy over an evaluation set divided by the mean electrical power draw of the accelerator during inference. Accuracy is calculated strictly on a binary basis per query—one if the output is deemed correct, zero if it fails. For benchmarks with verified ground truths (MMLU-Pro, SuperGPQA, and NaturalReasoning), scoring requires exact matches. For open-ended generative chat prompts drawn from WildChat, outputs are scored by an automated LLM judge comparing the local candidate against a reference answer produced by Qwen3-235B. Under standard chat evaluation protocol, the local model earns a score of one if it wins or achieves a tie.

Power measurements isolate the accelerator itself, sampling every 50 milliseconds via native vendor telemetry interfaces, which are then integrated over the full inference duration to calculate total energy consumption. The authors expand this into four distinct core metrics: 1. **Intelligence per Watt (IPW):** Steady-state accuracy delivered divided by active power draw in watts. 2. **Intelligence per Joule (IPJ):** Total accuracy normalized against total integrated energy, factoring in latency. 3. **Perplexity per Watt:** Model compression and predictive confidence scaled against steady-state power draw. 4. **Perplexity per Joule:** Language modeling confidence scaled against cumulative energy spent per token.

The evaluation suite was built by stress-testing over 20 open-weight local models against one million real single-turn prompts. The corpus consists of 500,000 real-world ChatGPT queries from WildChat, 500,000 reasoning questions from NaturalReasoning, 12,000 questions from MMLU-Pro, and 26,000 graduate-level questions spanning 285 academic disciplines from SuperGPQA. To understand real-world utility, each prompt was tagged across 22 occupation classes adapted from the Anthropic Economic Index. The tested model pool ranges from small open weights like Qwen3 (from 4B to 32B), GPT-OS 20B and 120B, Gemma 3, and IBM Granite 4, up against frontier references as of October 2025 (GPT-5, Gemini 2.5 Pro, and Claude Sonnet 4.5). Accelerator hardware spanned legacy enterprise cards (Quadro RTX 6000) through Nvidia B200 and the Apple M4 Max. All tests were executed on vLLM locked to a batch size of one, standardizing evaluation to isolate chip and model architecture rather than scheduler queuing efficiencies.

```mermaid flowchart TD accTitle: Intelligence per Watt Evaluation Architecture accDescr: Flowchart illustrating the four dataset sources feeding into model inference on vLLM, followed by accuracy and power telemetry measurement.

subgraph Data ["1 Million Query Corpus"] D1["WildChat (500k Chat)"] D2["NaturalReasoning (500k)"] D3["SuperGPQA (26k Grad-Level)"] D4["MMLU-Pro (12k Exam)"] end

subgraph Eval ["Standardized Execution"] V["vLLM Engine (Batch Size = 1)"] M["20+ Models (Local & Frontier)"] H["Accelerators (RTX 6000 to M4 Max & B200)"] end

subgraph Metrics ["Dual Metric Extraction"] Acc["Accuracy Scoring (Exact Match / Qwen3-235B Judge)"] Pwr["Telemetry Sampling (Vendor Telemetry @ 50ms)"] R1["Intelligence per Watt (Steady State)"] R2["Intelligence per Joule (Integrated Energy)"] end

Data --> V M --> V H --> V V --> Acc V --> Pwr Acc --> R1 Pwr --> R1 Acc --> R2 Pwr --> R2 ```

---

[05:45](https://www.youtube.com/watch?v=1fgiout-14U&t=345s) Local Model Capabilities, Reasoning Limits, and the 5.3× Trajectory

Examining the headline numbers reveals clear delineations in where small local models excel and where they hit hard walls. While an ensemble of local models covers 88.7% of all incoming requests, individual models exhibit steep falloffs. The strongest standalone model tested, GPT-OS 120B, achieves 71.4% total coverage. Dropping down in parameter scale shrinks capability further: Qwen3-14B answers roughly 60% of queries acceptably, while Qwen3-4B drops to about 50%. The 17-point performance advantage enjoyed by an ensemble stems from specialized pre-training and post-training mixtures across model families, allowing diverse small architectures to compensate for each other's blind spots. On three of the four core benchmarks, an ensemble of local models actually surpassed the cloud frontier tier. However, Theo warns that this is purely a "best-of-N selection effect"—the local ensemble pools answers from over twenty models, whereas the frontier reference pool had only three candidates.

The Achilles' heel of local inference remains high-level, complex reasoning. While the best standalone local model successfully handles 88.9% of everyday conversational chat, its accuracy drops to 64.9% when faced with complex reasoning datasets. When broken down by occupational domain within NaturalReasoning: - **Education:** ~80% coverage - **Computer & Mathematical:** 67% coverage - **Life & Physical Sciences:** 60% coverage - **Architecture & Engineering:** 41% coverage

At the absolute peak of problem difficulty, local progress slows to a crawl. The paper's text notes that for the top difficulty tier of reasoning, local model accuracy merely nudged upward from about 3% in 2023 to roughly 5% in 2025, even as performance on easy and medium tiers completely saturated.

Tracing the historical trend over two years highlights a compounded 5.3× increase in intelligence per watt. In 2023, the frontier pairing of Mixtral 8x7B running on an Nvidia Quadro RTX 6000 scored approximately $8 \times 10^{-4}$ accuracy per watt with 23% coverage. By 2024, Llama 3.1 8B on an RTX 6000 Ada achieved $1.8 \times 10^{-3}$ at 49% coverage. By 2025, GPT-OS 120B on the Apple M4 Max reached $4.2 \times 10^{-3}$ with 71% coverage—registering back-to-back annual multipliers of 2.27× and 2.32×. Decomposing these gains shows that model architecture improvements drove steady-state power efficiency (a 3.1× gain attributed to models versus 1.7× to chips). On an energy-per-joule basis, hardware took the lead, yielding an 18× total improvement with silicon advancements contributing 5.9× via sharp latency reductions. The fundamental physical catalyst enabling both curves was local memory capacity: consumer and workstation accelerator memory ballooned by 126× between 2012 and 2025, culminating in unified 128 GB memory buses capable of hosting a 120B mixture-of-experts model on an everyday office desk.

---

[09:01](https://www.youtube.com/watch?v=1fgiout-14U&t=541s) The Uncomfortable Truth: Laptop vs. Data-Center Accelerators

Despite the compounding efficiency gains of local models, comparing laptop silicon directly against dedicated data-center hardware on the identical model reveals a harsh performance disparity. When benchmarking the Qwen3 suite (across 4B, 8B, 14B, and 32B checkpoints), the Nvidia B200 data-center GPU delivers 1.4 times greater intelligence per watt than the Apple M4 Max. Specialized cloud accelerators widen the margin further: SambaNova's SN40L proves 1.78 times more efficient per watt on the 32B model.

When shifting the metric from steady-state watts to total energy per query (intelligence per joule), the laptop class fares significantly worse. The Nvidia B200 is 1.6 to 2.3 times more efficient per joule than the M4 Max, while the SambaNova SN40L is 6.5 to 7.4 times more efficient. Because data-center silicon finishes execution far quicker, total energy expended per response drops dramatically. The core performance bottleneck is not raw compute or FLOPs, but memory bandwidth: - **Nvidia B200 (HBM3e):** 4.8 to 8.0 TB/s - **Apple M4 Max (Unified Memory):** 546 GB/s

Crucially, these numbers reflect batch size one, a constraint that heavily flatters personal devices. While a single-user laptop is structurally trapped at batch size one during interactive use, cloud accelerators process large concurrently batched workloads. At a standard production batch size of 64, the B200's energy efficiency per joule surges by a factor of 11× to 20×, massively widening the operational efficiency gap. Interestingly, the paper identified a curious edge case in consumer hardware: the Apple iPhone 16 Pro, drawing roughly 12 watts, achieved 7 times higher intelligence per watt than a workstation GPU running the same 4B parameter model. However, because its mobile memory bus runs between 9 and 26 times slower, that apparent energy advantage almost completely evaporates when integrated over total latency into joules.

---

[10:44](https://www.youtube.com/watch?v=1fgiout-14U&t=644s) System-Level Routing and Methodological Caveats

The reality that personal chips lose to server silicon on a per-query basis raises an obvious question: how does shifting workloads to local devices reduce net global energy consumption? The authors demonstrate that energy savings stem not from chip efficiency, but from *model sizing*. High-end cloud inference typically throws frontier multi-hundred-billion parameter models at every mundane prompt. By inserting a lightweight classifier to route queries, small requests stay on efficient edge weights while heavy queries hit the cloud. Simulating a full production day of 80.2 million queries, the authors pitted a local ensemble of four small models on an M4 Max against a cloud server running Qwen3-235B on an Nvidia H200 at batch size 16.

```mermaid flowchart LR accTitle: Query Routing Architecture and Savings accDescr: Pipeline showing incoming queries routed between local M4 Max models and a cloud H200 frontier model.

Q["Incoming Queries (80.2M Daily Prompts)"] --> R{"Query Router (Target: ~80% Accuracy)"}

subgraph Local ["Edge Tier (Apple M4 Max)"] R -->|"80.7% Standard / Easy"| LMod["4 Small Local Models (≤20B Active Params)"] LMod --> Done["Direct Answer"] end

subgraph Cloud ["Frontier Tier (Nvidia H200)"] R -->|"19.3% Hard / Ambiguous"| CMod["Qwen3-235B (Batch Size = 16)"] LMod -.->|"Fallback on Low Confidence"| CMod CMod --> Final["Frontier Answer"] end ```

Under a theoretical "oracle" router that perfectly identifies which queries the local models can resolve, 80.7% of incoming traffic remains on the M4 Max. Compared to routing everything to the cloud flagship, overall energy consumption drops by 80.4%, compute demand drops by 77%, and operating costs plummet by 74%. Even using a realistic router operating at 80% routing accuracy: - **Energy consumption:** Reduced by 64% - **Compute demand:** Reduced by 62% - **Cost:** Reduced by 59%

Even a degraded 60% router captures approximately 48% energy savings. Misrouted queries that fail locally can fall back to the cloud, preserving overall system quality. The authors emphasize an engineering rule of thumb: developers should optimize routing accuracy up to roughly 80% and then stop, choosing instead to broaden the diversity of the local model ensemble. As Theo summarizes, "efficient infrastructure comes from routing, not from local hardware matching the cloud."

The segment closes by detailing several methodological limitations and caveats explicit in the paper: - **Single-Turn Limitation:** The study concentrates strictly on single-turn completions. An auxiliary sanity check run on 245 multi-turn agent tasks (across GAIA and TerminalBench 2) exhibited similar energy trends (cloud was 2.4× to 3× better per joule; local models matched accuracy within two percentage points), but local agent execution proved 12 to 15 times slower. - **Judge Ambiguity:** The paper exhibits internal contradictions regarding its LLM evaluation judge, citing GPT-4o in one appendix while listing Qwen3-235B within the limitations section. - **Scoring Leniency:** Open-ended conversational wins include ties against the reference answer, which inherently paints local capabilities in a generous light.

[13:08](https://www.youtube.com/watch?v=1fgiout-14U&t=788s) Limitations, Telemetry Caveats, and the M4 Max Anomaly

Continuing the audit of what the paper leaves unsettled, the hosts turn to measurement accuracy and physical power draw. The benchmark relies on software telemetry, which introduces an estimated 10% to 15% margin of error. Crucially, the telemetry profiles the accelerator alone, excluding the rest of the host system (such as motherboards, system fans, and power delivery). While this means relative comparisons among chips remain directionally solid, any absolute figures for energy consumption in joules must be treated as approximations.

The hosts then introduce their own striking empirical calculation that diverges from the paper's published numbers. Dividing the reported task success rate by the reported intelligence per watt in the paper’s data table implies an accelerator power draw of roughly 350 to 353 watts for Apple's M4 Max. However, Apple officially rates the entire Mac Studio chassis at a maximum wall draw of only 145 watts under full system load. If the benchmark’s telemetry significantly overstated the M4 Max's wattage, then local intelligence per watt is understated in the paper's findings. Consequently, the cloud's 1.4× efficiency lead (exhibited by Nvidia's B200) would shrink or even invert in favor of the local chip. This discrepancy reinforces the hosts' core warning: viewers should trust the relative rankings and multi-year trajectory rather than fixation on exact wattage numbers.

- **Telemetry limitations**: Software-based measurement carries a 10–15% inherent error margin and ignores host platform power draw outside the accelerator. - **The M4 Max calculation**: The paper's implied ~353 W for Apple's chip starkly contradicts Apple's official 145 W wall-power ceiling for the entire Mac Studio system. - **Impact on the hardware gap**: Correcting an overstated local power draw would narrow or flip the reported 1.4× B200 efficiency advantage over consumer silicon.

---

[14:02](https://www.youtube.com/watch?v=1fgiout-14U&t=842s) Snapshot Timing, Software Stacks, and Jevons Paradox

The hosts catalog three additional caveats highlighted by the research team: the temporal snapshot, software inference engines, and broader macroeconomic or systemic energy consequences. The study represents a frozen snapshot from October 2025; frontier models and small open-weight models have both advanced since publication. To compensate, the researchers open-sourced their evaluation and profiling harness on GitHub so external researchers can rerun benchmarks against newer weights, while simultaneously warning that escalating memory prices pose a severe structural headwind to local hardware adoption.

Regarding runtime infrastructure, the primary paper evaluations were conducted solely using vLLM. Cross-testing against alternative runtimes—such as SGLang and llama.cpp—resulted in variance between 3% and 12%, though accelerator rankings remained completely stable across all three engines. Finally, the researchers themselves raise Jevons paradox: shifting inference workloads out of mega-datacenters onto millions of distributed desktop chips simply redistributes electrical load across residential and municipal power grids rather than eliminating it. If dramatic intelligence-per-watt efficiency gains justify leaving autonomous local agents continuously active in the background 24/7, aggregate energy consumption could actually surge worldwide—a dynamic the paper acknowledges but leaves unquantified.

- **Snapshot timing**: Benchmark frozen in October 2025; open-source test harness allows continuous re-evaluation against newer releases. - **Hardware economics**: Rising high-bandwidth and unified memory costs flagged as an external threat to local device viability. - **Software engine parity**: Shifting between vLLM, SGLang, and llama.cpp yields a modest 3–12% shift without altering hardware rankings. - **Jevons paradox**: Efficiency gains encouraging always-on autonomous agents risk driving up aggregate global power consumption.

---

[14:50](https://www.youtube.com/watch?v=1fgiout-14U&t=890s) The Final Verdict: Local First, Router as the Future

When asked for a definitive one-sentence verdict on whether local LLMs will truly suffice in the future, the hosts deliver complementary conclusions. Mara votes "yes" for standard human usage: "For most of what people actually ask, the machine on the desk is already good enough, and the trend says it gets more so every year." Theo offers a nuanced architectural verdict: "Yes as the default first stop, no as a replacement." In his view, centralized frontier models will permanently command the hard reasoning tail, meaning the true winner of the coming decade is not an individual chip or model tier, but the intelligent router orchestrating the boundary between them.

```mermaid flowchart TD accTitle: Intelligent Query Routing Architecture accDescr: Incoming queries are classified, routing the vast majority to local consumer accelerators and offloading complex reasoning to cloud superclusters.

subgraph Client["Local User Machine"] Q["User Query"] --> R{"Dynamic Router"} R -->|"~80-89% Common / Moderate Queries"| L["Local Model Ensemble\n(≤20B Active Params)"] L --> S1["Fast, Private, Low-Cost Result"] end

subgraph Datacenter["Centralized Cloud Infrastructure"] R -->|"~11-20% Hard Reasoning Tail"| F["Frontier Cloud Model\n(e.g., Qwen3-235B / GPT-5 class)"] F --> S2["High-Compute Frontier Result"] end ```

The hosts weigh the comprehensive evidence accumulated across the paper:

| Dimension | Evidence For Local Viability | Evidence Against Complete Local Replacement | |:--- |:--- |:--- | | **Accuracy & Coverage** | Single-model coverage surged from 23% to 49% to 71% in three years; reaches 88.7% with a multi-model local ensemble. | The hardest tier of deep reasoning benchmarks (e.g., SuperGPQA) exhibited the least improvement. | | **System Efficiency** | Intelligence per watt improved 5.3× in two years; selective local routing yields 60–80% compute, energy, and cost savings. | Dedicated cloud chips remain up to 7.4× more energy-efficient per joule (e.g., SambaNova SN40L) on identical models; batching widens this gap. | | **Hardware Evolution** | Consumer device memory capacity expanded 126× across the past decade. | Memory bandwidth—not raw compute FLOPS—remains the critical edge bottleneck where consumer chips lag furthest behind datacenters. | | **Evaluation Scope** | Proven performance across 1,000,000 real-world queries from WildChat, MMLU-Pro, and NaturalReasoning. | All tests were strictly single-turn interactions, leaving multi-turn memory degradation and complex state unmeasured. |

Ultimately, two divergent performance curves will dictate the balance of power: whether local open-weight models can climb the hardest rungs of the reasoning ladder, and whether consumer silicon architectures can bridge the memory bandwidth chasm separating unified memory from datacenter HBM. The core question was never whether an ultraportable laptop could outperform a multi-megawatt server farm; it was whether desktop hardware delivers sufficient intelligence per watt for the specific prompt before it. Today, local hardware satisfies the vast majority of real-world queries—and that share continues to climb every single year.

## Notable quotes > [00:01](https://www.youtube.com/watch?v=1fgiout-14U&t=1s) Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

> [00:01](https://www.youtube.com/watch?v=1fgiout-14U&t=1s) a yes with a shape

> [00:01](https://www.youtube.com/watch?v=1fgiout-14U&t=1s) good enough inside a desk's power budget. — Theo

> [05:45](https://www.youtube.com/watch?v=1fgiout-14U&t=345s) best-of-N selection effect — Theo

> [10:44](https://www.youtube.com/watch?v=1fgiout-14U&t=644s) efficient infrastructure comes from routing, not from local hardware matching the cloud. — Theo

> [14:50](https://www.youtube.com/watch?v=1fgiout-14U&t=890s) For most of what people actually ask, the machine on the desk is already good enough, and the trend says it gets more so every year. — Mara

> [14:50](https://www.youtube.com/watch?v=1fgiout-14U&t=890s) Yes as the default first stop, no as a replacement. — Theo

> a router, not a winner — local first, frontier for the tail

## Mentioned Anthropic Economic Index, Apple, Apple M4 Max, Apple Mac Studio, Chris Ré, Claude Sonnet 4.5, GAIA, Gemini 2.5 Pro, Gemma 3, GPT-4o, GPT-5, GPT-OS 120B, GPT-OS 20B, IBM Granite 4, iPhone 16 Pro, John Hennessy, Leanim, Llama 3.1 8B, llama.cpp, Mara, MiniMax, Mixtral 8x7B, MMLU-Pro, NaturalReasoning, Nvidia B200, Nvidia H200, Nvidia Quadro RTX 6000, Nvidia RTX 6000 Ada, Qwen3-14B, Qwen3-235B, Qwen3-32B, Qwen3-4B, SambaNova SN40L, SGLang, Stanford Hazy Research Group, SuperGPQA, TerminalBench 2, Theo, Together AI, vLLM, WildChat

## Video description Can a local LLM on your own machine actually do the job, now and in the future? Loop State EP. 04 works through the Stanford / Together AI paper "Intelligence per Watt: Measuring Intelligence Efficiency of Local AI" (Saad-Falcon, Narayan, … Hennessy, Mirhoseini, Ré; arXiv 2511.07885) as a two-host Q&A: Mara asks, Theo brings the paper.

The paper measures 20+ local models (≤20B active parameters) on 9 accelerators across 1M real single-turn chat and reasoning queries, and proposes one metric for the question: intelligence per watt, mean accuracy divided by mean accelerator power.

What we cover • Why the cloud alone can't scale, and the mainframe → PC precedent • What intelligence per watt (and per joule) actually measures, and how a query is scored • The 1M-query benchmark: WildChat, NaturalReasoning, SuperGPQA, MMLU-Pro • 88.7% of queries answerable by some local model (71.4% for the best single model) — and where local fails (reasoning, hard tiers) • IPW ×5.3 in two years (×2.27, then ×2.32): models ×3.1, hardware ×1.7; coverage 23% → 49% → 71% • Laptop vs data-centre chip on the same model: B200 ×1.4 per watt, ×1.6–2.3 per joule; SN40L up to ×7.4; bandwidth is the bottleneck, batching widens the gap • Routing: 80.7% of queries to four small local models, the rest to Qwen3-235B — 60–80% less energy, compute and cost • What the paper does not settle: single-turn only, judge inconsistency, telemetry error, October-2025 snapshot, Jevons • Our own inference: the paper's table implies ~353 W for the M4 Max vs Apple's 145 W wall maximum • The verdict: a router, not a winner — local first, frontier for the tail

Chapters 00:00 Can your own machine run your AI in the future? 01:35 Why can't the cloud just keep scaling? 02:54 What does "intelligence per watt" actually measure? 04:21 How do you test a million real queries? 05:45 What can local models answer, and where do they fail? 07:25 How fast is intelligence per watt improving, and why? 09:01 Is a laptop chip as efficient as a data-centre chip? 10:44 Does routing between laptop and cloud actually save anything? 12:20 What does the paper not settle? 14:50 So, can the local LLM work in the future? 16:08 Sign-off

Paper: https://arxiv.org/abs/2511.07885 Code / profiling harness: https://github.com/HazyResearch/intelligence-per-watt Project page: https://hazyresearch.stanford.edu/intelligence-per-watt/ Apple Mac Studio power figures: https://support.apple.com/102027

All charts are redrawn from the paper's tables and figures. Numbers quoted are from the paper's v6 text (6 Sep 2026); where the paper's text and figures disagree, we say so on screen. The M4 Max power comparison is our inference, not the paper's claim.

Made with Leanim (Lean 4). Voices: MiniMax speech-2.8-hd. Pixel-art hosts, SVG charts and board animations authored for this episode.

#LocalAI #LLM #IntelligencePerWatt #EdgeAI #MachineLearning #Stanford #OpenSourceAI

[Source](https://www.youtube.com/watch?v=1fgiout-14U)

中文简报

斯坦福发现本地AI可处理88%查询并减少80%功耗将大部分日常工作负载路由至端侧小型模型可将算力需求降低多达80%，尽管在执行相同任务时，集中式数据中心芯片的能效仍高得多。本场讨论分析了实证研究结果，显示在两年内“每瓦智能”提升5.3倍的推动下，由小型本地模型组成的集成能够回答88.7%的日常闲聊与推理查询。然而，由于严重的内存带宽瓶颈，像Nvidia B200和SambaNova SN40L这样的专用数据中心加速器在每焦耳能效上仍以1.4倍至7.4倍的优势领先于Apple M4 Max等笔记本芯片。真正的能效提升并非源于笔记本芯片击败服务器芯片，而是来自智能路由机制：在本地处理约80%的常规查询，同时将复杂的推理任务卸载到云端，可将净能耗、计算量和成本削减60%至80%。主持人还指出了方法论上的局限，包括单轮评估的局限性，以及论文数据中一个明显的异常现象——尽管Apple官方给出的墙上电源额定功率为145 W，但该论文的数据推导出的M4 Max功耗约为353 W。斯坦福大学与Together AI的研究人员基于100万个真实的单轮查询对20多个小型本地模型进行了基准测试，发现活跃参数低于20B的本地模型集成可以令人满意地回答88.7%的工作负载。 (0:01) “每瓦智能”（准确率除以加速器功耗）在两年内提升了5.3倍，这主要得益于算法模型的进步（3.1倍），而非芯片硬件能效的提升（1.7倍）。 (5:45) 由于巨大的内存带宽差距，在运行相同模型时，数据中心加速器的每瓦能效仍比Apple M4 Max等高端笔记本芯片高出1.4倍，每焦耳能效高出多达7.4倍。 (9:01) 部署智能路由器将约80.7%的常见查询发送至本地边缘模型，并将云端前沿模型留给困难查询，可将总推理能耗降低最多80.4%，计算量减少77%，成本降低74%。 (10:44)

Reading summary

Research by Stanford's Hazy Research Group and Together AI demonstrates that small local model ensembles can resolve 88.7% of daily single-turn queries, yet true system efficiency relies on intelligent cloud-edge routing rather than local chips matching data-center hardware. Local Accuracy Gains and the Reasoning Ceiling (5:45) Between 2023 and 2025, "intelligence per watt"—defined as task accuracy divided by mean electrical power draw—improved 5.3× across a benchmark of one million real-world queries. Driven by architecture improvements and consumer unified memory reaching 128 GB, an ensemble of models under 20 billion active parameters covered 88.7% of evaluated queries, up from 23% for the top local model in 2023. However, single-model capability diminishes rapidly by scale: standalone GPT-OS 120B answered 71.4%, Qwen3-14B resolved roughly 60%, and Qwen3-4B reached approximately 50%. Localized models encounter severe limits when moving from open-ended chat to complex problem-solving. While the strongest standalone local model answered 88.9% of conversational chat prompts, its accuracy dropped to 64.9% on reasoning datasets. In occupational breakdowns, coverage ranged from 80% in education down to 41% in architecture and engineering. At the highest difficulty tier of reasoning, local accuracy only nudged from roughly 3% in 2023 to 5% in 2025, demonstrating that frontier cloud models remain necessary for the hardest intellectual tasks. Hardware Disparity and Memory Bandwidth Limits (9:01) Despite model-level efficiency strides, high-end consumer hardware remains less power-efficient than dedicated data-center accelerators executing the identical model. On the Qwen3 family, Nvidia's B200 GPU proved 1.4 times more efficient per steady-state watt than Apple's M4 Max, while SambaNova's SN40L was 1.78 times more efficient. Evaluated by total energy consumed per query (intelligence per joule), the B200 was 1.6 to 2.3 times more efficient, and the SN40L was 6.5 to 7.4 times more efficient. The researchers isolate memory bandwidth—not raw compute FLOPS—as the core operational bottleneck. The B200 achieves 4.8 to 8.0 TB/s using HBM3e, whereas the M4 Max reaches 546 GB/s. Furthermore, while client hardware is trapped at batch size one during interactive use, cloud servers process concurrent batches; at batch size 64, the B200's energy efficiency per joule surges by an additional 11× to 20×. In mobile testing, the Apple iPhone 16 Pro demonstrated 7 times higher intelligence per watt on a 4B model due to its 12-watt draw, but this advantage vanished once integrated over latency due to a memory bus 9 to 26 times slower than workstation cards. System-Level Routing to Slash Global Energy (10:44) Because individual client chips lag behind data-center silicon in raw physical efficiency, system-wide energy reductions require dynamic model sizing rather than moving all compute locally. In a simulation of 80.2 million daily requests, the authors tested routing queries between an M4 Max running four local models and an Nvidia H200 running Qwen3-235B at batch size 16. A theoretical oracle router retained 80.7% of incoming traffic locally, cutting overall energy consumption by 80.4%, compute demand by 77%, and operating expenses by 74%. Under a practical classifier with 80% routing accuracy, the system achieved a 64% energy reduction, a 62% compute reduction, and a 59% cost reduction, with failed local attempts falling back to the cloud. The authors suggest that engineering efforts should cap router optimization around 80% accuracy and redirect focus toward diversifying the local model ensemble. Telemetry Distortions, Benchmark Limits, and Systemic Risks (13:08) Several methodological caveats qualify the paper's headline findings. Telemetry sampled chip power every 50 milliseconds with a 10% to 15% error margin, excluding motherboard and cooling overhead. Crucially, the podcast hosts note an internal numerical contradiction: dividing the paper's reported success rate by its intelligence per watt yields an implied M4 Max accelerator draw of roughly 350 to 353 watts, conflicting with Apple's official 145-watt full-system wall limit for the Mac Studio. Correcting for an overstated local wattage would narrow or eliminate the cloud's 1.4× efficiency lead. Evaluation parameters also introduce bias: open-ended chat scoring credited ties as wins, and the study was confined to single-turn queries. Auxiliary tests on multi-turn agent benchmarks showed local models matched cloud accuracy within two percentage points but ran 12 to 15 times slower. Finally, the authors invoke Jevons paradox, noting that if local efficiency gains prompt users to keep autonomous desktop agents running continuously, aggregate electricity demand across local municipal grids could increase rather than diminish. The study indicates that edge AI does not need to outperform hyperscale clusters to succeed, paralleling the historical mainframe-to-PC transition where localized computing triumphed by becoming sufficient within personal thermal envelopes. Because consumer memory bandwidth and complex reasoning remain structural barriers, the authors and hosts conclude that the primary architecture of the coming decade will be the dynamic router managing the boundary between local edge models and centralized frontier clusters.

Source quotes

[0:01] Intelligence per Watt: Measuring Intelligence Efficiency of Local AI [0:01] a yes with a shape [0:01] Theo: good enough inside a desk's power budget. [5:45] Theo: best-of-N selection effect [10:44] Theo: efficient infrastructure comes from routing, not from local hardware matching the cloud. [14:50] Mara: For most of what people actually ask, the machine on the desk is already good enough, and the trend says it gets more so every year. [14:50] Theo: Yes as the default first stop, no as a replacement. a router, not a winner — local first, frontier for the tail

Original transcript

[0:01] Welcome to loop state. I'm Mara. Today's [0:04] paper is intelligence per watt from [0:07] Stanford's Hazy Research Group and [0:10] Together AI. 15 authors including John [0:13] Hennessy and Chris Ray. Theo, this [0:15] episode has one question. Can a local [0:18] model on your own machine actually do [0:20] the job now and in the future? [0:22] >> The papers answer in one line. Today, [0:25] some small local model answers 88.7% [0:29] of a million real single turn chat and [0:31] reasoning queries acceptably. 2 years [0:34] ago, the best local model covered 23% [0:38] and intelligence per watt improved 5.3 [0:41] times in those two years. [0:43] >> So, the grid behind me is 100 queries in [0:48] 2023. 23 of them could stay on the [0:51] laptop by 2025. 71 pick the best local [0:56] model for each query and 89 stay home. [1:00] >> That's the trajectory. But the same [1:02] paper measures that a laptop chip is [1:04] still at least 1.4 times less efficient [1:07] than a data center chip running the [1:09] identical model and up to seven times [1:11] worse per jewel. [1:13] >> So the answer is not a clean yes. [1:16] >> It's a yes with a shape. local for most [1:19] queries, cloud for the hardail, and a [1:21] router in between. We'll build that [1:23] answer piece by piece and stay strict [1:25] about what the paper measures versus [1:27] what it argues. [1:29] >> Then start with why anyone wants to [1:33] leave the cloud at all. Theo, the cloud [1:36] works. Um, frontier models answer almost [1:39] everything. Why does this paper want to [1:41] move queries off it? [1:43] >> Because demand grows faster than the [1:45] buildings. The introduction says [1:47] inference is heading from billions [1:49] toward trillions of queries a day and [1:52] providers struggle to add capacity at [1:54] that pace. Then it reaches for a [1:56] precedent. [1:57] >> Mainframes [1:59] and personal computers. [2:02] >> From 1946 to 2009, computing performance [2:06] per watt doubled roughly every year and [2:09] a half. Work moved from mainframes to [2:11] PCs. And the paper stresses the timing. [2:14] PCs never beat mainframes on raw [2:16] performance. They became good enough [2:18] inside a desk's power budget. [2:21] >> And the claim is that language models [2:24] are at the same kind of inflection. [2:27] >> Three trends in their framing. Small [2:29] models with at most 20 billion active [2:32] parameters that are competitive on many [2:34] tasks. Local chips like the Apple M4 Max [2:37] with enough unified memory to hold them [2:39] at interactive speed and a wave of local [2:42] first agent stacks. Um, so the laptop [2:46] doesn't have to win. It has to be good [2:49] enough per watt, which is why they need [2:51] a metric with both words in it. Define [2:54] it for me. Intelligence per watt sounds [2:58] like a slogan. [2:59] >> It's a ratio. Expected accuracy over the [3:02] query set divided by expected power draw [3:04] during inference in watts. Accuracy is [3:07] binary per query. One if the model got [3:10] it right, zero otherwise. [3:13] >> Right? According to whom [3:14] >> for benchmarks with ground truth, MMLU [3:18] Pro, Super GPQA, and Natural Reasoning [3:21] Exact Match for the Open Chat prompts [3:23] from Wild Chat, a language model judge [3:26] compares the local answer with a [3:27] reference answer written by Quen 3235B, [3:30] and the local model scores one if it [3:33] wins or ties. [3:34] >> A tie counts as a success. That's [3:37] generous, [3:38] >> is standard practice in chat evaluation, [3:40] and the paper says so. Hold that thought [3:43] for the boundary chapter. Power is [3:45] measured on the accelerator only, [3:47] sampled every 50 milliseconds through [3:49] the vendor's own telemetry and [3:51] integrated over the query for energy. [3:54] >> And you said there are four metrics, not [3:57] one. [3:58] >> Per watt and per jewel for accuracy and [4:01] for perplexity. Per watt is steady [4:04] state, how much correctness a chip [4:06] delivers at a given power draw. Per [4:08] jewel multiplies in the latency. So, [4:11] it's the total energy one answer costs. [4:13] The two tell different stories and we'll [4:16] see both. [4:17] >> Fine. Now, how do you push a million [4:19] queries through that? Where do a million [4:23] real queries come from? [4:26] >> Four sources. 500,000 real chat GPT [4:30] prompts from Wild Chat, 500,000 [4:33] reasoning problems from Natural [4:35] Reasoning, 12,000 from MMLU Pro, and 26 [4:39] 12,000 graduate level questions from [4:41] Super GPQA across 285 disciplines. [4:45] >> All single turn. [4:47] >> All single turn on purpose. Then a small [4:50] model tags each query with one of 22 [4:53] occupation categories from the anthropic [4:56] economic index. So coverage can be read [4:58] per domain. And the models and chips [5:02] >> over 20 local models Quinn 3 from 4 to [5:05] 32 billion. GPT OS 20B and 120B Gemma 3 [5:11] IBM Granite 4 Frontier references as of [5:14] October 2025 Claude Sonnet 4.5 Gemini [5:19] 2.5 Pro and GPT5 [5:22] accelerators from a 2019 Quadro RTX 6000 [5:25] up to a B200 with the Apple M4 Max as [5:29] the laptop class chip. [5:31] >> Same software everywhere. [5:33] >> VLM everywhere batch size one. So you [5:37] measure the model and the chip rather [5:39] than theuler. Remember batch size one. [5:42] It flatters the laptop and it matters [5:44] later. [5:45] >> Give me the headline number properly. [5:49] >> 88.7%. [5:51] That's coverage when each query goes to [5:53] the best local model for it. A single [5:55] model does worse. GPD O 12B alone covers [5:59] 71.4%. [6:01] Quinn 314B 60. Quinn 34B about 50. [6:07] >> So the ensemble is worth 17 points over [6:11] the best single model [6:13] >> because the models differ. Different [6:14] pre-training and post- training means [6:16] different strengths and routing exploits [6:18] that on three of four benchmarks. Best [6:21] of local even beats best of cloud. [6:23] >> Wait, local beats frontier. [6:26] >> Best of nor effect and the paper says so [6:29] itself. Best of local picks from over 20 [6:32] models. Best of cloud picks from three. [6:35] It's a selection effect, not a claim [6:37] that any small model beats GPT5. [6:40] >> Where does local fail? [6:44] >> Reasoning. The best single local model [6:46] covers 88.9% of chat but 64.9 of [6:50] reasoning. By domain on the reasoning [6:52] set, education is about 80%, computer [6:56] and mathematical 67, life and physical [6:59] science 60, architecture and engineering [7:02] 41. [7:03] >> And the hardest problems [7:07] >> slowest to move. The paper's text puts [7:09] the top difficulty tier of reasoning at [7:11] about 3% in 2023 and about five in 2025, [7:16] while the easy and middle tiers [7:17] saturated. Its figure reads higher. So [7:20] take the shape, not the digits. The [7:22] hardest tier is where local progress [7:24] lags. [7:25] >> Now the trend 5.3 times in two years. [7:30] Show me the three points. [7:32] >> Each year the best local model on the [7:35] best local chip 2023 Mixtrol 8x7B on a [7:39] Quadro RTX 6000 about 8 times 10 the [7:43] minus4 accuracy per watt 23% coverage. [7:47] 2024 Llama 3.18B on an RTX 6000 ADA 1.8 [7:53] * 10us 3 49%. [7:56] >> And 2025 [8:00] >> GPTO OS120B on an Apple M4 Max 4.2 * [8:04] 10us 3 71% coverage yearover-year 2.3 [8:10] times then 2.3 times again compounded [8:14] 5.3. [8:15] How much of that is models and how much [8:19] is chips? [8:20] >> Per watt models 3.1 times and hardware [8:24] 1.7. Per jewel the total is 18 times [8:28] with hardware at 5.9 because newer chips [8:31] also cut latency. Models drive the per [8:33] watt story. Hardware drives the perjoule [8:36] story. and the enabler underneath both [8:40] >> memory. Local accelerator memory grew [8:43] about 126 times from 2012 to 2025. [8:47] 128 GB of unified memory is what lets a [8:51] 120B mixture of experts with 20 billion [8:54] active parameters live on a desk. [8:56] >> Two curves compounding. Now the [8:59] uncomfortable chapter, [9:01] same model, laptop versus data center. [9:05] Who wins per watt? [9:07] >> The data center every time. Running Quen [9:10] 3 at 4, 8, 14, and 32B. The Nvidia B200 [9:15] gets 1.4 times the intelligence per watt [9:18] of the M4 Max. Samba Nova's SN40 L is [9:22] 1.78 times better on the 32B model. [9:26] >> And per jewel, [9:29] >> worse for the laptop, the B200 is 1.6 6 [9:32] to 2.3 times better per jewel. The SN40 [9:36] L 6 1/2 to 7.4 times. The gap widens [9:39] because the cloud chip also finishes [9:41] faster. [9:42] >> Why? [9:43] >> Memory bandwidth. HBM 3E moves 4.8 to 8 [9:48] tab a second. The M4 Max's unified [9:51] memory moves 546 GB plus dedicated [9:55] tensor units. The paper's phrase is that [9:57] the local bottleneck is bandwidth and [9:59] specialized compute, not raw flops. [10:02] >> And that's at batch size one, which you [10:05] told me to remember. [10:07] >> A cloud chip serves many users at once. [10:10] At batch 64, the B200's perjoule [10:13] efficiency rises 11 to 20 times. A [10:16] laptop serving one person cannot batch. [10:19] So the real perquery gap in production [10:21] is larger than 1.4. Does any device win [10:25] per watt? [10:27] >> The phone. Oddly, an iPhone 16 Pro at [10:30] about 12 watts gets 7 times the [10:33] intelligence per watt of a workstation [10:35] GPU on the same 4B model, but it's 9 to [10:39] 26 times slower. So, per jewel, most of [10:42] that advantage evaporates. [10:44] >> If the laptop is less efficient per [10:47] query, how can sending queries to it [10:50] save energy? Because the saving isn't [10:53] the chip, it's the model size. The paper [10:56] simulates a day of 80.2 million queries. [10:59] Local side, four small models on an M4 [11:02] Max. Cloud side, Quinn 3235B on an H200 [11:07] batched 16 at a time. So the baseline is [11:10] a realistic production server. And with [11:14] a perfect router, [11:16] >> the Oracle sends the 80.7% of queries [11:19] that local can answer to the small [11:21] models. Energy drops 80.4% [11:24] compute 77 cost 74 versus sending [11:28] everything to the big model. [11:30] >> Nobody has a perfect router. [11:34] >> An 80% accurate router, which the paper [11:36] calls a realistic target, still cuts [11:39] energy 64%. Compute 62 cost 59. A 60% [11:44] router saves about 48. Misouted queries [11:48] fall back to the cloud. So answer [11:49] quality holds. [11:51] >> So the router doesn't have to be [11:53] brilliant. [11:54] >> The paper's rule push router accuracy to [11:56] about 80% then stop and widen the local [11:59] ensemble instead. Model diversity buys [12:02] more than the last points of routing [12:03] accuracy. [12:04] >> It which inverts the previous chapter [12:08] >> exactly. per query local losses. At the [12:11] system level, routing across both wins. [12:14] Their sentence is that efficient [12:15] infrastructure comes from routing, not [12:18] from local hardware matching the cloud. [12:20] >> Boundaries, give me the list. [12:23] >> One, single turn only by design. They [12:26] add a sanity check on 245 multi-turn [12:30] agent tasks from GIA and Terminal Bench [12:34] 2. Same shape, cloud 2.4 to three times [12:37] better per jewel. local within about two [12:39] points of accuracy but 12 to 15 times [12:42] slower. They call it a sanity check not [12:44] a characterization [12:46] >> to the judge. [12:48] >> The chat reference answer comes from [12:50] Quinn 3235B [12:51] and the paper names the judge [12:53] inconsistently GPT 40 in one appendix [12:57] Quinn 3235B in the limitations. Either [13:01] way, on wild chat success means winning [13:03] or tying against a strong open model, [13:06] not against GPT5. [13:08] >> Three, the meters [13:10] >> software telemetry 10 to 15% error and [13:14] accelerator only. So the rest of the [13:17] machine is not counted. relative [13:19] comparisons hold absolute jewels are [13:22] approximate [13:24] >> and one we infer not the paper um divide [13:29] the success rate by the intelligence per [13:31] watt in their table and the M4 Max is [13:34] drawing about 350 watts. Apple rates the [13:40] whole Max Studio at 145 watts maximum at [13:45] the wall. We check that if local power [13:48] is overstated, local intelligence per [13:50] watt is understated and the 1.4 gap [13:53] shrinks or even flips. That cuts in [13:56] favor of local and it's another reason [13:58] to trust the rankings and the trends [13:59] rather than the absolute watts. [14:02] >> Four, the snapshot [14:04] >> October 2025. Frontier models moved [14:08] since, local models moved since. They [14:10] release the harness so the numbers can [14:12] be rerun and they flag rising memory [14:15] prices as a risk to the trend. [14:18] >> Um five the software stack [14:22] >> VLLM only across VLLM SG Lang and [14:26] Llama.cpp CPP the numbers shift 3 to 12% [14:31] and the rankings hold [14:33] >> and six the one they raise themselves [14:37] >> jevans local inference moves energy onto [14:40] consumer grids it doesn't erase it if [14:43] per watt gains justify always on local [14:45] agents total energy can go up they have [14:48] not quantified that [14:50] >> verdict time one sentence each mine yes [14:55] for most of what people actually ask the [14:57] machine on the desk is already good [14:59] enough and the trend says it gets more [15:02] so every year. [15:03] >> Mine yes as the default first stop no as [15:07] a replacement. The frontier keeps the [15:09] hard tail and the thing that works in [15:10] the future is the router between them. [15:13] >> Evidence for [15:15] >> coverage 23 49 71% in 3 years 89 with an [15:20] ensemble intelligence per watt 5.3 [15:23] times. local memory 126 times in a [15:27] decade and 60 to 80% system level [15:30] savings from routing [15:32] >> evidence against [15:34] >> the hardest reasoning tier moved least [15:37] per jewel a cloud chip is still up to [15:39] seven times more efficient on the same [15:41] model and batching widens that the [15:44] bottleneck is memory bandwidth not flops [15:47] and that is where the paper says local [15:49] chips must catch up and all of it is [15:51] single turn So what would change the [15:55] verdict? [15:56] >> Two curves in the paper's own terms. [15:59] Whether local models keep climbing the [16:01] reasoning difficulty ladder and whether [16:03] local chips close the bandwidth gap. [16:05] Those two decide how big most gets. [16:08] >> Takeaway. The question was never whether [16:11] a laptop beats a data center. It's [16:14] whether it's good enough per watt for [16:17] the query in front of it. Today that's [16:19] true for most queries and the share [16:22] climbs every year. [16:23] >> Open question. The hardest tier of [16:25] reasoning is where local models have [16:27] moved least. If they never reach it, the [16:30] router is permanent. If they do, the [16:32] cloud becomes the exception. Nobody [16:35] knows yet