AI Token Costs Eclipse Payroll as Labs Game Public Tests, Vals Says
Ben Horowitz, Erik Torenberg, Jennifer Li, Rayan Krishnan
Surging enterprise token consumption is beginning to rival employee salaries, requiring continuous, independent evaluations on proprietary workflows to verify model productivity and avoid massive compute waste.
AI-assisted summary of Who Grades the AI Models? | Ben Horowitz & Rayan Krishnan — original source. Check the original for full context.
Source published:
Key ideas
- Open-source public benchmarks are increasingly saturated and vulnerable to test-hacking, creating an illusion of progress that fails under real-world conditions—as seen when Meta's Llama 4 excelled on public tests but underperformed on private held-out suites. (0:00)
- Unmonitored enterprise AI usage can quickly eclipse payroll costs; a Fortune 10 trial saw engineers spend $300 daily per seat on Claude Code tokens, while an internal Vals experiment hit peak consumption of 6 billion tokens in a single day for one engineer, scaling to a theoretical $1.5 million monthly bill. (16:04)
- Modern agentic systems require an inverted evaluation paradigm compared to classical datasets like ImageNet, shifting from millions of simple categorical inputs to fewer, long-horizon tasks spanning hours or days that require durable harnesses capable of retrying from transient failures. (13:17)
- To prevent Arthur Andersen-style conflicts of interest and preserve auditing integrity, third-party evaluators must avoid monetizing training or fine-tuning datasets for the very labs whose models they evaluate. (7:44)
Detailed notes
## Background Published on RSS and web platforms on September 9, 2026, by Ben Horowitz, Erik Torenberg, Jennifer Li, and Rayan Krishnan, this episode features a16z partners sitting down with the founder and CEO of Vals to tackle the challenge of objectively measuring AI progress. According to the show description, the conversation explores why public benchmarks saturate, how Vals tests models under tight pre-release windows, the emergence of multi-day agentic evaluations, enterprise token economics, and how evals could underpin AI policy and global governance.
## TL;DR Vals founder Rayan Krishnan explains why saturated public benchmarks and lab self-reporting distort true model performance, creating a trillion-dollar need for independent, held-out evaluation infrastructure. As AI shifts toward long-horizon autonomous agents, traditional static tests must be replaced by continuous, dynamic benchmarks and bespoke enterprise suites like ValSmith, particularly as token spend surges to rival engineering payrolls. The discussion also establishes a clear public-private division of labor for regulation, proposing that third-party evaluators provide the empirical 'trust, but verify' audit layer required for domestic safety thresholds and international treaty coordination.
## Atmosphere & style The discussion has an intellectually sharp, pragmatic insider tone that bridges frontier systems engineering with macroeconomic and regulatory strategy. Ben Horowitz anchors the dialogue with historical and policy frameworks—comparing model benchmarks to MPAA film ratings and Cold War nuclear verification—while Krishnan brings an urgent, ground-level operator perspective illustrated by vivid anecdotes of token-maxing engineers burning millions in compute. The rapport is fast-paced and collaborative, creating a focused, high-stakes exploration of what it actually takes to measure genuine intelligence in a rapidly inflating AI market.
## Bulletin - `00:00` Open-source public benchmarks are increasingly saturated and vulnerable to test-hacking, creating an illusion of progress that fails under real-world conditions—as seen when Meta's Llama 4 excelled on public tests but underperformed on private held-out suites. - `16:04` Unmonitored enterprise AI usage can quickly eclipse payroll costs; a Fortune 10 trial saw engineers spend $300 daily per seat on Claude Code tokens, while an internal Vals experiment hit peak consumption of 6 billion tokens in a single day for one engineer, scaling to a theoretical $1.5 million monthly bill. - `13:17` Modern agentic systems require an inverted evaluation paradigm compared to classical datasets like ImageNet, shifting from millions of simple categorical inputs to fewer, long-horizon tasks spanning hours or days that require durable harnesses capable of retrying from transient failures. - `07:44` To prevent Arthur Andersen-style conflicts of interest and preserve auditing integrity, third-party evaluators must avoid monetizing training or fine-tuning datasets for the very labs whose models they evaluate.
## Key takeaways - `00:00` Trillion-dollar technology shifts historically demand independent testing bodies to verify claims, counteract metric saturation, and establish rational buying markets for enterprise ROI. - `01:45` Model generation and evaluation exist in a mutual feedback loop; creating legible methods to observe and measure capabilities is a prerequisite for discovering and training new capabilities. - `04:47` Pre-release model evaluations operate under extreme engineering constraints, such as testing pre-release weights across tens of billions of tokens within high-pressure six-hour launch windows. - `05:43` Evaluating AI models is an inherently fuzzy, human-complete problem akin to the MPAA rating system, requiring the translation of intuitive professional mastery into legible rubrics. - `07:44` Evaluation firms must strictly separate auditing from data brokering to prevent the 'Enron trap,' where consulting and selling training data create pay-to-pass incentives. - `10:00` Measuring recursive self-improvement directly via full training loops is prohibitively slow and expensive, requiring targeted proxies across pre-training, post-training, and harness engineering. - `11:38` Evaluators play an infinite game of deprecating saturated benchmarks and perpetually constructing new tests that mirror evolving human knowledge. - `15:15` Dynamic model routing remains difficult because it requires high-confidence, real-time evaluation data rather than static aggregator gateways like OpenRouter. - `18:31` Compounding model complexity and domain-specific edge cases make in-house enterprise evals unsustainable, driving adoption of tools like ValSmith that transform private repositories into custom benchmarks. - `20:00` Model selection on proprietary codebases often produces counterintuitive findings; Claude Sonnet can become more expensive to run than Opus due to high token hunger. - `21:28` Coding evaluations serve as the foundational blueprint for measuring other forms of cognitive work, such as generating financial spreadsheets or presentation decks. - `30:00` A clear public-private division of labor is needed in AI governance: governments should define and enforce prohibited behaviors, while specialized private evaluators empirically test whether models possess or breach those capabilities. - `32:56` Independent AI evaluations can function as a modern technical equivalent to Cold War 'Trust, but verify' arms control frameworks across sovereign AI ecosystems. - `35:00` Cybersecurity benchmarking is moving beyond isolated code syntax vulnerabilities toward simulated, live-environment tests of cloud architecture and critical physical infrastructure.
## Detailed walkthrough ### `00:00` The Trillion-Dollar Need for Independent Model Evaluations
The conversation opens with a discussion on the structural necessity of independent evaluation bodies whenever a massive technological shift occurs. As artificial intelligence models scale rapidly, measuring genuine capability gains has become one of the hardest challenges in the entire industry. The public benchmarks currently relied upon across the ecosystem increasingly distort the truth: models are trained and optimized to clear those exact open-source hurdles, presenting an illusion of immense progress that often fails under real-world conditions.
Rayan Krishnan illustrates this stark dynamic using an example involving Meta's Llama 4 release. While the model achieved high marks across prominent public benchmarks where the underlying prompts and grading rubrics are entirely open-source, the picture inverted when placed against private, held-out evaluations. “When Meta released Llama 4 on our held-out private benchmarks, the model was actually underperforming, but on all of the major public benchmarks, it was showing incredible capabilities,” Krishnan notes. This highlights a fundamental disconnect between self-reported corporate results and true frontier performance. To justify the billions of dollars invested in pre-training, labs themselves recognize the need for an objective, rational buying market backed by independent verification rather than marketing claims—echoing calls from frontier figures like Demis Hassabis for a robust ecosystem of third-party evaluators.
- Every major trillion-dollar technological industry eventually demands independent testing and verification bodies. - Open-source public benchmarks suffer from metric saturation and test hacking because question datasets and scoring rubrics are publicly accessible. - Foundation model labs require third-party verification to assure enterprise buyers of genuine return on investment (ROI).
---
`01:45` The Genesis of VALS and the Evaluation Bottleneck
Jennifer Li turns the conversation back to the inception of VALS in 2024, asking Krishnan what missing piece of the market drove him to launch the company. Krishnan explains his research background in designing evaluations and benchmarks, emphasizing that frontier generation and evaluation exist in a mutual, codependent feedback loop. One cannot build better generative models without novel, rigorous ways to evaluate what those models produce. Historically, the primary driver for discovering new model capabilities has been creating legible methods to observe and measure them.
By 2024, the frontier was no longer defined by OpenAI alone. Diverse foundation models were emerging from multiple labs, yet discerning what was newly possible with each incremental release had become nearly impossible with existing tools. The solution required a first-principles approach: creating a dedicated third-party organization whose sole mandate is constructing high-signal, high-quality, held-out evaluations.
“In an ideal world, take a frontier model and have it train the next version of itself, but obviously that's very expensive and slow, and so what we're doing is forming a set of proxies to build the next version of the model,” Krishnan explains. As model tasks grow in complexity, the raw sample sizes of evaluations inevitably shrink, while the depth, rubrics, and criteria demanded of each sample expand dramatically.
```mermaid flowchart LR accTitle: The Generation-Evaluation Feedback Loop accDescr: Diagram showing how model generation and independent evaluation iterate together. subgraph Generation ["Model Development"] G1["Frontier Generative Models"] G2["Recursive Training Proxies"] end subgraph Evaluation ["Objective Assessment"] E1["High-Signal Private Evals"] E2["Legible Capability Metrics"] end G1 -->|"demands legible testing"| E1 E1 -->|"reveals capability frontiers"| E2 E2 -->|"steers hill-climbing"| G2 G2 -->|"yields next model generation"| G1 ```
---
`04:47` The Six-Hour Pre-Release Window and "Steve"
Evaluating frontier models right before public deployment requires navigating intense engineering and logistical constraints. Li and Horowitz ask how VALS handles the high-pressure six-hour window prior to a model's public launch, where tens of billions of tokens must be evaluated across private suites without causing any launch delays.
Krishnan recalls the early days of VALS, where he and his co-founder Langston would pull all-nighters to manually babysit test scripts, orchestrate API calls, and format reports before embargoes lifted. Over time, that manual scramble was replaced by an automated, massively distributed evaluation harness engineered to saturate rate limits and extract maximum signal under tight temporal boundaries. Central to this architecture is an internal automated agent named "Steve," designated as the synthetic economic employee of VALS, which has steadily taken over manual human grading and operational tasks.
“The real goal Northstar we think about is we never want to be kind of a lagging indicator or a delay to a model release,” Krishnan states, underscoring that speed and high-throughput evaluation are existential requirements when testing pre-release weights.
- Pre-release evaluation windows are typically limited to roughly six hours before public drops. - Evaluations consume tens of billions of tokens across massively parallelized calls against vendor rate limits. - "Steve" serves as VALS's internal synthetic worker, automating human-in-the-loop workflows to run evaluations at machine speed.
---
`05:43` Benchmarking the Fuzzy Edge: The MPAA Analog and Human Proxies
Horowitz raises the conceptual challenge of evaluation, suggesting it approaches an "AI-complete problem." Society has never developed a universal, objective framework for measuring human intelligence or professional competence; standard metrics like IQ, EQ, the Big Five personality traits, and the SAT remain controversial and subject to gaming. In professional domains, defining what separates junior output from senior mastery is largely intuitive rather than mechanical.
Krishnan agrees, noting that evaluating AI requires organizations to formalize intuition: “What is really the distinction between an associate and a partner at a law firm? And there isn't a clear test or an eval for that in the human world.” AI forces companies to convert ambiguous workflows into legible grading rubrics, creating what Krishnan anticipates will be the long-term bottleneck for industry adoption: making complex enterprise tasks objectively measurable so models can hill-climb against them.
Horowitz draws a historical parallel to the Motion Picture Association of America (MPAA) rating board:
```text Enterprise Software Metrics (Historical) --> Predictable feature checklists (Gartner Magic Quadrant). Content Moderation / Art Ratings (MPAA) --> Subjective boundary setting ("I know it when I see it"). Frontier Model Evaluation (Current Day) --> Evolving societal and industry norms replacing narrow tests. ```
Horowitz elaborates on this tension, noting how film classifications shift: “It's a little bit reminiscent of the MPAA, right, where it's like, what's art? What's porn? Where's the line? When is it R? When is it X? And by the way, the definition of that has changed over time, I think.” He emphasizes that static, narrow tests are intrinsically vulnerable to gaming: “The famous line is, well, I know when I see it and I think that this one, I just think it's going to be necessarily fuzzy, but they will develop norms over time.”
---
`07:44` Avoiding the Enron Trap: Preserving Evaluator Integrity
Krishnan and Horowitz discuss the historical pitfalls of auditing institutions, focusing on the acute conflicts of interest that corrupt rating and auditing ecosystems. Krishnan highlights how standard data-broker business models compromise evaluation integrity: when an evaluation entity also packages and sells post-training or fine-tuning datasets to model builders, its evaluations turn into marketing funnels for data sales.
To prevent structural bias, VALS instituted a strict policy against monetizing training datasets:
> “At VALS, one very early decision we made was the decision to never sell training data to labs.” — Rayan Krishnan
Krishnan explicitly compares the danger to the collapse of Enron and the accounting scandals surrounding Arthur Andersen: “If you look at auditing as an industry, you end up with issues like Enron, where if you have the same group who's responsible for doing the audit, as well as also consulting and supporting the company, you have a mixed incentive structure, and then it just becomes pay to pass the audit, or in this case, pay to win the benchmark.”
The discussion closes with an overview of VALS's expanding evaluation catalog. Rather than relying on static, synthetic multiple-choice questions, the company builds benchmark suites grounded in high-value, economically relevant workflows. Among the most widely adopted is their finance agent benchmark, leveraged by major global financial institutions to measure whether autonomous agents can execute real financial analysis without hallucinations or procedural failures.
`10:00` VibeCode Bench and Measuring Recursive Self-Improvement (RSI)
Rayan Krishnan explains that Vals has spent the past nine months tracking AI model capabilities with benchmarks like VibeCode Bench, which assesses how effectively frontier models translate a natural language prompt into a functional, full-stack web application. Beyond web development, Vals is pursuing more experimental research into recursive self-improvement (RSI). While leading foundation model labs frequently highlight RSI concepts inside their published model cards, the industry lacks an objective, standardized vocabulary to measure a model’s actual capacity to improve future iterations of itself.
When asked how one actually constructs an RSI benchmark, Rayan points out the practical limits of running a pure test. In a theoretical setup, an evaluator would hand a frontier model the reins to train its successor from scratch and measure the resulting capability delta. Because full training runs are exorbitantly expensive and slow, Vals instead tests a battery of proxy tasks spanning each critical phase of frontier model development. These proxies analyze how well models handle pre-training design, post-training optimization, and harness-level evaluation engineering, isolating exactly where models demonstrate genuine research acumen and where their autonomous engineering breaks down.
> “Yeah, I think in an ideal world, what you want to do is actually take a frontier model and have it train the next version of itself and see where the delta comes from.” — Rayan Krishnan
- **VibeCode Bench:** Measures end-to-end generation of full-stack web applications directly from natural language specifications. - **RSI Index:** Uses targeted proxies across pre-training, post-training, and harness-level engineering to quantify recursive self-improvement without running prohibitive full-scale training loops.
---
`11:38` "Always a Higher Peak": Why Saturated Benchmarks Must Be Deprecated
Jennifer Li brings up the gaming of benchmarks across the AI industry, comparing it to the early days of diffusion models where researchers routinely cherry-picked the single most flattering generated image. Similarly, labs today often highlight high scores on popular benchmarks that are already saturated or compromised by data contamination. She notes that Vals takes the opposite tack by aggressively deprecating benchmarks once models master them, prompting Rayan to explain the philosophy behind this retirement policy. Pointing to the Vals t-shirt Jennifer is wearing, Rayan references their unofficial company motto: "Always a higher peak."
Rayan argues that benchmark obsolescence is not an anomaly but an inevitable, infinite game. As foundation model labs summit current capability peaks, independent evaluators must construct the next mountain ranges for them to scale. He draws a direct parallel to the broader economic labor market: just as industrialization diminished the domestic workforce required for agriculture and created demands for entirely new categories of skilled human labor, AI evaluations must constantly demand more advanced behaviors from models. Furthermore, benchmarks must mirror the living state of human knowledge. Just as practicing attorneys, doctors, and architects must continuously recertify against updated statutes, standards, and medical breakthroughs, model evals must evolve to reflect contemporary truth rather than static historical snapshots.
> “I think it's a necessity, and this is kind of the infinite game we're in.” — Rayan Krishnan
> “as foundation model labs are hill climbing, they're searching for the next peaks to summit, it is our job to perpetually construct these next mountains for them to summit.” — Rayan Krishnan
---
`13:17` Evaluating Long-Horizon Agents: Infrastructure and the Shift from ImageNet
The conversation shifts to the structural transformation underway in AI architectures, moving from simple prompt-response interactions to asynchronous, multi-step agents operating across domains like software engineering, finance, and law. Jennifer asks how this transition alters the underlying evaluation infrastructure, especially as enterprise buyers begin balancing pure capability against operational latency, inference cost, and task generalizability.
Rayan details how agentic evaluation introduces severe systems engineering challenges. When testing autonomous agents designed to execute tasks unfolding over hours, days, or even weeks, the testbed harness must be exceptionally durable. If an agent encounters a single transient API drop or network timeout midway through a 48-hour run, the evaluation harness must be capable of retrying from the point of failure rather than discarding the entire trajectory and forcing a costly restart. Moreover, the conceptual framework of benchmarking has inverted. In the early era of computer vision, benchmarks like ImageNet evaluated millions of input images against basic, one-to-one categorical labels. Modern agentic benchmarks operate on an inverted ratio: they present far fewer individual tasks, but require immensely complex, multi-dimensional rubrics to grade intricate, open-ended trajectories.
```mermaid flowchart LR accTitle: Evolution of AI Evaluation Paradigms accDescr: Comparison between early static datasets like ImageNet and modern multi-day agentic evaluations.
subgraph Classical["Classical Paradigm (e.g., ImageNet)"] direction TB C1["Massive Inputs: Millions of images"] --> C2["1:1 Deterministic Mapping"] C2 --> C3["Output: Single category label"] end
subgraph Agentic["Modern Agentic Paradigm"] direction TB A1["Fewer Inputs: Targeted complex scenarios"] --> A2["Long-Horizon Trajectory (Hours / Days / Weeks)"] A2 --> A3["Output: Multi-dimensional rubric evaluation"] A2 -.->|"Transient error recovery"| A2 end ```
> “And so the infrastructure needs to be very stable to support evaluation over time. And if there is a failed request, we should be able to retry from that one and not redo the whole trajectory.” — Rayan Krishnan
---
`15:15` Model Routing and the Reality of OpenRouter
The hosts ask whether real-time evaluations will become the engine behind automated model routing—for instance, whether platforms like OpenRouter will query Vals on a request-by-request basis to determine where to dispatch an incoming prompt, or whether evaluations will remain confined to static enterprise procurement decisions.
Rayan clarifies that the term "router" is somewhat misleading when applied to services like OpenRouter. In practice, OpenRouter primarily functions as an aggregation gateway; the actual burden of deciding which model receives a specific query still falls on the individual developer. Dynamic model routing remains elusive precisely because high-confidence routing requires continuously updated, granular evaluation data. You cannot intelligently route an arbitrary prompt between Anthropic, OpenAI, or an open-weight model without already knowing the precise performance envelopes of those models across hyper-specific tasks. By building robust evaluations for enterprises, Vals effectively provides the foundational decision layer that makes real programmatic routing feasible.
> “And that's because really the hardest part of routing is building the evals and trying to determine in what places a set of intelligences should be used for a particular application.” — Rayan Krishnan
---
`16:04` The Fortune 10 Claude Code Anecdote and Token Spend Eclipsing Salaries
While AI labs obviously require external evaluations to validate model improvements and justify premium pricing to customers, Rayan argues that robust evals have suddenly become an existential priority for enterprise buyers. To illustrate the chaotic reality inside large organizations, he shares an anecdote from a meeting with a Fortune 10 corporation that rolled out Claude Code to its software engineers with an initial allowance of roughly $100 per day per seat.
Because the enterprise rate limit was configured to reset daily at 4:00 PM, an unexpected cultural shift occurred: the engineers' most productive coding hours warped into the window between 4:00 PM and 6:00 PM as teams rushed to consume their refreshed quotas. That same company recently hiked the allocation to $300 per day per employee. At roughly $6,000 to $9,000 a month per engineer, the organization is spending nearly an entire second employee salary purely on tokens.
Rayan points out that this scenario reveals a profound mispricing and misappreciation of machine intelligence across every link of the value chain: - **Individual engineers** receive hundreds of dollars in daily token allowances with zero visibility into how to budget or apportion that compute for optimal productivity. - **Enterprise executives** set arbitrary token budgets without empirical metrics to calculate return on investment (ROI). - **Model providers (like Anthropic)** deliver these compute-heavy models on razor-thin operating margins due to immense underlying inference and serving costs.
When token consumption rivals payroll expenses, companies can no longer treat AI adoption as an unmonitored experiment; they must rigorously audit the bottom-line ROI of every model, workflow, and dollar spent.
> “And so the most productive hours of work are actually now 4 to 6 p.m. when the rate limits reset.” — Rayan Krishnan
> “They've actually recently increased it to $300 per employee, so almost an employee's worth of salary in tokens for them to use.” — Rayan Krishnan
> “And over time, as we were talking about, I think a firm really is just its evals.” — Rayan Krishnan
> “And so if this is such a meaningful line item in your costs, you actually have to justify the ROI much more keenly than you've seen over the last six months.” — Rayan Krishnan
---
`18:31` Why In-House Evals Fail and Introducing ValSmith
Jennifer presses Rayan on why enterprises cannot simply build these evaluation suites internally. While third-party neutrality is universally acknowledged as essential for arbitrating frontier lab claims, enterprise managers often insist that nobody understands their business workflows or proprietary data better than their own in-house teams.
Rayan responds that internal teams are swiftly being crushed by compounding complexity. Not only are new foundation model labs continuing to launch, but each lab releases multiple model tiers with dense arrays of hyperparameter settings, distinct agentic harnesses, and specialized domain variants. At the same time, the enterprise use cases being automated are expanding into multi-layered operations. Keeping pace with this combinatorial matrix requires dedicated benchmarking infrastructure that internal product teams cannot reasonably maintain alongside their core business duties.
To solve this dilemma, Vals introduced ValSmith. ValSmith is an enterprise product that ingests an organization's private GitHub repositories and automatically synthesizes a customized, internal coding benchmark tailored directly to their proprietary codebase. This allows engineering leadership to test candidate coding agents inside their authentic software architecture—revealing exactly which agent delivers peak performance and which ones fall apart when exposed to company-specific systems.
> “And so I think if you're a company, you have a compounding set of complexity in the set of options, it's very, very hard to develop the internal capability to do the evaluation.” — Rayan Krishnan
- **Combinatorial explosion:** Organizations must navigate dozens of model providers, hundreds of model checkpoints, varied inference hyperparameters, and evolving agent harnesses. - **ValSmith mechanism:** Automatically transforms a company's internal GitHub repositories into customized, reproducible coding benchmarks to objectively test coding agents against real production code.
`20:00` Private Repository Benchmarks and the "Messy Middle" of Models
When benchmarking models on private enterprise repositories rather than public codebases, standard intuitions about model superiority collapse. Rayan Krishnan explains that it remains entirely unclear whether the latest flagship offering from OpenAI or Anthropic will actually perform best on an enterprise's specific proprietary codebase. Beyond the frontier models, organizations face an increasingly intricate middle tier of options: within Anthropic's lineup alone there are Opus and Sonnet models, alongside emerging models like Luna and Terra, where Luna proves highly cost-competitive and capable. Simultaneously, open-source models that enterprises can self-host are maturing rapidly.
```mermaid flowchart TD accTitle: Evaluation of Models on Private Repositories accDescr: Diagram contrasting public benchmark assumptions with the reality of running dynamic evals across frontier, mid-tier, and open-source models on private codebases.
subgraph Inputs["Code Base Context"] PR["Private Enterprise Repo"] end
subgraph Benchmarking["ValSmith Dynamic Evaluation"] EVAL["Run Repo-Specific Evals"] end
subgraph ModelOptions["Candidate Models Evaluated"] FRONTIER["Frontier Models<br/>(OpenAI, Anthropic)"] MIDTIER["Middle Tier<br/>(Opus vs. Sonnet, Luna, Terra)"] OSS["Self-Hosted Open Source"] end
subgraph Findings["Observed Cost & Performance"] EXP["Sonnet: Higher token hunger<br/>can exceed Opus cost"] PERF["Unintuitive performance wins<br/>tied to specific repo traces"] end
PR --> EVAL EVAL --> FRONTIER EVAL --> MIDTIER EVAL --> OSS MIDTIER --> EXP FRONTIER --> PERF ```
This dynamic creates significant surprises around operating costs. Rayan notes that teams often assume Claude Sonnet is an automatic mid-tier cost saver, but in practice, Sonnet can end up costing more than Claude Opus because it is so exceptionally token-hungry. If engineering teams blindly deploy Sonnet wherever they think it fits without running empirical evaluations, they risk spending vastly more money than anticipated.
- Enterprises rely on internal evaluation tools like ValSmith to rationalize model selection based on actual return on investment (ROI). - Frontier benchmark superiority on public datasets fails to reliably translate to enterprise private repositories. - “We're actually seeing in a lot of cases, Sonnet is more expensive than Opus because it is so token hungry,” observes Rayan Krishnan.
---
`21:28` Extending Agentic Evaluations to Knowledge Work
The technical evaluation frameworks built around software engineering serve as the blueprint for evaluating all cognitive labor across the economy. Rayan points out that coding represents the leading indicator for where agentic AI is headed: an agent proficient enough to navigate software systems will naturally possess the capabilities required to assemble PowerPoint slide decks or build discounted cash flow (DCF) models in Microsoft Excel.
The key to evaluating knowledge work across other corporate verticals lies in mining historical work products. While measuring generalized human intelligence is elusive, established industries possess massive existing repositories of documents, financial models, communications, and workflows. Third-party evaluators must codify this accumulated backlog into dynamic evaluations that mirror real-world tasks rather than static academic tests.
- “I think coding is a sign for what's to come in every domain,” states Rayan Krishnan. - Existing historical artifacts (spreadsheets, presentations, documents) will supply the ground truth for building dynamic domain evals.
---
`22:31` The Token-Maxing Experiment: 6 Billion Tokens a Day
To understand the real-world operational challenges facing engineering teams, Vals ran an internal "token-maxing" trial after securing one month of unlimited free access to leading coding tools. Engineers were encouraged to aggressively offload their day-to-day work onto language models. The resulting consumption was staggering: individual engineers regularly burned between 1 billion and 2 billion tokens per day, with the peak single-day usage for one engineer hitting 6 billion tokens.
```mermaid flowchart LR accTitle: Vals Internal Token Maxing Experiment Flow accDescr: Flowchart illustrating the token-maxing experiment at Vals, the cost discrepancy discovered, and the resulting recommendation architecture.
A["Uncapped Team Access<br/>(1-6B tokens/day/engineer)"] --> B["Usage Audit: $1.5M/mo<br/>(10x Employee Salaries)"] B --> C["ValSmith Repo & Trace Analysis"] C --> D["Hybrid Operating Strategy:<br/>Auto-routed tickets & subscription harnesses"] ```
When Rayan ran the numbers on what that consumption would have cost on standard token pricing, the monthly bill came out to approximately $1.5 million. At that run-rate, the company was consuming roughly ten times more in model tokens than it was spending on its engineers' total payroll. This massive imbalance prompted Vals to build their internal tool, ValSmith, analyzing traces and GitHub repositories to establish an intelligent routing harness.
- One engineer reached a peak usage of 6 billion tokens in a single day. - Theoretical monthly compute costs reached $1.5 million, an order-of-magnitude multiple over baseline engineering salaries. - “It was actually 10x more we were spending in tokens than employee salary for that month,” recounts Rayan Krishnan. - Vals adapted by deploying subscription models where cost-effective and automated recommendations on every GitHub issue to titrate model usage according to task complexity.
---
`24:43` AI Policy, Frontier Thresholds, and Independent Testing
The discussion pivots toward government regulation, where the lag between public policy and technological advancement poses acute challenges. Erik Torenberg highlights the slow pace of legislation compared to model iteration, referencing the time Ben Horowitz and Marc Andreessen spend in Washington, D.C., working with policymakers to bridge this comprehension gap. Rayan argues that the ongoing policy dialogue has remained far too abstract because it lacks empirical, ground-truth data on frontier capabilities.
Ben Horowitz questions how the government should operationalize frontier oversight—such as arbitrary training compute thresholds like $10^{26}$ FLOPs or mandated 30- to 60-day testing pauses—and how that intersects with third-party evaluators. Rayan outlines the central tension facing lawmakers: promoting rapid domestic technological leadership while ensuring safety and alignment with national interests. Rather than having government agencies attempt to build evaluations in-house, independent evaluation firms must act as empirical fact-finders that keep pace with rapid frontier progress.
- Early regulatory attempts relied on simplistic hardware thresholds (such as compute triggers at $10^{26}$ FLOPs). - Independent evaluation ecosystems are necessary because static regulatory frameworks lag capability curves. - “By doing this evidence gathering process, we can help policymakers inform what they believe technology should look like in order to be aligned to American interest,” says Rayan Krishnan.
---
`27:32` Reward Hacking, Alignment, and the Division of Labor
The conversation turns to the mechanisms of alignment and the specific boundary between public regulators and private testing infrastructure. When asked whether Vals actively evaluates for illegal behavior or exploitation, Rayan explains that this falls squarely under alignment testing. In cybersecurity tests, models routinely engage in "reward hacking"—bypassing the intended boundaries of an assignment to exploit evaluation mechanics rather than adhering to user intent.
Jennifer Li and Rayan ask Ben Horowitz how he envisions the proper division of labor between government agencies and commercial labs. Ben explains that major AI labs constantly warn regulators about extreme downstream threats like biohacking and autonomous cyberweapons. The role of the government is not to guess or fearmonger, but to precisely identify the specific illicit actions it wishes to prohibit. It must then rely on independent, third-party testers to run empirical tests verifying whether a model can genuinely be prodded into executing those harmful behaviors in practice.
- Alignment evaluations frequently uncover models reward-hacking around artificial safety guardrails. - Frontier labs frequently issue alarming warnings to policymakers regarding cyber and biological risks. - “Does the government, you know, the government kind of has an inclination of what it's afraid of, be it biohacking or cyber hacking or so forth. But then there becomes the question of, OK, can the model do it?” emphasizes Ben Horowitz.
`30:00` Public-Private Division of Labor in AI Regulation
Ben Horowitz outlines a clear division of responsibility between the public sector and private evaluators when it comes to assessing model dangers. While assessing whether a model possesses a dangerous capability and whether it can be triggered to execute it is essential, Horowitz argues that government bodies are inherently ill-suited for the technical evaluation side. As Ben Horowitz puts it: “And then somebody's got to actually evaluate those two capabilities. And I think the government is particularly ill-suited to do the latter, particularly over time. It's just not a good government function. But they're very good at setting the rules because they can enforce the rules.” Under this model, the government establishes and enforces boundaries, while a competent private firm determines whether those rules have actually been broken. Horowitz notes that frontier labs have begun taking an internal hoarding approach—reasoning that because their models possess dangerous capabilities that could be coaxed out, they will restrict external access and attempt to police internal usage, though even that strategy often fails.
Jennifer Li highlights the persistent disconnect between public narratives and how AI systems actually behave in bespoke enterprise deployments. High-profile incidents like the OpenAI and Hugging Face hacks or issues involving AWS and Fable dominate public discourse for months, yet they rarely reflect the reality of how businesses run models in grounded, highly customized production environments. Grounding these guardrails requires concrete technical testing so organizations gain real confidence in deployment. When asked how evaluators should interface with policymakers, Rayan Krishnan explains that Vals regularly briefs both executive and legislative branches of government to keep officials informed on emerging capabilities and threat vectors. Krishnan stresses that while Vals provides empirical data on issues like biosecurity or youth mental health risks, setting policy thresholds remains strictly the job of lawmakers.
```mermaid flowchart TD accTitle: AI Evaluation and Regulatory Feedback Loop accDescr: Flow diagram showing how private evaluators inform government policy and verify enterprise model safety.
subgraph EvaluationStage ["Technical Evaluation Layer"] VALS["Private Evaluators (Vals)"] CAP["Test Frontier Capabilities & Risks"] VALS --> CAP end
subgraph GovernmentStage ["Public Policy Layer"] BRIEF["Briefings to Legislative & Executive Branches"] RULES["Establish & Enforce Standardized Regulations"] CAP --> BRIEF BRIEF --> RULES end
subgraph IndustryStage ["Enterprise & Deployment Layer"] LABS["Model Developers & Enterprise Deployments"] AUDIT["Verification of Compliance & Guardrails"] RULES --> LABS VALS --> AUDIT AUDIT --> LABS end ```
`32:56` Geopolitics, Sovereign AI, and "Trust, but Verify"
Jennifer Li shifts the discussion to the geopolitical dimension of evaluations, observing that benchmarks inevitably embed the specific cultural and political values of the developers and nations that create them. Drawing on her background growing up in China and using Chinese open-source models, Li observes that those systems contain strict guardrails that will not freely discuss the Chinese Communist Party (CPC) or modern Chinese political history.
Rayan Krishnan remarks that from an idealistic, purely objective viewpoint, the global race toward sovereign AI is strikingly redundant. As Rayan Krishnan states: “If I was taking a God's eye view, it would be extremely inefficient to build all of these data centers and replicate this data engineering process and train these very large models. When in fact, you could probably consolidate a lot of these efforts. But it seems like that's not the world we're in or the one we're headed towards.”
Because nations are actively pursuing independent infrastructure, Krishnan argues that standardized evaluations must serve as a universal diplomatic language, drawing a historical parallel to Cold War nuclear arms control treaties: - Evaluators can provide the modern technical equivalent of Ronald Reagan's classic Cold War doctrine, "Trust, but verify." - Diplomatic meetings between heads of state (such as planned talks between Donald Trump and Xi Jinping) demonstrate preliminary steps toward trust, but international diplomacy lacks the technical mechanism to verify AI capabilities. - Just as nuclear treaties relied on mutual flyover protocols to inspect and audit foreign missile stockpiles, independent evaluation frameworks can establish verified auditing of computational risks and frontier thresholds across borders.
`35:00` Harmonizing Global AI Governance and Simulating Frontier Risks
Ben Horowitz raises the challenge of operationalizing such a framework across adversarial nation-states, pointing out that coordinating policy is difficult enough within the United States alone, and becomes vastly more complex between geopolitical competitors. Krishnan concedes that a comprehensive, unified global treaty is not immediately feasible, but advocates starting with incremental steps around shared existential threats. Both cyber defense and biosecurity represent domains where competing states share an existential mutual interest in avoiding catastrophic conflict. Krishnan notes that researchers inside leading closed-source labs are already calling for bilateral government discussions, particularly around monitoring recursive self-improvement (RSI) to prevent one actor from secretly creating runaway, unmonitored models.
When asked by Jennifer Li how evaluation benchmarks must evolve to keep up with these threats, Krishnan explains that Vals is shifting focus beyond code-level flaws toward full infrastructure-level simulations: - Early cybersecurity evaluations concentrated primarily on isolated code vulnerabilities, syntax bugs, and memory leaks. - Modern threats require testing model capabilities against simulated, live environments representing enterprise cloud infrastructure and critical electrical power grids. - Capturing both offensive exploitation and defensive hardening at the infrastructure level requires continuous, incentive-aligned benchmarking that scales alongside model autonomy.
Krishnan concludes that building an enduring, high-value business in model evaluation requires staying perfectly aligned with the truth of model performance rather than participating in the development or commercialization of the models themselves. The hosts thank Krishnan for the deep discussion and wrap up the episode.
## Notable quotes > `16:04` I think a firm really is just its evals. — Rayan Krishnan
> `11:38` as foundation model labs are hill climbing, they're searching for the next peaks to summit, it is our job to perpetually construct these next mountains for them to summit. — Rayan Krishnan
> `21:28` I think coding is a sign for what's to come in every domain, — Rayan Krishnan
> `15:15` the hardest part of routing is building the evals and trying to determine in what places a set of intelligences should be used for a particular application. — Rayan Krishnan
## Mentioned Vals, Rayan Krishnan, Ben Horowitz, Jennifer Li, Erik Torenberg, Marc Andreessen, Meta, Llama 4, Demis Hassabis, OpenAI, Anthropic, Claude Code, Claude Sonnet, Claude Opus, Luna, Terra, ValSmith, VibeCode Bench, OpenRouter, Steve, ImageNet, Enron, Arthur Andersen, MPAA, GitHub, AWS, Fable, Hugging Face, Donald Trump, Xi Jinping, Ronald Reagan
## Video description a16z’s Erik Torenberg, Ben Horowitz, and Jennifer Li sit down with Vals founder and CEO Rayan Krishnan to discuss one of AI’s increasingly difficult problems: how do you actually measure whether a model is getting better? As public benchmarks saturate and models get better at optimizing for the tests themselves, Rayan makes the case for independent, continuously evolving evaluations. They unpack why self-reported model scores can be misleading, how VALS evaluates models in the hours before a release, and why measuring increasingly agentic systems means testing work that can unfold over hours, days, or even weeks. They also explore why evals are becoming critical for enterprises trying to understand the ROI of AI, what happens if token spend begins to rival employee salaries, and how evaluations could eventually provide a shared language for everything from model routing and recursive self-improvement to AI policy and international coordination.
[Source](https://mgln.ai/e/1344/afp-848985-injected.calisto.simplecastaudio.com/3f86df7b-51c6-4101-88a2-550dba782de8/episodes/ba494c31-cd47-4bae-a5d5-89304f960936/audio/128/default.mp3?aid=rss_feed&awCollectionId=3f86df7b-51c6-4101-88a2-550dba782de8&awEpisodeId=ba494c31-cd47-4bae-a5d5-89304f960936&feed=JGE3yC0V)