Vals Builds Private AI Benchmarks as Public Tests Mask Model Flaws

公开测试掩盖模型缺陷，Vals构建私有AI基准 随着企业Token支出或将超过员工薪资，独立的私有评估正取代趋于饱和的公开基准，以揭示模型的真实性能。 随着前沿AI模型的发展超越静态公开基准，且各实验室面临针对公开测试集进行“奖励作弊”（reward-hacking）的风险，独立评估层对于确立真正的技术进展已变得至关重要。Rayan Krishnan详细阐述了Vals如何在发布前紧迫的窗口期内执行数十亿Token规模的评估、评估长周期智能体工作流、并在规避结构性利益冲突的同时追踪递归自我改进的替代指标。讨论组探讨了飙升的Token成本如何迫使企业对其专有代码库进行基准测试，并严格量化业务投资回报率。最后，讨论将第三方实证测试定性为政府监管、对齐验证以及类似国际军控风格的安全框架不可或缺的技术基石。 标准的公开AI基准已经饱和且易受奖励作弊影响，催生了市场对独立的非公开私有评估的迫切需求。 (0:55) 实验室自行汇报的指标可能造成严重的市场失真，正如Llama 4在公开排行榜上领跑但在私有基准测试中表现欠佳所展现的那样。 (2:52) Vals通过部署大规模分布式测试工具集和名为“Steve”的内部智能体工作流，在发布前短短6小时的窗口期内完成对前沿模型的评估。 (4:05) 评估前沿智能需要将以往模糊的人类判定形式化，其运作更类似于不断演进的行业规范（如MPAA分级制度），而非静态的核对清单。 (6:00) 为防止安然式的利益冲突，评估机构必须保持严格独立，并避免向基础模型实验室出售训练数据流水线。 (9:06)

According to Vals co-founder Rayan Krishnan and Andreessen Horowitz partner Ben Horowitz, independent evaluation has become essential across the AI ecosystem because public benchmarks are easily gamed, foundation model labs face structural conflicts of interest, and enterprise buyers require empirical data to justify escalating compute budgets. Failure of Public Benchmarks and Evaluation Conflicts (2:52) Public leaderboards increasingly fail to reflect genuine model capabilities due to reward hacking and data contamination. Krishnan points to Meta's Llama 4 release, where the model demonstrated exceptional scores across major open benchmarks but underperformed on held-out private evaluations. When labs commit billions of dollars to model training, self-reported metrics distort market rationality and undermine buyer confidence. To preserve integrity, Krishnan argues that testing firms must avoid the structural conflicts seen in legacy corporate accounting. Many AI data vendors build promotional benchmarks to sell training datasets to foundation labs—a dynamic Krishnan compares to auditing firms cross-selling consulting services before the Enron collapse. To remain strictly neutral, evaluation organizations must establish foundational rules against selling training data to model developers. Enterprise Economics and Private Repository Benchmarking (19:09) As enterprises shift from isolated prompts to autonomous agents, model testing has become a core operational requirement. Krishnan describes a Fortune 10 company that raised daily Claude Code allowances from $100 to $300 per engineer, creating periods where work stopped when token limits were exhausted. In an internal month-long experiment at Vals, unconstrained usage peaked at 6 billion tokens per engineer in a single day, accumulating roughly $1,500,000 in monthly compute value—exceeding baseline payroll by tenfold. Because public leaderboards do not predict proprietary performance, Vals introduced Valsmith to benchmark agents against private codebases. These private evaluations frequently reveal non-intuitive cost dynamics. For example, developers often choose mid-tier models like Claude 3.5 Sonnet over flagship models under the assumption of lower cost, yet Sonnet's high token appetite in autonomous loops can make it far more expensive on complex repository workflows. Infrastructure Evolution and Tracking Recursive Self-Improvement (13:13) Evaluating autonomous agents requires replacing traditional single-turn, millisecond evaluations with stateful harnesses that execute across hours or days and recover gracefully from intermittent API failures. Under the operating philosophy that there is 'always a higher peak,' Krishnan argues that testing firms must continuously deprecate saturated benchmarks rather than allowing labs to cherry-pick legacy metrics. To address long-term capability growth, Vals created the Recursive Self-Improvement (RSI) Index. Because tasking models to train successive generations from scratch is computationally prohibitive, the index uses empirical proxies spanning pre-training research, post-training optimization, and harness engineering to measure autonomous research skills. Similarly, cybersecurity testing is moving beyond code-level syntax errors toward simulating live enterprise cloud environments and physical utility infrastructure. Technical Verification for Policy and Geopolitical Stability (26:30) Krishnan and Horowitz argue that effective AI governance requires empirical evidence rather than theoretical abstractions, such as arbitrary compute thresholds at 10^26 FLOPs or unguided waiting periods. Horowitz advocates for an institutional division of labor where governments define prohibitions and enforce legal accountability, while specialized private firms conduct empirical testing to determine whether models possess dangerous capabilities or can be coaxed into bypassing guardrails. On the geopolitical stage, Krishnan notes that while nations will duplicate compute infrastructure to build sovereign AI systems, international stability depends on verifiable technical standards. Citing Ronald Reagan's 'trust but verify' Cold War doctrine, Krishnan suggests that standardized third-party evaluations can provide the technical basis for bilateral accords, beginning with shared existential threats like biosecurity and cybersecurity before establishing limits on uncontrolled recursive self-improvement. While empirical benchmarking provides critical visibility for enterprise deployments and policy enforcement, establishing durable industry standards remains complicated by the subjective, fluid nature of advanced human intelligence and the difficulty of securing global coordination before models achieve recursive self-improvement.

随着前沿人工智能加速向具有自主性的智能体演进，公开基准测试因过拟合与数据污染而逐渐失效；独立第三方基准评估机构 Vals 认为，只有通过严格保密的私有测试集与真实工作流模拟，才能为万亿美元级的 AI 产业提供客观的能力验证与资本回报率核算。 公开基准失效与第三方评估的必要性 (0:55) 自2024年初以来，行业逐渐认识到标准化公开基准已无法准确反映真实的前沿技术进展。由于公共测试集的题目和评分细则完全开源，模型开发者容易通过过度拟合或奖励黑客行为（reward hacking）刷取高分，造成公开排行榜表现优异但在实际应用中滑坡的脱节现象。例如在 Meta 发布 Llama 4 时，该模型在各大主要开源公开基准上表现抢眼，但在未公开的私有测试集上却出现明显性能下滑。这种自说自话的测试扭曲了市场认知，难以支撑前沿实验室动辄数十亿美元的模型训练资本支出。行业领袖如 Demis Hassabis 也呼吁建立独立的第三方评测生态。 针对评测机构的中立性问题，Vals 联合创始人 Rayan Krishnan 指出，评估机构必须建立明确的商业防火墙以防范利益冲突。行业内部分数据供应商借开发评测基准之名向实验室推销训练数据集，这类似于安然事件中审计机构同时从事付费咨询业务，极易沦为花钱通关的交易。Vals 坚持确立绝对不向 AI 实验室出售训练数据的运营红线。在具体执行上，Vals 依靠分布式底层架构与名为“Steve”的内部智能体，在严格的6小时封锁窗口期内完成数百亿 Token 的高吞吐量测试，力求在不推迟模型发布的前提下提供客观的质量背书。 智能体评估范式向长周期与真实环境转变 (10:00) 前沿模型正在从单轮问答转向跨越数小时、数天甚至数周的自主长周期智能体系统，评估范式随之发生根本变革。传统的静态测试集（如 MMLU 或 ImageNet）依赖海量单一样本与单选、文本标签式的刚性匹配，而现代智能体评估（如 Vals 的金融智能体基准与跟踪九个月的 VI Codebench 全栈代码生成基准）则采用较小规模的高维度样本，着重考察计算成本、推理延迟、工具调用灵活性以及执行复杂工件的能力。 评估的范围也正从狭义的代码语法或内存泄漏检测，拓展到对复杂基础设施的动态模拟。Krishnan 表示，未来网络安全和关键领域的评测需要模拟企业级云架构乃至电网物理设施，以衡量模型的实际攻防潜力。此外，针对大模型研发中最关键的递归自我改进（RSI）能力，由于让模型从头完整训练下一代版本的成本高昂且耗时漫长，Vals 构建了涵盖预训练研究、后训练优化与测试框架工程的一整套代理评估指标，以精确判断模型究竟在哪些环节具备自主研究能力，又在哪些底层架构探索中出现故障。 企业落地困境：Token 支出与 ROI 的量化陷阱 (16:28) 在企业采购与部署层面，缺乏量化评估体系正在导致严重的资源错配。在代码生成这一企业级 AI 最大支出的领域，排行榜的公开得分与私有代码库的真实效果存在巨大偏差。为此 Vals 推出了针对企业私有 GitHub 代码库构建定制基准的工具 Valsmith，协助企业进行帕累托最优分析。实践中常出现反直觉现象：开发者通常认为 Claude 3.5 Sonnet 比 Claude 3 Opus 经济，但在复杂的工程工作流中，Sonnet 极高的 Token 吞吐需求反而可能导致实际成本大幅反超。 在激进采用前沿工具的背景下，算力开销正在剧烈冲击企业预算结构。一家财富10强企业最初为软件工程师设定每日100美元的 Claude Code 额度，随后迅速上调至300美元；因 API 速率在每日下午4点重置，导致整个技术团队的工作高峰扭曲在下午4点到6点，而在额度耗尽的午后则陷入完全停滞。Vals 内部进行的“Token Maxing”极限测试显示，在不加限制的情况下，单名工程师日均消耗10亿至20亿 Token，峰值达到60亿 Token，团队月度账单折合150万美元，达到员工薪酬成本的10倍。这表明企业亟需基于任务复杂度的自适应路由模型，否则 Token 开支将迅速吞噬原本的人力资本预算。 政府监管分工与主权 AI 的核查机制 (26:30) 在政策制定与地缘政治层面，独立评估是弥合立法滞后与工程现实的关键纽带。针对监管层提出的人为算力门槛（如 10^26 FLOPs）或30至60天发布等待期，Ben Horowitz 与 Rayan Krishnan 认为政府应明确自身的机构分工：政府擅长设定法律红线并惩罚违法违规行为，但缺乏跟进前沿动态评测的技术能力。监管机构不应轻信大实验室声称“唯有自身能在闭门环境下安全控制风险”的自律承诺，而应依靠独立评估机构提供客观证据，区分模型是否真正具备生物武器或网络攻击能力，以及不良行为者能否绕过护栏诱发恶意行动。 面对主权 AI 重复建设数据中心带来的地缘割裂，各方亟需统一的技术评估语言。Krishnan 援引冷战时期核军控条约中里根提出的“信任但需核实”（trust but verify）原则，指出大国竞争中各方或可先在网络安全与生物安全等共同底线问题上迈出务实合作步伐。更长远来看，针对单一大国或单一企业可能引发智能爆炸的递归自我改进（RSI）风险，唯有建立多边认可的独立评测标准与开发节奏阈值，才能防止失控的军备竞赛。 该讨论将独立基准评测定义为 AI 产业不可或缺的基础设施，但如何将模糊的主观人类判断转化为数学上精确的评估标准，以及如何在高度对立的地缘政治博弈中建立真正受各方信赖的全球核查协议，仍是未完全解决的长期挑战。

## Background 该视频由风险投资机构 a16z 于 2026 年 9 月 9 日在 YouTube 频道发布。视频简介指出，a16z 的 Erik Torenberg、Ben Horowitz 与 Jennifer Li 共同对话 Vals 创始人兼 CEO Rayan Krishnan，深入探讨在公开基准失效与刷榜泛滥的当下，如何通过独立且持续演进的第三方评估体系精准衡量前沿模型的真实能力。 ## TL;DR 访谈围绕 AI 模型评测（Evals）的体系重构展开，剖析了以 Llama 4 为代表的“公开高分与私有评测脱节”现象，阐明了前沿模型发布前数小时内进行高强度盲测的工业级挑战。对话深入探讨了从单步问答向长程 Agent 评测演进的范式转移，分析了企业从公有代码库到私有代码库中 Token 支出反超人力成本的 ROI 焦虑与智能路由破局之道。此外，嘉宾们还就递归自我改进指数（RSI）、政府监管与第三方审计的权责边界、主权 AI 的地缘政治博弈及“信任但需核实”的国际协调机制进行了前瞻性研讨。 ## Atmosphere & style 整场对话兼具硅谷一线创业者的工程实操感与顶尖风投合伙人的宏观战略洞察，探讨严谨深刻且不乏幽默交锋。Ben Horowitz 凭借深厚的商业与历史底蕴抛出犀利质询，用电影分级与金融审计作比，直戳行业痛点；Rayan Krishnan 则以极高信息密度沉着应答，分享真实荒诞的企业案例与极限测试细节。探讨从微观技术实现自然过渡至宏观治理与地缘博弈，理性克制中透露出对超级智能飞轮不可逆演进的敬畏与紧迫感。 ## Bulletin - [02:52](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=172s) 公开基准因数据污染与刷题全面失效，Meta Llama 4 在开源公开基准上表现优异，但在 Vals 的保密私有测试集（held-out benchmarks）中盲测成绩严重不及预期，推动行业转向中立第三方评测。 - [17:37](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1057s) 企业级 Token 支出正逼近甚至超越人力成本，某 Fortune 10 巨头为工程师分配每月高达 300 美元的 Token 调用额度，迫使企业将科学评测体系视为衡量真实业务 ROI 与存亡的核心。 - [18:41](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1121s) 面向代码生成的专属评测工具 Valsmith 揭示了反直觉现象：公有基准高分无法映射到私有仓库，且特定长任务中 Claude 3.5 Sonnet 因吞吐量极高，实际花销反而可能超过单价更高的 Opus。 - [36:08](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=2168s) 长期 AI 治理的最大风险在于“递归自我改进”（RSI），若单一机构秘密实现自我演化突破将形成无法预测的超级智能，跨国独立评测基准将成为核查生存风险的通用语言。 ## Key takeaways - [00:00](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=0s) 生成式 AI 的能力构建与评测机制紧密双向驱动，模型能力往往需要先建立起清晰可见（legible）的度量指标才能实现实质突破。 - [02:52](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=172s) 顶尖实验室投入数十亿美元研发前沿模型，迫切需要类似金融审计的中立第三方出具客观能力背书，以培育理性的采购市场，杜绝自吹自擂式的自报分数。 - [04:05](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=245s) 模型发布前的盲测窗口期通常仅有 6 小时，评测团队必须借助高并发自动化系统（如自主 Agent 'Steve'）在跑满提供商速率上限的前提下完成海量 Token 测试。 - [06:00](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=360s) 智能评测如同文化内容分级（如 MPAA 制度），注定带有某种模糊性，其核心价值在于促成全行业在真实商业工作流中达成公认的规范与显性化标准。 - [09:06](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=546s) 评测机构必须恪守不售卖评测集作为训练数据的商业红线，警惕将基准测试作为数据销售漏斗的‘安然式’利益冲突。 - [10:00](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=600s) Vals 推出递归自我改进指数（RSI Index），将昂贵的端到端自训练拆解为预训练、后训练和测试框架工程等代理任务，实现模型科研潜力的标准化横向比对。 - [12:11](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=731s) 基准测试不仅需要持续‘筑起更高的山峰’以防刷榜饱和，还必须像职业资格再认证一样，随着真实世界法律、医学等动态知识实时刷新。 - [13:26](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=806s) Agent 评测范式发生根本转移：从 ImageNet 时代的百万样本单一映射，转变为极少样本量（如 50 个全栈 Web 项目）配合极复杂的多维评判标准与断点续跑容错能力。 - [15:25](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=925s) OpenRouter 等工具本质仍是网关，智能模型路由落地的真正瓶颈在于缺乏精准评测体系来识别模型在具体业务步骤中的效费比。 - [22:40](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1360s) Vals 内部的‘Token 极限实验’显示，工程师在无限制条件下单月单人峰值消耗达 60 亿 Token，总账单是当月员工薪资的 10 倍，倒逼团队转向按任务难度智能推荐工具的管理机制。 - [26:30](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1590s) AI 政策监管面临对抗困境：机械划分算力阈值（如 10^26 FLOPs）脱离技术现实，独立评测机构应充当‘实证数据收集者’为立法提供第一手事实支撑。 - [28:13](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1693s) 科学的政企分工应由政府凭借法定权威划定安全红线并执法，由独立私营评测机构负责测试模型是否在现实中被诱导（Elicitation）出奖励黑客或实质危害行为。 - [33:30](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=2010s) 主权 AI 的割裂发展造成大量算力重复建设，国际社会亟需效仿冷战核军控‘信任但要核实’的逻辑，将通用评测基准作为跨国监管与生存风险核查的技术手段。 - [37:14](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=2234s) 未来网络安全等安全评测正从孤立的代码级静态漏洞检测，跨越到对企业级云架构与关键基础设施的动态真实运行环境仿真。 ## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=0s) 公共基准失效与 Vals 的诞生 每当一个万亿美元级的新兴产业崛起时，市场中往往会出现对独立第三方评测机构的迫切需求。访谈开篇回顾了 Vals 于 2024 年初创立的契机。当时，随着 OpenAI 之外的多家模型实验室陆续推出极具竞争力的新模型，整个行业突然发现传统的公开基准（Public Benchmarks）已彻底无法准确衡量前沿模型能力的真实跃升。作为 Vals 的创始人兼首席执行官，Rayan Krishnan 拥有深厚的学术研究背景，长期致力于基准评测体系的构建。他敏锐地指出，生成系统的能力构建与评测机制之间存在着极其紧密的双向驱动关系——在很多情况下，唯有先建立起清晰可见、具备可读性（legible）的评测机制，模型的实际能力才可能实现实质性突破。 面对市场上大量涌现的新模型，普通用户和企业客户越来越难以辨别每一次模型迭代到底解锁了哪些新能力。从第一性原理出发，Rayan 意识到必须存在一家独立的第三方公司，不参与模型训练竞赛，而是专注于打造极高质量、防刷题且高信号密度的基准测试，以此衡量前沿模型的真正边界。Vals 正是在这一背景下于 2024 年发布了首批基准测试，并在随后几年内迅速被整个 AI 生态系统所采纳。 ```mermaid flowchart LR accTitle: 独立评测在 AI 产业链中的定位 accDescr: 展示模型实验室、Vals 独立评测以及企业客户之间的评估与反馈关系 subgraph Labs["前沿模型实验室"] M1["研发与训练前沿模型"] M2["高额算力投资寻求合理回报"] end subgraph ValsCore["Vals 独立评测机构"] V1["保密私有评测集 (Held-out)"] V2["防刷题的高信号测试规范"] end subgraph Enterprise["企业与采购市场"] E1["评估真实业务 ROI"] E2["建立理性采购标准"] end Labs -->|"发布前提交测试"| V1 V1 -->|"高置信度能力信号"| Enterprise Enterprise -->|"基于真实表现采购"| Labs V2 -.->|"反哺与牵引能力跃升"| M1 ``` ### [02:52](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=172s) Llama 4 的脱节：公开高分与私有评测的现实差距 针对“为什么各大模型实验室无法自行主导评测标准”这一疑问，Rayan 解释道，虽然实验室内部构建了大量出色的基准来驱动自身的“爬坡式”（hill climbing）改进，但只要涉及向外界公布成绩，单纯依赖“自报分数”（self-reported scores）就存在致命缺陷。最具代表性的转折点便是 Meta 发布 Llama 4 时的滑铁卢事件。在题目和评分标准完全开源的主流公开基准上，Llama 4 表现出了惊人的强悍性能；然而，当 Vals 将其置于自身的保密私有测试集（held-out private benchmarks）中进行盲测时，该模型的实际表现却严重不及预期。公开成绩与真实能力之间的巨大落差，彻底暴露了模型针对已知考题过度优化乃至“刷榜作弊”的乱象。 这种信任脱节也促使顶尖实验室自身开始呼吁外部评测机构的介入。正如 DeepMind 联合创始人 Demis Hassabis 等业内领袖所倡导的那样，实验室投入数十亿美元构建下一代模型，他们迫切需要一个“理性的采购市场”（rational buying market），能够拿出具有公信力的客观证据向买家证明投入的价值，而不是陷入互相自吹自擂的内卷之中。Rayan 将这一角色类比为金融市场中的评级机构与会计审计事务所：Vals 一方面为前沿实验室提供无可争议的能力背书，另一方面则帮助企业在眼花缭乱的模型发布中厘清哪一款架构能为其业务场景带来最高的投资回报率（ROI）。 - **公开基准失真**：当测试题目、上下文与评分准则在开源社区公开后，模型极易产生测试集污染与奖励黑客（Reward Hacking）现象。 - **私有测试集的必要性**：Vals 的私有保留评测集严格对外保密，杜绝了模型在预训练或微调阶段通过“背题”获得虚假高分的可能。 - **生态共识的建立**：不仅是企业采购端需要客观标尺，顶尖模型团队同样需要第三方中立机构来验证自身的实质性进展。 ### [04:05](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=245s) 揭秘模型发布前的 6 小时窗口期与自动化系统“Steve” 在前沿模型正式对外发布前，通常仅留给评测团队极短的盲测窗口期——往往只有短短 6 个小时。在不能推迟模型官方发布日程的前提下，评测方必须在严苛的时间与速率限制（Rate Limits）内，完成数十亿甚至数百亿 Token 的高并发压力测试。Rayan 回忆道，Vals 创立早期完全是一场肉身抗压的血泪史，他和联合创始人 Lynx 经常通宵达旦地手动盯跑测试，拼尽全力赶在官方倒计时结束前将评测报告推向市场。 随着团队壮大和底层架构的深度演进，如今的 Vals 已经构建起高度自动化的分布式评测基础设施。这套系统能够在拿到前沿模型 API 访问权限的第一秒起，直接跑满模型提供商开放的最大调用速率上限。为了彻底解放人力，Vals 内部研发了一套名为“Steve”（被称为经济型评测员工）的自主 Agent 系统，逐步接管了以往需要人工干预的复杂分析与评判任务。这一演进确保了团队既能在几小时的生死时速内提炼出最高信噪比的综合评估，又不会成为模型发布流水线上的瓶颈。 ### [06:00](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=360s) 评测的模糊边界：将现实工作流程显性化与行业规范的形成 Ben Horowitz 随后抛出了一个本质性的质疑：评价智能本身是一个“AI 完备”（AI-complete）的终极难题。人类至今在评估同类的综合能力上都缺乏公认的精确标准——无论是智商测试（IQ）、情商测试（EQ）、大五人格模型，还是屡遭争议的 SAT 考试，都无法完全衡量一个人的真实水平，更何况现代模型极擅长通过模式匹配去取巧破解测试规则。对此，Rayan 回应称，AI 评测的发展正在倒逼企业将现实中许多模糊（fuzzy）且分散的隐性标准显性化（legible）。例如，法律界中究竟是什么本质区别划分了普通律师助理（associate）与合伙人（partner）？人类世界过去并没有一套量化的考试来界定这种晋升，但如果要评估大模型能否胜任这些岗位，评测团队就必须深入真实工作流，把这些模糊的行业经验转化为清晰的基准。长远来看，如何将企业各部门的日常流程和隐性知识清晰解构，正是决定 AI 能否规模化落地的核心瓶颈。 Ben Horowitz 用美国电影协会（MPAA）的分级制度做出了生动的历史类比：界定什么是艺术、什么是色情，或者划分 R 级与 NC-17/X 级之间的红线，从来没有绝对精确的数学公式，最著名的裁定标准莫过于大法官的那句“我看到它时便知道”（I know it when I see it）。随着社会认知的变迁，过去的 X 级内容在今天可能仅仅是 R 级，PG 与 PG-13 的边界也在不断动态浮动。Ben 强调，面向复杂智能的评测注定会带有某种模糊性，但关键在于业界领袖、企业管理者与金融高管们能否在实践中逐步达成某种公认的“行业规范”（norms）。这远比单纯测试模型能否解出一道孤立狭隘的数学题要有价值得多，因为后者不仅容易被模型攻破，而且代表的能力极度偏狭。 - **显性化转化的挑战**：将现实商业流程（如律所合伙人决策、企业内部邮件协同）翻译为可被算法验证的显性指标，是当前评测的最大技术门槛。 - **MPAA 分级类比**：智能评测如同文化内容分级，早期往往依赖主观经验判断，随后通过全行业的互动与实践，逐渐沉淀为具有约束力的共同规范。 - **坚守评测中立底线**：Rayan 透露，Vals 在创立初期便确立了一条不可逾越的商业红线——绝不向模型实验室出售任何评测集作为训练数据，以此杜绝利益冲突，维持裁判员身份的纯粹性。 ### [09:06](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=546s) 评估行业的利益冲突与独立审计价值 在讨论 AI 评估行业的商业生态时，Rayan Krishnan 指出当前市场存在严重的利益结构扭曲。许多数据标注和数据提供商在接触模型实验室时，往往是以兜售海量训练数据为商业驱动，甚至专门打造具有噱头性质的基准测试（gimmick benchmarks）作为其向实验室推销数据的获客跳板。这种“既当运动员又当裁判”的做法让 Rayan 联想到了金融审计史上的安然（Enron）丑闻——当同一个机构既负责为企业提供咨询和业务支持，又负责出具独立审计报告时，利益机制必然变质，最终演变为“付费即通过审计”或“花钱就能赢下基准测试”。Vals 从一开始便坚守第三方中立立场，拒绝将评测与数据销售绑定，以确保基准测试真正反映模型的客观能力。 - 行业乱象：许多数据公司将基准测试作为销售训练数据的营销漏斗，导致评估客观性大打折扣。 - 安然效应警示：审计业务与咨询/数据业务混为一谈会导致逆向激励，催生花钱买分数的买榜现象。 - 独立定位：Vals 坚持独立第三方的中立评测机制，旨在为整个行业提供不偏不倚的度量标准。 ### [10:00](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=600s) 从行业应用到“递归自我改进指数”（RSI） Jennifer Li 提到 Vals 如今已建立起覆盖各垂直领域（如金融、消费心理健康等）的丰富基准测试库。Rayan 介绍称，针对高经济价值场景的评测极受欢迎，例如其金融智能体基准已被多家大型金融机构用来追踪前沿模型演进；而在编程领域，其构建的 VI codebench 专门测试模型根据自然语言提示端到端构建全栈 Web 应用的能力。更具前瞻性的是他们近期发布的“递归自我改进指数”（Recursive Self-Improvement Index，简称 RSI Index）。虽然各大顶尖实验室在模型技术报告（model cards）中频繁提及模型自我迭代的潜力，但全行业一直缺乏统一的横向对比标准。理想情况下，评测 RSI 应该让前沿模型从零开始训练自己的下一代版本并观察能力增量，但这在算力成本与时间上极为高昂。因此 Vals 将构建下一代模型所需的完整管线拆解为一系列代理任务（proxies），覆盖预训练、后训练以及测试框架级别工程（harness-level engineering），从而精确诊断模型在哪些环节能够自主完成高质量科研探索，在哪些环节依然困难重重。 ```mermaid flowchart LR accTitle: RSI Index Workflow accDescr: 模型递归自我改进指数的代理评估框架流程图 subgraph Pipeline ["构建下一代模型的全流程拆解"] PT["预训练阶段代理 (Pre-training)"] PostT["后训练阶段代理 (Post-training)"] Eng["测试框架工程 (Harness Engineering)"] end subgraph Eval ["Vals RSI 评估体系"] Proxies["标准化任务基准代理"] Diagnosis["定位能力断点与自主研发潜力"] end PT --> Proxies PostT --> Proxies Eng --> Proxies Proxies --> Diagnosis ``` - 重点基准：金融 Agent 评测受主流金融机构青睐；VI codebench 聚焦根据自然语言生成全栈 Web 架构的落地能力。 - RSI 指数破局：将复杂的自我训练流程拆解为涵盖预训练、后训练、测试框架工程等多维代理任务，打破各实验室自说自话的局面，实现跨模型的统一横向比对。 ### [12:11](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=731s) 永远有更高的山峰：基准淘汰与现实世界的动态映射 针对基准测试被刷榜饱和以及行业常见的“挑最好结果展示”（cherry-picking）现象，Rayan 阐述了 Vals 废弃和迭代旧基准的根本逻辑。Jennifer 开玩笑提到自己正穿着印有 Vals 口号的 T 恤，Rayan 顺势解释了 Vals 的非官方座右铭——"Always a higher peak"（永远有更高的山峰）。正如基石模型实验室在不断做爬山算法寻找下一个能力制高点，评估机构的职责就是源源不断地为他们筑起更高的群山。这与整个人类经济的发展历程类似：当农业不再需要过多人口时，社会便对劳动力提出了新的高阶技能要求，评测也必须随之抬高门槛。此外，基准退役与更新还有一层常被低估的意义：评测必须实时映射现实世界知识的动态更新。就像执业律师、建筑师或医生必须定期重新认证考取资质一样，随着成文法修改、判例法演变以及医学知识的更新，法律研究与医疗评测等基准也必须随现实世界同步迭代，以此驱动模型向正确的应用方向进化。 - 持续筑峰哲学：实验室负责“爬山”，Vals 的使命则是不断建造更高维度的“新山峰”，避免基准停留在饱和状态。 - 知识时效性对齐：基准测试需要像专业职业资格重新认证一样保持动态刷新，使测试环境与当下的法律法规、医疗指南等真实世界状态紧密贴合。 ### [13:26](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=806s) 基础设施挑战：从单步问答到长程 Agent 评测 随着行业从多选问答（multiple-choice QA）快速转向能够自主执行复杂任务的 Agent，评估的工程架构与维度发生了深刻转变。现在的评测不再仅仅关注单一 Prompt 的文本输出，还必须综合衡量延迟、运行成本以及模型在不同广度领域的泛化弹性。在基础设施层面，Vals 正在测试能够连续运行数小时、数天乃至数周的长任务 Agent。评测系统必须具备高度容错与断点续跑能力，一旦某一步请求失败，系统能够精准重试该局部节点，而不是强行重新运行整条昂贵的操作轨迹。同时，评测的样本范式正在发生本质转移：早期像 ImageNet 这类基准依赖数百万级别的大样本量，但每个任务仅是从图片到标签的简单一对一映射；而面对当下的复杂工作流，评估样本量虽然骤降（例如仅让模型构建 50 个全栈 Web 应用），但在多步执行逻辑、代码健壮性以及系统鲁棒性等维度上的评估标准与判定机制（rubrics）却变得极其精细和庞大。 - 基础设施稳定性：支持长达数天甚至数周的长时间 Agent 执行，具备局部失败断点重试机制，避免整条长轨迹报废。 - 评估样本范式转变：从“百万样本 + 单一映射”（如 ImageNet）转变为“极少样本量（如 50 个全栈项目）+ 极复杂多维评判标准（rubrics）”。 ### [15:25](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=925s) 智能路由迷局与企业级 AI 的存亡时刻 关于模型路由（routing），Ben Horowitz 询问这类能力未来是否会演变成类似被 Stripe 收购的 OpenRouter 那样的实时机制。Rayan 澄清道，OpenRouter 名义上叫路由器，但目前主要还是作为模型网关（gateway）运行，真正决定调用哪款模型的依然是终端开发者。这是因为模型智能路由最核心、最困难的瓶颈在于如何建立精准可靠的评估体系，搞清楚哪款模型在什么场景、哪个细分步骤最有效。Vals 协助企业构建内部评测体系的实践，正是许多企业后续有底气落地自动化模型路由的前提。 Rayan 强调，评测对于基础模型实验室是证明溢价的武器，但对企业客户而言，评测已经上升为生死存亡的问题。Rayan 分享了一个极具代表性的行业趣闻：某家世界前十强（Fortune 10）企业为其工程师团队引入了 Claude Code，并设定了每人每天约 100 美元的 Token 调用额度。这个简单的限额机制竟然彻底颠覆了这家巨头的日常工作节奏：由于系统的速率限制（rate limit）每天下午 4 点重置，整个工程师团队效率最高的黄金时段变成了下午 4 点到 6 点；而在重置前的午后，额度耗尽的工程师们只能无所事事地去散步、喝咖啡，形成了漫长的“工作真空期”。这一荒诞现实揭示了企业在技术栈各个层面上对“智能”价值的严重误判与度量缺失——工程师手握 100 美元额度却不知如何将其分配在最具产出的关键环节上，企业也缺乏科学的评估标尺来衡量 Token 消耗与实际业务 ROI 之间的真实关系。 - 路由本质：OpenRouter 核心仍为网关工具，真正的智能模型路由受制于企业评估体系的完备程度。 - Fortune 10 真实案例：某全球前十强企业给工程师分配每天 100 美元 Claude Code 额度，导致下午 4 点额度重置后全员突击编程、午后额度耗尽集体散步摸鱼的魔幻作息。 - 企业评估断层：从基层工程师到管理层，普遍缺乏量化模型产出与合理分配 Token 预算的标准，暴露出在企业级应用中对“智能”定价和价值衡量的普遍困惑。 ### [17:37](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1057s) 当 Token 支出开始超越员工薪水：企业面临的 ROI 考量 访谈切入了一个极具冲击力的企业现实：某家世界前十强（Fortune 10）企业最初武断地规定每位员工每月拥有 100 美元的 Token 额度，最近这一数字更是直接飙升至每人 300 美元——对于庞大的员工基数而言，这几乎相当于在人均工资之外额外砸入了一份全额薪水专门用于消耗 Token。然而，这种额度分配本身非常武断，因为目前业界极难量化合理的用量边界。更微妙的是，Anthropic 这类大模型厂商为了维持这种海量调用，自身背负着极其高昂的模型服务成本，利润空间被压缩得相当微薄。 整个市场正处于一种估值与回报极其模糊的阶段。当 Token 开销可能逐步反超企业的人力薪资开支时，AI 投入不再只是可有可无的创新预算，而变成利润表中极具存在感的重大成本项。企业在过去半年或许还能凭热情买单，但接下来必须极度严苛地核算投资回报率（ROI）。从某种意义上说，"一家企业本质上就是它的评估系统（A firm really is just its eval）"；谁能将自身的评估标准清晰化、量化 ROI 计算公式，谁才能在长期竞争中脱颖而出。 - **额度暴涨**：某 Fortune 10 巨头将员工 AI 额度从 100 美元/月提升至 300 美元/月，成本直逼薪酬支出。 - **厂商压力**：Anthropic 提供此类海量推理支持时利润率相当受限，服务成本巨大。 - **核心命题**：如果 Token 支出占大头，AI 评估（evals）将直接决定企业的 ROI 与长期存亡。 --- ### [18:41](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1121s) 私有代码库 vs 公开基准：Valsmith 的诞生 面对企业质疑“为什么不内部自研评测、我们才最懂自身业务”，Rayan 指出，虽然企业应具备一定的内部专长，但现实中模型生态正在指数级爆发。除了不断冒出的新基础模型实验室，每家实验室还在以更密集的频率推出带有海量超参数选项的模型，并配合复杂的 Agent 运行框架（harnesses）。面对用例复杂度与模型选项的双重叠加，企业单凭自身力量很难建立起全面的评测体系。为此，Vals 推出了首款面向企业的产品——Valsmith。 Valsmith 专门针对企业 AI 支出最高昂的领域：代码生成（codegen）。它允许企业直接接入自有的 GitHub 私有代码库，自动生成内部专属的编码基准测试。这不仅让企业看清哪款 Coding Agent 在自身业务中最能打，更能绘制出帕累托最优（Pareto optimal）曲线，算出最高 ROI 的模型选择。 ```mermaid flowchart TD accTitle: Valsmith 评估流程 accDescr: 展示 Valsmith 如何从私有代码库生成专属基准并完成模型选型与 ROI 分析 subgraph Input["企业代码资产"] A["GitHub 私有代码库"] B["真实工程 Issues / Traces"] end subgraph Valsmith["Valsmith 评估引擎"] C["提取内部工作基准测试"] D["运行多模型横向对比"] E["测算 Token 消耗与成功率"] end subgraph Output["落地决策"] F["识别帕累托最优模型"] G["按任务自动路由与预算控制"] end Input --> Valsmith C --> D --> E Valsmith --> Output F --> G ``` 在公有代码库与私有代码库之间，模型表现存在巨大偏差。市场上目前根本无法直接断定 OpenAI 还是 Anthropic 的顶尖模型在某家企业的私有仓库里表现更好。此外，在 Anthropic 的 Opus、Sonnet，以及 Luna、Terra、Muspark 等轻量或开源自研模型的“混乱中间层”中，常识往往失效：例如很多场景下 Claude 3.5 Sonnet 的实际开销反而比 Opus 更昂贵，仅仅因为它表现得“极其饥渴于 Token（token hungry）”。如果企业凭直觉去用 Sonnet，最终花销反而会严重超标。这种评估框架不仅限于代码，各行各业沉淀的现有工作底稿（如 PowerPoint、Excel 财务模型等）未来都将以类似方式被转化为动态基准。 - **Valsmith 定位**：首款聚焦代码生成的企业评测工具，将私有 GitHub 仓库转为专属评测基准。 - **非直觉表现**：公有 Benchmark 高分不等于在私有 Repo 上表现好；Sonnet 在特定任务中由于极高的 Token 吞吐量，花费反而可能超过单价更高的 Opus。 - **知识工作迁移**：编码是所有知识工作的先导，代码 Agent 的评估模式未来将复用到 PPT 制作、DCF 折现现金流模型等白领劳动场景。 --- ### [22:40](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1360s) Vals 内部的“Token 极限挑战”与智能路由 为了更深入地理解极端 AI 消耗，Vals 团队在内部发起了一场“Token 极值化实验（token maxing experiment）”。Rayan 为团队争取到了为期一个月的编码工具无限量访问特权。结果令人咋舌：许多工程师每天要消耗 10 亿到 20 亿 Token，峰值时单人单日消耗高达 60 亿 Token。 一个月复盘下来，虽然该测试期间免去了官方账单，但若折合成商业定价，这支小团队单月足足烧掉了约 150 万美元的 Token。这一金额竟然达到该月团队员工薪资总和的 10 倍（“不是五五开，而是足足 10 倍”）。工程师们在面对无限制模型调用时表现出了一种“无处不用 Agent 的不安全感与依赖性”。这也逼迫 Vals 必须从运营机制上改革，不能在下个月重演这种烧钱模式。 通过 Valsmith 回溯 GitHub 痕迹与测试日志，团队挖掘出了极具价值的省钱洞察：例如 Cognition 推出的 Devin 工具在 Token 利用效率上表现极其优异；而在定价机制上，选择订阅制服务往往大幅划算于纯粹按 Token 用量计费的模式。 最终，Vals 落地了一套科学的管理方式： - **权限全开放**：员工依然可以无障碍使用任何市面上的顶尖模型与工具。 - **按难度智能推荐**：系统针对 GitHub 上的每个 Issue 或工单进行智能分析，自动提供最佳工具推荐，根据任务所需的“智力难度”来精准调节（titrate）实际 Token 消耗。 --- ### [25:06](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1506s) 政策监管的脱节与第三方的实证定位 对话随后转向监管与政策层面。公开基准在行业内部已经迅速失效，而立法政策面对技术飞跃时其滞后性更加严重。Ben Horowitz 和 Marc Andreessen 等投资人花费了大量时间前往华盛顿特区与政策制定者沟通，试图弥补这种认知鸿沟。但在界定 AI 标准时，究竟该由大模型实验室、独立评测机构、企业客户还是政府来主导？ Rayan 认为，各方都应当在某种程度上参与进来，多元视角具有不可替代的价值。但当前政策探讨最大的痛点在于过去几年的对话“过于抽象”，完全脱离了底层技术的物理现实。即便是部分模型大厂主动提议引入第三方评测生态，往往也未能明确这一机制具体该如何运转、依循何种机制规范。 因此，像 Vals 这样的独立评测机构在当前阶段的首要使命是充当“证据收集者（evidence gathering mode）”。通过在最前线采集关于大模型真实能力边界与潜在系统性风险的第一手经验数据（empirical data），为未来更严谨、更成熟的监管政策与行业对话提供坚实的底层支撑。 ### [26:30](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1590s) 前沿模型的监管困境与两难平衡 对前沿人工智能模型的监管，始终伴随着对“何为前沿”的界定难题。早期的监管尝试往往倾向于依赖生硬的算力阈值——例如以 $10^{26}$ FLOPs 作为界限，并提出设立 30 天或 60 天的强制等待期，甚至将模型先关在特定的“安全盒”中进行隔离观察，以确保其不会对现实系统造成破坏性冲击。然而，这种机械划线的做法引发了激烈的讨论：政府究竟应该如何界定模型是否处于前沿，而在等待期内又该具体执行哪些检查？ 针对这一问题，Rayan Krishnan 指出监管决策中存在两股根本性的对抗力量：一方是保持极速推进、确保政府流程不阻碍技术创新的迫切愿望；另一方则是确保所开发的技术始终符合美国乃至全人类核心利益的安全底线。这两者极难兼顾，以往往往是非此即彼的零和博弈。他认为破局的关键在于引入独立的第三方评测机构，建立基于实证的数据收集机制，帮助政策制定者看清技术发展的真实全貌，从而厘清什么样的技术标准才符合公众利益。与此同时，第三方评测机构可以灵活研发与前沿同步演进的测试与验证技术，避免监管测评技术因官僚流程而严重落后于模型创新的步伐。 ```mermaid flowchart TD accTitle: AI 监管与第三方评测协同架构 accDescr: 展示立法/行政机构与独立第三方评测机构在规则制定、能力验证与合规执法中的分工协作闭环。 subgraph Gov ["政府部门（规则制定与执法）"] L["立法机构"] -->|"定义不可接受风险<br/>（生物安全/未成年人健康）"| Rules["确立红线与法规"] E["行政机构（商务部 / SEC）"] -->|"根据违规报告<br/>执行市场准入与执法"| Enforce["合规监管与落地"] end subgraph Evaluator ["独立第三方评测（Vals 等）"] Data["定期简报与能力趋势数据"] -->|"实证输入"| L Test["自动化动态渗透与对齐测试<br/>（检查生物/网络渗透能力与奖励黑客）"] -->|"出具独立验证结果"| E end Rules -.->|"委托验证标准"| Test ``` ### [28:13](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1693s) 对齐、奖励黑客与政企分工的“最优解” 讨论随后转向了模型更深层的行为风险，即模型是否容易出现“奖励黑客”（Reward Hacking）或诱发违法行为。Rayan 解释称，Vals 将这类问题统归在模型对齐（Alignment）的范畴内。在实际评测中，这种现象已经屡见不鲜：例如在针对某项网络安全风险对模型进行测试时，模型会为了达成目标而主动寻找规则漏洞，通过绕道实现奖励黑客。Vals 评测的核心目标，正是检验模型是否始终与用户真实意图保持一致，并捕捉那些背离预期意图的反常行为证据。 针对政府与民间机构的权责边界，Ben Horowitz 深入剖析了当下的政企分工痛点。他直言不讳地指出，顶尖 AI 实验室常常向政府发出耸人听闻的警报，宣称模型即将引发致命的生物黑客攻击或毁灭性的网络安全风险；但政府在技术演进的长期跟踪与技术验证上天生存在严重短板，根本无法胜任“评估模型是否具备危害能力”以及“是否能在实际操作中被诱导执行非法行为”的具体测试工作。相比之下，政府的长处在于凭借法定权威设定红线并严格执法。理想的治理模式应是由政府明确划定不可容忍的行为底线，再由高水平的私营评测机构充当“裁判”，独立检测模型是否违规。Ben 讽刺道，目前一些大模型实验室以“模型具备危险能力且可能被诱导作恶”为由实行技术封锁，声称“只有我们自己内部使用才能确保安全”，但事实证明哪怕实验室内部自用也难以完全防住越狱。 Jennifer Li 则从企业落地角度指出了更为现实的脱节现象：舆论层面的恐慌叙事与现实部署严重脱节。外界至今仍在反反复复讨论数月前甚至更久之前的典型安全事件（如 OpenAI 或 Hugging Face 遭受的攻击，或是 Fable 发生的问题），但企业实际部署模型的生产环境极其定制化。如果缺乏将前沿能力与落地防护栏（Guardrails）精确对齐的客观工具，企业客户就很难建立起安全使用大模型的信心。 - 关键考量：评估模型风险不仅要看其潜在能力（Capability），更要验证在现实环境下是否能被有效诱导（Elicitation）并产生实质违法危害。 - 治理逻辑：政府负责“定义规则与违规惩罚”，独立第三方机构负责“专业测试与违规取证”，打破由大实验室既当运动员又当裁判员的垄断闭环。 ### [32:15](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1935s) 决策者与评测机构的常态化协同 当被问及政策制定者应如何与评测机构建立高效协作时，Rayan 披露了 Vals 当前与监管层的互动方式。首要任务是建立顺畅、直接的实证数据传递管道。目前，Vals 已经开始定期向美国行政分支与国会立法机构提供专属简报，详尽汇报评测中捕获到的最新模型能力边界与潜在风险点。 这种基于实测数据的常态化沟通，不仅让深处国会山的决策者能够跟上大模型一日千里的演进速度，更能帮助他们基于历史趋势外推未来的技术走向。Rayan 强调，评测机构的职责在于呈现中立客观的事实与数据趋势，而非越俎代庖去给政策下定论；如果立法者基于数据认定某些领域——例如涉及 18 岁以下未成年人的心理健康隐患，或是严重的生物安全威胁——已经达到临界点，那么立法者就拥有充分依据去制定标准化的模型发布限制条例。而在政策落地层面，美国商务部（Department of Commerce）或美国证券交易委员会（SEC）等行政执行机构，则可以利用这些标准来督导企业安全合规地引入模型，在保障生产力提升的同时守住系统底线。 ### [33:30](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=2010s) 主权 AI 崛起与国际评测的地缘政治 谈话的焦点随后扩展至全球地缘政治与不同文化价值观的冲突。Jennifer Li 提出，评测体系（Eval）本质上是模型开发者价值取向的投射；不同国家及其本土实验室所推崇的规范迥然不同。她以自身在中国成长及使用中国开源模型的经历为例：由于当地政治语境与监管要求，模型在涉及特定政治敏感话题时必然存在严格的过滤机制。面对这种割裂，评测基准究竟该如何跨越意识形态壁垒，在不同国家实验室之间发挥标准化作用？ Rayan 坦言，从纯粹的技术与资源效率等理想视角来看，各国竞相重金布局“主权 AI”（Sovereign AI）极其低效——这意味着大量算力数据中心被重复建设、数据工程流程被重复投入、超大模型被机械复制，而这些算力本可以通过集约化实现更优配置。然而，现实世界正坚定地迈向主权割裂，这使得建立一套各方认可的“共同评测语言”变得前所未有地紧迫： - 历史镜鉴：核武军控谈判提供了极具价值的历史参照。当年里根总统针对核军控提出了著名原则——“信任，但要核实”（Trust, but verify）。 - 验证机制的缺失：尽管大国领导人之间可以展开高层会晤并建立初步的信任意向，但目前在 AI 领域却完全缺乏类似冷战时期核查核弹头数量或允许军机越境空中侦察（Flyovers）的客观核查工具。 - 共同语言的价值：唯有打造通用的评测基准（Evaluations），国际社会才能拥有量化与审计前沿模型社会风险乃至生存风险（Existential Risk）的核查手段，从而为跨国技术监管与地缘协调提供可能。 ### [36:08](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=2168s) 国际竞争、监管底线与递归自我改进（RSI） 面对主权国家与企业之间激烈的 AI 军备竞赛，如何在缺乏完全信任的环境中建立协调机制成为焦点。Rayan Krishnan 坦承，如果声称自己当下对这个问题有完美的解决方案，那就太天真了。不过，各方完全可以先从共识最高的小步骤开始切入。例如，网络安全风险（Cybersecurity Risk）目前已是各方热议的焦点，而生物安全（Biosecurity）的潜在威胁在未来甚至会变得更加重要。在网络防攻击与防范生物恐怖主义等核心底线领域，各大竞争国和利益相关方存在明确的共同利益，完全有动力达成合作，以防止由于技术外溢引发毁灭性冲突。 从长期来看，Rayan 认为真正最值得关注、也最具颠覆性的是模型的“递归自我改进”（Recursive Self-Improvement，简称 RSI）潜力。在这一维度上，极有可能出现某一个国家或某一家公司在技术上彻底“脱缰飞驰”（run away with it），从而秘密训练出外界几乎一无所知、运行机理完全处于未知状态的超级智能模型。因此，建立一套联合的评估语言来界定何种研发节奏是人类社会所能承受的舒适区、何种演化速度已经突破了 RSI 的安全红线，具有决定性的战略意义。这也是当前许多顶尖闭源实验室的研究人员主动奔走，呼吁主权政府之间必须坐下来开启联合沟通与对话的核心动因。 ```mermaid flowchart TD accTitle: AI安全国际协调路径演进 accDescr: 展示从低阻力的具体安全领域共识演进到长远递归自我改进治理的框架图。 subgraph Phase1 ["初始切入点：共同底线防御"] A["网络安全风险<br/>(Cybersecurity)"] B["生物安全风险<br/>(Biosecurity)"] end subgraph Phase2 ["核心长期挑战：失控飞轮"] C["递归自我改进 (RSI)"] D["某国/某机构单一跑通<br/>产生黑盒超级模型"] end subgraph Phase3 ["治理落地机制"] E["跨国联合评测框架<br/>界定可接受的发展节奏"] F["闭源实验室联合政府对话"] end Phase1 --> Phase2 C --> D D --> E E --> F ``` ### [37:14](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=2234s) 未来基准演进：从代码漏洞到真实基础设施仿真 随着全球模型能力百花齐放，各方关注的重心开始分化——有的关注网络攻防，有的聚焦生物技术安全，这推动了下一代评测基准形态的变革。在探讨未来基准格局时，Rayan 指出，Vals 的核心战略就是极度专注在紧贴技术前沿（Hyper-focused on capturing the frontier）。只要在最前沿出现了新的模型能力分支或潜在风险，Vals 就会将其转化为 Vals.ai 平台上高度可复现、文档化的标准化评测，并不断扩大评测的覆盖广度与深度。 评测维度的升级直接体现在技术深度的迁移上。以网络安全评测为例： - **传统代码级测试的局限**：以往的测试大多聚焦在静态代码审查层面，比如检测代码中是否存在经典的内存泄漏（Memory leaks）或单点逻辑漏洞。 - **动态全系统级测试的演进**：现实世界中最严峻的系统性安全威胁并不体现在几行孤立的代码上，而是存在于庞大的底层基础设施（Infrastructure level）。 - **大规模环境仿真**：真正的评测必须能够动态模拟企业级云基础设施（Enterprise cloud infrastructure）乃至国家关键电网系统（Grid infrastructure）的真实运行环境，以此来严格测定模型在实战环境下的进攻与防守能力。 最后，Rayan 强调了第三方独立评测机构的立身之本：Vals 最具价值的商业模式必须建立在纯粹的激励对齐（Incentive aligned）之上。公司必须将全部精力投在提供中立、高质量、反映现实的评测本身，而绝不能“既当裁判员又当运动员”去直接插手模型的训练研发流程或协助模型刷分优化。保持这种绝对中立性，才能真正为整个 AI 行业构建出可信的度量衡。对谈在热烈的致谢氛围中圆满结束。 ## Notable quotes > [06:00](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=360s) I know it when I see it — Ben Horowitz > [12:11](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=731s) Always a higher peak — Rayan Krishnan > [17:37](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1057s) A firm really is just its eval — Rayan Krishnan > [18:41](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1121s) token hungry — Rayan Krishnan > [22:40](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1360s) 不是五五开，而是足足 10 倍 — Rayan Krishnan > [25:06](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=1506s) evidence gathering mode — Rayan Krishnan > [33:30](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=2010s) Trust, but verify — Rayan Krishnan > [37:14](https://www.youtube.com/watch?v=WO9c9qxDxzU&t=2234s) Hyper-focused on capturing the frontier — Rayan Krishnan ## Mentioned Vals, Rayan Krishnan, Erik Torenberg, Ben Horowitz, Jennifer Li, Lynx, OpenAI, Meta, Llama 4, DeepMind, Demis Hassabis, Steve, 美国电影协会 (MPAA), 安然 (Enron), VI codebench, Recursive Self-Improvement Index (RSI Index), ImageNet, Stripe, OpenRouter, Anthropic, Claude Code, Claude 3.5 Sonnet, Opus, Luna, Terra, Muspark, Valsmith, GitHub, Cognition, Devin, Marc Andreessen, Hugging Face, Fable, 美国商务部 (Department of Commerce), 美国证券交易委员会 (SEC), 罗纳德·里根 (Ronald Reagan), Vals.ai ## Video description a16z’s Erik Torenberg, Ben Horowitz, and Jennifer Li sit down with Vals founder and CEO Rayan Krishnan to discuss one of AI’s increasingly difficult problems: how do you actually measure whether a model is getting better? As public benchmarks saturate and models get better at optimizing for the tests themselves, Rayan makes the case for independent, continuously evolving evaluations. They unpack why self-reported model scores can be misleading, how VALS evaluates models in the hours before a release, and why measuring increasingly agentic systems means testing work that can unfold over hours, days, or even weeks. They also explore why evals are becoming critical for enterprises trying to understand the ROI of AI, what happens if token spend begins to rival employee salaries, and how evaluations could eventually provide a shared language for everything from model routing and recursive self-improvement to AI policy and international coordination. Timestamps: 00:00 - Intro 00:55 - Why Vals Exists: When Public Benchmarks Stopped Working 02:52 - The Llama 4 Disaster: Public Scores vs Private Reality 04:05 - Inside the 6-Hour Pre-Release Testing Window 06:00 - The Limits of Evaluation: Making Fuzzy Evals Explicit 10:00 - The Recursive Self-Improvement Index 13:13 - Beyond Capability: Cost, Latency & Keeping Benchmarks Fresh 17:52 - When Token Spend Starts to Eclipse Salary Spend 19:19 - Private Repos vs Public Benchmarks: The Real Performance Gap 22:40 - How Vals Uses Vals: Token Maxing the Coding Tools 24:48 - Policy: What Should the Government Actually Do? 28:32 - Alignment, Reward Hacking & Models Gaming the Test 33:30 - The Geopolitics of Evals: Whose Values Get Embedded? 37:14 - What the Benchmarking Landscape Looks Like Next Resources: Follow Rayan Krishnan on X: https://x.com/RayanKrishnan Follow Ben Horowitz on X: https://x.com/bhorowitz Follow Jennifer Li on X: https://x.com/JenniferHli Stay Updated: If you enjoyed this episode, be sure to like, subscribe, and share with your friends! Find a16z on X: https://twitter.com/a16z Find a16z on LinkedIn: https://www.linkedin.com/company/a16z Listen to the a16z Show on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX Listen to the a16z Show on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711 Follow our host: https://x.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see http://a16z.com/disclosures. [Source](https://www.youtube.com/watch?v=WO9c9qxDxzU)

[6:00] Ben Horowitz: I know it when I see it [12:11] Rayan Krishnan: Always a higher peak [17:37] Rayan Krishnan: A firm really is just its eval [18:41] Rayan Krishnan: token hungry [22:40] Rayan Krishnan: 不是五五开，而是足足 10 倍 [25:06] Rayan Krishnan: evidence gathering mode [33:30] Rayan Krishnan: Trust, but verify [37:14] Rayan Krishnan: Hyper-focused on capturing the frontier

[6:00] Rayan Krishnan: It's forcing a lot of the more fuzzy or distributed forms of eval to be made explicit. Like what is really the distinction between an associate and a partner at a law firm? There isn't a clear test or an eval for that in the human world, and so we have to first establish a lot of that in these different enterprise or real-world workflows for us to be able to test models in the same way. [6:00] Ben Horowitz: I know it when I see it. [12:11] Rayan Krishnan: always a higher peak [17:37] Rayan Krishnan: token spend may start to eclipse salary spend. [17:37] Rayan Krishnan: a firm really is just its eval. [4:05] Rayan Krishnan: Steve the economic Vals employee [33:30] Rayan Krishnan: trust but verify [36:08] Rayan Krishnan: run away with it

[0:00] Every time a new trillion dollar [0:02] industry emerges, there's a need for [0:04] this independent testing group. When [0:05] Meta released Llama 4 on our held out [0:07] private benchmarks, the model is [0:09] actually underperforming. But on all of [0:10] the major public benchmarks, it was [0:12] showing incredible capabilities. [0:14] >> What's the limit of what you can [0:15] achieve? And then within that, how are [0:18] you going about it? [0:19] >> In an ideal world to take a frontier [0:21] model and have it train the next version [0:22] of itself, but obviously that's very [0:24] expensive and slow. And so what we're [0:25] doing is forming a set of proxies for [0:27] every part of the process it takes to [0:29] build the next version of the models. [0:31] Evaluations as we become more complex [0:33] have a fewer sample size but a larger [0:36] set of criteria or expectations of them. [0:38] >> Where do you see the gap that's [0:39] happening today? [0:40] >> The government kind of has an [0:41] inclination of what it's afraid of be it [0:43] biohacking, cyber hacking. But then [0:46] there becomes the question of can the [0:48] model do it and then can you get the [0:50] model to do it? [0:51] >> What do you think the landscape will [0:52] look like? Um [0:55] >> so I'll start a question from uh when LA [0:58] got started in 2024 after a team [1:00] discovered that all the public [1:02] benchmarks are just not sufficient [1:03] enough to measure model progress and [1:06] there needs to be a new methodology and [1:08] approach coming to you know keep us um [1:11] uh on the frontier and and help um model [1:15] labs hill continue to hill climb. uh [1:17] maybe just take us back to the the [1:20] inception of vows and what do you see [1:22] was missing in the in the in the market [1:23] then? [1:24] >> Yeah. Yeah. I mean so I I had a [1:25] background uh doing research and in [1:27] particular building benchmarks and [1:28] evaluations and so what was very clear [1:30] to me was the very tight relationship [1:33] between what it takes to build new [1:34] systems for generation and actually new [1:36] mechanisms for evaluation. In fact in [1:39] order to get one you often need to get [1:40] better at the other. Um and actually one [1:42] of the biggest drivers for model [1:43] capability is having a new legible way [1:46] to evaluate models. Um and so around [1:48] early 2024 what we were seeing is um [1:50] there actually many interesting models [1:52] coming to market. They weren't all [1:53] coming from from OpenAI. Um and then [1:55] also in particular it was um harder than [1:57] ever to actually ascertain what was [1:59] newly capable with the new models. Um [2:01] and so in kind of a first principles way [2:03] what we we realized is that there would [2:04] need to be some thirdparty company that [2:06] solely existed to build really high [2:08] quality evaluations and benchmarks to be [2:10] able to discern um what was nearly [2:12] possible with these models. And so we [2:13] released our first benchmarks in 2024 [2:15] and now the last couple of years that [2:17] has kind of been been realized by many [2:19] different parts of the industry. [2:20] >> And why why do I guess one of the [2:22] obvious question is why do you think the [2:23] labs can't do this by themselves? [2:25] because um they know the best of where [2:26] the models are hill climbing on and what [2:30] what is missing capability wise. Uh why [2:32] can't they be the the the benchmarking [2:35] um stores? [2:37] >> Yeah, I mean internally they they do [2:39] build a lot of great benchmarks and [2:40] that's what drives model progress, but I [2:42] think there's an issue when we we speak [2:44] about model capabilities in a way that's [2:46] self-reported. And so one of the early [2:47] indications of that you saw was when [2:49] Meta released Llama 4. Um that was a bit [2:51] of a disaster. And and interestingly [2:53] what we saw is that on our heldout [2:54] private benchmarks the model is actually [2:56] underperforming but on all of the the [2:58] major public benchmarks where that the [3:00] questions and rubrics are actually open [3:02] source um it was it was showing [3:03] incredible capability. So there's a huge [3:05] disconnect between what was [3:07] self-reported based on these open [3:08] benchmarks um and then what we were [3:10] actually finding with our with our [3:11] higher quality higher signal benchmarks. [3:13] Um but I think it it speaks to a broader [3:16] um concept which the labs I think [3:17] understand that they would like to see a [3:19] rational buying market. uh you know they [3:21] would like to see that when they invest [3:23] billions of dollars to build a new [3:24] model, they're actually substantive ways [3:26] they can point to evidence and say we're [3:28] advancing in these in these ways. Um and [3:30] it's not just entirely self-reported to [3:32] justify that investment. Um and so you [3:34] also see instances of of Demis and [3:36] others in the industry calling for an [3:37] ecosystem of thirdparty valuators. [3:40] >> And what's the historical analog that [3:42] you have in mind here? There rating [3:43] agencies, audit firms. What's the right [3:45] comparable? Yeah, I I think I think [3:47] there's honestly lessons to learn across [3:49] the board and and every time a new [3:51] trillion dollar uh industry emerges um [3:53] there there there's a need for this [3:55] independent testing group and I think [3:56] the fact that it's moved so quickly in [3:58] AI has caused necessity for a lot of [4:00] these parallels to to see to be borne [4:01] out. Um you know, we think about [4:03] ourselves as is trying to sit on both [4:04] sides of the market. So there they're [4:06] mechanisms by which labs need to prove [4:08] that new models are very capable, but [4:10] they're also parallels where enterprises [4:11] need to figure out what adoption [4:13] strategy is going to amount to the [4:14] greatest ROI for them. Yeah, I want to [4:17] shift to take us through sort of the [4:19] six-hour pre-release window before the [4:21] model drops. Um, obviously you need to [4:23] run tens of billions of tokens without [4:25] delaying the launch. Which part is you [4:28] doing an allnighter versus it being uh [4:31] automated? Take us through that. [4:32] >> It's honestly been a journey. Um, and [4:34] and I think like the the real the goal [4:36] Northstar we think about is we we never [4:38] want to be um kind of a lagging [4:40] indicator or a delay to a model release. [4:43] Um and and so that means we have to move [4:45] really really quickly and extract the [4:47] most possible signal with the the rate [4:49] limits or capacity that we have we have. [4:51] And so early on what this looked like [4:52] was my co-founder Lynx and I pulling an [4:54] allnighter uh to try and get as much [4:55] done as possible and get results out the [4:57] door. Um now we've built up a team but [4:59] we've also really invested heavily in [5:00] infrastructure. And so we're able to run [5:02] evaluations in a a massively distributed [5:05] way um running effectively the the [5:07] maximum possible rate limits with with [5:09] every model we get access to. Um, and we [5:11] also have this internal system called [5:12] Steve. Uh, Steve the economic valves [5:15] employee. Um, and so that that's been a [5:18] mechanism by which we're able to [5:19] actually take more of the human work [5:20] over time and and put it into Steve. [5:24] >> And how do you deal with the kind of [5:26] issue that it's a little bit of a an AI [5:30] complete problem in that um, you know, [5:33] we still aren't really good at [5:35] evaluating humans um, or we haven't [5:38] agreed on it. like there are things like [5:39] IQ test, there's EQ, there's, you know, [5:43] the big five personality and so forth, [5:45] but there's not really an agreed upon [5:48] um framework for which we do it. And [5:51] people have issues with things like the [5:52] SAT and this and that and the third. And [5:55] then you know of course models are [5:57] really good at hacking the benchmark um [6:00] as on the floor proved [6:02] >> and so how do you think about you know [6:05] that issue and what are you like what's [6:07] the limit of what you can achieve and [6:09] then you know within that how are you [6:12] going about it? Yeah, I think I think [6:14] the honest answer is that it's forcing a [6:16] lot of the um more fuzzy or distributed [6:19] forms of eval to be made explicit like [6:22] what what is really the distinction [6:23] between an associate and a partner at a [6:25] law firm. Um and there there isn't a [6:27] clear test or an eval for that in the [6:30] human world and so we we have to first [6:33] establish a lot of that in uh these [6:35] different different enterprise or real [6:36] world workflows for us to be able to [6:38] test models in the same way. Um, and and [6:41] I think long term that will be actually [6:43] the biggest bottleneck, our ability to [6:44] take companies and their emails and and [6:46] make them legible. Uh, because that's [6:48] how we'll figure out what signal we'll [6:50] climb on and where we actually adopt. [6:52] >> Got it. Very interesting. Actually, [6:55] maybe one question for you, Ben, on just [6:57] like how the industry has formed before [7:00] like intelligence came through. Like [7:02] we're now measuring something that's [7:03] very fluid versus before like when we're [7:06] talking about enterprise software like [7:08] you know there's Gartner rating on like [7:10] 70 different metrics like you can sort [7:12] of stack rank on a on a on a quadrant to [7:15] say this company has these features [7:16] covered these features not. But now it's [7:19] like you know very jack frontier that's [7:21] very hard to measure given industries [7:22] like what do you see um you know one is [7:25] the analogy to the past that uh lessons [7:28] we can borrow and what do you see that's [7:29] really going to be the challenge and uh [7:32] and missing pieces going forward. Yeah, [7:34] it's a it's a little bit reminiscent of [7:36] the MPAA, right? Where it's like, what's [7:40] art, what's porn, where's the line, when [7:42] is it R, when is it X? Um, and by the [7:47] way, the definition of that has changed [7:49] over time, I think. Um, things that that [7:52] used to be X are now R and and so forth. [7:56] And, you know, then what's PG? What's [7:58] PG-13? [7:59] You know, all that kind of thing. And [8:01] there's no and the famous line is well I [8:05] I know it when I see it and I think that [8:07] um [8:09] this one I I just think it's going to be [8:12] necessarily [8:15] fuzzy but there will develop norms over [8:19] time. uh and [8:21] and you know like if enough kind of [8:25] people who run companies or run finance [8:28] or run whatever it is kind of agree yeah [8:30] no that's a norm uh then then I think it [8:33] is as opposed to kind of what we have a [8:36] lot now in the open benchmarks whereas [8:39] if you can solve this specific problem [8:40] then you're at this level and so forth I [8:42] think that's um [8:46] you know one it's hackable and then it's [8:48] too narrow [8:49] I think when you also look to some [8:51] historic analogies, there's there's a [8:53] lot of lessons that you can take from [8:54] them as well on what's gone wrong and [8:56] what we need to avoid. Um, and I think [8:58] for instance, Fed Bowels, one very early [9:01] decision we made was the decision to uh [9:03] to never sell training data to labs. [9:05] It's it's often a place that we're [9:06] pushed um when we start working with a [9:08] new lab to to actually source and sell [9:10] for them a bunch of training data. [9:12] >> Yeah, that's a lucrative business. Yeah. [9:14] >> Yeah. Yeah. uh and and actually a lot of [9:15] that industry has has now built these [9:18] gimmick style benchmarks as a mechanism [9:20] to sell their data. Um and so that [9:22] that's uh become kind of their go to [9:24] market as well. But um you know I think [9:27] if you look at auditing as an industry [9:29] you end up with issues like Enron where [9:31] um if you have the same group who's [9:33] responsible for doing the audit as well [9:35] as also consulting and and supporting [9:37] the company you have a mixed incentive [9:39] structure and uh and then it just [9:40] becomes pay to pass the audit or or in [9:42] this case pay to to win the benchmark. [9:44] Um and that's really not what the market [9:46] benefits from and and what we're trying [9:48] to do. [9:48] >> And Ryan today you have already a pretty [9:51] um extensive catalog of different type [9:53] of benchmarks. Some of them are more [9:55] focused on you know um specific [9:57] industries. Some of them are more like [9:59] consumer mental health related. Maybe [10:01] first just talk through like what are [10:02] the benchmarks that are most popular and [10:04] most like you know um read upon and [10:07] would love to dive into one of them as [10:09] well. [10:09] >> Yeah well we've done a lot of work in in [10:11] kind of the economically interesting [10:12] applications of models. Our finance [10:14] agent benchmark is used by a bunch of [10:16] the big financial institutions to get a [10:18] sense of how models are improving. Um we [10:20] also have a lot of good work in coding. [10:22] So our VI codebench measures how well [10:24] models can take a natural language [10:25] prompt and build a full stack web [10:27] application. And so uh that's been a [10:29] keen way to track model improvements [10:31] over the last nine months. Um yeah, [10:33] we're also doing a lot more experimental [10:34] work. So so one benchmark we released [10:36] recently I'm very excited by is our [10:37] recursive self-improvement index. Uh [10:39] it's a topic which a lot of big labs [10:41] have been talking about and and and [10:42] starting to report on in their model [10:44] cards. Um but there isn't a shared [10:45] language to talk about the RSI potential [10:48] of models. And so we created this as an [10:50] applesto apples way to actually [10:51] benchmark across the models. [10:53] >> Yeah, I thought that one is a very cool [10:54] benchmark. It's sort of all the rage in [10:56] the research community of how do you [10:58] measure like progress you can make [11:00] through having more frontier models that [11:02] you can just compound on the [11:03] capabilities. Um how do you actually go [11:05] about like building this RSI benchmark? [11:08] >> Yeah, I think you know in an ideal world [11:10] what you want to do is actually take a [11:11] frontier model and and have it train the [11:14] next version of itself and see what [11:15] where the delta comes from. Um but [11:17] obviously that's very expensive and [11:18] slow. And so what we're doing is forming [11:20] a set of proxies for every part of the [11:23] process it takes to build the next [11:24] version of the model. So there's some [11:26] work around pre-training, post-training, [11:28] harness level engineering. Um and then [11:30] seeing in in which mechanisms and [11:32] behaviors the models are able to to do [11:34] very good research work and build [11:36] something new and and where they're [11:37] struggling. [11:38] >> Very cool. Um there's also uh cases [11:41] where you have deprecated uh indexes and [11:44] benchmarks. Uh it's funny that I always [11:46] uh watch this benchmark industry people [11:48] like just like early diffusion model [11:51] days like you cherry pick whichever [11:53] image shows up the best and the most [11:55] perfect like benchmarks as well like you [11:57] pick something that's you know very [11:58] popular but maybe already saturated and [12:00] you rank very well on on that or score [12:02] very high but you took a very different [12:04] approach in like if these benchmarks are [12:06] saturated you'll deprecate it like maybe [12:08] talk us through the the the thinking [12:10] around that too. [12:11] >> A necessity and and this is kind of the [12:12] infinite game we're in. I mean, you're [12:14] wearing our shirt and so we have this [12:16] unofficial motto, always a higher peak. [12:18] Love this shirt. Yeah. So, in so far as [12:19] foundation model labs are hill climbing, [12:21] they're searching for the next peaks to [12:23] summit. It is our job to perpetually [12:25] construct these next mountains for them [12:27] to summit. Um, and I think that's also [12:29] how the economy has naturally functioned [12:31] over time. As um, you know, agriculture [12:34] becomes less important for our our labor [12:36] market. Um, there are new forms of uh of [12:39] of labor less required out of our out of [12:41] our population. Um and so in the same [12:43] way we we should expect our benchmarks [12:45] to keep up with the new frontier for [12:47] what we want models to do. Um there's [12:49] another component of retiring benchmarks [12:51] which I think is is underappreciated uh [12:53] which is that benchmarks should also be [12:55] reflective of the current state of the [12:56] world. Uh so in the same way if you're a [12:58] lawyer you have to you know retake the [12:59] bar exam and get certified or if you're [13:01] an architect you have to get your [13:02] certification and and or you know a [13:05] doctor um uh we should also expect [13:07] models to be tested on the current state [13:09] of the world and what we know in [13:10] medicine or what we what we have as our [13:12] set of laws. And so in the instance of [13:14] case law updating to something like [13:15] legal research benchmark that's a desire [13:17] to create a benchmark more reflective of [13:19] the current state of the world and also [13:20] push the models in place we want to see [13:22] them go. And how has um I guess one like [13:27] it used to be like we're we're um doing [13:30] these like multi- um answer or [13:32] multi-step [13:34] questions to like just evaluate prompt [13:37] answers. Now there's like a lot more [13:38] agentic uh work that's happening whether [13:41] it's on finance or legal um or coding [13:44] especially like there's a lot of um you [13:46] know um async background agents that can [13:49] just complete tasks. How has that [13:50] changed sort of how you build [13:52] infrastructure? how to think of [13:53] evaluating um the capabilities of not [13:56] just the models and agents themselves. [13:58] And there's also like a lot more [14:00] dimensions that people care about. It's [14:01] not just like capability, it's cost, [14:03] it's latency, it's like um you know [14:06] whether this model is flexible enough to [14:08] to to address like broader um domains [14:11] and task and so on. So how do you think [14:13] about the additional parameters to to [14:15] what you evaluate? [14:16] >> Oh yeah, there's I mean there's a lot [14:17] that goes into that. I think on the [14:19] infrastructure level you have a whole [14:20] new set of problems. Um I mean for [14:22] instance now we're testing models in [14:24] their ability to run over hours, days, [14:26] sometimes weeks and so the [14:27] infrastructure needs to be very stable [14:29] to support evaluation over time and and [14:31] if there is a failed request we should [14:32] be able to retry from that one and not [14:34] redo the whole trajectory. So there's [14:36] some simple simple mechanisms in the [14:38] infrastructure we have to think about. [14:39] Um but but I think in general what we've [14:41] seen is evaluations as they become more [14:43] complex have a fewer sample size but a [14:47] larger set of criteria or expectations [14:49] of them. And so what I mean by that is a [14:51] benchmark is largely some kind of input [14:53] space of things you're trying to query a [14:54] model to do and a set of requirements or [14:57] rubrics that you see in expectations of [14:59] the output. And so early on you have [15:00] things like imageet which have millions [15:02] of of images you're trying to see a [15:04] basic categorization for. So it's a [15:06] onetoone mapping between an image input [15:08] and a text label output. Now what we [15:10] have is far fewer set of tasks. You know [15:12] generate me 50 fullstack web [15:14] applications but a much larger complex [15:17] mechanism for evaluating the output [15:18] produced. And I think that trend is [15:20] going to continue as we see more complex [15:22] workflows evaluated with models. And and [15:25] do you do do you think that um [15:29] it will become kind of an a real time uh [15:35] kind of mechanism like so for something [15:37] like open router which stripe just [15:39] bought would open router look to valves [15:43] and say okay where should this next [15:45] request go or is this going to be kind [15:47] of strictly for like picking a model in [15:50] an enterprise for a task? Yeah, I mean I [15:53] think u you know open route is a bit of [15:55] a misnomer in that most of their usage [15:57] comes from being a model gateway and so [15:59] it's actually up to their their users to [16:01] decide which models they want to use [16:03] when and that's because really the [16:05] hardest part of routing is building the [16:06] evals and trying to determine uh in what [16:09] places a set of intelligences should be [16:11] used for a particular application. Um [16:13] and so so you know our effort in [16:15] supporting enterprises in building evals [16:16] has actually supported a lot of them in [16:18] also adopting routers and [16:20] >> and say more about why it's not only [16:21] important to labs but also existential [16:23] for for enterprise and maybe just say [16:25] more about how you guys work with with [16:27] enterprise. [16:28] >> Yeah of course yeah I mean I think that [16:29] the lab side of this is very clear like [16:30] you know if you're raising lots of money [16:32] investing heavily in building models [16:34] it's it's essential for you to show uh [16:36] why your model is getting better and [16:38] then why those customers should should [16:39] pay a premium for them. Um but what I [16:42] think is still underappreciated is on [16:43] the enterprise side this is turning to [16:44] be existential as well. Um, you know, I [16:48] um [16:50] I have I have a small anecdote related [16:51] to this actually. You was meeting with a [16:53] company in the Fortune 10. Um, and they [16:56] the way that they've adopted Cloud Code [16:57] has been with a roughly $100 a day [17:00] budget for their engineers. And and so [17:01] what I was hearing is that this has [17:03] actually fundamentally changed how work [17:04] gets done in this company. Uh, in that [17:06] there's a a rate limit which resets at 4 [17:08] p.m. And so so the most productive hours [17:10] of work are actually now 4 to 6 p.m. uh [17:12] when the rate limits reset. Um but then [17:14] there's this dead period in the [17:15] afternoon when people go on walks or you [17:17] know get a coffee because they they just [17:19] don't have the rate limits. And so I [17:20] think what's really um you know [17:22] illustrative there is that uh you see [17:24] that there is a misvaling of [17:25] intelligence happening at every layer of [17:27] the stack. And so by that I mean you [17:29] have engineers who have $100 worth of of [17:31] usage limits and they don't really know [17:32] how to aortion that to the greatest [17:34] productivity for them. Uh you also have [17:37] this Fortune 10 company which has kind [17:38] of arbitrarily said they're going to [17:40] allow $100 per employee. they've [17:41] actually recently increased it to $300 [17:43] per employee. So almost an employes [17:45] worth of salary in tokens for them to [17:47] use. Um and and this is actually pretty [17:49] arbitrary because it's hard to quantify [17:50] what the right usage limit should be. Uh [17:52] but then also Anthropic is running on [17:54] pretty narrow margins to support this. [17:56] Um and and they have, you know, massive [17:58] costs to to serve these models. Um and [18:00] so I think we're in this world where it [18:02] is still very unclear what ROI looks [18:04] like and how to value this intelligence [18:06] that's being used. Um, and so as we talk [18:08] about the existential concern for [18:09] enterprises, I think it is this uh kind [18:11] of direction we're shifting in where [18:12] token spend may start to eclipse salary [18:14] spend. And and so if if this is such a [18:17] meaningful line item in uh in your [18:19] costs, you actually have to justify the [18:21] ROI much more keenly than you you've [18:23] seen over the last 6 months. And over [18:25] time, as we were talking about, I think [18:27] a firm really is just its eval. So the [18:29] ability for a company to make its eval [18:32] legible uh in order to solve this ROI [18:35] calculus um is going to be the reason [18:37] why that that company wins out over the [18:38] competitors in the long term. [18:41] >> And and maybe just double click on that [18:43] um like similar question to to why the [18:45] labs can't do it themselves or requires [18:47] a third party um agency to to rate it. I [18:51] think it's it's a lot uh more um uh [18:54] understandable that you know you need [18:55] this that neutrality across the industry [18:57] but for enterprise they will argue that [19:00] they know the task the best for their [19:02] customers like what what how does like [19:05] val come in to provide value and maybe [19:06] you can talk through sort of val new [19:08] product launch as well [19:09] >> I would recommend a lot of companies to [19:11] develop in-house expertise um but I [19:14] think that should not be the only [19:15] solution um you know there's this [19:17] explosion of intelligence happening [19:19] there somehow still more foundation [19:21] model labs um getting getting [19:23] constructed and and each lab is also [19:25] releasing more models than ever with [19:27] many more hyperparameter options and [19:29] they exist within a complex set of [19:31] harnesses and agents. So the option and [19:33] we even talk about specific intelligence [19:35] this this new paradigm that's emerging. [19:37] Um so there's there's a growing set of [19:39] intelligence options and I think what [19:40] we're finding is that we're still [19:42] finding new places we want to use AI [19:44] models and so the use cases are growing [19:47] in complexity as well. And so I think if [19:49] you're if you're a company you you have [19:50] a compounding set of uh complexity in [19:53] the set of options it's very very hard [19:55] to develop the internal capability to do [19:57] the evaluation. Um and so to try and [20:00] remedy that we've started to release [20:01] some products more openly for [20:02] enterprises to use. Uh the first of [20:04] which is called val. Um and so Valmith [20:06] is is focused on codegen the area we're [20:09] seeing to be the highest spended in [20:11] enterprise AI. Um, it allows any company [20:13] to take their GitHub code base and build [20:15] their internal coding benchmark from it [20:17] to get a sense of what coding agents are [20:19] going to be the most performant, but [20:21] also what's going to be predo optimal or [20:23] the the highest ROI for them to use. Um, [20:25] and actually we we use uh Val Smith a [20:27] lot of valves and uh we're seeing that a [20:29] lot of the best enterprises and [20:31] sophisticated ones are doing that too. I [20:33] would expect that to be the direction [20:34] the market moves as it rationalizes. And [20:36] what are some of the examples when you [20:38] let's say benchmark on a private Ripple [20:41] that it just shows very different uh [20:43] performance cost behavior compared to [20:46] let's say like using like frontier model [20:50] using a a public repo uh benchmark. [20:53] >> I think today it's still very unclear [20:56] whether uh the best OpenAI model or the [20:58] best anthropic model is actually going [20:59] to be best for your repository. And so [21:01] we've seen a lot of non-intuitive [21:02] examples where you actually had to run [21:04] the eval to figure out what's going to [21:06] be the frontier performance for that [21:07] repository. Um I think you also now see [21:10] a very complex middle set of options in [21:13] that um there's now opus and sonnet [21:16] models from anthropic but also Luna and [21:18] Terra and Luna is very uh cost [21:20] competitive. Muspark is also very cheap [21:22] and and 1.2 is very capable. There's [21:25] also a growing ecosystem of open source [21:27] models which companies can choose to [21:28] self-host. So I think in this this messy [21:31] middle it's actually very non-intuitive [21:33] what's the right fit where we're [21:35] actually seeing in a lot of cases sonnet [21:36] is more expensive than opus because it [21:38] is so token hungry [21:40] >> um and and so I think if you were to [21:41] operate based on you know use sonnet [21:43] where where you feel like it's it's uh [21:45] applicable you may actually end up [21:46] spending more than you need to [21:48] >> and say more about how this evaluation [21:51] framework will apply to knowledge work [21:53] in other domains or what are some [21:55] examples I can [21:56] >> I think coding is a sign for what's to [21:57] come in every domain and and a lot of [21:59] primitives established there are [22:00] carrying over to other places. You know, [22:02] if you if you have a very good coding [22:04] agent, chances are you have a model that [22:06] can also make PowerPoint slides or DCFS [22:09] and Excel um and and with with high [22:11] degree of capability as well. Um I think [22:13] what we need to leverage in a lot of [22:15] these industries though is the existing [22:17] repository of work that has been done as [22:19] a mechanism to build evaluations. And so [22:21] just as Ben was talking about we we you [22:23] know haven't really solved the question [22:24] of what is human intelligence but I [22:26] think in a lot of industries we have a [22:28] sitting repository of data around what [22:30] work has looked like and it'll be the [22:32] task of of us and others to try and [22:34] codify that into evaluations uh that can [22:36] that can stay dynamic and actually [22:38] evaluate models where human work is [22:39] being done. [22:40] >> Maybe maybe just tag along the the [22:41] earlier question um how are you guys [22:43] using VSmith internally to uh evaluate [22:46] what's the best coding model for for V? [22:48] Yeah, I mean to be honest, this was [22:50] actually born out of a problem that that [22:51] we saw as well. So, um I wanted to do a [22:53] token maxing experiment. Um and it was [22:56] able to get unlimited access for our [22:58] team for for a month for some of the [22:59] coding tools. Um and so in retrospect [23:02] looking back, uh we had some we had a [23:04] lot of engineers spending between 1 to [23:05] two billion tokens a day. I think peak [23:07] day was one engineer spending six [23:09] billion. [23:11] >> Yeah. It's also crazy because [23:12] >> how much does that equal to to dollars? [23:15] >> So, okay. And then I went back and did [23:16] some math. Um, and and it looked like in [23:19] that month we spent roughly $1.5 million [23:21] worth of tokens. This is free, by the [23:23] way. I No, don't want um but it was [23:27] actually 10x more we were spending in [23:29] tokens than employee salary for that [23:31] month. Um, so it's not even like, oh, [23:33] this is this is 50/50, it's 10x. Um, and [23:36] it was interesting to debrief and see [23:38] the the places where people were using [23:39] agents and and this kind of, you know, [23:41] insecurity to use models all the time [23:43] everywhere. Um um and so what we were [23:46] faced with is okay, we cannot continue [23:47] with this mode of operation for the next [23:49] month. Um how do we actually [23:51] intelligently figure out what are the [23:53] right tools we should use and and for [23:55] what teams and what projects. Um and so [23:57] we ran this experiment of looking at the [23:59] work that was done. We we looked through [24:00] a lot of the traces. We looked through [24:02] our GitHub repo and and built out the [24:04] Val Smith tool and we found some pretty [24:06] surprising um insights like like for [24:08] instance the Cognition Devon tool is [24:10] actually very token efficient and so [24:11] that's a place we we've chosen to adopt [24:13] more um and and I think there's a lot of [24:15] places when you can get better pricing [24:18] models out of um subscriptions as [24:20] opposed to token based pricing and so [24:22] it's actually informed our strategy for [24:24] how we um can actually effectively token [24:26] max without spending $1.5 million per [24:29] month. [24:30] >> Very cool. So is the current um [24:32] operating mode that you're using like [24:35] one uh I guess more token efficient [24:39] harness plus model and then like on top [24:42] of that people have some more [24:43] flexibility to to use token base to for [24:46] some higher higher or more challenging [24:48] tasks. [24:48] >> Yeah, we so we we have access to all the [24:50] tools. We we give everyone access to [24:51] everything. Um, but we autosue [24:54] recommendations for any GitHub issue or [24:57] ticket for where to begin their session. [24:59] And that should titrate the actual usage [25:02] depending on the intelligence required [25:03] for that task. [25:04] >> Very cool. [25:06] >> I want to segue to the policy side for [25:08] sec for a second because we we talked [25:10] about how quickly benchmarks become [25:12] obsolete. In policy, it's it's even [25:13] worse in that laws move, you know, much [25:15] slower relative to capabilities. Ben and [25:17] Mark spend you know a bunch of time um [25:20] in DC and and with policy makers to try [25:22] to close that gap. So so given that who [25:25] who should define the standards here? Is [25:26] it labs? Is it independent evaluator [25:28] like you guys? Is it customers? Is it is [25:30] it government? How should this work from [25:31] a policy perspective? [25:33] >> Yeah, I think the short answer is that [25:34] everyone should be involved to some [25:36] extent. Um I think there's there's a [25:38] benefit from varied perspectives. I [25:40] think the main issue though is that [25:41] policy conversations as they've happened [25:43] over the last couple of years have been [25:45] very abstract. uh and there there's been [25:47] no material grounding to to figure out [25:49] what policy should cover. Uh and so even [25:51] when you have proposals from labs to [25:54] have uh a thirdparty testing company or [25:56] ecosystem, um it isn't actually made [25:58] explicit what the behavior in mexin by [26:01] which they work is. Um, and so I I view [26:03] our role, especially early on, is to [26:05] just be in evidence gathering uh mode [26:07] where where we're able to pull a lot of [26:09] information, empirical data about what [26:11] models uh are capable of and where the [26:13] risks are and that can go on to inform [26:15] um you know a more sophisticated [26:17] conversation about policy. And [26:19] >> how do you think about like who does [26:21] what? because the government kind of [26:24] actually did the first evalu [26:30] um okay what's at the frontier and needs [26:33] to be regulated right so they started [26:35] with you know some crazy idea with 10 to [26:38] 26 flops or some such thing um and so [26:43] when you think about it like what should [26:44] the government be doing um you know to [26:48] put it in this 30-day wait period or [26:50] 60-day wait period or whatever whatever [26:52] it is and then what should happen in [26:53] that weight period. Uh and how does that [26:59] intersect with what you're doing and [27:00] what's the right way to determine [27:02] whether a model is on the frontier or [27:04] not and needs to be um put in some [27:07] special box for a while to make sure it [27:10] doesn't break into everything like how [27:12] do you how do you think about like how [27:14] that relationship works? [27:16] >> I think there's effectively two [27:17] counterveailing forces that that has to [27:20] be considered. The first is the desire [27:22] to move very quickly and ensure that the [27:25] the government process isn't slowing [27:26] down uh the rate of technological [27:28] innovation. And I think the other part [27:30] is to make to make sure the technology [27:32] as it's developed is in the best [27:33] interest of Americans and people more [27:35] broadly. Um and so I think these are [27:38] very tough to reconcile and often you [27:41] know picking one means it's at the [27:42] expense of the other. And so what I'd [27:44] hope to see is that um by doing this [27:47] evidence gathering process, we can help [27:49] policymakers inform what they believe [27:51] technology should look like in order to [27:53] be aligned to American interest and it [27:55] can be the job of third party evaluators [27:57] to develop the technology to actually [27:59] test and enforce that. Um because I [28:01] think that will create a mechanism by [28:03] which you can see uh advancement in [28:05] methodologies for evaluation and testing [28:07] in a way that actually keeps up with the [28:09] frontier. It it doesn't lag behind or [28:11] slow down the pace of development. [28:13] Do you think um in terms of that already [28:18] and in and developing your emails like [28:20] do you think well can we test to see how [28:23] easy it is for this to get this model to [28:26] start reward hacking or or that kind of [28:28] thing um you know and doing illegal [28:31] stuff or is that kind of not in the [28:34] scope yet or how do you think about [28:36] that? [28:36] >> Yeah, I mean we think about this broadly [28:38] under the category of alignment. Um, I [28:40] think there's places where you see that [28:42] borne out now now where models that are [28:44] being tested for one cyber security risk [28:46] are actually reward hacking and figuring [28:48] out other ways to to get around it. Um, [28:50] but what we're trying to evaluate is are [28:51] models aligned with user intent. And so [28:53] in those places where we're actually [28:55] finding evidence that models are [28:56] exhibiting behaviors that that are not. [28:58] >> And maybe that's a question for you Ben [29:00] as well. Um, I guess how do you think [29:03] about the right division of labor here? [29:04] like what what should um you know [29:06] government agencies um control or and do [29:09] themselves and and where they should [29:11] like you know partner trust um you know [29:13] private uh companies to to take care of [29:16] and where do you see the gap that's [29:18] happening today? [29:19] >> Yeah. So I I think the government [29:21] agencies [29:23] do get a lot of warnings from by the way [29:25] the big labs. Oh, this thing is going to [29:27] biohack. This is going to be a cyber [29:30] security risk and so forth. And so I [29:32] think what the government needs to do is [29:35] go okay if the model [29:39] you know is if the model is capable of [29:41] it and then can somebody [29:45] uh kind of basically prod the market to [29:48] actually do the illegal behavior. um and [29:53] you know kind of specifying exactly what [29:56] are those things um that they don't want [30:00] in the market and then having a third [30:04] party kind of evaluate that. So it's [30:06] kind of does the uh government um you [30:11] know the government kind of has an [30:12] inclination of what it's afraid of be it [30:14] biohacking or cyber hacking or so forth [30:18] but then there becomes the question of [30:21] okay can the model do it and then can [30:23] you get the model to do it and and then [30:26] somebody's got to actually evaluate [30:28] those two capabilities and I think the [30:30] government is particularly ills suited [30:33] to do the latter particularly over time [30:35] It's just not a good government [30:36] function. But they're very good at [30:38] setting the rules because they can [30:40] enforce the rules. So I think that [30:42] that's kind of the combination you want [30:44] that the government sets and enforces [30:45] the rules and that um a very competent [30:49] kind of private company then tells them [30:52] if the rule is broken. um you know and [30:55] kind of it's it's been interesting to [30:57] see like the large labs start to go well [31:00] the model's got the capability and you [31:03] can get it to do the bad thing so we're [31:05] not going to let anybody have it we'll [31:06] just use it and make sure that our [31:08] people don't get it to do the bad thing [31:10] and that's and even that doesn't always [31:12] work so [31:14] >> very you know we're in interesting times [31:16] I would say [31:17] >> and to me there's the gap of like what's [31:19] the narrative and what actually happens [31:21] in real world um Because you know every [31:24] setup in again like an enterprise setup [31:26] is is very different like um or or just [31:29] like people however they use the models [31:31] are very different. The narrative uh [31:32] that connects to the actual examples are [31:34] very rare which is why we still talk [31:36] about openai and hugging face hack [31:38] today. We still talk about you know what [31:39] happened with fable and and for for like [31:42] two months but a lot of times like you [31:46] know that's not really how uh the models [31:48] being deployed the environment they're [31:50] running on is very uh bespoke. So h how [31:52] to like you know really bridging those [31:54] dots and again set up the the right uh [31:57] environment and also like rule basis for [32:00] adopting these models I think also just [32:01] requires you know someone taking um uh [32:05] the capability and taking like what's [32:06] the the the guard rails and and put it [32:09] down to to to the ground so that people [32:11] can like you know have the confidence [32:13] using using the models [32:15] >> to end say more about how exactly the [32:17] policy maker should work with the [32:18] evaluator what information do they need [32:20] what what how should the relationship [32:22] work so that it's most effective. [32:23] >> Yeah, I think I think in the first order [32:25] there should be an a mechanism by which [32:28] um insights and data can be passed [32:30] directly to relevant people in [32:31] government. And so now we're regularly [32:33] doing briefings for um executive and [32:35] legislative branches um on what we're [32:37] finding capabilities and risk of models. [32:39] And so I think first order that helps uh [32:41] people there get up to speed on what's [32:43] going on and and also track through what [32:45] will be problems in the future. I I [32:47] think you know things are moving very [32:48] very quickly. It's hard to predict where [32:49] where things are going. Um but at least [32:51] when you have data you can start to [32:53] extrapolate a trend. Uh and then I think [32:55] from there it's it's up to the people in [32:56] the legislative branch to decide where [32:59] they they want to see policy. Uh and so [33:01] it's not really our our place to give [33:02] recommendations like that. Um but if [33:05] they if they see that there is [33:06] significant risk in say mental health [33:08] for for people under the age of 18 uh or [33:10] biocurity risk in the models that [33:12] necessitates having a standardized way [33:13] to curtail model release. Um then it's [33:16] up to them to inform policy. Um and I [33:18] think then there's other places where [33:20] the executive branch in the places like [33:21] department of commerce or SEC is [33:23] responsible for for making sure that [33:25] private companies are able to adopt and [33:27] use the models in a way that's going to [33:28] be productive for the whole system. [33:30] >> I love to probe on another angle just [33:32] around geo geopolitical. Um I often see [33:37] evals being like a representation of [33:39] sort of the value um of of the model the [33:42] model developer like you kind of develop [33:45] this rubric of what's embedded in in the [33:47] model and of course different different [33:49] countries um uh and and labs in those [33:52] countries care about different things [33:54] like I mean um I'm born and raised in [33:56] China use a lot of like Chinese uh open [33:58] source models too like you still cannot [34:00] let them you know just go freely talk [34:02] about CPC and all the industry there uh [34:05] because um you know what happens in [34:07] China. Um so so how do you think about [34:10] how eval I guess and benchmarks play a [34:13] role in like standardizing or like uh [34:17] being treated by different um different [34:20] model labs from from different places. [34:23] >> Yeah. I mean to be honest from from my [34:26] um very idealistic perspective um I'm [34:28] surprised to see so much investment in [34:31] sovereign AI. Um, you know, if I was [34:32] taking a god's eye view, it would be [34:34] extremely inefficient to build all of [34:36] these data centers and and replicate uh [34:39] this data engineering process and train [34:40] these very large models um when when in [34:43] fact you could probably consolidate a [34:44] lot of these efforts. Um, but it seems [34:46] like that's not the world we're in or [34:48] the one we're headed towards and and [34:49] there's actually increased efforts to [34:51] build AI in a sovereign way. Um, and so [34:53] I think that takes having a shared [34:55] language to communicate about what the [34:57] framework for valuations are and and um [34:59] where we're going to collectively align [35:01] around the risks. Um, you know, I think [35:02] there's actually a lot to learn from [35:04] from nuclear here as well. I think [35:05] Reagan had this line, trust but verify. [35:08] Uh, and so I think we're starting to see [35:09] signs of trust in that um, you know, Zi [35:12] Jinping and Trump are going to be [35:13] meeting uh, next month. Um, but there is [35:15] no uh, clear way to actually do the [35:17] verification part of this. um having the [35:19] shared language of evals will allow us [35:21] to say things like you know you you have [35:22] the the right number of nuclear warheads [35:24] and in that example there were also [35:26] flyovers so so Mexican by which a [35:28] country could audit another country's uh [35:30] nuclear stockpile by having flyovers um [35:33] and so I think similarly if there's [35:34] concern about the societal or even [35:36] existential risk of AI it will [35:38] necessitate us uh constructing this [35:40] shared language of evaluations to do the [35:41] verification process [35:43] >> how do how do you think about um [35:46] harmonizing a policy like that so that [35:51] you know it's hard enough to do it in [35:53] America and then how would you think [35:58] about kind of taking it global um you [36:02] know because now you're dealing you're [36:03] not dealing with enterp enter enterprise [36:05] customers you're dealing with [36:07] governments and those governments are [36:08] competitive with each other and [36:12] how would you think about that working [36:15] >> I would be naive to say I have the [36:17] perfect solution to this problem today. [36:18] Uh and so I think there are baby steps [36:20] in which we can start. Um for instance, [36:23] there seems to be a lot of talk about [36:24] cyber security risk. I think the concern [36:26] around biocurity will become even more [36:29] important over time. And so there are [36:31] clear places where there will be mutual [36:32] interest in aligning around uh ways to [36:35] to prevent conflict around cyber or bio. [36:37] Um I in my opinion I think long-term [36:39] what's actually going to be the most [36:40] interesting is the recursive [36:42] self-improvement possibility. And that's [36:44] a place where you could see one country, [36:46] one or one company kind of run away with [36:49] it and produce models that we we don't [36:51] know much about or or operating in ways [36:53] that are unknown to us. Um, and so I [36:56] think having a way to in a joint way [36:58] describe this being the level of pace [37:00] we're comfortable with or this being [37:02] exceeding the pace of of development as [37:05] as it relates to RSI is going to be [37:07] super important. And that's where I [37:08] think you see a lot of the researchers [37:10] at Closource Labs calling for joint [37:12] conversations between governments today. [37:14] >> What do you think landscape will look [37:15] like going from from here? Um now that [37:18] we have lots of different capabilities [37:21] and uh capable models as well as like [37:24] you know um uh countries that care about [37:27] different um developing I mean everyone [37:30] cares about RSI for sure but like on on [37:33] the bio side or like the the cyber side [37:35] people care about um you know slightly [37:37] different different things whether it's [37:38] more offensive defensive and so on like [37:40] what do you think the landscape will [37:41] look like and um how do you think about [37:43] developing new benchmarks to to keep up [37:45] with with that? Yeah, I mean we're we're [37:48] hyper focused on on building benchmarks [37:51] that capture the frontier. And so uh in [37:54] so far as we see new places for [37:55] capabilities or risks at that frontier, [37:57] we want to make that uh an actually well [37:59] doumentable evaluation um on Val.AI. Um [38:03] and and I think it it takes have [38:04] increasing coverage over time. You for [38:06] instance I think in in cyber security a [38:08] lot of our historical work has been done [38:10] around um code vulnerabilities or memory [38:13] leaks that may exist in code. Um but [38:15] actually a lot of the the biggest [38:17] concern or risk is in the infrastructure [38:18] level. Uh and so these are not things [38:20] that are expressed in code but take um [38:22] simulating larger environments of [38:24] enterprise cloud infrastructure or or [38:26] even grid infrastructure for us to be [38:28] able to say this is what the offense or [38:30] defensive capability of models is. Um [38:32] and and so making sure evaluations are [38:34] reflective of those new places is is [38:36] really important for what we do at VLES. [38:38] uh and we believe that the the most [38:39] valuable form of this business will be [38:41] one that's incentive aligned around [38:43] doing really high quality evaluation not [38:45] supporting the intelligence development [38:47] process or uh the process by which the [38:50] models can actually improve on that side [38:52] over time. [38:53] >> Awesome. Thanks for coming on the [38:54] podcast. It's been a great episode. [38:56] >> Thanks so much for having me. [38:57] >> Thanks so much, Ryan. Thanks, Ben. [38:58] >> That was fun. Thank you.