UK AI Security Institute Adopts EvalEval Infrastructure to Publish Reproducible Benchmark Data

UK AI Security Institute Adopts EvalEval Infrastructure to Publish Reproducible Benchmark Data The UK AI Security Institute (AISI) is utilizing the EvalEval Coalition's infrastructure to openly share evaluation results, configurations, and transcript-level data. The initiative aims to enhance reproducibility and standardize reporting across AI evaluations using the Every Eval Ever (EEE) schema and Evaluation Cards. UK AISI has adopted EvalEval's Every Eval Ever schema and Evaluation Cards to publish benchmark results openly. The initial data release covers five main benchmarks—HealthBench, FrontierMath, Humanity's Last Exam, SWE-Bench Pro, and Terminal-Bench 2.0—across models including Claude Opus 4, 4.5, 4.6, and GPT-5, 5.2, 5.4. The release accompanies AISI's paper examining how inference-time compute and evaluation protocols influence frontier model benchmark performance. Two cyber evaluations, Cyber CTFs and The Last Ones, are also included to provide verified reference points for researchers. Need for Reproducible Evaluation Reporting As AI deployment accelerates, model evaluations often suffer from inconsistent reporting across diverse formats and outlets, frequently omitting the configuration details necessary to reproduce results. Running evaluations from scratch can be prohibitively expensive. EvalEval addresses this through its Every Eval Ever reporting schema and Evaluation Cards platform, unifying benchmark metadata, model metadata, and run data into standardized, interpretable records. AISI Benchmark Data and Research Findings AISI is providing verified results, context, and setup configurations tied to its paper, 'How Inference Compute Shapes Frontier LLM Evaluation.' The release includes main experiment data on benchmarks such as FrontierMath, Humanity's Last Exam, HealthBench, SWE-Bench Pro, and Terminal-Bench 2.0, evaluated across Claude Opus 4, 4.5, 4.6, and GPT-5, 5.2, 5.4, as well as cyber evaluation datasets. Findings on Humanity's Last Exam highlight how benchmark scores shift based on evaluation protocols and inference-time compute, demonstrating sustained task-solving as token counts increase when models receive oracle feedback. Collaborative Goals and Infrastructure The partnership builds on past collaboration originating at a joint workshop alongside NeurIPS 2025 and complements AISI's existing technical initiatives, including OptStop for evaluation efficiency and HiBayES for hierarchical Bayesian modeling. By standardizing transcript analysis and elicitation data, the collaboration aims to supply reference points that allow researchers, developers, and policymakers to compare models across differing evaluation conditions.

英国AI安全研究所与EvalEval合作推动基准测试结果的可复现性 英国AI安全研究所（UK AISI）宣布采用EvalEval的基础设施公开分享模型评估结果，以促进AI评估科学的可复现性与可验证性。双方通过统一的架构与平台，解决当前评估报告格式分散且难以复现的问题。 UK AISI通过EvalEval的Every Eval Ever模式与Evaluation Cards平台公开经核实的评估结果与配置信息。 本次发布涵盖HealthBench、FrontierMath、Humanity's Last Exam等五项主要基准测试及网络安全评估，涉及Claude与GPT系列多款前沿模型。 公开的数据配合AISI关于推理算力如何影响LLM评估的研究论文发布，提供交互上下文与配置以供行业参考与横向比较。 解决评估复现难题的共享基础设施 随着AI部署加速，模型评估成为衡量系统性能的重要依据，但目前评估结果往往散布在不同渠道且缺乏足够的信息来复现，重新测试成本也极其昂贵。EvalEval通过Every Eval Ever（EEE）模式和Evaluation Cards平台，为评估结果与解读信息提供通用结构。AISI此前已推出提升评估效率的OptStop和增强统计严谨性的HiBayES，此次合作进一步推动了评估报告标准的统一。 AISI发布的数据集与模型覆盖范围 在本次合作中，AISI通过Evaluation Cards公开了具体评估方法和发现。主要实验数据覆盖HealthBench、FrontierMath、Humanity's Last Exam、SWE-Bench Pro和Terminal-Bench 2.0五项基准，测试对象涵盖Claude Opus 4、Claude Opus 4.5、Claude Opus 4.6以及GPT-5、GPT-5.2、GPT-5.4等模型。此外还包含Cyber CTFs与The Last Ones两项网络安全评估数据。这些数据伴随AISI关于推理计算和评估协议对模型表现影响的研究一同发布，为学界提供了可比对的参考基准。

The EvalEval Coalition is thrilled to share that the UK AI Security Institute (AISI) is using EvalEval's infrastructure to openly share evaluation results, supporting more reproducible and verifiable evaluation science. AISI and EvalEval have previously collaborated on research that began at a joint workshop alongside NeurIPS 2025 , and feedback from the Institute has helped shape the Every Eval Ever (EEE) schema . This next phase of the collaboration puts that shared infrastructure into practice. Why reproducible evaluation reporting matters As AI deployment accelerates, evaluations are becoming increasingly important sources of evidence about model and system performance. Yet results are reported across many formats, platforms, and outlets, often without enough information to reproduce them. Running the evaluations again may itself be prohibitively expensive. EvalEval's mission is to improve this ecosystem through a shared reporting schema, Every Eval Ever , and an open platform, Evaluation Cards , that brings evaluation results and the information needed to interpret them into a common structure. This builds naturally on AISI's work to make evaluation more efficient through OptStop , more statistically rigorous through HiBayES , and more standardised in areas including transcript analysis and capability elicitation. Together, AISI and EvalEval are working to diagnose gaps in evaluation reporting and build shared infrastructure to close them. What AISI is sharing Transcript-level transparency matters not only for reproducibility, but also for analysis and diagnosis. In this new phase of the collaboration, AISI is making publicly reported evaluation methods and findings available through Evaluation Cards where appropriate. The release includes verified results, context, and configuration information for the five benchmarks in the paper's main experiment: HealthBench FrontierMath Humanity's Last Exam SWE-Bench Pro Terminal-Bench 2.0 These results cover six frontier models: Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2, and GPT-5.4. The release also includes results from two related cyber evaluations—Cyber CTFs and The Last Ones—which use a different, partially overlapping set of models. The data accompany AISI's paper, How Inference Compute Shapes Frontier LLM Evaluation , which studies how benchmark performance depends on inference-time compute and evaluation protocol. Performance on Humanity's Last Exam changes with evaluation protocol and inference compute. Each curve shows the cumulative share of attempted tasks solved within a given token count, using the earliest observed success per task. When models received correctness feedback from an oracle after each attempt, they continued to solve additional tasks as token use increased. When results are openly released with setup information, researchers and practitioners can examine individual studies more closely and compare findings across the wider ecosystem. Where other reports lack these details, releases like AISI's provide verified reference points for interpreting evaluations in context—for example, by helping researchers understand how setup choices may influence reported performance. As more evaluators adopt EEE, open comparisons like these can support broader and more reliable meta-research. AISI's Terminal-Bench 2.0 results alongside other reported evaluations for the same models, under different evaluation setups. We are excited about this adoption and look forward to further standardising and sharing evaluations with AISI and other AI evaluation organisations. Contribute to the shared mission Model developers: Report verified evaluation results . Evaluation developers: Report benchmarks and run data using the Every Eval Ever schema . Evaluation, governance, and policy researchers: Explore Evaluation Cards by benchmark or model, or use it to examine the state of evaluation reporting as a whole. About the EvalEval Coalition The EvalEval Coalition is a research community developing scientifically grounded research and robust deployment infrastructure for the evaluation ecosystem. Its goal is to improve evaluation science, address the lack of consensus around documenting evaluation applicability and utility, and broaden coverage of the impacts that matter for scientific research and policy analysis. The coalition's flagship projects include Every Eval Ever , a shared schema and repository for evaluation results, and Evaluation Cards , which combines benchmark metadata, evaluation-run data, and model metadata into interpretable records. Together, they make it easier to understand when apparently similar scores were produced under meaningfully different conditions. About the UK AI Security Institute The UK AI Security Institute is a research organisation within the UK government's Department for Science, Innovation and Technology. Its mission is to equip governments with a scientific understanding of the risks posed by advanced AI. AISI conducts research and builds infrastructure to understand advanced AI capabilities and impacts, develop and test mitigations, and inform policy. Further reading How Inference Compute Shapes Frontier LLM Evaluation HiBayES: Improving LLM evaluation with hierarchical Bayesian modelling HiBayES paper OptStop paper Every Eval Ever Evaluation Cards