Community Hackathon Uses AI Agents to Reproduce Over 2,200 ICML 2026 Papers

Community Hackathon Uses AI Agents to Reproduce Over 2,200 ICML 2026 Papers In an open hackathon organized by Hugging Face and alphaXiv, more than 1,200 participants used coding agents to test the reproducibility of ICML 2026 papers claim by claim. Across 19 days, participants audited 2,226 papers—roughly one-third of the conference—publishing over 6,800 Trackio logbooks that revealed verified claims, contested findings, and mathematical or experimental errors. ICML 2026 accepted 6,352 papers out of 23,918 submissions, doubling the prior year and straining volunteer reviewer capacity. A total of 1,221 participants produced 6,816 Trackio logbooks auditing 2,226 papers across 35,908 judged claims using an automated GLM-5.2 judge. 51% of examined papers had at least one claim verified, while 23% had at least one claim falsified or contested. Independent re-verification confirmed multiple paper errors, including broken mathematical proofs, discrepancy between theory and code loss functions, and evaluation metrics diluted by padding. Organizers conclude human oversight remains essential to guide agents, detect scale-dependent flaws, and make perceptual evaluations. Scaling Paper Review with Coding Agents The exponential growth of machine learning submissions—reaching 23,918 submissions and 6,352 accepted papers at ICML 2026—has outpaced traditional peer review capacity, with reviewers sometimes admitting to not checking proofs carefully. To test whether coding agents could bridge this gap, Hugging Face and alphaXiv organized an open challenge from July 15 to August 2, 2026. Participants indexed core claims from accepted papers, deployed autonomous agents like Claude Code, Codex, and Cursor, and published transparent Trackio logbooks containing code, artifacts, and execution traces. Hackathon Scale and Reproduction Statistics During the event, 1,221 community members launched 2,962 cloud jobs via Hugging Face compute credits and published 6,816 logbooks covering 34% of the conference. An automated judge running GLM-5.2 evaluated 35,908 claims. Aggregated results showed 51% of examined papers (1,103 papers) had at least one claim independently verified, with 266 fully reproduced. However, 23% (496 papers) had at least one claim falsified or contested, including 242 papers where independent teams reached opposing verdicts. Another 502 papers yielded only toy-scale evidence, and 280 remained inconclusive, frequently due to missing artifacts. Confirmed Falsifications and Author Responses Organizers re-verified claimed falsifications by re-deriving mathematics and re-implementing experiments from the original texts. Confirmed issues included a spotlight paging paper whose additive robustness term proved to be logarithmic rather than constant, an attention theorem that failed after 224 steps despite passing finite-horizon checks, and a continual learning paper whose code implemented forward KL divergence while its theory analyzed reverse KL divergence. Another paper showed a deflated perplexity because 66% of evaluated label positions were EOS padding tokens. When contacted, several authors confirmed the findings, and multiple arXiv corrections are underway. Some participant falsifications were also debunked, such as a claimed speed discrepancy caused by an arithmetic batch-size bug in the reproduction code. The Continuing Necessity of Human Guidance The challenge demonstrated clear boundaries for fully autonomous agents, which struggled with local loops, scale-dependent phenomena, and units mismatches. The most reliable reproductions occurred when humans steered the workflow—identifying flawed premises, redirecting the agent, or conducting perceptual assessments that numerical metrics missed, such as reviewing extreme image quantization. Organizers liken the emerging human role to principal investigators who configure the environment, ask the right questions, and supervise the agents executing the experiments.

社区借助AI智能体复现ICML 2026逾2200篇论文及核心发现 Hugging Face与社区动用1200余名研究者及其代码智能体，对ICML 2026录用的逾三分之一论文展开逐项声明复现，揭示了顶会论文的真实复现率、具体技术漏洞以及人类在自动化实验时代不可替代的管理与纠错角色。 1221名参与者使用各类代码智能体生成了6816份日志，尝试复现2226篇论文的35908项声明。 受检论文中51%有声明获得验证，23%存在被证伪或争议声明，且有242篇论文出现独立团队结论相反的现象。 经过核实确认了多项理论漏洞，包括数学证明失效、反例隐藏在长步数之后、理论推导与代码损失函数不符，以及评估指标被填充标记稀释。 纯智能体存在尺度盲区与局部死循环等局限，最高质量的研究产出依赖人类进行方向引导与感知评估。 活动背景与执行机制 ICML 2026共收到23918篇投稿并录用6352篇，投稿量翻倍使得志愿审稿资源严重受限。为检验论文质量，挑战赛于2026年7月15日至8月2日开展，先提取论文核心声明，允许参与者自由选用Claude Code、Codex等代码智能体运行实验。所有运行结果通过Trackio日志完全公开代码、产物与执行轨迹，并由开源模型GLM-5.2自动进行无信任的逐项声明裁决。 大规模复现统计结果 在尝试复现的2226篇论文中，51%（1103篇）至少有一项声明得到独立证实，其中266篇实现全部声明完全复现；23%（496篇）至少有一项声明被证伪或存疑，包括49篇全部声明被推翻。值得注意的是，有242篇论文在不同团队的独立复现中得出了截然相反的结论，说明可复现性具有对抗性。其余论文则受限于未公开代码数据而仅能进行小规模玩具实验，或因材料缺失无法定论。 核实确认的论文缺陷与证伪案例 组织方对35项证伪声称进行了对抗性核查。一项分页算法论文声称鲁棒性为H_k + O(1)，但实测与推导均表明误差项呈对数增长，实际为H_k + Θ(log k)；某注意力理论声称粒子必然向原点塌缩，但反例直到第224步甚至数千步后才显现，导致短时间测试出现误判；自蒸馏论文全篇理论基于逆KL散度，实际开源代码默认却运行前向KL散度；另有论文因评估中66%的标记为填充符号，导致困惑度被拉低三倍。 作者反馈与复现本身的失误 组织方已与多篇论文的作者沟通，多个团队确认了复现指出的问题，两篇论文已在arXiv提交修正版本。与此同时，核查也发现了复现本身的误判：曾有参与者声称某方法比基准慢两倍，经查是参与者在复现时将单轨迹时间与50批次时间混淆导致的计算错误，修正归一化后反而印证了原论文8倍加速的结论。 智能体局限与人类管理者的角色 实验表明纯智能体存在明显瓶颈，极易在有限视界停止检查、陷入死循环或因单位换算错误得出虚假结论。此外，部分任务如量化模型生成的图像质感，无法单凭数值指标断定是否崩溃，仍需人类进行感知层面的评判。人机协作的最佳范式在于人类扮演导师角色，负责搭建实验环境、质疑前提假设并提供反馈，进而引导智能体高效执行实验。

Back in July, we ran a hackathon where more than 1,200 community members brought their own coding agents and tried to reproduce the papers published at ICML 2026, claim by claim. In 19 days, participants published 6,816 Trackio logbooks reproducing 2,226 papers, about a third of the conference 🤯 In this post, we're sharing what we learned from running this hackathon, and what it suggests about the role humans will play when agents are doing the research experiments. More papers than anyone can review Questions about how reproducible AI research really is are older than the current AI wave. But these questions are exacerbated by scale. ICML 2026 received 23,918 submissions and accepted 6,352 papers , roughly double the previous year, continuing an exponential trend that is at least partly driven by AI agents making it faster to run experiments and write them up. Reviewing capacity has not doubled along with it. Reviewers at most conferences are volunteers who may not have the time or expertise to fully review a paper. Here is a review of one accepted ICML 2026 spotlight paper, in the reviewer's own words: "My low confidence score is because I did not check all the proofs carefully." Note that this paper got strong scores and a spotlight . Keep it in mind, because we will come back to this exact paper later in the post, and to what happened when we finally did check the proofs carefully. What has changed, though, is that the same technology driving the flood of submissions can also help us keep up with it. Coding agents like Claude Code, Codex, Cursor, and Pi can now read a paper, write the code, launch the experiments, and report back on what they found. Checking a paper carefully used to cost a reviewer a weekend; an agent can attempt it in an afternoon, in parallel, thousands of times over. So the question we wanted to ask was: if we actually re-examined a major conference at scale, and tried to reproduce every paper, what would we find? The hackathon (July 15 - August 2nd) Rather than audit papers ourselves, we opened it up to the whole community, with all the diversity of agent frameworks, compute budgets, and scientific taste that brings. From July 15 to August 2, 2026, the ICML 2026 Open Reproductions challenge worked like this: Pick a paper. We indexed all 6,341 accepted ICML 2026 papers with their abstracts and extracted the core scientific claims of each one, so an agent could start from a concrete, checkable target rather than a 40-page PDF. Multiple people reproducing the same paper was encouraged. Bring your own agent. Participants used Claude Code, Codex, Cursor, OpenResearch's orx , and everything in between. We provided a streamlined interface so an agent could pull the paper, its claims, and the challenge instructions with a single command. Reproduce, then publish everything. Every run produced a Trackio logbook: a static Hugging Face Space containing the write-up, the code that ran, the artifacts it produced, and (optionally) the full agent execution trace uploaded as a Hugging Face Dataset. The auditing process itself had to be auditable. Get judged. An automated Logbook Judge (running an open-weights model, GLM-5.2) re-read every logbook and issued a per-claim verdict: verified , falsified , toy (evidence at reduced scale), or inconclusive . The judge was explicitly instructed to treat each logbook's self-assessment as untrusted. Participants received $20 in Hugging Face compute credits to run experiments on HF Jobs ; across the challenge, participants launched 2,962 cloud jobs. Where a full reproduction was impossible, for example when a paper's dataset was proprietary or its checkpoints unreleased, participants ran toy reproductions on synthetic data mimicking the original's properties. Here is what a finished reproduction looks like: By the numbers, this hackathon was probably the largest attempted reproduction of a scientific conference: 1,221 community members joined the organization 6,816 reproduction logbooks published 2,226 papers attempted, 34% of the entire conference , many by several independent teams 35,908 claims judged, with all verdicts frozen in a public dataset at challenge close 2,962 HF Jobs launched; 274 full agent-trace datasets published on Hugging Face What we found Aggregating the claim-level verdicts per paper: 51% of examined papers (1,103) had at least one claim independently verified. Of those, 266 papers were fully reproduced, with every extracted claim verified, and 632 more were partially reproduced with nothing falsified. In total, 3,978 individual claims were confirmed with real experiments. 23% of examined papers (496) had at least one claim falsified or contested. That includes 49 papers where all claims were falsified and nothing could be verified, and, maybe most interestingly, 242 papers where independent reproduction teams reached opposite verdicts on the same claims. Reproducibility is not binary; it is adversarial. The remainder sat in the middle: 502 papers with toy-scale evidence only, and 280 where nothing could be established either way (missing artifacts were the most common cause). Reproductions done well Some papers came through the gauntlet looking great, and the community's best logbooks are worth reading in their own right: "Flat Minima and Generalization: Insights from Stochastic Convex Optimization" was reproduced by 20 independent teams, 12 of which verified every claim. The one linked included and published the full agent trace. "A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness" had 14 of 17 logbooks verify every claim. A paper about unreliable LLM judges holding up under scrutiny by LLM agents :) Falsifications, and what happened when we checked them 35 participants formally claimed they had falsified something. We adversarially re-verified every claimed falsification: re-reading the paper, re-reading the logbook, and re-deriving the math or re-implementing the experiment from the paper's own text. A few of the confirmed falsifications, linking to the logbook that found it: The paging paper from the introduction. The reviewer who did not check the proofs carefully? The paper, "Towards Optimal Robustness in Learning-Augmented Paging," claims its algorithm achieves robustness H k + O ( 1 ) H_k + O(1) . One participant's logbook measured the additive term growing like 0.38 ln ⁡ k 0.38 \ln k and located the exact step of the proof that breaks. Our own re-implementation extended the sweep to k = 1,024 and confirmed the growth at roughly nine sigma. The true robustness is H k + Θ ( log ⁡ k ) H_k + \Theta(\log k) . A theorem that falls after step 224. "Attention's forward pass and Frank-Wolfe" proves that token particles collapse to the origin whenever the origin starts inside their convex hull. Three independent teams found counterexamples, with violations first appearing at t = 224, ~3,800, and 6,416 steps, which neatly explains why everyone else "verified" the claim: finite-horizon checks stop too early. The cleanest counterexample is stated in exact rational arithmetic, so there is no floating-point ambiguity to hide behind. The authors confirmed the same day and are working on a fix. Theory written for one loss, results produced by another. In "Self-Distillation Enables Continual Learning," the paper's central equation and its entire theory section analyze reverse KL divergence, but the released code's default, which per the authors produced all the paper's results, computes forward KL. The logbook that caught it also failed to reproduce the paper's headline +4pp result under the authors' own code and data. The authors have already uploaded a clarified version to arXiv. An evaluation diluted by padding. In "Do Transformers Need Three Projections?", a participant discovered that ~66% of evaluated label positions were EOS padding tokens that train to near-zero loss, deflating perplexity roughly threefold. The abstract's "3.1% quality cost for 50% cache reduction" becomes roughly 9.4% once corrected. 🚨 False falsifications. Sometimes we found flaws in an attempted reproduction. One logboook claimed dramatically, "the paper's method is 2x slower than the baseline"; this turned out to be an arithmetic bug in the reproduction : per-trajectory time compared against per-batch-of-50 time. Correctly normalized, the participant's own data confirms the paper's claimed 8x speedup. Talking to authors We have begun writing to the authors of every confirmed finding, with a simple framing: here is what we found, here is all the evidence, do you agree or is our analysis wrong? The early responses have been very positive: So far authors have confirmed findings on multiple papers, two arXiv corrections are in flight, and in one case an author had quietly fixed the error in a new arXiv version a month before the challenge found it, which we count as independent convergence 🤗 The role of humans The most interesting question that this hackathon raises is: do humans still have a role in reviewing papers? We think so, for several reasons: Pure agent execution hits real limits. Agents got stuck in local loops, misread scale-dependent behavior (several "verified" verdicts on the paging paper came from checks that stopped before the log-k growth became visible), and occasionally built an entire falsification on top of a units mismatch. The challenge's most reliable results came from workflows where a human was steering: re-pointing the agent, questioning an assumption, or deciding that an experiment's premise was wrong before burning a week of compute on it. Some evaluation is irreducibly human, for now. Our human-in-the-loop winner is the clearest example. The paper claimed stable image generation under extreme quantization. Numerical metrics said "no collapse"; whether the images were actually usable was a perceptual question. The agent built a purpose-built review UI, and the human personally judged all 128 image pairs, with the annotations committed to the repo and the agent validating their consistency afterward. The published agent trace captures the whole exchange, down to the participant asking how the review tool works and coming back with "I have gone over the pairs and put the csv in the repo, please check." So what are our roles as human reviewers? We think it is to manage intelligence effectively. Much like a professor or principal investigator (PI) sets up an environment where grad students can do good work, with compute, harnesses, data access, and targeted feedback at the right moments, the participants who got the most out of their agents were the ones who built the right environment and asked the right questions , then let the agents do the running. Thank you To the 1,221 people who joined, the winners, the authors who responded with grace, and our organizers at Hugging Face and alphaXiv: thank you. Every logbook, verdict, trace, and artifact from the challenge is public, starting from the challenge Space . We think this is the largest open, claim-by-claim audit of a machine learning conference to date, and we would love for it not to hold that record for long. Stay tuned for future reproduction events. 🤗