NYT Engineer Says Puzzle Benchmarks Contradict Tech Industry AGI Claims

纽时工程师称字谜基准测试驳斥科技界AGI论调 前沿模型在ARC-3和隐式填字游戏中的成功率骤降至个位数，而未经训练的人类却能取得100%的成绩，这反驳了通用人工智能已经到来的主张。 Shafik Quoraishee探讨了游戏系统、约束满足与机器学习的交叉领域，解释了为何人类文字谜题对现代AI而言仍是一项持续存在的挑战。他剖析了填字游戏的架构约束、Dr.Fill等算法求解器的演变，以及大语言模型在解析隐式线索时遇到的具体失效模式。通过直接类比ARC-AGI基准测试——普通人在ARC-3上的正确率达到100%，而前沿AI模型却在7%到30%之间挣扎——他阐明了二维约束传播和少样本归纳推理如何揭示当代AI与真正通用智能之间的认知鸿沟。 《纽约时报》填字游戏及游戏产品组合成为探索莫拉维克悖论（Moravec's paradox）的重要基准，在该悖论中，对人类而言直觉性的推理被证明对AI极其困难。 (0:04) 《纽约时报》保持着严格的编辑规则，即所有刊登的填字游戏完全由人类创作者纯手工打造，绝不通过AI生成。 (2:26) 标准填字游戏的求解严重依赖约束满足，其中填入交叉字母可以通过最小剩余值（MRV）和集束搜索等启发式方法来剪枝分支因子。 (5:10) 像Dr.Fill这样基于启发式驱动的非神经网络求解器，通过在2021年赢得美国填字游戏锦标赛，证明了其对标准美式填字游戏的掌握能力。 (5:10) 隐式填字游戏对大语言模型来说仍是一个未解的前沿难题，因为它们需要识别变位词、同音词和字谜等双重线索机制，而不仅是标准释义匹配。 (10:35)

According to New York Times staff engineer Shafik Quoraishee, the cognitive mechanics of crosswords and grid-based puzzles expose significant reasoning deficits in modern artificial intelligence, demonstrating that true general intelligence remains unrealized despite advances in algorithmic search. Grid Mechanics and Heuristic Constraint Solvers (2:26) The New York Times maintains a strict editorial standard requiring all crosswords to be human-constructed rather than generated by artificial intelligence. Under longtime editor Will Shortz, puzzles follow rigorous topological rules: 180-degree rotational symmetry, a minimum word length of three letters, complete letter checking where every character belongs to both an across and a down entry, and full interlock preventing isolated subgraphs. These design requirements turn grid completion into an exponential constraint-satisfaction problem, where each filled letter restricts candidate entries across intersecting slots. To navigate these search trees, non-neural computational solvers apply the Minimum Remaining Values (MRV) heuristic and beam search to prune unviable branches. This algorithmic approach reached a major milestone in 2021 when the automated solver Dr.Fill won the American Crossword Puzzle Tournament (ACPT). Dr.Fill relied on parallel constraint satisfaction and candidate scoring rather than generative large language models, demonstrating that standard crosswords can be effectively solved using structured heuristics. Cryptic Clues and Spatial Graph Architectures (10:35) While standard American crosswords are largely computationally solved, cryptic crosswords remain a formidable barrier for machine learning. Cryptic clues combine a straightforward definition with an explicit wordplay recipe, such as anagrams, homophones, or containment structures. Large language models struggle with this dual-layered logic, regularly misidentifying clue mechanisms and hallucinating invalid rationalizations. To bridge the gap between linguistic interpretation and geometric constraints, Quoraishee designed an experimental architecture called the Cryptic Solver GNN. Rather than treating the puzzle purely as a text-generation task, this model frames the grid as a connected graph. By combining Graph Neural Networks with semantic word embeddings, the architecture coordinates letter placements against the physical coordinates and intersecting constraints of the board. The ARC-AGI Generalization Collapse (14:56) The reasoning failures seen in cryptic crossword solving mirror AI bottlenecks on the ARC-AGI (Abstraction and Reasoning Corpus) benchmark, which tests few-shot inductive reasoning on novel visual grids. Both formats require models to deduce implicit, local transformation rules and propagate them across a two-dimensional grid without relying on memorized training data. While frontier AI models effectively saturated the initial ARC-1 benchmark at roughly 97% and scored between 90% and 92.5% on ARC-2, their performance collapsed on ARC-3. On ARC-3, Claude Opus achieved approximately 30% and frontier GPT models fell to around 7%. In contrast, untrained everyday humans tested in San Francisco achieved a 100% solve rate from only a few demonstration examples, illustrating the fragility of current machine heuristics when confronted with novel inductive tasks. Quoraishee argues that this performance gap challenges claims by industry figures, such as Nvidia CEO Jensen Huang, that AGI has effectively arrived. Because games like crosswords require associative vocabulary, shifting constraints, and cultural subtext, they provide an objective benchmark showing that contemporary models still lack the flexible reasoning characteristic of human cognition.

《纽约时报》资深工程师兼编辑 Shafik Quoraishee 指出，尽管人工智能在计算棋类和标准化任务中屡获突破，但在面对高度依赖常识联想、双重语境机制与空间拓扑约束的填字游戏时仍面临严峻瓶颈。这种局限不仅生动印证了莫拉维克悖论，更与前沿大模型在抽象推理基准 ARC-3 上的断崖式性能滑坡相呼应，表明当前模型仍未具备人类无需预先训练即可展现的少样本归纳泛化能力。 填字游戏的网格约束与启发式求解演进 (2:26) 《纽约时报》自 1942 年起持续刊登填字游戏，并延伸出 Spelling Bee、Wordle 与 Connections 等数字产品。Quoraishee 强调，该报填字游戏坚持由人工独立构思与设计，绝不采用生成式 AI 自动化制作。标准填字游戏具有极度严格的几何与语言学约束：盘面必须具备 180 度旋转对称性、词长至少三字母、所有字母必须在横向与纵向两向完全交叉（Checked），且黑色格子不得将网格分割为孤立子图。在 1993 年加入的编辑 Will Shortz 主持下，谜题构建了从周一（最易）到周六（最难）逐步升级的难度曲线，周日则以更大尺寸提供中等难度体验。 在计算维度上，填字游戏属于高分支因子的约束满足问题。早期非神经求解器主要依托词频统计、束搜索（Beam Search）以及最少剩余值（MRV）启发式剪枝策略，利用字母交叉快速压缩周围候选词空间。这一传统规则演进路线在 2021 年达到顶峰，非生成式 AI 求解系统 Dr.Fill 凭借并行冲突驱动的约束满足机制赢得了美国填字游戏锦标赛（ACPT），延续了从 Deep Blue（1997 年战胜卡斯帕罗夫）、AlphaGo（2016 年战胜李世石）到 AlphaStar（星际争霸）的游戏 AI 探索历程。 神秘填字难题与图神经网络求解架构 (10:35) 相较于已被传统算法基本攻克的标准填字游戏或包含多重歧义的薛定谔网格（如 1996 年同时兼容 CLINTON 与 BOBDOLE 的美国大选谜题），神秘填字游戏（Cryptic Crosswords）构成了前沿 AI 的核心认知壁垒。神秘填字的每一条线索均包含一个字面定义与一套文字游戏构造公式，涵盖易位构词（Anagram）、同音词、双重定义或包容拼接（Charade）等复杂机制，例如用“Father's drink”对应双重释义答案 POP，或以指示词“wild”指示重排“among”得出 MANGO。 主流大语言模型在解析神秘填字时表现极其脆弱，不仅频繁混淆不同线索构造的元规则，还常为错误答案编造虚假的归纳逻辑。针对这种语言学与二维空间交织的复合约束，Quoraishee 开发了实验性架构 Cryptic Solver GNN。该系统将填字盘面建模为相互联结的图结构，借助图神经网络与语义词嵌入技术，将线索的语义表征直接映射到棋盘的几何几何约束空间中，协同推演全盘解集。 ARC-AGI 评测揭示的模型推理能力断崖 (14:56) 神秘填字游戏所暴露的模型短板，与旨在评估通用人工智能的抽象与推理语料库（ARC-AGI）基准形成了紧密映射。两者在结构上高度相似：均要求解题者基于二维网格中的局部线索推导不可见的全局规则，并在坐标空间中实现规则传播。机器学习模型在面对 ARC 任务时，往往极难基于极少量示例推导出物体的空间平移、多阶段触发机制或基于非显式颜色的运动矢量。 根据评测数据，前沿模型在初代 ARC-1 基准上已接近饱和（达 97% 解决率），在 ARC-2 上顶级推理模型与 Claude Opus 亦能分别达到约 90% 与 92.5% 的解决率。然而在最新的 ARC-3 评测中，前沿 AI 的表现出现灾难性滑坡：Claude Opus 的解决率跌至约 30%，顶级 GPT 系列模型更降至约 7%。与此形成鲜明对比的是，旧金山街头随机抽测的普通人类受试者在未经专门训练的前提下，仅凭两到三个示范案例即可全部掌握规则，测试解决率达到 100%。 莫拉维克悖论与 AGI 的通用泛化验证 (19:47) 人类受试者轻松应对未知规则与前沿模型性能暴跌的对照，直接驳斥了部分科技界领袖（如英伟达首席执行官黄仁勋在财报或行业讨论中提出的“AGI 实际上已经到来”）的乐观论断。Quoraishee 指出，智能代理在面对分布外数据时依然依赖脆弱的表面启发式经验，缺乏真正的动态泛化与逻辑迁移能力，再次验证了莫拉维克悖论——计算机能够轻易处理人类感到繁重的超大规模计算，却在人类直觉性的常识联想和情境推理面前步履维艰。 演讲者总结认为，诸如《纽约时报》填字游戏这样高度融合文化潜台词、双关隐喻和严格约束传播的网格游戏，并非简单的消遣工具，而是检验通用人工智能的试金石。追踪模型在处理此类细微文化语境与动态规则交互时的表现，能够剥离单纯依靠预训练参数记忆带来的表象能力，为学术界与产业界度量 AI 究竟何时才能跨越通用认知鸿沟提供清晰而严谨的评估视角。 填字游戏与 ARC 基准的测试结果共同表明，真正的通用抽象归纳能力目前仍是人工系统的显著盲区。尽管预训练大模型在海量已知数据与固定结构中表现卓越，但当系统被迫在缺乏庞大训练集的前提下独立推理全新元规则并协调空间约束时，其推理链条便极易断裂。游戏领域所提供的严苛约束和隐喻空间，将持续充当检验模型是否真正获得通用认知灵活性的试金石。

## Background 本视频由频道 Ai4 于 2026 年 8 月 18 日在 YouTube 平台发布，由《纽约时报》资深游戏与机器学习工程师 Shafik Quoraishee 带来分享。视频简介指出，尽管现代大模型在语言和编码上取得了突破，但《纽约时报》填字游戏依然是衡量 AI 认知灵活性与语言推理的坚固难关，可作为检验通用人工智能（AGI）进展的重要基准试验场。 ## TL;DR 演讲从《纽约时报》纯人工构造的填字游戏规范与约束满足（CSP）求解机制切入，回顾了倒立摆、Alpha 系列以及填字求解器 Dr.Fill 的发展历程。随后深入探讨了密语填字游戏（Cryptic Crosswords）这一前沿阵地，展示了 LLM 在面对隐喻和规则重组时高达 100% 的有效失败率，以及基于 GNN 空间求解的新尝试。最后，演讲引入 ARC-AGI 抽象推理基准，通过在最新测试中人类普通受试者达成 100% 而前沿大模型骤降至 7%~30% 的残酷代差，有力驳斥了 AGI 已经实现的论调，证明游戏仍是衡量真实认知鸿沟的核心标尺。 ## Atmosphere & style 整场演讲兼具计算机科学的严谨与益智解谜的趣味，语调轻松幽默且叙事层层递进。主讲人风格从容坦诚，在现场翻页笔失灵等意外小插曲中以自嘲化解，互动感十足。从对游戏 AI 辉煌历史的肯定逐渐转向对当下 AGI 炒作的清醒反思，给观众带来一种既硬核烧脑又真实接地气的思辨体验。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=4s) 填字游戏融合了语言学、博弈论与前沿计算科学，是衡量通用人工智能（AGI）认知弹性与语言推理鸿沟的理想测试基准。 - [00:54](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=54s) 填字游戏直观展现了“莫拉维克悖论”——人类凭借常识与直觉轻而易举完成的任务，当前极其强大的 AI 却往往难以胜任。 - [02:30](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=150s) 《纽约时报》填字游戏坚持 100% 由人类手工设计，遵循严格的 180 度旋转对称、横纵全相交、全连通图等几何与图论约束规范。 - [05:10](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=310s) 填字游戏求解属于高维“约束满足问题”（CSP），通过最小剩余值（MRV）启发式与束搜索（Beam Search）剪枝，可有效化解指数级分支因子带来的搜索组合爆炸。 - [08:57](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=537s) 基于冲突约束与评分驱动的传统求解器 Dr.Fill 在 2021 年夺得全美填字游戏锦标赛头名，标志着传统启发式搜索在常规网格求解上的顶峰。 - [10:35](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=635s) 如 1996 年大选“薛定谔谜题”等变体，体现了人类在极度受限的网格几何中创造多重解且完全自洽的语言智慧。 - [11:16](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=676s) 面对需要破解规则操作、字谜重组与深层文化隐喻的“密语填字游戏”（Cryptic Crosswords），现代大语言模型几乎遭遇 100% 的有效失败，常一本正经输出详尽却错误的伪推理。 - [13:50](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=830s) 针对密语谜题的空间与语义双重约束，基于图神经网络（GNN）的求解架构可将线索与交叉槽位抽象为图拓扑，结合多维词嵌入实现空间-语义联合推理。 - [14:56](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=896s) ARC-AGI 抽象推理基准要求系统在极少样本下自发摸索涌现规则，直击当前智能体在环境机制归纳与因果理解上的致命瓶颈。 - [17:46](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=1066s) ARC-3 基准测试中前沿大模型表现遭遇断崖式暴跌（骤降至 7%~30%），而街头随机受试人类达成 100% 成功率，打破了“AGI 已经实现”的神话，印证了 AI 距离真实通用智能仍存在巨大代差。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=4s) 个人背景与通过填字游戏探索通用人工智能（AGI） 演讲者 Shafik Quoraishee 开门见山地提出了本场分享的核心议题：借助《纽约时报》（The New York Times）填字游戏来深化对通用人工智能（AGI）的认知。尽管语言学、博弈论与前沿计算看似跨度极大，但填字游戏恰恰是拆解这些复杂交集的理想切入点。在进入正题前，Shafik 简要回顾了自己的职业履历。他目前是《纽约时报》的资深游戏与机器学习工程师，负责填字游戏、各类益智游戏产品以及底层基础设施的技术架构。在此之前，他曾先后在 NBA、Business Insider 和 MTV 任职，拥有深厚的媒体与工程背景，早期更是一名专注于计算机视觉（Computer Vision）领域的机器学习研究员，随后逐步将技术视野延伸至游戏智能领域。 ### [00:54](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=54s) 莫拉维克悖论、纽约时报游戏生态与经典游戏 AI 《纽约时报》的游戏板块拥有深厚的历史积淀，其中最具代表性的填字游戏早在 1942 年便已创办，近年来更推出了 Spelling Bee、Wordle 以及 Connections 等风靡全球的爆款。《纽约时报》的填字游戏不仅是一项智力消遣，更成为审视人工智能演进的独特棱镜。本场演讲探讨填字游戏的核心视角是“莫拉维克悖论”（Moravec's Paradox）——即“人类轻而易举能够做到的事情，极其强大的 AI 却往往难以胜任”（"humans are still good at doing things that very powerful AI still is not capable of doing"）。Shafik 此前曾专门就《纽约时报》的 Connections 游戏撰文探讨过这一悖论背后的认知鸿沟。 从 20 世纪 50 和 60 年代起，游戏一直是人工智能算法迭代与展示高光时刻的核心试验场。以街机经典《吃豆人》（Pac-Man）为例，其幽灵幽默而精准的围堵机制，本质上是工业界最早落地的简单 AI 之一——有限状态机（Finite State Machine, FSM）。这种通过规则与状态转换实现的控制逻辑，构成了早期游戏智能的基础形态。 ```mermaid flowchart LR accTitle: Crossword Solving Constraint and Search Architecture accDescription: 展示填字游戏从线索输入、约束传播到分支修剪与束搜索的计算求解过程 subgraph Input["题目与线索输入"] A["填字网格与线索"] B["启发式评分 / 词频分析"] end subgraph ConstraintEngine["约束满足核心"] C["MRV 最小剩余值启发式"] D["双向相交约束传播"] end subgraph SearchOptimization["搜索与求解"] E["指数级分支决策树"] F["束搜索 Beam Search 剪枝"] G["最优网格解"] end A --> C B --> C C --> D D --> E E --> F F --> G ``` ### [02:30](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=150s) 纽约时报填字游戏的规则结构与难度曲线 在解析填字游戏的技术求解之前，Shafik 特别声明并强调了一条根本原则：**《纽约时报》所有的填字游戏皆由人类纯手工创作，过去不是、未来也绝不会由 AI 生成**。 标准的纽约时报填字游戏遵循极其严格的构造与几何规范： - **180度旋转对称性**：将整个棋盘旋转 180 度后，黑白格的几何构型完全保持一致。 - **横纵全相交（Interlock）**：网格中的每一个字母都必须同时隶属于一个横向（Across）线索和一个纵向（Down）线索。 - **最短单词限制**：所有填入词项的长度至少为 3 个字母，严禁出现 1 或 2 个字母的过短词汇。 - **全局连通图（All-over Interlock）**：在图论意义上，棋盘必须是一个完整的连通图，黑块绝不能将网格分割为两个或多个孤立的子区域。 自 1993 年起担任《纽约时报》填字游戏主编的传奇人物 Will Shortz（威尔·肖茨）不仅把关着谜题质量，还是“美国填字游戏锦标赛”（American Crossword Puzzle Tournament, ACPT）的主办人。纽约时报填字游戏有着明确的周度难度递进机制：周一最为简单，旨在让玩家收获解题自信；随后难度与主题复杂性逐日攀升，至周五、周六达到烧脑巅峰；周日则转为尺寸更大（通常为 21x21）但难度介于周中水平的大型主题谜题，形成循环往复的智力挑战节奏。 ### [05:10](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=310s) 约束传播、分支因子与束搜索（Beam Search）求解 从计算科学的角度来看，填字游戏是一个典型且极度稠密的“约束满足问题”（Constraint Satisfaction Problem, CSP）。求解过程中存在显著的“约束传播”（Constraint Propagation）效应：当解题者率先确定某一部分的单词时，与之相交区域的候选字母便受到严格限定，从而极大地缩减了剩余网格中的潜在词库空间。 然而，填字游戏的搜索树极为复杂。随着网格推进，可能匹配的词汇组合呈现出巨大的“分支因子”（Branching Factor），导致搜索树的可能状态呈指数级急剧扩张。这种高密度的组合爆炸使得传统穷举搜索在计算耗时上无法承受。为了在有限时间内找到可行路径，专用的搜索优化算法——例如束搜索（Beam Search）——被广泛引入。束搜索通过在每一步评估中仅保留最具前景的有限个候选分支，剪除海量无意义的无效搜索路径，大幅提升了求解效率。 对于人类解题者而言，提升解题能力的路径通常包括： - **模式识别与快速检索**：长期训练大脑中的词汇召回与线索识别机制。 - **洞悉出题风格与文字游戏**：熟悉不同构造者（Constructors）常用的双关语、隐喻以及出题范式。 - **词汇量扩展与循序渐进**：通过广泛阅读拓宽知识面，并从周一的简单网格开始逐级迭代练习。 ### [07:13](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=433s) 从强化学习到超人类博弈：CartPole、深蓝与 Alpha 系列 在梳理 AI 征服游戏的演进史时，Shafik 提到了强化学习（Reinforcement Learning）的经典基准环境——倒立摆问题（CartPole）。这个让小车在轴线上通过反复尝试最终学会平衡直立木杆的模型，直观展示了 AI 如何通过与环境交互自我纠错以解决物理空间任务。 纵观现代 AI 发展史，各类棋类与电竞游戏曾多次见证里程碑式的技术突破： - **深蓝（Deep Blue, 1997 年）**：IBM 研发的国际象棋系统击败世界棋王加里·卡斯帕罗夫（Garry Kasparov），代表了基于大规模启发式搜索与专家评估函数的顶峰。 - **AlphaGo（2016 年）**：DeepMind 打造的围棋程序击败世界顶尖棋手李世石（Lee Sedol），创造了游戏 AI 史上最震撼的突破，证明了深度神经网络与蒙特卡洛树搜索（MCTS）结合后在复杂直觉博弈中的自学习威力。 - **AlphaZero**：摆脱了人类对弈棋谱的束缚，仅凭基础规则通过完全自博弈（Self-Play）迅速成长为横跨围棋、国际象棋与将棋的超人类霸主。 - **AlphaStar**：进军拥有即时、非完全信息特性的复杂实时战略游戏《星际争霸 2》（StarCraft II），其运行时类似大脑 MRI 扫描般的神经网络激活图，清晰展现了系统在微操与大局宏观决策上的深度思考逻辑。 ### [08:57](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=537s) 启发式求解与 Dr.Fill 的锦标赛突破 在现代大语言模型与生成式 AI 兴起之前，填字游戏的自动化求解依赖于高度精密的启发式规则体系： - **词频启发式（Frequency-based Heuristics）**：根据特定长度词汇在语言语料库中的出现频率，优先推测并放置高频词。 - **最小剩余值启发式（Minimum Remaining Values, MRV）**：优先处理那些受当前相交字母约束最严重、可用候选词数量最少的网格位置，促使约束迅速向全盘扩散。 - **候选词动态评分机制（Scored Candidates）**：对每一个可能填入的词项进行概率与置信度打分。 2021 年，填字游戏 AI 迎来了历史性的一幕：由 Matt Ginsberg 研发的知名求解器 **Dr.Fill** 赢得了美国填字游戏锦标赛（ACPT）的头名。Dr.Fill 本质上是一个冲突驱动的约束求解系统（Conflict Constraint-based Solver），它能够在并行计算中快速锁定棋盘上的冲突区域，结合机器学习评分与启发式树搜索，以极高速度求得全盘最优解。尽管 Dr.Fill 斩获全场最高分，但根据赛事组委会规则，AI 无法正式领取人类专享的锦标赛奖金。这一历史性突破不仅展现了传统约束求解技术的惊人极限，也为后续探讨通用认知与语言推理打下了坚实铺垫。 ### [10:35](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=635s) 填字游戏的奇特变体与薛定谔谜题 填字游戏的魅力不仅在于传统网格，还在于其千奇百怪的形式——有的被设计成鸭子形状，有的采用六边形单元格，还有一种被称为“薛定谔填字游戏”（Schrödinger's crossword）的绝妙设计。在这类谜题中，同一个线索可以同时容纳多种答案。一个著名的历史轶事发生在 1996 年美国总统大选期间：当时的大选在比尔·克林顿（Bill Clinton）与鲍勃·多尔（Bob Dole）之间展开，那期《纽约时报》填字游戏的一道关键线索设计得极为精巧，不论最终胜选者是克林顿还是多尔，这两个名字都能天衣无缝地填入网格，并且与其交叉的所有纵向横向单词全部成立。这种精湛的谜题构造，展示了人类在受限网格中创造多维语言可能性的极致智慧。 - **鸭子形状与六角网格谜题**：打破传统方形网格的几何约束，对空间布局与词汇填充提出了全新的挑战。 - **1996 年大选的薛定谔谜题**：无论填入“CLINTON”还是“BOB DOLE”，网格都能自洽成立，成为报刊填字游戏史上的经典传奇。 --- ### [11:16](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=676s) 密语填字游戏：LLM 无法逾越的高墙 随后，演讲转向了难度更高的“密语填字游戏”（Cryptic Crosswords）。与常规谜题直截了当的提示不同，密语填字要求解题者先“破解线索本身的隐藏规则”，然后才能知道要填入什么词。例如： - 线索“Father's drink”（父亲的饮料），答案是“POP”（既有父亲之意，又是汽水饮料）； - 线索“Fruit from wild among”（来自 wild among 的水果），答案是“MANGO”（将“among”一词进行“wild”乱序字母重组即得“mango”，且它确实是一种水果）。 更复杂的线索如《卫报》（The Guardian）曾经刊登的“Individual who's busy protecting a pit boss”（忙碌保护矿井工头的人），答案则是“DOGEATER”。如果想要入门，像 The Daily Cryptic 这样的网站会逐步拆解线索结构，引导玩家识别哪些词代表词根，哪些词是操作指示。 当前的大型语言模型（LLM）在面对密语填字游戏时几乎遭遇了“100% 的有效失败”。大语言模型在尝试解析这些线索时，往往会一本正经地输出详尽的推理长篇大论，解释为什么自己认为答案是某一个词，但最终结果却是错误的。常规填字游戏在很大程度上已被现代 AI 攻克，但密语填字依然是坚固的前沿阵地（frontier）。AI 经常混淆词性、误判线索类型，分不清眼前的线索究竟是字谜（anagram）、藏头藏尾词（hidden word）、双关语还是同音异义词（homophone），更不用说破解线索中深藏的文化隐喻了。 --- ### [13:50](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=830s) 空间求解器：利用图神经网络（GNN）攻坚密语线索 针对密语填字游戏在空间与隐喻层面的双重约束，演说者展示了自己构建的一款定制化解决方案——“Cryptic Solver GNN”（基于图神经网络的密语求解器）。如果将密语填字游戏抽象为一个图结构（Graph），网格交点与线索槽位化为节点，彼此之间的空间制约与字母共享化为边，便能够将这一语言难题转化为空间几何问题。 ```mermaid flowchart LR accTitle: Cryptic Solver GNN Architecture subgraph Input["输入特征"] A["密语线索文本"] B["网格空间几何"] end subgraph Embedding["特征嵌入与图构建"] C["Word Embeddings<br/>(语义与规则表征)"] D["Graph 拓扑网格<br/>(交叉约束约束图)"] end subgraph Model["GNN 空间推理"] E["Cryptic Solver GNN<br/>(空间-语义联合推断)"] end subgraph Output["解空间输出"] F["候选答案定位与填入"] end A --> C B --> D C --> E D --> E E --> F ``` 该系统结合了词嵌入（Word Embeddings）与网格谜题本身的空间几何属性，试图在多维嵌入空间中定位线索应该落在何处、如何与特定的几何空间产生关联。从可视化结果来看，谜题线索能够从图的嵌入空间中被推断出来。演说者指出，填字游戏的求解过程本质上是“约束传播”（Constraint Propagation）——随着网格中填入的字母越来越多，未解区域的搜索空间与约束随之收紧。尽管这是演讲者个人的实验性研究，但他卖了个关子，表示性能细节留待“第二部分”再深入探讨。 --- ### [14:56](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=896s) 智能体遭遇真正瓶颈：从 ARC-AGI 审视通用智能 这是否意味着只要套用 Agent 智能体框架，就能顺理成章地解决所有填字与逻辑难题？答案是否定的。演讲者借此引入了衡量 AGI 进展的核心标尺——ARC 测试（ARC-AGI，抽象与推理基准）。该基准由 ARC 基金会设计，专门用来探测 AI 距离真正的通用人工智能究竟还有多远边界。 ARC 题目由基于方格几何图案的空间谜题构成。每道题目仅给出极少数（通常只有几组）输入-输出示范样本，要求系统从中自发归纳出“涌现的规则系统”（emergent rule sets），并据此在问号区域绘制出正确的答案图形。ARC 对 AI 构成了极大的挑战，因为玩家必须“在玩游戏的同时自己摸索出游戏的底层规则”。AI 经常彻底误解游戏法则： - 搞不清什么机制触发了机关、什么移动了物体； - 搞不清究竟需要按动那个白色方块多少次才能打开通往下一个区域的门； - 陷入颜色与空间几何变换的误区——在这些谜题中，颜色不一定代表颜色本身，它可能代表移动方向、拓扑阻隔或状态转换。 人类凭借常识与直觉，在仅观察极少数演示后便能瞬间抓住潜在的隐式关联规则，而这恰恰是当前各类 AI 系统的死穴。 --- ### [17:46](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=1066s) ARC 基准的残酷代差：人类 100% 与 AI 骤降的滑铁卢 演说者用详实的基准演进数据，剖析了当代前沿大模型在归纳推理上的真实表现，揭示了模型面对未知规则时惊人的脆弱性： | 基准测试版本 | 发布时间 | 前沿大模型表现 | 人类普通受试者表现 | |:--- |:--- |:--- |:--- | | **ARC-1** | 数年前发布 | GPT-5/6（测试模型）达成 **97%** 饱和解题率（基本被攻克） | 接近 100% | | **ARC-2** | 2025 年 | GPT Soul（开启最大推理）达到 **~90%**；Claude Opus 5 达到 **92.5%** | 接近 100% | | **ARC-3** | 最新版本 | Claude Opus 5 暴跌至 **~30%**；GPT-5/6 骤降至仅 **~7%** | **100%**（旧金山街头随机非专业人群） | 在展示 ARC-3 数据时，演讲者的翻页笔（clicker）突然卡壳，他半开玩笑地一边摆弄遥控器一边揭晓谜底：“人类在 ARC-3 上的解题成功率是 100%。”值得强调的是，这一成绩并非来自经过专业数学训练的专家，而是在旧金山街头随机邀请的普通路人，他们在毫无特异知识背景的前提下，仅看几道示例就能解出几乎所有谜题。 演说者尖锐地指出，很多声称攻克谜题的 Agentic AI 系统根本不是“泛化”（generalization），它们只是被定向微调，能够凑巧解决某一特定谜题，但面对下一个全新规则的谜题时便立刻崩溃。如果系统自称具备通用智能，它就必须具备人类这种凭极少范例触类旁通的能力。 演讲的尾声充满反思：英伟达 CEO 黄仁勋（Jensen Huang）曾在财报电话会上被问及“通用人工智能（AGI）何时能够实现”，黄仁勋当时宣称“它已经实现了”。然而，ARC-3 以及复杂的语言填字游戏无情地戳破了这一神话——如果普罗大众在街头能轻松搞定的空间与语义隐喻，最顶尖的超算 AI 依然无法解答，就证明“我们离真正的 AGI 依然存在巨大的鸿沟”。尽管手中的翻页笔在最后关头彻底罢工，演说者依然风趣地总结道：无论是填字游戏还是几何谜题，游戏不仅是一种消遣，更是丈量认知差距、检验 AI 迈向真实通用智能最绝佳的试验场。 ## Notable quotes > [00:54](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=54s) humans are still good at doing things that very powerful AI still is not capable of doing — Shafik Quoraishee > [02:30](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=150s) 《纽约时报》所有的填字游戏皆由人类纯手工创作，过去不是、未来也绝不会由 AI 生成 — Shafik Quoraishee > [11:16](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=676s) Father's drink > [11:16](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=676s) Fruit from wild among > [11:16](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=676s) Individual who's busy protecting a pit boss > [11:16](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=676s) 100% 的有效失败 — Shafik Quoraishee > [14:56](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=896s) 在玩游戏的同时自己摸索出游戏的底层规则 — Shafik Quoraishee > [17:46](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=1066s) 人类在 ARC-3 上的解题成功率是 100%。 — Shafik Quoraishee > [17:46](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=1066s) 它已经实现了 — 黄仁勋 > [17:46](https://www.youtube.com/watch?v=IMLhEbZZWz8&t=1066s) 我们离真正的 AGI 依然存在巨大的鸿沟 — Shafik Quoraishee ## Mentioned Shafik Quoraishee, The New York Times（《纽约时报》）, Will Shortz（威尔·肖茨）, Matt Ginsberg, Dr.Fill, Garry Kasparov（加里·卡斯帕罗夫）, Lee Sedol（李世石）, Jensen Huang（黄仁勋）, Bill Clinton（比尔·克林顿）, Bob Dole（鲍勃·多尔）, Deep Blue（深蓝）, DeepMind, AlphaGo, AlphaZero, AlphaStar, Pac-Man（《吃豆人》）, Wordle, Spelling Bee, Connections, CartPole, StarCraft II（《星际争霸 2》）, ARC-AGI, The Guardian（《卫报》）, The Daily Cryptic, American Crossword Puzzle Tournament (ACPT), Claude Opus 5, GPT-5, GPT-6, GPT Soul, NBA, Business Insider, MTV ## Video description Shafik Quoraishee - Senior Games/ML Engineer, The New York Times Artificial Intelligence has achieved remarkable progress in language modeling, reasoning, and problem-solving—yet the New York Times Crossword remains one of the most enduring challenges for even the most capable AI systems. This talk examines how crosswords serve as a fascinating testbed for advancing cognitive flexibility, linguistic reasoning, and multimodal understanding in AI. While massive models trained on terawatts of computation and vast text corpora can summarize articles or generate code, they often falter at the nuanced, context-rich inferences required to solve human word puzzles. The crossword demands something deeper: the ability to interpret puns, cultural references, idioms, and abstract associations—skills that reveal the gap between specialized AI and true general intelligence. Through exploring the way humans effortlessly connect disparate ideas to decode clues, we gain insight into the creative and associative reasoning that current systems still struggle to emulate. Through examples from the New York Times Crossword and its variants, this session will illustrate why such linguistic puzzles can be more than a pastime—the can be a benchmark for measuring progress toward Artificial General Intelligence and for developing APIs and agentic frameworks that move AI closer to genuine cognitive versatility. [Source](https://www.youtube.com/watch?v=IMLhEbZZWz8)

[0:54] Shafik Quoraishee: humans are still good at doing things that very powerful AI still is not capable of doing [2:30] Shafik Quoraishee: 《纽约时报》所有的填字游戏皆由人类纯手工创作，过去不是、未来也绝不会由 AI 生成 [11:16] Father's drink [11:16] Fruit from wild among [11:16] Individual who's busy protecting a pit boss [11:16] Shafik Quoraishee: 100% 的有效失败 [14:56] Shafik Quoraishee: 在玩游戏的同时自己摸索出游戏的底层规则 [17:46] Shafik Quoraishee: 人类在 ARC-3 上的解题成功率是 100%。 [17:46] 黄仁勋: 它已经实现了 [17:46] Shafik Quoraishee: 我们离真正的 AGI 依然存在巨大的鸿沟

Shafik Quoraishee: The crossword demands something deeper: the ability to interpret puns, cultural references, idioms, and abstract associations—skills that reveal the gap between specialized AI and true general intelligence. Shafik Quoraishee: Through examples from the New York Times Crossword and its variants, this session will illustrate why such linguistic puzzles can be more than a pastime—the can be a benchmark for measuring progress toward Artificial General Intelligence [2:26] Shafik Quoraishee: never will be AI generated [10:35] Father's drink [10:35] Fruit from wild among [10:35] Individual who's busy protecting a pit boss [10:35] Shafik Quoraishee: While standard American-style crosswords are largely considered solved by computational heuristics, cryptic crosswords represent a true frontier for artificial intelligence. [14:56] Shafik Quoraishee: Both ARC tasks and crossword grids require an agent to derive unseen rules and propagate local actions across a shared coordinate space. [17:28] Shafik Quoraishee: Everyday people tested off the street in San Francisco scored 100% without any specialized preparation or domain knowledge, picking up the underlying logic from just a handful of few-shot examples. [19:47] Shafik Quoraishee: Tracking how models handle the nuanced rules and shifting constraints of such games provides a clear, entertaining lens into where current AI excels and where the critical gaps toward cognitive versatility still need to be bridged.

[0:04] Today I'm going to be talking about um advancing  the understanding of general intelligence through [0:08] the New York Times crossword. Um a lot of things  that are mixed together into one, but uh I think [0:16] we'll we'll we'll we'll we'll we'll break it down.  And so um just a little bit of introduction to [0:21] myself first. Um I'm a staff engineer uh editor  editor at the New York Times. I work on our cross [0:28] word puzzles and our games and a few other uh  pieces of our infrastructure. Um, prior to the [0:34] times I worked at the NBA, Business Insider and  MTV. So, I have a background in media and uh I was [0:41] formerly an MLAI researcher working in the area of  computer vision and I took that into the area of [0:46] games moving forward. Um, there's a YouTube link  for myself and my ex link. Um, okay. All right. [0:54] So, just a little bit about the times. Um, how  many people play the New York Times games here? [0:59] Probably a few people. Okay. So, as you know that  we're uh games we're well known for, including [1:05] the crosswords, which since 1942, that was um our  staple game includes Spelling Me and Wordle, just [1:11] as a reminder. Um, and now jumping a little far  ahead into the AI portion of uh everything. Um, [1:19] the the concept the main principle that I'm going  to be discussing the crosswords through is the [1:24] idea of Moravex paradox. So I don't know how many  people have heard of this paradox before but uh [1:29] in AI in the world of AI it basically means that  humans are still good at doing things that very [1:34] powerful AI uh still is not capable of doing.  Um and so I wrote a a Times article about this [1:41] um last year when it came to our game connection.  So if anyone's interested you can uh check that [1:46] out. Um and then why am I doing this? Um we I etc.  uh because I'm really interested in games and AI [1:53] and uh really games is the place where AI really  started to shine and become um a thing uh in the [2:01] 50s and 60s and it became uh more powerful um as a  result of a lot of iteration over uh the world of [2:10] games. Uh so this is uh for example uh Pac-Man.  So, if you're familiar with how Pac-Man works, [2:15] this is a uh one of the simplest AIs that was ever  put into production called a finite state machine. [2:21] Um, I'm sure people who are into computer science  here recognize this, but this is basically the way [2:26] Pac-Man works. And now I'm going to go into the  crossword. So, let's go into the topics I'm going [2:30] to be talking about. I'm going to introduce you to  crossword puzzles. How many people here play cross [2:34] word puzzles? Anybody? Okay, a few. Good. Um, I'm  going to talk about how people solve crosswords, [2:41] current AI systems and the crosswords, uh, plus  my own system that I work on because I do research [2:46] on the in that area and then hallmarks of, uh,  general intelligence, a little bit about the ARC [2:51] prize. Um, so a few things to keep in mind and  and you might not have known this, but I if I'll [2:57] iterate them. Uh, the New York Times crosswords  are all humanmade. Um, they are not and never [3:02] will be AI generated. So, I just have to put that  out there in case you're not sure about that. Um, [3:07] we aren't going to go into crossword construction,  but it's a really interesting area in its own [3:11] right. Um, and there'll be a part two of this  talk to talk about that. Um, AI changes super [3:16] quickly and everything that I'm saying here could  be outdated real quickly. Um, and the history is [3:21] here something I am really interested in kind  of elucidating. So that's uh why I want to keep [3:27] this as a historical record. So what is a cross  word puzzle? This is a cross word puzzle. Um, [3:32] it's a grid made of across and down entries that  intersect through shared letters. and the clues [3:37] give you ideas of what should fit in the grid.  So, pretty straightforward. The New York Times [3:41] cross word puzzle is a special kind of cross word  puzzle. Uh, one our characteristics is that it has [3:46] 180°ree rotational symmetry. So, if you rotate the  cross word on its side, it looks pretty much the [3:51] same uh geometry- wise. Uh, every letter appears  both in an across and a down clue. Um, that's not [3:57] true of all crosswords. Um, all entries must be  at least three letters. So, no super tiny letter [4:02] words, two-letter words. And all over interlock  means that the crossword grid has to be completely [4:08] connected. The black squares can't separate the  grid into two parts or more in any other place. [4:13] And if you're familiar with connected graphs and  graph theory, this is generally the idea. Um, so I [4:19] don't know if anyone recognizes this guy, but this  is Will Schwarz, the editor of the New York Times [4:24] crosswords. Um he's been there since at the times  since uh 1993 and he's uh been the host of the [4:31] American Crossword Puzzle Tournament, which is an  awesome tournament. If you're into crosswords and [4:35] want to uh play competitive games with other cross  word enthusiasts, you should play that. Um and I'm [4:40] I bring that up because we're going to be talking  about solving crossword puzzles. So this is just [4:45] the difficulty distribution of the New York Times  crosswords. If you've played them, you know that [4:49] the Monday crossword is the easiest cross word,  and everybody loves playing a Monday to make [4:53] them feel good for the week. It all escalates  until Saturday um by increasing grid difficulty [4:59] and theme difficulty and people like really find  the Saturday the Friday one pretty tough until it [5:04] starts to go back down to an easier grid on Sunday  with a larger grid space and then uh back to [5:10] Monday. So I'm going to start off with a few terms  and uh physics or physical characteristics of the [5:16] New York Times crosswords or any crosswords. Um  one of them is called constraint propagation. So [5:20] if you ever play a cross word puzzle, this is sort  of an exaggerated example I created. But if you [5:25] solve a part of a cross word puzzle, what happens  is it becomes easier to solve the rest of the [5:29] crossword puzzle becomes the rest of the crossword  puzzle becomes more constrained because the number [5:33] of clues um potential clues reduces in the clue  space. Um so uh that's because and that relates [5:41] to the so-called branching factor of a cross word  puzzle. So as you uh search through the crossword [5:46] puzzle space, this is another visualization of  what happens as the number of possible answers [5:52] increase in the space of the crossword puzzle.  So you see this sort of tree diagram becomes [5:56] exponentially large um as the number of uh as  the number of iterations of words goes on and [6:02] you can sort of find a path to solving in this  red path but it be it's very dense a dense tree [6:07] and because it's a a dense tree is very difficult  computationally to solve in a reasonable amount of [6:13] time. Specialized algorithms were developed uh  such as beam search um in order to solve dense [6:19] tree problems. Is anybody familiar with the idea  of beam searching? Um, okay, a few people here. [6:24] So, that means this is a way of optimizing uh your  path through searching of a cross word puzzle or [6:30] any kind of dense tree space to find the best way  through uh by pruning a lot of unnecessary choices [6:36] um in your solver. Okay, so how do people get  um good at solving crosswords? Like anything, [6:41] you practice a lot. Uh right, so you build your  recognition and recall. Um you learn common clue [6:47] styles. This is pretty important because uh  constructors have certain styles in which they [6:51] uh create clues for the crosser puzzles and word  play formats. You should read more to expand your [6:56] vocabulary and you should do that anyway not just  for the crosswords but that helps with crosser [7:00] puzzles. Um and then you start with the easier  grids like the Mondays and the other you know [7:05] easy grids and you practice your uh getting better  over time with iteration over the easy grids. So [7:13] a little bit about um crosswords and AI. So this  is what the meat of the presentation is going to [7:17] be talking about. Anybody recognize this uh here  the simulation? Okay, great. If you're into RL, [7:23] this is the cart pole problem. One of the most  simple RL problems out there. This is an actual [7:27] simulation of a cart trying to balance a pole on  its um axis. So that after several iterations, [7:33] the cart can balance the pole, but it does take  a lot of iteration. It's a classic reinforcement [7:38] learning problem that trains AI of a real physical  space to um solve a problem. And um basically [7:45] games and AI have been pretty much solved in many  cases through reinforcement learning throughout [7:49] the history the recent history of AI and gaming.  So this uh Deep Blue um that actually wasn't [7:55] reinforcement learning per se, but Deep Blue was  the uh chess uh system in 1997 that beat Gary [8:02] Kasparov, the chess master uh grandmaster at the  time. Um Alph Go, everybody probably remembers in [8:08] 2016 Alph Go winning Go against Lisa Doll. um one  of the biggest upsets actually and most surprising [8:17] uh in AI in game playing history because Go is  considered a challenging game to play. So it [8:22] proved that AI can train itself to play a game  repeatedly well and Alpha Zero essentially does [8:28] that. It can train itself. It doesn't need human  guidance. It can learn the game and eventually [8:33] it can um beat pretty much everybody it plays  at the game. This is also a a version of Alph [8:39] Go called Alpha Star uh which is uh meant  to play Starcraft. And as you can see that [8:44] uh the neural activations um in the game kind of  like an MRI show you where the game is thinking [8:50] where the AI is thinking rather and um making  moves in the game accordingly. So, so a few things [8:57] about again back to crosswords as we're talking  about um crosswords uh essentially um are were a [9:04] heristic problem before they became an AI problem.  So there are certain heristics that are associated [9:09] with solving a cross word efficiently. You have  frequency based heristics. So basically words that [9:14] appear more frequently in certain lengths should  go in certain places. Um minimum values is another [9:20] heristic that you would use to solve as I said uh  as you solve a crossword um it becomes easier to [9:26] find places where you can put more words because  the constraints propagate through the rest of the [9:30] crosswords that's called the MRV heristic and um  scored candidates basically there's different ways [9:35] of scoring potential words and what you have here  is a sort of solver that I created to simulate [9:41] solving how an a mechanical heristic algorithm  or quote AI unquote would solve the crossword [9:47] puzzle. So, um, in 2021, the American cross  word puzzle tournament was actually solved by a, [9:53] uh, AI solver called Dr. Phil. And Dr. Phil is  is a basically a conflict constraintbased solver [10:01] which looks for conflicting areas in a crossword  board and simultaneously in parallel solves these [10:06] areas using all these heristic techniques. Um,  and it was a pre-generative AI system. It's more [10:11] of like a a system which then um kind of like  finds optimal spaces to fill words in. And uh it [10:19] uses a little bit of ML and it's evolved since  then, but um it didn't it won the tournament, [10:24] but it did not get the prize because only humans  are supposed to get prizes for winning tournaments [10:29] uh according to the rules. So just a little bit of  a you know kind of um display of why this problem [10:35] is interesting is because there's different kinds  of crosswords. So there's a duckshaped cross word [10:39] puzzle. There's a hexagonal cell-shaped cross word  puzzle. There's another cross word puzzle called [10:44] Schrodinger's cross word which is a special kind  of cross word which can have multiple clues. So [10:49] in 1996 the winner of the presidential election  um was Bill Clinton. But if in just in case it [10:55] was Bob Dole this uh uh who was the other who was  the um you know who was running against this cross [11:01] word clue could have filled up uh Bill Clinton or  Bob Dole and it still would have allowed the cross [11:06] word puzzle to uh survive and be a viable cross  word puzzle. thought it was a little interesting [11:10] piece of trivia because that's what um we did.  Um so now we move to another type of crossword [11:16] called the cryptic crossword. Cryptic crossword  has anybody ever played cryptic crosswords before? [11:22] Okay. So if so cool. So if you one or two people  cryptic crossword puzzles are crossword puzzles [11:27] where you actually have to solve the clue  itself uh before you can solve what goes in [11:31] the puzzle. So this is basically a type of clue.  So father's drink the answer would be pop. So you [11:36] have to you know get it. Um uh another one is  fruit from wild among. Uh the answer would be [11:44] uh mango. Um so basically the clue itself tells  you how to solve the what the word should be. [11:51] So wild among means like you scramble the word  among and you get mango and mango is a fruit. [11:56] So and then it gets more complicated because those  are simple cryptic clues but the cryptic clues get [12:01] uh more advanced like individual who's busy  protecting a pit boss. The answer is doineer. [12:05] comes from the Guardian. Um, one of their paper,  one of the clues from a while back. Um, there's [12:12] a a site called The Daily Cryptic in case you're  interested in solving cryptic crossword puzzles [12:16] that shows you how to solve a clue and break it  down. Um, you can take a clue and uh it'll show [12:21] you where you should think about certain parts of  the clue and then, you know, um, eventually you'll [12:26] you'll get good at solving cryptic crosswords,  maybe. Um, but the point is that LLMs fail to [12:32] solve cryptic crosswords effectively 100% of  the time. And that's because cryptic crosswords, [12:38] um, as I'm going to lead into, uh, are basically  a system where you have to infer what the rule of [12:44] the clue is to solve the crossword. And this is an  example of an LLM trying to solve that particular [12:50] clue for that I showed you before, and it didn't  get it, but it did try to explain why it thought [12:54] um, the answer was a particular answer. And then,  um, yeah. So basically the idea is the reason I'm [13:01] focusing on cryptic crosswords is because solving  the regular cross word puzzles is kind of the [13:06] solved problem but the cryptic crossword puzzles  is kind of a little bit of a what I would call a [13:10] frontier um because uh there are cryptic crossword  data sets but um many times the more complicated [13:17] clues are are uh more difficult for AI to solve.  Um AI can mclassify, misapply word types. It can [13:25] sometimes not be sure if something is supposed to  be an anagram type clue, a hidden word type clue, [13:30] homophone type clue, etc. And so also cultural  references. I mean AI gets better at this over [13:35] time. And I'm saying, you know, like the  language models, the foundational models, [13:39] they get better at solving puzzles, but still  the cryptic clues can um confuse AI, whereas [13:45] many humans who get good at solving cryptic clues  can actually figure these out in a better. And [13:50] so a way to solve this is for example if you are  building a sort of spatial solver. This is a way [13:56] you can if if anybody has done things with graph  neural networks before you can see where I'm going [14:01] to. If I represent the cryptic crosswords as a  graph uh then I can use a graph neural network [14:06] system. Here's an example of one that I built,  a custom one called a cryptic solver GNN. Um, [14:12] which is specifically designed to unravel  cryptic clues. And it uses word embeddings [14:17] and it uses spatial geometry of the puzzle  itself to figure out where things should go, [14:22] um, and how the cryptic clue associates with a  particular geometric space. This is sort of like [14:28] a visualization of, you know, what happens with  the solving. It's just kind of meant to be pretty, [14:32] but it's basically showing you that the  clue can be kind of inferred from the [14:36] embedding space of the of the graph. Um, and  again, it's a spatial problem. So, it kind of [14:41] tries to figure out where the clue would fit  and then finds the appropriate clue. Um, so, [14:47] um, you know, how does this perform? This is on  my own little experiment, but we can talk about [14:52] that at some other point in the future if you're  interested. Um, wait for a part two to that. Um, [14:56] but now all these things, can agents just solve  all these things, right? Can you put an agent [15:01] against the cross word and it solves the cross  word? Well, yeah, probably. But um that's not [15:06] I don't know. Is that the end of the story? Well,  maybe not because it so everybody's familiar with [15:10] the ARC tests here, right? A r agi. Uh okay, if  you're not, these are really important because the [15:16] arc tests kind of are the limitation at which um  we sort of decided that uh general intelligence, [15:22] the AI is not quite at the boundary of general  intelligence. So the ARK foundation um creates uh [15:29] uh these tests uh these spatial puzzles with sort  of emergent rule sets for um AI to solve. So each [15:35] puzzle is uh has just a few example pairs and then  you have to kind of figure out what the pattern is [15:40] and the AI or whatever will a human will figure  out what's in that question block there. Um but [15:47] this benchmark is a real and it's a very important  one because again thi this benchmark um is a kind [15:54] of has many iterations and has essentially stumped  AI for quite quite a bit and I'm going to go into [16:01] the details there but this is a ge a square  geometry based puzzle-shaped benchmark and so [16:07] the reason it's um difficult it has been difficult  for AI to solve this perfectly is because this ben [16:13] this the puzzles in this benchmark are such that  you have to kind of figure out what the rules of [16:19] the game are as you play the game. So, um the arc  failures um that AI typically has is that it fails [16:26] to interpret the rules in the game appropriately  to figure out like for example what opens what, [16:31] what moves what, how many times you have to  press that white thing in order to open the door [16:36] so it can go through uh another area. Um, this is  important because um, there's a lot that has to do [16:42] with object transformation and rules about color  space. Colors don't always mean colors. Colors [16:48] can mean direction. Colors can mean something  different. You have to kind of figure out what [16:52] happens what um, with a color in order to move the  puzzle forward. So, um, and again, this is where [16:58] indirect relationships are difficult uh, for AI to  solve, but people can identify the underlying rule [17:05] after seeing only a handful of demonstrations. And  that's where the really important piece is. Um if [17:10] if AI is supposed to be general then it should be  up to the level where people can generalize. So [17:17] like crosswords arc tasks rely on grid structures  where local actions affect patterns. And so again [17:22] this pattern inference um is sort of what the  goal of the puzzle solving is. And it is related [17:28] to as I said the constraint propagation of a cross  word puzzle. For example, the like as the puzzle [17:35] um gets more and more solved, the constraints  reduce. And that is that is one example of a [17:40] type of rule that an AI system has to contend with  if it's being built to solve um th this puzzle or [17:46] any any puzzle. And so moving forward, here's sort  of how AI is doing against the ARC benchmarks as [17:52] it is. Um the first ARC benchmark came out several  years ago. AI has basically crushed it. The GPT 56 [18:00] soul model um is has saturated the benchmark  at um 97% solve rate. Um the ARK2 benchmark [18:06] which came out in 2025 um has about a 90% solve  rate with GPT soul um uh with max reasoning and [18:14] about um 925 with quad opus 5 right recently uh  discovered results but the AR 3 quad opus 5 has [18:22] about a 30% solve rate on AR 3 um and GPT56 has  about a 7% solve rate on AR 3. What percentage [18:32] do you think that human beings have on the R3  benchmark? Does anybody can take a guess? Well, [18:39] I'll just tell you that if I can move this clicker  forward. We're just waiting for this clicker to [18:46] move forward. I don't know if this clicker is  dead or not, but it's not moving forward. But [18:51] I will tell you that human beings on the next  slide have 100% solve rate on the ARC uh AGI uh [18:57] 3 benchmark. And um that's because um and and this  is not with specialized training. Uh there was a [19:04] test that was taken such that humans on in the  street of San Francisco without any specialized [19:09] knowledge were able to solve this with only a  few examples. Pretty much all the puzzles and [19:14] Agentic AI and other uh AI systems uh can you know  have been uh trained against the benchmark but [19:22] really the agentic systems can only solve certain  puzzles. For example, that's not generalization. [19:27] the agentic system can solve a particular puzzle  and then it will fail to solve the next puzzle [19:33] um in many cases and so if the system is supposed  to be uh general then it should be able to solve [19:38] all puzzles with only a few examples. So um and  again this uh the clicker is like you know doing [19:47] something funny right now. But the idea is that  um over time the over time we see that uh we still [19:53] have a large gap between um you know where we are  when we talk about having general intelligence [19:59] and where we need to be uh with AI systems. And  I think there was a point when Jensen Wong had [20:05] actually said that um he was asked in a a call  whether um whether AGI when would AGI be solved [20:13] when would general intelligence be solved and  he said it's already happened but as we can see [20:17] this benchmark kind of demonstrates that it's not  quite happened yet. You know if average everyday [20:22] people can solve things that AI systems cannot  solve. And so, um, again, I'm I'm fighting with [20:28] the clicker to move this forward, but I'm pretty  much at the end of my presentation. And I just [20:33] wanted to leave you with that, um, just, uh, the  idea that, um, you can infer a lot from games and, [20:40] uh, different complexities of games, including  crosswords, as to where AI might be at. And things [20:45] like the crosswords are good tests of how an AI  system can generalize in the future. and maybe [20:51] um just a fun way of kind of seeing that yeah  there are gaps that still need to be filled. So [20:57] um that's the end of my presentation. Um we'll  take questions later if everyone's anyone's [21:01] interested but uh this clicker has stopped  and thank you very much. Appreciate your time.