Google Research Head Rejects Hinton AI Vision, Urges Active Alignment

谷歌研究负责人驳斥Hinton的AI愿景，呼吁积极推进对齐 被动的技术惯性必然导致失败，除非社会主动解决认知信任与价值对齐困境，而不是将先进AI视作不可避免的主宰。 Pilar Manchón探讨了人类快速的技术蜕变，并警告称美好的未来不会自动实现，需要深思熟虑且以人为本的指引。她详细介绍了AI赋能的变革性科学突破——例如AlphaFold、AlphaProteo以及像Genie这样的生成式世界模型，同时探讨了对自主智能体建立信任的心理架构。Manchón剖析了内部与外部价值对齐的复杂性（包括迈达斯问题），并反驳了知名AI领军人物的宿命论范式。最后，她引用Alice Walker的话，呼吁与会者发挥能动性，成为未来的积极构建者，而非被动的旁观者。 当前围绕 AI 的公共讨论在文明崩溃与盲目乐观两极之间分化，面对这场由算力、能源、基础设施、数据与人才驱动的历史性变革，需要建立批判性思维而非被动接受极端叙事。 (0:04) 人类正迎来蜕变为“蝴蝶还是飞蛾”的关键转折期，这一走向绝不能依靠自动驾驶的惯性，而必须坚持以人为本的导向。 (3:40) AI 正深度赋能科学研究：深度学习系统预测出 220 万种新晶体结构，计算磁体成本降至 20%；AlphaFold 将耗时数年的蛋白质结构解析推进至数周解决 2 亿种，并为 AlphaProteo 从头设计蛋白质奠定基础。 (8:01) 生成式技术正从静态提示词演进为可实时交互的世界模型 Genie，配合 AI Co-Scientist 等智能体将长达数月的文献整合压缩至数小时。 (11:27) 随着自主智能体普及，核心瓶颈从算力转向多维度的信任校准，并需直面外部对齐的“迈达斯问题”与内部对齐挑战，以避免技术失控，每个人都应成为未来的架构师。 (15:05)

Speaking at Ai4, Pilar Manchón argues that humanity faces an unprecedented structural transformation driven by artificial intelligence, requiring active human agency rather than passive reliance on technological inertia. Scale of Transformation and Scientific Disruption (8:01) Humankind stands at the threshold of a rapid structural shift powered by foundational elements including compute, energy, infrastructure, data, and talent. Rather than functioning as an isolated discipline, artificial intelligence operates as an underlying catalyst across multiple domains, including education systems, human-computer interaction, creative arts, and physical sciences. In material sciences, deep learning systems have computationally predicted 2.2 million new crystal structures, with 395,000 verified as thermodynamically stable, while computing magnets have been produced in the UK at 20% of their historical manufacturing cost. Structural biology exemplifies this acceleration. AlphaFold resolved the 3D structures of 200 million cataloged proteins in weeks—a task that historically required a four-to-five-year doctoral thesis per single protein structure. Google has advanced this capability with AlphaProteo, transitioning from mapping natural proteins to designing novel, de novo proteins for scientific and therapeutic use. Computational optimization is similarly applied to semiconductor architecture, aviation contrail management, and municipal disaster forecasting. Interactive World Models and Agentic Workflows (11:27) Generative capabilities have advanced beyond static text, image, and video generation toward dynamic environments. Google DeepMind's Genie allows users to enter prompts and navigate the generated video interactively in real time, offering consistent world models applicable for training embodied robots and running spatial simulations. In enterprise workflows, tools like the AI Co-Scientist compress literature reviews from months to hours by sweeping research papers, validating findings, and synthesizing hypotheses with user notes. Concurrently, autonomous agents are shifting into consumer logistics. Systems such as AI Mode can parse rental listings under price constraints and contact landlords directly. However, these systems remain imperfect and capable of generating convincing synthetic outputs, which Manchón warns requires persistent user vigilance and critical evaluation rather than blind trust. The Architecture of Trust and Biological Efficiency (15:05) As autonomous agents proliferate, Manchón identifies trust rather than compute power as the primary bottleneck. Human trust varies significantly across applications, demanding different thresholds for financial or educational tasks compared to conversational tools. Furthermore, physical embodiment shifts human psychological comfort, and conversational agents risk blurring boundaries because humans are evolutionarily predisposed to treat natural language speakers as conscious entities. This trust dynamic coincides with engineering bottlenecks. While industrial hyperscale data centers require megawatts or gigawatts, the human brain executes perception, reasoning, and memory on roughly 23 watts of power. Manchón notes that biological mechanisms, such as high-density DNA data storage, provide potential blueprints for altering the carbon, thermal, and architectural limits of modern silicon arrays. Inner, Outer, and Midas Alignment Dilemmas (20:11) Ensuring models act according to human values introduces theoretical and practical obstacles split into outer and inner alignment. Outer alignment contends with the difficulty of defining collective values amidst pluralism, systemic bias, moral uncertainty, and reward hacking. Manchón illustrates this via the 'Midas problem': an AI can act like a helpful 'fairy godmother' that interprets intent to elevate outcomes, or an 'evil genie' that fulfills literal commands to the user's detriment. Inner alignment addresses the technical enforcement of these multidimensional human values within model architectures once defined. To institutionalize safety against extreme risks, Google operates under published AI Principles and its Frontier Safety Framework. However, Manchón observes that general comprehension of human value alignment remains alarmingly low across engineering and business ecosystems, requiring a broader conceptual reframing around human dimensions. Contrasting her perspective with frameworks from Geoffrey Hinton, Joanna Bryson, and Yann LeCun, Manchón rejects the notion that humans should accept a subordinate status to machine intelligence. Citing Alice Walker, she concludes that individuals must exercise agency within their respective domains to act as deliberate architects of the technology rather than passive spectators.

Google的Pilar Manchón在Ai4大会上提出，人工智能正引发人类历史上规模最深广、速度最迅猛的结构性剧变；她强调人类决不能被动等待技术自然演变，而必须主动把握技术能动性，将系统牢牢锚定在以人为本的价值体系与信任架构中，成为未来的“架构师”而非旁观者。 转型规模与人本主义抉择 (0:04) 当前围绕人工智能的公共讨论呈现两极分化，图灵奖得主与诺贝尔奖得主之间亦存在针锋相对的观点，一方认为文明面临崩溃风险，另一方则主张无需过度担忧。演讲者指出真实情况处于两者之间，人类正站在依赖算力、能源、基础设施、数据和人才五大核心支柱驱动的历史性转折点上，这一变革正以跨越数量级的非线性速度重塑教育、科研和交通等领域，并因合成内容的扩张而模糊了客观事实与虚构信息的界限。 面对这场被称为“蜕变”的巨变，人类社会面临走向光明（蝴蝶）还是自毁（飞蛾）的关键抉择。现场调查显示没有任何观众认为仅凭技术惯性或被动旁观就能自然换来理想结局。演讲者明确主张，对抗虚假幻觉与盲目乐观的唯一途径是保持批判性思考，坚决避免将技术演进演化为脱离人类利益的孤立复杂度竞争或异化为不可控的“异质智能”，确保人工智能始终服务于人类社会的福祉。 驱动科学突破与现实任务落地 (8:01) 在材料与生命科学领域，深度学习展现了颠覆传统科研周期的巨大潜力。根据演说引用的数据，系统已成功计算预测出220万种新晶体结构，其中395,000种被证实具有热力学稳定性；英国研究团队借此将计算专用磁体的制造成本降至原先的20%。在结构生物学领域，以往一名博士生需耗费4至5年才能解析单个蛋白质结构，而AlphaFold在数周内便预测解析了全部已编目的2亿种蛋白质三维结构，Google更进一步开发出AlphaProteo以实现从头定制蛋白质设计。 在日常应用与学术研究层面，自主代理与生成式模型正在改变工作流形态。DeepMind推出的Genie模型实现了实时可交互的三维世界模型生成，支持具身机器人与空间导航训练；面向大众的AI Mode可直接代劳租赁房源筛选与房东对接等繁复事务；AI Co-Scientist能在数小时内完成传统需耗时数月的文献检索与假设验证。不过演讲者同时提醒，这类系统现阶段仍存在不完善与不可靠之处，用户仍需保持警惕与审慎核查。 智能体时代的信任与认知边界 (15:05) 随着行业步入自主代理时代，系统落地面临的核心瓶颈已从单纯的算力堆叠转移到人机信任的建立。演讲者强调信任具有多维差异性：将系统接入涉及个人资产的银行账户，与用于医疗诊断、儿童教育或体育训练，其信任阈值截然不同；一旦代理获得机械实体并在物理空间中穿行，人类的心理感知与舒适度也将发生剧烈变化。此外，开发者侧重的技术验证指标往往与普通用户在易用性、伦理透明度及现实相关性层面的诉求存在明显错位。 认知科学研究正在探索不同人际信任模型对人机交互的影响。传统计算机遵循确定性代码，而现代生成式模型通过自然语言进行互动，极易触发人类将对话方视作“人”的演化心理机制。然而此类智能体仅仅具备拟人发音与语言表达能力，本质上并不具备真正的人类意识与同理心，如何化解这一心理投射张力是构建可靠智能体工作流必须解决的课题。 能效启示与价值对齐的双重困境 (18:38) 人脑在深度推理、感知与记忆运行时仅消耗约23瓦功率，这与当前动辄消耗兆瓦乃至吉瓦级电力的工业数据中心形成鲜明反差。演讲者指出，探索生物机理对改进计算架构、散热限制以及开发高密度DNA生物数据存储具有关键借鉴意义。与此同时，技术部署迫切需要面对“人类价值对齐”这一认知普及率极低的现实瓶颈，Google据此设立了AI原则与前沿安全框架（Frontier Safety Framework）以应对极端技术风险。 对齐难题在技术上被划分为外部对齐与内部对齐。外部对齐涉及价值定义的多元化与“迈达斯问题”（即系统机械执行字面指令带来恶果的“邪恶精灵”现象，对比深谙人类本意的“神仙教母”机制），需处理规范指定偏差、奖励黑客行为以及将浅层表面偏好误认为核心深层价值等问题；内部对齐则聚焦于即便价值达成共识，底层模型架构如何在执行中被严格约束。因此，演讲者建议采用更接地气的“人类维度”来重新定义这一复杂议题。 超越从属范式与重获主体能动性 (21:54) 针对高级人工智能与人类关系的构想，业内存在数种代表性假说：Geoffrey Hinton曾提出类似“AI母亲”的照护与管辖隐喻，Joanna Bryson主张将系统置于类似于公司或城市治理的制度框架中，Yann LeCun则将其单纯界定为可解决的工程问题。Pilar Manchón对此表达了明确异议，断然拒绝接受人类必须在未来科技生态中沦为“从属存在”或弱势地位的预设假说。 演讲者援引作家Alice Walker的名言“人们放弃力量最常见的方式就是认为自己毫无力量”，呼吁各领域的专业人士与构建者认识到自身所掌握的知识与能动性。在大模型迅速渗透各行各业的当下，每个人都必须做出根本抉择：是充当被动接受结果的旁观者，还是主动承担起设计规则、监督技术落地的未来架构师。 本场演说明确了人工智能在科研赋能与流程自动化上的巨大成效，但也警示了失衡舆论与被动心态的危害。未解的核心挑战在于：在不同文化与多元价值严重分化的背景下，全球社会能否在智能体大规模融入物理与金融系统前，就核心人类维度和可落地的内部对齐机制达成有效共识，并建立起经得起考验的信任防线。

## Background 本视频由 Ai4 于 2025 年 8 月 29 日在 YouTube 平台发布，内容为 Google 研究战略高级总监 Pilar Manchón 的主题演讲。根据页面介绍，演讲聚焦于智能体 AI（Agentic AI）如何从被动工具转变为主动伙伴，并呼吁大众理解自身在这一历史性转型中的角色，成为塑造未来的积极建筑师。 ## TL;DR Pilar Manchón 指出人类正处于由 AI 驱动的最深远转型期，AI 已在材料科学、蛋白质结构解析（AlphaFold）及实时交互式世界模拟等领域实现跨越式突破。然而，面对技术狂飙，建立基于认知科学的“人类中心信任”以及攻克外在与内在“价值对齐困境”至关重要。她呼吁全社会坚守“以人为本”的底线与批判性思维，拒绝盲目乐观或在智能生态中自我矮化，主动担当共同未来的真正缔造者。 ## Atmosphere & style 演讲兼具宏大的科学视野与冷静的哲学思辨，彻底跳出了常规业务汇报的框架，呈现出极具穿透力的思想厚度。Pilar Manchón 谈吐沉稳自信且善于发问，通过现场犀利的互动打破听众对技术的盲目崇拜或被动漠视。整场演讲的情绪弧线从对前沿突破的惊叹逐渐转向对伦理与信任危机的严肃拷问，最后在激昂有力的行动呼吁中收尾，带给听众深层的认知震撼与责任担当感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=4s) 人类正处于历史上最深刻、最快速的根本性转型节点，AI 带来的变革不仅关乎生产力和效率的提升，更是在重塑工作、教育、人机交互与人类自身。 - [03:40](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=220s) AI 技术的蜕变面临走向“蝴蝶”或“飞蛾”的抉择，若缺乏人类的主动干预与批判性思维引导，技术自发导向美好结局的概率极低。 - [03:40](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=220s) 坚守“以人为本的 AI”是技术发展的唯一底线，研发目标应是让人类更具人性、让社会变得更好，而非创造脱离控制的异构智能。 - [08:47](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=527s) AI 正在成为驱动科学突破的底层支柱，AlphaFold 在数周内预测并解析 2 亿种蛋白质结构，为生物学与新材料研发带来了前所未有的加速。 - [12:16](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=736s) 实时交互式虚拟环境与视频生成技术打破了物理维度的限制，为 AI 智能体的空间感知、环境交互与具身机器人训练提供了高保真模拟平台。 - [12:16](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=736s) 智能体深度介入现实生活与科研工作，AI Co-Scientist 等工具能够交叉验证海量文献并生成科学假设，将传统数月的科研周期压缩至数小时。 - [15:05](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=905s) 多智能体协同落地的核心基石在于“信任”，自然语言的拟人化特性容易模糊机器意图的判断，需从认知科学层面解构人机信任机制与伦理边界。 - [18:38](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=1118s) 人类大脑仅需 23 瓦功耗即可完成极度复杂的认知运算，破译人脑机制将为未来突破数据中心能耗瓶颈及发展生物计算提供关键启示。 - [18:38](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=1118s) 人类价值对齐面临外在定义难题（如迈达斯国王困境与奖励作弊）与内在约束的双重挑战，需要以容纳更多复杂性的“人类维度”来替代单一指标。 - [21:49](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=1309s) 人类绝不能在自己缔造的技术生态中退让为依附性的次等存在，必须发挥自身主动权，挺身成为未来的真正缔造者而非被动旁观者。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=4s) 站在巨变的十字路口：如何看待 AI 带来的分歧 Google 研究战略高级总监 Pilar Manchón 开篇指出，这次演讲的内容将与大家以往听到的技术分享有所不同。她希望引导听众跳出日常工作中面临的具体业务与技术挑战，从更广阔、不同的视角去审视当前所处的时代节点。 当前围绕人工智能的讨论呈现出极度两极分化的态势：光谱的一端认为 AI 时代天下太平、毫无后顾之忧；而另一端则充斥着末日论调，坚信世界即将走向终结。真实的未来大概率介于这两者之间。然而令人困惑的是，当诺贝尔奖得主和图灵奖得主各自站在完全相反的立场上各执一词时，人们究竟该相信谁？Manchón 指出，人类正处于整个人类历史上最深刻、最广泛、最快速的根本性转型边缘。这种变革不仅关乎创造力、生产力和效率的指数级提升，更在以难以预料的方式重塑各行各业与市场格局。 在这场变革中，数据、基础设施、能源、算力以及人才构成了核心要素。但随之而来的深层议题涵盖了方方面面： - **工作与人才的未来**：如何定义未来的劳动力与核心技能。 - **人机交互（HCI）的演进**：在过去短短几年内，人与机器的交互方式发生了颠覆性飞跃。 - **教育与下一代成长**：不仅包括个人的终身学习，更关乎下一代应当学习什么、如何确保他们掌握正确的工具并为未来做好准备。 - **内容生成的挑战**：涉及艺术、创意、新闻与信息传播，特别是如何应对虚假信息与“另类事实”（alternative facts）的泛滥。 - **人类与社会的未来**：Manchón 强调，“我们不仅是在构建技术，我们实际上是在塑造下一代的我们自己（building the next generation of ourselves）”，而多数人对此尚未有充分的清醒认知。 ```mermaid flowchart TD accTitle: AI 变革驱动要素与未来影响领域 subgraph Inputs["核心驱动要素"] D["数据 (Data)"] I["基础设施与算力 (Infra & Compute)"] E["能源供给 (Energy)"] T["顶尖人才 (Talent)"] end subgraph Impact["重塑的关键领域"] W["工作与交互形态 (Work & HCI)"] ED["终生教育与下一代 (Education)"] C["科学与内容生态 (Science & Content)"] S["人类与社会未来 (Society & Humankind)"] end Inputs --> Impact ``` --- ### [03:40](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=220s) 蜕变时刻：做造就未来的“建筑师”还是被动等待 AI 并非只是又一项普通的新兴技术，它是未来所有知识和科学领域的“底层支柱（backbone）”。随着科学各领域全面被 AI 赋能，突破性进展正以史无前例的规模和速度爆发。人类正经历一场前所未有的剧烈“蜕变（metamorphosis）”，但 Manchón 抛出了一个核心隐喻：这场蜕变最终孵化出来的，究竟会是一只美丽的“蝴蝶（butterfly）”，还是一只扑火的“飞蛾（moth）”？ 现场互动中，Manchón 询问在场听众有多少人在使用或构建 AI 工具，绝大多数人都举起了手；但当问及“有多少人确信自己能改变世界”时，举手的人明显变少。她强调，当前既是一个需要敢于“做大梦（dream big）”的时刻，也是一个必须保持怀疑、警惕幻象（mirages）并坚守批判性思维（critical thinking）的时刻： - **感知与现实的边界模糊**：AI 已经强大到能够逼真地凭空构建“现实”，人们极易迷失在生成的虚像中。 - **批判性思维的必要性**：面对海量学习资源和 AI Agent 每日给出的回答，使用者必须时刻审视内容的真实性与掌控权归属。 - **超越纯数学维度的理解**：仅仅弄懂大语言模型（LLM）背后的数学公式与运行机理，远远不足以理解这项技术对整个人类社会构成的深远冲击。 当问及“如果不加干预、任由技术自由发展，我们有多大把握能迎来‘蝴蝶’而非‘飞蛾’的结局”时，全场举手人数为零。这表明：若不主动施加正面影响与纠偏，通向美好未来的概率几乎为零。因此，坚守“以人为本的 AI（Human-Centered AI）”是唯一底线——研发 AI 不是为了创造某种脱离控制的“异构智能（alien intelligence）”，而是为了让人类变得更具人性（more humane）、让社会变得更好，防止陷入“为了技术而做技术”的泥潭。 --- ### [08:47](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=527s) 科学跨越与多模态生成：从 AlphaFold 到无限创造 Manchón 盘点了目前 AI 已经在现实与前沿科研中带来的实质性赋能，从商业加速、疾病筛查、AI 生成图像的水印追踪，到环境预测与低碳减排： - **新材料发现**：AI 助力发现了 220 万种新材料，其中多达 39.5 万种具备高稳定性；英国最新研究利用相关技术制造计算磁体，将成本压缩至原先的 20%。 - **气候与城市治理**：涵盖洪水与野火预测、航线优化以减少航司碳排放、交通流量调控以及城市绿化规划等。 - **AlphaFold 的科研革命**：过去解析单个蛋白质的 3D 结构需要耗费一名博士生整整 4 到 5 年的研究生涯；而 AlphaFold 问世后，在短短数周内便成功预测并解析了全部 2 亿种蛋白质的 3D 结构，相当于将数千年的科研工作量瞬时交付给全球的药物研发与生物学研究者。在此基础上，业界更进一步利用 AlphaProteo 等技术去无中生有地设计自然界不存在的新蛋白质。 - **大脑解密与生成式多模态创作**：作为一名受过专业训练的认知科学家（cognitive scientist），Manchón 指出解析人脑是终极目标之一。如今在创作端，用户仅凭自然语言输入即可生成任意图像和视频片段，影视工作者更可将这些镜头无缝串联成完整的叙事场景，诸如 Genie 这类前沿工具正在打破创意的物理边界。 ### [12:16](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=736s) 实时交互式世界与科研加速：智能体应用的全面爆发 Google 最近推出的前沿工具不仅能够生成视频，还允许用户以实时交互的方式探索虚拟环境。用户可以发出指令要求以第一人称视角漫游冰岛的河流，并即时交互式地调整运镜方向，甚至能够体验从悬崖跳伞的极限模拟。这种技术的突破为 AI 智能体（Agents）打开了全新的维度——当智能体能够在高度一致且动态生成的虚拟世界中导航与交互时，空间感知、仿真模拟、游戏设计以及机器人具身训练（Robot Training）都将迎来质的飞跃。 在日常生活中，这类智能体正在深度介入现实任务。Pilar Manchón 举例提到，用户可以让 Google 智能体帮忙在奥克兰（Oakland）寻找租金低于特定预算且配有洗衣设施的公寓，智能体不仅能筛选房源，还能代为与房东沟通。谈及智能购物助手时，她打趣道现场的女性朋友一定会由衷喜爱这种能提前预判并满足需求的工具。而对于科研人员而言，最令人振奋的是 AI 在科学发现上的加速能力。她分享了自己数月前便开始使用的 **AI Co-Scientist** 工具：在传统的硕士或博士研究中，仅文献调研就要耗费数月，而现在该工具不仅能全面检索、交叉验证文献，还能将学界成果与研究者本人的构想有机结合，将数月的工作压缩到短短几小时内完成。 ```mermaid flowchart TD accTitle: AI Co-Scientist and Agentic Workflow subgraph S1["现实与虚拟交互应用"] A["实时交互视频生成"] --> B["空间导航与机器人仿真"] C["生活助理智能体"] --> D["自动化找房与生活决策"] end subgraph S2["科学发现与认知加速"] E["海量文献深度检索"] --> F["学术成果交叉验证"] F --> G["灵感与假设自动生成"] end B --> H["多智能体协作生态"] D --> H G --> H ``` - **实时视频交互**：突破传统视频生成的单向播放，支持用户在虚拟世界中实时变更视角与交互路径，为机器人训练和具身智能提供高保真模拟环境。 - **生活场景落地**：智能体能够自主解析复杂限制条件（如预算、特定设施），并代表用户直接与现实世界的服务方沟通。 - **AI Co-Scientist**：通过自动化文献梳理、可靠性校验与构想整合，极大缩短科研探索周期，将“数月工作压缩为数小时”。 --- ### [15:05](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=905s) 智能体时代的核心命题：建立认知层面的“人类中心信任” 在前景一片光明的浪潮下，Pilar Manchón 提出了发人深省的质问：我们是否因为对未来的美好憧憬而变得盲目乐观？整场大会都在热议智能体的编排优化与协同作业，但所有系统能够运转的最关键基石在于——**信任（Trust）**。如果缺乏信任，没有人会愿意将财务管理、病历隐私或子女教育全权托付给 AI。评估一个 AI 教师、AI 健身教练或 AI 税务律师时，人类所权衡的信任维度各不相同；而当智能体拥有物理实体（Embodied AI）并在现实世界中穿梭时，人类对它们的感知与防备又会发生剧烈变化。 企业不能仅仅停留在内部合规与治理流程的“自我信任”，而必须从用户视角出发，构建真正让人放心的产品。Pilar Manchón 所在的研究团队正致力于从认知科学层面解构信任机制：人类对母亲、配偶的信任，与对银行家的信任截然不同；人类对传统遵循代码指令的计算机的信任，也与对使用自然语言对话的 Gemini 或 ChatGPT 的信任大相径庭。自然语言赋予了机器拟人化的外壳，却也模糊了人类对实体意图的判断边界，因为人类的大脑经过数十万年进化专门用于识别人类行为，在面对“听起来像人但又不是人”的智能体时，信任的度量变得极其复杂。 - **多维信任评估**：不同领域的智能体（金融、医疗、教育）面临完全不同的信任门槛与风险预期。 - **具身智能的感知转变**：拥有物理形态或移动能力的智能体，对人类心理安全感与交互边界提出了更高要求。 - **语言拟人化的信任迷局**：自然语言交流使人类误将对同类的信任心理投射到模型上，必须通过严格的认知框架来规范产品的人性化表达与伦理边界。 --- ### [18:38](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=1118s) 23瓦大脑的启示与人类价值对齐的深层困境 面对未知，人类正迎来解密基因组、探索宇宙与深海的黄金时代。Pilar Manchón 指出，人类大脑仅凭 **23 瓦** 的极低功耗就能完成极其复杂的认知运转，如果未来的数据中心与生物数据存储（Biodata Storage）能够破译这种高效机制，算力架构将发生革命性重塑。然而，技术狂飙的同时必须严防伦理脱缰。当现场仅有极少数观众了解“人类价值对齐（Human Value Alignment）”时，她直言“这令人感到遗憾（That's sad）”，并强调了 Google 在更新前沿安全框架（Frontier Safety Framework）与践行 AI 原则上的责任感。 她深入拆解了价值对齐的双重困境： 1. **外在对齐（Outer Alignment）**：核心在于“定义什么价值以及由谁定义”。不同文化背景下的价值多元性、道德不确定性、偏好与本质价值的混淆，使得量化人类价值异常困难。这就好比经典的“迈达斯国王难题（Midas Problem）”——我们期望智能体像“仙女教母”一样领会真实意图并提供明智建议，但系统却可能像“邪恶神灯精灵”一样，机械死板地执行字面指令，最终带来毁灭性后果（如规范制定缺陷、奖励黑客行为 Reward Hacking）。 2. **内在对齐（Inner Alignment）**：即便人类能够完美定义价值，又该如何在数学和架构层面确保复杂模型在推理过程中始终严格恪守这些准则，而不产生隐蔽的偏离？ ```mermaid flowchart LR accTitle: AI Alignment Challenges subgraph Outer["外在对齐 (Outer Alignment)"] O1["价值多元性与道德不确定性"] O2["规范设定问题 (Specification Problem)"] O3["奖励作弊 (Reward Hacking)"] end subgraph Metaphor["迈达斯困境 (The Midas Problem)"] M1["仙女教母：理解意图并优化结果"] M2["邪恶精灵：字面执行导致灾难后果"] end subgraph Inner["内在对齐 (Inner Alignment)"] I1["模型内部表征与目标一致性"] I2["复杂推理下的约束机制"] end Outer --> Metaphor Metaphor --> Inner ``` - **高能效生物计算**：人脑仅消耗约 23 瓦能量，揭示了生物存储与超低功耗计算在未来替代庞大数据中心能耗的巨大潜力。 - **迈达斯困境**：形象比喻了 AI 系统因缺乏上下文理解与真正同理心，在字面执行指令时可能导致的灾难性反噬。 - **人类维度（Human Dimensions）**：Pilar 主张用“人类维度”替代抽象单薄的“人类价值”，更全面地容纳人类社会的复杂性。 --- ### [21:49](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=1309s) 拒绝沦为次等存在：做未来的缔造者而非旁观者 在谈及人类与高级人工智能的关系时，学界与工业界领袖存在诸多流派：Geoffrey Hinton 倾向于将超级 AI 设想为某种“母性 AI（AI Mother）”般的引导力量；Joanna Bryson 教授倡导将其纳入类似城市或企业的治理体系；而 Yann LeCun 则坚称这纯粹只是一个工程问题。Pilar Manchón 明确表示自己对这三派观点均不认同。她强调，人类绝不能在自己亲手缔造的新生态系统中主动退让，沦为低人一等的依附存在。 她引用了著名作家爱丽丝·沃克（Alice Walker）的名言：“人们放弃力量最常见的方式，就是认为自己毫无力量（The most common way people give up their power is by thinking they don't have any）。” 她向全场观众发起质问：无论此前是否举手认为自己在改变世界，每个人在各自的专业领域和企业组织中都拥有无可替代的主动权。在人类自我增强与破译生命奥秘的关键节点上，每个人都面临着根本性的抉择——是成为这场智能化转型的被动旁观者，还是挺身而出，成为我们共同未来的真正缔造者（Architects of our future）。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=4s) 我们不仅是在构建技术，我们实际上是在塑造下一代的我们自己（building the next generation of ourselves） — Pilar Manchón > [03:40](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=220s) 人类正经历一场前所未有的剧烈“蜕变（metamorphosis）”，但这场蜕变最终孵化出来的，究竟会是一只美丽的“蝴蝶（butterfly）”，还是一只扑火的“飞蛾（moth）”？ — Pilar Manchón > [03:40](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=220s) 做大梦（dream big） — Pilar Manchón > [03:40](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=220s) 研发 AI 不是为了创造某种脱离控制的“异构智能（alien intelligence）”，而是为了让人类变得更具人性（more humane）、让社会变得更好，防止陷入“为了技术而做技术”的泥潭。 — Pilar Manchón > [12:16](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=736s) AI Co-Scientist — Pilar Manchón > [15:05](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=905s) 人类对母亲、配偶的信任，与对银行家的信任截然不同；人类对传统遵循代码指令的计算机的信任，也与对使用自然语言对话的 Gemini 或 ChatGPT 的信任大相径庭。 — Pilar Manchón > [18:38](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=1118s) That's sad — Pilar Manchón > [21:49](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=1309s) The most common way people give up their power is by thinking they don't have any — Alice Walker > [21:49](https://www.youtube.com/watch?v=ff9wPUgkk7Y&t=1309s) 是成为这场智能化转型的被动旁观者，还是挺身而出，成为我们共同未来的真正缔造者（Architects of our future）。 — Pilar Manchón ## Mentioned Google, Pilar Manchón, Ai4, AlphaFold, AlphaProteo, Genie, AI Co-Scientist, Gemini, ChatGPT, Geoffrey Hinton, Joanna Bryson, Yann LeCun, Alice Walker ## Video description Pilar Manchón, Senior Director of Research Strategy, Google Abstract We are standing at the threshold of a new era, where intelligent agents don't just answer our questions, but actively build our future alongside us. This fundamental shift from passive tool to active partner is poised to catalyze massive change, impacting our societies, markets, and even our philosophy of life. Agentic AI represents the next great leap in human progress, and its foundations are being laid today. This presentation will showcase key examples, revealing how these tools empower individuals and communities to tackle grand challenges. You will see how this technology is helping architect everything from sustainable cities to personalized education. Join this session to understand your place in this transformation and discover how you can become an architect of our future. [Source](https://www.youtube.com/watch?v=ff9wPUgkk7Y)

[0:04] Pilar Manchón: 我们不仅是在构建技术，我们实际上是在塑造下一代的我们自己（building the next generation of ourselves） [3:40] Pilar Manchón: 人类正经历一场前所未有的剧烈“蜕变（metamorphosis）”，但这场蜕变最终孵化出来的，究竟会是一只美丽的“蝴蝶（butterfly）”，还是一只扑火的“飞蛾（moth）”？ [3:40] Pilar Manchón: 做大梦（dream big） [3:40] Pilar Manchón: 研发 AI 不是为了创造某种脱离控制的“异构智能（alien intelligence）”，而是为了让人类变得更具人性（more humane）、让社会变得更好，防止陷入“为了技术而做技术”的泥潭。 [12:16] Pilar Manchón: AI Co-Scientist [15:05] Pilar Manchón: 人类对母亲、配偶的信任，与对银行家的信任截然不同；人类对传统遵循代码指令的计算机的信任，也与对使用自然语言对话的 Gemini 或 ChatGPT 的信任大相径庭。 [18:38] Pilar Manchón: That's sad [21:49] Alice Walker: The most common way people give up their power is by thinking they don't have any [21:49] Pilar Manchón: 是成为这场智能化转型的被动旁观者，还是挺身而出，成为我们共同未来的真正缔造者（Architects of our future）。

[0:04] Pilar Manchón: Nobel laureates tell you one thing and other Turing Award winners telling you exactly the opposite thing. [0:04] Pilar Manchón: alternative facts [3:40] Pilar Manchón: butterfly or a moth [3:40] Pilar Manchón: Zero. [3:40] Pilar Manchón: alien intelligence [15:05] Pilar Manchón: age of agents [18:38] Pilar Manchón: That's sad. [20:11] Pilar Manchón: fairy godmother [20:11] Pilar Manchón: evil genie [20:11] Pilar Manchón: human dimensions [21:54] Geoffrey Hinton: AI mother [21:54] Pilar Manchón: lesser being [21:54] Alice Walker: the most common way people give up their power is by thinking they don't have any

[0:04] This uh presentation that I have here [0:05] for you today is probably a little [0:08] different from what you have seen so [0:09] far. And you might be wondering what's [0:12] what's the focus? What is it that you [0:14] want to get out of this presentation [0:16] today [0:18] and I'm hoping to make you think think [0:21] from a different perspective think about [0:23] different things. Think beyond the many [0:26] challenges and the many interesting [0:29] things that you have seen here and [0:31] you're probably facing in your [0:32] respective jobs. [0:35] Think about where we stand, the [0:37] different perspectives [0:39] where you look at one side of the [0:41] spectrum and there is nothing to worry [0:43] about in a world of AI or the other side [0:46] where the world is about to end. Yeah, [0:49] the reality is probably somewhere in the [0:51] middle, right? But whose perspective do [0:55] you believe? Especially when you have [0:57] Nobel laureates tell you one thing and [1:00] other touring award winners telling you [1:03] exactly the opposite thing. Right? Who [1:06] do we believe? What kind of stand do we [1:08] have? What do you think about this? [1:11] We are on the verge of the most deepest, [1:16] broadest, fastest transformational [1:19] fundamental shift that we have seen in [1:22] the whole of humanity. [1:24] Things are changing and they are [1:25] changing in an order of magnitude never [1:27] seen before. We're talking about [1:30] creativity, productivity, efficiency, [1:33] the transformation of industries and [1:36] markets, not organically, not just [1:38] faster, but in ways that are very hard [1:41] to anticipate. We have to think really, [1:44] really hard to anticipate how the world [1:46] is changing and not only how our [1:48] products and services will be evolving [1:50] with it. And we know we have key [1:53] ingredients like data, infrastructure, [1:56] energy, compute, and of course talent, [2:00] right? [2:01] The future of work and talent is what [2:04] we're discussing here today as well. The [2:07] future of science, the future of [2:09] transportation, the future of human [2:12] computer interaction that has evolved so [2:14] much in just the last couple of years. [2:18] the future of education. And we [2:20] understand education not only in terms [2:22] of our lifelong journey of mindset and [2:27] growth and educating ourselves, but the [2:29] next generations coming over. What are [2:32] they going to be learning? What tools [2:33] are they going to have? How are we going [2:35] to make sure that they learn the right [2:36] kind of things and they are prepared for [2:38] the future? the future of content [2:40] generation and that touches anything [2:42] from creativity, art and so on to news [2:48] to information to facts or alternative [2:52] facts. And that's part of the challenge, [2:54] right? [2:56] Is this the future of humanity, the [2:58] future of humankind, [3:01] the future of society? [3:03] I think it is. I think we're not just [3:06] building technology. I think that we're [3:07] building the next generation of [3:09] ourselves. And I don't think everybody [3:12] is fully aware of that yet. [3:14] AI is not just another technology. It's [3:17] not just another emerging technology. [3:20] It's actually the backbone of the [3:22] future. It's empowering [3:25] everything else. Every field of science [3:28] and knowledge is now empowered by AI. [3:31] And we're going to be seeing [3:33] breakthroughs, advances, and things at a [3:36] speed, at a scale that we have never [3:38] seen before. [3:40] We're in the midst of the biggest [3:42] metamorphosis that humankind has ever [3:45] gone through because of its size, of the [3:48] scale, and most of all because of the [3:50] speed at which is happening. The [3:53] question is, is it going to be a [3:55] butterfly or a moth that comes after? [4:00] That's part of who we have to decide. [4:02] And it's not just some of us. It's all [4:05] of us. But can all of us have something [4:07] to say about that matter? So what do we [4:10] do? What do you do? How many of you here [4:14] are building tools with AI? [4:18] Many of you. How many of you are just [4:20] using AI for your professional personal [4:24] activities? So one way or another, if [4:27] you're here today, you're using AI, [4:29] hoping to use AI, building things with [4:32] AI. [4:33] Are you changing the world? How many of [4:36] you are changing the world? [4:39] We have some architects here. How many [4:42] of you hope to change the world? [4:46] That's what I like. [4:50] In my opinion, it's a time to dream. [4:52] It's a time to think big. It's a time to [4:55] think that we have the most powerful [4:57] technology that has ever been created [4:59] and it's evolving and it's changing and [5:02] it's not 100% reliable yet, but it's [5:05] getting better and better and better and [5:07] we can do fantastic and imaginable [5:09] things with it. So why not dream big, [5:13] right? To build that future. But it's [5:16] also a time to question, [5:18] time to think about the mirages ahead of [5:20] us, a time to see where everything that [5:23] we see might not be exactly what we [5:26] think that we are. It's all about the [5:27] perception. And now the tools are so [5:30] good at creating new realities that how [5:33] do we know that that's the reality that [5:36] we're kind of living by? [5:39] So it's the time to pri prioritize [5:41] critical thinking. talking about [5:44] education before. I think that we have [5:47] an enormous opportunity to learn about [5:49] anything that we want. There are so many [5:52] resources at our disposal. But then at [5:54] the same time, what are you learning? [5:56] Who's in control of that? Are you sure [5:58] that the content that you're learning [5:59] are the right kind of contents? [6:03] Critical thinking is something that we [6:06] need to preserve not only for ourselves [6:08] but also for the next generation coming. [6:10] And how do you do that? Do you apply [6:12] critical thinking to every single [6:14] response that you're getting from these [6:15] agents every day? Maybe [6:19] it's also time to make choices. And some [6:21] of those choices are difficult choices, [6:23] right? We have a lot of tools. We have a [6:26] lot of opportunities. We have a lot of [6:27] challenges. We have [6:30] ambition. We have things to do. We have [6:33] different responsibilities. But what [6:35] kind of choices are we making based on [6:37] what we know? And how much do we know in [6:40] order to make those choices? [6:43] Is getting an understanding of the [6:45] mathematics behind the large language [6:47] models and understanding how that works [6:50] enough to have a good perspective of [6:52] what these tools imply for society? Is [6:56] that enough? [7:00] We're walking a very fine line. We're [7:03] walking in an extremely interesting time [7:07] and it's a time to be bold but also [7:09] responsible, right? It's a time to be [7:11] excited but at the same time to make [7:13] sure that we're taking the right steps, [7:15] making the right choices and going in [7:18] the right direction [7:21] because we are definitely moving and [7:22] we're moving fast in a direction. Is [7:24] that the direction that we should go? [7:27] How many of you are confident that if we [7:29] do nothing and if we just let just let [7:32] things go the way they are, we're going [7:35] to get to a fantastic place to the [7:37] butterfly as opposed to the moth place? [7:44] Zero. [7:45] So the chances of us succeeding [7:49] without [7:51] doing something in favor of the world [7:53] that we wonder are pretty much zero in [7:55] everybody's opinion. And what are we [7:57] doing in order to make that change? [8:01] One of the main things is to take into [8:03] consideration a human center AI is a [8:06] must. It's not about just building a new [8:09] entity, a new type of intelligence. Some [8:11] people call it like an alien [8:13] intelligence and so on. It's about [8:16] helping us be more human, more humane. [8:20] It's about helping society. It's about [8:22] helping us be better. Whatever better [8:26] means for us, right? But if we lose [8:29] track of the human centricity in this [8:32] equation, we might end up developing [8:35] tech for the sake of tech, right? And [8:38] that's probably not what we want if we [8:40] want to have technology at the service [8:43] of humanity and not the other way [8:45] around. [8:47] We're doing so much. It's such an [8:49] exciting moment. [8:52] So, we're accelerating the growth uh for [8:54] millions of businesses, detecting [8:56] diseases of all kinds, watermarking and [8:59] identifying AI generated um images, [9:03] understanding proteins linked to cancer [9:05] and autism, discovering millions of new [9:08] materials, 2.2 millions of new [9:11] materials, 395,000 of them are stable. [9:15] Recently in the UK they have discovered [9:17] new ways of doing this magnets for [9:19] computing at a 20% cost of what used to [9:23] before. I mean it's crazy that it's now [9:26] all over the news. [9:28] Forecasting floods and wildfires. Making [9:30] translation easy and accessible. Mapping [9:33] the world making it more immersive. [9:35] Finding roots. I mean we all use the [9:37] GPS, right? I don't know how to get [9:39] anywhere anymore without the GPS. I [9:41] don't know about you. Automatic cheap [9:43] design for better performance and cost. [9:48] Helping airlines to contaminate less. [9:53] Working with mathematical models. [9:56] Predicting the weather in more accurate [9:58] ways. Helping traffic to generate fewer [10:03] emissions or helping cities plan how [10:06] much green they have in those cities to [10:08] make them more environmentally friendly. [10:11] or be helping with wildfires. How many [10:14] of you have heard of Alpha Fall? Please, [10:16] all of you raise your hands. No. No. No. [10:18] Come on. Alpha Fall is a beautiful tool. [10:22] This I love talking about this because [10:24] it's the great example of the kind of [10:27] changes and disruption that are coming. [10:30] It used to take one PhD, it's all PhD to [10:35] decipher the 3D structure of one [10:38] protein. That's four or five years of [10:40] work of one full PhD student. [10:44] When Alpha fall came along, it basically [10:48] decipher the 3D structure of all 200 [10:51] million proteins in a matter of weeks. [10:54] That's millennia of worth of research [10:57] that is now at the availability is [11:00] available for all the world for [11:03] researchers in drugs in medicine in [11:06] biology in everything. [11:08] We have just leapfrogged enormously and [11:12] you know once all these proteins have [11:14] been decoded why not invent new ones new [11:17] ones that don't exist anymore with alpha [11:19] proteo. Imagine this apply to any field [11:22] of science and knowledge. [11:24] [Music] [11:27] I didn't say any of this before, but I'm [11:29] a cognitive science scientist by by [11:32] training. So deciphering the human brain [11:37] is an ultimate goal, right? How much can [11:40] we learn from our own biology in this [11:43] case? And now we have tools where if you [11:46] can imagine it, if you can say you can [11:50] get a picture, any picture [11:53] or you can get a video just by saying [11:57] using natural language with it. I'm [12:00] hoping that you're all familiar with [12:01] these tools, right? [12:03] Or if you are more in the uh movie [12:07] business then creating sequences and [12:10] scenes and joining all these little [12:12] videos together and Genie one of the [12:16] latest tools that we have launched where [12:19] not only can you generate video you can [12:22] actually interact with a video in real [12:24] time. [12:26] So you can say something like I want to [12:29] see a first first person view of a river [12:32] in Iceland and actually telling what [12:35] direction you wanted to move the camera [12:38] interactively [12:41] or see what it would be like if you were [12:43] crazy enough to jump off a mountain. [12:48] And what does this imply for the world [12:51] of agents? [12:53] When you can actually tell an agent to [12:55] interact with a virtual world that you [12:58] are building and is consistent [13:00] throughout [13:02] spatial navigation, simulations, games, [13:06] the training of robots. [13:09] You are [13:12] in the middle of a huge transformation. [13:14] Like I said before, you're witnessing [13:16] all these changes and hopefully using [13:18] them too. [13:20] The AI mod with Google that I'm hoping [13:22] many of you have used already, right? [13:24] Having your own agent to actually do [13:27] things for you like find apartments in [13:29] Oakuckland that uh have a rent lower [13:32] than X and have laundry available. And [13:35] it goes and finds them and talks to the [13:37] landlords for you. [13:41] Love it. Shopping [13:44] There are many women here. I don't know [13:45] about the gentlemen, but we certainly [13:47] love the chopping [13:49] and how much nicer is it to have a tool [13:52] that helps you kind of anticipate [13:53] anything and everything that you might [13:55] be interested in. But what as a [13:58] researcher, one of the most interesting [13:59] things is accelerating the discovery of [14:03] new things. Have you heard of the AI cos [14:05] scientist? [14:06] Oh, I found out a little way before we [14:10] launched it and I've been using it for [14:12] months now and I love it. I mean any of [14:15] you who have done any research, a [14:17] masters, a PhD now it's like going [14:20] through the literature alone takes you [14:22] forever. Now these things goes through [14:24] the literature finds it for you [14:26] validates which ones are the best [14:28] actually puts together little ideas with [14:30] your own. It's fantastic. It's It's [14:33] months of work translated to a few [14:36] hours. It's a wonderful world. Is it [14:41] kind of ish? Yeah. Because it's [14:43] wonderful and but we're not 100% safe, [14:46] right? So it's is [14:49] [Music] [14:51] we being foolish by thinking that [14:54] because it's wonderful and because of [14:56] the promise of this fantastic future, [14:59] it's guaranteed. Are we doing enough? [15:02] Are you doing enough? [15:05] We are in the age of agenda. This is [15:08] this is what this whole conference is [15:10] about, isn't it? We've been talking [15:12] agents and over and over all kinds of [15:14] agents, how to optimize them, how to do [15:18] collaborations of orchestration of [15:20] multiple agents and so on. But what is a [15:22] key ingredient of all these agents? [15:28] Do you want to have an agent that [15:30] handles your financials for you if you [15:32] don't trust it? [15:35] Right? Do you have an agent that takes [15:38] care of your medical information if you [15:41] don't trust it [15:43] or it takes care of teaching your [15:46] children if you don't trust it? How do [15:50] you trust it? This is the same kind of [15:51] trust for all these kind of agents. Are [15:54] you worried about the same thing? If [15:55] you're hiring a teacher, if you're [15:57] hiring a coach, if you're hiring a tax [15:59] attorney, is it the same kind of trust? [16:04] How can we trust these agents? And does [16:06] that change at all if this agent is [16:08] embodied? If it moves in the world, if [16:10] it interacts in the world, if it has [16:12] some kind of embodiment of any kind, [16:16] how does our own perception change? [16:20] How can we make sure that our products [16:21] and services if you are in the world of [16:25] generating products and services are [16:27] actually trustworthy? [16:29] Not only trustworthy from our [16:32] perspective because we're all [16:33] trustworthy and we do the right things [16:34] and we have our governance processes and [16:36] we try and we test and we do all those [16:38] things but trustworthy from the point of [16:40] view of the users. They need to embed [16:43] these agents in their everyday lives in [16:46] things that are very very important to [16:48] them. [16:50] And we go back to the human centricity [16:52] of this challenge where generating trust [16:55] becomes the goal. They have to be easy [16:58] to use and easy to understand. They have [17:01] to be useful and relevant, ethical and [17:05] responsible. They're the foundation of [17:07] the future. [17:09] We have the opportunity to unleash that [17:11] potential maximize social benefit being [17:14] riskaware never forgetting that and [17:16] democratizing access. [17:19] But there is a hierarchy in all these [17:21] things and there is a set of priorities [17:23] that we have to take into consideration [17:25] when we do our business plans and when [17:27] we build these technologies. Right? [17:29] That's why part of the research in my [17:31] group is about creating a framework that [17:34] helps us understand at a cognitive [17:36] level. How does trust work? [17:40] Is a trust that you have in your mother [17:42] or your wife the same trust that you [17:45] have on your banker? [17:48] Kind of different, [17:50] right? How does that work? Do you trust [17:54] your computer to do what you tell it to [17:56] do with instructions? And is it the same [17:59] that you trust any of the agents like [18:01] Gemini, Chhattivity or any of the other? [18:03] Is it the same kind of trust? [18:05] When you use natural language to [18:07] interact with these entities, [18:11] our ability to trust them becomes a [18:13] little blurry [18:15] because we are trained to understand how [18:17] people function. We've been training all [18:20] our lives to understand and read people [18:23] and we kind of understand how computers [18:25] work. But would you get an agent that [18:27] actually sounds like a person but not [18:30] quite? H how how do you measure that [18:33] trust? How do you make sure that you're [18:34] doing the right thing? [18:38] We have a bold and exciting future [18:40] ahead. We have so many opportunities, so [18:42] many challenges that we can solve, so [18:44] many things that it's unbearable if you [18:47] get excited easily. And you can probably [18:48] tell I get excited easily. [18:51] There's such a world out there in terms [18:53] of what we can do. [18:55] We can decode the human genome. [18:58] We can decode. How does this thing work [19:02] with only 23 watts? [19:05] 23 watts. What can you do with 23 watts? [19:08] Nothing. [19:10] Or can we? What would be the future of [19:13] our data centers and our compute if we [19:15] were able to decipher how we can deal [19:18] with this biodata storage? How much [19:22] information is in our DNA? Decoding the [19:24] universe, decoding the abyss. [19:27] We're in a new world with all rules. [19:30] That's why it's important that we focus [19:32] on human value alignment. How many of [19:34] you know what human value alignment [19:36] means? [19:39] That's sad. [19:41] Very few of you know that. And that's [19:44] why I decided to focus on this today. At [19:47] Google, we have always been focused on [19:50] doing the responsible thing. That's why [19:51] we have the AI principles about being [19:54] bold and being responsible and [19:55] collaborative. And we recently updated [19:58] our frontier safety framework to make [20:00] sure that not only are we doing the [20:02] right thing, but everybody in the world [20:04] that's looking at us is trying to do the [20:06] right thing. But just to educate those [20:09] of you that do not understand the [20:11] severity and the seriousness of [20:13] alignment, we have the align the outer [20:16] alignment problem which is if I ask you [20:19] what are human values I probably get 50 [20:23] different answers here and how do you [20:25] measure them [20:28] and whose values your values my values [20:30] western society [20:33] well what kind of values do we want too [20:36] Right? And what about the Midas problem? [20:40] Right? If we have this magnificent tools [20:42] that you can ask them to do anything for [20:45] you, anything for you using natural [20:47] language, you can have the scenario with [20:49] the fairy godmother that's trying to [20:51] help you and give you a hand. And if you [20:53] say something, it's going to give it to [20:54] you and it's going to enhance it and [20:56] it's going to advise you and it's going [20:57] to do it better for you. Or you can have [21:00] the evil genie [21:02] that gives you exactly what you didn't [21:04] need and you didn't want and you were [21:05] all worse off before you started. And [21:08] that's part of the challenge that we [21:09] face. Right? [21:12] In the outer alignment, we have the [21:13] specification problem, the reward [21:15] hacking, the value of pluralism, moral [21:18] uncertainty, bias and discrimination, [21:20] preferences being confused with values. [21:23] What are they? And actually a layman [21:25] understanding of what's going on. That's [21:27] why I call them human dimensions instead [21:30] of human values. [21:32] From the inner perspective, how on earth [21:35] do you make if even if you were able to [21:37] define all these values, how on earth [21:40] are you going to be able to actually [21:42] make the models respect those values? [21:45] Well, there are some points of consensus [21:47] and contention. [21:49] I'm almost out of time, so I'm going to [21:51] accelerate a lot. [21:54] But one of the critical things is that [21:56] many of you probably heard Hinton say [21:58] about the mother AI mother as a vision. [22:02] Professor Vyron um Bryson advocates for [22:06] a situation more like governance based [22:09] like a city or corporation and young lun [22:12] advocates for not this is just an [22:13] engineering problem. I personally [22:15] disagree with all three of them because [22:18] we don't have to resign ourselves to [22:20] being the lesser being in that new [22:22] ecosystem that we're building. Alice [22:24] Walker used to say that the most common [22:26] way people give up their power is by [22:28] thinking they don't have any. Do you [22:30] have any power? Some of you tell were [22:33] said before that you were changing the [22:35] world. You were architecting the world. [22:37] Those of you who didn't raise your [22:38] hands, do you have any power? [22:42] That's the question. That's what I would [22:43] like for you to think about. What role [22:45] do you play in your areas of knowledge [22:48] or in your companies where you have an [22:50] empower? We're in the verge of decoding [22:52] humanity, augmenting ourselves. We don't [22:55] have to resign to being the lesser being [22:59] in that new ecosystem. [23:01] We're architects of our future. And the [23:04] question is whether you're going to be [23:06] an expectator or you're going to be an [23:08] architect of that future. Thank you so [23:10] much. [23:12] [Applause]