Alphabet’s X Deploys Multimodal AI to Accelerate Industrial Spinouts

Alphabet 旗下 X 部署多模态 AI 加速工业项目分拆 解决住房、能源和材料科学领域的物理瓶颈，需要构建将多模态规划器与确定性仿真工具相连接的领域中间抽象层，而不仅仅是扩展模型参数。 Divya Choudhary 阐明了“登月工厂”X 如何将多模态 AI 与智能体推理相结合，以应对需要激进 10 倍解决方案的重大现实问题。通过梳理多模态系统从解耦编码器到智能体视觉-语言-动作（Vision-Language-Action）框架的演进，演讲解释了连接物理仿真工具与多感官输入的特定领域“连接纽带”的关键作用。结合电池化学发现、自动化可持续住宅设计、电网现代化和气候适应力等具体案例研究，Choudhary 展示了闭环智能体工作流如何将复杂的现实约束转化为可验证、可扩展的解决方案。 X 的“登月”理念核心在于采纳当下看似不合理或不太可能的设想，并持续进行研发，直至其成为不可避免的现实基础设施，正如 Waymo 规模扩展至每月提供超过 100 万次自动驾驶出行服务所证明的那样。 (0:04) “登月”项目依据严格的三部分准则进行评估：解决影响数百万人或数十亿人的重大问题、提出 10 倍级的激进解决方案，以及运用结合仿真、软件与 AI 的突破性技术。 (3:14) 应用 AI 的竞争优势不仅在于训练更大的模型，更在于工程化构建“连接纽带”——即能够将受物理法则支配的复杂现实问题转化为计算上可验证且经济上可行的表征的抽象层。 (6:51) 多模态 AI 经历了四个不同世代的演进，从解耦的单模态编码器与跨模态桥接，发展到原生标记化以及第四代视觉-语言-动作（VLA）智能体架构。 (10:29) 现代科学智能体依托持续的“思考、行动与反思”循环运行，将多模态规划器与模块化反思引擎，以及有限元分析（Finite Element Analysis）和分子组合求解器等确定性工具结合在一起。 (14:42)

Divya Choudhary of X, The Moonshot Factory, argues that multimodal AI's primary potential lies in solving high-stakes physical-world crises by pairing foundation models with intermediate domain representations, simulation engines, and deterministic verification tools. The Moonshot Framework and Domain Translation Layer (3:14) X evaluates potential projects against a strict three-part rubric: targeting a global problem affecting millions or billions of people, proposing a radical solution offering a 10x improvement over existing methods, and leveraging breakthrough technology uniting physics, simulation, and AI. Choudhary notes that X simultaneously explores dozens of concepts, intentionally terminating the vast majority to concentrate resources on technically and economically viable ideas. This methodology supported the transition of Waymo from an improbable autonomous vehicle experiment into a commercial service providing over one million driverless rides per month, alongside spinouts like 280 Earth (direct air carbon capture) and Heritable (climate-resilient crops). Rather than competing solely on model parameter scale, Choudhary states that X focuses on building the "connective tissue"—an abstraction layer that converts messy, physics-governed physical environments into computationally verifiable and economically affordable representations. Multimodal AI functions as the core substrate of this layer, translating heterogeneous physical and sensory inputs into structured domain pipelines. From Decoupled Encoders to Agentic Systems (10:29) Choudhary outlines four generations in the architectural development of multimodal AI. Generation 1 relied on decoupled vision and text encoders that generated isolated embeddings, allowing rapid vector retrieval but offering no cross-modal reasoning. Generation 2 introduced cross-modal bridges, such as projection layers and Q-Formers, to feed visual features into frozen language models. Generation 3 established native unified tokenization across text, audio, video, and raw sensor feeds. Modern Generation 4 systems combine Vision-Language-Action (VLA) architectures with agentic workflows. Because context window limits have largely dissolved across coordinated agents, systems now operate via continuous "Think, Act, and Reflect" cycles. Multimodal planners ingest sensory tokens, execute Chain-of-Thought or Tree-of-Thought reasoning, and dispatch tasks to specialized external tools—such as Finite Element Analysis (FEA) and physics engines—whose empirical outputs are evaluated by custom reflection engines before updating plans. Material Optimization and Molecular Intelligence (21:02) Applied agentic loops are demonstrated in battery chemistry optimization, where X deployed a closed loop between a proposal model and an automated simulation evaluator. The proposal model uses Retrieval-Augmented Generation across scientific literature to design novel ion combinations and electrode formulations. Automated physical and chemical simulators then test these candidates against strict ionic conductivity and structural stability thresholds, discarding unviable compounds early. Choudhary states that automating this loop has compressed material discovery cycles from years of laboratory trial-and-error down to weeks or hours. Similar molecular-level reasoning informs Project Matera, which identifies and categorizes specific molecular structures in scrap streams rather than sorting by gross item type, addressing the observation that 75% of materials extracted from the earth are returned as waste. Additionally, Project A-Life applies multimodal models and biological domain knowledge to design novel cells for precision manufacturing of critical compounds. Solving Built-Environment and Climate Challenges (24:04) Single-modality AI fails in physical planning because real-world domains require balancing disparate, conflicting constraints. Addressing the building sector—which Choudhary notes accounts for roughly 37% of global carbon emissions while facing a projected shortage affecting 3 billion people by 2030—X engineered an intermediate domain-specific language. This layer abstracts local municipal zoning laws, structural engineering constraints, climate risk metrics, and material footprints into internal vector spaces linked to computational geometry toolboxes. In a real-world test on a Phoenix lot, generative agents optimized setbacks, density, and parking requirements to produce code-compliant, structurally sound architectural layouts. Beyond housing, multimodal models support critical infrastructure. Project Tapestry integrates disparate national electrical data to optimize grid power flows and mitigate clean-energy interconnection bottlenecks that often exceed six years. Project Bellwether combines atmospheric and geospatial data streams to forecast severe storms and wildfires, partnering with agencies like the National Guard as the United States faces 18 or more annual billion-dollar weather disasters. Choudhary concludes that solving intractable physical problems requires moving beyond generic foundation models toward intermediate representations that strictly enforce physical laws and domain constraints, supported by deterministic tools that make generative planning verifiable and safe to deploy.

来自X（The Moonshot Factory）的Divya Choudhary指出，多模态人工智能已超越消费级对话助手，正在通过构建连接物理定律与计算模型的中间层，推动能源、住房、气候与材料等全球性重大难题的颠覆性解决。 登月工厂的评估公式与技术转化 (0:04) X（The Moonshot Factory）设立的核心目标是发明并推出能够实质改善人类生存环境的登月级技术，将看似不可思议的概念转化为日常基础设施。Choudhary阐述了其严格的三要素评估框架：项目必须直面影响数百万乃至数十亿人的全球重大危机（如电网不稳或住房匮乏），方案必须具备带来10倍跃升潜力的颠覆性手段，同时依托结合了物理模拟与先进AI的突破性技术。X通常并行探索数十个项目并果断终止大部分尝试，仅保留极少数具备可行性的方案。 该路径的典型代表是自动驾驶项目Waymo，其已从最初被行业广泛质疑的试验演化为全球最大规模的无人驾驶出租车服务，月均完成超过100万次出行。在过去18个月中，X孵化出了应对可持续住房危机的Anori、研发抗逆农作物的Heritable、搭建低成本直接空气碳捕集设施的280 Earth、利用光束进行无线通信的Taara，以及专注海洋监测的Tidal和供应链空间智能的Chorus等独立实体。 多模态架构演进与智能体闭环 (10:29) 多模态AI架构经历了四代演化：从第一代依赖独立编码器生成向量以进行快速检索但缺乏推理能力的系统，到第二代引入Q-Former等跨模态投影层，再到第三代将文本、音频与传感数据进行端到端统一标记，直至第四代融合视觉-语言-动作（VLA）与智能体抽象的架构。Choudhary强调，早期大型语言模型将卫星图像、化学分子式或三维坐标强行压平为纯文本，导致了巨大的信息损失，而第四代架构则将高层规划与底层专业工具调用有效解耦。 在落地层面，系统通过“思考、行动、反思”（Think, Act, Reflect）循环应对受物理法则支配的真实环境。多模态规划器接收传感器输入并利用交错思维链或思维树展开推理，调度有限元分析（FEA）、物理引擎或分子合成API等专业工具；随后，“反思”引擎对稳定性、导电性等执行结果展开严密验证并形成反馈。这一模块化工具库具备高度复用性，例如电池化学评估模块可无缝迁移至合成生物学管线。 材料科学与可持续住房的落地突破 (21:02) 多模态智能体显著加速了材料研发周期的验证效率。在电池化学优化案例中，系统构建了包含提议模型与仿真评估器的双引擎闭环：提议模型利用检索增强生成（RAG）遍历化学文献以生成新型离子组合和电极配方，随后自动物理与化学模拟器对候选化合物的离子电导率和结构稳定性执行门槛筛选。这一自动化筛选机制成功将过去需要数年实验室试错的发现流程缩减至数周或数小时。 面对占全球碳排放约37%且预计到2030年面临30亿人住房缺口的建筑领域，X开发了面向建筑设计的中间领域特定语言作为连接组织。传统的单模态模型难以兼顾法律法规、气候风险、结构受力及材料足迹等异构约束，而该系统将多样化条件转化为内部向量嵌入并对接计算几何引擎。在亚利桑那州凤凰城的实地测试中，系统自动将楼层数、退让红线、车位指标及居住密度等多重限制综合求解，直接生成符合规范与力学要求的建筑平面图。 电网基建、气候防灾与分子制造应用 (28:50) 在基础设施与生态韧性领域，多模态AI同样发挥着系统调度作用。Project Tapestry针对新能源并网通常耗时六年以上及断电频发的现状，整合全美电网多源数据以优化潮流分布、自动化物理巡检并预测设备故障。面对美国连续四年每年遭遇18起以上损失超十亿美元天气灾害的严峻形势，Project Bellwether汇集地理空间与大气气象数据，预测山火及极端风暴，并已与蒙大拿州及国民警卫队建立减灾合作。 在微观制造与环保方面，X推进了Matera与A-Life项目。鉴于人类从地球开采的资源中75%最终沦为废弃物，Matera通过多模态视觉直接识别分拣特定分子结构，将低级废品提升为高纯度再生资源；A-Life则将多模态AI与细胞生物学融合，用于定向设计制造关键分子的细胞，实现以更低成本和更高效率生产关键化合物。 Choudhary总结认为，多模态AI的核心价值在于将抽象推理与确定性物理仿真工具紧密结合。然而，系统在不同专业领域落地时，依然高度依赖针对特定场景设计的通用中间语言，如何持续构建并验证这些能够无损翻译复杂现实规律的中间表征，仍是该技术持续扩展的关键挑战。

## Background 本内容由 YouTube 频道 Ai4 于 2026 年 8 月 17 日发布。页面描述显示，本次演讲由来自 X (The Moonshot Factory) 的首席 AI 研究科学家 Divya Choudhary 主讲，聚焦登月工厂的前沿探索。 ## TL;DR 演讲系统阐述了 X (The Moonshot Factory) 如何运用多模态 AI 与智能体（Agentic）架构攻克全球性重大难题。演讲者回顾了多模态架构从解耦编码到 VLA 智能体控制的演进历程，并详细介绍了高层规划与专用工具解耦的“思考-行动-反思”闭环。通过电池化学优化、可持续建筑设计（Project Anori）、电网优化（Tapestry）、气候灾害预测（Bellwether）以及分子级垃圾回收（Matera）等实战案例，深入展示了 AI 如何作为连接现实物理世界与数字智能的“结缔组织”，将看似不可能的构想转化为势在必然的现实。 ## Atmosphere & style 整场演讲呈现出严谨、前瞻且充满使命感的科技布道氛围。演讲者以沉稳自信且条理清晰的逻辑展开叙述，将宏大的“登月愿景”层层拆解为高度可落地的工程与科学架构。听众在跟随其穿梭于电池化学、可持续建筑与气候预测等跨学科硬核案例时，能深切感受到尖端前沿科技重塑物理世界所带来的震撼与工业美感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=8qApsMR7cDU&t=4s) X (The Moonshot Factory) 的核心使命是通过高淘汰率迭代机制，将当下看似不理性或不可能的构想转化为未来势在必然的技术现实，Waymo 便是从内部实验演进为月订单超百万单的典型代表。 - [03:14](https://www.youtube.com/watch?v=8qApsMR7cDU&t=194s) 登月项目立项严格遵循“三要素筛选法则”，必须同时满足影响数亿人的全球重大难题、带来 10 倍飞跃的激进颠覆性方案，以及融合物理仿真、软件与 AI 的突破性技术路径。 - [08:44](https://www.youtube.com/watch?v=8qApsMR7cDU&t=524s) X 的 AI 定位并不局限于训练参数量更大的基座大模型，而是致力于构建连接混乱物理世界与数字智能的“结缔组织”抽象表征层，实现高可验证性与成本可控。 - [10:29](https://www.youtube.com/watch?v=8qApsMR7cDU&t=629s) 多模态 AI 架构经历了从解耦编码器、跨模态桥接、原生统一标记化到视觉-语言-动作（VLA）智能体控制调度的四代演进历程。 - [14:32](https://www.youtube.com/watch?v=8qApsMR7cDU&t=872s) 多模态智能体采用“思考-行动-反思”（Think-Act-Reflect）闭环，并将高层规划与底层专用物理/化学工具调用清晰解耦，彻底打破了单模型上下文窗口的限制。 - [18:07](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1087s) 端到端多模态执行链路由规划器交织 CoT/ToT 推理完成全局决策与工具路由，沉淀出具备高度跨项目复用性的工具分类体系，并强调全流程的可验证性。 - [21:02](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1262s) 在电池化学优化中，基于文献 RAG 的分子提议模型与专用物理仿真器形成闭环评估，将原本长达数年的研发周期大幅缩减至数周乃至数小时。 - [24:04](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1444s) 可持续建筑设计项目通过研发统一抽象的“连接语言”，融合法规、气候风险与结构力学等多维异构约束，借助生成式 AI 自主输出合规且低碳的建筑方案。 - [28:43](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1723s) Tapestry 项目利用多模态 AI 整合统一全美电网数据并自动化巡检，旨在攻克新增能源接入耗时过长的瓶颈，提高能源网络调配效率与韧性。 - [29:52](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1792s) Bellwether 项目聚合卫星遥感与传感器等多模态地理空间数据，显著提升对森林野火等极端气候灾害的预测精度，已与美国国民警卫队展开实战合作。 - [31:09](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1869s) Matera 与 A-Life 分别在分子微观层面实现废弃物的精准识别分选与新型细胞/分子的定制化生物制造，颠覆传统材料回收与生命科学制造范式。 - [33:37](https://www.youtube.com/watch?v=8qApsMR7cDU&t=2017s) 演讲总结出多模态 AI 登月的三大启示：多模态感知理解能力的爆发式演进、适配特定专业领域的精确表征构建，以及结合智能体规划器在复杂约束优化下的巨大潜力。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=8qApsMR7cDU&t=4s) X (The Moonshot Factory) 的使命：将“不可思议”变为“势在必然” 来自 X (The Moonshot Factory) 的首席 AI 研究科学家 Divya Choudhary 首先概述了人工智能领域的深刻转变。就在几年前，AI 模型的核心能力还主要局限于文本理解，而如今 AI 已经能够理解并综合处理多感官表征（multi-sensory representations），多模态 AI（Multimodal AI）由此为高精尖的科研及专业领域带来了颠覆性变革。 X 的宏观使命看似简单，执行起来却极具挑战性——发明并推出能够真正让世界变得更美好的“登月级”（Moonshot）前沿技术。这意味着要通过一套严密的机制去评估、筛选、甚至主动“杀死”（kill）绝大多数项目，并对少数具备潜力的项目加倍下注。其核心目标是“在当下把看似完全不理性或不可能（irrational or improbable）的事情，转变成未来势在必然（inevitable）的现实”。 Waymo 便是这一理念最典型的成功代表。Waymo 最初只是 X 内部的一个自动驾驶实验项目，在早期大家普遍认为这几乎是一项不可能成功的构想；而如今，Waymo 已经发展为全球最大规模的无人驾驶出租车服务，月服务订单量突破 100 万单。 - **Waymo**：从内部实验演变为全球最大无人驾驶出行服务，达成每月 100 万次行程。 - **多元化毕业项目生态**： - **Verily**：生命科学与健康医疗。 - **Wing**：自主无人机低空配送系统。 - **280 Earth**：直接空气碳捕集（Direct Air Capture），实现永久且经济的固碳。 - **Intrinsic**：工业机器人智能控制软件平台。 - **Tidal**：利用水下计算机视觉监测与保护海洋生态系统。 --- ### [03:14](https://www.youtube.com/watch?v=8qApsMR7cDU&t=194s) 登月项目的“三要素筛选法则” 很多人误以为登月技术类似于科幻小说，但实际上每一个项目在立项前都经过了严密的三重标准评估，只有三者的交集才能构成一个真正的 Moonshot： ```mermaid flowchart TD accTitle: Moonshot Evaluation Criteria subgraph CRITERIA["登月项目评估三角"] P["全球重大难题<br/>(Huge Problem)"] S["激进颠覆性方案<br/>(Radical Solution)"] T["突破性技术路径<br/>(Breakthrough Technology)"] end P --- S S --- T T --- P CRITERIA --> M["Moonshot 登月项目诞生"] ``` 1. **全球重大难题（Huge Problem）**：必须直面世界正在经历的严重问题，其影响范围需覆盖数以亿计甚至数十亿人群（如全球垃圾循环、电网与能源分配、供应链脆弱性、以及全球可持续住房短缺）。 2. **激进颠覆性方案（Radical Solution）**：绝不满足于现有技术基础上的修修补补；如果一项方案相比现有手段无法带来 10 倍（10x）的飞跃，就不能称之为激进方案。团队必须跳出当下软硬件的局限，去设想未来可被发明出来的新范式。 3. **突破性技术路径（Breakthrough Technology）**：不仅依赖单一技术，通常是物理仿真、前沿软件工程与核心 AI 能力的深度融合。 X 的探索本质上是一个高淘汰率的迭代过程，绝大部分概念会在早期被主动关停，只有极少数经过验证的项目才能最终独立分拆（Spin-out）。在过去 18 个月内，已有多个项目完成分拆独立运营： - **Aalyria / Taara**：利用高速光束（Light beams）突破传统限制，提供超高带宽的电力与无线光通信网络。 - **Heritable**：通过前沿计算提升作物应对气候变化的韧性，保障可持续农业。 - **Chorus**：构建端到端智能化感知的新一代物流供应链。 - **Anori**：运用多模态 AI 解决全球建筑与可持续住房危机（Sustainable housing）。 --- ### [08:44](https://www.youtube.com/watch?v=8qApsMR7cDU&t=524s) 构建现实世界与 AI 之间的“结缔组织” X 在 AI 与机器学习领域的前沿定位，并不局限于单纯训练参数量更大的基座大模型，而是致力于构建连接真实物理世界与数字智能的“结缔组织”（Connective Tissues）。现实世界的问题充满物理定律约束、极度杂乱（messy）且充满工程边界，X 的目标是建立一层高度可验证、可执行且成本可控的抽象表征层（Abstraction Layer），让多模态 AI 系统能够真正接入并重塑这些现实物理系统。 --- ### [10:29](https://www.youtube.com/watch?v=8qApsMR7cDU&t=629s) 多模态架构演进：从解耦编码到 VLA 智能体控制 早期的 LLM 受限于纯文本输入，面临着严重的“文本转换瓶颈”（Text conversion bottleneck）。当处理图像、连续传感器流或 3D 几何图谱时，如果强行把结构化/非结构化物理数据压缩成文本描述，必然会导致严重的特征信息丢失。现代 AI 的突破正是源于统一多模态标记化（Unified Tokenization）技术，使得来自不同物理模态的数据能够被映射到共享的表示空间中，从而催生了高效的多智能体（Agentic）协作架构。 ```mermaid flowchart LR accTitle: Multimodal AI Architecture Evolution subgraph G1["第一代: 解耦编码器"] E1["独立 Vision/Text 编码器"] --> R1["检索表现优异 / 无推理能力"] end subgraph G2["第二代: 跨模态桥接"] E2["冻结单模态编码器"] --> P2["Q-Former / 投影层桥接"] end subgraph G3["第三代: 原生统一基座 (至2024)"] E3["文本/音频/视频/传感器"] --> T3["端到端统一 Token 空间"] end subgraph G4["第四代: VLA 与 Agent 调度"] T4["统一 Tokenization"] --> A4["视觉-语言-动作 (VLA) 规划与专业 Agent 派发"] end G1 --> G2 --> G3 --> G4 ``` 多模态架构的四代演进路径： 1. **第一代（Decoupled Encoders，解耦编码器时代）**：文本与视觉采用各自独立的 Encoder 生成嵌入（Embeddings）。虽然因向量维度紧凑而非常适合高速检索，但模型本身具备“零推理能力”（Zero reasoning）。 2. **第二代（Bridged Representations，桥接时代）**：开始尝试连接两种模态。通常冻结一个成熟的 Encoder（如冻结 Vision Backbone），通过 Q-Former 或线性投影层（Projection Layers）将视觉表征映射对齐到语言模型的潜空间中。 3. **第三代（Native Unified Foundations，原生统一底座时代，至 2024 年）**：实现端到端的统一标记化（Unified Tokenization）。文本、音频、连续视频流以及工业传感器序列在同一个原生多模态架构中统一被 Token 化。 4. **第四代（VLA Agentic Control，视觉-语言-动作智能体控制时代）**：在统一多模态 Token 的基础上，叠加高阶抽象规划能力与视觉-语言-动作（VLA）执行机制。系统不仅能理解感知信号，还能进行任务拆解与逻辑规划，将具体指令精准分发调度给高度专业化的垂类 Agent 执行。 ### [14:32](https://www.youtube.com/watch?v=8qApsMR7cDU&t=872s) 智能体架构的核心逻辑与“思考-行动-反思”闭环 在多模态应用中，X（The Moonshot Factory）团队采用智能体（Agentic）架构来应对真实世界中的复杂性。这类架构的首要目标是将现实世界中混乱、不规整的表征转换为可量化、可验证的具体问题。其底层通用范式是经典的“思考-行动-反思”（Think-Act-Reflect）闭环。然而，如何设计“思考”、如何定义“行动”、如何构建专用的“反思”机制，正是 X 团队所谓“连接组织”（Connective Tissue）与独特卖点（USP）的体现。为了完成高质量的反思，团队构建了一系列高度专业化的引擎，例如基于物理学的仿真器、有限元分析（FEA）模拟器，以及专门用于寻找不同分子最佳组合以合成新型电池的化学工具。正是这些能够将混乱的输入转化为真实、可验证方案的专业能力，构成了推动项目落地的核心闭环。 在架构设计上，团队将高层“规划”（Planning）与底层“专用工具调用”（Specialized Tool Calling）进行了清晰的解耦。这种解耦带来了更多的调控杠杆：解耦程度越高，系统优化的自由度就越大。在自动化房地产等行业的复杂项目中（例如 Project Anori），整体流程完全由基于工作流的高层规划所主导；而每一次具体的工具调用，则由系统根据上下文决定在何时触发何种工具，并设定专业领域可接受的输出阈值。得益于多智能体架构，各智能体可以分头处理不同维度的任务，以往由于模型上下文窗口（Context Window）限制所造成的性能瓶颈已被彻底打破，开发者能够针对各个环节进行独立迭代开发。 - **Think-Act-Reflect 闭环**：通过感知输入驱动思考，调用物理或化学专用引擎执行与模拟，再利用专业指标反思并校正结果。 - **解耦设计优势**：将高层规划与底层工具分离，使得任务编排更加灵活，可针对特定工具设置专用的质量置信度阈值。 - **突破上下文限制**：不再强求单一大模型单次吞吐所有上下文，而是通过多智能体分工协同处理海量异构数据。 ### [18:07](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1087s) 端到端多模态智能体执行流程与跨项目复用 从工程视角来看，整个智能体执行链路从多模态传感器输入开始，数据首先进入多模态规划器（Multimodal Planner）。该规划器的职责是充分利用不同的数据模态进行全局决策，并交织融合思维链（Chain of Thought, CoT）与思维树（Tree of Thought, ToT）能力，以实现对多模态词元（Tokens）的深度理解和感知。随后，智能体根据所理解的任务自动完成工具选择（Tool Selection）。在此全流程中，“可验证性”（Verifiability）贯穿始终：必须持续验证规划器是否按预期分解目标、思维链是否符合人类严密的逻辑演进，以及工具调用是否准确匹配。 ```mermaid flowchart TD accTitle: Multimodal Agent Execution Pipeline subgraph S1["感知与规划层"] A["多模态感知输入 (Sensory Inputs)"] --> B["多模态规划器 (Multimodal Planner)"] B --> C["思维链 / 思维树 (CoT / ToT 推理)"] end subgraph S2["工具决策与路由层"] C --> D["自主工具选择 (Tool Selection)"] D --> E["统一协调与路由协议 (Routing Protocol)"] end subgraph S3["执行与反馈层"] E --> F1["神经符号 / 神经 API"] E --> F2["物理仿真 / FEA 模拟器"] E --> F3["执行机构 / GUI 坐标"] F1 & F2 & F3 --> G["动作执行并改变环境状态"] end ``` 工具分类体系（Tool Taxonomy）随着攻克的问题域不断拓宽而持续丰富，且具备极高的跨项目可复用性。例如，在电池化学项目中积累的分子搭配与评估工具，可直接迁移至生命科学领域（例如旨在构建更健康细胞的新型分子合成项目）。这些底层工具的形式非常多元，既可以是神经符号（Neural-Symbolic）接口或神经 API，也可以是 GUI 坐标操作、物理执行机构驱动或专业仿真器。最终，由一套统一的协议收集并路由所有的工具调用结果，执行具体动作，进而推动外部环境状态发生转移。 ### [21:02](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1262s) 电池化学优化：利用 RAG 与多智能体仿真颠覆研发周期 为了展现多模态 AI 在实体科学领域的威力，演讲者详细剖析了电池化学优化项目。电池材料的探索面临着离子电导率与化学稳定性的严苛双重要求：若两个候选分子的结合无法达到特定的物理阈值，合成出的电池就会极不稳定。针对这一痛点，团队构建了一套由两个核心部分组成的多多模态智能体系统： 首先是一个基于化学文献与学术论文 RAG（检索增强生成）的“提议模型”（Proposal Model）。该模型深入海量科学论文中提取有效洞见，自动生成用于电池离子的候选提议；其次，这些候选方案会被输送至多台仿真器中进行评估，测量其离子电导率和结构稳定性。系统通过这种持续提优候选变体的反馈回路，筛选出兼具高电导率与高稳定性的最优配对。这一多智能体仿真平台彻底改写了研发节奏，将以往通常需要耗费数年的新型电池研发周期，断崖式缩短到了“数周甚至是数小时”。 - **双核心系统架构**：基于化学文献 RAG 的“变体提议智能体”负责生成分子方案；专用仿真引擎负责评估性能与安全阈值。 - **研发周期质的飞跃**：通过自动化高置信度多目标筛选，直接替代了传统实验室大量盲目试错的过程，使研发效率提升了数个数量级。 ### [24:04](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1444s) 可持续建筑设计：以多约束统一表征重塑人居建造 另一个凸显多模态 AI 价值的核心项目是可持续建筑设计（Sustainable Building Design）。建筑行业对全球环境影响深远，产生了全球约 37% 的碳排放。与此同时，全球正面临严重的住房短缺危机，预计到 2030 年将有约 30 亿人口需要充足的住房保障。X 团队设定的激进目标是：在大幅缩短房屋建设周期的同时显著削减建造成本，实现兼具“可持续性”（Sustainable）与“可负担/易获取性”（Attainable）的人居环境。 然而，建筑设计是一项极其庞杂的系统工程，无法依赖训练单一模型来解决。它需要融合多重异构数据体系： 1. **法规数据**：因行政辖区（Jurisdiction）而异的强制性建筑规范； 2. **风险数据**：当地天气模式与气候灾害风险； 3. **结构数据**：确保建筑稳固的物理力学法则与材料强度； 4. **生态与环境数据**：材料可持续性、项目对周边生物多样性的影响，以及建筑本身的环境舒适度设计。 为了打破不同数据类型之间的孤岛，团队研发了一种独立于各模态之上的“连接语言”（Representation/Language）。该语言将各类专业领域的约束统一抽象为内部向量表征，并结合计算几何（Computational Geometry）工具箱来验证设计的合规性与结构合理性。在这一统一语言基底之上，系统构建起了可自动化进行方案设计与规范审查的 AI 引擎。演讲者展示了一个在亚利桑那州凤凰城（Phoenix）真实地块上的应用案例：生成式 AI 模型在完全理解该地块的容积率、停车配比、住宅规划以及特定辖区法规等多重严苛约束后，自主生成了结构合理、兼顾环保与可负担性的多层建筑完整设计方案。 ### [28:43](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1723s) Tapestry 项目：重塑可视化电网与能源网络 在建筑领域之后，X（The Moonshot Factory）内部利用多模态 AI 的另一个重磅项目是 **Tapestry**。该项目致力于让整个电网变得清晰可见，以确保各类可靠能源的高效接入与调配。电力中断在过去始终是一个棘手的现实难题，以往接入一种新的能源甚至需要耗费长达 6 年以上的时间。针对这一巨大瓶颈，Tapestry 构建了一个由 AI 驱动的平台，旨在整合并统一全美范围内的电网数据，优化并加速能源流动。 面对这种国家级规模的复杂电网问题，多模态 AI 展现出了不可替代的价值。它不仅能综合处理各类异构数据流，还能实现基础设施巡检和故障停电的自动化诊断与响应，极大提升了能源网络的韧性。 - **并网瓶颈**：以往新增能源接入电网平均需要 6 年以上。 - **核心能力**：通过多模态 AI 统一全美电网数据，自动化巡检关键基础设施，加速电力调配并降低断电风险。 --- ### [29:52](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1792s) Bellwether 项目：利用多模态地理空间数据预测气候灾害 另一个应对全球性挑战的关键项目是 **Bellwether**。该项目专注于预测气候变化以及可能危及地球生态环境的各类自然灾害。近年来极端天气与气候灾害频发，美国在连续 4 年内就遭遇了超过 18 起十亿美元级别的重大灾害（billion-dollar disasters），这亟需一种全新的技术解决方案。 针对气象预测中数据永远不足的痛点，Bellwether 汇聚并利用了海量的多模态地理空间数据（Geospatial Data）。在气候预测领域，整合的数据模态越多，模型的预测结果就越精准。Bellwether 能够精准预测森林野火（Wildfire）等各类复杂气候风险，目前已与美国国民警卫队（National Guard）以及蒙大拿州（State of Montana）政府等多方展开落地合作。 ```mermaid flowchart TD accTitle: Bellwether 气候多模态预测架构 subgraph Input["多模态数据输入"] A["卫星遥感图像"] B["气象监测传感器数据"] C["地理空间环境数据"] end subgraph Core["Bellwether AI 预测引擎"] D["多模态表征融合与分析"] end subgraph Output["预测输出与实际应用"] E["森林野火风险预测"] F["极端气候灾害预警"] G["协同合作<br/>(国民警卫队 / 蒙大拿州)"] end A --> D B --> D C --> D D --> E D --> F E --> G F --> G ``` - **灾害背景**：美国连续 4 年发生 18 起以上损失超十亿美元的极端气候灾害。 - **数据策略**：融合尽可能多的地理空间数据模态，弥补单一气象数据维度的不足。 - **合作伙伴**：已与美国国民警卫队、蒙大拿州政府等机构建立实战化合作。 --- ### [31:09](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1869s) Matera 与 A-Life：分子智能垃圾回收与生物精准制造 在物质与生命科学领域，X 还孵化了 **Matera** 和 **A-Life** 项目。Matera 的核心理念在于运用“分子智能”（Molecular Intelligence）来革新全球垃圾管理。通常人们对垃圾的分类仅停留在宏观部件或材料类别层面，而 Matera 构建的多模态 AI 技术则深入到分子层面识别垃圾中存在的不同分子，并按照分子特征对废弃物进行精细化归类与回收。这种细粒度分选大幅提升了废弃物的可重复利用率。演讲者提到一个令人震惊的数据：人类从地球上开采掠夺的资源中，有整整 75% 最终又被直接丢弃回了地球表面，而 Matera 正试图用多模态 AI 改变这一现状。 与 Matera 相呼应，**A-Life** 项目则探索生物制造（Manufacturing of Biology）的未来。该项目利用先进的生物系统，以极高精度制造出具有特定期望特征的新型细胞和目标分子。这种前沿的登月构想需要极其深厚的生物学专业积淀，通过多模态 AI 系统整合跨学科数据，使得以“更快、更好、更低成本”（"faster, better and cheaper"）的方式进行生物制造成为可能。 - **Matera 分子级回收**：突破传统材质分类，在分子层面识别与分选废弃物，改善“人类从地球获取的 75% 物质最终沦为垃圾”的严峻现状。 - **A-Life 生物制造**：结合深厚生物学专长与多模态模型，高精度定制制造细胞与分子。 --- ### [33:37](https://www.youtube.com/watch?v=8qApsMR7cDU&t=2017s) 总结与三大核心启示：多模态 AI 的登月方法论 在回顾了众多落地于现实世界的登月项目后，演讲者总结了本次分享希望大家带走的三大核心启示（Three Takeaways）： 1. **多模态能力迎来了跨越式爆发**：多模态 AI 模型的理解与感知能力过去几年实现了惊人增长，并将持续演进。这使得许多几年前被认为“绝无可能解决”的难题，如今在多模态语言与感知理解的赋能下迎刃而解。 2. **构建适配领域特性的精确表征（Representation）是破局秘诀**：面对具有高度专业领域特异性（Domain-Rich）的问题，研发的核心在于找到最合适的表征方式。这种表征既能赋予模型强大的泛化能力，又能在具体业务层面保持严格的可验证性（Verifiability）。 3. **AI 在抽象头脑风暴与约束优化上的巨大潜力**：现代 AI 在面对复杂约束条件下的方案优化与发散性构想时表现卓越，而这一能力在配合“智能体工具调用（Agentic Calls）与规划器（Planner）”后得到了前所未有的放大，使多模态系统在跨项目的实际落地中更可靠、更具实用性。 演讲最后，讲者展示了 X 的官方网站与个人联络方式，鼓励与会者进一步探索更多 X 工厂正在探索的前沿登月项目，并对现场听众表达了由衷的感谢。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=8qApsMR7cDU&t=4s) 在当下把看似完全不理性或不可能（irrational or improbable）的事情，转变成未来势在必然（inevitable）的现实。 — Divya Choudhary > [03:14](https://www.youtube.com/watch?v=8qApsMR7cDU&t=194s) 如果一项方案相比现有手段无法带来 10 倍（10x）的飞跃，就不能称之为激进方案。 — Divya Choudhary > [08:44](https://www.youtube.com/watch?v=8qApsMR7cDU&t=524s) 致力于构建连接真实物理世界与数字智能的“结缔组织”（Connective Tissues）。 — Divya Choudhary > [14:32](https://www.youtube.com/watch?v=8qApsMR7cDU&t=872s) 解耦程度越高，系统优化的自由度就越大。 — Divya Choudhary > [21:02](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1262s) 将以往通常需要耗费数年的新型电池研发周期，断崖式缩短到了“数周甚至是数小时”。 — Divya Choudhary > [24:04](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1444s) 预计到 2030 年将有约 30 亿人口需要充足的住房保障。 — Divya Choudhary > [29:52](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1792s) 在气候预测领域，整合的数据模态越多，模型的预测结果就越精准。 — Divya Choudhary > [31:09](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1869s) 人类从地球上开采掠夺的资源中，有整整 75% 最终又被直接丢弃回了地球表面，而 Matera 正试图用多模态 AI 改变这一现状。 — Divya Choudhary > [31:09](https://www.youtube.com/watch?v=8qApsMR7cDU&t=1869s) 以“更快、更好、更低成本”（"faster, better and cheaper"）的方式进行生物制造成为可能。 — Divya Choudhary > [33:37](https://www.youtube.com/watch?v=8qApsMR7cDU&t=2017s) 面对具有高度专业领域特异性（Domain-Rich）的问题，研发的核心在于找到最合适的表征方式。这种表征既能赋予模型强大的泛化能力，又能在具体业务层面保持严格的可验证性（Verifiability）。 — Divya Choudhary ## Mentioned Divya Choudhary, X (The Moonshot Factory), Ai4, Waymo, Verily, Wing, 280 Earth, Intrinsic, Tidal, Aalyria, Taara, Heritable, Chorus, Anori, Tapestry, Bellwether, Matera, A-Life, 美国国民警卫队 (National Guard), 蒙大拿州政府 (State of Montana) ## Video description Divya Choudhary - Lead AI Research Scientist, X, The Moonshot Factory [Source](https://www.youtube.com/watch?v=8qApsMR7cDU)

[0:04] Divya Choudhary: 在当下把看似完全不理性或不可能（irrational or improbable）的事情，转变成未来势在必然（inevitable）的现实。 [3:14] Divya Choudhary: 如果一项方案相比现有手段无法带来 10 倍（10x）的飞跃，就不能称之为激进方案。 [8:44] Divya Choudhary: 致力于构建连接真实物理世界与数字智能的“结缔组织”（Connective Tissues）。 [14:32] Divya Choudhary: 解耦程度越高，系统优化的自由度就越大。 [21:02] Divya Choudhary: 将以往通常需要耗费数年的新型电池研发周期，断崖式缩短到了“数周甚至是数小时”。 [24:04] Divya Choudhary: 预计到 2030 年将有约 30 亿人口需要充足的住房保障。 [29:52] Divya Choudhary: 在气候预测领域，整合的数据模态越多，模型的预测结果就越精准。 [31:09] Divya Choudhary: 人类从地球上开采掠夺的资源中，有整整 75% 最终又被直接丢弃回了地球表面，而 Matera 正试图用多模态 AI 改变这一现状。 [31:09] Divya Choudhary: 以“更快、更好、更低成本”（"faster, better and cheaper"）的方式进行生物制造成为可能。 [33:37] Divya Choudhary: 面对具有高度专业领域特异性（Domain-Rich）的问题，研发的核心在于找到最合适的表征方式。这种表征既能赋予模型强大的泛化能力，又能在具体业务层面保持严格的可验证性（Verifiability）。

[0:04] Divya Choudhary: to invent and launch moonshot technologies that can actually make world a better place to live. [3:14] Divya Choudhary: Huge Problems, Radical Solutions, and Breakthrough Tech [6:51] Divya Choudhary: connective tissue [14:42] Divya Choudhary: Think, Act, and Reflect [31:09] Divya Choudhary: 75% of everything we strip from earth actually goes back into the earth

[0:04] Welcome to the talk. I am Diva Chadri and I  am highly privileged to be representing X the [0:10] moonshot factory today. U today what I want to  talk about is the shift that we have seen in the [0:18] AI world. Just a few years ago, the AI models  were very good at understanding text, but now [0:26] they can understand multi-ensory representations  and comprehensions. And that's what multimodal AI [0:34] has brought uh in and has revolutionized what we  can achieve from that. So throughout this talk, [0:41] you'll see a lot of applications of multimodal  AI in highly scientific and domainrich fields. [0:50] So I said I come from X the moonshot factory and  a little bit about what we do is let me start from [1:00] our mission. Uh our mission it's it's very simple  to state but extremely difficult to execute. uh [1:09] we say that our overarching mission is to invent  and launch moonshot technologies that can actually [1:17] make world a better place to live. [snorts] And  you'll see across our projects this reflecting [1:23] in different ways that we choose our projects, we  kill our projects, we double down on some of them. [1:32] What we aim at is turning something that's  completely irrational or improbable at [1:40] this point of time into something that  will become inevitable in the future. [1:45] Uh and that [snorts] could be said through a  story of one of our spinouts which is Whimo. [1:52] Whimo started as a autonomous driving project  within X and back then it just was an idea that [2:01] would not work most likely. That's what people  thought about it. And today you can see that [2:07] Whimo is the the the largest scale driverless  taxi service that is available right now doing [2:14] 1 million rides every month. So that's what we  mean by converting improvable into inevitable. [2:22] There are multiple of other projects that we have  launched. A few of them very familiar like they [2:30] span multiple domains. So you can see Verily in  life sciences to Whimo we talked about it to wing [2:37] which is autonomous drone delivery to maybe 280  earth which is basically doing carbon capturing [2:44] from the air to intrinsic which is industrial  robots um to tidle. title is basically maintaining [2:52] the ocean ecosystem uh through monitoring through  computer vision and a lot of all these projects [2:58] uh you'll see and you you hear about it they  are targeting some of the problems that were [3:05] there when they started and they have solved  it. How do we think about building a moonshot? [3:14] Usually people say are all these moonshots more  like a science fiction. Uh but they aren't. [3:22] There are a lot large portion of this which is  highly thought into each of our projects. Uh [3:31] before we dive into any of the investigation, we  evaluate our investigation into three criterias. [3:39] The first one is it has to solve a huge problem  that the world is facing right now. It should and [3:45] when we say huge it should affect millions  if not billions of people and we look for [3:53] solving that problem through radical solution  uh a solution that exists right now that might [3:59] be solving the problem or part of the problem  right now. If we cannot get a 10x leap from that [4:07] uh from our solution with our solution then that's  not a radical solution that we're looking at. So [4:13] we train ourselves to think about solutions in  that way and then there is always a breakthrough [4:19] technology where we hope that that is a technology  that will help us solve that. We don't start from [4:26] there. So that's how the three the intersection of  these three things play into making a moonshot. Uh [4:35] we talked about a huge problem, right? So the  huge problem actually presents itself in the [4:41] world and we have plenty of them. Uh so we all  agree on that, right? So the first thing you can [4:50] see there are plenty of these problems that we  have just put on this slide which is more like [4:55] uh recycling waste. The global waste is a problem.  Nobody uh has a very efficient solution about it. [5:03] So we are tackling one of our projects is tackling  that. Then there are problems around efficient [5:09] supply chain. There is problem around electricity  and the grid facility across the world. there is [5:15] problem around sustainable housing and we have  projects around all of that. So that's how we [5:21] identify the real huge problem in the world.  And when we say the uh radical solution, [5:30] it is not always obvious that the radical solution  will be something that's that's out there. [5:38] So we train ourselves to think in a way that what  could be a solution that can change this or what [5:45] could be a solution that can happen in the future  doesn't exist right now but can be invented in the [5:51] future that can solve that. Um so it's not just  limited with the physical or software technologies [6:00] that we have right now. And then the breakthrough  technology is always uh something that will be a [6:09] combination of a physical simulation software and  a lot of AI to drive all of these projects. When [6:19] we look at any of the moonshot ideas working on  them is actually an iterative process. So we at [6:27] any point of time we are looking at multiple  of these ideas and then only a few of them [6:34] survive. We intentionally kill multiple of  them and double down on few of them that [6:41] survive to actually spin out as an independent  project or companies that you get to know about. [6:51] These are just a few of the projects that  have actually spun out in the last 18 months [6:57] uh out of X as independent companies and  independent projects. Anori is one of the [7:05] projects that's very close to my heart. I've  worked on it intentionally. Uh and they are [7:11] trying to solve sustainable housing. So there  is a scarcity of housing. We'll talk in detail [7:17] about it as an example of how multimmodal AI has  been applied there. But they were trying to solve [7:23] more [snorts] about all the housing crisis that  exists right now around the world. Heritable is a [7:30] project that is trying to create uh more resilient  crops. So making agriculture more sustainable and [7:38] resilient. 280 Earth is a project that is removing  carbon from the atmosphere. uh finding newer ways [7:46] to do that uh very affordably and permanently.  So those are the things that we there could be [7:52] other projects that could be doing it but um that  might affordable might be a 10x factor for some of [7:59] our projects. Tara is [laughter] another project  that is uh independent and it is out there. uh [8:07] they're trying to solve how do we get the fast  and abundant um electricity and connectivity uh [8:14] across using the light beams and title is another  project that is uh looking at how do we do ocean [8:22] monitoring ocean life monitoring sustainably and  chorus is improving how we make the supply chain [8:30] more intelligent. So you can read all about these  projects on our website. Uh but the these are a [8:35] few of them that have recently graduated. So  what sets X um as the leading edge of AI and [8:44] ML? Uh and what we'd like to think about is we  are trying to build not just larger models, not [8:55] just the AI capacity to solve few of the existing  problems, but we are trying to look further ahead [9:04] where we are trying to build connective tissues  that will help us move a real problem in the world [9:12] into a domain that is that's verifiable, that  is doable and is affordable at the same time. [9:20] So we are building that representation layer  or that abstraction [clears throat] layer [9:25] which helps us connect these problems and solve  them for real and we want to cover every aspect [9:32] of that and that's why you'll see that a lot of  multimodal AI and the different parts of the AI [9:39] that I'll talk about plays very nicely into this  uh whole idea of building the connective tissue. [9:49] So goal for like pivoting from what moonshot does.  I hope that's clear right now uh and has got you [9:58] interested into what we will talk about next.  We will talk focus more on a little bit of the [10:04] technical part which is how we moved from the text  understanding to multiensory comprehension in U AI [10:12] world which is the multimmodal AI integration  techniques and throughout after that I'll [10:18] give examples of how we at X have driven this  through using multimodal AI across our projects. [10:29] uh to begin this I wanted to talk a little  bit about the paradigm shift that we have seen [10:35] uh in the multimodality in the beginning there was  always use of LLM was restricted by using just the [10:43] text prompt um that was the beginning of LLMs and  that led to a text conversion bottleneck for a lot [10:51] of the applications so if you're looking into a  problem that involves you looking into images or [10:59] sensor data or maybe 3D graphs and others. It was  very difficult to convert them all into a textual [11:06] representation and not lose information while  doing that and put that into an LLM as a prompt. [11:13] So that has shifted now. Now we have more high  highly rich and multimodal unified tokenization [11:23] approaches that help us get tokens that actually  embed all the information coming from different [11:31] modalities of the data and that is what the shift  has actually uh enabled us with and that has also [11:41] given birth to this agentic system that uh is  using these unified tokenization. Um so we use [11:49] agents for a lot of our activ like our uh tasks  and when you use multimodal uh tokens it becomes [12:00] even easier to direct to different agents  depending on their specific specializations. [12:08] Now these the shift that I wanted to talk about  is there definitely is the multimodal AI there's [12:17] definitely unified tokenization and all the  technology that has come up but what is extremely [12:23] important is how do you apply it to convert a  real messy phys physical world physics governed [12:30] problem into something that can actually use that  and that's what will be something that I'll talk [12:37] about this is the architecture evolution how we  have grown from uh decoupled encoders where you [12:46] had a vision encoder a text encoder you'll get all  these embeddings separately they were ex excellent [12:54] at retrieval because embedding sizes are smaller  they will help you retrieve very fast but then [12:59] there was zero reasoning in the beginning then  we moved to gen two of this architecture which [13:06] is actually building bridges so that you  can combine two modalities together. And [13:12] that actually very beginning we started with  freezing one of the encoders and trying to [13:19] uh train the other uh through Q former or other  projection layers and then that actually was [13:27] little bit easier to use different modalities  but still not the best. Then we came with Gen 3 [13:34] which was native unified foundations for all  these unified tokenizations where you will [13:41] actually have an end toend tokenization across  different modalities. So you'll have text, audio, [13:47] video, other sensor data if you want to have  all of that will be combined into a unified [13:52] tokenization. That was till 2024. And then in the  last couple of years we have seen that this has [14:02] uh turned into an um VLA uh agentic um control  where you have these unified tokenization but [14:12] you also have their ways to actually the  abstractions to use them into an agentic [14:18] mode where you can direct you can do the planning  and you can direct the output to actually go to [14:25] agents that are highly specialized And all these  specializations can be written uh and enhanced at [14:32] our end in this agentic architecture. I wanted to  talk about how we think about this problem. Uh we [14:42] have used the we use agentic architecture uh for  all the multimodal applications. uh there is a [14:50] need for you to convert this messy representation  of a problem into an actual verifiable problem uh [15:00] which is quantified and verifiable. How do you do  that using an agentic representation is it's very [15:07] common practice to use the think act and reflect  uh cycle where you have to now that this is where [15:15] the role of X and what I was talking about  connective tissue uh comes into picture where [15:21] how do you design the think part how do you design  the act part and how do you design the reflect now [15:27] what happens how many things do you have which  are specialized to actually do reflect We have [15:33] engines that are specialized in physics- based  simulations. Uh we have engines which are FEA uh [15:41] uh finite uh estimate estimations uh simulators.  There are tools that we have built which are more [15:49] around finding what are the the best combination  of different molecules uh that combine together to [15:56] form batteries. uh and all of this specialization  that I'm talking about is the connective tissue [16:03] that is the USB of any project that you want to  solve that converts from that messy world into a [16:12] real world. Uh so that's the real loop that we try  to apply to different projects to solve this and [16:21] then we try to decouple the planning part from the  specialized tool call part. So these days even all [16:29] the agentic system that you see have a huge focus  on how do we make our tool calls much better. [16:37] uh how do you combine the planning part and  reasoning part with a tool call part uh in the [16:44] same agentic system and the more you decouple  it the more levers you have to make it better [16:55] and you'll see across the projects that we have  tried that the planning part is something that [17:01] is actually governing the overall flow of how the  problem needs to be solved. If you're automating [17:08] the entire real estate industry uh like uh the  project Anori um you'll have to do a planning [17:14] which is based on the workflows and then with each  specialized tool call you are supposed to decide [17:22] on what is the right call right right tool to be  called at what instance and the thresholds for [17:28] when you say that the tool has returned an output  that is acceptable in these domain specific tools [17:36] um they actually are not governed by context  now because the context window part that we we [17:43] had as a bottleneck earlier those are no longer  there because of you can use multiple agents to [17:50] do multiple things and multiple aspects of it and  you can actually develop each of them separately [17:55] and that is where the um the beauty of the problem  solving comes. The this is the flow. This is how [18:07] it looks like in if I had to give a picture of  how the flow works. Uh you will you'll get some [18:15] sensory inputs. uh you put that into a multimodal  planner which is the goal of the multimodal [18:22] planner is to actually use the different  modalities in the best way that's possible [18:27] to take a decision at hand and then you interle  this chain of thought uh and uh chain of thought [18:34] or tree of thought capabilities in order to come  up with the actual perception or the actual um [18:42] comprehension of the tokens And then you do a tool  selection. So your agent is then looking at the [18:50] real problem that it is trying to solve through  these prompts and do the tool selection by itself. [18:57] Uh and there will be a verifiable property on each  of these steps that your planner is doing what [19:03] it is supposed to do. Your uh chain of thought is  actually working the way that you would logically [19:10] think of thinking as a human. your tool selection  is happening the way it is supposed to happen. And [19:16] so that's why verifiability becomes integral part  of each one of these components. And once you do [19:22] tool selection, this tool taxonomy is something  that uh can be built and grown as you dive into [19:31] more domains of the problems and the best thing  about it is that you can keep reusing them across [19:37] projects that have an overlap of the problem sets.  So if you have worked in the battery domain and [19:45] you know about different molecules and different  combinations that go together uh you can also [19:51] maybe use parts of it into understanding the newer  molecular synthesis that can happen that has maybe [19:59] that is what are the part of the other projects  like a life where they are trying to build new [20:04] cells that can do things that we want like work  exactly the healthier way that we would like to [20:11] Uh so we can they're definitely reusable  across projects and they can they can be like [20:18] uh neural uh symbolic or neural APIs or they  could be like GUI uh coordinates or they could [20:26] be physical actuators or other simulators that you  can build. And then after these tool calls happen, [20:33] you actually have this some kind of a protocol  that collects this together, routes them together, [20:40] uh and then have the action executed and you  have the the shift of the environment. So [20:46] this is the entire flow that that's very  common across uh the agentic execution. [20:54] Now this this was the brief of the multimodal  AI technology part. I wanted to show something [21:02] that is how we have used this across different  applications uh and the projects in addex. I was [21:12] talking about the battery chemistry optimization.  So this will also give you a sense of the diverse [21:19] range [snorts] of projects that we do at X. And  one of the project is based on battery chemistry [21:26] optimization uh where we built a multimodal AI uh  agentic system that is that has two components. [21:35] There is one proposal model that is trying  to use a rag which is based on the literature [21:42] in chemistry uh and all the uh the scientific  papers that can actually be useful in battery [21:50] uh optimization and it is trying to come  up with different uh proposals of the ions [21:57] that can be used for a battery and then that  is those different proposals have thresholds [22:03] that we have defined And they're supposed to be a  good candidate that you can evaluate on different [22:11] um simulators in order to understand their ionic  conductivity, their stability once they become [22:16] a battery because um you can't combine if they  do not reach a particular threshold. You cannot [22:23] combine two different uh proposed variants to make  a battery that might not be stable. Right? So, [22:31] so you do you go through this loop of optimizing  this proposed variance and then find out what is [22:38] the top variant that's both uh scoring high in  the ionic conductivity as well as um stability. [22:46] And just this uh multi-agent um simulation has  reduced years of uh effort into finding newer [23:00] uh uh variants that can make batteries to  just weeks or even hours of running this. [23:07] So that's one way that how you use this multimodal  AI into um reducing and making the systems more [23:18] efficient. This is the uh the project in in its  full form working trying to come up with different [23:27] uh suggestions for the variants and it tries to  optimize on getting you the best pairs that can [23:36] be combined together uh based on their thresholds  that you see there on the confidence level that [23:43] we have on that uh projection. and we do this  get the projections and pass them through the [23:51] actual evaluators. The second project that I  wanted to talk about that can actually give [23:57] you a sense of how again multimodal AI plays a  vital role is sustainable building design. Uh [24:04] building design is a global problem. It leads to  around 37% of uh carbon emission in the world. uh [24:13] it's difficult to put that number into context  that buildings just designing the buildings and [24:18] building buildings actually leads to that much of  emission and we also know that there is shortage [24:24] of housing in this world and by 2030 we'll need  around 3 billion uh people would need adequate [24:32] housing. So the definitely a huge problem to be  solved and the radical solution that we thought [24:40] about is we will try to reduce the time that it  takes to build a housing and also the the amount [24:50] of expenditure that you have to do to build  a housing. So both sustainable and attainable [24:57] housing. Now in order to achieve that there this  entire housing domain is has multiple types of [25:07] data that you have to deal with. Uh it's not one  type of data and you train a specific model for [25:14] it and it works. You have regulatory data which  is governing how your houses can be it. Every [25:21] jurisdiction will have a different regulation  and you have to abide by it. There could be [25:26] risk data which is based on the weather and the  climate in that area. There is structural data [25:32] which is governing how strong the building is and  you have to abide by that structural law as well. [25:39] There could be materials data, biodiversity data,  uh how you're contributing to the biodiversity of [25:45] the community that you're building and then  other aspects like design of the building and [25:51] uh environmental uh um factors that can contribute  to making the building nicer or stronger depending [26:01] on what you put into it. So all of this data needs  to be used and somehow has to be balanced in order [26:12] to create something that's actually worth living,  actually worth uh our promises of attainability [26:19] and sustainability. And in order to solve this  problem, what we thought about is as connecting [26:27] tissue uh that we were talking about, we thought  about building some kind of representation that [26:33] will help us move from one data type to the other  data type and still get all of them together as [26:40] multiple constraints that the model has to solve.  And that's where you'll see that we try to create [26:46] a language which is different from all these  modalities that sits on top of this and tries [26:54] to con combine or do a planning of converting  these different problems into subdomains like [27:03] data generation. uh we do internal vector uh  representation of all these problems and we [27:11] come up with performance metrics with that I was  telling you about verifiability. Once you have [27:16] your own representation you need to have some  kind of performance metrics to say that this [27:21] is the right representation of the data that you  have and then you do some tools because this is [27:27] uh computation it can involve computational  geometry. So you have different toolbox in [27:32] your computational geometry in order to  understand that the building uh design is [27:36] the right design and it is structurally sound.  And then from there we define AI engine that can [27:44] do the design that can do compliance. All of  that aspect that we talked about can then be [27:51] built on top of this language. uh once you  have the the ground built in that language [28:00] and this is uh one of the uh work  that we did where this is a multi- [28:06] constraint optimization design generative AI  problem that this building is being built by [28:16] uh models the multimodel agents that  understand all these constraints and given this [28:23] lot. This is a real lot in Phoenix. Uh it can  actually build depending on your constraints [28:30] that these are these many floors. This is  my uh jurisdiction uh regulation and this [28:37] is the area that I can use for residential  for parking and all other factors. It will [28:43] create buildings that are structurally  sound and meets the performance metrics. [28:50] Other projects uh around X that use  multimodal AI is like tapestry. Tapestry is [28:57] uh trying to make the electric grid visible for uh  for all kinds of reliable energy. Uh power outages [29:06] uh they had been a real problem in the past and  it has been difficult. It takes around six plus [29:12] years to connect new energy sources and uh  that's why there was this huge problem that [29:20] we wanted to solve and we have an AI uh enabled  platform that actually unifies grid data across [29:28] um uh the country and it tries to  optimize how the energy flow will happen [29:34] um and accelerate that. So this is not uh a  small scale problem but using the multimodality [29:42] aspect it actually helps to automate the  infrastructure inspection and uh outages. [29:52] Then there is another project called bell  weather. Um it the bell weather is basically [29:58] around predicting what will happen the changes  in the climate uh or other issues that can happen [30:07] that can harm the planet. Um so the weather and  the climate disasters they have been increasing [30:14] in the past. Uh there around 18 plus billion  disasters in the US over four consecutive years [30:21] and it needed another uh solution. So we use the  AI using the geospatial. So here also there could [30:28] be different you can think about modalities  of data that you would need in order to solve [30:33] something that's weather or climate related.  Um I think any modality like as many number [30:40] of modalities you would uh have the better your  predictions will become. Uh you'll always have [30:45] lesser data when you're predicting weather. Right?  So it all it harnesses all the geospatial data [30:53] uh and it tries to predict um wildfire all  kinds of different uh weather predictions that [30:59] can happen and they are um already partnering  with national guard state of Montana and more. [31:09] Then there is project called Matera uh and  they are about how do we do use the molecular [31:16] intelligence uh to make uh global waste much more  uh when we do waste management it the management [31:28] part of it can be done much more better if you  know the molecular intelligence of the waste. [31:34] uh and this is a unique idea in terms  of how material recycles uh the waste. [31:42] We have built um we have built technology which  is again multimodal which tries to identify the [31:49] different molecules that exist in each waste so  that you can categorize the different molecules [31:56] and waste as per the molecules and not just by  the um by the actual component or the type of [32:02] material that it has used. So once you do that  sorting what happens is they become a lot more [32:10] reusable as compared to what happened earlier.  Uh this was uh news to me that 75% of everything [32:17] we strip from earth actually goes back into the  earth. So we uh that's how we uh do our waste [32:25] management right now. Uh and all of this again  uses multimodal uh AI system. And then there [32:33] is another project called a life uh which is uh  doing kind of a manufacturing of biology a part of [32:45] uh uh of creating new cells um that are that can  have features that we want them to. So these uh bi [32:55] biological systems uh can actually manufacture  molecules uh with um high precision and [33:04] uh right now this moonshot idea is still something  that is not very common and it requires very deep [33:14] biological expertise. So you can think about  different areas of data collection that you [33:21] can do for this and actually use a multimodal  um system that can help you use that deeper [33:28] expertise in order to create something  that that's faster, better and uh cheaper. [33:37] And overall I talked about I know a lot  of projects um a lot of uh applications [33:44] of multimmodal AI. Uh if there are three  things that I would like you to take away [33:50] uh from this is that the multime model uh  capabilities of um models and AI systems [33:59] uh have been increasing tremendously and they  will increase even further and that has enabled [34:06] us to actually see different problems that were  actually impossible to be solved a few years ago [34:15] be solved world now by using this multimodality  and multimodal langu uh understanding. So that [34:22] is a significant growth that we have seen in the  community uh and in the research community and the [34:27] AI world. Now creating the right representation  uh for domain specific problems is the actual [34:36] uh recipe for solving these problems. Whenever you  have very specific domainrich problem to solve, [34:45] uh what we focus on is creating the right  representation that can help us make it more [34:51] generalizable and still have verifiability at a  specific level. And then uh the last thing that [34:59] I wanted to highlight is uh how what is the  AI's potential for abstract brainstorming and [35:07] uh constraint optimization with and all of  this is enhanced with the right set of this [35:13] tools that you can actually build now. uh  and this we used to build this earlier as [35:19] well but with the agentic calls that can  actually call these tools very easily with [35:25] a planner this becomes even much more usable  uh and much more reliable uh across projects [35:34] and these were all different projects that we  do and these were uh just few of the glimpses [35:42] of the projects that we uh handle at X um and  you feel free to read more about it. Um, and [35:50] uh, this is where you can connect with me if  you would like more information. And this is [35:56] uh, X's website where you can learn more  about. Thank you so much for your time.