Transformer Co-Author Kaiser Sees New AI Models Breaking Big Tech Grip

Sophia Dew, Lukasz Kaiser

Lukasz Kaiser argues today's AI centralization is merely a temporary engineering shortcut, predicting algorithmic breakthroughs will enable specialized models to match massive systems on far less data.

AI-assisted summary of Can Open Source Keep AI Power From Concentrating? — original source. Check the original for full context.

Source published: 2026-09-07

Key ideas

Lukasz Kaiser argues that current AI power concentration is an artifact of the transformer architecture's appetite for massive data and compute, rather than an inherent property of intelligence. (0:00)
Big tech relies on brute-force engineering and multi-billion-dollar clusters because unpredictable fundamental research discoveries cannot be scheduled onto commercial product roadmaps. (0:00)
A single consumer-grade RTX 5090 GPU now packs more compute power than the entire 8-GPU cluster used at Google Brain to invent the original transformer in 2017. (2:37)
As commercial labs like OpenAI pivot toward productization, independent and open-source researchers have a clear opening to discover smarter, data-efficient architectures beyond monolithic models. (2:37)

Detailed notes

## Background Published by Sophia Dew on September 7, 2026, this audio podcast episode reports from the Open Source AI Summit in San Francisco. The program examines whether open-source AI can prevent power from consolidating among a handful of tech giants, featuring an in-depth conversation with 'Attention Is All You Need' co-author Lukasz Kaiser on model architecture, compute access, and specialized models.

## TL;DR Sophia Dew interviews researcher Lukasz Kaiser on whether modern AI's extreme centralization is an inevitable fate or merely an artifact of the current transformer architecture. Kaiser argues that big tech's reliance on billion-dollar clusters and indiscriminate web scraping is a brute-force engineering shortcut, whereas biological intelligence proves that deep expertise can be acquired from far smaller, targeted datasets. Because major commercial labs have shifted toward consumer product delivery, Kaiser asserts that independent and open-source researchers—now armed with consumer hardware more powerful than the cluster that created the transformer—have a vital opportunity to develop the algorithmic breakthroughs needed for a distributed, multi-model future.

## Atmosphere & style The discussion has a reflective, grounded, and encouraging atmosphere, cutting through the prevailing fatalism surrounding big tech's capital dominance with intellectual optimism. Kaiser speaks with the patient perspective of a founding transformer researcher, balancing realistic appraisals of current model limitations with genuine enthusiasm for the next wave of algorithmic discovery. It feels like an inspiring reality check for independent developers and academics, reframing today's monolithic corporate data centers as an early, transitional growing pain rather than the permanent ceiling of artificial intelligence.

## Bulletin - `00:00` Lukasz Kaiser argues that current AI power concentration is an artifact of the transformer architecture's appetite for massive data and compute, rather than an inherent property of intelligence. - `00:00` Big tech relies on brute-force engineering and multi-billion-dollar clusters because unpredictable fundamental research discoveries cannot be scheduled onto commercial product roadmaps. - `02:37` A single consumer-grade RTX 5090 GPU now packs more compute power than the entire 8-GPU cluster used at Google Brain to invent the original transformer in 2017. - `02:37` As commercial labs like OpenAI pivot toward productization, independent and open-source researchers have a clear opening to discover smarter, data-efficient architectures beyond monolithic models.

## Key takeaways - `00:00` The extreme concentration of AI resources into big tech firms stems from the specific demands of the transformer paradigm rather than a permanent rule of artificial intelligence. - `00:00` The successful rise of coding agents marked a turning point that incentivized giant corporations to build walled gardens around massive compute clusters. - `00:00` Corporate product timelines favor brute-force scaling and web scraping over high-risk, unpredictable basic research breakthroughs. - `00:00` Transformers cannot yet become specialized domain experts when trained on narrow datasets without suffering severe performance degradation. - `02:37` Biological intelligence proves that world-class capability does not require centralized monolithic models or consuming the entire internet. - `02:37` Open-source and academic researchers are uniquely positioned to tackle fundamental algorithmic research as leading labs shift their focus to product delivery. - `02:37` Hardware deflation has lowered the barrier to algorithmic research, allowing a single modern consumer GPU to outpower the cluster that produced the original transformer. - `04:47` The machine learning industry temporarily sidelined foundational architecture exploration after 2017 because scaling transformers offered quick commercial returns. - `04:47` Astronomical data center and scaling costs are driving the industry back toward basic research into smaller, specialized, and collaborative models. - `04:47` Current model uniformity and centralized API dependencies represent a transitional phase on the path toward customized, decentralized intelligence.

## Detailed walkthrough ### `00:00` The Concentration of AI and the Transformer Paradigm

Sophia Dew introduces the episode from the Open Source AI Summit in San Francisco, framing the defining question of modern artificial intelligence: as AI becomes more powerful, the resources required to build it are concentrating into fewer and fewer hands, raising the question of whether open source can create a more distributed future. She speaks with Lukasz Kaiser, one of the co-authors of the landmark 2017 paper "Attention Is All You Need," who suggests that the intense centralization we see today may simply be an artifact of the current technological architecture rather than an inherent, permanent rule of intelligence itself. Kaiser points out that a major turning point occurred late last year and early this year when coding agents genuinely started to work. However, because modern systems demand massive data centers and endless troves of data, giant corporations have built walled gardens around them and monetized them, naturally centralizing power.

Kaiser notes that despite their scale, current models are still nowhere near smart enough. While making systems fundamentally smarter with far less data requires basic research breakthroughs, corporate incentives run in the opposite direction. Research discoveries are inherently risky and cannot be scheduled like product roadmaps. Because businesses cannot rely on unpredictable research breakthroughs to meet targets, big tech firms choose the reliable brute-force engineering route: spending billions of dollars, building colossal clusters, and scraping every corner of the web. Kaiser cautions that transformers are not even ten years old, and their fundamental limitation is that while they are reasonably smart when trained on the entire internet, they cannot simply be given narrow, targeted data to become domain experts without becoming "stupid." As Lukasz Kaiser explains: “Now that is very concentrating. You go bigger, bigger, you need billions of dollars. You need to scrape data from every corner of the Internet.”

```mermaid flowchart TD accTitle: Big Tech Scaling vs Algorithmic Breakthrough accDescr: Flowchart illustrating the diverging paths between the current capital-intensive scaling model and the research-driven paradigm.

subgraph CurrentParadigm["Current Engineering Trap"] A["Transformers Need All Data"] --> B["Billion-Dollar Data Centers & Massive Scraping"] B --> C["Power Concentrates in Mega-Caps"] end

subgraph FutureParadigm["Alternative Research Path"] D["Fundamental Research Breakthroughs"] --> E["Smaller Data & Specialized Architecture"] E --> F["Decentralized, Distributed Intelligence"] end ```

`02:37` Biological Proof and the Opportunity for Open Source

Kaiser argues that an algorithmic alternative to brute-force scaling unquestionably exists because biological intelligence serves as an undeniable proof of concept. Human brains do not need to absorb the entire internet to achieve world-class capability, nor is humanity powered by a single centralized brain; instead, individual humans specialize deeply in specific fields and outperform massive models within those niches. He acknowledges that researchers have not yet figured out how to replicate this in machines—it may require modifying loss functions, data curation, or training mechanics rather than just attention layers—but the search space remains wide open.

With pioneering labs like OpenAI shifting their posture from pure exploratory research centers into massive commercial product companies, a vacuum has opened up that academia and open-source researchers are uniquely positioned to fill. Kaiser illustrates how accessible the entry point has become through his own personal hardware setup: a single consumer-grade desktop GPU now eclipses the entire compute infrastructure his team originally used at Google Brain to invent the transformer architecture. Kaiser highlights this shift in hardware capability: “So I bought a 5090 RTX GPU. It has more power than the eight GPU machines we used as a team to do the Transformers research. So you can buy and have one GPU that has all the power we used to design Transformer.” While an individual card will not train a frontier LLM from scratch, it provides more than enough horsepower to invent novel architectures and explore new machine learning paradigms.

- Humanity serves as existing biological proof: people learn complex tasks from limited data and achieve deep expertise without relying on a monolithic generalist brain. - OpenAI’s transition toward commercial product delivery leaves fundamental open-ended exploratory research open for independent developers, academic labs, and open-source contributors. - Hardware deflation allows a single modern consumer GPU, such as an RTX 5090, to exceed the combined compute power of the entire 8-GPU cluster used to develop the transformer in 2017.

`04:47` Looking Beyond the Monolithic Era

Addressing why he remains fundamentally optimistic, Kaiser reiterates that intelligence works best when distributed across varied, collaborating units rather than funneled through a single massive computer. Classical machine learning concepts, like model ensembles, already hint at how collections of specialized models can achieve superior results over a single giant system. Machine learning progress did not freeze in 2017 with the transformer; rather, because transformers produced immediate commercial dividends when fed internet-scale data, the industry temporarily abandoned foundational architecture research in favor of scaling up compute. Now that the costs of scaling have become unsustainably astronomical, researchers are being forced to turn back toward fundamental research.

Kaiser addresses the pessimism felt by builders who look at the current landscape and see only mega-corporations, multi-billion-dollar data centers, and paid API subscriptions that users cannot modify or control. He emphasizes that this setup is merely a transitional snapshot of the technology in its infancy, comparing today's blandly uniform AI outputs—where every model tells the exact same joke about atoms—to early-stage technical artifacts that will disappear once models can learn specialized domains and develop distinct personalities. Addressing those demoralized by big tech's current lead, Lukasz Kaiser offers an encouraging view: “It's a step in the technology, but it's a step towards something much more fun where everyone can have their own model maybe. We'll learn from much smaller data. The models will be experts in different domains like we are, and maybe one will be making jokes this way and another another way.” Concentration occurred because it was an easy engineering shortcut that demonstrated what was possible, but the next era belongs to researchers discovering smarter algorithms that democratize AI.

## Notable quotes > `02:37` You can buy and have one GPU that has all the power we used to design Transformer. — Lukasz Kaiser

## Mentioned Sophia Dew, Lukasz Kaiser, Open Source AI Summit, Attention Is All You Need, Google Brain, OpenAI, RTX 5090, Transformer

## Video description MTS host Sophia Dew visits the Open Source AI Summit in San Francisco to ask researchers and founders across the AI stack a central question: can open source prevent AI power from concentrating in the hands of a few companies? Lukasz Kaiser, co-author of Attention Is All You Need, argues that today’s concentration may be a feature of the current technological paradigm rather than a permanent feature of AI. Transformers reward enormous amounts of data and compute, but future breakthroughs could make smaller, more specialized models far more capable. Across conversations with researchers and builders working on open models, infrastructure, and applications, Sophia explores why China has taken the lead in open-weight models, whether the U.S. needs more open-model startups, what it means for companies to own their own intelligence, and where openness alone falls short, particularly when access to compute remains concentrated.

[Source](https://mgln.ai/e/1344/afp-848985-injected.calisto.simplecastaudio.com/3f86df7b-51c6-4101-88a2-550dba782de8/episodes/ad5068cd-5d10-4465-bc38-31449e172e26/audio/128/default.mp3?aid=rss_feed&awCollectionId=3f86df7b-51c6-4101-88a2-550dba782de8&awEpisodeId=ad5068cd-5d10-4465-bc38-31449e172e26&feed=JGE3yC0V)

中文简报

Transformer合著者Kaiser称新型AI模型有望打破科技巨头垄断 Lukasz Kaiser认为当前的AI中心化仅是一种暂时的工程捷径，并预测算法上的突破将使专业化模型能够依托少得多的数据比肩庞大系统。 Sophia Dew就现代AI的极度中心化究竟是不可避免的宿命，抑或仅仅是当前Transformer架构的产物，对研究员Lukasz Kaiser进行了专访。Kaiser认为，科技巨头对价值数十亿美元的计算集群以及无差别网络数据抓取的依赖，只是一种暴力破解式的工程捷径，而生物智能恰恰证明，仅凭规模小得多且针对性更强的数据集就能习得深厚的专业知识。由于各大商业实验室已将重心转向消费级产品的交付，Kaiser断言，如今手握性能超越昔日打造Transformer集群的消费级硬件的独立研究者与开源研究者，迎来了关键机遇去开创算法突破，推动走向分布式、多模型的未来。 Lukasz Kaiser认为，当前AI权力的集中是Transformer架构对庞大数据和算力渴求的产物，而非智能本身的固有属性。 (0:00) 科技巨头依赖暴力工程和数十亿美元的集群，是因为无法将不可预测的基础研究发现排入商业产品路线图。 (0:00) 如今一块消费级RTX 5090 GPU的算力，已超过2017年Google Brain用来发明最初Transformer的整个8卡GPU集群。 (2:37) 随着OpenAI等商业实验室转向产品化，独立研究者和开源研究者迎来了明确的契机，去探索超越单体模型的更智能、更高数据效率的架构。 (2:37)

Reading summary

Transformer co-author Lukasz Kaiser argues that the current concentration of AI power in mega-corporations is a temporary artifact of brute-force scaling rather than an inherent law of intelligence, creating an opening for open-source researchers to pioneer decentralized architectures. The Engineering Trap of Brute-Force Scaling (0:00) According to Kaiser, the current centralization of artificial intelligence stems directly from the limitations of the transformer architecture, which is not yet ten years old. While transformers perform well when trained across internet-scale data, narrowing their training data to domain-specific information causes them to fail. Consequently, making these systems smarter currently requires either basic research breakthroughs or brute-force engineering. Because fundamental discoveries are inherently unpredictable and cannot be scheduled on commercial product roadmaps, large technology companies have favored brute force. By deploying billions of dollars to construct colossal data centers and scrape web data, major firms have built centralized, monetized walled gardens. Kaiser emphasizes that this concentration reflects a corporate preference for reliable engineering shortcuts over risky foundational research. Biological Precedent and Hardware Accessibility (2:37) Kaiser points to human biology as empirical proof that intelligence does not require monolithic scale. Biological intelligence functions without absorbing the entire internet or relying on a single centralized system; individual human brains attain world-class expertise in specialized domains using limited data. While researchers have yet to replicate this in machines—which may require rethinking loss functions, data curation, or training mechanics—the design space remains open. As pioneering institutions like OpenAI pivot toward commercial product operations, an exploratory research gap has opened for academia and open-source contributors. Kaiser notes that hardware deflation makes novel architectural exploration accessible: a single consumer-grade GPU, such as an RTX 5090, provides more compute power than the entire eight-GPU machine his team used at Google Brain to invent the transformer in 2017. Although insufficient for training frontier models from scratch, such hardware is adequate for developing new architectures. The Shift Toward Distributed, Specialized Models (4:47) Kaiser maintains that intelligence operates most effectively when distributed across diverse, collaborating components rather than concentrated in a single massive system, pointing to classical machine learning ensembles as precedent. Although the artificial intelligence sector temporarily deprioritized architectural exploration to capitalize on the commercial returns of scaling transformers, unsustainably astronomical costs are now compelling researchers to revisit fundamental questions. Addressing developer demoralization surrounding big tech's current dominance, Kaiser characterizes uniform outputs and multi-billion-dollar clusters as early-stage technical artifacts. He anticipates a transition toward an ecosystem where individuals can train specialized, smaller-data models that feature distinct personalities and domain capabilities, arguing that algorithmic breakthroughs will ultimately dismantle centralized concentration. While Kaiser forecasts that algorithmic innovation will eventually democratize AI, researchers have not yet identified the specific algorithmic frameworks—whether through alternative loss functions, training mechanics, or architectures—required to enable models to master specialized domains from narrow datasets.

精炼阅读摘要

Transformer 架构合著者 Lukasz Kaiser 在与 Sophia Dew 对话时指出，当前人工智能向少数科技巨头集中的局面并非智能本质的必然规律，而是现有架构依赖海量数据暴力工程扩展的阶段性产物；随着算力扩展成本难以为继，基础算法研究与开源社区有望推动人工智能转向去中心化与专门化的未来。架构局限与工程捷径催生算力集中 (0:00) Kaiser 指出，尽管代码智能体在近期已展现出实用价值，但当前人工智能产业呈现出的极端中心化，本质上是企业商业激励机制与技术局限共同导致的结果。因为基础科学突破具有不可预测性、无法按商业路线图规划，科技巨头普遍放弃了高风险的基础研究，转向可预测的工程蛮力路径——耗资数十亿美元建设庞大集群并抓取全网数据以筑起竞争壁垒。然而，诞生不足十年的 Transformer 架构存在根本短板：尽管它在吞吐全网海量数据时表现出泛化能力，但如果仅依靠特定狭窄领域的精选数据进行训练，模型不仅无法成为垂直专家，反而会陷入退化。这种必须依赖全网数据才能生效的机制，直接将算力与控制权推向了能够承担天文数字资本开销的科技寡头。生物智能参照与开源硬件门槛下降 (2:37) Kaiser 强调，人类的生物智能证明了暴力扩展之外必然存在更优的算法路径。人类大脑无需吞下整个互联网的数据即可在复杂领域达到世界级水平，人类社会也依靠具备垂直专长的个体协作运转，而非依赖单一大一统的通用中央大脑。虽然让机器复现这种能力可能需要对损失函数、数据筛选机制或训练方式进行根本性改良，但该探索领域目前仍极为广阔。与此同时，随着 OpenAI 等开创性研究实验室向商业产品公司转型，基础理论探索领域出现了真空，为学术界和开源开发者留出了独特的进入空间。硬件层面的持续通缩进一步降低了门槛：Kaiser 以自身配置为例说明，如今单张消费级 RTX 5090 GPU 的计算性能，已经超过了其团队当年在 Google Brain 发明 Transformer 架构时所使用的整台 8 卡 GPU 机器。尽管单卡无法从零预训练前沿大模型，但已足以支撑研究人员探索颠覆性的新架构。超越单一巨型系统走向专门化协作 (4:47) 针对开发者对巨头垄断、昂贵算力中心以及封闭 API 的悲观情绪，Kaiser 认为当前的单体模型中心化形态仅仅是技术婴儿期的过渡产物。在 Transformer 展现出即时商业回报后，整个行业一度搁置了基础架构探索，全力投入算力扩展；但经典机器学习中的模型集成概念早就证实，多个专业化模型协同作战的效果往往优于单一庞大系统。目前算力堆叠的成本正逼近不可承受的极限，正倒逼研究人员重返底层算法。Kaiser 表示，当前所有大模型给出千篇一律的平庸回答（例如讲出完全相同的原子冷笑话）只是技术早期的残留特征；随着算法允许模型在较小规模的数据下高效吸收专业知识，未来的智能生态将转向每个人拥有专属模型、不同系统具备鲜明性格并各具专业深度的去中心化格局。 Kaiser 认为，当前算力向少数巨头集中只是行业在技术初期走捷径的工程结果，并不代表人工智能的终局；下一阶段的技术变革将取决于底层算法的突破，学术界与独立开源研究者有望在更分散、更高效的智能架构中重塑技术格局。

Detailed notes — zh

## Background 该内容由 Sophia Dew 与 Transformer 论文合著者 Lukasz Kaiser 于 2026 年 9 月 7 日发布在 RSS 音频播客平台上。节目的官方简介指出，Sophia Dew 探访了旧金山开源 AI 峰会，与全栈研究员及创始人探讨开源能否遏制 AI 权力向少数科技巨头集中的核心命题，并由 Lukasz Kaiser 深入剖析算力集中是否仅为当前技术范式的阶段性产物。 ## TL;DR 节目围绕当下前沿大模型因依赖海量算力与互联网全量数据而导致的科技垄断展开深度讨论。Lukasz Kaiser 指出，巨头堆叠规模只是规避科研风险的商业选择，Transformer 必须通吞全网数据的局限证明了当前架构并非智能终局。他以人类大脑低能耗、专精化的学习方式为对照，说明未来的正确路径将由高效率算法、小数据集专精模型以及分布式集成所引领。现代单张消费级 GPU 的算力已超过当年发明 Transformer 的整台机器，这为学术界与开源社区打破巨头黑盒垄断提供了重回底层算法创新的历史契机。 ## Atmosphere & style 对话洋溢着思辨性与技术乐观主义，既有对当下科技寡头垄断算力现实的冷静剖析，又充满对底层科学突破的由衷热忱。访谈风格深入浅出、推心置腹，Lukasz 凭借其作为 Transformer 奠基人的敏锐洞察，以购买单张 RTX 5090 显卡做研究等极具画面感的细节打破了技术虚无主义。整期播客在严谨的学术逻辑中流淌着人本主义温度，从对巨头流水线式规模扩展的疲惫反思，自然过渡到对去中心化、多元化个性智能形态的生动憧憬，带给听众一种穿越技术迷雾后的通透与振奋。 ## Bulletin - `00:00` Transformer 共同作者 Lukasz Kaiser 指出，当前的算力与模型垄断并非 AI 的固有宿命，而是科技巨头规避科研不确定性、依赖数十亿美元堆砌参数与数据的特定技术阶段产物。 - `02:30` 当前顶尖大模型在专精领域仍缺乏自适应效率，必须包揽全网数据才具备实用智能，而人类大脑的存在性证明了依靠极低能耗和极小数据量亦能形成专家级认知。 - `02:30` 硬件算力呈现断层式跃升，单张消费级显卡（如 RTX 5090）的计算能力已超越 2017 年最初研发 Transformer 所使用的 8 卡 GPU 集群总和，为独立研究者打破巨头垄断留出窗口。 - `05:00` 未来智能架构有望从集中式单体大模型转向分布式专精模型集成（ensembles），每个人都可能拥有在小数据集上训练且具备独特个性的专属专家模型。 ## Key takeaways - `00:00` 去年底至今年初代码智能体（coding agents）实用化，成为加速超大规模算力训练与商业变现的关键时间节点。 - `00:00` 科技巨头选择无限扩大规模（Scaling up）是因为基础科研突破具有高度不确定性，堆算力与全网数据是一条商业上最具确定性的路线。 - `00:00` 现有顶尖大语言模型在 Lukasz 看来仍然“远远不够聪明”，Transformer 架构诞生不足十年，在小数据集与狭窄领域上的学习能力依然极其笨拙。 - `02:30` 人类大脑是无需全知却能在各领域达到专精深度的最佳“存在性证明”，印证了低能耗、小样本高效学习在算法机制上的可行性。 - `02:30` 高效智能的实现不仅依赖模型架构本身，更取决于损失函数（loss）、训练策略与数据组织的底层综合创新。 - `02:30` 头部实验室逐步转变为被产品交付和商业化重压捆绑的大型企业，对纯粹底层研究的忽视反而为学术界与开源社区让渡了探索空间。 - `02:30` 一张现代消费级显卡（如 RTX 5090）的算力已超越最初设计 Transformer 论文时的 8 卡机器集群，足以支持独立研究者展开颠覆性的架构创新实验。 - `05:00` 人类社会由各具专长的大脑网络协同构成，这表明受限数据下的最优知识学习形态很可能是分布式模型集成（ensembles）而非全能单体大模型。 - `05:00` 当暴力扩展（Scaling）的成本壁垒难以为继时，算力的高昂现实将倒逼研究界重返机器学习的底层理论探索。 - `05:00` 未来的 AI 生态将更加多元化，个人专属小模型能在小数据上完成专精训练，打破当下通用黑盒模型输出同质化、机械化的局限。 - `05:00` 算力高度集中只是技术初期证明潜力的过渡性权宜方案，随着算法科学的演进，AI 权力的去中心化与普惠化是必然趋势。 ## Detailed walkthrough ### `00:00` 算力集中与 Transformer 架构的技术局限随着人工智能的能力日益强大，构建前沿模型所必需的数据与算力资源正在迅速向少数头部科技巨头集中。在旧金山举办的开源 AI 峰会（Open Source AI Summit）上，MTS 主持人 Sophia Dew 带着一个核心议题走访了 AI 产业链各环节的研究人员与创始人：开源能否真正打破科技寡头的垄断，推动 AI 走向更去中心化、权力分散的未来？作为划时代论文《Attention Is All You Need》的共同作者之一，Lukasz Kaiser 针对当下算力与模型权力高度集中的现状做出了深度剖析。他指出，在去年年底到今年年初之间出现了一个极其关键的时间节点——代码智能体（coding agents）开始真正发挥实用价值。然而，当代 AI 的底层逻辑是必须在造价极其昂贵的数据中心里、使用海量数据进行训练，这直接导致少数拥有雄厚资本的大企业掌握了绝对主导权并借此收费盈利。Lukasz 坦言，即使是现有的顶尖模型在他看来也“远远不够聪明”，行业本需要更多底层研究突破来让模型用更少的数据变得更智能。但在商业逻辑中，科学研究的突破具有极高的不确定性。大企业为了规避风险，选择了一条不需要等待科研突破的确定路线——不断把规模做大（scaling up）。这种无限扩充规模的做法需要数十亿美元的资金支持，以及扫荡、抓取互联网每一个角落的数据，从而不可避免地加剧了资源的集中化。然而 Lukasz 强调，这种集中只是当前技术演进特定阶段的产物，绝非 AI 的固有宿命。Transformer 架构从诞生到现在甚至不足十年，它唯有在吞下整个互联网的数据时才会展现出相当的智能水平；一旦试图让它仅在有限的狭窄领域内学习，它就会显得十分笨拙。Lukasz Kaiser 直言： > “Transformers are really good if you train them on the whole internet, and then they're reasonably smart. But if you try to say, you just learn about this... then they're just stupid, right?” - 关键拐点：去年底至今年初，代码智能体进入可用阶段，加速了对超大规模算力训练的依赖。 - 商业扩张 vs 科研探索：大企业选择依赖数十亿美元资本“堆算力、堆数据”进行 Scaling，而非押注不可预测的基础研究突破。 - 架构短板：Transformer 必须包揽全网数据才能表现良好，缺乏在极小数据集上高效学习专精知识的自适应能力。 ```mermaid flowchart TD accTitle: 当前 Transformer 扩展困境与开源科研机会 accDescr: 展示了大公司依靠资本推行高成本 Scaling 与开源社区依托基础科研寻找高效算法的路线差异。 subgraph BigTech ["科技巨头路线：资本驱动"] A["巨额资本与大规模数据中心"] --> B["抓取全网互联网数据"] B --> C["暴力扩大参数规模 (Scale Up)"] C --> D["高昂服务订阅费与权力集中"] end subgraph ResearchAlt ["科研探索路线：算法驱动"] E["基础研究与架构创新"] --> F["小数据集高效学习"] F --> G["分布式/领域专家模型"] G --> H["去中心化与开源赋能"] end ``` --- ### `02:30` 人类大脑作为参照：寻找下一代算法突破与开源机遇面对 Sophia 提出的关于是否存在某种算法或科研突破能让小型机构与巨头平起平坐的疑问，Lukasz 斩钉截铁地表示，这类算法不仅存在，而且人类自身就是最好的实证。人类并不是通晓一切的全才，但各个领域的专家在特定深度上往往能超越那些庞大无匹的通用大模型。他分析道，通往高效率智能的道路不仅仅关乎模型结构，还涉及损失函数（loss）、训练策略与数据组织的综合创新。 Lukasz 也观察到了前沿实验室重心的转移。他回忆自己加入 OpenAI 时，那里还是一个非常纯粹的研究型实验室；而现在的巨头实验室已经演化为背负沉重产品发布与商业化压力的庞大企业，对基础前沿研究的关注度相对分散。这种转向反而给学术界、大学以及开源运动让渡了宝贵的探索空间。为了说明算力民主化的可能性，Lukasz 分享了一个生动的个人细节：他最近个人单飞做研究时，购买了一张 RTX 5090 显卡。令人惊叹的是，这一张现代消费级 GPU 的计算能力，已经超越了当年他们整个团队研发 Transformer 论文时使用的 8 卡 GPU 机器集群的总和。Lukasz Kaiser 强调： > “So you can buy and have one GPU that has all the power we used to design Transformer. So you can do things with it.” 尽管单张显卡无法从零预训练超大规模语言模型（LLM），但它已经完全足够让独立研究人员进行颠覆性的架构创新和实验。他坚信机器学习研究仍有广阔未被发掘的疆域，呼吁更多研究者重新将目光投入到根本性的理论探索中。 - 存在性证明（Existence Proof）：人类大脑证明了存在能够依靠极低能耗、极小数据量在特定领域形成超高专业能力的学习机制。 - 算力代际飞跃：一张 RTX 5090 消费级显卡的性能已超过 2017 年打造 Transformer 时所使用的 8 卡计算节点。 - 创新空间转移：大实验室逐渐被产品与交付捆绑，使得学术界与开源社区迎来了接棒底层前沿研究的历史契机。 --- ### `05:00` 走出大模型垄断的悲观叙事：迈向分布式专家模型的新未来在谈及对未来为何仍保持乐观时，Lukasz 进一步阐述了他对智能形态的终极设想。他指出，人类社会并没有一个所谓的“全人类大统一大脑”，而是由无数各具专长的大脑协同构成的网络。在固定的数据约束下，学习知识最高效的路径或许根本不是集中式单体大模型，而是大量独立强悍、联合后更为强大的分布式模型组合（ensembles）。许多人看到当前算力中心被巨头垄断、普通用户只能被动购买无法干预的黑盒订阅服务，难免对 AI 的未来产生悲观情绪。但 Lukasz 认为这只是暂时的技术过渡阶段。机器学习的理论进展并未在 2017 年终结；过去几年行业被数据 Scaling 的红利所吸引，如今当这种暴力堆砌变得昂贵到难以为继时，算力成本的现实壁垒将倒逼学术界与工程师重新聚焦底层基础研究。他描绘了一个更加多元、有趣的 AI 未来：未来每个人都可能拥有专属于自己的小模型，这些模型在极小的数据集上完成学习，并且在特定领域具备专家级的造诣。与当下大语言模型千篇一律的机械反应不同（比如每次让它讲笑话，它总是在讲关于原子的老梗），未来的模型将具有各异的性格与幽默感。Lukasz Kaiser 在总结当下的垄断格局与未来的科研红利时精辟地说道： > “It's a step in the technology, but it's a step towards something much more fun where everyone can have their own model maybe. We'll learn from much smaller data. The models will be experts in different domains like we are, and maybe one will be making jokes this way and another another way.” 权力的暂时集中只是因为那是此前最容易走通的商业路径，它证明了 AI 的巨大潜能；但随着科研突破的跟进，这种集中形态必将被打破。 - 智能形态的本质：分布式专精模型集成（Ensemble）比单一大模型更有可能是实现高智力密度的正确路径。 - 同质化弊端：当前超大通用模型受制于训练范式，缺乏个性化特征与特定情境的深度表现（例如同质化的冷笑话）。 - 发展周期判断：算力垄断是技术初期的“权宜之计”（convenient thing），基础科学研究的追赶终将使算力与智能普惠化。 ## Notable quotes > `00:00` Transformers are really good if you train them on the whole internet, and then they're reasonably smart. But if you try to say, you just learn about this... then they're just stupid, right? — Lukasz Kaiser ## Mentioned Sophia Dew, Lukasz Kaiser, Open Source AI Summit, Attention Is All You Need, Transformer, OpenAI, RTX 5090 ## Video description MTS host Sophia Dew visits the Open Source AI Summit in San Francisco to ask researchers and founders across the AI stack a central question: can open source prevent AI power from concentrating in the hands of a few companies? Lukasz Kaiser, co-author of Attention Is All You Need, argues that today’s concentration may be a feature of the current technological paradigm rather than a permanent feature of AI. Transformers reward enormous amounts of data and compute, but future breakthroughs could make smaller, more specialized models far more capable. Across conversations with researchers and builders working on open models, infrastructure, and applications, Sophia explores why China has taken the lead in open-weight models, whether the U.S. needs more open-model startups, what it means for companies to own their own intelligence, and where openness alone falls short, particularly when access to compute remains concentrated. [Source](https://mgln.ai/e/1344/afp-848985-injected.calisto.simplecastaudio.com/3f86df7b-51c6-4101-88a2-550dba782de8/episodes/ad5068cd-5d10-4465-bc38-31449e172e26/audio/128/default.mp3?aid=rss_feed&awCollectionId=3f86df7b-51c6-4101-88a2-550dba782de8&awEpisodeId=ad5068cd-5d10-4465-bc38-31449e172e26&feed=JGE3yC0V)

Source quotes

[0:00] Lukasz Kaiser: Transformers are really good if you train them on the whole internet, and then they're reasonably smart. But if you try to say, you just learn about this... then they're just stupid, right?

Source quotes

[2:37] Lukasz Kaiser: You can buy and have one GPU that has all the power we used to design Transformer.

Original transcript

[0:00] AI is becoming more powerful, but the resources needed to build it are increasingly concentrated. Does it have to stay that way? MTS host Sophia Du heads to the Open Source AI Summit in San Francisco to ask researchers and founders across the AI stack whether open source can create a more distributed future for AI. Lucas Kaiser, co-author of the landmark Attention is All You Need paper, argues that today's concentration may be a property of our current technology, feature of AI. Transformers thrive on enormous amounts of data and compute, but they're less than a decade old, and the next breakthrough could change those economics. From open models and access to compute, to companies owning their own intelligence, Sophia explores where AI power is actually concentrating, what openness can change, and what it still leaves unsolved. the paper that introduced the transformer that completely changed modern day AI. Where is modern day AI and where is power currently concentrating? There was a very important moment sometime around end of last year, start of this year, where the coding agents started to work. The nature of this AI is that you need to train it on very expensive data centers and on a lot of data. So there's big companies that do this, and then they charge for it, right? That concentrates it in the sense to get the To me, even the current models are not smart enough. So there's still... could be much smarter. There may be some research breakthroughs that are needed to make them smarter with less data, but since research breakthroughs You know, sometimes they come, sometimes they don't. They're not like a business proposition. But the big companies are like, okay, but you can do things without research, which is to go bigger, bigger, bigger. Now that is very concentrating. You go bigger, bigger, you need billions of dollars. You need to scrape data from every corner of the Internet. So the current state of the technology is changing. a bit concentrating, but we should remember that it's just the current state. Sure, transformers are great, but they're not even 10 years old. I'm sure will be other things. maybe will make it possible to train with less data, more diverse models. Transformers are really good if you train them on the whole internet, and then they're reasonably smart. But if you try to say, you just learn about this... then they're just stupid, right? It doesn't work that well. But it doesn't work because maybe we need a research breakthrough that will make it work. But the transformer needs to be everything. And that's a problem, but it's a research problem that hopefully gets solved. With the transformer, architecturally, you need a lot of resources. And the more resources you have as a company, as a model, the more powerful you can be. But do you think there is a type of algorithmic breakthrough or some type of research breakthrough that will allow smaller players to really compete? I mean, we know it exists. We are the proof, right? Humans are not experts in everything. But in the field, there are experts. They're very good, sometimes better than our super huge models. So it exists. I don't think we have found it technically. It may even have attention layers somewhere, right? It may be because it's not just the model. It's also the loss and the data and how it's trained. There's a lot of factors. So we don't really know exactly where to look. That's the thing in research. But on the other hand, I think more and more people are looking. I hope. I think the labs have a little bit... OpenAI, when I joined, was a very research lab. Now it's a research lab a bit, but it's also a very big company that needs to do products and stuff. So it has less focus on researching. But this gives the chance to the open source movements. That's a chance for academia to universities. I recently started doing things on my own also. And it's funny. So I bought a 5090 RTX GPU. It has more power than the eight GPU machines we used as a team to do the Transformers research. So you can buy and have one GPU that has all the power we used to design Transformer. So you can do things with it. Thank you. big LLM, but you can... research and experiment on things. And I think many people should. I believe there is still much more in machine learning research than... My last question for you is what makes you feel optimistic that the future of AI is going in a direction that will be truly open and broadly empowering for everyone? If you look at the world, humans are the most amazing computers in the world, right? Our brains are still unmatched by the models. And we are not generalists. And there's not like one big brain of humanity. I think fundamentally, it might be that given a fixed amount of data, the best way to learn from it is to have a lot of distributed models that are strong on its own, but even stronger when they're. And if you look at some basic research, there are actually reasons to believe it. Like I was just saying, [5:00] talking about ensembles and things like that. So, yes, transformers are great when you feed them all of the Internet. But we know there is an algorithm that's even better and it can learn from much smaller data and be amazing at the things it's learning. This we are the proof. We have not found it yet like. how to do this in computers very well. But we will. I mean, progress in machine learning has not stopped in 2017. Of course, because it's so good, everyone has focused on the big data. But now that it's so expensive, maybe we will focus back more on more fundamental research. And yeah, I'm absolutely convinced it can work and it just needs research and effort. I think many people sometimes look at the current state of AI and get a little pessimistic, that it's like, you know, it's just... big companies, big data centers, everything's so big, and then you get a subscription to something you cannot do much about. And... To me, it's like, okay, but that's just the current state. It's just this year, next year. It's a step in the technology, but it's a step towards something much more fun where everyone can have their own model maybe. We'll learn from much smaller data. The models will be experts in different domains like we are, and maybe one will be making jokes this way and another another way. It's like currently when you ask a big language model about a joke, it's always the same, right? Something about atoms. Because it's an outcome of a technology and the technology will change when the research catches up. And also for researchers, there is like just a huge opportunity to catch up with this. So, you know, like just don't worry about, okay, it got concentrated because that was a convenient thing to do. And it showed that great things can be done. That's good. research breakthroughs. Thanks for listening to this episode of the A16Z podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our substack at a16z.substack.com. Thanks again for listening, purposes only and is not a recommendation to buy, hold, or sell any investment or financial product. This podcast has been produced by a third party and may include paid promotional advertisements, other company references, and individuals unaffiliated with A16Z. Such advertisements, companies, and individuals are not endorsed by AH Capital Management LLC, A16Z, or any of its affiliates. Information is from sources deemed reliable on the date of publication, but A16Z does not guarantee its accuracy. you you