贾扬清与硅谷独角兽高管：生成式AI因边际成本与不确定性解构巨头垄断

贾扬清与硅谷独角兽高管：生成式AI因边际成本与不确定性解构巨头垄断 大模型本质是具有随机性的统计学黑盒，传统软件的确定性API已被打破，应用层企业必须通过工程手段控制高昂的线性算力成本并重构QA标准才能生存。 本视频围绕生成式AI时代的产品开发展开了深度对谈。嘉宾们指出，GenAI产品正经历从传统软件的确定性API向“模糊输入与模糊输出”的范式转变，这使得质量控制（QA）和评估成为核心挑战。同时，讨论深入分析了AI软件高昂且呈线性的边际成本结构，分享了通过供应链思维、智能路由调度（如Martian）以及底层工程优化（如OpusClip）来降低算力开销的实用策略。此外，基于开源模型（如Llama 3）的崛起和边际收益递减规律，嘉宾们预测企业级AI市场将走向去中心化，而初创公司的长期护城河将依赖于深度的行业纵深数据与复合AI系统设计。 大语言模型的边际计算成本与调用量呈刚性线性增长，迫使开发者必须借鉴供应链优化思维；通过采用 OpenAI 预留吞吐量、云厂商竞价实例（Spot Instances）以及重构底层渲染流水线，可将算力成本降低 60% 至 80%。 (29:15) Martian 等路由工具运用供应链聚合与医疗分流逻辑，将散装零售 API 流量打包为高确定性的大宗采购，并在入口端将约 90% 的基础任务导向平价模型、仅留 10% 复杂任务给 GPT-4，实现高达 85% 的整体成本降幅。 (33:05) 计算架构正经历集中与分布式的“钟摆效应”：消费级市场（ToC）趋向集中于 ChatGPT 等单一入口，而企业级市场（ToB）因严格的数据隐私和定制需求，将演化为由开源模型与专业基础设施构成的去中心化生态。 (43:22) 在缺乏绝对标准答案的生成式应用中，模型评估是系统落地的核心；OpusClip 等团队采用类似“草莓农场”的工业化筛选思路，利用低价模型批量生成候选结果，再通过专门的评估 Prompt 多维度打分过滤出前 20% 的优质内容。 (65:09)

Building generative AI products requires abandoning traditional deterministic software assumptions in favor of statistical systems, managed unit economics, and compound architectures. Industry leaders across infrastructure, routing, and applications argue that sustainable enterprise advantages stem from domain context and cost-reduction pipelines rather than raw foundation models. The Shift from Deterministic Logic to Statistical Fuzziness (13:57) Traditional software engineering relies on translating human ambiguity into deterministic, reproducible logic governed by strict APIs and JSON formatting. Leap AI CEO Yangqing and veteran technologist Ho explain that large language models invert this paradigm by introducing "fuzzy input, fuzzy output" computation. Because outputs behave like non-deterministic black boxes, conventional quality assurance—relying on established happy paths and boundary test cases—fails; altering a single character in a prompt can disrupt results unexpectedly. This lack of reproducibility poses severe adoption barriers for enterprises, which require near-flawless reliability. Panelists observe that developers must treat AI as inherently statistical rather than deterministic. Instead of chasing pixel-level interpretability or absolute input-output guarantees, engineers must design statistical bounds, guardrails, and structural constraints—such as native function calling—to safely integrate non-deterministic outputs into existing business workflows. Inverting Software Economics and Infrastructure Optimization (27:15) Unlike standard cloud software where infrastructure scales sublinearly, LLM computation introduces direct linear marginal costs: every single API query incurs an unavoidable expense (such as roughly 10 cents per GPT-4 query). Without rigorous operational engineering, application developers risk operating at severe structural deficits and simply "working for OpenAI." Panelists draw on Tesla's scaling trajectory—validating an expensive premium vehicle before re-engineering the supply chain for mass-market affordability—to argue that AI startups must launch expecting to eliminate up to 80% of operational overhead. Achievable efficiencies include negotiating committed throughput discounts, shifting to spot instances, rewriting compute-heavy codebases (such as an audio-video FFmpeg rewrite saving 60% to 80% of compute), and implementing intelligent model routing. By functioning like a medical triage system, routers direct roughly 90% of routine queries to inexpensive models like GPT-3.5, reserving expensive models like GPT-4 for the remaining 10%, cutting overall API expenditures by up to 85%. Decentralization, Diminishing Returns, and Open Weights (43:22) While consumer AI exhibits centralization around single interfaces like ChatGPT due to user convenience and minimal price resistance ($20 per month), enterprise AI is unfolding as a decentralized ecosystem. Corporate security requirements and privacy restrictions prevent enterprises from sending proprietary data to external monolithic APIs, prompting the adoption of fragmented internal search tools and specialized models. Furthermore, the panel disputes early fears that a single provider like OpenAI could monopolize computing value. Training foundation models displays diminishing marginal returns: doubling compute investments from $1 billion to $2 billion fails to double model intelligence. Concurrently, highly optimized open-weights releases like Meta's Llama 3 8B model demonstrate that efficient, smaller architectures can rival older, massive models, disrupting hyper-monopolistic economies of scale. Compound Systems, Domain Moats, and AI-Driven Evaluation (49:46) Raw foundation models resemble an 18-year-old Einstein—brilliant at foundational tasks, but ineffective in a corporate enterprise without domain onboarding and context. Referencing the UC Berkeley and Databricks framework on "compound AI systems," speakers argue that economic moats are created not by base model weights, but by agents, external data retrieval, and proprietary task understanding that cannot be scraped from the open web. Because evaluating generative outputs is as nuanced as evaluating human intelligence, static benchmarks like MMLU and GSM8K are insufficient for complex conversational products. To bridge this, teams employ automated multi-tiered frameworks. Opus Clip CTO Jay illustrates this with a strawberry-harvest analogy: generating 100 candidate video clips with inexpensive GPT-3.5 prompts, then running specialized evaluation prompts across multiple quality dimensions to filter the top 20%. However, panelists caution that AI grading has blind spots, noting top vision models still fail to identify glaring visual artifacts. The panel emphasizes that long-term generative AI viability depends on accepting statistical uncertainty rather than forcing deterministic logic onto foundation models. Developers succeed not by relying on raw API prompts, but by building compound systems, aggressive routing architectures, and proprietary domain-evaluation loops.

## Background 本内容由 YouTube 频道“课代表立正”于 2026 年 4 月 25 日发布，为原会员专享转公开的线下技术圆桌录像。活动由 Statsig 赞助支持，汇聚了斯坦福客座讲师徐老师（Howie Xu）、Lepton AI 创始人贾扬清、OpusClip 联合创始人兼 CTO 吴中杰以及 Martian 创始工程师 Jason Hu 四位嘉宾，深入探讨生成式 AI 时代的产品研发、算力成本控制、市场集中度以及落地工程挑战。 ## TL;DR 圆桌全方位拆解了 GenAI 产品与传统软件在确定性、QA 测试逻辑及边际成本结构上的本质分歧。OpusClip 结合自身增长经历分享了如何通过长期协议、自研渲染管线以及“AI 评测 AI”的漏斗筛选机制实现高达 80% 的成本削减与高质量交付；Martian 则展示了利用智能模型路由在简单与高难任务间分流的省钱策略。贾扬清从计算机系统与供应链第一性原理出发，指出 AGI 落地必然依赖去中心化的复合 AI 系统而非单一模型垄断，并呼吁工程界坦然接受模型的概率统计本质，用统计学安全护栏与外层约束替代对绝对可解释性的执念。 ## Atmosphere & style 现场氛围既有硅谷一线技术极客的硬核交锋，又充满轻松幽默的创业者自嘲与调侃。四位嘉宾覆盖了底层 Infra、中间件路由与上层爆款应用的互补视角，彼此借星巴克买咖啡、草莓分选流水线或“18岁爱因斯坦入职”等鲜活隐喻拆解复杂系统概念，台下笑声与掌声不断。整场对话节奏紧凑扎实，既直面了初创公司在算力账单与巨头夹缝中的生存焦虑，又在尾声将工程实践升华为拥抱统计学概率世界的豁达哲学，给听众带来通透且极具启发性的临场体验。 ## Bulletin - [28:29](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1709s) GenAI 应用可实现 80% 以上的综合降本：通过锁定预置吞吐量、采用公有云竞价实例结合自研工程管线（如自研重构音视频渲染），并叠加 Martian 智能路由将约 90% 的简单任务分流至轻量模型，推理与算力成本能够被系统性压缩。 - [23:05](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1385s) 生成式 AI 正分化出两条演进路径：一是顺应“模糊输入到模糊输出”的原生 AIGC 与对话交互，二是通过 JSON 结构化输出与函数调用（Function Calling）给大模型“戴上手铐”，使其安全嵌入企业现有的确定性逻辑系统。 - [42:42](https://www.youtube.com/watch?v=Lt-lVe957hc&t=2562s) 算力架构与市场格局正处于钟摆轮回：ToC 领域流量高度向 ChatGPT 等单一入口收敛，而 ToB 领域受制于数据隐私与垂直业务上下文，注定走向去中心化多模型协同，打破单一基座模型的绝对垄断神话。 - [50:00](https://www.youtube.com/watch?v=Lt-lVe957hc&t=3000s) 基础大模型如同刚毕业的“年轻爱因斯坦”，无法独立创造商业价值：真正解决产业痛点需要依赖由 RAG、智能体（Agent）及私有领域知识构筑的“复合 AI 系统”（Compound AI Systems），而应用落地的头号瓶颈已转向动态模型评估（Evaluation）。 ## Key takeaways - [13:54](https://www.youtube.com/watch?v=Lt-lVe957hc&t=834s) 底层商业常识从未在 GenAI 时代失效，深刻理解用户需求（Jobs-to-be-Done）、精美直观的 UI/UX 以及冷启动渠道依旧是产品留存的核心护城河，仅提供对话框无法支撑完整业务。 - [18:02](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1082s) 企业级软件的核心支柱是精确性与可重现性，而大模型天生具备概率随机性；单字符变动即可导致输出突变，使传统 QA 测试体系面临颠覆，必须严格限制输入与输出空间。 - [21:20](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1280s) 消费级（ToC）产品虽对输出瑕疵具备更高的主观容错空间，但缺乏客观衡量标准导致反馈循环模糊，算法工程必须持续跨越用户主观满意的心理及格线。 - [23:05](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1385s) 传统计算机科学致力于将模糊输入转化为确定逻辑，而大模型打破了这一定律；当前技术界正分化为“拥抱模糊输出的原生应用”与“利用结构化约束强行固化逻辑”的双轨路径。 - [26:30](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1590s) 大模型调用的边际成本呈现残酷的线性递增关系，打破了传统云架构的亚线性成本优势，单纯封装公有 API 也面临传统网络效应与数据飞轮弱化的困境。 - [28:29](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1709s) GenAI 产品定价策略需类比特斯拉的降本曲线倒推设计：早期单位推理成本极其昂贵，但团队应预判工程重构与采购议价后约 80% 的成本下降空间以设定商业模型。 - [33:05](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1985s) 算力供应链遵循“确定性换取更低价格”的铁律：零散突发的 API 请求成本最高，而通过聚合零散需求进行大宗批发采购、结合智能路由难易分流，可直接削减高达 85% 的推理开销。 - [38:23](https://www.youtube.com/watch?v=Lt-lVe957hc&t=2303s) 计算机架构在大型机（集中）、PC（分散）与公有云（集中）之间反复摇摆；受数据隐私、长尾定制需求驱动，大模型生态正迎来向边缘端与私有模型分散的钟摆阶段。 - [42:42](https://www.youtube.com/watch?v=Lt-lVe957hc&t=2562s) AI 格局存在 ToC 与 ToB 的结构性分野：大众搜索与消费级聊天因用户惰性极度中心化收敛，而企业级服务因数据孤岛与专有知识壁垒必然长期保持去中心化。 - [45:52](https://www.youtube.com/watch?v=Lt-lVe957hc&t=2752s) 大模型算力堆叠面临边际收益递减规律，且缺乏类似搜索引擎“真实用户点击与反馈”的绝对垄断护城河，开源模型能力的快速追赶正在瓦解专有闭源模型的垄断壁垒。 - [50:00](https://www.youtube.com/watch?v=Lt-lVe957hc&t=3000s) 通用大模型必须嵌入结合了 RAG、Agent 与企业工作流的复合 AI 系统（Compound AI Systems），才能将通用智力转化为具有实际经济价值的专业生产力。 - [51:12](https://www.youtube.com/watch?v=Lt-lVe957hc&t=3072s) 初创企业抵御巨头侵蚀的防线在于持续积累深层领域知识与业务上下文，护城河不是融钱躺平的静态资产，而是依靠高速数据反馈与 A/B 测试拉开的时间差。 - [01:02:29](https://www.youtube.com/watch?v=Lt-lVe957hc&t=3749s) 模型评估（Evaluation）是生成式 AI 应用落地的首要难题，评测机制正从简单的静态多选题（如 MMLU）向复杂的推理链验证、沙箱代码执行与多维自动化框架演进。 - [01:05:09](https://www.youtube.com/watch?v=Lt-lVe957hc&t=3909s) OpusClip 验证了“AI 评测 AI”的草莓分选漏斗机制：利用廉价模型多路并发生成大量候选，再由评估 Prompt 多维打分筛选排名前 20% 的结果，但多模态模型对显性视觉缺陷仍存在盲区。 - [01:12:19](https://www.youtube.com/watch?v=Lt-lVe957hc&t=4339s) 工程界应接纳大模型的概率统计本质，而非执着于学术化的微观可解释性；正确的系统设计是在不可解释的黑盒外部建立起严密的统计学安全护栏与边界控制。 ## Detailed walkthrough ### [00:18](https://www.youtube.com/watch?v=Lt-lVe957hc&t=18s) 开场暖场与主办方介绍 活动在现场嘉宾与观众的就座与调试麦克风声中拉开序幕。主办方代表登台致辞，热烈欢迎大家参加由 Statsig 赞助举办的“AB Talk”暨社交酒会活动。 ```mermaid flowchart TD accTitle: GenAI 研讨会生态全景 accDescr: 展示从底层基础设施、中间件到上层应用及各方代表人物的生态全貌 subgraph Infra["底层基础设施 (Infra)"] Lepton["Lepton.ai<br/>贾扬清"] end subgraph Middleware["中间层路由 (Middleware)"] Martian["Martian<br/>Jason Hu"] end subgraph App["应用层产品 (Application)"] Opus["OpusClip<br/>吴中杰 (Jay)"] end subgraph Strategy["产业洞见与投资 (Strategy)"] Howie["斯坦福客座讲师 / 投资人<br/>徐老师 (Howie Xu)"] end Lepton -->|提供算力与推理| Middleware Middleware -->|模型调度优化| App Strategy -.->|行业趋势观察| Infra Strategy -.->|指导与孵化| App ``` 主办方解释了活动命名的渊源：Statsig 是一家面向产品研发的现代数据平台，核心提供功能开关（Feature Flags / Feature Gates）、A/B 测试实验平台以及深度数据分析等关键工具。Statsig 之所以发起这场社区技术活动，是因为他们不仅服务于传统科技企业，更深度支持了诸如 OpenAI、Anthropic 和 Character.ai 等头部生成式 AI 公司。伴随生成式 AI 浪潮的席卷，主办方希望搭建一个交流场域，汇聚开发者与行业专家，共同探讨在 GenAI 时代如何实践最优秀的产品研发方法论。随后，活动负责人还向大家介绍了负责统筹的活动经理 Morgan、招聘负责人 Jesse 以及负责开发者关系的 Brock，并提到公司在华盛顿州贝尔维尤（Bellevue）完全实行现场办公模式，开放了大量技术岗位。 ### [08:25](https://www.youtube.com/watch?v=Lt-lVe957hc&t=505s) 四位重磅全明星嘉宾介绍 主持人随后依次隆重介绍了今晚参与圆桌讨论的四位核心嘉宾： 1. **徐老师（Howie Xu）**：硅谷深谙技术与商业的资深老兵。早年在 VMware 亲手创办了网络虚拟化事业部，如今该部门已发展为数十亿美元营收的庞大支柱业务。此后，他经历过多次创业、在多家行业龙头担任高管，还在顶级风投机构 Greylock Partners 出任驻场企业家（EIR），同时在斯坦福大学客座授课并指导知名 B2B 孵化器 Alchemist Accelerator。Howie 近期创办了自己的 YouTube 频道《Byte into Future》，并在短短两周后即将前往纽约证券交易所（NYSE）现场进行面对面专访与直播。 2. **贾扬清（Yangqing Jia）**：Lepton AI 创始人兼 CEO。Lepton AI 是当前业界性能极顶尖的生成式 AI 基础设施服务平台之一。令人称道的是，基于 Lepton 自身的底层架构，贾扬清曾在不到一周的时间内用仅约 500 行代码搭建出一款 LLM 搜索引擎并全面开源，甚至在 Twitter 上与 Perplexity.ai 的 CEO 展开了一场轰动社区的切磋与讨论。在此之前，他曾任阿里巴巴副总裁及 Facebook（Meta）AI 基础设施总监，在加州大学伯克利分校读博期间主导开发了 Caffe，后续在业内主导/推动了 Caffe2、ONNX，并联合主导开发了如今全球最流行的机器学习框架 PyTorch。 3. **Jason Hu**：Martian 的创始工程师。Martian 是去年异军突起的 LLM 路由器（Router），其核心能力在于能够为用户输入的不同任务动态挑选性价比最高、表现最优的底层大模型，目前已被包括 OpenAI 和亚马逊在内的 300 多家公司的开发者采纳。此前他曾在芝加哥 Human+AI 实验室及字节跳动从事前沿研究，同时还是旧金山湾区最大的前沿 AI 社区之一 aligns.ai 的发起人。 4. **吴中杰（Jay Wu）**：OpusClip 的联合创始人兼 CTO。团队在 2023 年从原先的直播工具果断转型为当前的 AI 视频切片工具，并于当年 6 月正式发布。在不到一年的时间里，OpusClip 的年度经常性收入（ARR）已达到惊人的 1500 万美元，展现了现象级的飞速增长。在此之前，吴中杰曾任 Airbnb 工程经理以及 LinkedIn 核心工程师，拥有全面的大规模工程研发与管理实战经验。 ### [10:30](https://www.youtube.com/watch?v=Lt-lVe957hc&t=630s) 现场互动插曲：贾扬清安利“Elmo Chat”与各方打趣 在主持人介绍完贾扬清的履历并提到与 Perplexity 的公开讨论时，贾扬清幽默地笑着澄清那场互动“其实是一场非常友好的对话，大家可以去推特爬楼看原文”。由于主持人顺带推荐了 Howie 的 YouTube 频道，贾扬清打趣表示“既然大家都在插播广告”，他也要借机安利一下 Lepton 的落地成果。 贾扬清提到，很多人常常好奇：“你们这些做 AI Infra（基础设施）的公司，平时到底能做出些什么？”为此，团队推出了一款非常精巧实用的 Chrome 浏览器扩展——**Elmo Chat**（访问域名为 `elmo.chat`，名称灵感来源于芝麻街的 Elmo）。安装后，它可以直接帮用户总结当前网页上的各类内容，包括长篇新闻、推特讨论帖、Reddit 社区帖子、YouTube 视频甚至 PDF 文档。贾扬清强调，该插件完全免费、体验极佳，并且后端完全稳定运行在 Lepton AI 自身的推理平台上，以此直观展现其基础设施在真实高并发场景下的承载能力。 随后，Jason Hu 也不甘示弱地呼吁现场观众关注 aligns.ai 的官方 Twitter（@aligns_ai），并邀请大家常去旧金山参加他们的线下技术聚会。 轮到介绍 OpusClip 时，现场还爆发了一阵欢乐的互动：主持人提到自己是 OpusClip 的重度忠实付费用户，而 Howie 也在使用。主持人打趣问 Howie 到底是不是付费会员，Howie 笑着确认：“我当然付了费！因为我最近刚开了 YouTube 频道，全靠它剪视频，所以大家快去关注我的频道，关注我就等于是在支持 OpusClip！” ### [13:54](https://www.youtube.com/watch?v=Lt-lVe957hc&t=834s) 核心议题开启：打造 GenAI 产品有本质不同吗？ 在介绍完圆桌嘉宾覆盖“应用层、中间件层与底层 Infra”的互补背景后，主持人正式抛出第一个宏观且核心的问题：“**打造 GenAI 产品，和打造传统软件产品相比，是否存在本质上的不同？**” 吴中杰（Jay）首先从 OpusClip 真实且残酷的创业增长实践切入，深入拆解了“不变的商业与产品底色”与“剧烈变化的工程与协作形态”： #### 1. 那些未曾改变的底层逻辑 - **用户需求与产品定义（Jobs-to-be-Done）**：底层商业常识从未失效。开发者依旧必须深刻洞察用户到底需要完成什么任务。许多人误以为在 GenAI 时代，只要给用户一个类似 ChatGPT 的单行文本输入框就足够了、界面设计不再重要，吴中杰直言“这完全是大错特错的认知”。在 OpusClip 的实践中，精美、直观、低门槛的 UI/UX 依然是留存与口碑的护城河。 - **冷启动与营销渠道（Distribution & Marketing）**：产品的爆发并非纯粹靠天上掉馅饼。OpusClip 之所以在发布后能迎来火箭般的裂变，核心在于他们在上一款不怎么成功的直播工具创业期，就与大量内容创作者和网红（Influencers）建立了深厚互信。当 OpusClip 推出时，这些创作者自发且狂热地免费制作视频安利，因为向粉丝推荐这款强大的生产力工具“会显得他们很酷”。如果创业者在营销和渠道上一无所有，即便产品同等优秀，也极容易沦为跟风者并被先发者淹没。 #### 2. GenAI 产品面临的剧烈差异与全新挑战 - **用户对瑕疵的容忍度暂时更高**：相较于成熟稳定的传统软件领域，用户当前对 AI 产品的容忍阈值明显更宽容。因为大家深知这是前沿技术，即便模型偶有抽风或存在小 Bug，只要它能在某些关键环节成倍提升工作效率，用户依然乐意使用。 - **算力成本压力与规模效应的生死线**：在创立初期，OpusClip 每次给用户处理视频所消耗的单位算力成本高得惊人，是如今的 5 倍甚至 10 倍以上。团队必须拼尽全力通过架构优化去压缩单位推理成本以实现正向现金流，“否则我们本质上只是在给 OpenAI 打工”。吴中杰特别警示，许多跟进的竞品如果无法迅速获取海量用户以形成规模效应，就无法摊薄固定开销和获得更好的 API 议价/部署效率，最终会被可怕的推理算力账单拖垮。 - **团队协作模式与 Prompt Engineering 的崛起**：在传统 AI 时代，团队通常需要一群硬核算法研究员（Research Engineers）在实验室里苦熬训练私有模型；而在当下的提示工程（Prompt Engineering）时代，核心开发流程发生了质变。团队不再需要那么多纯粹训练模型的算法工程师，取而代之的是深度理解业务的“Prompt 专家”——他们通常由产品经理（PM）甚至产品运营人员转型而来。他们需要紧密配合工程团队，不断迭代、微调 Prompt，这一新型工种彻底改变了跨职能团队的协作路径。 随后，现场将话筒交由其他嘉宾，展开进一步视角的交锋探讨。 ### [18:02](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1082s) 企业级软件的精度困境与生成式 AI 带来的 QA 重构 在对比传统软件与生成式 AI 软件时，硅谷徐老师（Howie Xu）着重分析了他最为熟悉的企业级软件领域。对于企业软件而言，精确度（Preciseness）、正确性（Correctness）、可预测性（Predictability）以及可重现性（Reproducibility）是不可动摇的底线。然而，生成式 AI（GenAI）应用天生就不具备可重现性。 在 Text-to-Code、生成数据表格或计算结果等企业级业务场景中，哪怕百万分之一的错误率都是不可接受的，更不用说达到 99% 正确率（每一百次错一次）这种在一般标准下看似不错的水平。面对这种特质，工程团队不能怨天尤人，而必须彻底重新思考架构设计：通过严格限制输入空间（Input Space）和输出空间（Output Space），寻找能让生成结果远比单纯调用 OpenAI API 更加可控、可预测的工程路径。 这一范式转换对软件质量保障（QA）体系带来了颠覆性的冲击。传统软件的 QA 逻辑非常清晰：工程师先验证常规的“黄金路径”（Happy Path），然后系统化地测试边界条件（Corner Cases）——例如极小值、极大值、0、负数等。如果系统在输入为 5、6、8 时都能正确运行，开发者基本能推断输入为 10 和 11 时大概率也是正确的，只需防范 0 或 -1 等极端边界即可。但在生成式 AI 领域，文本的语义空间难以捉摸，仅仅多加一个字符，输出结果就可能因为底层机制的随机性而发生剧烈突变。究竟什么是“正确”，什么算“错误”，由谁来进行数据标注，目前行业内根本没有统一的标准化答案。整个行业必须从底层重新定义在企业级软件中究竟该如何做 QA。 ```mermaid flowchart LR accTitle: Traditional Software vs GenAI QA Approaches accDescr: Comparison of input predictability and testing between traditional software engineering and generative AI software. subgraph TRAD["传统软件测试 (确定性逻辑)"] direction TB T_IN["确定输入 (Happy Path & 边界)"] --> T_LOGIC["确定性逻辑与代码"] T_LOGIC --> T_OUT["严格可预测输出<br/>(验证 0 / 负数 / 极大值)"] end subgraph GENAI["生成式 AI 测试 (非确定性黑盒)"] direction TB G_IN["模糊输入 (变动单字符即可剧变)"] --> G_MODEL["概率黑盒模型"] G_MODEL --> G_OUT["非结构化/随机输出<br/>(缺乏绝对正确基准)"] end ``` 这种质量与稳定性的两难，直接导致企业级 GenAI 应用在落地生产环境时举步维艰。CNBC 在 2023 年 12 月曾发表过一篇题为《2023 年是英伟达赚取巨额利润、其余各方进行崇高实验的一年》（Hefty profits for NVIDIA and lofty experiments for the rest）的报道，精准揭示了当下的产业现状：英伟达赚得盆满钵满，而其余绝大多数公司都在进行各种宏大的“实验”，真正进入 Production 阶段的落地应用寥寥无几。核心根源就在于生成式 AI 软件的底层特性与传统企业软件的要求存在根本性的冲突。 - 企业级软件的核心支柱是精确性与可重现性，而大模型天生具有随机性和非确定性。 - 传统 QA 依靠常规路径加极值用例即可建立信心；LLM 哪怕微调一个字符都可能打破既有逻辑，令传统测试体系失效。 - 限制模型的输入输出空间，是让其达到企业级可用度的必经工程手段。 --- ### [21:20](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1280s) C端产品的容错空间与模糊的满意度门槛 转向消费级（C端）软件时，徐老师指出其面临的挑战虽然类似，但在容错度上拥有天然的缓冲优势。以吴中杰（Jay）创办的 OpusClip 为例，当用户上传一段长视频并要求其生成切片剪辑时，世界上并没有唯一的“标准答案”。系统剪出的视频片段是 1 分钟还是 1 分 30 秒，用户通常都能接受，拥有更大的自由度（Leeway）和容错空间。徐老师坦言自己本身就是 OpusClip 的付费用户，只要剪辑合理，视频时长稍有出入完全不会影响体验。 然而，这种容错度并不意味着消费级产品的问题更容易解决。时长无论长短，最终还是要跨过用户的“满意度阈值”（Satisfaction Threshold）。如果剪辑出来的 1 分半钟视频持续低于用户预期，用户依然会果断弃用；只有持续超出预期，产品才能形成黏性。由于没有客观对错的绝对衡量标准，开发团队的反馈循环（Feedback Loop）变得极其模糊，如何通过算法工程稳定、可预测地跨越用户的主观心理及格线，反而成了一道深不见底的技术与产品难题。 --- ### [23:05](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1385s) 贾扬清剖析：从“模糊输入到确定逻辑”到“双轨演进” 针对徐老师提出的范式转变，主持人总结道旧有的开发体系无法直接沿用。随后，贾扬清从数学与计算机科学本质的角度，对这一现象给出了高度精炼的抽象总结。 贾扬清指出，传统计算机科学的核心使命，始终是将“模糊粗糙的输入”（Sloppy Input，如抽象的人类需求、自然语言）极快地转换为“严密固定的逻辑世界”（Fixed Logical World，如可执行代码、严格定义的 JSON 格式或标准化 API）。然而 ChatGPT 的爆火彻底打破了这一铁律，它将 AI 变成了一个黑盒，输出结果也变成了“模糊随意的输出”（Sloppy Output）。这种松散性用来写写同人小说尚可，但若要直接将大模型的输出挂载到数据库查询等严肃工程系统中，就会引发巨大的断层。基于这种颠覆，当前技术界正在分化出两条平行的演进路径： 1. **全新范式：Sloppy Input 到 Sloppy Output** 直接在“模糊输入到模糊输出”的链条上构建原生体验。大量的 AIGC 应用、Chatbot 乃至于新一代 AI 搜索皆属于此类。传统的搜索引擎返回的是固定的网页链接和确定结果，而 AI 搜索给出的则是带有不确定性甚至可能偶尔出错的综合回答。由于人类日常与物理世界交互时本身就极其适应这种“模糊性”（Sloppiness），优秀的优秀产品经理正在抓住这一特性，创造出全新的交互维度和功能体验。 2. **重塑旧轨：用技术给大语言模型“戴上手铐”** 技术界仍在竭力捍卫传统逻辑世界的确定性，即努力将模糊输入重新规范为确定的逻辑输出。典型的方向包括代码生成（Code Generation）以及自然语言转 SQL（Text-to-SQL）。为了将这些新技术平滑嵌入现有的工业流水线，工程师们正在通过 JSON 结构化输出（JSON Output）、函数调用（Function Calling）等工程手段给大模型戴上“手铐”（Shackles/Handcuffs），建立严密的防御工事与周边外围工具，逼迫模型吐出确定性的系统结果。 在讨论中，主持人笑着提到自己此前与 Jason 录播客时曾用过“Fuzzy Input / Fuzzy Output”的表述，贾扬清风趣地回应说，自己用的“Sloppy”纯属现场随口杜撰，并非行业标准名词，引来现场一阵笑声。 ```mermaid flowchart TD accTitle: The Two Evolution Paths of GenAI Software accDescr: Two pathways branching from LLM technology: native fuzzy applications and constrained deterministic logic. INPUT["模糊粗糙的输入<br/>(Sloppy / Fuzzy Input)"] subgraph PATH1["路径一：原生 AIGC 交互"] direction TB P1_ENGINE["生成式大语言模型"] --> P1_OUT["模糊随意的输出<br/>(Sloppy Output)<br/>• 对话机器人<br/>• AI 搜索<br/>• 创意生成"] end subgraph PATH2["路径二：重塑确定性逻辑 (戴上手铐)"] direction TB P2_CTRL["约束层<br/>• Function Calling<br/>• JSON Mode<br/>• DSPy 框架"] --> P2_OUT["严格逻辑输出<br/>(Fixed Logical World)<br/>• Text-to-Code<br/>• Text-to-SQL<br/>• 数据库查询"] end INPUT --> PATH1 INPUT --> PATH2 ``` --- ### [26:30](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1590s) 计算模式的多样化、边际成本剧变与网络效应重构 接续贾扬清的分析，Martian 创始工程师 Jason Hu 指出，计算体系正在呈现多样化的模态融合： - 传统软件：纯粹确定性的计算系统； - 结构化大模型：利用 JSON Formatted Output 强行提升 LLM 确定性的模式； - 新兴架构：如斯坦福大学 NLP 团队主导的 DSPy 项目，试图用类似 PyTorch 的架构来组织计算流，但底层全部交由大模型文本生成驱动； - 极致模糊形态：如 ChatGPT 纯粹基于概率与创造力的对话计算模式。这些不同的计算形态正在各自最契合的场景中寻找生存空间。 随后，Jason 提出了区分传统软件与 AI 软件的另外两大底层维度： **一、 边际成本（Marginal Cost）的本质差异** 传统软件的算力成本是使用量的“亚线性函数”（Sublinear Function）。开发者花 100 美元买下一台服务器，无论是服务 100 个人、1,000 个人还是 10,000 个人，硬件成本几乎恒定，边际成本接近于零。但在大模型时代，调用 GPT-4 时，调用一次可能需要 10 美分，调用两次就是 20 美分，呈现极其残酷的“线性递增函数”（Linear Function）。Martian 在对接众多应用层客户时发现，现在的开发者对成本极其敏感，其敏感程度远远超过了过去对 AWS 等传统云服务基础设施的关注，因为 AI 的调用成本居高不下，规模效应的成本红利被打破。 **二、 传统网络效应（Network Effects）的弱化与断层** 在传统的 Google 或 Facebook 体系中，推荐引擎会随着使用人数的增加沉淀更多数据，从而让推荐变得越来越精准，产生强大的双向正向反馈飞轮。然而，当大量新型应用仅仅是简单调用 OpenAI API 时，这种通过自身业务数据积累反哺核心模型竞争力的传统网络效应正在遭受严峻挑战。 ### [28:29](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1709s) AI 产品的成本曲线弯曲：从特斯拉模式到 80% 的成本削减 主持人针对 GenAI 应用中的成本痛点进一步追问。如果企业仅仅调用外部 AI API，使用自身专有数据来提升模型效果十分困难，而在商业模式上，API 成本随着用户量呈线性增长也是极其致命的挑战。主持人向 OpusClip 联合创始人兼 CTO 吴中杰（Jay Wu）提问：此前提到规模经济（Economy of Scale），OpusClip 是如何把原本线性的成本增长曲线“向下弯曲”成亚线性（Sub-linear）的？ 吴中杰将 GenAI 产品的成本控制与特斯拉（Tesla）的降本路径进行了类比。特斯拉最初打造的是售价高达 10 万美元的高端车型 Model S，随后推出相对亲民但配置较高的双电机版 Model 3，最后再推基础版车型以彻底压低价格。这一过程绝非一味交付质量更差或偷工减料的车，而是在生产制造过程中不断优化工程技术、革新供应链，从而把制造成本逐步打下来。 打造 AIGC 产品也是同样的底层逻辑。在起步阶段，产品的单位成本结构往往极其昂贵：用量低、OpenAI 等大模型厂商不会给予任何折扣。然而，随着调用规模扩大，团队能从供应商处拿到极具竞争力的商务条件，同时在内部针对 GPU 算力集群和底层算法展开工程优化。吴中杰指出，综合这一系列举措，企业整体通常能够预期实现大约 80% 的成本节约。正因如此，创业团队在一开始制定产品定价时，必须提前将这 80% 的降本空间折算进去；早期即使在账面上处于亏损运营的状态，只要后续持续投入工程研发进行系统优化，最终就能获得健康的利润空间。 - **特斯拉类比**：高端切入（Model S，10 万美元级）建立品牌与技术，通过供应链与工艺改进过渡到大众走量款（Model 3），实现规模降本。 - **GenAI 成本预判**：初始阶段因用量少缺乏议价权，单位成本极高；综合谈判与工程优化后整体可实现约 80% 的成本压缩。 - **产品定价策略**：定价必须按“降本 80% 之后的稳态成本”倒推设计，允许早期阶段承受战略性亏损。 --- ### [31:08](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1868s) 成本优化的三大拆解：商务谈判、云资源与工程重构 在被问及这 80% 的成本降幅具体来自于大模型厂商谈判、传统公有云议价，还是内部工程优化时，吴中杰对成本拆解做了非常详尽的剖析。首先在大模型供应商层面，OpenAI 提供了名为“预置吞吐量”（Provisioned Throughput）的产品形态，企业只需承诺在一个月或三个月内维持特定的每分钟 Token 调用量（Tokens per Minute, TPM），就能获得 5% 到 50% 不等的折扣优惠。 在传统云厂商（如 AWS、Google Cloud 等）的算力开销上，降本空间同样巨大。若从纯粹的按需计费（On-Demand）实例切换到长期承诺的预留实例，通常能获得高达 70% 的降价；如果团队进一步采用极度廉价的抢占式/竞价实例（Spot Instances），再结合一定的长期保底协议，还能在此基础上再榨出大约 20% 的成本降幅。 更关键的杠杆在于深度的自研算法与工程重构。吴中杰分享了一个具体的内部实战案例：OpusClip 聘请了一位音视频工程领域的顶尖专家，从底层自研了一套高度优化的代码，彻底替换掉了系统中原本极其庞大臃肿的开源 FFmpeg 处理流程。单单是这一项架构改造，就直接将 OpusClip 核心渲染管线（Rendering Pipeline）的算力成本削减了 60% 到 80%。此时主持人插话打趣，提到 OpusClip 近期也签约成为了 Statsig 的企业级客户，好奇大规模 A/B 测试是否也有助于成本优化；吴中杰则笑着回应称已在问答环节准备了更详细的案例，稍后展开解答。 | 优化维度 | 具体举措 | 预期降本幅度 | |:--- |:--- |:--- | | **大模型 API 采购** | 采购 OpenAI 预置吞吐量（Provisioned Throughput），锁定 1~3 个月 TPM | **5% ~ 50%** | | **公有云算力实例** | 从按需计费切换至长期预留，或在 Spot 竞价实例基础上叠加承诺协议 | **70% 基础降幅**，Spot 额外再挤出 **20%** | | **自研工程管线** | 顶尖音视频专家重写核心管线，彻底替代庞大的 FFmpeg 渲染链路 | 渲染管线成本立降 **60% ~ 80%** | --- ### [33:05](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1985s) 贾扬清的星巴克隐喻：供应链的铁律与确定性溢价 Lepton AI 创始人贾扬清则从宏观基础设施的视角指出，自己其实“完全不担心 AI 的成本问题”。他认为，供应链管理在传统工业界早已是被研究透彻的成熟学科，有着完备的知识体系，而供应链中最核心的一条基本铁律在 AI 领域完全适用：“**确定性会换来更低的价格**”（Predictability is rewarded with a lower price）。 贾扬清抛开复杂的英伟达 GPU 内核和底层优化，用买星巴克咖啡做了一个生动形象的比喻： 1. **单点零售**：一个人随机走进星巴克点一杯拿铁，需要支付全额零售价（如 5 美元）； 2. **突发零售巨额需求**：一个人突然冲进星巴克大喊“我现在就要 100 杯拿铁”，不仅单杯价格依然是 5 美元毫无折扣，顾客需要痛苦地漫长等待，吧台咖啡师也手忙脚乱、苦不堪言； 3. **高确定性供应**：如果一家 100 人的公司负责人提前告诉星巴克，“在未来一整年里，每天早晨 9:00 请准时把 100 杯咖啡送到公司前台”，星巴克不仅能给出极佳的折扣，履约与配送效率也会非常出色。 贾扬清解释道，在零售端零买永远最贵，而在 Costco 这样的批发平台按可预测的节奏批量采购，自然能拿到批量折扣。当前很多 GenAI 应用还处于起步探索期，调用模式完全属于第一种“即时零售”状态，甚至由于缺乏用量预测，经常会突然给基础设施供应商砸来“突发 100 杯咖啡”的峰值流量，导致上下游都异常痛苦。随着应用落地走向成熟、流量走向平稳规律，传统供应链的经济学规律必然全面生效，算力与模型价格必将大幅下挫。 贾扬清顺势为同台的 Martian 打了一个“免费广告”。Martian 所做的事情在逻辑上极其清晰：海量开发者在零散使用各种大模型 API，属于典型的零售流量；Martian 将这些分散的零售流量聚合成庞大的规模化流量，再向底层算力与模型供应商（例如 Lepton AI）进行大宗采购（Bulk Purchase）。这种模式极大提升了供应链的确定性，如同 Booking.com 聚合零散客流后能从万豪酒店拿到 10% 的佣金一样。贾扬清透露，Lepton AI 已经在与 Martian 洽谈此类批发采购合约。 --- ### [36:47](https://www.youtube.com/watch?v=Lt-lVe957hc&t=2207s) 智能路由与分级诊疗：Martian 如何通过难易分流再省 85% Martian 的创始工程师 Jason Hu 对贾扬清的观点表示高度赞同，并补充指出：除了在商务层面“聚合零售需求换取批发折扣”之外，Martian 还依靠一套**智能模型路由**（Smart Routing）算法在技术层面实现降本。 Jason 以开发一个“数学助考/解题应用”（Math GPT）为例：在该应用所接收的全部用户问题中，通常有 90% 属于非常简单的基础运算，GPT-3.5 Turbo 就能完美回答；只有 10% 属于真正高难度的复杂数学难题，必须动用 GPT-4 才能解决。在没有动态路由的情况下，开发者为了保证最底线的用户体验，往往只能将 100% 的流量全部兜底打到 GPT-4 上。然而，GPT-4 的调用价格比 GPT-3.5 Turbo 昂贵约 30 倍，这种“高射炮打蚊子”的做法导致成本居高不下。Martian 的做法是：对每个输入的 Prompt 进行自动化的语义难度判别，简单问题分流至 GPT-3.5 Turbo，难题才唤醒 GPT-4。凭借高达 30 倍的单价差距，系统综合下来能为开发者立省约 85% 的推理成本。 ```mermaid flowchart TD accTitle: Martian 智能模型路由工作流 accDescr: 展示用户输入如何通过难度分类被路由到轻量或大型模型，实现成本节约。 subgraph Ingest["用户请求接入"] A["用户 Prompt 输入"] --> B{"Martian 难度识别器"} end subgraph Routing["模型分流决策"] B -->|"简单问题 (约 90%)"| C["GPT-3.5 Turbo / 轻量模型"] B -->|"高难问题 (约 10%)"| D["GPT-4 / 旗舰大模型"] end subgraph Output["最终交付"] C --> E["合并输出 & 节约 85% 成本"] D --> E end ``` 贾扬清随即呼应了这一方案，指出这同样是现实供应链中“分级诊疗”机制的完美复刻。如果一个人平时有点头痛脑热的小病，绝不会立刻冲向斯坦福医院挂顶尖专家的门诊，因为那对医生和患者而言都极其昂贵且浪费资源；正确的做法是先看社区家庭医生（Family Doctor）。家庭医生负责初步分析与分诊：遇到普通问题，开一片阿司匹林就能解决；只有遇到疑难杂症，才会将病人精准转诊至专科专家手中。这种传统智慧在 AI 基础设施和应用分流中展现出了惊人的效率。 --- ### [38:23](https://www.youtube.com/watch?v=Lt-lVe957hc&t=2303s) 历史的钟摆：算力架构在集中与分布式之间的六十年轮回 随着对传统历史规律的讨论渐深，主持人将话题引向核心命题：在技术发展史上，历史总是惊人地相似，那么**生成式 AI 的发展演进究竟会走向高度集中化（Centralized），还是分布式/去中心化（Decentralized）？** 受邀做开场发言的斯坦福客座讲师徐老师（Howie Xu）回顾了过去 50 至 60 年间的整体计算机架构演进史。他指出，计算架构的历史本质上就是一个在“集中”与“分布”之间来回摆动的钟摆（Pendulum Swing）： - **大型机时代（集中式）**：早期以 IBM 大型机（Mainframe）为中心，所有计算集中在机房中，用户通过哑终端访问； - **PC 与客户端/服务器时代（分布式）**：上世纪 70 年代，比尔·盖茨提出“让每个家庭、每张办公桌上都有一台电脑”的宏愿。在那个大型机独霸天下的年代，IBM 的高管们对此嗤之以鼻，甚至嘲笑盖茨的想法；然而 PC 和 Client/Server 架构迅速崛起，算力被彻底分发至终端； - **云计算时代（回归集中式）**：几十年过去，世界仿佛兜了一个大圈，重新回到了“集中式大型机”的怀抱——AWS、Microsoft Azure 和 Google Cloud 这三大公有云巨头，实质上就是当代的“三大超级大型机”。 徐老师感慨道，站在 GenAI 爆发的当口，计算架构的钟摆再次面临方向选择。面对全新的大模型范式，究竟是会继续巩固云端巨头的绝对集中，还是会再次分散到边缘端与本地开源生态，历史究竟会嘲笑哪一方，目前仍充满了悬念。 ### [39:51](https://www.youtube.com/watch?v=Lt-lVe957hc&t=2391s) 算力钟摆假说：集中与去中心化的历史循环 探讨大语言模型（LLM）的未来格局时，主持人与嘉宾们引用了计算架构史上反复出现的“钟摆效应”。如果去问 OpenAI 首席执行官 Sam Altman，他的论调非常干脆：大家没必要费心折腾自己的专属模型，只要调用 OpenAI API，做点微调（fine-tuning）和提示词工程（prompt engineering）就完事了；因为无论你怎么练，OpenAI 迭代出的新模型能力注定更强、表现更好。但在很多实际业务中，这个假设并不成立——大量客户出于数据合规与商业机密考量，绝对不可能把核心数据拱手相让，因此拥有私有模型是硬性刚需。 从技术史的视角看，OpenAI 现在的地位很像当年的大型机（Mainframe）。技术架构的演进总是在集中与分散之间反复摇摆，其核心驱动力永远是“创新的瓶颈在哪里”： - **IBM 大型机转向 PC 架构**：当年 IBM 大型机生态陷入沉闷乏味，可用性极差，难以激发新的创造力；用户迫切希望用自己的方式做事，从而催生了个人电脑与分布式创新的大爆发。 - **PC 混乱催生云端回流**：当去中心化创新泛滥成灾，导致管理与运维陷入地狱（Management hell），整个行业又重新收束回集中化的云端架构。 当前整个行业正由中心化巨头垄断（OpenAI 占据绝对优势），但这违背了技术创新的本性。OpenAI 单一厂商无法覆盖成百上千种细分业务场景与定制功能需求。因此，未来几年将见证大量模型跑在边缘侧（Edge）、移动终端或 PC 端，走向高度去中心化。然而，当大家都意识到维护成千上万个私有模型成本过高、难以管理时，行业大概率又会在几年后重回类似“AI 云时代”的中心化架构。 ```mermaid flowchart LR accTitle: 计算与AI架构的钟摆循环 accDescr: 展示从集中到分散再回归集中的钟摆演进过程 subgraph Phase1["集中式阶段"] A["IBM 大型机 / OpenAI 集中垄断"] end subgraph Phase2["去中心化阶段"] B["PC 架构 / 边缘端与本地私有模型"] end subgraph Phase3["新集中式阶段"] C["云计算 / 未来统一的 AI 云服务"] end A -->|"可用性差 / 垄断压制创新 / 数据隐私"| B B -->|"模型泛滥 / 管理地狱 / 运维成本高"| C ``` --- ### [42:42](https://www.youtube.com/watch?v=Lt-lVe957hc&t=2562s) 贾扬清谈 ToC 与 ToB 的分野：从搜索范式看 AI 格局 贾扬清提出了另一个分析视角：中心化还是去中心化，不能脱离具体商业属性（ToC 与 ToB）一概而论，这一规律在搜索领域早已被验证： - **ToC 领域高度中心化**：日常面向大众的公开网络搜索，Google 牢牢统治着 95% 以上的市场份额，“Google”甚至直接演变成了搜索的动词代名词。消费者本性是惰性的，不愿货比三家；面对 AI 聊天机器人亦是如此，大众普遍觉得 ChatGPT 体验足够好，便欣然每月支付 20 美元（相当于 5 杯星巴克咖啡的钱），导致 C 端流量极度向 ChatGPT 收敛。 - **ToB 领域高度去中心化**：在企业内部工作环境中，没有任何一款所谓“企业级 Google”能够一统天下。员工查找内部文档时，用 Notion 的就在 Notion 内部搜，用 Quip 的就在 Quip 里面搜，抑或依赖自研内部检索系统以及 Elastic、MongoDB 等基础设施供应商提供的垂直方案。 在企业级 AI 市场，同样的去中心化趋势正在加速上演。当下 GPT-4、Claude 与 Google 旗下的模型已经在企业端展开白热化争夺。未来不仅基础模型供应商百花齐放，更会有层出不穷的垂类应用和企业内部自研模型协同运转。面对这一必然走向碎片化与去中心化的 ToB 生态，企业对底层基础设施工具的需求将迎来井喷——正如当年互联网建站浪潮催生了 Databricks 与 Snowflake 这批数据基础设施巨头一样，Lepton AI 的核心目标就是打造服务于企业级去中心化 AI 市场的底座。 --- ### [45:52](https://www.youtube.com/watch?v=Lt-lVe957hc&t=2752s) 规模经济、搜索信号与 OpenAI 垄断神话的破灭 针对中心化问题，现场嘉宾从第一性原理（First Principles）切入，指出决定一个行业最终格局的根本要素是该业务的**规模经济效应（Economies of Scale）**与网络效应。 - **搜索与社交的极致垄断**：社交平台（Facebook、微信）与搜索引擎之所以极度垄断，是因为用户规模直接强化产品壁垒。当年微软 Bing 团队为了追赶 Google，全员 7×24 小时高强度投入巨量资源，却依然在绝望中落败。 - **Google 排序的绝密第一信号**：贾扬清在此补充了一个内幕技术细节。大约五年前 Google Brain 团队宣传深度学习赋能搜索时，对外宣称深度学习（AI）已经成为搜索排序的“第三大核心信号”，却对最重要的第一大信号讳莫如深。这个第一大信号正是**真实的人类反馈与点击数据（Human Feedback）**。当某个用户搜索后点击了特定结果，就构成了最强正反馈；Google 垄断了全网 90% 以上的搜索流量，这种由庞大用户点击形成的数据护城河，让任何对手都无法通过纯算法完成超越。 然而，大语言模型并不具备这种无限滚雪球式的规模经济： - **边际收益递减（Declining Marginal Return）**：OpenAI 无法凭借无限制堆叠算力永久通吃全球 GDP。在模型训练中，投入 20 亿美元训出的模型，绝对无法达到 10 亿美元模型两倍的智商，算力与数据带来的性能提升正在遭遇严重的边际效应递减。 - **开源高效技术的灵感扩散**：先进算法理念无法被任何单一巨头垄断。例如 Meta 开源的 Llama 3 8B 模型，在团队内部 Benchmark 评测中表现极为优异，能以极高推理效率击败许多此前参数量大得多的旧模型。 - **AGI 落地对场景上下文的刚性依赖**：真正的通用人工智能落地，高度依赖于垂直行业的私有上下文和专业知识（Context & Specific Knowledge），仅凭一个通用黑盒基座模型，根本无法直接通盘解决现实世界千差万别的具体难题。 ### [050:00] 复合 AI 系统与去中心化 AGI 的实现路径 Jason 提出了一个生动的类比：基础大模型就像一个 18 岁的爱因斯坦刚从大学毕业，聪明绝顶、领悟极快，但你直接让他去 Google 或 Martian 上班，他根本干不了实际的工作。企业内部存在着海量的背景信息、业务流程规范、微妙的人际交往与团队协作管理，新员工都需要经历入职培训（Onboarding），而 OpenAI 等模型厂商不可能掌握企业内部专有的细分知识。因此，市场必须依赖第三方 SaaS（不论是 ToB 还是 ToC），将通用的通用智能转化为垂直领域的专业智能，以此完成具备真正经济价值的工作。 这一观点直接契合了加州大学伯克利分校学者与 Databricks 联合创始人发表的关于复合 AI 系统（Compound AI Systems）的核心论文。该论断指出：最优秀的业务表现与落地成果，从来不是单纯依赖最新、最聪明的单体基础模型来实现的，而是依赖于将智能体（Agent）、检索增强生成（RAG）、思维链（Chain of Thought）以及外围工具链编织在一起的复合 AI 架构。真正的通用人工智能（AGI）必将通过这种去中心化的应用构建生态落地，而非由单一巨头垄断一切。 ```mermaid flowchart TD accTitle: 复合 AI 系统的落地架构 accDescr: 展示从通用智能通过企业上下文与复合架构转化为实际经济价值的过程 subgraph Base ["底层基础模型"] LLM["通用大模型<br/>（如 18 岁的爱因斯坦）"] end subgraph CompoundSystem ["复合 AI 系统中间层"] RAG["RAG 检索增强"] Agent["Agent 智能体规划"] COT["Chain of Thought 提示工程"] end subgraph DomainContext ["企业领域知识"] Rules["内部业务规则与工作流"] Context["专有数据与上下文"] end subgraph Output ["商业价值闭环"] Value["具有实际经济效益的业务产出"] end LLM --> CompoundSystem Rules --> CompoundSystem Context --> CompoundSystem CompoundSystem --> Output ``` --- ### [051:12] 初创公司的护城河：领域知识与持续迭代 吴中杰接着探讨了初创企业在面对巨头时的生存空间与护城河问题。几乎所有风投在考核创业团队时都会质疑：“大模型厂商如果推出类似功能，你凭什么活下来？”在 ChatGPT 问世后 Jasper 增长断崖式下跌的案例，让行业对这个问题格外警惕。中杰同意 Jason 的观点，认为深入垂直场景的领域知识（Domain Knowledge）和业务上下文（Context）是初创团队最强大的壁垒。 以 OpusClip 为例，怎样才算剪辑出一段节奏优秀、吸引用户的短视频，这背后的审美标准和工程规则并不会显式地出现在互联网公开网页上。这套认知是在不断打磨产品、跟客户持续交互，并通过大量的 A/B 测试（团队重度依赖 Statsig 等测试平台）从实际反馈数据中沉淀出来的。中杰特别提醒，创业公司永远不要妄想有一劳永逸的“终极黄金护城河”；如果你融了钱就躺平混日子（rest and vest），很快就会被对手淘汰出局。护城河不是静态资产，而是一个靠快速迭代、持续积累深层领域知识动态拉开时间差的过程。 --- ### [053:14] 观众问答开始：在信息爆炸时代如何建立技术认知 在主持人提示时间超出、进入观众 Q&A 环节后，现场一位技术从业者提出了一个贴近普通开发者的问题：在各种新技术和喧嚣铺天盖地涌现的 AI 时代，普通从业人员应当借助哪些工具或信息渠道，才能穿透噪音、保持对前沿动态的敏锐感知？ 徐老师（Howie）直言，这也是他和主持人私下反复探讨的心结。互联网在头十年确实极大地帮助了人类获取知识，但过去几年里，全社会的“信噪比”（Signal-to-Noise Ratio）究竟是在提升还是下降，答案变得非常模糊。对于普通人来说，建立严苛的筛选机制至关重要： - **精读优于泛读**：很多人总在追问书单，认为读的书越多越好，但徐老师宁可把一本好书精读三遍，也不愿走马观花读三本平庸的书。这就像 Llama 3 等模型再三证明的技术事实——高质量数据对最终能力的贡献远超纯粹堆砌数量。 - **追求极高内容标准**：徐老师制作视频与技术播客的目标，是产出听众即便反复听上十遍依然觉得有所收获的硬核内容，学习者也应当用同样的高标准来反向挑选自己的信息源。 --- ### [057:43] 社交媒体生态的“带毒”特性与平台分流策略 贾扬清接过了信息渠道的具体落地话题，分享了自己日常获取资讯的工作流，并调侃了各个平台的生态偏好： - **Twitter / X**：适合观察最前沿的酷炫技术与开源发布，但同时充斥着大量抓马（Drama），氛围酷似《唐顿庄园》或《老爸老妈的浪漫史》这种肥皂剧。 - **Hacker News**：适合技术极客交流，能看到很多深度硬核的代码和底层架构争论。 - **LinkedIn**：职场人士往往打扮得光鲜亮丽，适合寻找严肃的商业合作和商业化动态。 - **优质播客与长文**：遇到长文本则借助 LLM（如 Lepton 搭建的 Chat 服务）进行提炼总结。 徐老师随即补充了 Andrej Karpathy（卡帕西）几个月前发过的一条推文：“刷推文就像从互联网上随便下载一个可执行文件（.exe），你根本不知道里面有没有恶意软件。”在以吸引眼球为核心机制的平台上，发言往往越极端、越有争议才能获得越多的曝光和流量。贾扬清也幽默附和，暗讽了某位在 X 上极具攻击性而广为人知的搜索公司 CEO（暗指 Perplexity 创始人 Aravind Srinivas），并提醒大家在吸收信息时，必须主动做“去偏见化”（Undo the biases）处理。 --- ### [059:52] 传统团队落地 GenAI 业务的核心卡点 现场下一位观众提出了一个普遍的企业级痛点：一家已有成熟业务的公司，内部验证了几个基于 LLM 的概念验证（POC），但在将其转化为真实商业价值时遇到了巨大鸿沟。由于 LLM 基础设施与传统机器学习（ML）架构截然不同，构建跨部门支撑产品团队的 AI 基础设施时，需要牢牢抓住哪些核心考点？ 吴中杰率先开麦解答。他坦言 OpusClip 作为一家专注产品落地的应用型公司，并没有像底层模型厂商那样动用数万张 GPU 去从零训练超大模型，而是以产品效能和用户留存为最底层的度量指标，在有限算力下寻找最优的架构解法。 ### [01:01:13](https://www.youtube.com/watch?v=Lt-lVe957hc&t=3673s) 技术人切忌“拿锤子找钉子”：理解用户真实需求 吴中杰（Jay）指出，虽然大家都在讨论 GPU 等底层算力，但开发产品的核心目的始终是用 AI 打造出真正有用的工具。因此，至关重要的一步是深刻理解用户的真实痛点，并借助 AI 的力量去解决它们。如果脱离了对客户问题的理解，这就好比“手里拿着一把闪亮、崭新又强大的锤子，满大街去找钉子”，最终根本无法产生实质性的商业影响力。 在场的很多从业者都是技术背景出身，Jay 建议技术人员更要主动去培养产品感觉（product sense），阅读如《创新者的窘境》（*The Innovator's Dilemma*）以及“待办任务理论”（Jobs to Be Done）等经典商业书籍。这类商业认知能帮助技术人跳出单纯的技术视角，去洞察客户并发现真正的产品机会。AI 本质上是一门让你打造出更强大锤子的新技术，但最关键的永远是：“找到那些过去没有被又旧又烂的破锤子敲好的钉子”。 --- ### [01:02:29](https://www.youtube.com/watch?v=Lt-lVe957hc&t=3749s) GenAI 落地的首要关卡：评估（Evaluation）从静态到动态 在 Jay 的发言基础上，Martian 联合创始人 Jason Hu 补充道，从工程实操角度来看，**评估（Evaluation）是构建生成式 AI 应用中最重要、也是必须首先解决的核心问题**。在与众多应用层团队合作的过程中，Jason 发现许多公司甚至连如何评估自己的 AI 应用都没搞清楚，就盲目扎进各种花哨的技术方案中。但在生成式 AI 时代，评估的难度极其巨大，因为评估大语言模型（LLM）的本质，几乎就等同于评估一个人类的智力与表现。系统需要一套稳定、鲁棒且一致的指标体系，以判断模型究竟是在进步还是在退化。 现场主持人随即追问，在评估流水线（Evaluation pipeline）上，除了像统计分析（statistic）等常用手段，是否有推荐的工具或供应商方案。Jason 坦言，由于大语言模型天生的复杂性，这并不是一个能靠单一工具解决的问题。目前业界的评估体系呈现出“从静态到动态”的光谱演进： - **静态基准测试（如 MMLU）**：让大模型做多选题（Multiple-choice questions），这类测试形式固定，结果容易解析与量化打分。 - **中间复杂度与推理任务（如 GSM8K）**：让模型解答数学应用题。这类题目往往无法通过简单的数字比对完成解析，而是包含一整套推理链条（reasoning process），往往需要引入人类专家或其他 AI 评测系统进行综合裁决。 - **代码生成评测**：需要直接在沙箱环境中执行生成的代码，根据运行结果来验证正误。 - **极度细微的开放式交互（如评估 ChatGPT 或通用聊天机器人）**：这与评估一个人类的交谈逻辑非常类似。Martian 内部针对此类复杂场景研发了一整套评测工具集与框架，综合融合了**人工标注（Human Annotation）**、**机器辅助标注（Machine-assisted Annotation）**与**全自动评估（Automated Evaluation）**等多种手段。 --- ### [01:05:09](https://www.youtube.com/watch?v=Lt-lVe957hc&t=3909s) 模糊输出的解法：AI 评测 AI 与草莓分级的“漏斗哲学” 面对极难量化的“模糊输出（fuzzy output）”，Jay 分享了 OpusClip 内部非常独特的实战做法：**用 AI 来评估 AI**。很多用户惊叹于 OpusClip 生成的短视频切片质量与趣味性远超竞品，但 Jay 透露，这并不是因为他们掌握了什么不可告人的“魔法 Prompt”，他们的主力模型其实是极其便宜的 GPT-3.5（成本相较于 GPT-4 便宜了将近 95%）。 ```mermaid flowchart TD accTitle: OpusClip AI 视频切片生成与打分筛选流程 accDescr: 展示了从 4 组生成 Prompt 产出 100 个视频切片候选，再由评估 Prompt 进行多维度打分，最后挑选前 20% 交付给用户的过程。 subgraph GEN["生成阶段 (GPT-3.5)"] P["输入长视频"] --> P1["Prompt 策略 A"] P --> P2["Prompt 策略 B"] P --> P3["Prompt 策略 C"] P --> P4["Prompt 策略 D"] P1 & P2 & P3 & P4 --> CAND["生成约 100 个候选切片<br/>(Candidates)"] end subgraph EVAL["评测打分阶段"] CAND --> CRIT["多维度评估 Prompt 组合<br/>- 趣味性与故事性<br/>- 开头 Hook 是否吸睛<br/>- 缺陷检测 (如突然截断)<br/>- 广告内容识别"] CRIT --> SCORE["综合打分与排序<br/>(Weighted Scoring & Ranking)"] end subgraph OUT["分级交付"] SCORE --> TOP["挑选排名前 20% 优质切片<br/>交付给用户"] SCORE --> DROP["丢弃其余 80% 平庸/缺陷切片"] end ``` 为了让平价模型产出极致质量，OpusClip 采用了一套“广种薄收”的多层机制： 1. **多路并发生成**：针对同一个长视频，同时运行至少 4 组完全不同策略的 Prompt，竭尽全力发掘不同维度的剪辑点，生成约 100 个候选切片片段。 2. **多维度交叉打分**：引入另一组专门的评估 Prompt 对每个候选切片打分。评估指标涵盖：是否具备故事感、开头是否有足够吸引人的“黄金钩子（Hook）”、结尾是否突兀截断，以及是否夹杂了赞助商广告等硬伤。 3. **综合排序取 Top 20%**：将各项得分加权汇总后进行严格排序，只提取排名前 20% 的顶级切片交付给最终用户。 Jay 用草莓农场的分拣体系生动比喻这一逻辑：农场里采摘出来的草莓，最好的那一批打包卖给 Safeway 等高档超市，烂掉的直接丢弃，中等平庸的则流向廉价菜摊。做 AI 产品也是如此——让底层系统用多路尝试去死磕，再用 AI 进行自动化层级筛选，挑选出最顶级的产出。 但 Jay 也指出了这种“AI 评测 AI”的巨大局限。他在 Twitter（@Jay of Opus）上曾发布过一个案例：他拿一张 AI 生成的手工匠人图片进行测试，图中人物的头部与脖子明显错位断裂，肉眼一眼就能看出异常。然而，无论是拿给 GPT-4、Gemini 还是 Anthropic 的 Claude 检查，所有模型的回答出奇一致：“这张图片非常正常，看起来合情合理。”哪怕你在追问中明确引导“他的头部和颈部是不是有问题？”，AI 依然嘴硬回答“完全没毛病”。这说明多模态与多智能体评测固然有效，但目前绝对不能做到 100% 盲目依赖。 --- ### [01:08:25](https://www.youtube.com/watch?v=Lt-lVe957hc&t=4105s) 分布式训练的不确定性与所谓的“可解释性困境” 在观众提问环节，一名正在与 AWS 团队合作从事机器学习系统可靠性（Reliability）和可观测性（Observability）研究的现场同学抛出了一个硬核问题：随着模型规模暴增，分布式训练（如采用张量并行 Tensor Parallelism 或流水线并行 Pipeline Parallelism）成为必然。然而在相同数据、相同初始权重的情况下，采用不同分布式策略训练出来的模型输出往往存在巨大差异，数值误差在深层网络中不断累积，导致 AWS 等大厂的工程师在排查故障时难以定位。他询问嘉宾，在分布式系统训练中究竟该如何提升可靠性与可解释性。 贾扬清（Yangqing Jia）情绪激动地抢麦回应，直言“对此我有太多话要说了”： 首先，**浮点计算不可复现性绝不是 AI 时代的新鲜事**。早在 1970 年代的高性能科学计算（HPC）领域，科学家们就已经明确认识到在实际数值计算中完全复现几乎是不可能的。贾扬清笑称当前的 AI 社区（包括他自己在内）有点像“被宠坏的孩子”，遇到问题不愿去已有成熟学科中寻找现成方案。在数值计算中，浮点数加法连结合律都不满足——`1 + 2 + 3 + 4` 的精度结果往往不同于 `4 + 3 + 2 + 1`；指数运算中如果指数高达 62，底层数值早就溢出（OOM/Overflow）了。科学计算早就提出用**统计度量（statistical measurement）**来评估数值稳定性（numerical stability）。面对分布式差异，工程师应当正视并引入数值稳定性的成熟框架，而不是幻想绝对确定性。 其次，针对大家常谈的**模型可解释性（Interpretability）**，贾扬清直率地批评其“带有过度的学院派色彩（outrageously academic）”，在工业实践中，大家根本不在意这种所谓的可解释性： > “我们做计算机视觉的，你来用像素（Pixel）给我解释一下究竟什么是‘猫’？我们非要强迫计算机视觉系统搞出一套‘具有可解释性’的分类逻辑，那反过来解释我们人类的大脑呢？大家会说‘猫有尖耳朵’，那什么叫‘尖’？什么又叫‘耳朵’？” 贾扬清总结道，现实世界中很多复杂系统本质上就是自然复杂且不可完全解释的，这在大部分实际场景中是完全可以接受的。就像人们每天下班开车回家，没有哪个驾驶员能 100% 洞悉车辆内部每一颗金属零件是否发生了微观金属疲劳，但这并不妨碍汽车作为一种高度成熟的工业工具被全人类广泛使用。 ### [01:12:19](https://www.youtube.com/watch?v=Lt-lVe957hc&t=4339s) 拥抱概率世界：从确定性代码到统计学安全护栏的范式转变与圆桌结语 圆桌讨论在最后回归到了整场活动最初抛出的核心问题——在生成式 AI 时代，产品与系统架构究竟发生了什么根本性的转变？嘉宾对此进行了一场极具技术洞见的哲学总结：传统软件工程建立在严格的确定性之上，工程师调用一个接口清晰的 API，给定固定的输入，就必然会得到固定的输出。但在大模型时代，这种底层规则已经被彻底打破。AI 在本质上是纯粹的统计学产物（Statistical），许多输出和内部机理天然就无法完全可解释。与其执着于消除所有不确定性，工程师和产品经理更应该坦然接受这一事实，学会在不可解释的黑盒外部建立“统计学安全边界”（Statistically Safe Box）与安全护栏（Security Fences），只要系统经过统计学约束后能够在预设的边界内稳定运转，那就是一种行之有效的工程解法。 这种转变要求整个行业重新思考系统设计的容错哲学与安全边界。在做完这番精彩的阐述后，嘉宾幽默地向大家致歉，笑称自己刚才可能“吐槽、长篇大论得有点太多了（ranted too much）”。主持人随即回应并给出了极高的评价，认为这一观点为今晚的讨论画上了一个完美的句号——“这个世界已经变了，我们只需要接受它”，正是当下拥抱生成式 AI 最通透的思维模型。随后，主持人宣布当晚的圆桌交流正式结束，现场场地将开放至晚上 9 点供参会者自由社交拓展人脉，并号召全场观众用热烈的掌声感谢徐老师（Howie Xu）、贾扬清、吴中杰（Jay Wu）以及胡 Jason 带来的干货分享。伴随着全场的掌声与致谢声，活动在充满科技前沿探索热情的氛围中圆满落幕。 ```mermaid flowchart TD accTitle: 传统确定性编程与统计学AI架构对比 accDescr: 展现从传统的固定输入输出函数调用，向以大模型为核心配合外围统计学安全护栏的架构演进。 subgraph Deterministic["传统软件工程（确定性）"] A["固定输入 (Fixed Input)"] --> B["预定义 API / 函数"] B --> C["确定输出 (Fixed Output)"] end subgraph Probabilistic["生成式 AI 工程（统计学）"] D["多模态 / 自然语言输入"] --> E["统计学黑盒 (LLM Core)"] E --> F{"安全护栏检测 (Guardrails)"} F -->|"落在可控统计边界内"| G["安全可用结果 (Safe Output)"] F -->|"触发异常边界"| H["降级拦截 / 重新采样"] end ``` - **确定性与统计学的本质分野**：传统编程追求“明确 API 定义下的固定输入与固定输出”；而 AI 系统本质上属于概率统计驱动，不具备绝对的一致性与全流程可解释性。 - **构建安全护栏（Security Fences）**：面对模型的不确定性与不可解释性，正确的工程思路是将模型装入经过统计学度量与验证的受控盒子里，通过外层规则、边界监控与护栏机制确保最终业务安全。 - **活动收尾与致谢**：主持人宣布活动主流程圆满结束，现场预留充足的社交时间至晚间 9 点，观众在热烈的掌声中向硅谷徐老师、贾扬清、吴中杰和 Jason 等分享嘉宾致意。 ## Notable quotes > [10:30](https://www.youtube.com/watch?v=Lt-lVe957hc&t=630s) 其实是一场非常友好的对话，大家可以去推特爬楼看原文 — 贾扬清 > [13:54](https://www.youtube.com/watch?v=Lt-lVe957hc&t=834s) 这完全是大错特错的认知 — 吴中杰 > [13:54](https://www.youtube.com/watch?v=Lt-lVe957hc&t=834s) 否则我们本质上只是在给 OpenAI 打工 — 吴中杰 > [18:02](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1082s) 2023 年是英伟达赚取巨额利润、其余各方进行崇高实验的一年 — 徐老师 (Howie Xu) > [33:05](https://www.youtube.com/watch?v=Lt-lVe957hc&t=1985s) 确定性会换来更低的价格 — 贾扬清 > [57:43](https://www.youtube.com/watch?v=Lt-lVe957hc&t=3463s) 刷推文就像从互联网上随便下载一个可执行文件（.exe），你根本不知道里面有没有恶意软件。 — 徐老师 (Howie Xu) > [01:01:13](https://www.youtube.com/watch?v=Lt-lVe957hc&t=3673s) 手里拿着一把闪亮、崭新又强大的锤子，满大街去找钉子 — 吴中杰 > [01:01:13](https://www.youtube.com/watch?v=Lt-lVe957hc&t=3673s) 找到那些过去没有被又旧又烂的破锤子敲好的钉子 — 吴中杰 > [01:02:29](https://www.youtube.com/watch?v=Lt-lVe957hc&t=3749s) 评估（Evaluation）是构建生成式 AI 应用中最重要、也是必须首先解决的核心问题 — Jason Hu > [01:08:25](https://www.youtube.com/watch?v=Lt-lVe957hc&t=4105s) 我们做计算机视觉的，你来用像素（Pixel）给我解释一下究竟什么是‘猫’？我们非要强迫计算机视觉系统搞出一套‘具有可解释性’的分类逻辑，那反过来解释我们人类的大脑呢？大家会说‘猫有尖耳朵’，那什么叫‘尖’？什么又叫‘耳朵’？ — 贾扬清 > [01:12:19](https://www.youtube.com/watch?v=Lt-lVe957hc&t=4339s) 这个世界已经变了，我们只需要接受它 — 主持人 ## Mentioned Statsig, Howie Xu, 贾扬清, 吴中杰, Jason Hu, Lepton.ai, OpusClip, Martian, OpenAI, Anthropic, Character.ai, VMware, Greylock Partners, Stanford University, Alchemist Accelerator, Byte into Future, Perplexity.ai, Alibaba, Facebook, Caffe, Caffe2, ONNX, PyTorch, Airbnb, LinkedIn, ByteDance, aligns.ai, Elmo Chat, DSPy, Tesla, AWS, Google Cloud, Booking.com, Marriott, IBM, Microsoft Azure, Notion, Quip, Elastic, MongoDB, Databricks, Snowflake, Google Brain, Bing, Llama 3, Jasper, Andrej Karpathy, Hacker News, Twitter/X, Gemini, Claude ## Video description Howie Xu, Guest Lecturer, Stanford First, we have our guest, Howie. Howie is a seasoned technology veteran in Silicon Valley, with a career that began by founding VMware's networking division, now a multi-billion-dollar revenue business. His diverse experience spans founding startups, holding executive positions at industry-leading companies, serving as an Entrepreneur in Residence at Greylock Partners, and mentoring and teaching at prestigious institutions such as Stanford University and leading B2B accelerator Alchemist. Howie has recently started a YouTube channel, "Byte into Future." Make sure to subscribe now, as he will host some amazing entrepreneurs and do live streaming from the iconic New York Stock Exchange floor in just two weeks. Yangqing Jia, Cofounder and CEO, Lepton.ai Next, I'm pleased to introduce Yangqing, the founder and CEO of Lepton.AI, one of the top-performing generative AI infrastructure service platforms. Remarkably, using Lepton's infrastructure, Yangqing developed a large language model-based search engine with just 500 lines of code in less than a week, which he then open-sourced (and sparked a debate with the CEO of perplexity.ai). Prior to Lepton, Yangqing was a Vice President at Alibaba and the Director of AI Infrastructure at Facebook. There, he created Caffe2 and co-led the development of PyTorch, which has become the most popular machine learning framework. Zhongjie(Jay) Wu, Cofounder and CTO, OpusClip Jay is Cofounder and CTO of OpusClip. His startup pivoted from a streaming tool to the current AI Video clipping product in 2023 and launched in June that year. In less than a year, OpusClip is reaching 15M Annual Recurring Revenue and is growing fast. (If you are familiar with startups, you should know how crazy that growth is.) I am a heavy user of OpusClip myself and love the product. Before that, Jay was an engineering manager at Airbnb and Engineer at Linkedin with a broad spectrum of engineering practices. Jason Hu, Founding Engineer, Martian Last but not least, we have Jason, the founding engineer at Martian, a fast-growing LLM router launched last year. Martian router is about selecting the best AI solution for your tasks, and is already adopted by developers from over 300 companies, including OpenAI and Amazon. Before Martian, Jason did research at the Chicago Human+AI Lab and ByteDance. He leads one of the largest AI communities in the Bay Area, aligns.ai, with speakers and community members from OpenAI, NVIDIA, Stanford, and more. Tonight, we will focus our talk on product development with GenAI. We all know that GenAI is going to be disruptive, but it can only disrupt through products. We have a forty minute panel discussion, followed by 20 minutes of live Q&A from the audience. Questions Is building GenAI products fundamentally different? Is adoption faster than mobile adoption? Is the cost manageable? Is it going down? Are we in a S curve, and if so, what stage are we in? Will the adoption of AGI be a centralized process or a distributed effort? Open Source models vs. closed source models What role will smaller players and middleware companies play in this process? How is it going to impact big tech companies like FAANG? Is it going to make them more dominant? Hot take — is Perplexity.ai a hype or a true demand? What are the top three challenges of GenAI we need to solve for this year and next year? ──────────────────────────────────── 嘉宾 / Guest：贾扬清（Yangqing Jia） PyTorch创始人之一，前Meta Director，阿里VP，Lepton AI创始人 @ Lepton AI 更多嘉宾访谈：https://www.lizheng.ai/guests 加入课代表社区：https://www.superlinear.academy ──────────────────────────────────── [Source](https://www.youtube.com/watch?v=Lt-lVe957hc)

[10:30] 贾扬清: 其实是一场非常友好的对话，大家可以去推特爬楼看原文 [13:54] 吴中杰: 这完全是大错特错的认知 [13:54] 吴中杰: 否则我们本质上只是在给 OpenAI 打工 [18:02] 徐老师 (Howie Xu): 2023 年是英伟达赚取巨额利润、其余各方进行崇高实验的一年 [33:05] 贾扬清: 确定性会换来更低的价格 [57:43] 徐老师 (Howie Xu): 刷推文就像从互联网上随便下载一个可执行文件（.exe），你根本不知道里面有没有恶意软件。 [61:13] 吴中杰: 手里拿着一把闪亮、崭新又强大的锤子，满大街去找钉子 [61:13] 吴中杰: 找到那些过去没有被又旧又烂的破锤子敲好的钉子 [62:29] Jason Hu: 评估（Evaluation）是构建生成式 AI 应用中最重要、也是必须首先解决的核心问题 [68:25] 贾扬清: 我们做计算机视觉的，你来用像素（Pixel）给我解释一下究竟什么是‘猫’？我们非要强迫计算机视觉系统搞出一套‘具有可解释性’的分类逻辑，那反过来解释我们人类的大脑呢？大家会说‘猫有尖耳朵’，那什么叫‘尖’？什么又叫‘耳朵’？ [72:19] 主持人: 这个世界已经变了，我们只需要接受它

[17:58] Ho: 2023 is a year that Hefty profits for NVIDIA and lofty experiments for the rest [23:23] Yangqing: sloppy input sloppy output kind of situation [53:42] read a good book three times rather than read three books [61:17] Jay: trying to find a nail using a flashy new powerful Hammer [68:25] Shway: AI is naturally just statistical. It's not like if you call a function, it has a defined API, a fixed input, a fixed output. No longer—this world has changed and we just need to accept it.

[0:18] founding engineer [0:20] Jason CER [0:25] J large language model [0:28] router [0:30] for large language [0:35] model [0:58] clip [1:07] for destion [1:56] open reward planning [2:17] fundamental [2:20] difference [2:22] out sloppy in out [2:44] centralized [2:51] decentralized Computing history [3:07] evaluation [3:23] evaluate evaluate [3:36] REM [3:44] solution solution [4:03] course Builders mindset [4:42] hey [4:43] everyone Hi everyone we're going to get [4:46] started in just a minute or two here so [4:50] if you want to grab a drink or a snack [4:52] and then find a [4:54] seat [4:57] sorry we're going to get started in just [4:59] a minute or two if you want to find a [5:00] drink or a snack and then find a seat [5:03] that would be awesome thank you [5:12] [Music] [5:45] ttin [5:53] one testing [5:57] two testing one [6:11] while [6:36] all [6:39] right [6:42] hello let's try I think [6:51] it's [6:55] check check check check check check can [6:59] you hear me [7:00] okay could you yeah hello hello yeah [7:06] nice all right thank you everyone [7:08] welcome to AB talk and uh at the Social [7:12] Hour uh a lot of people ask me why AB [7:16] talks it is because this event is [7:18] sponsored by our company Stasik Stasik [7:21] is a modern data platform for prodct [7:24] development and we offer uh essential [7:26] tools such as uh future Gates [7:29] experimentation and analytics U sesc is [7:32] hosting this event for the sake of the [7:35] community and uh also because we are we [7:38] provide services to a lot of AI [7:40] companies such as openi anthropic and [7:43] character. AI so we want to learn from [7:46] the AI Community about the best [7:48] practices in product development uh in [7:51] the Gen geni era so that is the house [7:56] cleaning uh our team number Morgan is [7:58] the invent manager there if you need [8:00] anything just go to Morgan Jesse is our [8:04] uh head of head of recruiting we are [8:07] 100% onsite at Bev but if are interested [8:11] we have a lot of position open and Brock [8:14] our uh yeah that relationship yeah so if [8:19] you need anything just spend our well of [8:21] us uh let me do a quick intro about our [8:25] speakers tonight we are honored to [8:27] invite h y [8:30] Jason and J here so I prepared a little [8:33] bit introduction if you don't know them [8:35] before so first ho is seist technology [8:40] veteran in Silicon Valley with a career [8:42] that began by founding wiw's Network [8:45] division nor now a multi-billion dollar [8:48] Revenue business his diverse experience [8:50] spans funding startups holding exective [8:53] positions as industry leading companies [8:57] serving as an entrepreneur in Residence [8:59] at Greylock partners and mentoring and [9:01] teaching at prestigious institutions [9:04] such as Stanford University and leading [9:07] B2B accelerator Alchemist H has recently [9:10] started a YouTube channel it's called [9:13] fight into future make sure to subscribe [9:15] now and he will host some amazing [9:17] entrepreneurs and do live streaming from [9:20] the iconic New York St New York Stock [9:22] Exchange floor in just two weeks so bite [9:25] into future make sure to subscribe next [9:28] I'm pleased to introduce yangqin the [9:30] founder and CEO of LEP AI one of the top [9:34] performing Genera generative AI [9:36] infrastructure service platforms [9:39] remarkably using lpt's INF [9:41] infrastructure y developed a large [9:44] language model based search engine with [9:47] just 500 lines of code in less than a [9:49] week which he then open sourced and I [9:53] want to add and Spark spark debate with [9:56] the CEO of perplex perplexity [9:59] a very friendly conversation friendly [10:01] conversation you can check the Twitter [10:04] uh [10:04] threads prior to lepton uh yaning was a [10:08] vice president and Alibaba and the [10:10] director of AI infrastructure at [10:12] Facebook there he created Cafe Cafe too [10:16] uh Cafe 2 Cafe was actually back in the [10:18] grass school days back at BIR across the [10:21] oh nice and onx among other things and [10:24] he collat the development of PCH which [10:28] has become the most popular our machine [10:29] learning [10:30] framework yeah actually since we're [10:32] already making advertisement with you [10:34] know YouTube channels and Har and things [10:35] like that um uh we recently well we're [10:38] in air infra company so people always [10:40] wonder what air infra companies do and [10:42] uh uh we uh we we we mail made a really [10:46] cute little uh Chrome extension called [10:48] Elmo chat Elmo as you know s Street Elmo [10:51] so if you go to elmo. chat you can [10:53] basically download install it as Chrome [10:55] extension what it does is it summarizes [10:57] every single web page for you whether it [10:59] be news articles we have the news posts [11:02] play the discussions YouTube videos PDFs [11:04] and things like that so completely free [11:06] this is basically our way of basically [11:08] introducing something uh that could be [11:10] used by users and run completely on the [11:13] on left platform so if you're interested [11:15] check it out this elmo. Che and yeah [11:17] completely free PR high quality and I [11:19] apologize my big stupid head covering [11:22] the screen no this uh perfect and I [11:25] actually was kind of packging that uh in [11:27] my first question because the first [11:28] question is the about product that's a [11:30] great platform thank [11:32] jump and next one we have Jason Jason is [11:37] the founding engineer uh at marshan um [11:41] and marshan is a fast growing llm router [11:45] launched last year if you don't know [11:46] what router is mar router is about [11:49] selecting the best AI solution for our [11:52] tasks and is already adopted by [11:54] developers from over 300 companies [11:56] including openi and Amazon before Mar [11:59] Jason did research at Chicago human plus [12:02] AI lab and bance he he leads one of the [12:05] largest AI communities in the Bay Area a [12:08] lens. with speakers and communities [12:11] members from openi theia Stanford and [12:14] more since we're also doing ads everyone [12:16] please follow our Ali AI Twitter AIG ns. [12:21] and we constantly hold events uh in C so [12:23] everyone's [12:25] welcome yeah and uh the last one I don't [12:27] think you even need to do that because [12:29] the introduction is about your product [12:32] Jay is the co-founder and CTO of opos [12:35] clip his startup pivoted from a [12:37] streaming to to the current AI video [12:39] clipping product in 2023 and launched in [12:43] the J in June that year in less than a [12:45] year opas clip is reaching 15 million [12:49] annual recurring revenue and is growing [12:51] fast so if you are familiar with start [12:53] apps you should know how crazy this [12:55] growth is I am a heavy user of opos clip [12:58] myself [12:59] and I just found out that H is also one [13:03] but I'm a paying member and I think H is [13:05] still on thee right [13:07] now paying user oh you already paid [13:10] because I started this YouTube [13:12] channel speaking of commercial make sure [13:15] you subscribe because that's also [13:17] supporting the Opus as well nice yeah [13:21] and before that Jay was an engineer [13:23] manager at arbnb and engineer at linkoln [13:26] with a broad spectrum of engineering [13:28] practices so that's our the introduction [13:31] as you can see we invited uh speakers in [13:34] the Gen AI space Not only in the product [13:38] but also middleware and infra so today [13:41] we hope to go to some deep questions I [13:44] prepare a list of questions but uh even [13:46] if it just cover one but cover a lot of [13:49] perspectives I think it's still good so [13:52] I'll leave 20 minutes for audience QA as [13:54] well all right so I'll just start and [13:57] the first question I'll start with J um [14:00] is building gen products fundamentally [14:05] different um [14:07] so check check is it [14:10] working okay so yeah there are [14:12] definitely things that are different and [14:14] some things are [14:18] um thanks um so some of the things are [14:22] different some of them things stay the [14:23] same so let me talk about this uh things [14:26] that doesn't change so when you build a [14:28] product um you still need to understand [14:31] deeply what the user need so you can [14:34] refer to these uh theorems such as jobs [14:37] to be done and uh of of course uh so [14:41] also like things like design and [14:43] marketing is still very important uh [14:45] when people talk about uh geni um [14:49] applications um you might think that we [14:52] need a line of input like chat GPT and [14:55] you don't need the design uh that's [14:57] pretty much wrong at least in my product [14:59] we still build relatively uh good [15:02] beautiful product that is easy to use [15:05] and also uh marketing is important uh we [15:08] had a growth uh that was pretty good [15:10] because we built relationship with [15:12] different influencers uh during the [15:15] previous not so successful product uh [15:17] when we launched this product people [15:20] that we knew were so excited that they [15:22] decided to uh talk about our product for [15:24] free uh because it makes them cool and [15:27] they they really love the product but if [15:29] you don't know these people and when you [15:31] don't you're not so ready about [15:33] marketing then you might be a follower a [15:36] similar product might just be uh a [15:39] leader position in the market and you uh [15:42] may not have these uh free traffic so of [15:45] course there are uh things that uh are [15:48] different for example uh user tolerance [15:51] is you uh typically higher compared to [15:54] uh more um incumbent spaces uh people [15:58] know know that it's a AI product and [16:01] they know that it's very new that the [16:03] technical uh aspect could be different [16:05] and challenging so people are so excited [16:08] that it gives them a lot of productivity [16:10] but at the same time they are okay with [16:12] some of the bug and also uh cost can be [16:15] a lot of a concern so we started by uh [16:18] having 5x more uh even 10x more cost per [16:23] uh processing power we provide to the [16:25] user in the beginning and we tried [16:28] really hard to to bring it down so that [16:30] we can reach uh Cash Cash positive [16:32] otherwise we would just be uh working [16:35] for open AI basically uh and uh at the [16:38] same times um I can see a lot of our [16:42] competitors they were not having the uh [16:45] Mass scale effect which economy of scale [16:48] so they could be really suffer from uh [16:51] higher cost and in the long term um much [16:55] higher uh cost efficiency can be really [16:58] important for your product so uh [17:02] something that is uh very different is [17:05] um prompt engineering becomes a thing [17:08] and it's quite new and has some imple uh [17:12] implications uh during your [17:14] collaboration because before you might [17:16] have a bunch of research Engineers uh [17:19] they they are AI engineer they train the [17:21] model and they finish everything but [17:23] during the prompt engineering era you [17:26] probably don't need uh many hardcore AI [17:29] Engineers but you need people who can [17:32] write prompts and these people could be [17:35] product managers or product operation [17:38] people they need to work in joint with [17:41] uh your AI engineer so to write prompts [17:44] tune the prompts and make things work so [17:47] that's pretty much um I think answer [17:50] from me thanks you can do fre style or [17:53] we can do around maybe clockwise so are [17:58] sure [17:59] um I think it's very very different from [18:02] the last generation of the software um [18:05] both on the enterprise software and then [18:07] consumer software uh let me talk a [18:10] little bit more on the enterprise [18:12] software which I know about so if you [18:15] think about enterprise software [18:16] preciseness correctness is very [18:18] important the [18:20] predictability [18:21] reproducibility is very important and [18:23] then there's no reproducibility with uh [18:26] gen application [18:30] now some use cases like you know text to [18:33] Cod generating a you know table [18:35] generating the result you cannot be [18:37] wrong even one out of a million times [18:40] that alone one out of 100 times which is [18:42] you're already pretty good if you are [18:44] right 99 out of the 100 times so you [18:47] have to rethink about how do you do it [18:49] right it's not the end of the world [18:51] because what you're doing what you're [18:52] going to do is actually you limit the [18:54] input space output space and somehow [18:56] figure out a way so that the result is [18:58] actually a lot more predictable uh way [19:01] more than you know open a API allow you [19:03] to do um so you have to re rethink about [19:05] it that's sort of under under just the [19:08] software side of design and then you [19:10] know we all know that a QA is another [19:12] aspect an important aspect of the [19:14] software and the QA thing is totally [19:17] different like especially if it this is [19:19] text based like what is correct what is [19:21] not correct uh who's going to label it [19:24] for you who's going to tell you this is [19:26] correct in the in the traditional sof [19:29] you sort of do a happy path and then you [19:31] figure out some Corner case right you [19:33] extremely large number you extremely low [19:36] number zero minus you know whatever so [19:39] you kind of know that if if this this [19:42] software is right for five and a six and [19:44] eight you know that if input is 10 and [19:46] 11 it's likely correct but you need to [19:49] figure out whether when it's zero input [19:51] is zero or minus one or you know [19:53] infinitely large number whether that's [19:55] still correct but in the in the in the [19:57] generative it's unclear to me if I [19:59] change you know add one character maybe [20:01] the output is totally different for one [20:03] reason or another right so so it's a lot [20:06] less uh predictable so actually as I [20:09] think we as industry need to rethink [20:11] about how to do even QA uh what does [20:13] that what does that mean to do QA in the [20:15] enterprise software so I'm just giving [20:17] you a few examples so one of the reasons [20:19] you see that you know there are not [20:22] enough of the Enterprise um gen [20:25] application being in production is [20:28] because you know the kind of the issues [20:30] I was talking about in December uh 2023 [20:34] there was an article from [20:37] CNBC um you can search it it's it's [20:39] basically the title of the article is [20:41] 2023 is a year that you know Hefty Pro [20:45] uh Hefty uh profits for NVIDIA and then [20:48] lofty experiments for the rest which [20:51] captured pretty accurately what was [20:54] going on in 2023 I'm pretty sure that [20:57] 2024 is going to be lot well some were [20:59] better whether it's a lot better or a [21:01] lot worse um and if if you think about [21:04] why it's sort of a so many um lofty [21:08] experiments but a lot less production uh [21:11] it has to do with the software nature is [21:13] a very different um so so that's just [21:18] that right and then of course you know I [21:20] I'm not necessarily the expert in the uh [21:23] consumer software I think that there is [21:25] something very similar in that uh it's a [21:27] very different now for the consumer [21:29] software there's one thing that [21:30] potentially you have an advantage is um [21:33] like the kind of things that um J is [21:36] doing right you know you gener you know [21:38] basically I give you a video you [21:40] generate some clips but there's no [21:42] correct answer right it could be you [21:43] know one and a half minute it could be [21:45] one one minute um so you have more [21:48] leeway you know to sort of make sure [21:51] that customers or the clients or myself [21:53] I'm actually a customer right you know [21:55] even if you give me one minute or a [21:57] minute and a half that's fine right so [22:00] so you have some leway you there's more [22:03] tolerance I think J mentioned it um but [22:08] that doesn't mean that um the that does [22:12] not mean the problem is much easier for [22:15] instance right just because I care a [22:17] little bit less whether the end result [22:19] should be one minute and a half and then [22:20] one minute but ultimately there's still [22:22] a satisfaction right so the women and a [22:25] half video always you know um below my [22:28] expectation it's still going to be uh [22:31] it's still not you know you know if if [22:33] that's the case I wouldn't use the [22:35] software right you know I feel like [22:36] above my expectation I use it but how do [22:39] you make sure your software is actually [22:41] reach that above that threshold uh and [22:44] then predictably get to that state it's [22:46] actually not easy uh your feedback loop [22:49] is a little bit fuzzy in terms of what [22:51] is correct what is not correct so [22:53] overall my feeling is the uh software is [22:56] very very different in of AI you know [23:00] error than the previous [23:03] generation thank you and one thing I [23:05] pick up from the answer is seems like J [23:08] is a paradigm shift and for a paradigm [23:10] shift is fundamentally new then maybe [23:13] the old way of doing things cannot be [23:15] just continued you have to think about a [23:17] new way which uh we can deep dive in my [23:20] question number three but before [23:23] that yeah I think how actually made a [23:25] really good summarization already but uh [23:27] let me just it in a mathematical way um [23:31] I think it's going to be different [23:32] mainly because the way of interaction is [23:35] different um in the old days computer [23:37] science is all about going from the [23:40] sloppy input our abstract needs and [23:43] things like that very quickly into a [23:46] logical World whether it be a piece of [23:48] code some sort of Json some sort of apis [23:51] and things like that right we have very [23:53] sloppy input and then very fixed [23:56] output and now suddenly this whole just [23:58] changed with ch gbt you find that this [24:01] AI has become a blackbox and some sort [24:04] of really sloppy random output right I [24:06] mean for fuction it's fine but it just [24:10] you know like how do you actually [24:11] connect chbd output to like a database [24:13] query right so as a result I think [24:16] basically two things started emerging [24:17] which wasn't you know like wasn't some [24:19] important stuff back in the days where [24:21] we all work with API and computers talk [24:23] to each other one is basically we [24:25] started building applications on top of [24:27] the sloppy input sloppy output kind of [24:29] situation right there's and all kinds of [24:32] aigc companies there are chatbots there [24:34] are you know like say what even search [24:36] become kind of floppy because you just [24:38] search for something that gives you an [24:40] answer it doesn't really give you fixed [24:42] results or things like that fixed Pages [24:43] it just give you a sloy answer which [24:45] which even isn't isn't actually like [24:48] completely correct so that's basically [24:50] user experience starts changing we see [24:52] quite a lot of really awesome product [24:54] managers actually utilizing those kind [24:56] of tools to create new experiences new [24:59] um like new new functionalities and [25:01] things like that because our world when [25:04] we interact with this world we are so [25:06] comfortable with [25:08] liness and that's basically one thing [25:10] the other thing the second thing is [25:11] basically we are starting actually only [25:14] starting to basically solidify the old [25:18] way but using a technology we still want [25:21] to go from a sloppy World sloppy input [25:24] to a fixed logical world like code [25:27] generation ation is one right and you [25:30] might have heard about you know like say [25:31] language to SQL which is another one and [25:33] there's a lot of other examples and [25:35] that's why we see a lot of work starting [25:37] to actually kind of put in shackles on [25:40] the on uh put in handcuffs on the large [25:43] language modules we have Jon output we [25:46] have function call and things like that [25:48] at the very bottom we're we're defending [25:51] the old model which is a sloppy input [25:54] and a fixed output a fixed logical world [25:56] because that is still hugely important [25:59] so I think you know like B the first one [26:01] is completely new s input s output [26:03] enabled by aigc and llm and the second [26:06] one is basically since we want to take [26:08] these some new technology into the [26:10] existing work pipeline then there's [26:11] quite a lot of peripheral tools and [26:13] functionality and things like that that [26:15] need to be done I think we started two [26:17] different part nice this reminds me of [26:20] podast did with Jason and I use the word [26:23] fzy input fzy output this SL is the [26:26] industry stand kind of equivalent I made [26:28] it up so it's not standard for sure cool [26:31] yeah yeah I think Ying's point is very [26:33] very interesting and we're seeing [26:34] different modal modalities um in the [26:37] like how L computation interact with [26:40] traditional computation right on one [26:42] hand we have you know the traditional [26:43] software which is totally deterministic [26:45] we have you know Json formatted output [26:48] which try to make LM more deterministic [26:50] we have something more like you know DSP [26:53] which is a project L by Stanford NP [26:55] people which is to try to replicate this [26:57] p for architecture but the underlying [27:00] computation is all based on text B on LM [27:03] and we have the most fuzzy ones right [27:06] like chbt mode creat I think it's very [27:09] interesting to see what kind of those [27:10] modes of computation would Prevail in [27:12] what kind of scenario um and um some [27:15] other points I would like to bring so I [27:17] think the difference between AI software [27:19] and traditional software the first is [27:22] the difference in marginal cost I think [27:24] in traditional software like the usage [27:27] um increased [27:28] the cost is a sublinear function of usit [27:32] right like I can buy a $100 server and [27:35] it can serve uh I said 100 people it's [27:37] the same cost a th people is the same [27:39] cost um a 10,000 make the same cost so [27:41] it's like maybe more a little bit more [27:43] expensive but not that much but for LM [27:46] like I call one time gp4 can be a 10 [27:49] cents two times 20 cents a linear [27:51] function actually we are working with [27:53] many application clients and they are [27:55] much much more ctive than um they are [27:58] with let's say Ad or traditional Cloud [28:01] off because it's the marginal cost of [28:03] using AI is so so High um and another [28:07] thing I would like to mention is um I [28:10] think we need to think about to what [28:11] extent AI applications can have Network [28:13] fact have they enjoyed by traditional um [28:16] let's say social media right because [28:18] let's say in Google or Facebook um their [28:20] recommendation engin um can be better [28:23] and better if more people using it [28:24] there's more data on the platform um [28:26] however for someone using openi API it's [28:29] very hard for you to use your own data [28:31] to improve the performance of the AI API [28:33] you're calling right so what would be [28:36] the most in the long term for a [28:37] application company um I think this is a [28:40] task that every application company can [28:42] figure [28:43] out I want to double click um on the [28:47] cost thing because uh yeah the linear [28:49] relationship is uh Bome and uh I heard [28:55] Jay you mentioned economy of scale so [28:57] must have done something to reduce the [28:59] cost to maybe try to bend the curve to [29:02] sub linear and you are infra company so [29:05] must you know have a lot of experiences [29:08] in optimizing so any uh insights so how [29:11] to get the cost to a linear [29:15] relationship um so from from our usage [29:18] um the way to look at building the infra [29:21] o optimizing the cost is like basically [29:24] uh the same as the uh company Tesla so [29:27] they started out by building a car that [29:29] sells for uh 500 uh50 ,000 uh Model S I [29:36] think it's more than that it's a 100, [29:39] and then you start to build they start [29:41] to build model 3 which is a a relatively [29:44] high they start to uh sell the do motor [29:46] version and then they start to sell a [29:49] more basic version and drive the cost [29:51] down so it's not that they are giving [29:53] worse and worse cars uh or or simpler [29:56] cars but they also uh during their [29:59] production um they optimize all of their [30:02] techniques optimize their uh supply [30:05] chain and the cost goes down so it's the [30:08] same as when you build a aigc related [30:11] product uh in the beginning you may you [30:13] may have a very expensive uh CA [30:16] structure uh your usage is low and open [30:19] ey don't give you discounts and the more [30:22] you use it you start to get good deals [30:25] from vendors such as open AI you get con [30:28] offers from uh Google Gemini and you [30:31] also start to optimize your clusters uh [30:34] you optimize the usage of your gpus and [30:37] all these things come up together you [30:39] can project maybe a 80% saving on your [30:42] cost overall so uh you need to project [30:46] your pricing uh to be uh to to so that [30:52] after you have a saving of 80% you make [30:55] a reasonable profit so in the beginning [30:57] then you probably are operating at loss [31:00] but as as long as you put engineering [31:02] effort into optimization uh you will get [31:05] you will get much better uh [31:08] costs I have a question so from the 80% [31:10] cost reduction what are the percentage [31:12] that's from negotiating with AI [31:14] providers like openi what's the [31:16] percentage that's from negotia with a [31:18] traditional like a A or [31:21] GC um so open AI has this product called [31:26] um provisioned throughput [31:28] which basically you commit to a certain [31:31] number of tokens per minute for three [31:34] months or one month they give you a [31:36] discount on the um on on on the cost [31:40] which is roughly uh it depends on how [31:43] much but can range from 5% to up to [31:46] 50% and uh traditional Cloud providers [31:51] uh they usually offer if you are using [31:54] uh on demand like is uh instances they [31:59] typically can can go down by 70% if you [32:02] are using these super cheap on uh spot [32:06] instances you probably still can squeeze [32:08] an extra 20% out if you sign up on some [32:11] of their commitment program so uh and [32:15] then on top of that you have C cluster [32:17] optimizations you have algorithm [32:20] optimizations in my case uh I hired a [32:24] superstar in uh audio video engineering [32:27] and he wrote a to replace my gigantic [32:30] FFM Peg code and saved the 60 to 80% [32:35] during the the the rendering pipeline so [32:37] they are all huge uh savings I have to [32:40] do a plugin here oppos clip Al recently [32:44] sent with stasic to be our Enterprise [32:46] user so use uh stasic expensively does [32:51] AD testing [32:53] help I'm actually prepared to answer [32:55] that during the QA part yeah [32:58] yeah I can't give more detailed answer [33:00] when we talk about that to the yeah um [33:05] I'm actually not particularly worried [33:07] about cost um the reason is simple um [33:10] Supply chains is all from and there's [33:12] all kinds of existing knowledge that [33:13] does it and we're actually chatting just [33:16] now right about this uh there's one [33:18] simple rule in supply chain that applies [33:20] in AI as well which is predictability is [33:24] rewarded with a lower price right now [33:27] let me take what all those kind of crazy [33:29] ganizations NOA Kels and things like [33:31] that those are crazy let me just throw [33:32] that right let's make coffee some right [33:35] you walk into Starbucks you want to get [33:39] latte randomly right what you get $5 [33:43] right or something like that about that [33:45] price now another case is you walk into [33:49] Starbucks was like I want 100 cups of [33:51] latte right [33:52] now what will happen $5 do the same [33:57] price and but you'll suffer and uh the [34:00] bartend or whatever the person is called [34:03] they will suffer as well because you [34:04] have to wait for a long time for that [34:05] 100 cups to come but if you tell [34:09] Starbucks for the next whole year every [34:12] day in the morning I'm basically running [34:13] the company we have 100 people I'm every [34:15] morning at 9:00 a.m. I'm going to be [34:17] asking for 100 Cups Coffee delivered to [34:21] my friend to my friend what price you [34:24] you be getting you were getting a very [34:26] good discount and delivery time is good [34:30] right because you are actually doing a [34:31] very good predictability in supply chain [34:34] uh this whole thing happens all the time [34:36] and then you know like basically for [34:37] retail the price is always higher and [34:39] then if you go to car school or [34:41] basically if you have bulk purchases and [34:42] like that the predictable rate you're [34:44] getting [34:45] discount today a lot of these AI [34:48] applications are just starting and [34:49] basically we're just the on Dem mon uh [34:52] the the the uh the the Starbucks first [34:54] sitation right or in some cases because [34:57] people are not read predicting the usage [34:59] then sometimes there's 100 cup Starbucks [35:01] concentration which makes everyone's [35:03] life [35:04] miserable but I mean uh like like you [35:06] just mentioned right as we are seeing a [35:09] more [35:09] mature adoption of the technology [35:12] adoption of the product and as we are [35:14] having more regular traffic then the [35:16] conventional wisdom of supply chain just [35:19] kicks in and I think price is going to [35:21] be significantly dropping down uh and [35:24] then plug in one unpaid at of transal [35:26] Martian right so margin does is a very [35:28] simple logic correct me if I'm wrong [35:31] there's all kinds of retail traffic [35:33] people who are seriously using all those [35:36] like a like apis and things like that [35:38] Mar is like what guys just don't go to [35:40] individual places just come to me right [35:42] and then Maran did does exactly one [35:45] thing which is they converted retail [35:48] traffic into bulk [35:50] purchase which gives us a better [35:53] predictability in the supply chain and [35:56] so we left an AI for example we are both [35:59] providers right we both providers of [36:01] apis and then basically if people come [36:02] in and basically say like just they want [36:04] to do retail then of course they're the [36:05] price but if let's say hopefully down [36:07] the road Martian I'm actually starting [36:09] to talk about contract um if Martian [36:12] basically says hey I have this [36:13] predictable traffic right could you give [36:15] me discount you know Marriot down the [36:17] road actually gives booking.com 10% [36:21] Commission because they Agate traffic [36:24] and they are doing fedility right all [36:26] those kind of things start King in and [36:27] then made for for all kind of Industries [36:30] we already made supply chain extremely [36:32] efficient we're just not there yet [36:34] application wise and infra wise for AI [36:37] but it's not a very difficult thing I [36:39] think it's a it's something that will [36:41] happen and is happening right and uh and [36:43] I think that's why I don't really worry [36:45] about [36:46] cost um just to add a point to Yi I [36:49] think yes um aggregating retail demand [36:52] is one big better problem we have um but [36:54] we have another smart routing logic that [36:57] also does the cost reduction logic right [36:59] so let's say we're building like a math [37:01] GB so 90% of the math problems as on the [37:05] app is very simple it can be answered by [37:07] gp45 and 10% are very hard you must use [37:10] gp4 to answer it and to ensure user [37:13] experience people usually just uh use [37:15] gp4 to do to serve every client which is [37:19] very very expensive right gp4 is 30 [37:21] times more expensive than G by Puro so [37:24] what we do is that for each new prompt [37:26] we are automatically recognize whether [37:28] it's a hard problem or easy problem for [37:30] easy problem we go to gp5 Turbo for hard [37:33] problems we go to gg4 and due to the 30X [37:36] cost difference so we can like save like [37:39] 85% of the cost so that's another way [37:42] Martian can also help with the cing of [37:44] but yeah retail traffic application is [37:46] definitely one of them yeah I would like [37:47] to Echo that mainly because these kind [37:49] of things happen in conventional supply [37:51] chain as well uh if we want to if we're [37:53] a little bit sick uh you don't go to [37:55] specialist in like say like say St [37:57] passway immediately because those are [37:59] terribly expensive for the doctor and [38:01] for us as well but family doctor family [38:03] doctors what what are they doing they [38:05] are doing initial analysis like Martian [38:07] does plus routing they route you to [38:10] Specialists if needed or they basically [38:12] say well take a p like aspirin and be [38:14] done with it right those kind of things [38:16] are like like like conventional wisdom [38:18] in Supply chains and they're actually [38:20] like doing really well in AI as well [38:22] nice I like the analogy because uh we [38:25] just had a conversation with h before [38:27] for the this panel that we often see [38:30] history repeat itself yeah and U [38:33] actually it's time to move on to the [38:36] next topic about history repeting itself [38:38] and I would like to start with h u is [38:41] generative AI going to be centralized or [38:44] decentralized because um like we [38:46] mentioned PC actually started with ADM [38:50] being centralized and then became [38:52] decentralized then cloud is becoming [38:55] centralized uh how to [38:58] generaliz yeah I think the you know the [39:01] context just a little bit more was about [39:04] in general computation history in the [39:06] last 50 60 years we went from IB main [39:09] frame to PC you know client server and [39:12] then to the cloud in many ways it's a [39:15] you know a pendulum swing from [39:17] centralization centralized architecture [39:19] to distribute it and then U Back to [39:22] centralized in the cloud days you know [39:24] when when IBM you know in IBM days um [39:28] and then Bill Gate said in the 7 dayss [39:31] that we wanted to I wanted to bring PC [39:33] to every house and then you know IBM [39:35] guys you know laughed at him so but what [39:38] happened you know what 56 year later is [39:42] essentially we went back to that we have [39:44] three main FRS you know AWS asro and [39:46] then gcp so we essentially went back so [39:49] it's unclear who's going to laugh at the [39:51] mo at the latest but but I think what [39:54] what um we discussed was it's going to [39:56] be a pendulum back and the forth coming [39:58] back to the Gen you know the this is a [40:02] debate for a while that is that what's [40:04] the model right that if you ask Sam [40:07] Alman his his argument is look you know [40:10] you guys don't need to fo around just [40:13] you know use open AI open AI API do some [40:16] fine training pump pump engineering uh [40:18] be done with it there's no reason for [40:20] you to have your own model because [40:22] whatever our model is going to be a lot [40:24] more capable a lot better anyways [40:28] um I think that's uh you know in many [40:31] cases that's probably true but you know [40:32] in many other cases that's going to be [40:34] completely UNT right because you know [40:37] there's no way I'm going to for some [40:38] customer there's no way I'm going to [40:39] give you the data so for that reason I I [40:42] need to uh have my own model um and then [40:46] but in the context of the history [40:47] repeats itself I feel like you know [40:49] opening eyes like MRE right you know [40:52] there will be you it will be distributed [40:54] you know but it will be back and forth [40:56] you know what happens every time the [40:58] hist when history repeats itself is [41:00] always where's the Innovation bottom [41:02] that's real that's the real gist so I [41:05] think about why IBM INF you know went to [41:07] the PC architecture right that's because [41:10] there's not much interesting going on [41:12] with IB it's the boring you know [41:14] usability is so so horrible so people [41:17] wanted to have my own way of doing [41:18] things so a lot more Innovations but [41:21] then you know too much Innovation that's [41:23] a management held and then we went back [41:25] to the uh centralized so I think you [41:28] know as as soon as you know someone is [41:30] dominating monopolizing things it's a [41:33] human nature it's it's a you know [41:35] Innovation will happen po up somewhere [41:37] else right so so my view is there will [41:40] be um there will be a lot of the model [41:42] running on the edge or the equivalent of [41:45] the mobile or the PC end uh that's just [41:48] because you know uh open a is not going [41:51] to allow so many different use cases so [41:54] many different features um and it will [41:57] enable data but but at some point we'll [41:59] realize why do we manage so many models [42:01] and it's crazy so we will go back to the [42:04] centralized model so so that's the [42:06] context e and I were discussing you know [42:09] if you if he he was asking you is that a [42:11] centralized I think you know we are in [42:13] the centralized model predominantly but [42:15] we are kind of a you know we are going [42:17] to enable the a lot more decentralized [42:20] distributed models but at some point [42:22] right you know it doesn't make sense for [42:24] each of us to develop our own model so I [42:26] will go back [42:27] equivalent Cloud days it will come but [42:29] it's not there so we're are more moving [42:31] towards the other direction side um it [42:34] will take a few years for us to realize [42:36] that's that doesn't make any sense that [42:38] will be a few years from [42:42] now yeah in terms of centralization and [42:44] decentralization I'll probably uh skip [42:47] the technology part for another day and [42:49] uh and uh and basically give you like [42:51] one example I think centralization and [42:53] decentralization could also be uh [42:55] discussed on a different uh different uh [42:58] Dimension so let's say search right [43:02] every day we do search and things like [43:03] that what is a search engine you [43:07] use there a word right Google becomes a [43:09] word everyone I hope I hope I mean uh [43:12] statistics show that 97 or something [43:14] like 95% of people when they're doing [43:17] search they use Google right okay now [43:20] let me ask you an alternative question [43:22] and you'll probably realize that huh so [43:25] you're doing work right during our work [43:28] we have documentations we have all kind [43:29] of different kind of things what search [43:32] engine do you use for [43:34] that I will probably say more than half [43:37] of us are saying well it's probably not [43:39] Google right because Google doesn't have [43:40] it if I use notion I use notion as a [43:42] search right if I use quip which is God [43:45] forbid that's such an old thing um quip [43:48] has a search although it's bad right I [43:50] mean every Everything has a search and [43:52] maybe some of your companies you have a [43:53] search yourself right and there are some [43:55] suppliers like elastic and other [43:57] platforms that are mongodb that are [43:59] providing search as well so very quickly [44:02] we find that in the one end or what we [44:05] called the TOC world it's very [44:07] centralized it's search it's Google [44:09] right and in the Tob world when we're [44:11] actually dealing with business is [44:13] massively decentralized we actually [44:16] don't really have the Google of [44:17] Enterprise search we have a lot of [44:18] Enterprise search Solutions and I kind [44:20] of feel that AI well let's say Chat bar [44:22] right it's going to be similar in the to [44:25] world for me as a consumer I'm kind of [44:27] lazy I don't really want to shop around [44:30] for multiple choices and things like [44:31] that people Ching it's good yeah I'll [44:33] pay $20 per month which is five cups of [44:35] starbs coffee which is not too much [44:37] right uh and and I just happily use it [44:39] and that's why we're seeing a really big [44:42] convergence onto chat gbt for us to you [44:45] just like to to see conversation part uh [44:47] in the Enterprise world today things are [44:50] already happening there's a huge fight [44:52] there's a huge fight between GPD or gp4 [44:55] and cloud and you know Googles all kinds [44:57] of different kind of like Solutions and [45:00] I my believe is there's only going to be [45:03] much more much much more players in this [45:05] world not only these kind of providers [45:07] but also models applications you know [45:10] something capabilities built inhouse [45:12] from the Enterprises themselves right so [45:14] in the Tob World things are going to be [45:16] massively decentralized now plug in a [45:18] little bit of our what we see is [45:20] basically we see that the 2B world is [45:22] going to be decentralized and as a [45:24] result people will be needing tools [45:26] right uh when every company is actually [45:29] doing their websites and things like [45:30] that well database VI and those things [45:33] become a thin and then data dataor [45:35] snowflake like all those kind of [45:37] companies start emerging what about AI [45:39] there's got to be some sort of company [45:41] that does it and uh just to throw away [45:43] all those kind of details we strive to [45:45] be that company to help the [45:47] decentralized to be worked yeah than you [45:52] I think um the previous two speakers [45:54] have great really great answers I want [45:55] to offer the answer for from a slightly [45:57] different perspective I think from the [45:59] first principal point of view right what [46:02] determines the end state of industry is [46:05] basically only one thing that is the [46:07] economy of scale of that business so why [46:10] would you know social media and search [46:12] engine a to of the biggest industries [46:13] are extremely monopolized like Google is [46:15] the only player like Facebook the only [46:17] player why because that business has [46:19] huge economy of skill and because the [46:21] more uh the more user use Google the [46:24] better Google search engine is I was [46:25] talking to some early um team leader at [46:28] working on Microsoft them right he [46:30] described to the desperation like they [46:33] they put much more resource into U you [46:35] know search engine trying to catch up [46:37] working 724 but they still are not able [46:41] to catch up Google like Google if you [46:43] don't me mind me chiming in on this one [46:45] there's a technical reason behind that [46:47] uh do you know what is the biggest [46:49] signal for Google to rank the uh the the [46:53] the search results um about 5 years ago [46:57] go or like when dis believe as what [46:58] Google Google brain was craftly saying [47:01] Google brain deep learning is uh [47:03] supporting Google search they said in [47:05] their post you can actually search for [47:07] it they said AI or deep lant has become [47:10] the third largest signal the third [47:12] important most important signal to rank [47:14] search they never talk about what is the [47:17] biggest [47:18] one and the biggest one turned out to be [47:22] human feedback if I search for something [47:24] and I click something then it means [47:26] means that other people are more likely [47:28] to think that that one is important and [47:30] because Google is occupying more than [47:32] 90% of the traffic no one else is able [47:35] to compete with it because I [47:37] mean why do I think that search result [47:40] is good because the other people think [47:42] it's good it's natural it doesn't even [47:44] have anything to do with AI so that's [47:46] what you mean by uh Ben that's exactly [47:48] the case That's a classic definition of [47:51] econ skill or NW effect right and [47:53] similar with you know Facebook with [47:55] WeChat with all those social media [47:57] um and now the question is to understand [48:00] does foundation model LM has this [48:02] calling of skill um I have to admit like [48:05] about one year ago when I first see chbp [48:08] I was so scared that you know open can [48:10] be a company that literally take 95% of [48:13] world GDP G uh GDP right because it [48:16] seems to be a business with huge economy [48:19] and you put more data more compute into [48:21] one model and this one model can solve [48:23] every problem in the world what's the [48:25] business of a bigger economy [48:27] um but gradually when I you know dig [48:29] more into the industry and you know [48:30] train more myself and serve more clients [48:33] I discover it's actually another case [48:35] right I think um I think people are [48:38] discovering more and more new ways of [48:40] training model that are way more [48:42] efficient and those new techniques are [48:45] because they are like inspiration right [48:47] it's not possible for open ey to [48:50] monopolize those new ideas I think one [48:52] very good example is llama 3 I'm sure [48:54] many of the audience right here has [48:56] heard about it we have Benchmark in L [48:58] Lama 3 8 billion internally and it's [49:00] very good it's way better than many [49:03] previous models which was much larger [49:05] parameter size and it's very efficient [49:08] right and it's not being produced by [49:09] open a and I think at any any time Point [49:14] um the model uh the amount of resource [49:17] you put into training a model is [49:19] actually not uh the economy skill is not [49:22] that big right I put one billion into [49:24] training a model versus I put2 billion [49:26] to trainer model the second one is not [49:29] two times smarter right it's like a [49:31] declining marginal return in putting [49:33] more computer and resource so I think [49:35] this is one of the fundamental reasons [49:37] why I think OPI is not going to take [49:39] over the world because um this business [49:41] does not have that huge of economic [49:43] skill and another reason I think OPI is [49:46] not going to take over the world is um [49:49] AGI will not be built by AGI requires [49:52] Contex and specific knowledge to be [49:54] useful in um specific scenario and we [49:57] can think of um the smartest model in [50:00] the world as some genius college [50:02] graduate like Einstein his 18 years old [50:05] it's he's very very smart he can pick up [50:07] things very uh quickly but he's not [50:09] useful to I say work at Google work at [50:11] Martian right we have so many internal [50:13] context internal rules internal um [50:16] subtle you know like interpersonal thing [50:19] management everything like that's needs [50:21] some on boarding process and it's not [50:23] possible for OPI to get acquire that [50:25] part of the knowledge so there needs to [50:27] be you know third party SAS um third [50:30] party you know whether 2B or 2C to [50:32] modify the general purpose intelligence [50:34] into domain specific intelligence that's [50:36] actually doing the economically useful [50:38] work um and there is a very good paper [50:41] written by you know Berkeley people and [50:43] uh you know data break Co Founders [50:45] called you know compound AI systems [50:47] right that's exactly exactly a thesis [50:50] the best revs are not delivering by the [50:52] latest Cutting Edge the most the [50:54] smartest Foundation model but rather the [50:57] compound a systems that you know agents [50:59] rap um you know uh Chain of Thought and [51:02] other applications that build on top of [51:04] those um that's how we achieve real AGI [51:08] and that process will be decentralized [51:10] um in in asset [51:12] yeah J well it's a long discussion so [51:16] I'll keep it short so um it's a related [51:20] uh thought about [51:22] decentralization which is when you are [51:24] building a startup people or DCS will [51:27] ask you how are what's your mode and how [51:30] how about these big guys will defeat you [51:33] uh these uh large model companies they [51:35] will be so powerful that you you your [51:38] your application is is not going to be [51:41] uh useful this isn't especially true [51:44] when Jasper basically died after chat GP [51:47] came out the thing is uh I think in when [51:50] you talk about decentralization I I [51:52] agree with Jason's uh topic which is [51:56] domain knowledge or context is very [51:59] important it could be your kind of your [52:02] mode at least for you to survive for a [52:05] while for example in our company how to [52:08] clip a good video is getting more and [52:11] more becoming of our mode and these [52:13] domain knowledge they not they don't [52:16] appear very apparently on the internet [52:19] and they come up with deep knowledge [52:21] when you build your product when you [52:22] interact with your customers you get [52:24] your feedback uh a lot from AB testing [52:28] and uh [52:30] so yes yes I I I we use extensively uh [52:34] we use stattic extensively for a testing [52:38] so uh these data become very valuable to [52:41] your company and win you time to have [52:45] higher mode so there is no golden mode [52:48] you can have you start you keep building [52:50] mod and people eventually if you if you [52:54] lie down and and rest vest then uh you [52:57] will soon be uh out competed so yes so [53:02] the domain knowledge is a great thing [53:04] for you to survive as a startup thank [53:07] you for the mentioning I do think AB [53:09] testing is about measurement and [53:11] learning and that will become OTE um [53:14] time flies very fast when we talk about [53:16] de technical topics we already over time [53:20] because we want to save some time for [53:21] audience as well actually prepare six [53:23] questions we just got two uh the rest of [53:26] the questions are what are the [53:27] challenges what what is the time the [53:30] total addressable Market of J and what [53:33] role proprietory data play and which [53:36] companies are well positioned to get [53:38] this proprietary data and how like is [53:41] there any effective uh tricks in doing J [53:45] but I guess because we don't have time [53:49] yeah uh we can save that for other days [53:52] if you can interview the guys on our [53:53] social media subscribe to stasic if [53:56] you're interested in this kind of topics [53:59] um and then also buy the future BT the [54:02] future do it now yeah um we're going to [54:05] open to the audience for Q&A just raise [54:08] your hand if you are [54:19] yeah thank you for all the speakers [54:22] amazing discussion um I can start with a [54:25] very non-technical question so in one of [54:28] uh K's video how you foresee or forecast [54:32] the AI we before it hits the sure so uh [54:36] I'm looking for advice or tools or [54:38] resources for normal like Tech workers [54:41] like me uh not to fores but how to keep [54:44] most updated to in this AI environment [54:47] thank [54:48] you but I just give the answer [54:52] already uh I think I think on the most [54:55] serious [54:56] side um that's a very good question I [55:00] think this is something Eugene and I [55:01] actually talked offline and online quite [55:04] a bit that there is so much noise um I [55:09] think this is one of the interesting [55:10] angles I've actually been thinking about [55:12] this for quite a few years in the recent [55:14] past that did the internet help [55:17] us to acquire new knowledges or good [55:21] knowledges or or or not in the first 10 [55:24] years or so the answer to me was is [55:26] obvious right you know it's so easy to [55:28] to get knowledge Google give me access [55:31] to everything and anything on the planet [55:33] in the last few years I'm not sure about [55:35] that um I think when I saying I'm not [55:38] sure I'm not saying myself or urnal I'm [55:40] just saying that in general the [55:42] population do we improve our the the the [55:47] signal to racial uh signal to noise [55:50] Ratio or not it's not very clear to me [55:54] so the actionable thing for everyone [55:57] here including myself is we need to be [55:58] sensitive about how to where to acquire [56:01] the knowledge uh and how to spend time [56:05] and then how to think about this right [56:07] so one thing you know it could be [56:09] controversial um so every time I do an [56:12] interview or not towards the end of the [56:14] question is always often about which [56:16] book do you read and I want to [56:18] recommend um and as if and also there [56:21] there's also implied theory that the [56:24] more books you read it the better I'm [56:26] not sure about that I feel like you know [56:28] today um I feel like I rather read a [56:31] good book three times rather than read a [56:34] three books um it's like a training data [56:37] right you know is that a lot of data the [56:39] more the better or you have high quality [56:42] data I think uh llama 3 has taught you [56:45] know I mean not just llama 3 but you [56:47] know llama 3 T taught us again that the [56:49] high quality data uh makes a lot of [56:52] difference so I close with one thing [56:55] that you and I discussed in uh uh a [56:58] podcast another podcast the video [57:00] interview um before that um whenever [57:05] he's doing a program or am I doing a [57:07] program I almost feel like I wanted to [57:09] do something so that people can listen [57:11] to 10 times and it still feel like [57:13] they're learning I think that's probably [57:15] you know the bar is very high I don't [57:17] know I can do get to that point but [57:19] that's almost the bar I wanted to get to [57:21] and I almost feel like when when we [57:22] acquire knowledge you on the internet [57:25] that's that's something we should should [57:26] be uh thoughtful about it not just the [57:28] the more the marit that's that's one [57:30] thing I've been thinking in the last few [57:32] years you know in some ways internet [57:34] allow us to acquire knowledge too fast [57:36] too easy is that really good thing so [57:39] that's a that's a question for all of us [57:40] to think about but that's a very good [57:43] question um I'd like to basically [57:46] provide some sort of like practical [57:47] answer so uh Twitter for really cool [57:50] Technologies and a little bit of drama [57:52] like go or you know Downtown Abby or How [57:55] I Met Your Mother kind of style uh [57:57] Hacker News for cool technology and Nery [58:01] discussions or things like that uh [58:03] LinkedIn for serious business [58:05] opportunities and stuff like that and [58:07] then of course you know really awesome [58:09] podcasts like these guys right and I [58:11] think that's basically like what I do [58:12] for like collecting information because [58:14] there's just like so much information [58:15] flying around and and then well if you [58:17] find some articles to leond then pull up [58:19] ELO chat and you can just like the [58:20] summarizations but normally those [58:22] websites are actually providing a good [58:23] coverage the reason is I think people [58:26] and platform co-evolve and uh you know [58:28] for Twitter you just can't post some [58:30] serious business stuff but LinkedIn [58:32] people are always you know dressing up [58:34] polished and then publish it right so so [58:36] so so those kind of platforms naturally [58:38] gives you different angles and uh and [58:40] help you filter through a bunch of [58:42] information [58:43] already since you mentioned the Twitter [58:45] Andre capassi actually made a tweet a [58:47] few months back maybe two months ago he [58:50] said something like tweet is very much [58:52] like downloading the file executable on [58:54] the Internet it's clear that's a mware [58:57] or not uh that's that's the very point I [58:59] was trying to make but that but yanin [59:01] said is right in the sense that what [59:03] sort of what kind of information uh [59:06] reasonably popular on which website but [59:08] my early point is more like you know [59:10] when you acquire the knowledge you need [59:12] to be careful right you know is that a [59:15] executable um you know you download it [59:17] from internet to think about it I mean [59:20] you have to be outrageous on Twitter to [59:22] be popular and that's why Twitter is [59:23] like full of outrageous I mean there's [59:25] one very famous uh search Company CEO [59:28] who appears to be a big D but you know [59:31] he's popular right so that's why I mean [59:33] every platform has has biases I I I do I [59:36] do too so yeah right and and and I think [59:39] how we very nicely pointed out that like [59:41] we should undo the biases while we are [59:42] actually consuming those information [59:44] sorry for the comment about that de [59:46] don't tell [59:49] him our next [59:52] question yeah uh first of all I want to [59:55] thank you for all for for sharing your [59:57] expertise and wisdom with us today um [60:00] and second um I think at this time like [60:04] all the companies are trying to find how [60:07] they can use gen to empower uh the [60:10] business to like stay relevant in the [60:13] game um so I want to the setting is like [60:16] this um you have a company you have a [60:19] few pretty good use cases that could [60:21] leverage uh LM um and there's actually [60:24] like proof of conep but there's still [60:27] some Gap to really bring it to like real [60:29] business value and you're trying to [60:31] build a team like more horizontally to [60:34] empower those products um and since LM [60:37] is so different from like traditional ml [60:40] infrastructure I'm curious to like learn [60:43] like what do you think is the key [60:45] components and consideration you think [60:48] you must get right to make that happen [60:50] but I think we mentioned like evaluation [60:53] monitoring cost yeah I just want to hear [60:56] more about [60:59] that maybe I can start because I I buil [61:03] product uh that is so so we don't have [61:07] very deep AI um in terms of uh training [61:10] large models using tens of thousands of [61:13] gpus but uh we still use AI to to build [61:17] a useful product so it's important to [61:21] understand the real needs of your users [61:23] and then harness the power of of AI for [61:26] that I think if you don't understand why [61:30] what what problems or what pain uh pain [61:34] points your customer have it it's it's [61:36] basically trying to find a nail using a [61:39] a flashy new powerful Hammer which you [61:42] don't really uh make any impact so [61:46] finding and and as a lot of people uh [61:50] here are technical background it's even [61:53] more important for you to learn learn [61:56] about the product sense and uh to read [62:00] books like innovator dilemma jobs to be [62:04] done that kind of a uh business [62:05] knowledge and it will be really [62:07] essential for technical people to [62:10] understand customers and find uh [62:13] opportunities for building a product [62:15] then AI comes into hand to you as a new [62:18] technology to build a shiny new hammer [62:21] and the important thing is find the the [62:24] the nails that hasn't been uh hit by [62:26] other old bad [62:29] hamers um to add up on Jay's answer I [62:31] think um from a more practical stand [62:33] Point evaluation is probably the first [62:36] and the foremost the most important [62:37] thing in building like ji use cases [62:40] right because we are collaborating with [62:41] many application side companies I see so [62:44] many companies they don't even [62:45] understand how do they evaluate um their [62:48] I use case and then just jump directly [62:50] into trying all those fancy techniques I [62:53] think and in evaluation the J AG is very [62:56] very difficult right like basically it's [62:59] how it's How we evaluate humans it's [63:01] similar to how we evaluate LM right but [63:03] we need a consistent robust metric to [63:06] see whether we are improving our [63:08] performance or you know deteriorating [63:10] our performance to to to build upon from [63:13] the starting point I think um less [63:16] people realize the importance of [63:18] evaluation than you know what should be [63:21] optim do you do you have any tools or [63:24] vendors you would have recommend you [63:25] feel like a disc compon do some of the [63:28] evaluation much better provide a [63:30] evaluation solution much [63:33] better I [63:35] think it is a framework what I think [63:38] evaluation is a very long you know [63:40] pipeline right I think stasic is one [63:44] component clearly that's a very good [63:47] tool anything else you know on this you [63:49] know in this pip I'm just curious what [63:51] do you see yeah there's no simple answer [63:54] due to the innate complex of LM but um [63:58] so I think the wave of evaluation range [64:00] from more you know static to more [64:03] Dynamic on the most static side we have [64:06] evaluation benchmarks like mlu which is [64:08] literally to let LMS um do multiple [64:11] choice questions right those are very [64:12] straightforward very easy to evaluate [64:15] and uh slightly more complex and more [64:17] Nuance we have like GSM AK where we ask [64:20] LM to solve math problems but the [64:23] solution to a math problem sometimes [64:25] it's not a single number where you can [64:26] part and query it's like a whole [64:28] reasoning process so we need some other [64:31] humans or other AI systems to evaluate [64:34] that and similar we have like how to [64:36] evaluate the code generation ability of [64:39] we need to actually execute the code [64:41] right I think the most nuan scenario is [64:43] like how do we evaluate CHT or how do we [64:46] evaluate a general chat box similar to [64:48] how do we evaluate a human right so um [64:51] at Martian we have a entire Suite of [64:53] toolings and framework to to do those [64:56] kind of complex evaluations that's [64:58] basically a combination of the human um [65:01] annotation machine assist annotation and [65:04] shortly automated evaluation so that's a [65:07] fairly complex [65:09] to um we we do have a extensive need and [65:14] experience in handling fuzzy output so [65:18] it's very interesting that or intuitive [65:20] for us to use AI to evaluate AI so what [65:25] we do is uh so we outpace our [65:29] competitors in terms of the clips [65:31] generate the quality of the clips how [65:33] interesting are they so it's not because [65:36] we have a really magical prompts we use [65:40] gbt 3.5 which is a pretty stupid model [65:44] and it's it's it's cheap it's almost [65:46] like 95% off based on gp4 so the way we [65:51] do is we have many prompts that try to [65:55] clip a video but just not one but at [65:57] least four so they have different ways [65:59] to do it and they try different they try [66:03] really hard and every video we probably [66:06] have like a 100 Clips candidate and how [66:09] do we pick the best is we use another [66:12] set of prompts this set of prompt is uh [66:15] evaluating the quality of these clips [66:18] with multiple dimensions for example are [66:21] they interesting is it the story does it [66:24] start with a very intriguing cook does [66:26] it have defect like sudden ending is it [66:29] a sponsor message these kind of uh uh uh [66:33] scoring and we give these uh scoring uh [66:36] a combination and and finally combine [66:40] them into a final score and then we rank [66:43] the candidates we only pick the 20% so [66:46] in any industrial process for example uh [66:50] if you are working on a farm you have a [66:52] lot of produce like uh strawberry when [66:56] you sell it you sell the good ones to [66:58] Safe Way and then you throw away some of [67:00] the bad ones and the mediocre ones you [67:03] probably send to a a worse uh tier [67:06] grocery so it's the same when when you [67:09] are building a product you want to try [67:11] your you want your system try harder and [67:14] you want to get the good ones and one [67:15] way to do it is using AI to evaluate [67:19] well there is a a caveat there though [67:21] some AIS uh don't catch the trick so I [67:24] have a uh example example on my uh [67:26] Twitter which is J of Opus you that's my [67:30] only interesting post I have is it's an [67:33] AIG generated image of some Craftsman [67:37] and the the person's head and the neck [67:40] are obviously disconnected you can you [67:42] can you can see from the video uh you [67:44] can see from the image but I asked this [67:47] image uh with chat GB with gbd4 with [67:51] Gemini with uh uh an anthropic and all [67:55] of them say oh it's it's a very okay uh [67:58] it's a very okay image uh it make very [68:01] much sense nothing is weird and even [68:04] after you prompt them is there something [68:06] wrong with their head and their neck and [68:09] the the AI will say it's very normal so [68:11] you you can 100% rely on this but it's [68:14] helpful framework to try harder and then [68:18] evaluate using Ai and then um and then [68:21] get the best results nice I like I love [68:25] keep hear in those answers but [68:26] unfortunately time is The Limited [68:28] resource here we have one last question [68:30] so uh thank you for offering this [68:33] opportunity uh I think uh my question is [68:37] a little bit technical I think many [68:39] audience with technical background might [68:40] be interesting uh the thing is I'm [68:43] recently doing a research with professor [68:45] and his collaborate team in AWS on the [68:48] re reliability and observability for [68:52] machine Learning System and this uh my [68:55] question is kind related to hway is [68:57] sharing about QA part you know the QA [69:00] part so uh for example now we know that [69:05] in order to train a model uh efficiency [69:09] we have to train in the dist distributed [69:11] strategies like using the sensor parm or [69:16] pipeline parm but this will has uh we [69:19] have a backfire which like given a fixed [69:23] input like a fixed data and fix [69:26] parameter or fixed weight of the modu [69:28] but with using this distribute uh [69:31] strategy the output of model will be [69:34] huge difference and it's hard for the [69:37] developers to debug within or build this [69:39] and we now uh heard many complaints from [69:43] Amazon's developers they said we cannot [69:46] locate this failure because the you know [69:49] the numeric difference will accumulate [69:52] but during this process there is no such [69:55] a very efficient way to discover it so [69:58] my question is about how to improve the [70:02] liability and [70:04] affability especially in distribute [70:06] system training yeah thank you so much [70:09] yeah man I have so much to say about [70:11] yeah I'm already starting to basically [70:13] first of all this is not new problem uh [70:16] starting from 1970s there is the [70:18] scientific computation or HPC world that [70:20] already knows that when you are actually [70:22] doing numerical computation things are [70:24] just not possible to reproduce right and [70:27] uh I think right now basically I mean [70:29] the AI Community are a little bit me [70:32] included are a little bit of a spoiled [70:34] child and they're not willing to look at [70:37] existing solutions for that there's one [70:39] existing solution which is numerical [70:40] stability right if you add four float [70:43] numbers together if you do 1 two 3 4 or [70:45] 4 32 1 things are different and but you [70:47] can statistically measure the stability [70:49] which is a very important thing in [70:52] numerical computation right if you do [70:54] exponent of something then you want to [70:56] make sure that that something is not [70:58] outrageously large if you do [71:01] x62 you're going to be o anyway right so [71:04] all those kind of things are actually [71:05] already there so I would basically argue [71:07] that um the solutions there we just need [71:10] to basically employ numerical stability [71:12] into it that's one part the second thing [71:14] that you mentioned is basically about [71:16] about uh about interpretability right [71:18] that's also something that's really kind [71:20] of like outrageously academic I would [71:22] say yeah people don't care about [71:24] interpretability yeah and let me give [71:26] you one example right okay so we're all [71:28] comp Vision or things like that explain [71:30] to [71:31] me what a cat [71:34] is in pixel right we forcing computers [71:38] to basically say we need to have a inter [71:41] interpretable computer vision system [71:44] that classifies a cat now was like well [71:47] interpret our own brain they'll [71:49] basically say oh yeah cat has you know [71:51] like pointy ears I was like what is [71:54] pointy right what is a near there are [71:57] something that is [71:59] naturally complex and un like [72:02] uninterpretable and I think that just [72:04] fine in many cases and uh I mean if you [72:08] think about it you're driving the car [72:09] back home today right now we don't know [72:13] if that car is you know like might break [72:15] down or not right maybe there's just a [72:17] fatigue in the in the middle and it [72:19] breaks down right there are things that [72:21] are not interpretable it's just you know [72:23] like as long as we have a more [72:25] statistically safe way to put things in [72:27] a safe box then it's fine and I think [72:30] this is kind of like one fundamental [72:32] looping back to the very first question [72:33] what is changing in the ER of AI versus [72:36] not ai ai is naturally just statistical [72:39] it's not like if you call a function [72:41] does Define API a fixed input fixed [72:43] output no longer this world has changed [72:46] and we just need to accept it and then [72:48] build security fenes around those kind [72:50] things so that statistically or [72:52] basically after this bounds things are [72:54] working properly [72:57] okay thank you so much sorry S I rented [72:58] too much oh no I think with that it's a [73:02] very nice ending and I like the Lear uh [73:04] that the world is CH has changed we just [73:07] need to accept it uh so we have the a [73:10] room until 900m uh feel free to network [73:13] social after the talk uh with that let's [73:17] give it up to our speakers [73:21] [Applause] [73:26] all right thank you everyone thank you [73:28] thank you [74:24] go [74:54] e [75:23] e for [75:28] [Music] [76:06] spe [76:48] spe [77:10] spe [77:53] spe for